[OpenIndiana-discuss] Understanding O_DSYNC and zfs_nocacheflush on ZFS

[Yi Zhang]

Hi all,

I'm trying to achieve the same effect of UFS directio on ZFS and here
is what I did:

1. Set the primarycache of zfs to metadata and secondarycache to none,
recordsize to 8K (to match the unit size of writes)
2. Run my test program (code below) with different options and measure
the running time.
a) open the file without O_DSYNC flag: 0.11s
b) open the file with O_DSYNC flag: 147.26s
c) same as b) but also enabled zfs_nocacheflush: 5.87s

My questions are:
1. With my primarycache and secondarycache settings, the FS shouldn't
buffer reads and writes anymore. Wouldn't that be equivalent to
O_DSYNC? Why a) and b) are so different?
2. My understanding is that zfs_nocacheflush essentially removes the
sync command sent to the device, which cancels the O_DSYNC flag. Why
b) and c) are so different?
3. Does ZIL have anything to do with these results?

Thanks in advance for any suggestion/insight!
Yi


#include <fcntl.h>
#include <sys/time.h>

int main(int argc, char **argv)
{
    struct timeval tim;
    gettimeofday(&tim, NULL);
    double t1 = tim.tv_sec + tim.tv_usec/1000000.0;
    char a[8192];
    int fd = open(argv[1], O_RDWR|O_CREAT|O_TRUNC, 0660);
    //int fd = open(argv[1], O_RDWR|O_CREAT|O_TRUNC|O_DSYNC, 0660);
    if (argv[2][0] == '1')
        directio(fd, DIRECTIO_ON);
    int i;
    for (i=0; i<10000; ++i)
        pwrite(fd, a, sizeof(a), i*8192);
    close(fd);
    gettimeofday(&tim, NULL);
    double t2 = tim.tv_sec + tim.tv_usec/1000000.0;
    printf("%f\n", t2-t1);
}