[OpenIndiana-discuss] Understanding O_DSYNC and zfs_nocacheflush on ZFS

[Yi Zhang]

Just want to add that I know O_DSYNC and directio (or disabling FS
caching) are two different things. The reason why I wanted to play
with O_DSYNC is that a) doesn't seem to give the desired directio
result (running the experiment on UFS with directio took 2s).

On Sat, Feb 5, 2011 at 10:20 AM, Yi Zhang <yizhang84 at gmail.com> wrote:
> Hi all,
>
> I'm trying to achieve the same effect of UFS directio on ZFS and here
> is what I did:
>
> 1. Set the primarycache of zfs to metadata and secondarycache to none,
> recordsize to 8K (to match the unit size of writes)
> 2. Run my test program (code below) with different options and measure
> the running time.
> a) open the file without O_DSYNC flag: 0.11s
> b) open the file with O_DSYNC flag: 147.26s
> c) same as b) but also enabled zfs_nocacheflush: 5.87s
>
> My questions are:
> 1. With my primarycache and secondarycache settings, the FS shouldn't
> buffer reads and writes anymore. Wouldn't that be equivalent to
> O_DSYNC? Why a) and b) are so different?
> 2. My understanding is that zfs_nocacheflush essentially removes the
> sync command sent to the device, which cancels the O_DSYNC flag. Why
> b) and c) are so different?
> 3. Does ZIL have anything to do with these results?
>
> Thanks in advance for any suggestion/insight!
> Yi
>
>
> #include <fcntl.h>
> #include <sys/time.h>
>
> int main(int argc, char **argv)
> {
>    struct timeval tim;
>    gettimeofday(&tim, NULL);
>    double t1 = tim.tv_sec + tim.tv_usec/1000000.0;
>    char a[8192];
>    int fd = open(argv[1], O_RDWR|O_CREAT|O_TRUNC, 0660);
>    //int fd = open(argv[1], O_RDWR|O_CREAT|O_TRUNC|O_DSYNC, 0660);
>    if (argv[2][0] == '1')
>        directio(fd, DIRECTIO_ON);
>    int i;
>    for (i=0; i<10000; ++i)
>        pwrite(fd, a, sizeof(a), i*8192);
>    close(fd);
>    gettimeofday(&tim, NULL);
>    double t2 = tim.tv_sec + tim.tv_usec/1000000.0;
>    printf("%f\n", t2-t1);
> }
>