[OpenIndiana-discuss] NAS R/W performance puzzle

[Bob Friesenhahn]

On Mon, 24 Mar 2014, Reginald Beardsley wrote:

> My apologies, I thought the references to /dev/zero, /dev/null and bs=65536 made it clear I was using dd.

At least for tests on the server side.

> I increased the file size to 65 GB and reran the tests.  It turns 
> out the read operation wall time can vary by several minutes.  For 
> the 6.5 GB test that makes the thoughput rather low.  On the 65 GB 
> tests I got 42-59 MB/s read for 10 trials.  This is pretty much what 
> I expect from gigabit using a cheap SOHO switch w/ no other traffic.

The thing about this performance test approach is that if your server 
only has 4 GB of RAM and the file size is 65 GB then you are 
preventing the zfs ARC from adding much value on the server side other 
than to buffer up individual full zfs blocks.  You are essentially 
testing what happens when your server is running out of memory and 
not testing the normal case.

If the zfs blocksize is 128k, then the 64k dd blocksize is resulting 
in two reads per zfs block.  This can potentially incur more latency, 
except for the caching (and possibly read-ahead) in the ARC.  The NFS 
client negotiates the maximum block size to use (could be 1 MB in some 
modern implementations).  Hopefully it is using the same block size as 
'dd' is requesting rather than a smaller size since then additional 
latencies would be incurred.  Some other NFS clients use much smaller 
block sizes by default and require tunables to use larger ones.

I suggest that you tell dd to use the same block size as the 
underlying zfs filesystem.  This will usually produce the best 
performance in such benchmarking.

The largest influence of block size will be when writing on top of an 
existing huge file.  There will be considerably less performance when 
writing using 64k blocks on top of 128k zfs blocks because of the 
shortage of memory (whole file is not already cached) and ZFS's COW 
algorithm (needs to read the underlying zfs block before it can write 
a new one).

Bob
-- 
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/