[OpenIndiana-discuss] NAS R/W performance puzzle
bfriesen at simple.dallas.tx.us
Mon Mar 24 19:16:56 UTC 2014
On Mon, 24 Mar 2014, Reginald Beardsley wrote:
> My apologies, I thought the references to /dev/zero, /dev/null and bs=65536 made it clear I was using dd.
At least for tests on the server side.
> I increased the file size to 65 GB and reran the tests. It turns
> out the read operation wall time can vary by several minutes. For
> the 6.5 GB test that makes the thoughput rather low. On the 65 GB
> tests I got 42-59 MB/s read for 10 trials. This is pretty much what
> I expect from gigabit using a cheap SOHO switch w/ no other traffic.
The thing about this performance test approach is that if your server
only has 4 GB of RAM and the file size is 65 GB then you are
preventing the zfs ARC from adding much value on the server side other
than to buffer up individual full zfs blocks. You are essentially
testing what happens when your server is running out of memory and
not testing the normal case.
If the zfs blocksize is 128k, then the 64k dd blocksize is resulting
in two reads per zfs block. This can potentially incur more latency,
except for the caching (and possibly read-ahead) in the ARC. The NFS
client negotiates the maximum block size to use (could be 1 MB in some
modern implementations). Hopefully it is using the same block size as
'dd' is requesting rather than a smaller size since then additional
latencies would be incurred. Some other NFS clients use much smaller
block sizes by default and require tunables to use larger ones.
I suggest that you tell dd to use the same block size as the
underlying zfs filesystem. This will usually produce the best
performance in such benchmarking.
The largest influence of block size will be when writing on top of an
existing huge file. There will be considerably less performance when
writing using 64k blocks on top of 128k zfs blocks because of the
shortage of memory (whole file is not already cached) and ZFS's COW
algorithm (needs to read the underlying zfs block before it can write
a new one).
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
More information about the OpenIndiana-discuss