[OpenIndiana-discuss] ISCSI SSD Performance

[Jim Klimov]

On 2013-02-18 05:18, Grant Albitz wrote:
> I would like to discuss one more item:
>
> Based on the writes below vs the reads it seems like I am able to get more data out of a w/s as apposed to a read per second. I may just be misunderstanding the results but the disks themselves are rated for higher read performance then write. I am just wondering if there is a zfs tunable I can play with. The disks seem have much more queued up during writes then reads as well.


Well, earlier I wrote that writing zeroes (as in "dd if=/dev/zero")
may be interpreted by ZFS differently that nay other block. Upon more
thought, many modern SSDs include their own compression algorithms,
often not tunable. Hence your writes of zeroes might return as soon
as they hit the SSD's cache and are ignored in terms of heavy IO.

I'd suggest you try a large blob of data like concatenation of several
gigabytes of movie files (preferably different in order to rule out
compression) or of your typical data - VM images, database files, etc.

Speaking of the latter, your tests might be more meaningful on a later
stage, if you also ran them with ZFS recordsize/volblocksize that would
be typical for your real expected workload (i.e. 8-16-64kb typical for
DB and for VM storage) - you'd have higher metadata storage overheads
and lower RMW costs for random writes.

As for writes diminishing as you add drives... I don't have a consistent
theory, but it likely has to do with HDD-oriented HBA controller chip
being overwhelmed with SSD's "unexpected" random write performance.
Typically one HBA is recommended per 8 SSDs max, especially if you
don't use IT mode but fake RAID hacks which need more work from circuit
logic. Also, the PCI buses may become contended which is somewhat
proven by steady read speeds. If your motherboard (chipset) does have
several independent buses, you may roughly double performance by putting
different HBAs on different ones - or even by making sure that your
networking used for storage networking and the one HBA in your build
are on different buses. See diagrams in your MoBo documentation, that
is all usually drawn out.

Good luck,
//Jim