[OpenIndiana-discuss] Recommendations for fast storage

[Jay Heyl]

On Tue, Apr 16, 2013 at 2:25 PM, Timothy Coalson <tsc5yc at mst.edu> wrote:

> On Tue, Apr 16, 2013 at 3:48 PM, Jay Heyl <jay at frelled.us> wrote:
>
> > My question about the rationale behind the suggestion of mirrored SSD
> > arrays was really meant to be more in relation to the question from the
> OP.
> > I don't see how mirrored arrays of SSDs would be effective in his
> > situation.
> >
>
> There is another detail here to keep in mind: ZFS checks checksums on every
> read from storage, and with raid-zn used with block sizes that give it more
> capacity than mirroring (that is, data blocks are large enough that they
> get split across multiple data sectors and therefore devices, instead of
> degenerate single data sector plus parity sector(s) - OP mentioned 32K
> blocks, so they should get split), this means each random filesystem read
> that isn't cached hits a large number of devices in a raid-zn vdev, but
> only one device in a mirror vdev (unless ZFS splits these reads across
> mirrors, but even then it is still fewer devices hit).  If you are limited
> by IOPS of the devices, then this could make raid-zn slower.
>

I'm getting a sense of comparing apples to oranges here, but I do see your
point about the raid-zn always requiring reads from more devices due to the
parity. OTOH, it was my impression that read operations on n-way mirrors
are always issued to each of the 'n' mirrors. Just for the sake of
argument, let's say we need room for 1TB of storage. For raid-z2 we use
4x500GB devices. For the mirrored setup we have two mirrors each with
2x500GB devices. Reads to the raid-z2 system will hit four devices. If my
assumption is correct, reads to the mirrored system will also hit four
devices. If we go to a 3-way mirror, reads would hit six devices.

In all but degenerate cases, mirrored arrangements are going to include
more drives for the same amount of usable storage, so it seems they should
result in more devices being hit for both read and write. Or am I wrong
about reads being issued in parallel to all the mirrors in the array?