[OpenIndiana-discuss] Recommendations for fast storage

[Timothy Coalson]

On Wed, Apr 17, 2013 at 7:38 AM, Edward Ned Harvey (openindiana) <
openindiana at nedharvey.com> wrote:

> You also said the raidz2 will offer more protection against failure,
> because you can survive any two disk failures (but no more.)  I would argue
> this is incorrect (I've done the probability analysis before).  Mostly
> because the resilver time in the mirror configuration is 8x to 16x faster
> (there's 1/8 as much data to resilver, and IOPS is limited by a single
> disk, not the "worst" of several disks, which introduces another factor up
> to 2x, increasing the 8x as high as 16x), so the smaller resilver window
> means lower probability of "concurrent" failures on the critical vdev.
>  We're talking about 12 hours versus 1 week, actual result of my machines
> in production.
>

Did you also compare the probability of bit errors causing data loss
without a complete pool failure?  2-way mirrors, when one device completely
dies, have no redundancy on that data, and the copy that remains must be
perfect or some data will be lost.  On the other hand, raid-z2 will still
have available redundancy, allowing every single block to have a bad read
on any single component disk, without losing data.  I haven't done the math
on this, but I seem to recall some papers claiming that this is the more
likely route to lost data on modern disks, by comparing bit error rate and
capacity.  Of course, a second outright failure puts raid-z2 in a much
worse boat than 2-way mirrors, which is a reason for raid-z3, but this may
already be a less likely case.

Also, as for time to resilver, I'm guessing that depends largely on where
bottlenecks are (it has to read effectively all of the remaining disks in
the vdev either way, but can do so in parallel, so ideally it could be the
same speed), and how busy the pool is with other things - how does the
bandwidth sharing during resilver work?

Tim