[OpenIndiana-discuss] Recommendations for fast storage

[Jan Owoc]

On Wed, Apr 17, 2013 at 12:57 PM, Timothy Coalson <tsc5yc at mst.edu> wrote:
> On Wed, Apr 17, 2013 at 7:38 AM, Edward Ned Harvey (openindiana) <
> openindiana at nedharvey.com> wrote:
>
>> You also said the raidz2 will offer more protection against failure,
>> because you can survive any two disk failures (but no more.)  I would argue
>> this is incorrect (I've done the probability analysis before).  Mostly
>> because the resilver time in the mirror configuration is 8x to 16x faster
>> (there's 1/8 as much data to resilver, and IOPS is limited by a single
>> disk, not the "worst" of several disks, which introduces another factor up
>> to 2x, increasing the 8x as high as 16x), so the smaller resilver window
>> means lower probability of "concurrent" failures on the critical vdev.
>>  We're talking about 12 hours versus 1 week, actual result of my machines
>> in production.
>>
>
> Did you also compare the probability of bit errors causing data loss
> without a complete pool failure?  2-way mirrors, when one device completely
> dies, have no redundancy on that data, and the copy that remains must be
> perfect or some data will be lost.  On the other hand, raid-z2 will still
> have available redundancy, allowing every single block to have a bad read
> on any single component disk, without losing data.  I haven't done the math
> on this, but I seem to recall some papers claiming that this is the more
> likely route to lost data on modern disks, by comparing bit error rate and
> capacity.  Of course, a second outright failure puts raid-z2 in a much
> worse boat than 2-way mirrors, which is a reason for raid-z3, but this may
> already be a less likely case.

Richard Elling wrote a blog post about "mean time to data loss" [1]. A
few years later he graphed out a few cases for typical values of
resilver times [2].

[1] https://blogs.oracle.com/relling/entry/a_story_of_two_mttdl
[2] http://blog.richardelling.com/2010/02/zfs-data-protection-comparison.html

Cheers,
Jan