[OpenIndiana-discuss] Replacing both disks in a mirror set

Tue Oct 9 00:44:11 UTC 2012

Good point on split vs detach. Unfortunately this particular misinformation
seems widespread :( 

-----Original Message-----
From: Richard Elling [mailto:richard.elling at richardelling.com] 
Sent: Monday, October 08, 2012 8:39 PM
To: Discussion list for OpenIndiana
Subject: Re: [OpenIndiana-discuss] Replacing both disks in a mirror set

On Oct 8, 2012, at 4:07 PM, Martin Bochnig <martin at martux.org> wrote:

> Marilio,
> 
> 
> at first a reminder: never ever detach a disk before you have a third 
> disk that already completed resilvering.
> The term "detach" is misleading, because it detaches the disk from the 
> pool. Afterwards you cannot access the disk's previous contents 
> anymore. Your "detached" half of a mirror can neither be imported, nor 
> mounted and also not even rescued (unlike a disk with a "zpool 
> destroy"ed disk). If I ever mentally recover from a zfs encryption 
> caused 2TB (or 3 years!) data loss, then I may offer an implementation 
> with less ambigous naming to Illumos.
> 
> 
> "zpool detach" suggests, that you could still use this disk as a 
> reserve backup copy of the pool you were detaching it from.

No it doesn't -- there is no documentation that suggests this usage.

> And that
> you could simply "zpool attach" it again, in case the other disk would 
> die.

You are confusing zpool detach and zpool split commands.
 -- richard

> 
> Unfortunately, this is not the case.
> Well, you can of course attach it again. Like any new or empty disk.
> But only if and only if you have enough replicas, and that's not what 
> one wanted if one fell in this misunderstanding trap.
> And there are no warnings in the zpool/zfs man pages.
> 
> 
> What you want:
> 
> zpool replace <poolname> <vdev to be replaced> <new vdev> But last 
> weekend I lost 7 years of trust that I had in ZFS.
> Because Oracle Solaris 11/11 x86 with an encrypted and gzip-9 
> compressed mirror cannot be accessed anymore after VirtualBox forced 
> me to remove prower from the host machine.
> Since then a 1:1 mirror of 2TB disks cannot be mounted anymore. It 
> always ends in a kernel panic due to a pf in 
> aes:aes_decrypt_contiguous_blocks.
> 
> Well: TITANIC IS UNSINKABLE!
> The problem is, that scrub doesn't find an error, and so has nothing 
> to auto-repair.
> Even zpool attach sucessfully completes resilver, but the newly 
> resilvered disk contains the same error. Be aware that ZFS is not free 
> of bugs.
> If it stays like that (I contacted some folks for help), then my trust 
> in ZFS has destroyed, VAPORIZED 3 years of my work and life.
> 
> So, back to your question: To be as cautious as possible, what I would 
> do in your case:
> 
> 
> 0.)  zpool offline <poolname> <vdev you want to replace>
> 
> 1.) Physically remove this disc (important, because I have seen cases, 
> where zfs forgets that you offlined a vdev after a reboot)
> 
> 2.) AFTER (!IMPORTANT!) you physically disconnected the disc to be 
> replaced, "zpool detach it" or alternatively take "zpool replace 
> <poolname> 
> <oldvdev_that_you_disconnected_BEFOREinordertokeepitasbafailsafebackup
> !>
> <newvdev>
> 
> 3.) Depending on if you did detach or replace in step 2.), "zpool 
> attach <poolname> <Firstvdevofthispool> <newvdev>  or ommit this step, 
> if you took "zpool replace" in step 2.)
> 
> 
> NEVER TRUST ZFS TOO MUCH.
> What I do from now on: For each 1:1 mirror that I have I will take a 
> third disk, resilver it, offline and physically disconnect it, and 
> store it at a secure place.
> 
> Because if you have this much bad luck as I had last weekend, ZFS 
> replicates the data corruption, too.
> And then you could have 1000 discs mirrored, they would all contain 
> the corruption.
> For this reason, you are only on the safe side, if you physically 
> disconnect a third copy!
> 
> 
> 
> Good luck!
> %martin
> 
> 
> 
> 
> 
> 
> On 10/8/12, Maurilio Longo <maurilio.longo at libero.it> wrote:
>> Dan Swartzendruber wrote:
>>> I'm not understanding your problem.  If you add a 3rd temporary 
>>> disk, wait for it to resilver, then replace c1t5d0, let the new disk 
>>> resilver, then detach the temporary disk, you will never have less 
>>> than 2 up to date disks in the mirror. What am I missing?
>>> 
>> 
>> Dan,
>> 
>> you're right, I was trying to find a way to "move" the new disk in 
>> the failing disk bay instead of simply replacing the failing one :)
>> 
>> Thanks for the advice!
>> 
>> Maurilio.
>> 
>> --
>> __________
>> |  |  | |__| Maurilio Longo
>> |_|_|_|____|
>> 
>> 
>> 
>> _______________________________________________
>> OpenIndiana-discuss mailing list
>> OpenIndiana-discuss at openindiana.org
>> http://openindiana.org/mailman/listinfo/openindiana-discuss
>> 
> 
> 
> --
> regards
> 
> %martin bochnig
>  http://wiki.openindiana.org/oi/MartUX_OpenIndiana+oi_151a+SPARC+LiveDVD
>    http://www.youtube.com/user/MartUXopensolaris
>
http://www.facebook.com/pages/MartUX_SPARC-OpenIndiana/357912020962940
>        https://twitter.com/MartinBochnig
>          http://www.martux.org (new page not yet online, but pretty 
> soon)
> 
> _______________________________________________
> OpenIndiana-discuss mailing list
> OpenIndiana-discuss at openindiana.org
> http://openindiana.org/mailman/listinfo/openindiana-discuss

--

Richard.Elling at RichardElling.com
+1-760-896-4422

_______________________________________________
OpenIndiana-discuss mailing list
OpenIndiana-discuss at openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss