[OpenIndiana-discuss] in desperate need of fsck.zfs

[Richard Elling]

On Jul 24, 2012, at 2:04 PM, Ray Arachelian wrote:

> On 07/24/2012 02:41 PM, Jim Klimov wrote:
>> 
>> Did you try to "zpool import -o readonly=on" and using TXG rollback?
> Can't do that now, the pool is actually imported, and it won't let me
> export it, nor offline it.
> 
>> If your drives wrote crap during disconnection, it is possible that
>> some previous ZFS transaction is (more or less) consistent on-disk
>> and was not overwritten.
> 
> I suspect this might be the case, as the weird behavior survived a few
> reboots.  It's as if it's imported half way and then some flag gets set
> saying it's not safe to touch the zpool, so everything fails after that
> with an I/O error - but at the low level, I can access it just fine via
> dd, and to some extend the zdb command.  (Although zdb core dumps with
> out of memory.  Not sure if it's because of the bad metadata, or if the
> the box is too low on RAM.)

zdb is not intended to be run against a live file system. But it also can
be caught by a massive corruption in the metadata tree, which is what
it appears that you have. metadata is already double redundant, by
default, so the corruption runs deep.

>> 
>> You might want to back up ZFS labels of each disk somewhere, because
>> a rollback import clears knowledge of TXGs past some number (i.e.
>> CurrentTXG-10); it is possible you'd want to recover that label and
>> try a different offset into the past.
> 
> I've found something interesting here: 
> http://sigtar.com/2009/10/19/opensolaris-zfs-recovery-after-kernel-panic/
> I plan on booting off the live ISO off another machine that has more
> memory, and attempting it when I get home.  (I can remotely access the
> machine, but there's no ILO, so can't force a power off, insert a boot
> DVD, etc.)
> 
> How do I backup/restore zfs labels?

Your zfs labels are ok, otherwise you could not import the pool.
 -- richard

--
ZFS Performance and Training
Richard.Elling at RichardElling.com
+1-760-896-4422