[OpenIndiana-discuss] summary: in desperate need of fsck.zfs

[Ray Arachelian]

So, I found a way to mount the zpool's volume read only.

The problem was with the cached view of the zpools here:
/etc/zfs/zpool.cache

Deleting that file and rebooting the host got rid of the locks on the
zpool - doing this however means all your zpools will be offline except
the rpool, until you manually import them.  Once you do, all the error
counts vanish, though it does keep track of the last scrub status.

When I tried to import the pool in r/w mode, it hung - maybe it would
have finished, but I gave up waiting after 10 minutes, so mounted it
read only with: zpool import -F -f -o readonly=on data  and that
returned immediately and I was able to see the files.

I then reran my rsync script and I see it hung the box in the same spot,
so that file has a bad block or bad metadata.  I then powered off the
JBOD and rebooted, removed teh zpool cache file, rebooted again, and
turned on the job. 

Next, I created an exception file list and added it to the rsync command
via the --exclude-from=FILE option and am running it again.  If that was
the only bad file, I'm fine with that as it was an old backup of a
vmware guest that I have elsewhere.  (I'm running rsync inside a script
with nohup, and doing tail -f nohup,out elsewhere, so if it hangs again
on some other file, I can just add that file to the exclude list and
restart the process.)

Hopefully I'll be able to pull most of the data off the drive.

I suspect that maybe either the JBOD, or the USB driver doesn't proper
error handling, or handles errors in such a way that the whole JBOD
stops responding to OI.  Might be a driver issue, might be crappy
firmware in the JBOD, might be a lame USB port on this motherboard,
etc.  The zpool handler notices the I/O errors and marks the drives I/O
error count in the pool cache file, which then prevents all zpool access
because it thinks its unsafe to touch.  Removing the file removes this
lock...

Hopefully switching to eSATA might be better (or maybe it won't work at
all depending on the port multiplier this JBOD uses and the OI drivers,
etc.)