[OpenIndiana-discuss] zfs corruption

[Scott LeFevre]

File this under "heal thy self".  I discovered a way to fix my problem
without destroying the entire pool and reloading from a backup and I'm
posting here so it may help someone else in the future.

On my tank1 pool that was showing corruption, I had about 15% capacity
left.  I created a new dataset/filesystem named tank1/media2 and slowly
moved everything from tank1/media in 10-12% chunks over several days.
Once everything was moved over, I deleted tank1/media and renamed
tank1/media2 to tank1/media and re-established my shares.  At this
point, the pools showed that it was still corrupted but it no longer
showed the name of the dataset/filesystem.  After my regularly schedule
scrub ran without error, the tank1 pool corruption disappeared and show
its clean!

As a side bonus with some forward planning, I was able to better
compress my data and better distribute the data across all drives which
gave me better throughput.
I hope this is helpful to someone in the future.

Scott LeFevre


On Tue, 2012-02-21 at 18:59 -0500, Scott LeFevre wrote:

> I'm fairly new to openindiana and zfs.  I've been running it for a year
> or more with a lot of success.  I'm currently running oi 151a.  I have a
> server setup w/ 9 drives in a single zpool at version 28.  Last week, I
> added a ssd drive and partitioned it for a zpool log and cache.  Over
> the weekend, the server locked up and I had to power cycle the box.
> When it powered up, it wouldn't mount one of the datasets under tank1
> and wouldn't go any further.  I booted to the oi_151a cd image, imported
> the tank1 pool where it showed the ssd log partition had 8 crc errors.
> Using zpool status -v tank1, it also showed the pool online but 
> errors: Permanent errors have been detected in the following files:
> 
>         tank1/media:<0x0>
> 
> I cleared the device errors with zpool clear tank1 and exported the pool
> tank1.  When I rebooted to my oi 151a installation, I imported the pool
> and spot checked the file system which seems to be fine.  
> 
> I then tried to remove the zil partition using zpool remove tank
> c2t6d0p0.  The command completes but doesn't remove the device nor does
> it give an errors as to why.  I've tried offlining and detaching the
> drive and the removing with no luck.  I ran a scrub over the weekend but
> it found no errors.
> 
> So, I'm stuck with a zpool that looks acts like its fine but shows
> errors and a zil that I don't fully trust that I can't remove.  I'm open
> to suggestions on how to clean up the error and remove the zil.  I've
> searched the web for similar errors and it seems the general consensus
> is to back, destroy, create, and restore the pool.  That's all well and
> fine but I don't have that option at the moment.  Here's what the pool
> looks like today.
> 
> # zpool status -v tank1
>   pool: tank1
>  state: ONLINE
> status: One or more devices has experienced an error resulting in data
> 	corruption.  Applications may be affected.
> action: Restore the file in question if possible.  Otherwise restore the
> 	entire pool from backup.
>    see: http://www.sun.com/msg/ZFS-8000-8A
>   scan: scrub repaired 0 in 12h55m with 0 errors on Mon Feb 20 23:41:17
> 2012
> config:
> 
> 	NAME        STATE     READ WRITE CKSUM
> 	tank1       ONLINE       0     0     0
> 	  raidz1-0  ONLINE       0     0     0
> 	    c2t1d0  ONLINE       0     0     0
> 	    c1t1d0  ONLINE       0     0     0
> 	    c1t0d0  ONLINE       0     0     0
> 	    c1t3d0  ONLINE       0     0     0
> 	    c1t4d0  ONLINE       0     0     0
> 	    c1t5d0  ONLINE       0     0     0
> 	  raidz1-1  ONLINE       0     0     0
> 	    c2t0d0  ONLINE       0     0     0
> 	    c1t2d0  ONLINE       0     0     0
> 	    c2t2d0  ONLINE       0     0     0
> 	logs
> 	  c2t6d0p0  ONLINE       0     0     0
> 	cache
> 	  c2t6d0p1  ONLINE       0     0     0
> 
> errors: Permanent errors have been detected in the following files:
> 
>         tank1/media:<0x0>
> 
> 
> Scott LeFevre
> 
> _______________________________________________
> OpenIndiana-discuss mailing list
> OpenIndiana-discuss at openindiana.org
> http://openindiana.org/mailman/listinfo/openindiana-discuss