[OpenIndiana-discuss] Zpool replacing forever

[Andrew Gabriel]

On 20/01/2019 15:11, Gary Mills wrote:
> The status of rpool is like this:
>
>              NAME                STATE     READ WRITE CKSUM
>              rpool               DEGRADED    23     0    15
>                mirror-0          DEGRADED    46     0    30
>                  replacing-0     DEGRADED     0     0    20
>                    c5t0d0s0/old  FAULTED      0     0     0  corrupted data
>                    c5t0d0s0      ONLINE       0     0    20
>                  c5t1d0s0        ONLINE       0     0    76
>      
>      errors: 5 data errors, use '-v' for a list
>
> To get here, I initially had a single SSD as rpool.  It had started
> developing errors.  The number of errors was increasing.  That was
> c5t0d0s0 .  I installed a second SSD, c5t1d0s0, and added it to rpool,
> making it a mirror.  The resilver copied all of the data, including
> the errors.
>
> Then, I removed the bad SSD and replaced it with another new one, at
> the same device name.  I started the replacement with the command:
>
>      # zpool replace -f rpool c5t0d0s0
>
> That started another resilver, as I expected.  However, it didn't
> remove the old SSD from the mirror.  As far as I can tell, it did
> copy all of the data.  What do I do now?
>
> All five of the data errors are either in old snapshots or old BEs.
> I can destroy them, if that will help.

You will need to delete the corrupt files (and snapshots if the corrupt 
files are in any) and then run a scrub so zfs knows there is no more 
corrupt data in the zpool.

zfs is keeping the old disk around as a ghost, in case you can put it 
back in and zfs can find good copies of the corrupt data on it during 
the resilver. It will stay in the zpool until there's a clean scrub, 
after which zfs will collapse out the replacing-0 vdev. (In your case, 
you know there is no copy of the bad data so this won't help, but in 
general it can.)

So to fix, delete the corrupt files, and any snapshots they're in.
Then run a scrub.
When the scrub finishes, ZFS will collapse out the replacing-0 vdev.

-- 
Andrew Gabriel