[OpenIndiana-discuss] Migrating ZFS RAIDZ1 volume to larger disks

[Reginald Beardsley]

I decided it was time to replace the disks in my Solaris 10 u8 system.  The setup is a bit unusual, so please bear with me.

The setup is an HP Z400 with three 1 TB disks with SMI labels.  Slice s0 on each disk is 100 GB and s1 is the rest of the disk.  The s0 slices form a mirrored rpool.  The s1 slices form a RAIDZ1 pool.

This has worked just fine for 3 years.  But an attractive price on 2 TB disks and the age of the disks led me to decide to replace the drives.

I have built 3 systems with this configuration, Solaris 10 u8, OI 151_a8 and Hipster 2017.10.  It is necessary to relabel the initial install disk when creating this configuration  because of the behavior of the installer, but aside from tedium, easy to do.

After backing up the mirrored root pool and the RAIDZ1 export pool, I shut down, pulled one of the disks, replaced it and attempted to boot.  This produced a "bad PBR" message from the HP Z400 BIOS.  So I tried changing a different disk.  This led to an "unable to find boot pool" message.  

I replaced the original disk, booted and removed a disk, replaced it with another disk.  Now my troubles start.  Zpool steadfastly refuses to see the replacement disk.  It appears in the "format -e" selection list with the same designation as the original device, but attempts to attach or replace the device fail with a "no such device" error.

On the mirrored pool I tried:

zpool replace <pool>  <dev>

zpool detach <pool> <dev.
zpool attach <pool> <dev. <dev>

I did not try to migrate the RAIDZ1 slice.

I *think* the HP BIOS is causing the problem, but that's just because the BIOS always demands I hit [F1] when any device has been changed.

My expectation is that I would have simply replaced a drive, resilvered, replaced another drive, resilvered, etc until all 3 drives had been replaced.  But I cannot get it to boot in a degraded state.  Rather clearly not all  of the disks on the mirror has a GRUB boot, but the "unable to find pool" message suggests there is another issue.  I'm hoping that resolving that will let me boot in a degraded state with the BIOS having done its perverse bit of twiddle.  All  three disks have identical labels so far as I can determine using format(1).

Because of personal UI preferences updating from u8 does not appear to be an option.  I run twm which seems to be quite impossible to do on OI because of all the dependencies without having to thrash my way through the jungle of starting X11.  I did that as part of building  X11R3  at work  25 years ago before all the vendors supplied X11 and have no desire to repeat the experience.  CDE just made things worse. Aside from twm, my only requirement is the Sun/Forte development tools.  I've used the debuggers on all the major workstations (and lots of obscure systems) and nothing comes close to the functionality of the CLI dbx on Solaris.  I *really* like "print sin(a)*tan(b)*exp(c)" actually working.

Reg