[OpenIndiana-discuss] Recovering from a controller failure

[Matt Boswell]

Hi all,

We had one of our SAS expander cards in a storage system running OpenIndiana go bad last night.  We replaced it, but in the process, we had to move the RAID controller to a new slot on the motherboard.  This caused all devices to have a new designation; i.e. c2t10d0 became c5t10d0.

Our pools are online with no data loss, thankfully, but one is degraded because it is hanging on to a couple of devices with the old controller designation.  When I try to replace the "old" device with the "new" one, I get an error that the "new" device belongs to the pool already.  I think zfs is reading metadata from the device but doesn't realize that the two pools are one and the same.

root at hermes:~# zpool status zfs1
  pool: zfs1
state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: scrub repaired 0 in 89h58m with 0 errors on Fri Jan  2 17:58:21 2015
config:

        NAME           STATE     READ WRITE CKSUM
        zfs1           DEGRADED     0     0     0
          mirror-0     DEGRADED     0     0     0
            c5t11d0    ONLINE       0     0     0
            c2t10d0    UNAVAIL      0     0     0  cannot open
          mirror-1     DEGRADED     0     0     0
            c5t13d0    ONLINE       0     0     0
            c2t5d0     UNAVAIL      0     0     0  cannot open
          mirror-2     DEGRADED     0     0     0
            c2t3d0     UNAVAIL      0     0     0  cannot open
            c5t16d0    ONLINE       0     0     0
          mirror-3     ONLINE       0     0     0
            c5t17d0    ONLINE       0     0     0
            c5t18d0    ONLINE       0     0     0
        logs
          mirror-4     ONLINE       0     0     0
            c5t20d0s1  ONLINE       0     0     0
            c5t21d0s1  ONLINE       0     0     0
        spares
          c2t14d0      UNAVAIL   cannot open
          c2t12d0      UNAVAIL   cannot open

errors: No known data errors
root at hermes:~# zpool replace zfs1 c2t10d0 c5t10d0
invalid vdev specification
use '-f' to override the following errors:
/dev/dsk/c5t10d0s0 is part of active ZFS pool zfs1. Please see zpool(1M).


When I try with '-f' I get the same thing.

I've tried offlining c2t10d0 and then onlining c5t10d0 but I get a message that there is no such device in the pool.  Any way to do this or do I just have to format the devices and then add them to the degraded pool and wait for a full resilver?

M

________________________________
The information contained in this email may be confidential and/or may be covered under the Privacy Act, 5 USC 552(a), and/or the Health Insurance Portability and Accountability Act (PL 104-191) and its various implementing regulations and must be protected in accordance with those provisions.. It has been sent for the sole use of the intended recipient(s). If the reader of this message is not an intended recipient, you are hereby notified that any unauthorized review, use, disclosure, dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please contact the sender by reply email and destroy all copies of the original message. To contact our email administrator directly, send an email message to helpdesk at medsphere.com. Thank you.