[OpenIndiana-discuss] spool replace says the disk has a different sector alignment

[Francis Swasey]

On Nov 19, 2013, at 2:30 PM, Francis Swasey <Frank.Swasey at uvm.edu> wrote:

> On Nov 19, 2013, at 1:50 PM, Stefan Müller-Wilken <stefan.mueller-wilken at acando.de> wrote:
> 
>> Hi there,
>> 
>> have you looked at http://wiki.illumos.org/display/illumos/ZFS+and+Advanced+Format+disks ? Or quoting http://thr3ads.net/zfs-discuss/2012/09/2101915-cannot-replace-X-with-Y-devices-have-different-sector-alignment as  another source, you could also try fdisk to compare your two devices:
> 
> Yes, I have looked at the first one.  and the fdisk -G shows me:
> 
> root at bujbod1:~/blocksize# fdisk -G /dev/rdsk/c5t0d0
> * Physical geometry for device /dev/rdsk/c5t0d0
> * PCYL     NCYL     ACYL     BCYL     NHEAD NSECT SECSIZ
>  60788    60788    0        0        255   504   512   
> root at bujbod1:~/blocksize# fdisk -G /dev/rdsk/c5t1d0
> * Physical geometry for device /dev/rdsk/c5t1d0
> * PCYL     NCYL     ACYL     BCYL     NHEAD NSECT SECSIZ
>  60788    60788    0        0        255   504   512   
> 
> And it is the c5t0d0 that it can't replace.
> 
> What I have discovered since my first email is I have all these messages in /var/adm/messages for all the disks in this zpool:
> 
> Nov 19 14:24:33 bujbod1 zfs: [ID 447730 kern.warning] WARNING: Disk, '/dev/dsk/c5t10d0s0', has a block alignment that is larger than the pool's alignment
> 
> I'm guessing that the firmware upgrade I applied to this system a month ago changed something critical to ZFS on the IBM M5120 (rebranded LSI) raid controller.


Reading further into http://blog.delphix.com/gwilson/2012/11/15/4k-sectors-and-zfs/, I get to the zdb command...

root at bujbod1:~# zdb -l /dev/dsk/c5t0d0s0 | grep ashift
        ashift: 12
        ashift: 12
        ashift: 12
        ashift: 12
root at bujbod1:~# zdb -l /dev/dsk/c5t1d0s0 | grep ashift
        ashift: 9
        ashift: 9
        ashift: 9
        ashift: 9

And here is the issue... Whatever got changed in the M5120's firmware now makes zfs want to use ashift=12 instead of ashift=9 (which it did when the pool was created).  

Suggestions for how I get myself out of this mess?

Thanks,
  Frank