[OpenIndiana-discuss] spool replace says the disk has a different sector alignment

Tue Nov 19 23:34:02 UTC 2013

Wups, I meant that the page may help querying the reported block sizes on
openindiana - I have used it successfully on 151a7 by building from the
source file, I didn't try the binary.

Tim

On Tue, Nov 19, 2013 at 5:31 PM, Timothy Coalson <tsc5yc at mst.edu> wrote:

> I would hope that attaching one new drive wouldn't make the controller
> suddenly decide all other disks also had 4k sectors - he said he was
> getting "larger block size" for all existing disks in the pool.
>
> Perhaps it would help to get an answer from the disks about their reported
> block sizes: hook one of each model in question up to a pure HBA (or, if
> applicable, a native SATA port in AHCI mode) and query the physical/logical
> block sizes.  This may help for doing so on solaris:
>
>
> http://solaris.kuehnke.de/archives/18-Checking-physical-sector-size-of-disks-on-Solaris.html
>
> If it turns out that they had 4k physical blocks all along, it may help
> performance to rebuild the pool.  I'm unsure whether you can use sd.conf to
> make a 4k disk pretend to be a 512 disk (I have only used it to do the
> opposite), but if rebuilding the pool isn't an option, or you just want to
> get it back to full redundancy before trying additional things, that might
> be something to try.
>
> Tim
>
>
>
> On Tue, Nov 19, 2013 at 4:48 PM, Reginald Beardsley <pulaskite at yahoo.com>wrote:
>
>> It's probably the new drive, NOT the controller.  I ran into this when I
>> RMAed a 3 TB HGST drive. The replacement had 4k sectors instead of the 512
>> byte sectors of the failed drive.  HGST was unable to supply a 512 sector
>> disk.
>>
>> Unfortunately, as far as I know you'll have to rebuild the entire pool.
>>  I'd suggest buying new 4k sector disks for the new pool and keeping the
>> old ones as spares for pools built with 512 sector disks.
>>
>> You may need to add entries in sd.conf to notify the driver that the
>> drives are 4k.  That seems to depend upon what the drives report to the
>> driver.  Plan on this being tedious and time consuming.  You might get
>> lucky, but it's best to be prepared.  I suggest being especially vigilant
>> about backups until you've got it resolved.  Make sure you don't overwrite
>> good ones with bad ones.
>>
>> I *think* I wrote up everything I learned on the page Klimov created, but
>> don't hesitate to ask if you have questions.  If you have to do the sd.conf
>> bit make sure you read George Wilson's page linked from Klimov's page.
>>
>> Good luck,
>> Reg
>>
>> --------------------------------------------
>> On Tue, 11/19/13, Francis Swasey <Frank.Swasey at uvm.edu> wrote:
>>
>>  Subject: Re: [OpenIndiana-discuss] spool replace says the disk has a
>> different sector alignment
>>  To: "Discussion list for OpenIndiana" <
>> openindiana-discuss at openindiana.org>
>>  Date: Tuesday, November 19, 2013, 1:46 PM
>>
>>
>>  On Nov 19, 2013, at 2:30 PM, Francis Swasey <Frank.Swasey at uvm.edu>
>>  wrote:
>>
>>  > On Nov 19, 2013, at 1:50 PM, Stefan Müller-Wilken
>>  <stefan.mueller-wilken at acando.de>
>>  wrote:
>>  >
>>  >> Hi there,
>>  >>
>>  >> have you looked at
>> http://wiki.illumos.org/display/illumos/ZFS+and+Advanced+Format+disks
>>  ? Or quoting
>> http://thr3ads.net/zfs-discuss/2012/09/2101915-cannot-replace-X-with-Y-devices-have-different-sector-alignment
>>  as  another source, you could also try fdisk to compare
>>  your two devices:
>>  >
>>  > Yes, I have looked at the first one.  and the
>>  fdisk -G shows me:
>>  >
>>  > root at bujbod1:~/blocksize# fdisk -G /dev/rdsk/c5t0d0
>>  > * Physical geometry for device /dev/rdsk/c5t0d0
>>  > * PCYL     NCYL
>>     ACYL     BCYL
>>     NHEAD NSECT SECSIZ
>>  >  60788    60788    0
>>        0
>>  255   504   512
>>  > root at bujbod1:~/blocksize# fdisk -G /dev/rdsk/c5t1d0
>>  > * Physical geometry for device /dev/rdsk/c5t1d0
>>  > * PCYL     NCYL
>>     ACYL     BCYL
>>     NHEAD NSECT SECSIZ
>>  >  60788    60788    0
>>        0
>>  255   504   512
>>  >
>>  > And it is the c5t0d0 that it can't replace.
>>  >
>>  > What I have discovered since my first email is I have
>>  all these messages in /var/adm/messages for all the disks in
>>  this zpool:
>>  >
>>  > Nov 19 14:24:33 bujbod1 zfs: [ID 447730 kern.warning]
>>  WARNING: Disk, '/dev/dsk/c5t10d0s0', has a block alignment
>>  that is larger than the pool's alignment
>>  >
>>  > I'm guessing that the firmware upgrade I applied to
>>  this system a month ago changed something critical to ZFS on
>>  the IBM M5120 (rebranded LSI) raid controller.
>>
>>
>>  Reading further into
>> http://blog.delphix.com/gwilson/2012/11/15/4k-sectors-and-zfs/,
>>  I get to the zdb command...
>>
>>  root at bujbod1:~# zdb -l /dev/dsk/c5t0d0s0 | grep ashift
>>          ashift: 12
>>          ashift: 12
>>          ashift: 12
>>          ashift: 12
>>  root at bujbod1:~# zdb -l /dev/dsk/c5t1d0s0 | grep ashift
>>          ashift: 9
>>          ashift: 9
>>          ashift: 9
>>          ashift: 9
>>
>>  And here is the issue... Whatever got changed in the M5120's
>>  firmware now makes zfs want to use ashift=12 instead of
>>  ashift=9 (which it did when the pool was created).
>>
>>  Suggestions for how I get myself out of this mess?
>>
>>  Thanks,
>>    Frank
>>  _______________________________________________
>>  OpenIndiana-discuss mailing list
>>  OpenIndiana-discuss at openindiana.org
>>  http://openindiana.org/mailman/listinfo/openindiana-discuss
>>
>>
>> _______________________________________________
>> OpenIndiana-discuss mailing list
>> OpenIndiana-discuss at openindiana.org
>> http://openindiana.org/mailman/listinfo/openindiana-discuss
>>
>
>