[OpenIndiana-discuss] zpool replace says the disk has a different sector alignment
Timothy Coalson
tsc5yc at mst.edu
Wed Nov 20 00:57:43 UTC 2013
On Tue, Nov 19, 2013 at 6:18 PM, Frank Swasey <Frank.Swasey at uvm.edu> wrote:
> Today at 5:31pm, Timothy Coalson wrote:
>
> I would hope that attaching one new drive wouldn't make the controller
>> suddenly decide all other disks also had 4k sectors - he said he was
>> getting "larger block size" for all existing disks in the pool.
>>
>
> No, these are Seagate ST4000NM0023 drives, they've always really been 4K
> drives - but the controller used to report they were 512 byte sector
> drives. My problem is I don't have a spare system to move this data to
> (and this is the backup system, so no destroy it and recover from backups
> path). But, clearly, I either have to find a way to have the controller
> pretend these are 512 byte sector drives again, or bite that bullet and
> rebuild.
>
Is it possible to reflash the controller to the older firmware that
reported them as 512? Not the first thing I'd want to try, but...
>
>
>> If it turns out that they had 4k physical blocks all along, it may help
>> performance to rebuild the pool. I'm unsure whether you can use sd.conf
>> to
>> make a 4k disk pretend to be a 512 disk (I have only used it to do the
>> opposite), but if rebuilding the pool isn't an option, or you just want to
>> get it back to full redundancy before trying additional things, that might
>> be something to try.
>>
>
> I tried repeatedly to change the blocksize via sd.conf to pretend they are
> 512 byte sector disks this afternoon and I have not succeeded yet. What I
> have right now is:
>
> sd-config-list =
> "IBM ServeRAID M5120 ", "physical-block-size:512";
>
That looks correct to me, comparing to what I have in mine (though I don't
have identification strings with trailing spaces).
> Either I'm doing it wrong or that parameter is not allowed to be less than
> the reported physical block size.
>
Found one post I half-remembered over on illumos zfs that says the sd
driver doesn't allow that:
http://www.listbox.com/member/archive/182191/entry/19:299/20130326114258:D38D379E-962B-11E2-AF17-BE9C6E434AE2/
Thanks,
> Frank
>
>
>> Tim
>>
>>
>>
>> On Tue, Nov 19, 2013 at 4:48 PM, Reginald Beardsley <pulaskite at yahoo.com
>> >wrote:
>>
>> It's probably the new drive, NOT the controller. I ran into this when I
>>> RMAed a 3 TB HGST drive. The replacement had 4k sectors instead of the
>>> 512
>>> byte sectors of the failed drive. HGST was unable to supply a 512 sector
>>> disk.
>>>
>>> Unfortunately, as far as I know you'll have to rebuild the entire pool.
>>> I'd suggest buying new 4k sector disks for the new pool and keeping the
>>> old ones as spares for pools built with 512 sector disks.
>>>
>>> You may need to add entries in sd.conf to notify the driver that the
>>> drives are 4k. That seems to depend upon what the drives report to the
>>> driver. Plan on this being tedious and time consuming. You might get
>>> lucky, but it's best to be prepared. I suggest being especially vigilant
>>> about backups until you've got it resolved. Make sure you don't
>>> overwrite
>>> good ones with bad ones.
>>>
>>> I *think* I wrote up everything I learned on the page Klimov created, but
>>> don't hesitate to ask if you have questions. If you have to do the
>>> sd.conf
>>> bit make sure you read George Wilson's page linked from Klimov's page.
>>>
>>> Good luck,
>>> Reg
>>>
>>> --------------------------------------------
>>> On Tue, 11/19/13, Francis Swasey <Frank.Swasey at uvm.edu> wrote:
>>>
>>> Subject: Re: [OpenIndiana-discuss] spool replace says the disk has a
>>> different sector alignment
>>> To: "Discussion list for OpenIndiana" <
>>> openindiana-discuss at openindiana.org>
>>> Date: Tuesday, November 19, 2013, 1:46 PM
>>>
>>>
>>> On Nov 19, 2013, at 2:30 PM, Francis Swasey <Frank.Swasey at uvm.edu>
>>> wrote:
>>>
>>> > On Nov 19, 2013, at 1:50 PM, Stefan Müller-Wilken
>>> <stefan.mueller-wilken at acando.de>
>>> wrote:
>>> >
>>> >> Hi there,
>>> >>
>>> >> have you looked at
>>> http://wiki.illumos.org/display/illumos/ZFS+and+Advanced+Format+disks
>>> ? Or quoting
>>> http://thr3ads.net/zfs-discuss/2012/09/2101915-cannot-replace-X-with-Y-
>>> devices-have-different-sector-alignment
>>> as another source, you could also try fdisk to compare
>>> your two devices:
>>> >
>>> > Yes, I have looked at the first one. and the
>>> fdisk -G shows me:
>>> >
>>> > root at bujbod1:~/blocksize# fdisk -G /dev/rdsk/c5t0d0
>>> > * Physical geometry for device /dev/rdsk/c5t0d0
>>> > * PCYL NCYL
>>> ACYL BCYL
>>> NHEAD NSECT SECSIZ
>>> > 60788 60788 0
>>> 0
>>> 255 504 512
>>> > root at bujbod1:~/blocksize# fdisk -G /dev/rdsk/c5t1d0
>>> > * Physical geometry for device /dev/rdsk/c5t1d0
>>> > * PCYL NCYL
>>> ACYL BCYL
>>> NHEAD NSECT SECSIZ
>>> > 60788 60788 0
>>> 0
>>> 255 504 512
>>> >
>>> > And it is the c5t0d0 that it can't replace.
>>> >
>>> > What I have discovered since my first email is I have
>>> all these messages in /var/adm/messages for all the disks in
>>> this zpool:
>>> >
>>> > Nov 19 14:24:33 bujbod1 zfs: [ID 447730 kern.warning]
>>> WARNING: Disk, '/dev/dsk/c5t10d0s0', has a block alignment
>>> that is larger than the pool's alignment
>>> >
>>> > I'm guessing that the firmware upgrade I applied to
>>> this system a month ago changed something critical to ZFS on
>>> the IBM M5120 (rebranded LSI) raid controller.
>>>
>>>
>>> Reading further into
>>> http://blog.delphix.com/gwilson/2012/11/15/4k-sectors-and-zfs/,
>>> I get to the zdb command...
>>>
>>> root at bujbod1:~# zdb -l /dev/dsk/c5t0d0s0 | grep ashift
>>> ashift: 12
>>> ashift: 12
>>> ashift: 12
>>> ashift: 12
>>> root at bujbod1:~# zdb -l /dev/dsk/c5t1d0s0 | grep ashift
>>> ashift: 9
>>> ashift: 9
>>> ashift: 9
>>> ashift: 9
>>>
>>> And here is the issue... Whatever got changed in the M5120's
>>> firmware now makes zfs want to use ashift=12 instead of
>>> ashift=9 (which it did when the pool was created).
>>>
>>> Suggestions for how I get myself out of this mess?
>>>
>>> Thanks,
>>> Frank
>>> _______________________________________________
>>> OpenIndiana-discuss mailing list
>>> OpenIndiana-discuss at openindiana.org
>>> http://openindiana.org/mailman/listinfo/openindiana-discuss
>>>
>>>
>>> _______________________________________________
>>> OpenIndiana-discuss mailing list
>>> OpenIndiana-discuss at openindiana.org
>>> http://openindiana.org/mailman/listinfo/openindiana-discuss
>>>
>>> _______________________________________________
>> OpenIndiana-discuss mailing list
>> OpenIndiana-discuss at openindiana.org
>> http://openindiana.org/mailman/listinfo/openindiana-discuss
>>
>>
> --
> Frank Swasey | http://www.uvm.edu/~fcs
> Sr Systems Administrator | Always remember: You are UNIQUE,
> University of Vermont | just like everyone else.
> "I am not young enough to know everything." - Oscar Wilde (1854-1900)
> _______________________________________________
> OpenIndiana-discuss mailing list
> OpenIndiana-discuss at openindiana.org
> http://openindiana.org/mailman/listinfo/openindiana-discuss
>
>
More information about the OpenIndiana-discuss
mailing list