[OpenIndiana-discuss] Buggy mptsas or multipath or something ...

[Mark]

I have spent more than a week battling my first sas2 storage build.

It may be a bug somewhere, but I'm not sure what to point a finger at
but I can reproduce it.

The goal is about 50Tb of usable storage built mainly with Supermicro 
kit, with a supported controller. I have build a number of these with 3G 
sas components and, with the right disks, they work great.

Apart from the "standard" issues with sata disks and expanders, and 
working out what to avoid, things have been pretty good on sas 
generation 1, but things have moved on to generation 2.

It's been deja vu for an IT fossil like me, as it's like going back to 
the bad old days of fibre channel, where vendors all built gear to the 
same standard, but none of it would talk to anyone else's.

The first encounter was this nice "vision multiplier", 4 devices for the 
price of one...   http://kb.lsi.com/KnowledgebaseArticle16414.aspx

Having overcome that hurdle, it was on to the next.

With any of my 2Tb Sas disks installed, the oi_147 text installer simply 
exited to a locked up state when enumerating the disks.
So did Solaris 11 text installer, so it is not just oi.

Dropping to the shell first and checking out what disks were visible was 
revealing.
cfgadm only showed the two for the os.
format saw all four, well sort of.

AVAILABLE DISK SELECTIONS:
	0. c1t5000C5003434E723d0 <SEAGATE-ST32000444SS-0006 cyl 60798 alt 2 hd 
255 sec 252>
	/scsi_vhci/disk/@g5000c5003434e723

Now that's weird, this is multipath.

But neither of the os disk's are !!
and they are connected to the same controller and backplane.

2. c2t5000C5000A85BA89d0 <IBM-ESXS-ST373455SS-BA2D cyl 72629 alt 2 hd 2 
sec 897>
	/pci at 0,0/pci8086,4027&7/pci8086,3500&0/pci,3510&0/pci1000,3020 at 0/iport&f0/disk at w5000c5000a85ba89,0

I know that sas2 allows multilane operation between the controller and 
expanders if the vendor chooses to implement it, so multipath may be 
needed as it is enabled with the installer mpt_sas.conf, but something 
isn't quite right here.

With only the two os disk plugged in, the installation completed 
normally, as did the reboot.
I then disabled the multipath option from /kernel/drv/mpt_sas.conf and 
rebooted and things still worked, so I took a deep breath and plugged in 
the rest of the disks.
No crash, and a big bunch of "normal" disks appeared.

I disabled the multipath as on one of the many "loops" over several 
days, I had done this and crashed the server when the disks were added
with multipath left at the installer default of enabled.

check cfgadm - all listed the same as the OS, and also good in format.

I should also add I started this install with two controllers intending 
to use multipath, but having had so much "fun" even getting to this 
point, it will stay single path until I have a spare month or two to 
take it further.

Hardware:
SM 847E26 chassis, X7DWN+ m/b and 2 x LSI 9211-8i fw P9(patched) in IR 
mode, 30 x ST32000444SS.

So when is multipath the wrong path ?
When the installer takes a $ each way.

And I don't even have two controllers installed, but there "may" be more 
than one path, if you are the right device that is.

I conclude the mptsas driver will probably require more work as vendor 
uptake on the many new sas2 features available appear.

Mark.