[OpenIndiana-discuss] Only one SSD detected on expander

[Scott Marcy]

Hi all,

I'm new to the list and OI, but I've been running a Solaris box with some version of ZFS on it for a couple years. I recently acquired a Supermicro 6047R-E1R36N. It has an X9DRi-LN4F+ motherboard, dual E5-2620s and 48GB of RAM. I soon learned that I needed a different HBA (it comes with an LSI 2108-based card that doesn't have any JBOD passthrough mode), so I replaced that with an LSI SAS9211-8i, flashed with the latest "IT" firmware.

Being new to SAS expanders (which this server has two of), I quickly learned that some older 2TB drives I had on my older Solaris box simply will not work reliably on the expander. (They would very quickly lock up the entire file system once the load got heavy.) However, I've had much better luck (so far) with some newer 3TB SATA drives (Seagate ST3000DM001). I'm still not convinced that these are going to be reliable in the long run, having now read about a few SATA-on-SAS expander horror stories (and my own experience with the 2TB drives), but this isn't a production system (yet) so I'm willing to see how it goes.

But the one problem I can't seem to figure out are the Samsung 830 SSD drives I have for ZIL and ARC. I have 2 mirrored 128GB 830s as the boot pool, but they're on the built-in SATA ports, not the expander ports. Then I have four other 128GB 830s which I installed into 3.5" hot-swap adapters and these are the ones causing problems. The HBA sees the drives just fine—they're listed during POST and I can access them via the card's built-in BIOS. But once OI (151a5) is up and running, it only sees ONE of the 830s. And here's where it gets weird: if I remove all but one of the SSDs from the expander ports, it will see whichever one SSD is connected—it does not matter which port the drives are on. Also, regardless of the port, the attachment path always seems to be the same in the OS. It's like the OS thinks all four drives are the exact same physical drive.

$ cfgadm -al
Ap_Id                          Type         Receptacle   Occupant     Condition
c6                             scsi-sas     connected    configured   unknown
c6::es/ses0                    ESI          connected    configured   unknown
c6::smp/expd0                  smp          connected    configured   unknown
c6::w5002538043584d30,0        disk-path    connected    configured   unknown  <-- this is the SSD
c6::w5000c500513a0efe,0        disk-path    connected    configured   unknown
c6::w5000c5005168b2f0,0        disk-path    connected    configured   unknown
c6::w5000c5005169a612,0        disk-path    connected    configured   unknown
c6::w5000c5005169cc6a,0        disk-path    connected    configured   unknown
c6::w5000c500515199bd,0        disk-path    connected    configured   unknown
c6::w5000c500517042d9,0        disk-path    connected    configured   unknown
c8                             scsi-sas     connected    configured   unknown
c8::es/ses1                    ESI          connected    configured   unknown
c8::smp/expd1                  smp          connected    configured   unknown
c9                             scsi-sas     connected    unconfigured unknown
sata0/0::dsk/c4t0d0            disk         connected    configured   ok
sata0/1::dsk/c4t1d0            disk         connected    configured   ok
sata0/2                        sata-port    empty        unconfigured ok
sata0/3                        sata-port    empty        unconfigured ok
sata0/4                        sata-port    empty        unconfigured ok
sata0/5                        sata-port    empty        unconfigured ok


$ format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c4t0d0 <ATA-SAMSUNGSSD830-3B1Q cyl 19932 alt 2 hd 224 sec 56>
          /pci at 0,0/pci15d9,626 at 1f,2/disk at 0,0
       1. c4t1d0 <ATA-SAMSUNGSSD830-3B1Q cyl 19932 alt 2 hd 224 sec 56>
          /pci at 0,0/pci15d9,626 at 1f,2/disk at 1,0
       2. c7t5000C500513A0EFEd0 <ATA-ST3000DM001-9YN1-CC4B-2.73TB>
          /scsi_vhci/disk at g5000c500513a0efe
       3. c7t5000C5005168B2F0d0 <ATA-ST3000DM001-9YN1-CC4B-2.73TB>
          /scsi_vhci/disk at g5000c5005168b2f0
       4. c7t5000C5005169A612d0 <ATA-ST3000DM001-9YN1-CC4B-2.73TB>
          /scsi_vhci/disk at g5000c5005169a612
       5. c7t5000C5005169CC6Ad0 <ATA-ST3000DM001-9YN1-CC4B-2.73TB>
          /scsi_vhci/disk at g5000c5005169cc6a
       6. c7t5000C500515199BDd0 <ATA-ST3000DM001-9YN1-CC4B-2.73TB>
          /scsi_vhci/disk at g5000c500515199bd
       7. c7t5000C500517042D9d0 <ATA-ST3000DM001-9YN1-CC4B-2.73TB>
          /scsi_vhci/disk at g5000c500517042d9
       8. c7t5002538043584D30d0 <ATA-SAMSUNG SSD 830-3B1Q-119.24GB>
          /scsi_vhci/disk at g5002538043584d30

This last one is the SSD. If I remove all the SSDs from the expander ports, disk #8 does not show up (nor does it show in cfgadm). If I put one SSD into any expander port, disk 8 is back. If I add additional SSDs, I still only have disk 8. If I remove the first SSD (while others are still attached), disk 8 still appears. In all cases, I can run a read test on the disk (format -> analyze -> test) without any errors, so I think it's actually accessing one of the physical drives.

How do I get OI to see the other 3 SSDs?

Any pointers are greatly appreciated. Thanks!

-Scott