[OpenIndiana-discuss] Only one SSD detected on expander
Scott Marcy
oi at mscott.org
Wed Oct 17 13:30:48 UTC 2012
So one more data point that seems to suggest that I'm getting duplicate names from these drives. When I attach at least one of the Samsung SSDs to each expander (front and back), I get this:
$ cfgadm -al
Ap_Id Type Receptacle Occupant Condition
c6 scsi-sas connected configured unknown
c6::es/ses0 ESI connected configured unknown
c6::smp/expd0 smp connected configured unknown
c6::w5002538043584d30,0 disk-path connected configured unknown <-- SSD
c6::w5000c500513a0efe,0 disk-path connected configured unknown
c6::w5000c5005168b2f0,0 disk-path connected configured unknown
c6::w5000c5005169a612,0 disk-path connected configured unknown
c6::w5000c5005169cc6a,0 disk-path connected configured unknown
c6::w5000c500515199bd,0 disk-path connected configured unknown
c6::w5000c500517042d9,0 disk-path connected configured unknown
c8 scsi-sas connected configured unknown
c8::es/ses1 ESI connected configured unknown
c8::smp/expd1 smp connected configured unknown
c8::w5002538043584d30,0 disk-path connected configured unknown <-- SSD
c9 scsi-sas connected unconfigured unknown
sata0/0::dsk/c4t0d0 disk connected configured ok
sata0/1::dsk/c4t1d0 disk connected configured ok
sata0/2 sata-port empty unconfigured ok
sata0/3 sata-port empty unconfigured ok
sata0/4 sata-port empty unconfigured ok
sata0/5 sata-port empty unconfigured ok
Note that both c6 and c8 have the exact same name for these drives. I *CAN* access the two drives separately, but why on earth am I getting these duplicates? This means I can only reliably access one Samsung SSD per expander.
Thanks.
-Scott
On Oct 16, 2012, at 7:29 PM, Scott Marcy <oi at mscott.org> wrote:
> Hi all,
>
> I'm new to the list and OI, but I've been running a Solaris box with some version of ZFS on it for a couple years. I recently acquired a Supermicro 6047R-E1R36N. It has an X9DRi-LN4F+ motherboard, dual E5-2620s and 48GB of RAM. I soon learned that I needed a different HBA (it comes with an LSI 2108-based card that doesn't have any JBOD passthrough mode), so I replaced that with an LSI SAS9211-8i, flashed with the latest "IT" firmware.
>
> Being new to SAS expanders (which this server has two of), I quickly learned that some older 2TB drives I had on my older Solaris box simply will not work reliably on the expander. (They would very quickly lock up the entire file system once the load got heavy.) However, I've had much better luck (so far) with some newer 3TB SATA drives (Seagate ST3000DM001). I'm still not convinced that these are going to be reliable in the long run, having now read about a few SATA-on-SAS expander horror stories (and my own experience with the 2TB drives), but this isn't a production system (yet) so I'm willing to see how it goes.
>
> But the one problem I can't seem to figure out are the Samsung 830 SSD drives I have for ZIL and ARC. I have 2 mirrored 128GB 830s as the boot pool, but they're on the built-in SATA ports, not the expander ports. Then I have four other 128GB 830s which I installed into 3.5" hot-swap adapters and these are the ones causing problems. The HBA sees the drives just fine—they're listed during POST and I can access them via the card's built-in BIOS. But once OI (151a5) is up and running, it only sees ONE of the 830s. And here's where it gets weird: if I remove all but one of the SSDs from the expander ports, it will see whichever one SSD is connected—it does not matter which port the drives are on. Also, regardless of the port, the attachment path always seems to be the same in the OS. It's like the OS thinks all four drives are the exact same physical drive.
>
> $ cfgadm -al
> Ap_Id Type Receptacle Occupant Condition
> c6 scsi-sas connected configured unknown
> c6::es/ses0 ESI connected configured unknown
> c6::smp/expd0 smp connected configured unknown
> c6::w5002538043584d30,0 disk-path connected configured unknown <-- this is the SSD
> c6::w5000c500513a0efe,0 disk-path connected configured unknown
> c6::w5000c5005168b2f0,0 disk-path connected configured unknown
> c6::w5000c5005169a612,0 disk-path connected configured unknown
> c6::w5000c5005169cc6a,0 disk-path connected configured unknown
> c6::w5000c500515199bd,0 disk-path connected configured unknown
> c6::w5000c500517042d9,0 disk-path connected configured unknown
> c8 scsi-sas connected configured unknown
> c8::es/ses1 ESI connected configured unknown
> c8::smp/expd1 smp connected configured unknown
> c9 scsi-sas connected unconfigured unknown
> sata0/0::dsk/c4t0d0 disk connected configured ok
> sata0/1::dsk/c4t1d0 disk connected configured ok
> sata0/2 sata-port empty unconfigured ok
> sata0/3 sata-port empty unconfigured ok
> sata0/4 sata-port empty unconfigured ok
> sata0/5 sata-port empty unconfigured ok
>
>
> $ format
> Searching for disks...done
>
>
> AVAILABLE DISK SELECTIONS:
> 0. c4t0d0 <ATA-SAMSUNGSSD830-3B1Q cyl 19932 alt 2 hd 224 sec 56>
> /pci at 0,0/pci15d9,626 at 1f,2/disk at 0,0
> 1. c4t1d0 <ATA-SAMSUNGSSD830-3B1Q cyl 19932 alt 2 hd 224 sec 56>
> /pci at 0,0/pci15d9,626 at 1f,2/disk at 1,0
> 2. c7t5000C500513A0EFEd0 <ATA-ST3000DM001-9YN1-CC4B-2.73TB>
> /scsi_vhci/disk at g5000c500513a0efe
> 3. c7t5000C5005168B2F0d0 <ATA-ST3000DM001-9YN1-CC4B-2.73TB>
> /scsi_vhci/disk at g5000c5005168b2f0
> 4. c7t5000C5005169A612d0 <ATA-ST3000DM001-9YN1-CC4B-2.73TB>
> /scsi_vhci/disk at g5000c5005169a612
> 5. c7t5000C5005169CC6Ad0 <ATA-ST3000DM001-9YN1-CC4B-2.73TB>
> /scsi_vhci/disk at g5000c5005169cc6a
> 6. c7t5000C500515199BDd0 <ATA-ST3000DM001-9YN1-CC4B-2.73TB>
> /scsi_vhci/disk at g5000c500515199bd
> 7. c7t5000C500517042D9d0 <ATA-ST3000DM001-9YN1-CC4B-2.73TB>
> /scsi_vhci/disk at g5000c500517042d9
> 8. c7t5002538043584D30d0 <ATA-SAMSUNG SSD 830-3B1Q-119.24GB>
> /scsi_vhci/disk at g5002538043584d30
>
> This last one is the SSD. If I remove all the SSDs from the expander ports, disk #8 does not show up (nor does it show in cfgadm). If I put one SSD into any expander port, disk 8 is back. If I add additional SSDs, I still only have disk 8. If I remove the first SSD (while others are still attached), disk 8 still appears. In all cases, I can run a read test on the disk (format -> analyze -> test) without any errors, so I think it's actually accessing one of the physical drives.
>
> How do I get OI to see the other 3 SSDs?
>
> Any pointers are greatly appreciated. Thanks!
>
> -Scott
>
>
> _______________________________________________
> OpenIndiana-discuss mailing list
> OpenIndiana-discuss at openindiana.org
> http://openindiana.org/mailman/listinfo/openindiana-discuss
More information about the OpenIndiana-discuss
mailing list