[OpenIndiana-discuss] disconnected drives, how to avoid in the future?

[Geoff Nordli]

Running OI151a on a Supermicro X8DTH-6F board, which has an  LSI 2008 8-Port 
6Gbps SAS controller, with 8 SAS internal drives, plus an SSD for the boot 
disk.   I am running the mpt_sas driver.  
    
The server became unresponsive to any commands (couldn't even do a remote 
reboot).  I did a hard reset on server, which seems to have resolved the 
issues (scrub showed no errors and iostat is OK). 

My question, any ideas why this happened and if there is something I can do to 
avoid this issue in the future.  

some errors in the /var/adm/messages:

Jan  9 09:58:50 ml-oi1 scsi: [ID 107833 kern.warning] WARNING: 
/pci at 0,0/pci8086,3410 at 9/pci15d9,400 at 0 (mpt_sas0):
Jan  9 09:58:50 ml-oi1  Disconnected command timeout for Target 16

iostat -En showed lots of different errors (mostly "No Device").

c4t0d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: INTEL SSDSA2CT04 Revision: 0362 Serial No: 
CVPR136307X1040 
Size: 40.02GB <40020664320 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 10 Predictive Failure Analysis: 0 
c4t2d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ASUS     Product: DRW-24B1ST   a   Revision: 1.04 Serial No:  
Size: 0.00GB <0 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 2 Predictive Failure Analysis: 0 
c2t5000C50033F5BF9Fd0 Soft Errors: 0 Hard Errors: 5 Transport Errors: 0 
Vendor: SEAGATE  Product: ST1000NM0001     Revision: 0001 Serial No: 
Z1N00DS70000S12 
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 5 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 
c2t5000C50033F5BD7Fd0 Soft Errors: 0 Hard Errors: 4 Transport Errors: 0 
Vendor: SEAGATE  Product: ST1000NM0001     Revision: 0001 Serial No: 
Z1N00DSE0000S11 
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 4 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 
c2t5000C50033F5BFFBd0 Soft Errors: 0 Hard Errors: 4 Transport Errors: 0 
Vendor: SEAGATE  Product: ST1000NM0001     Revision: 0001 Serial No: 
Z1N00J5Z0000S12 
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 4 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 
c2t5000C50033F5BE3Bd0 Soft Errors: 0 Hard Errors: 6 Transport Errors: 0 
Vendor: SEAGATE  Product: ST1000NM0001     Revision: 0001 Serial No: 
Z1N00DRT0000S12 
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 6 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 
c2t5000C50033F5C5E7d0 Soft Errors: 0 Hard Errors: 5 Transport Errors: 0 
Vendor: SEAGATE  Product: ST1000NM0001     Revision: 0001 Serial No: 
Z1N00J680000S12 
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 5 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 
c2t5000C50033F5D607d0 Soft Errors: 0 Hard Errors: 10 Transport Errors: 0 
Vendor: SEAGATE  Product: ST1000NM0001     Revision: 0001 Serial No: 
Z1N00E2W0000S12 
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 10 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 
c2t5000C50033F5C16Fd0 Soft Errors: 0 Hard Errors: 2 Transport Errors: 0 
Vendor: SEAGATE  Product: ST1000NM0001     Revision: 0001 Serial No: 
Z1N00DTC0000S12 
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 2 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 
c2t5000C50033F5BFBBd0 Soft Errors: 0 Hard Errors: 597 Transport Errors: 0 
Vendor: SEAGATE  Product: ST1000NM0001     Revision: 0001 Serial No: 
Z1N00J310000S12 
Size: 1000.20GB <1000204886016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 597 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 


Here is an example from an fmdump -eV:

Jan 09 2012 10:04:31.334477741 ereport.io.scsi.cmd.disk.dev.rqs.derr
nvlist version: 0
        class = ereport.io.scsi.cmd.disk.dev.rqs.derr
        ena = 0xc3ca9ccb73e00c01
        detector = (embedded nvlist)
        nvlist version: 0
                version = 0x0
                scheme = dev
                device-path = 
/pci at 0,0/pci8086,3410 at 9/pci15d9,400 at 0/iport at 80/disk at w5000c50033f5bfb9,0
                devid = id1,sd at n5000c50033f5bfbb
        (end detector)

        devid = id1,sd at n5000c50033f5bfbb
        driver-assessment = retry
        op-code = 0x28
        cdb = 0x28 0x0 0x11 0x5d 0x75 0xf9 0x0 0x1 0x0 0x0
        pkt-reason = 0x0
        pkt-state = 0x37
        pkt-stats = 0x0
        stat-code = 0x2
        key = 0x6
        asc = 0x29
        ascq = 0x2
        sense-data = 0x70 0x0 0x6 0x0 0x0 0x0 0x0 0xa 0x0 0x0 0x0 0x0 0x29 0x2 
0x2 0x0 0x0 0x0 0xdd 0xba
        __ttl = 0x1
        __tod = 0x4f0b2c2f 0x13efb9ad



thanks,

Geoff