[OpenIndiana-discuss] dell r730xd + SATA

[Andrew Gabriel]

On 28/04/2017 22:57, Nikola M wrote:
> On 04/28/17 09:09 PM, jason matthews wrote:
>>
>> Is anyone using the R730XD with its 3x port expanders successfully
>> with SATA drives? Yes, I am aware of the conventional wisdom.
> I'll pass some general knowledge, that may not be suitable for your
> exact hardware, but could be used as the pointers when you compare your
> specs.
>
> Using SAS port expanders is unadvised because they include chips in them
> that can go crazy (with their own small firmwares), and with expanders,
> the disk controller is not in direct control of the drives and that can
> do really bad for ZFS.
> That goes especially if using SAS expanders with SATA drives, like a
> nightmarish situation, the worst case, that is actually not supported
> under illumos. (And especially mixing SAS and SATA drives on the same
> SAS expander)
> Even using SATA drives with SAS controller on direct ports, without
> expanders, is discouraged and not a good idea.
> SAS controller -- SAS drives.  SATA controller - SATA drives.
>
> The controller is best to be working in JBOD mode. Don't use Hardware
> RAID levels, use controllers that can do JBOD. (Take care, since JBOD
> usually gets intentionally disabled on Hardware RAID controllers, so you
> have a vendor lock-in for their hardware..)
> Software RAID is what ZFS is for, to elevate one from hardware
> constrains. So you can just pop up disks from one machine and pop them
> in into another JBOD machine and it just works without any
> configuration. (even between x86 and SPARC - ZFS is endian agnostic).
> Not possible with hardware RAID controlling drives.
>
>> I have a number of R730s working with the 2x expanders and Intel DC
>> S37{0,1}0 SSDs. It is time to order again and the 24 bay 730XD is a
>> seductress in terms of storage options. I just dont know if it will
>> work well mixing expanders and SATA.
> No it won't. It might look ok, or you could seem happy, but at the first
> sign of trouble or reporting some bug, you could learn the hard way,
> that is not supported configuration to use SATA drives with SAS
> expanders. And avoid configuration with the expanders at all, but do
> direct connecting drives to the controller ports).
>
> Btw, internet is full with expander warnings, hope this helps.

It might help to explain when and why it doesn't work.

SATA drives think nothing of dropping the phy (physical interface) 
whenever they fancy, e.g. drive controller resetting itself, or doing 
some error recovery. Generally, their firmware is pretty grotty in this 
area, and this is expected and ignored. It's not a problem when you have 
a one-to-one link to the host controller.

The problem arises when you go through a SAS expander.

SAS drives don't drop their phy if they can possibly avoid it, as they 
know it's expensive for the SAS fabric. When a SATA drive does this, the 
expander ends up internally re-enumerating all the drives to work out 
who went and who is still there, which [multi-]paths still work, etc. 
This will normally cause all their phys to be dropped.

What this means in practice is that once you get a SATA drive which 
starts going bad, its phy will start going up and down. The expander 
will take all the other SATA drive's phys up and down too as it keeps 
re-enumerating to work out which drives are still there. The host will 
now start to see lots of transport errors and timeouts across all the 
SATA drives. The bottom line is, when one SATA drive starts going bad, 
you will usually find you get errors reported against lots of them, and 
there's no way to find the culprit without taking the storage array down 
and testing access to each disk one at a time. That's a non-starter in 
any Enterprise environment.

There are some limited cases where SATA drives can work, and that's 
where there's only one of them on the SAS expander. Sometimes this was 
used to put a SATA SSD into an otherwise all-SAS array.

If you want to build a cheap array that actually works, you should use 
nearline SAS drives rather than SATA drives.

-- 
Andrew Gabriel