[OpenIndiana-discuss] dell r730xd + SATA

[Jason Matthews]

Andrew,

First let me say, you response was both thoughtful and articulate. Thanks.

However, I am familiar with all of these legacy issues. Some seven or 
eight years has passed since all this bruja-ha went down. I am 
interested in examining how modern high grade equipment performs today.

This notion of it is not being supported is non-sense. It 100% 
transparent to the operating system. The question is, how well will it 
work. In 2009, not so well. How about today? Once manufacturers produce 
gear that works as advertised and that gear interoperates with others  
then it is suddenly supported. So let's just set that aside for now. It 
is a silly circular ascertain (forgive me if I am mashing up your 
response with Nikola's).

What you are basically describing is forwarding and/or firmware issues 
in the expanders and drives. Intel, in my opinion did a fantastic job 
with their firmware. Perhaps it doesnt drop phy as you suggest. Perhaps 
the expanders are better at dealing with issues.


I suppose my point is, we are not going to know how well these 
configurations work unless someone looks at the issue periodically. I 
think eight years is enough time to give it another look.


j.



On 4/28/2017 4:29 PM, Andrew Gabriel wrote:
> On 28/04/2017 22:57, Nikola M wrote:
>> On 04/28/17 09:09 PM, jason matthews wrote:
>>>
>>> Is anyone using the R730XD with its 3x port expanders successfully
>>> with SATA drives? Yes, I am aware of the conventional wisdom.
>> I'll pass some general knowledge, that may not be suitable for your
>> exact hardware, but could be used as the pointers when you compare your
>> specs.
>>
>> Using SAS port expanders is unadvised because they include chips in them
>> that can go crazy (with their own small firmwares), and with expanders,
>> the disk controller is not in direct control of the drives and that can
>> do really bad for ZFS.
>> That goes especially if using SAS expanders with SATA drives, like a
>> nightmarish situation, the worst case, that is actually not supported
>> under illumos. (And especially mixing SAS and SATA drives on the same
>> SAS expander)
>> Even using SATA drives with SAS controller on direct ports, without
>> expanders, is discouraged and not a good idea.
>> SAS controller -- SAS drives.  SATA controller - SATA drives.
>>
>> The controller is best to be working in JBOD mode. Don't use Hardware
>> RAID levels, use controllers that can do JBOD. (Take care, since JBOD
>> usually gets intentionally disabled on Hardware RAID controllers, so you
>> have a vendor lock-in for their hardware..)
>> Software RAID is what ZFS is for, to elevate one from hardware
>> constrains. So you can just pop up disks from one machine and pop them
>> in into another JBOD machine and it just works without any
>> configuration. (even between x86 and SPARC - ZFS is endian agnostic).
>> Not possible with hardware RAID controlling drives.
>>
>>> I have a number of R730s working with the 2x expanders and Intel DC
>>> S37{0,1}0 SSDs. It is time to order again and the 24 bay 730XD is a
>>> seductress in terms of storage options. I just dont know if it will
>>> work well mixing expanders and SATA.
>> No it won't. It might look ok, or you could seem happy, but at the first
>> sign of trouble or reporting some bug, you could learn the hard way,
>> that is not supported configuration to use SATA drives with SAS
>> expanders. And avoid configuration with the expanders at all, but do
>> direct connecting drives to the controller ports).
>>
>> Btw, internet is full with expander warnings, hope this helps.
>
> It might help to explain when and why it doesn't work.
>
> SATA drives think nothing of dropping the phy (physical interface) 
> whenever they fancy, e.g. drive controller resetting itself, or doing 
> some error recovery. Generally, their firmware is pretty grotty in 
> this area, and this is expected and ignored. It's not a problem when 
> you have a one-to-one link to the host controller.
>
> The problem arises when you go through a SAS expander.
>
> SAS drives don't drop their phy if they can possibly avoid it, as they 
> know it's expensive for the SAS fabric. When a SATA drive does this, 
> the expander ends up internally re-enumerating all the drives to work 
> out who went and who is still there, which [multi-]paths still work, 
> etc. This will normally cause all their phys to be dropped.
>
> What this means in practice is that once you get a SATA drive which 
> starts going bad, its phy will start going up and down. The expander 
> will take all the other SATA drive's phys up and down too as it keeps 
> re-enumerating to work out which drives are still there. The host will 
> now start to see lots of transport errors and timeouts across all the 
> SATA drives. The bottom line is, when one SATA drive starts going bad, 
> you will usually find you get errors reported against lots of them, 
> and there's no way to find the culprit without taking the storage 
> array down and testing access to each disk one at a time. That's a 
> non-starter in any Enterprise environment.
>
> There are some limited cases where SATA drives can work, and that's 
> where there's only one of them on the SAS expander. Sometimes this was 
> used to put a SATA SSD into an otherwise all-SAS array.
>
> If you want to build a cheap array that actually works, you should use 
> nearline SAS drives rather than SATA drives.
>