[OpenIndiana-discuss] AMCC/3Ware 9550SX-8LP

[Edward Ned Harvey (openindiana)]

> From: Sašo Kiselkov [mailto:skiselkov.ml at gmail.com]
> 
> On 01/24/2013 02:39 PM, Edward Ned Harvey (openindiana) wrote:
> >
> > Based on my extensive work benchmarking zfs systems, I can say this:  The
> default write performance on a plain old sas/sata card (without SSD) is
> horrible by comparison to the following alternatives:
> >
> > You get a huge increase if you add the HBA write buffer.
> > You get better still (by just a little) if you disable the HBA write buffer (go
> back to the plain old SATA card) and add a SSD log device instead.
> > You get the best performance (of course) if you eliminate both the HBA
> write buffer and SSD log device, while also disabling ZIL.  But this
> configuration is only "safe" for certain types of usage.
> 
> Just a quick question and I think I know the answer: you mean *sync*
> write performance, right? 

Correct.


> Otherwise none of those above matter and the
> HBA should act as just a fat pipe from RAM to disks.

When you're doing async writes (or sync writes with ZIL disabled, which are in effect, async writes) then the HBA write-back actually hurts your performance a little bit.  Like 5-10% or so.

The conclusion to draw is:

(a)  The absolute fastest thing you can do is use a dumb controller, and sync=disabled.  This maximizes performance for both sync & async writes.  But only if it's safe for you to run in that configuration.  Otherwise ...

(b)  The second fastest thing you can do is use a dumb controller, and a dedicated log device.  This again maximizes performance for both sync & async, but now it's safe for whatever your usage patterns.

(c)  The third fastest thing is to have a smart HBA with NV write-back.  This hurts async write performance slightly, but the sync write performance is far better than the dumb & naked bare-bones solution.

(d)  The dumb & naked bare-bones solution is to use a dumb controller, with ZIL enabled (which is default), and no special log device.  Your async writes are optimized, but your sync writes are a dog, which tend to also drag your async writes into the gutter with them.  Believe it or not, dedicating a HDD slog to a HDD pool actually improves performance signfiicantly just because you're able to isolate the sync writes to a device while the async can still hit the pool without interference.  But this is kind of a dumb configuration, so we don't talk about it much except for entertainment purposes.  ;-)

So basically, no matter what you do, you're not going to significantly impact the async write performance.  The sync performance is the only thing you have to think about optimizing.

The performance delta from (d) to (c) is approximately proportional to (c) vs (b), which is approximately proportional to (b) vs (a).  A reasonable ballpark is to say approx 1.75x to 2x jump on each of those deltas.

Long story short:  This was all a response to the intended-to-be-tangential comment "Who uses hardware raid anymore now that we have zfs?"  And the answer is, "Anybody who has an HBA, doesn't have an SSD, and can't or won't disable the ZIL."  This is a surprisingly high number of people, because people previously bought (and continue to buy) servers that have HBA's, which later get provisioned to run zfs.  And then lots of times, they don't know or can't spring for the slog solution.