[OpenIndiana-discuss] write speeds faster with no ZIL and L2ARC

[Lucas Van Tol]

I played around using a thumb drive as a SLOG vs no SLOG; and small IO 
speed was still significantly higher (2-3x faster for untar operations via NFS, very small files);
 even though the max write bandwidth to that pool tanked horribly (down 
to ~10 mbps from gigabit line speed on large files).

I can't recall what version that was ( I think it was OpenSolaris < snv_120) but that may still be the case.   I guess sometimes writes that bypass ZIL w/o a SLOG still get written to the SLOG if it's there?
If so, your older SSD's may be limiting max write bandwidth, but they should still help a lot for small write operations.
MLC SSD's aren't very fast if they are flushing their memory based write cache constantly, which may be why your new SSD's don't help either.  The SLOG should be flushing it's cache on pretty much every write...
The SLOG is largely there to provide random write speed; i.e. lots very small writes.     
Those will generally drop like a rock without the SSD, even if they are getting kind of old; so you may still want the SLOG around if you are doing small writes.
If you aren't doing small writes; than a SLOG probably won't help you much in general, and you should try leaving it with the SLOGS offline.

You might try re-flashing the firmware on the intel SSD's; effectively performing a full drive 'trim' operation if just removing and re-adding them didn't already do that.  
This could restore some of their speed, I've seen it help the Intel X25-e's before, but not necessarily. 
If you stick the pair of the Intel SSD's as SLOG devices, it should raise the pools write bandwidth back up while still having the small write performance from the SLOG devices.

As for the read cache devices; I don't know how they would slow your pool down. 
They should still have good read speeds for serving up some metadata; and writes to the cache shouldn't be affecting the rest of the pool, which is why any SSD should be able to help there.

You can also try running latencytop; is that installed by default now?
That should say if your system is spending much time writing ZIL data or not.   
'ZFS ZIL writer I/O' should be the thing to watch for.
See if that goes away without the SLOG SSD's; but this would be something to test with production and not just a couple file copies.


-Lucas Van Tol

> From: gregory at youngblood.me
> Date: Mon, 27 Jun 2011 14:55:07 -0700
> To: openindiana-discuss at openindiana.org
> Subject: Re: [OpenIndiana-discuss] write speeds faster with no ZIL and L2ARC
> 
> 
> On Jun 27, 2011, at 2:46 PM, Chris Mosetick wrote:
> 
> > Lucas, thank you for your ideas. For over a week I looked at the entirety of
> > iostat's output, including asvc_t, %b, %w as you have suggested. Nothing in
> > the output seems to be able to explain the transfer speed slowness we
> > started experiencing. I'm also completely aware that I can keep adding log
> > and cache devices to a single pool. Doing so ignores the truth of the
> > matter. Things were humming along just fine with the two X25-E's as log and
> > cache, and now things move faster without them configured. I'm pretty sure
> > that ZFS is supposed to move at reasonable speeds when on 7200rpm SATA
> > disks, and move even faster when low latency log and cache devices are
> > configured. At least that's how it has been for me.
> 
> Solaris (and OpenSolaris) used to seem like it worked really well with 7200 SATA drives. SAS would be faster, of course, but the SATA were a good option. Of course, lots of concurrent IO could bog it down, and made the SSD for ZIL a requirement. 
> 
> I haven't been following the full thread, but is it possible the SSDs are nearing end of life due to limits on write longevity, perhaps it's taking longer for writes to complete now? Is it possible to put a brand new SSD in as a ZIL and see what happens? I don't know, kind of taking a stab in the dark here. 
> 
> Hope you get it nailed down. I'll be watching to see what you discover. 
> 
> Greg
> _______________________________________________
> OpenIndiana-discuss mailing list
> OpenIndiana-discuss at openindiana.org
> http://openindiana.org/mailman/listinfo/openindiana-discuss