[OpenIndiana-discuss] Interesting question about L2ARC

[Dan Swartzendruber]

(snipped my OP)

At first glance it's hard to tell why your l2arc is failing to fill up, but
my suspicion is that it has something to do with your workload. As a recap,
here's how the l2arc works:

 * there is a feed thread (l2arc_feed_thread) that periodically scans
   the end of the MRU/MFU lists of the ARC in order to capture buffers
   before they are evicted and write them to l2arc

 * l2arc by default only caches non-prefetch data (i.e. random-reads),
   since it is primarily a tool to lower random access latency, not
   increase linear throughput (the main pool is assumed to be faster
   than l2arc in bulk read volume)

It is somewhat suspicious that your l2arc only contains 9GB of data. Run the
following command to check your l2arc growth in response to your
workloads:

# while sleep 2; do echo -------; kstat -m zfs -n arcstats | grep l2 ;\ done

Look for the l2_size parameter. If your workload's random-access portion
fits entirely into the ARC, then l2arc isn't going to do you any good.
If you do want to cache prefetched data in the l2arc as well (because your
l2arc devices have cumulatively higher throughput than your main pool), try
setting l2arc_noprefetch=0:

# echo l2arc_noprefetch/W0t0 | mdb -kw

Be advised though, that this might start an unending heavy writing frenzy to
your l2arc devices, which might burn through your l2arc device's flash write
cycles much faster than you'd want.

***

Saso, I think you might be on to something here.  This is a handful of VM's,
none doing anything all that disk intensive, so any disk I/O will tend to be
somewhat random.  I'm not so worried about premature death of the device,
since it's a brand-new device, and we're not talking hundreds of terabytes
per month or anything.  My pool is definitely NOT throughput stressed
(rarely exceeding a 20-30 MB/sec at most), so latency is more of a priority
to me.  I'll give this a try, thanks!