[OpenIndiana-discuss] Interesting question about L2ARC

[Dan Swartzendruber]

I got a 256GB Crucial M4 to use for L2ARC for my OI box.  I added it to 
the tank pool and let it warm for a day or so.  By that point, 'zpool 
iostat -v' said the cache device had about 9GB of data, but (and this is 
what has me puzzled) kstat showed ZERO l2_hits.  That's right, zero.

  kstat | egrep "(l2_hits|l2_misses)"
         l2_hits                         0
         l2_misses                       1143249

The box has 20GB of RAM (it's actually a virtual machine on an ESXi 
host.)  The datastore for the VMs is about 256GB.  My first thought was 
everything is hitting in ARC, but that is clearly not the case, since it 
WAS gradually filling up the cache device.  Maybe it's possible that 
every single miss is never ever being re-read, but that seems unlikely, 
no?  If the l2_hits was a small number, I'd think it just wasn't giving 
me any bang for the buck, but zero sounds suspiciously like some kind of 
bug/mis-configuration.  primarycache and secondarycache are both set to 
all.  arc stats via arc_summary.pl:

ARC Efficency:
          Cache Access Total:             12324974
          Cache Hit Ratio:      87%       10826363       [Defined State 
for buffer]
          Cache Miss Ratio:     12%       1498611        [Undefined 
State for Buffer]
          REAL Hit Ratio:       68%       8469470        [MRU/MFU Hits Only]

          Data Demand   Efficiency:    85%
          Data Prefetch Efficiency:    59%

For the moment, I gave up and moved the SSD back to being my windows7 
drive, where it does make a difference :)  I'd be willing to shell out 
for another SSD, but only if I can gain some benefit from it.  Any 
thoughts would be appreciated (if this is too esoteric for the OI list, 
I can try the zfs discussion list - I am starting here because of common 
platform with the rest of the audience...)