[OpenIndiana-discuss] [developer] Re: Memory usage concern

[Richard Elling]

On Oct 22, 2012, at 9:13 AM, James Carlson <carlsonj at workingcode.com> wrote:

> Daniel Kjar wrote:
>> I have this problem with any VM running on either Sol10 Nevada,
>> Opensolaris, openindiana.  I have the ARC restricted now but for some
>> reason, and 'people at sun that know these things' have mentioned it
>> before when sun still existed, when something needs a big devoted chunk
>> of ram, zfs fails miserably at giving up its cache.  If I don't limit
>> the ARC, a few days of even a single VM running causes the system to
>> start stalling.  With the ARC limit, everything is peaches.
> 
> My understanding is that the two mechanisms are only superficially similar.
> 
> UFS was (and is) in bed with the VM implementation, so it's using memory
> pages as the cache in the same way that the VM keeps paged-in data in
> memory.  UFS "giving up" memory that's needed for applications is really
> the same thing as saying that the VM reallocates memory pages when it
> needs to.

UFS cache is also mostly clean data.

> 
> ZFS, though, is different.  It's not as much in bed with the VM.  The
> ARC is allocated memory, done in the same way that any other kernel
> service can allocate memory.  There is a back-pressure "help! I need
> more space" mechanism on normal kernel allocations, and this is what the
> ARC relies on to give up memory when necessary, but it occurs much
> further down the line than ordinary VM page reallocation.  Thus, when it
> does happen, the damage has already been done -- the VM has already been
> exhausted of easy-to-find pages, and we're already in a bad state.

In a ZFS system, the ARC could contain a large amount of dirty data that
needs to be written to the pool. Giving back memory can involve actual
disk I/O.

> So, yes, limiting the ARC is (unfortunately) a good thing.

Yes, and can be used to emulate UFS ala SunOS 3.x.

Also, note that the size of an ARC shrink request is tunable, but doesn't scale
well to large or small machines. In other words, if there is a sudden demand
for more memory, the ARC will shrink by 2^arc_shrink_shift. If you need much
more than that, another shrink will occur. This can appear to increase the time
needed to shrink and there is no real mechanism to do otherwise, given the 
existing VM design.
 -- richard

--

Richard.Elling at RichardElling.com
+1-760-896-4422