[OpenIndiana-discuss] [developer] Re: Memory usage concern

[Brendan Gregg]

(removing developer)

On Mon, Oct 15, 2012 at 4:01 PM, Cedric Blancher <
cedric.blancher at googlemail.com> wrote:

> On 16 October 2012 00:52, Jason Matthews <jason at broken.net> wrote:
> >
> >
> >
> > I should also mention that over all network performance degrades over
> time
> > and the only fix I have sound so far is to shut down the zones, destroy
> the
> > vnics, recreate them, and then reboot the zones.
> >
> > I do that about every four hours to keep the response times reasonable.
> It
> > really sucks.
>
> IMO you blame the wrong people. You can have the same kind of problems
> with any Illumos-based distribution if you activate a zone and let the
> machine just sit there for a week or so or have a lot filesystem
> activity using mmap(). Either way the machines will choke themselves
> to memory starvation. The only workaround we found are regular reboots
> (every 24h), or limit the ZFS ARC to an absolute minimum.
>

Is this bare-metal or under a VM? We have many mmap()-based workloads
running in production, bare-metal, without issue. The ZFS ARC will reduce
its size (reap) when the system is low on memory. If it doesn't, and needs
workarounds, that's a bug. It can be diagnosed by checking kstats
(arcstats, or using arcstat.pl) and using DTrace.

And by choke, you mean the system pages out applications, such that you
have a rate of anonymous page-ins? Ie, "vmstat -p 1" and having a rate of
"api"?

A full "echo ::kmastat | mdb -k" before reboot may show where the memory is
or isn't.

Brendan

-- 
Brendan Gregg, Joyent                      http://dtrace.org/blogs/brendan