[OpenIndiana-discuss] Swap/stack space continuously shrinking

[Richard L. Hamilton]

Once a week, save the output of

df -h -F tmpfs

and

top -o size -b -c 10

Look for filesystem growth in the former, and a process that's getting larger in the latter.

If it's the kernel, you'd have to use mdb -K from the console or boot with kmdb, and then use  ::findleaks

The following covers some of that, if differently; but their use of mdb is for a process, not for the kernel.
https://www.joyent.com/blog/bruning-questions-where-s-my-memory-going

For the kernel, google for

mdb findleaks

and look around a little.

> On Feb 21, 2018, at 06:27, Markus Wernig <listener at wernig.net> wrote:
> 
> Hi all
> 
> I have 2 OI installations in kvm/qemu virtual machines. Both are showing
> the same symptom: The available swap/stack space is continuously
> decreasing, up to the point (roughly every 6-7 weeks) where none is left
> and no new processes can be forked. Meaning a forced reboot, as also no
> login shells can be spawned any more.
> I have already moved /tmp to a separate dataset in the global zone (and
> to plain directories within the two other zones).
> 
> What I see is a slow, continuous decrease in the sisze of all file
> systems marked as swap (~6.5 GB after reboot):
> 
> df -h | grep swap over the last 24 hours:
> 
> swap               6.45G  1020K      6.45G     1%    /etc/svc/volatile
> swap               6.45G    60K      6.45G     1%    /var/run
> ..
> swap               5.94G  1020K      5.94G     1%    /etc/svc/volatile
> swap               5.94G    60K      5.94G     1%    /var/run
> ..
> swap               5.89G   204K      5.89G     1%    /etc/svc/volatile
> swap               5.89G    12K      5.89G     1%    /var/run
> ..
> swap               5.62G   204K      5.62G     1%    /etc/svc/volatile
> swap               5.62G    12K      5.62G     1%    /var/run
> ..
> swap               5.04G  1020K      5.04G     1%    /etc/svc/volatile
> swap               5.04G    48K      5.04G     1%    /var/run
> 
> The swap device on the other hand remains stable:
> 
> # swap -lh
> swapfile             dev    swaplo   blocks     free
> /dev/zvol/dsk/rpool/swap 270,2        4K     512M     512M
> 
> As nothing seems to get stored in that space, I assume it must be used
> by something else (processes not freeing it).
> 
> Does anybody have an idea on how to start debugging this? All I get in
> the log files are these messages, starting some hours before the boxes
> stand still:
> 
> Feb 14 03:51:09 xfer-srv05 tmpfs: [ID 518458 kern.warning] WARNING:
> /etc/svc/volatile: File system full, swap space limit exceeded
> Feb 14 03:51:10 xfer-srv05 last message repeated 1 time
> Feb 14 04:03:20 xfer-srv05 genunix: [ID 470503 kern.warning] WARNING:
> Sorry, no swap space to grow stack for pid 8106 (httpd)
> Feb 14 04:28:36 xfer-srv05 genunix: [ID 470503 kern.warning] WARNING:
> Sorry, no swap space to grow stack for pid 8920 (httpd)
> Feb 14 05:58:20 xfer-srv05 genunix: [ID 470503 kern.warning] WARNING:
> Sorry, no swap space to grow stack for pid 11050 (httpd)
> Feb 14 05:58:24 xfer-srv05 last message repeated 1 time
> 
> Thanks for any ideas.
> 
> /markus
> 
> 
> 
> _______________________________________________
> openindiana-discuss mailing list
> openindiana-discuss at openindiana.org
> https://openindiana.org/mailman/listinfo/openindiana-discuss
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 874 bytes
Desc: Message signed with OpenPGP
URL: <http://openindiana.org/pipermail/openindiana-discuss/attachments/20180221/7c06b86b/attachment.bin>