[OpenIndiana-discuss] Swap/stack space continuously shrinking

[Markus Wernig]

Hi all

I have 2 OI installations in kvm/qemu virtual machines. Both are showing
the same symptom: The available swap/stack space is continuously
decreasing, up to the point (roughly every 6-7 weeks) where none is left
and no new processes can be forked. Meaning a forced reboot, as also no
login shells can be spawned any more.
I have already moved /tmp to a separate dataset in the global zone (and
to plain directories within the two other zones).

What I see is a slow, continuous decrease in the sisze of all file
systems marked as swap (~6.5 GB after reboot):

df -h | grep swap over the last 24 hours:

swap               6.45G  1020K      6.45G     1%    /etc/svc/volatile
swap               6.45G    60K      6.45G     1%    /var/run
..
swap               5.94G  1020K      5.94G     1%    /etc/svc/volatile
swap               5.94G    60K      5.94G     1%    /var/run
..
swap               5.89G   204K      5.89G     1%    /etc/svc/volatile
swap               5.89G    12K      5.89G     1%    /var/run
..
swap               5.62G   204K      5.62G     1%    /etc/svc/volatile
swap               5.62G    12K      5.62G     1%    /var/run
..
swap               5.04G  1020K      5.04G     1%    /etc/svc/volatile
swap               5.04G    48K      5.04G     1%    /var/run

The swap device on the other hand remains stable:

# swap -lh
swapfile             dev    swaplo   blocks     free
/dev/zvol/dsk/rpool/swap 270,2        4K     512M     512M

As nothing seems to get stored in that space, I assume it must be used
by something else (processes not freeing it).

Does anybody have an idea on how to start debugging this? All I get in
the log files are these messages, starting some hours before the boxes
stand still:

Feb 14 03:51:09 xfer-srv05 tmpfs: [ID 518458 kern.warning] WARNING:
/etc/svc/volatile: File system full, swap space limit exceeded
Feb 14 03:51:10 xfer-srv05 last message repeated 1 time
Feb 14 04:03:20 xfer-srv05 genunix: [ID 470503 kern.warning] WARNING:
Sorry, no swap space to grow stack for pid 8106 (httpd)
Feb 14 04:28:36 xfer-srv05 genunix: [ID 470503 kern.warning] WARNING:
Sorry, no swap space to grow stack for pid 8920 (httpd)
Feb 14 05:58:20 xfer-srv05 genunix: [ID 470503 kern.warning] WARNING:
Sorry, no swap space to grow stack for pid 11050 (httpd)
Feb 14 05:58:24 xfer-srv05 last message repeated 1 time

Thanks for any ideas.

/markus