[OpenIndiana-discuss] Swap/stack space continuously shrinking

[Jim Klimov]

On February 21, 2018 11:27:43 AM UTC, Markus Wernig <listener at wernig.net> wrote:
>Hi all
>
>I have 2 OI installations in kvm/qemu virtual machines. Both are
>showing
>the same symptom: The available swap/stack space is continuously
>decreasing, up to the point (roughly every 6-7 weeks) where none is
>left
>and no new processes can be forked. Meaning a forced reboot, as also no
>login shells can be spawned any more.
>I have already moved /tmp to a separate dataset in the global zone (and
>to plain directories within the two other zones).
>
>What I see is a slow, continuous decrease in the sisze of all file
>systems marked as swap (~6.5 GB after reboot):
>
>df -h | grep swap over the last 24 hours:
>
>swap               6.45G  1020K      6.45G     1%    /etc/svc/volatile
>swap               6.45G    60K      6.45G     1%    /var/run
>..
>swap               5.94G  1020K      5.94G     1%    /etc/svc/volatile
>swap               5.94G    60K      5.94G     1%    /var/run
>..
>swap               5.89G   204K      5.89G     1%    /etc/svc/volatile
>swap               5.89G    12K      5.89G     1%    /var/run
>..
>swap               5.62G   204K      5.62G     1%    /etc/svc/volatile
>swap               5.62G    12K      5.62G     1%    /var/run
>..
>swap               5.04G  1020K      5.04G     1%    /etc/svc/volatile
>swap               5.04G    48K      5.04G     1%    /var/run
>
>The swap device on the other hand remains stable:
>
># swap -lh
>swapfile             dev    swaplo   blocks     free
>/dev/zvol/dsk/rpool/swap 270,2        4K     512M     512M
>
>As nothing seems to get stored in that space, I assume it must be used
>by something else (processes not freeing it).
>
>Does anybody have an idea on how to start debugging this? All I get in
>the log files are these messages, starting some hours before the boxes
>stand still:
>
>Feb 14 03:51:09 xfer-srv05 tmpfs: [ID 518458 kern.warning] WARNING:
>/etc/svc/volatile: File system full, swap space limit exceeded
>Feb 14 03:51:10 xfer-srv05 last message repeated 1 time
>Feb 14 04:03:20 xfer-srv05 genunix: [ID 470503 kern.warning] WARNING:
>Sorry, no swap space to grow stack for pid 8106 (httpd)
>Feb 14 04:28:36 xfer-srv05 genunix: [ID 470503 kern.warning] WARNING:
>Sorry, no swap space to grow stack for pid 8920 (httpd)
>Feb 14 05:58:20 xfer-srv05 genunix: [ID 470503 kern.warning] WARNING:
>Sorry, no swap space to grow stack for pid 11050 (httpd)
>Feb 14 05:58:24 xfer-srv05 last message repeated 1 time
>
>Thanks for any ideas.
>
>/markus
>
>
>
>_______________________________________________
>openindiana-discuss mailing list
>openindiana-discuss at openindiana.org
>https://openindiana.org/mailman/listinfo/openindiana-discuss

Do you have anything writing to tmpfs (e.g. /tmp in global or local zones)? Note that unlike Linux, there is no default service to wipe older files here - so if you have e.g. regular jobs dumping something into /tmp and not cleaning up after themselves, that would eat your virtual memory and stay in it until deleted (or until reboot kills the tmpfs).

Jim

--
Typos courtesy of K-9 Mail on my Android