[OpenIndiana-discuss] Swap/stack space continuously shrinking

[Markus Wernig]

Hi all

Thanks for the pointers, will try to follow up on them.

The fact that Solaris is interleaving disk and memory-backed storage
into the tmpfs hadn't struck me until now, as there never seems to be an
increase of the used space in these file systems, but just a decrease of
the available overall (presumably memory) space.

Just one other question: Shouldn't an increase in memory consumption by
any process be visible in the output of tools like eg. top? (Which it
isn't: no matter how low the available "swap" space reported by df
becomes, top keeps reporting ~6GB free memory:

# top
Memory: 8192M phys mem, 5900M free mem, 512M total swap, 512M free swap
ARC:    816M Total, 607M MRU, 159M MFU, 1579K Anon, 4144K Header, 45M Other

# df -h -F tmpfs
Filesystem             Size   Used  Available Capacity  Mounted on
swap                  4.98G  1020K      4.98G     1%    /etc/svc/volatile
swap                  4.98G    48K      4.98G     1%    /var/run)

best
/markus

On 21.02.2018 12:27, Markus Wernig wrote:
> Hi all
> 
> I have 2 OI installations in kvm/qemu virtual machines. Both are showing
> the same symptom: The available swap/stack space is continuously
> decreasing, up to the point (roughly every 6-7 weeks) where none is left
> and no new processes can be forked. Meaning a forced reboot, as also no
> login shells can be spawned any more.
> I have already moved /tmp to a separate dataset in the global zone (and
> to plain directories within the two other zones).
> 
> What I see is a slow, continuous decrease in the sisze of all file
> systems marked as swap (~6.5 GB after reboot):
> 
> df -h | grep swap over the last 24 hours:
> 
> swap               6.45G  1020K      6.45G     1%    /etc/svc/volatile
> swap               6.45G    60K      6.45G     1%    /var/run
> ..
> swap               5.94G  1020K      5.94G     1%    /etc/svc/volatile
> swap               5.94G    60K      5.94G     1%    /var/run
> ..
> swap               5.89G   204K      5.89G     1%    /etc/svc/volatile
> swap               5.89G    12K      5.89G     1%    /var/run
> ..
> swap               5.62G   204K      5.62G     1%    /etc/svc/volatile
> swap               5.62G    12K      5.62G     1%    /var/run
> ..
> swap               5.04G  1020K      5.04G     1%    /etc/svc/volatile
> swap               5.04G    48K      5.04G     1%    /var/run
> 
> The swap device on the other hand remains stable:
> 
> # swap -lh
> swapfile             dev    swaplo   blocks     free
> /dev/zvol/dsk/rpool/swap 270,2        4K     512M     512M
> 
> As nothing seems to get stored in that space, I assume it must be used
> by something else (processes not freeing it).
> 
> Does anybody have an idea on how to start debugging this? All I get in
> the log files are these messages, starting some hours before the boxes
> stand still:
> 
> Feb 14 03:51:09 xfer-srv05 tmpfs: [ID 518458 kern.warning] WARNING:
> /etc/svc/volatile: File system full, swap space limit exceeded
> Feb 14 03:51:10 xfer-srv05 last message repeated 1 time
> Feb 14 04:03:20 xfer-srv05 genunix: [ID 470503 kern.warning] WARNING:
> Sorry, no swap space to grow stack for pid 8106 (httpd)
> Feb 14 04:28:36 xfer-srv05 genunix: [ID 470503 kern.warning] WARNING:
> Sorry, no swap space to grow stack for pid 8920 (httpd)
> Feb 14 05:58:20 xfer-srv05 genunix: [ID 470503 kern.warning] WARNING:
> Sorry, no swap space to grow stack for pid 11050 (httpd)
> Feb 14 05:58:24 xfer-srv05 last message repeated 1 time