[OpenIndiana-discuss] [developer] Re: Memory usage concern

[Brendan Gregg]

On Mon, Oct 15, 2012 at 5:52 PM, Jason Matthews <jason at broken.net> wrote:

>
>
> -----Original Message-----
> From: Brendan Gregg [mailto:brendan.gregg at joyent.com]
>
>
> > A full "echo ::kmastat | mdb -k" before reboot may show where the
> > memory is or isn't.
>
> I am not a point where I would normally reboot it, but we are pretty well
> deep into memory utilization for the size of the apps that run...
>
> On the proxy tier, it seems that the DCE cache is way out of out of line.
> What are the tunables here?
>

This sounds like the DCE cache cleanup issue:
http://smartos.org/2012/02/28/using-flamegraph-to-solve-ip-scaling-issue-dce/

A "netstat -dn | wc -l" should show how many entries are in the cache.

Brendan



>
>
> Thanks,
> j.
>
>
> Here is a dump from a box on the proxy tier...
>
> root at www003:~# echo ::kmastat |mdb -k |sort -nrk 5 |head
> kmem_va_4096                4096 2964721 2964736 12143558656B   3023570
> 0
> dce_cache                    152 73389856 73389862 11561725952B  73389938
> 0
> zio_data_buf_131072       131072  43616  43644 5720506368B    893451     0
> kmem_va_16384              16384  51737  51744  847773696B     51969     0
> zio_buf_16384              16384  51386  51475  843366400B   5324046     0
> tcp_conn_cache              1808  17352  88512  181272576B 1597914226     0
> kmem_va_8192                8192  22097  22112  181141504B     37356     0
> streams_dblk_1040           1152     42 106253  124346368B 1582704292     0
> kmem_bufctl_cache             24 3527424 3527541   86519808B   3750227
> 0
> arc_buf_hdr_t                176 345516 345576   64339968B   1199529     0
> root at www003:~# echo ::memstat |mdb -k
> Page Summary                Pages                MB  %Tot
> ------------     ----------------  ----------------  ----
> Kernel                    3721822             14538   44%
> ZFS File Data             1448069              5656   17%
> Anon                        84512               330    1%
> Exec and libs                5085                19    0%
> Page cache                  15805                61    0%
> Free (cachelist)            22861                89    0%
> Free (freelist)           3072717             12002   37%
>
> Total                     8370871             32698
> Physical                  8370870             32698
>
>

-- 
Brendan Gregg, Joyent                      http://dtrace.org/blogs/brendan