[OpenIndiana-discuss] [developer] Re: Memory usage concern

[Jason Matthews]

-----Original Message-----
From: Brendan Gregg [mailto:brendan.gregg at joyent.com] 


> A full "echo ::kmastat | mdb -k" before reboot may show where the 
> memory is or isn't.

I am not a point where I would normally reboot it, but we are pretty well
deep into memory utilization for the size of the apps that run...

On the proxy tier, it seems that the DCE cache is way out of out of line.
What are the tunables here? 


Thanks,
j.


Here is a dump from a box on the proxy tier...

root at www003:~# echo ::kmastat |mdb -k |sort -nrk 5 |head
kmem_va_4096                4096 2964721 2964736 12143558656B   3023570
0
dce_cache                    152 73389856 73389862 11561725952B  73389938
0
zio_data_buf_131072       131072  43616  43644 5720506368B    893451     0
kmem_va_16384              16384  51737  51744  847773696B     51969     0
zio_buf_16384              16384  51386  51475  843366400B   5324046     0
tcp_conn_cache              1808  17352  88512  181272576B 1597914226     0
kmem_va_8192                8192  22097  22112  181141504B     37356     0
streams_dblk_1040           1152     42 106253  124346368B 1582704292     0
kmem_bufctl_cache             24 3527424 3527541   86519808B   3750227     0
arc_buf_hdr_t                176 345516 345576   64339968B   1199529     0
root at www003:~# echo ::memstat |mdb -k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    3721822             14538   44%
ZFS File Data             1448069              5656   17%
Anon                        84512               330    1%
Exec and libs                5085                19    0%
Page cache                  15805                61    0%
Free (cachelist)            22861                89    0%
Free (freelist)           3072717             12002   37%

Total                     8370871             32698
Physical                  8370870             32698

And here is one from the database tier...

root at db001:~# echo ::kmastat |mdb -k |sort -nrk 5 |head
zfs_file_data_8192        8192 2991810 7365904 60341485568B 3495907153     0
kmem_va_4096              4096 4965999 6615520 27097169920B  73888636     0
zio_data_buf_8192         8192 2991120 2991810 24508907520B 3108123433     0
kmem_va_16384            16384 216265 1442656 23636475904B 224812598     0
zfs_file_data_buf     24814600192B 24814600192B 24814600192B 15532060657
arc_buf_hdr_t           176 56809623 61605434 11469811712B 489897048     0
zio_buf_131072            131072  83335  83370 10927472640B 624255183     0
zio_buf_16384              16384 213952 214677 3517267968B 866096686     0
kmem_alloc_32            32 54596040 95296625 3122679808B 4035614613     0
kmem_va_8192                8192  61443 213888 1752170496B 483211721     0
root at db001:~# echo ::memstat |mdb -k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   10971334             42856   44%
ZFS File Data             6059601             23670   24%
Anon                      5686726             22213   23%
Exec and libs                9304                36    0%
Page cache                  53574               209    0%
Free (cachelist)           140715               549    1%
Free (freelist)           2225873              8694    9%

Total                    25147127             98230
Physical                 25147125             98230