[OpenIndiana-discuss] KVM Failed to allocate memory: Resource temporarily unavailable
Jonathan Adams
t12nslookup at gmail.com
Thu Dec 10 10:10:01 UTC 2015
If you have the disk space, you might be as well creating a swap size of
memory x 2. Old Solaris used to suggest you set it to memory x 4, but that
was when memory was limited and expensive.
On 10 December 2015 at 10:01, Fucai Liang （BLCT） <fcliang at baolict.com>
wrote:
> Thanks for your help！
>
> root at oi01:~# swap -sh
> total: 8.1G allocated + 21M reserved = 8.1G used, 16G available
> root at oi01:~# swap -l
> swapfile dev swaplo blocks free
> /dev/zvol/dsk/rpool/swap 100,2 8 12574712 12574712
> root at oi01:~#
>
>
> 16G swap ， so KVM can locked 16G memory .
>
> thanks again .
>
>
> ------------------------------
> Fucai Liang
>
>
>
>
> On Dec 10, 2015, at 17:24, Jonathan Adams <t12nslookup at gmail.com> wrote:
>
> > how much swap do you have?
> >
> > KVM automatically allocates the same amount of swap as memory when you
> > start an instance, if you don't have the free swap it won't work.
> >
> > Jon
> >
> > On 10 December 2015 at 04:04, Fucai Liang （BLCT） <fcliang at baolict.com>
> > wrote:
> >
> >>
> >>
> >> Hello, guys：
> >>
> >> I has a server running oi_151.1.8, the server have 32G memory .
> >>
> >>
> >> root at oi01:~# prtconf | grep Memory
> >> Memory size: 32760 Megabytes
> >> root at oi01:~# echo "::memstat" | mdb -k
> >> Page Summary Pages MB %Tot
> >> ------------ ---------------- ---------------- ----
> >> Kernel 451736 1764 5%
> >> ZFS File Data 78429 306 1%
> >> Anon 19849 77 0%
> >> Exec and libs 586 2 0%
> >> Page cache 2170 8 0%
> >> Free (cachelist) 10432 40 0%
> >> Free (freelist) 7821076 30551 93%
> >>
> >> Total 8384278 32751
> >> Physical 8384277 32751
> >> root at oi01:~#
> >>
> >>
> >> I lauch a KVM VM by
> >>
> >>
> >> #!/bin/sh
> >>
> >> qemu-kvm \
> >> -vnc 0.0.0.0:2 \
> >> -cpu host \
> >> -smp 2 \
> >> -m 8192 \
> >> -no-hpet \
> >> -localtime \
> >> -drive file=/dev/zvol/rdsk/rpool/svrinit,if=virtio,index=0 \
> >> -net nic,vlan=0,name=e1000g0,model=e1000,macaddr=2:8:20:bd:ae:01
> \
> >> -net
> >> vnic,vlan=0,name=e1000g0,ifname=vnic01,macaddr=2:8:20:bd:ae:01 \
> >> -vga std \
> >> -daemonize
> >>
> >>
> >> root at oi01:~# echo "::memstat" | mdb -k
> >> Page Summary Pages MB %Tot
> >> ------------ ---------------- ---------------- ----
> >> Kernel 461061 1801 5%
> >> ZFS File Data 78446 306 1%
> >> Anon 2131265 8325 25%
> >> Exec and libs 918 3 0%
> >> Page cache 2255 8 0%
> >> Free (cachelist) 10015 39 0%
> >> Free (freelist) 5700318 22266 68%
> >>
> >> Total 8384278 32751
> >> Physical 8384277 32751
> >> root at oi01:~#
> >>
> >>
> >> then I launch second KVM VM,
> >>
> >> #!/bin/sh
> >>
> >> qemu-kvm \
> >> -enable-kvm \
> >> -vnc 0.0.0.0:3 \
> >> -cpu host \
> >> -smp 2 \
> >> -m 8192 \
> >> -no-hpet \
> >> -localtime \
> >> -drive file=/dev/zvol/rdsk/rpool/svr03,if=virtio,index=0 \
> >> -net nic,vlan=0,name=e1000g0,model=e1000,macaddr=2:8:20:bd:ae:03
> \
> >> -net
> >> vnic,vlan=0,name=e1000g0,ifname=vnic03,macaddr=2:8:20:bd:ae:03 \
> >> -vga std \
> >> -daemonize
> >>
> >>
> >>
> >> I got err message :
> >> Failed to allocate memory: Resource temporarily unavailable
> >>
> >> -bash: fork: Not enough space
> >>
> >> .
> >>
> >> the same ietmes happen on OmniOS , the following is a OmniOS-discuss
> mail
> >> .
> >>
> >> That caused by availrmem not enough for memory locked .
> >>
> >> Is that mean I can only use about 14G memory for KVM VM in a 32G memory
> >> installed server?
> >> OS can only locked about 14G memory .
> >>
> >> ——————FOLLOWING--------------
> >>
> >> Hello, guys：
> >>
> >> I has a server running OmniOS v11 r151016. the server have 32G memory .
> >> I star tow kvm virtual machines by running the following commands:
> >>
> >> qemu-system-x86_64 -enable-kvm -vnc 0.0.0.0:12 -cpu host -smp 4 -m 8192
> >> -no-hpe
> >>
> >>
> >> qemu-system-x86_64 -enable-kvm -vnc 0.0.0.0:11 -cpu host -smp 2 -m 4096
> >> -no-hpe
> >>
> >> one use 8G memory and the other one use 4G memory.
> >>
> >> now the memory usage of the system as following:
> >>
> >> root at BLCC01:/root# prtconf | grep Memory
> >> Memory size: 32760 Megabytes
> >> root at BLCC01:/root# echo "::memstat" | mdb -k
> >> Page Summary Pages MB %Tot
> >> ------------ ---------------- ---------------- ----
> >> Kernel 549618 2146 7%
> >> ZFS File Data 668992 2613 8%
> >> Anon 3198732 12495 38%
> >> Exec and libs 1411 5 0%
> >> Page cache 4402 17 0%
> >> Free (cachelist) 10578 41 0%
> >> Free (freelist) 3950545 15431 47%
> >>
> >> Total 8384278 32751
> >> Physical 8384277 32751
> >> root at BLCC01:/root# swap -sh
> >> total: 12G allocated + 35M reserved = 12G used, 6.8G available
> >> root at BLCC01:/root# swap -l
> >> swapfile dev swaplo blocks free
> >> /dev/zvol/dsk/rpool/swap 263,2 8 8388600 8388600
> >> root at BLCC01:/root#
> >>
> >>
> >> root at BLCC01:/root# prctl $$
> >>
> >> project.max-locked-memory
> >> usage 12.0GB
> >> system 16.0EB max deny
> >> -
> >> project.max-port-ids
> >> privileged 8.19K - deny
> >> -
> >> system 65.5K max deny
> >> -
> >> project.max-shm-memory
> >> privileged 8.00GB - deny
> >> -
> >> system 16.0EB max deny
> >> -
> >>
> >>
> >>
> >>
> >>
> >>
> >> #prstat -J
> >>
> >> PROJID NPROC SWAP RSS MEMORY TIME CPU PROJECT
> >> 1 5 12G 12G 38% 1:07:23 5.6% user.root
> >> 0 43 72M 76M 0.2% 0:00:59 0.0% system
> >> 3 5 4392K 14M 0.0% 0:00:00 0.0% default
> >>
> >>
> >>
> >> then I start the third vm (4G memory), it got the following error :
> >>
> >>
> >> qemu-system-x86_64 -enable-kvm -vnc 0.0.0.0:2 -cpu host -smp 2 -m 4096
> >> -no-hpet
> >>
> >> qemu_mlock: have only locked 1940582400 of 4294967296 bytes; still
> >> trying...
> >> qemu_mlock: have only locked 1940582400 of 4294967296 bytes; still
> >> trying...
> >> qemu_mlock: have only locked 1940582400 of 4294967296 bytes; still
> >> trying...
> >> qemu_mlock: have only locked 1940582400 of 4294967296 bytes; still
> >> trying...
> >> qemu_mlock: have only locked 1940582400 of 4294967296 bytes; still
> >> trying...
> >> qemu_mlock: have only locked 1940582400 of 4294967296 bytes; still
> trying…
> >>
> >>
> >> I got 15G free memory in the system, why qemu-system-x86_64 can not
> locked
> >> enough memory ？
> >>
> >> Thanks for your help !
> >>
> >> sorry for my poor english !
> >>
> >>
> >>
> >>
> >> -----------------------------------
> >> fcliang
> >>
> >>
> >>
> >> Thank for your help!
> >>
> >> when the server boot up, it has 7989066 pages availrmem. after I
> launch
> >> one VM (8Gmemory), availrmem decrease to 4756624 .
> >>
> >>
> >>
> >> 7989066-4756624 = 3232442
> >>
> >> 3232442/256 = 12626.7265625 / 1024 = 12.3G
> >>
> >>
> >>
> >> root at BLCC01:/root# mdb -ke 'availrmem/D ; pages_pp_maximum/D'
> >> availrmem:
> >> availrmem: 7989066
> >> pages_pp_maximum:
> >> pages_pp_maximum: 325044
> >>
> >> root at BLCC01:/root# qemu-system-x86_64 -enable-kvm -vnc 0.0.0.0:12 -cpu
> >> host -smp 4 -m 8192 -no-hpe
> >>
> >>
> >>
> >> root at BLCC01:/root# mdb -ke 'availrmem/D ; pages_pp_maximum/D'
> >> availrmem:
> >> availrmem: 4756624
> >> pages_pp_maximum:
> >> pages_pp_maximum: 325044
> >> root at BLCC01:/root#
> >>
> >>
> >> That mean the VM use 12.3G availrmem , how it happens ?
> >>
> >> Thank !
> >>
> >>
> >>
> >>
> >> ------------------------------
> >> fcliang
> >>
> >>
> >>
> >>
> >> On Dec 2, 2015, at 1:37, Joshua M. Clulow <josh at sysmgr.org> wrote:
> >>
> >>> On 1 December 2015 at 09:11, Dan McDonald <danmcd at omniti.com> wrote:
> >>>>> On Dec 1, 2015, at 12:03 PM, Fucai.Liang <fcliang at baolict.com>
> wrote:
> >>>>> then I start the third vm (4G memory), it got the following error :
> >>>>> qemu-system-x86_64 -enable-kvm -vnc 0.0.0.0:2 -cpu host -smp 2 -m
> >> 4096 -no-hpet
> >>>>>
> >>>>> qemu_mlock: have only locked 1940582400 of 4294967296 bytes; still
> >> trying...
> >>>>> qemu_mlock: have only locked 1940582400 of 4294967296 bytes; still
> >> trying...
> >>>>> qemu_mlock: have only locked 1940582400 of 4294967296 bytes; still
> >> trying...
> >>>>> qemu_mlock: have only locked 1940582400 of 4294967296 bytes; still
> >> trying...
> >>>>> qemu_mlock: have only locked 1940582400 of 4294967296 bytes; still
> >> trying...
> >>>>> qemu_mlock: have only locked 1940582400 of 4294967296 bytes; still
> >> trying…
> >>>>>
> >>>>> I got 15G free memory in the system, why qemu-system-x86_64 can not
> >> locked enough memory ？
> >>>> What does "vmstat 1 5" say prior to your launch of the third VM?
> >>>
> >>> I suspect it will show you have free memory available, but that what
> >>> is really happening is getting here:
> >>>
> >>>
> >>
> https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/vm/seg_vn.c#L7989-L8002
> >>>
> >>> This is likely failing in page_pp_lock() because "availrmem" has
> >>> fallen below "pages_pp_maximum":
> >>>
> >>>
> >>
> https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/vm/vm_page.c#L3817-L3818
> >>>
> >>> We set this value here, though it can be overridden in "/etc/system":
> >>>
> >>>
> >>
> https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/vm/vm_page.c#L423-L436
> >>>
> >>> You can look at the current values with mdb:
> >>>
> >>> mdb -ke 'availrmem/D ; pages_pp_maximum/D'
> >>>
> >>> Increasing this value doesn't seem to be without risk: I believe that
> >>> it can lead to memory exhaustion deadlocks, amongst other things. I
> >>> don't know if it's expected to be tuneable without a reboot.
> >>>
> >>>
> >>> Cheers.
> >>>
> >>> --
> >>> Joshua M. Clulow
> >>> UNIX Admin/Developer
> >>> http://blog.sysmgr.org
> >>
> >> _______________________________________________
> >> OmniOS-discuss mailing list
> >> OmniOS-discuss at lists.omniti.com
> >> http://lists.omniti.com/mailman/listinfo/omnios-discuss
> >>
> >>
> >>
> >>
> >> ------------------------------
> >> Fucai Liang
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> openindiana-discuss mailing list
> >> openindiana-discuss at openindiana.org
> >> http://openindiana.org/mailman/listinfo/openindiana-discuss
> >>
> > _______________________________________________
> > openindiana-discuss mailing list
> > openindiana-discuss at openindiana.org
> > http://openindiana.org/mailman/listinfo/openindiana-discuss
>
> _______________________________________________
> openindiana-discuss mailing list
> openindiana-discuss at openindiana.org
> http://openindiana.org/mailman/listinfo/openindiana-discuss
>
More information about the openindiana-discuss
mailing list