[OpenIndiana-discuss] Appalling disk performance for KVM
Andy Lubel
alubel at gmail.com
Wed Jan 4 05:43:39 UTC 2012
Use default block size. Larger the better. As for pinpointing issues, try iostat -xnc and look at iowait. Another thing that can help is some dtrace in opt/dtt and look at your write sizes. Run atto disk bench in the vm if it's windows.
I would expect-a-guess a single sequential write around 200 and a read at about 300 mb/sec for that setup. Peel back the layers of KVM if needed to track the issue.
On Jan 3, 2012, at 23:50, Russell Hansen <russhan at new-swankton.net> wrote:
> I'm seeing absolutely terrible disk performance for my virtual machines running under KVM.
>
> My ZFS pool is 6 Western Digital AV-25 500GB disks arranged in 3 mirrored vdevs connected to an LSI 9210 (reflashed IBM M1015) in IT mode.
>
> My KVM startup script:
>
> /usr/bin/qemu-kvm \
> -enable-kvm \
> -smp 2 \
> -m 1024 \
> -no-hpet \
> -rtc base=localtime,driftfix=slew \
> -drive file=/dev/zvol/dsk/oi_data/kvm/system/disk0,if=virtio,format=raw,cache=none,index=0 \
> -drive file=/dev/zvol/dsk/oi_data/kvm/system/disk1,if=virtio,format=raw,cache=none,index=1 \
> -net nic,vlan=0,name=lan0,model=virtio,macaddr=$MAC0 \
> -net vnic,vlan=0,name=lan0,ifname=$ETH0 \
> -net nic,vlan=1,name=int0,model=virtio,macaddr=$MAC1 \
> -net vnic,vlan=1,name=int0,ifname=$ETH1 \
> -vnc $LAN_ADDR:$1 \
> -monitor telnet:127.0.0.1:444$1,server,nowait \
> -usbdevice tablet \
> -daemonize
>
> My zvol settings:
>
> NAME PROPERTY VALUE SOURCE
> oi_data/kvm/sachiel/disk1 type volume -
> oi_data/kvm/sachiel/disk1 creation Tue Nov 8 11:56 2011 -
> oi_data/kvm/sachiel/disk1 used 298G -
> oi_data/kvm/sachiel/disk1 available 844G -
> oi_data/kvm/sachiel/disk1 referenced 139G -
> oi_data/kvm/sachiel/disk1 compressratio 1.00x -
> oi_data/kvm/sachiel/disk1 reservation none default
> oi_data/kvm/sachiel/disk1 volsize 144G local
> oi_data/kvm/sachiel/disk1 volblocksize 8K -
> oi_data/kvm/sachiel/disk1 checksum on default
> oi_data/kvm/sachiel/disk1 compression off default
> oi_data/kvm/sachiel/disk1 readonly off default
> oi_data/kvm/sachiel/disk1 copies 1 default
> oi_data/kvm/sachiel/disk1 refreservation 149G local
> oi_data/kvm/sachiel/disk1 primarycache all default
> oi_data/kvm/sachiel/disk1 secondarycache all default
> oi_data/kvm/sachiel/disk1 usedbysnapshots 11.8G -
> oi_data/kvm/sachiel/disk1 usedbydataset 139G -
> oi_data/kvm/sachiel/disk1 usedbychildren 0 -
> oi_data/kvm/sachiel/disk1 usedbyrefreservation 147G -
> oi_data/kvm/sachiel/disk1 logbias latency default
> oi_data/kvm/sachiel/disk1 dedup off default
> oi_data/kvm/sachiel/disk1 mlslabel none default
> oi_data/kvm/sachiel/disk1 sync standard default
> oi_data/kvm/sachiel/disk1 refcompressratio 1.00x -
>
> Representative iostat output while moving some 4GB of digital photos (approx 2MB each):
>
> capacity operations bandwidth
> pool alloc free read write read write
> ------------------------- ----- ----- ----- ----- ----- -----
> oi_data 373G 1019G 71 377 569K 3.98M
> mirror 124G 340G 15 127 125K 1.34M
> c2t50014EE2AFBDC3CBd0 - - 2 71 118K 1.34M
> c2t50014EE25A683C10d0 - - 0 69 7.01K 1.34M
> mirror 124G 340G 31 123 256K 1.33M
> c2t50014EE20518D4F6d0 - - 4 49 132K 1.33M
> c2t50014EE2AFBD87B4d0 - - 4 48 123K 1.33M
> mirror 124G 340G 23 125 189K 1.32M
> c2t50014EE205154E0Bd0 - - 3 52 35.1K 1.32M
> c2t50014EE25A6829D6d0 - - 4 51 156K 1.32M
> ------------------------- ----- ----- ----- ----- ----- -----
>
> A Bonnie result from a napp-it install:
>
> Version 1.03c ------Sequential Output------ --Sequential Input- --Random-
> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> 2011.12.12 32G 61586 99 151550 16 95819 17 49188 99 273833 18 1166 3
> ------Sequential Create------ --------Random Create--------
> -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> 16 32070 99 +++++ +++ +++++ +++ 32736 100 +++++ +++ +++++ +++
> 2011.12.12,32G,61586,99,151550,16,95819,17,49188,99,273833,18,1166.1,3,16,32070,99,+++++,+++,+++++,+++,32736,100,+++++,+++,+++++,+++
>
>
> As you can see, it basically hovers around 4-6 MB/s average with an approximate 10KB I/O size so it appears I'm 100% iops limited. Do I need to recreate my zvols with a larger volblocksize to get decent performance or is it something else I need to do on the KVM config side or within the virtual machine?
>
> I have one Windows Server 2003 (32-bit) VM and 2x Windows Server 2008 R2 (64-bit) VMs and they all behave the same way when it comes to disk perf.
>
> Thanks,
>
> -Russ
> _______________________________________________
> OpenIndiana-discuss mailing list
> OpenIndiana-discuss at openindiana.org
> http://openindiana.org/mailman/listinfo/openindiana-discuss
More information about the OpenIndiana-discuss
mailing list