[OpenIndiana-discuss] Appalling disk performance for KVM

[Open Indiana]

What you could have done was:
1. use Virtualbox for your virtual OS's
2. create a minimal C: drive for the OS only
3. create 3 iscsi drives on OpenIndiana from a ZFS drive
4. import these drives as storage in your VM's

Apart from step 1 you could do it right now. I guess you will see improved
performance, going up to 50MB/s




-----Original Message-----
From: Russell Hansen [mailto:russhan at new-swankton.net]
Sent: woensdag 4 januari 2012 5:51
To: Discussion list for OpenIndiana
Subject: [OpenIndiana-discuss] Appalling disk performance for KVM

I'm seeing absolutely terrible disk performance for my virtual machines
running under KVM.

My ZFS pool is 6 Western Digital AV-25 500GB disks arranged in 3 mirrored
vdevs connected to an LSI 9210 (reflashed IBM M1015) in IT mode.

My KVM startup script:

/usr/bin/qemu-kvm \
-enable-kvm \
-smp 2 \
-m 1024 \
-no-hpet \
-rtc base=localtime,driftfix=slew \
-drive
file=/dev/zvol/dsk/oi_data/kvm/system/disk0,if=virtio,format=raw,cache=none,
index=0 \ -drive
file=/dev/zvol/dsk/oi_data/kvm/system/disk1,if=virtio,format=raw,cache=none,
index=1 \ -net nic,vlan=0,name=lan0,model=virtio,macaddr=$MAC0 \ -net
vnic,vlan=0,name=lan0,ifname=$ETH0 \ -net
nic,vlan=1,name=int0,model=virtio,macaddr=$MAC1 \ -net
vnic,vlan=1,name=int0,ifname=$ETH1 \ -vnc $LAN_ADDR:$1 \ -monitor
telnet:127.0.0.1:444$1,server,nowait \ -usbdevice tablet \ -daemonize

My zvol settings:

NAME                       PROPERTY                        VALUE
SOURCE
oi_data/kvm/sachiel/disk1  type                            volume
-
oi_data/kvm/sachiel/disk1  creation                        Tue Nov  8 11:56
2011           -
oi_data/kvm/sachiel/disk1  used                            298G
-
oi_data/kvm/sachiel/disk1  available                       844G
-
oi_data/kvm/sachiel/disk1  referenced                      139G
-
oi_data/kvm/sachiel/disk1  compressratio                   1.00x
-
oi_data/kvm/sachiel/disk1  reservation                     none
default
oi_data/kvm/sachiel/disk1  volsize                         144G
local
oi_data/kvm/sachiel/disk1  volblocksize                    8K
-
oi_data/kvm/sachiel/disk1  checksum                        on
default
oi_data/kvm/sachiel/disk1  compression                     off
default
oi_data/kvm/sachiel/disk1  readonly                        off
default
oi_data/kvm/sachiel/disk1  copies                          1
default
oi_data/kvm/sachiel/disk1  refreservation                  149G
local
oi_data/kvm/sachiel/disk1  primarycache                    all
default
oi_data/kvm/sachiel/disk1  secondarycache                  all
default
oi_data/kvm/sachiel/disk1  usedbysnapshots                 11.8G
-
oi_data/kvm/sachiel/disk1  usedbydataset                   139G
-
oi_data/kvm/sachiel/disk1  usedbychildren                  0
-
oi_data/kvm/sachiel/disk1  usedbyrefreservation            147G
-
oi_data/kvm/sachiel/disk1  logbias                         latency
default
oi_data/kvm/sachiel/disk1  dedup                           off
default
oi_data/kvm/sachiel/disk1  mlslabel                        none
default
oi_data/kvm/sachiel/disk1  sync                            standard
default
oi_data/kvm/sachiel/disk1  refcompressratio                1.00x
-

Representative iostat output while moving some 4GB of digital photos (approx
2MB each):

                              capacity     operations    bandwidth
pool                       alloc   free   read  write   read  write
-------------------------  -----  -----  -----  -----  -----  -----
oi_data                     373G  1019G     71    377   569K  3.98M
  mirror                    124G   340G     15    127   125K  1.34M
    c2t50014EE2AFBDC3CBd0      -      -      2     71   118K  1.34M
    c2t50014EE25A683C10d0      -      -      0     69  7.01K  1.34M
  mirror                    124G   340G     31    123   256K  1.33M
    c2t50014EE20518D4F6d0      -      -      4     49   132K  1.33M
    c2t50014EE2AFBD87B4d0      -      -      4     48   123K  1.33M
  mirror                    124G   340G     23    125   189K  1.32M
    c2t50014EE205154E0Bd0      -      -      3     52  35.1K  1.32M
    c2t50014EE25A6829D6d0      -      -      4     51   156K  1.32M
-------------------------  -----  -----  -----  -----  -----  -----

A Bonnie result from a napp-it install:

Version 1.03c       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
%CP
2011.12.12      32G 61586  99 151550  16 95819  17 49188  99 273833  18
1166   3
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
%CP
                 16 32070  99 +++++ +++ +++++ +++ 32736 100 +++++ +++ +++++
+++
2011.12.12,32G,61586,99,151550,16,95819,17,49188,99,273833,18,1166.1,3,16,32
070,99,+++++,+++,+++++,+++,32736,100,+++++,+++,+++++,+++


As you can see, it basically hovers around 4-6 MB/s average with an
approximate 10KB I/O size so it appears I'm 100% iops limited.  Do I need to
recreate my zvols with a larger volblocksize to get decent performance or is
it something else I need to do on the KVM config side or within the virtual
machine?

I have one Windows Server 2003 (32-bit) VM and 2x Windows Server 2008 R2
(64-bit) VMs and they all behave the same way when it comes to disk perf.

Thanks,

-Russ
_______________________________________________
OpenIndiana-discuss mailing list
OpenIndiana-discuss at openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss