[OpenIndiana-discuss] [zfs-discuss] slow speed problem with a new SAS shelf
Sašo Kiselkov
skiselkov.ml at gmail.com
Mon Jul 23 16:09:59 UTC 2012
Hi,
Have you had a look iostat -E (error counters) to make sure you don't
have faulty cabling? I've bad cables trip me up once in a manner similar
to your situation here.
Cheers,
--
Saso
On 07/23/2012 07:18 AM, Yuri Vorobyev wrote:
> Hello.
>
> I faced with a strange performance problem with new disk shelf.
> We a using ZFS system with SATA disks for a while.
> It is Supermicro SC846-E16 chassis, Supermicro X8DTH-6F motherboard with
> 96Gb RAM and 24 HITACHI HDS723020BLA642 SATA disks attached to onboard
> LSI 2008 controller.
>
> Pretty much satisfied with it we bought additional shelf with SAS disks
> for VMs hosting. New shelf is Supermicro SC846-E26 chassis. Disks model
> is HITACHI HUS156060VLS600 (15K 600Gb SAS2).
> Additional controller LSI 9205-8e was installed in server and connected
> with JBOD.
> I connected JBOD with 2 channels and setup multi path first, but when i
> noticed performance problem i disabled multi path and disconnected one
> cable (for sure it is not multipath cause the problem).
>
> Problem description follow:
>
> Creating test pool with 5 pair of mirrors (new shelf, SAS disks)
>
> # zpool create -o version=28 -O primarycache=none test mirror
> c9t5000CCA02A138899d0 c9t5000CCA02A102181d0 mirror c9t5000CCA02A13500Dd0
> c9t5000CCA02A13316Dd0 mirror c9t5000CCA02A005699d0 c9t5000CCA02A004271d0
> mirror c9t5000CCA02A004229d0 c9t5000CCA02A1342CDd0 mirror
> c9t5000CCA02A1251E5d0 c9t5000CCA02A1151DDd0
>
> (primarycache=none) to disable ARC influence
>
>
> Testing sequential write
> # dd if=/dev/zero of=/test/zero bs=1M count=2048
> 2048+0 records in
> 2048+0 records out
> 2147483648 bytes (2.1 GB) copied, 1.04272 s, 2.1 GB/s
>
> iostat when writing look like
> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
> 0.0 1334.6 0.0 165782.9 0.0 8.4 0.0 6.3 1 86
> c9t5000CCA02A1151DDd0
> 0.0 1345.5 0.0 169575.3 0.0 8.7 0.0 6.5 1 88
> c9t5000CCA02A1342CDd0
> 2.0 1359.5 1.0 168969.8 0.0 8.7 0.0 6.4 1 90
> c9t5000CCA02A13500Dd0
> 0.0 1358.5 0.0 168714.0 0.0 8.7 0.0 6.4 1 90
> c9t5000CCA02A13316Dd0
> 0.0 1345.5 0.0 166669.3 0.0 9.0 0.0 6.7 1 92
> c9t5000CCA02A102181d0
> 1.0 1317.5 1.0 164456.9 0.0 8.5 0.0 6.5 1 88
> c9t5000CCA02A004271d0
> 4.0 1342.5 2.0 166282.2 0.0 8.5 0.0 6.3 1 88
> c9t5000CCA02A1251E5d0
> 0.0 1377.5 0.0 170515.5 0.0 8.7 0.0 6.3 1 90
> c9t5000CCA02A138899d0
>
> Now read
> # dd if=/test/zero of=/dev/null bs=1M
> 2048+0 records in
> 2048+0 records out
> 2147483648 bytes (2.1 GB) copied, 13.5681 s, 158 MB/s
>
> iostat when reading
> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
> 106.0 0.0 11417.4 0.0 0.0 0.2 0.0 2.4 0 14
> c9t5000CCA02A004271d0
> 80.0 0.0 10239.9 0.0 0.0 0.2 0.0 2.4 0 10
> c9t5000CCA02A1251E5d0
> 110.0 0.0 12182.4 0.0 0.0 0.1 0.0 1.3 0 9
> c9t5000CCA02A138899d0
> 102.0 0.0 11664.4 0.0 0.0 0.2 0.0 1.8 0 15
> c9t5000CCA02A005699d0
> 99.0 0.0 10900.9 0.0 0.0 0.3 0.0 3.0 0 16
> c9t5000CCA02A004229d0
> 107.0 0.0 11545.4 0.0 0.0 0.2 0.0 1.9 0 13
> c9t5000CCA02A1151DDd0
> 81.0 0.0 10367.9 0.0 0.0 0.2 0.0 2.2 0 11
> c9t5000CCA02A1342CDd0
>
> Unexpected low speed! Note the busy column. When writing it about 90%,
> when reading it about 15%
>
> Individual disks raw read speed (don't be confused with name change. i
> connect JBOD to another HBA channel)
>
> # dd if=/dev/dsk/c8t5000CCA02A13889Ad0 of=/dev/null bs=1M count=2000
> 2000+0 records in
> 2000+0 records out
> 2097152000 bytes (2.1 GB) copied, 10.9685 s, 191 MB/s
> # dd if=/dev/dsk/c8t5000CCA02A1342CEd0 of=/dev/null bs=1M count=2000
> 2000+0 records in
> 2000+0 records out
> 2097152000 bytes (2.1 GB) copied, 10.8024 s, 194 MB/s
>
> The 10-disks mirror zpool read slower than a single disk.
>
> There is no tuning in /etc/system
>
> I tried test with FreeBSD 8.3 live CD. Reads was the same (about
> 150Mb/s). Also i tried SmartOS, but it can't see disks behind LSI
> 9205-8e controller.
>
> For compare this is speed from SATA pool (it consist of 4 6-disk raidz2
> vdev)
> #dd if=CentOS-6.2-x86_64-bin-DVD1.iso of=/dev/null bs=1M
> 4218+1 records in
> 4218+1 records out
> 4423129088 bytes (4.4 GB) copied, 4.76552 s, 928 MB/s
>
> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
> 13614.4 0.0 800338.5 0.0 0.1 36.0 0.0 2.6 0 914 c6
> 459.9 0.0 25761.4 0.0 0.0 0.8 0.0 1.8 0 22
> c6t5000CCA369D16860d0
> 84.0 0.0 2785.2 0.0 0.0 0.2 0.0 3.0 0 13
> c6t5000CCA369D1B1E0d0
> 836.9 0.0 50089.5 0.0 0.0 2.6 0.0 3.1 0 60
> c6t5000CCA369D1B302d0
> 411.0 0.0 24492.6 0.0 0.0 0.8 0.0 2.1 0 25
> c6t5000CCA369D16982d0
> 821.9 0.0 49385.1 0.0 0.0 3.0 0.0 3.7 0 67
> c6t5000CCA369CFBDA3d0
> 231.0 0.0 12292.5 0.0 0.0 0.5 0.0 2.3 0 18
> c6t5000CCA369D17E73d0
> 803.9 0.0 50091.5 0.0 0.0 2.9 0.0 3.6 1 69
> c6t5000CCA369D0EA93d0
>
> PS. Before testing i flash last firmware and bios to LSI 9205-8e. It
> come with factory 9 version. I flashed version 13.5.
> Now I think that it was not worth such a hurry.
> Then i downgrade it to version 12. Read speed remains the same.
> Now controllers versions
>
> # ./sas2flash -listall
> LSI Corporation SAS2 Flash Utility
> Version 12.00.00.00 (2011.11.08)
> Copyright (c) 2008-2011 LSI Corporation. All rights reserved
>
> Adapter Selected is a LSI SAS: SAS2008(B1)
>
> Num Ctlr FW Ver NVDATA x86-BIOS PCI
> Addr
>
> ----------------------------------------------------------------------------
>
>
> 0 SAS2008(B1) 12.00.00.00 0c.00.00.04 07.23.01.00 00:05:00:00
> 1 SAS2308_2(B0) 12.00.00.00 0c.00.00.04 07.23.01.00 00:84:00:00
>
> Any suggestions or thoughts ?
>
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
More information about the OpenIndiana-discuss
mailing list