[OpenIndiana-discuss] [zfs-discuss] slow speed problem with a new SAS shelf

[Sašo Kiselkov]

On 08/27/2012 12:58 PM, Yuri Vorobyev wrote:
> 27.08.2012 14:43, Sašo Kiselkov пишет:
> 
>>> Is there any way to disable ARC for testing and leave prefetch enabled?
>>
>> No. The reason is quite simply because prefetch is a mechanism separate
>> from your direct application's read requests. Prefetch runs on ahead of
>> your anticipated read requests and places blocks it expects you'll need
>> in the ARC, so obviously by disabling the ARC, you've disabled prefetch
>> as well.
>>
>> You can get around the problem by exporting and importing the dataset
>> between testing runs, which will clear the ARC, so do:
>>
>> # dd if=/dev/zero of=testfile bs=1024k count=10000
>> # zpool export sas1
>> # zpool import sas1
>> # dd if=testfile of=/dev/null bs=1024k
> 
> Thank you very much, Sašo.

You're very welcome.

> Now i see hardware works without problem.
> 
> I create another 10-disks pair of mirrors zpool for testing:
> 
> root at atom:/# zpool export sas2 ; zpool import sas2
> root at atom:/# readspeed /sas2/5g
> 5120+0 records in
> 5120+0 records out
> 5368709120 bytes (5.4 GB) copied, 5.73728 s, 936 MB/s
> root at atom:/# zpool export sas2 ; zpool import sas2
> root at atom:/# readspeed /sas2/5g
> 5120+0 records in
> 5120+0 records out
> 5368709120 bytes (5.4 GB) copied, 5.63869 s, 952 MB/s

Sounds about right, that's ~94MB/s from each drive. Have you tried
running multiple dd's in parallel? ZFS likes to have its pipelines
fairly saturated, so chances you'll get higher total performance numbers
with multiple parallel readers, like this:

(create multiple files like /sas2/5g_1, /sas2/5g_2, /sas2/5g_3, etc...)
# zpool export sas2 && zpool import sas2
# readspeed /sas2/5g_1 & readspeed /sas2/5g_2 & readspeed /sas2/5g_3

The simply sum up the MB/s from each dd operation. Also, if possible,
use larger files, not just 5GB - something on the order a few 100GBs.
You can then watch your pool's performance via "zpool iostat sas2 5"
(that's how I usually do it).

Cheers,
--
Saso