[OpenIndiana-discuss] Inefficient zvol space usage on 4k drives

[Richard Elling]

On Aug 8, 2013, at 11:34 AM, Lionel Cons <lionelcons1972 at gmail.com> wrote:

> On 8 August 2013 17:11, Richard Elling <richard.elling at richardelling.com> wrote:
>> On Aug 7, 2013, at 2:50 PM, Jason Lawrence <jjlawren at gmail.com> wrote:
>> 
>>> This might be a better question for the Illumos group, so please let me know.
>>> 
>>> I have a zvol for a KVM instance which I felt was taking up too much space. After doing a little research, I stumbled upon http://support.freenas.org/ticket/2383 and repeated the test on my machine. I'm running a RAIDZ2 pool with eight "advanced format" 4k sector drives. I created the pool with ashift=12 with this in mind.
>>> 
>>> root at hostname:~# zfs create -V 20g -o volblocksize=32k storage/testbed/32k
>>> root at hostname:~# zfs create -V 20g -o volblocksize=8k storage/testbed/8k
>>> 
>>> root at hostname:~# dd if=/dev/zero of=/dev/zvol/rdsk/storage/testbed/32k bs=1048576 count=20400
>>> 20400+0 records in
>>> 20400+0 records out
>>> 21390950400 bytes (21 GB) copied, 493.836 s, 43.3 MB/s
>>> 
>>> root at hostname:~# dd if=/dev/zero of=/dev/zvol/rdsk/storage/testbed/8k bs=1048576 count=20400
>>> 
>>> 20400+0 records in
>>> 20400+0 records out
>>> 21390950400 bytes (21 GB) copied, 548.916 s, 39.0 MB/s
>>> 
>>> root at hostname:~# zfs list | grep testbed
>>> storage/testbed              64.2G  5.79T   307K  /storage/testbed
>>> storage/testbed/32k          21.3G  5.79T  21.3G  -
>>> storage/testbed/8k           42.8G  5.79T  42.8G  -
>>> 
>>> 
>>> The 8k blocksize zvol takes up twice the space of the 32k blocksize zvol. As 8k is the default, this must be affecting others...
>> 
>> This is expected. For 4K sector sizes and > 9 disks per raidz2 set:
>>        volblocksize = 8k, raidz2 writes 2 data (8k) + 2 parity (8k)  [like mirroring]
>>        volblocksize = 32k, raidz2 writes 8 data (32k) + 2 parity (8k) [like RAID-6]
>> 
>> What most folks miss is:
>>        volblocksize = 4k, raidz2 writes 1 data (4k) + 2 parity (8k) [like triple mirroring]
>> 
>> so, don't do 4k recordsize with 4K sector disks unless you are mirroring.
> 
> What about disks (i.e. iSCSI connected to a hardware raid; the larger
> sector size is intentional to improve throughput) which have even
> larger sectors, e.g. 8k sectors?

In my experience, most iSCSI targets advertise 512 byte sectors, regardless of the
back end. For COMSTAR, this can be changed, so it is an easy way to experiment
with different "physical" block sizes.
 -- richard

--

Richard.Elling at RichardElling.com
+1-760-896-4422