[OpenIndiana-discuss] RAIDZ performance

[Sašo Kiselkov]

On 02/20/2013 08:05 PM, Reginald Beardsley wrote:
> On an N40L running oi_151a7 w/ four ST2000DM001 drives I'm seeing
> a large drop in performance for RAIDZ2 vs RAIDZ1 which surprises me.
> 
> The discussions google found were not entirely enlightening and not
> OI based.  How much CPU does a small OI based ZFS server need?
> 
> For 3 disk RAIDZ1 I get 189-199 MB/s and 179 MB/s for 4 disk RAIDZ1.
> But for 4 disk RAIDZ2 I get 109-118 MB/s.  I expected some loss in
> performance, but not that much.  These are measured writing 64 GB
> of /dev/zero to the RAIDZ filesystem from a console window.  
> 
> For a 256 GB file I got 111 MB/s writing and 279 MB/s reading 4
> disk RAIDZ2.
> 
> System is unloaded w/ 4 GB of ECC DRAM using a 4-way mirror rpool
> in the s0 slices w/ RAIDZ in the s1 slices.
> 
> Does the drop in RAIDZ2 write performance correspond to other
> people's experience? If so, why such a large hit? top suggests the
> system has spare CPU capacity (30-50% idle). For my current needs
> things look good, but I'd like to understand why a bit better.
> Or if there is some tuning I should do. 

RAID-Z2 is quite CPU intensive, since it switches from XOR parity (for
RAID-Z1) to a Reed-Solomon error-correcting code (in order to be able to
survive multiple drive failures). The N40L's piddly CPU (2 cores @
1.4GHz) probably doesn't like that very much. 30-50% idle means 1 core
completely saturated and the other running up to ~50% saturation. Try to
switch to RAID-Z1 and watch as your CPU utilization drops dramatically -
that's the source of your pain.

Cheers,
--
Saso