[OpenIndiana-discuss] ISCSI SSD Performance

[Grant Albitz]

I found the write culprit, write back cache was enabled in the raid0s being presented to zfs

-----Original Message-----
From: Grant Albitz 
Sent: Sunday, February 17, 2013 9:23 PM
To: openindiana-discuss at openindiana.org
Subject: RE: [OpenIndiana-discuss] ISCSI SSD Performance

Here are some interesting results.

If I put 8 ssds in a raid 10 I get about 1.2Gbps write

Read is less but good at 1.1GBps, which I don't understand why its less but here is the iostat for the reads, I am listing just one drives iostat results to keep the message shorter. In all cases all of the drives are consistent with each other.

   r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
1120.6    0.0 142248.1    0.0  0.0  2.2    0.0    2.0   2  38 c1t0d0

If I double the ssds to 16 the writes go down drastically to 421MBps. There doesn't seem to be a specific device dragging me down. I can understand diminishing returns as far as drives go but I wouldn't expect adding drives to drastically cut the performance.

r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0  427.4    0.0 52047.4  0.0  9.7    0.0   22.7   1  98 c1t0d0

The reads stay the same

r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
  550.2    0.0 70019.3    0.0  0.0  0.9    0.0    1.6   1  25 c1t0d0


If I add all 24 drives to the raid 10 I see the following for writes: 355MB/S

r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0  233.2    0.0 28410.7  0.0  9.4    0.0   40.5   0  97 c1t0d0


And reads stay the same at 1.1GBps
  r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
  369.0    0.0 47029.6    0.0  0.0  0.7    0.0    1.9   1  20 c1t0d0

Presently I have each drive in a raid 0 presented through a perc h710. I suppose it could just be tripping over itself. I originally had a h310 in the server but the drivers weren't really mature enough and I was actually having the entire system go down.


Just wondering if anyone has any recommendations. At this point I could just start creating multiple pools