[OpenIndiana-discuss] very slow zpool
Martin Frost
me at cs.stanford.edu
Fri Jan 1 01:02:17 UTC 2016
Thanks, Bob and Ian.
First, dedup is off everywhere.
Below is the output of the longer iostat command for a few intervals.
There are some small error counts under trn (transport errors);
these counts are for 604 days, so they seem pretty low.
Otherwise nothing jumps out at me here. Let me know if
you see anything interesting below. The scrub is still running,
so it is probably generating almost all the activity below.
Martin
tty cpu
tin tout us sy wt id
0 79 0 4 0 96
extended device statistics ---- errors
---
r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b s/w h/w trn
tot device
0.1 17.4 0.0 0.1 0.0 0.0 0.0 0.5 0 0 0 0 0
0 c3t0d0
0.1 17.2 0.0 0.1 0.0 0.0 0.0 0.5 0 0 0 0 0
0 c3t1d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0
0 c3t2d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0
0 c3t3d0
94.9 10.3 3.5 0.0 0.0 1.0 0.0 9.8 0 35 0 0 2
2 c1t5000C50035063BC3d0
106.3 14.3 3.5 0.0 0.0 1.1 0.0 8.8 0 35 0 0 0
0 c1t5000C5003505A723d0
106.3 14.5 3.5 0.0 0.0 1.1 0.0 9.0 0 36 0 0 0
0 c1t5000C5003504E1F3d0
92.5 10.4 3.4 0.0 0.0 1.0 0.0 9.7 0 34 0 0 2
2 c1t5000C50035070BE3d0
107.5 15.0 3.6 0.0 0.0 1.1 0.0 8.9 0 37 0 0 0
0 c1t5000C5003505BF17d0
106.8 15.3 3.6 0.0 0.0 1.2 0.0 9.7 0 37 0 0 0
0 c1t5000C5003503C847d0
93.6 10.4 3.5 0.0 0.0 1.1 0.0 10.4 0 35 0 0 2
2 c1t5000C50035015617d0
105.8 15.0 3.5 0.0 0.0 1.1 0.0 9.3 0 36 0 0 0
0 c1t5000C5003505EC87d0
107.4 14.8 3.6 0.0 0.0 1.1 0.0 9.1 0 36 0 0 0
0 c1t5000C50034FE2237d0
107.8 15.0 3.6 0.0 0.0 1.2 0.0 9.4 0 36 0 0 0
0 c1t5000C50035012E7Bd0
92.8 10.6 3.5 0.0 0.0 1.1 0.0 10.5 0 35 0 0 1
1 c1t5000C5003506371Bd0
93.4 10.4 3.5 0.0 0.0 1.1 0.0 10.2 0 34 0 0 0
0 c1t5000C5003505066Bd0
105.4 14.8 3.5 0.0 0.0 1.1 0.0 9.4 0 36 0 0 0
0 c1t5000C5003504BF8Bd0
93.1 10.7 3.5 0.0 0.0 1.1 0.0 10.5 0 34 0 0 1
1 c1t5000C5003506835Bd0
94.0 10.5 3.6 0.0 0.0 1.1 0.0 10.9 0 36 0 0 2
2 c1t5000C5003505954Fd0
94.9 10.2 3.6 0.0 0.0 1.1 0.0 10.9 0 35 0 0 2
2 c1t5000C50035043D4Fd0
108.6 15.4 3.6 0.0 0.0 1.2 0.0 9.5 0 37 0 0 0
0 c1t5000C50034FCD30Fd0
93.5 10.8 3.6 0.0 0.0 1.1 0.0 10.6 0 36 0 0 2
2 c1t5000C5002604233Fd0
tty cpu
tin tout us sy wt id
0 79 0 4 0 96
extended device statistics ---- errors
---
r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b s/w h/w trn
tot device
0.2 8.9 0.0 0.0 0.0 0.0 0.0 0.4 0 0 0 0 0
0 c3t0d0
0.1 9.0 0.0 0.0 0.0 0.0 0.0 0.5 0 0 0 0 0
0 c3t1d0
0.0 0.1 0.0 0.0 0.0 0.0 0.0 3.8 0 0 0 0 0
0 c3t2d0
0.0 0.1 0.0 0.0 0.0 0.0 0.0 3.8 0 0 0 0 0
0 c3t3d0
83.4 11.4 2.5 0.0 0.0 0.6 0.0 6.3 0 27 0 0 2
2 c1t5000C50035063BC3d0
99.6 12.8 2.7 0.0 0.0 0.7 0.0 6.5 0 30 0 0 0
0 c1t5000C5003505A723d0
100.0 12.6 2.8 0.0 0.0 0.7 0.0 6.2 0 29 0 0 0
0 c1t5000C5003504E1F3d0
84.4 11.1 2.5 0.0 0.0 0.6 0.0 6.3 0 27 0 0 2
2 c1t5000C50035070BE3d0
99.8 12.7 2.7 0.0 0.0 0.7 0.0 6.6 0 29 0 0 0
0 c1t5000C5003505BF17d0
99.2 12.5 2.7 0.0 0.0 0.7 0.0 6.5 0 30 0 0 0
0 c1t5000C5003503C847d0
83.7 11.5 2.6 0.0 0.0 0.7 0.0 6.9 0 28 0 0 2
2 c1t5000C50035015617d0
99.1 12.9 2.7 0.0 0.0 0.7 0.0 6.3 0 29 0 0 0
0 c1t5000C5003505EC87d0
97.7 12.8 2.6 0.0 0.0 0.7 0.0 6.1 0 28 0 0 0
0 c1t5000C50034FE2237d0
99.0 12.8 2.7 0.0 0.0 0.7 0.0 6.2 0 29 0 0 0
0 c1t5000C50035012E7Bd0
86.9 11.3 2.6 0.0 0.0 0.7 0.0 7.2 0 28 0 0 1
1 c1t5000C5003506371Bd0
88.8 11.5 2.7 0.0 0.0 0.7 0.0 6.7 0 28 0 0 0
0 c1t5000C5003505066Bd0
100.1 12.9 2.8 0.0 0.0 0.7 0.0 6.6 0 30 0 0 0
0 c1t5000C5003504BF8Bd0
84.8 11.4 2.6 0.0 0.0 0.6 0.0 6.4 0 27 0 0 1
1 c1t5000C5003506835Bd0
86.2 11.3 2.6 0.0 0.0 0.7 0.0 6.8 0 28 0 0 2
2 c1t5000C5003505954Fd0
84.3 11.7 2.6 0.0 0.0 0.6 0.0 6.7 0 28 0 0 2
2 c1t5000C50035043D4Fd0
98.4 12.5 2.7 0.0 0.0 0.7 0.0 6.6 0 29 0 0 0
0 c1t5000C50034FCD30Fd0
88.1 10.9 2.7 0.0 0.0 0.7 0.0 7.3 0 29 0 0 2
2 c1t5000C5002604233Fd0
tty cpu
tin tout us sy wt id
0 79 0 4 0 96
extended device statistics ---- errors
---
r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b s/w h/w trn
tot device
0.0 2.6 0.0 0.0 0.0 0.0 0.0 0.5 0 0 0 0 0
0 c3t0d0
0.0 2.6 0.0 0.0 0.0 0.0 0.0 0.5 0 0 0 0 0
0 c3t1d0
0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0
0 c3t2d0
0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0
0 c3t3d0
102.8 13.8 3.2 0.0 0.0 0.8 0.0 6.9 0 31 0 0 2
2 c1t5000C50035063BC3d0
100.1 15.6 3.1 0.0 0.0 0.9 0.0 7.5 0 31 0 0 0
0 c1t5000C5003505A723d0
97.9 15.1 3.1 0.0 0.0 0.8 0.0 7.1 0 30 0 0 0
0 c1t5000C5003504E1F3d0
102.0 14.1 3.1 0.0 0.0 0.8 0.0 6.6 0 31 0 0 2
2 c1t5000C50035070BE3d0
103.1 15.1 3.2 0.0 0.0 0.8 0.0 7.0 0 31 0 0 0
0 c1t5000C5003505BF17d0
98.2 14.5 3.1 0.0 0.0 0.9 0.0 7.6 0 31 0 0 0
0 c1t5000C5003503C847d0
104.5 13.6 3.1 0.0 0.0 0.9 0.0 7.4 0 32 0 0 2
2 c1t5000C50035015617d0
99.8 15.3 3.1 0.0 0.0 0.8 0.0 7.4 0 31 0 0 0
0 c1t5000C5003505EC87d0
98.1 14.7 3.0 0.0 0.0 0.8 0.0 7.2 0 30 0 0 0
0 c1t5000C50034FE2237d0
98.6 14.8 3.1 0.0 0.0 0.8 0.0 7.2 0 30 0 0 0
0 c1t5000C50035012E7Bd0
103.5 13.5 3.1 0.0 0.0 0.9 0.0 7.4 0 32 0 0 1
1 c1t5000C5003506371Bd0
102.7 13.4 3.1 0.0 0.0 0.9 0.0 7.5 0 33 0 0 0
0 c1t5000C5003505066Bd0
95.4 14.4 3.1 0.0 0.0 0.8 0.0 7.2 0 30 0 0 0
0 c1t5000C5003504BF8Bd0
101.4 13.8 3.1 0.0 0.0 0.8 0.0 7.0 0 31 0 0 1
1 c1t5000C5003506835Bd0
103.3 13.4 3.1 0.0 0.0 0.9 0.0 7.5 0 32 0 0 2
2 c1t5000C5003505954Fd0
104.1 13.9 3.2 0.0 0.0 0.9 0.0 7.3 0 31 0 0 2
2 c1t5000C50035043D4Fd0
97.1 14.6 3.0 0.0 0.0 0.9 0.0 7.7 0 31 0 0 0
0 c1t5000C50034FCD30Fd0
103.4 13.0 3.2 0.0 0.0 0.9 0.0 7.8 0 33 0 0 2
2 c1t5000C5002604233Fd0
On Thu, Dec 31, 2015 at 1:32 PM, Ian Collins <ian at ianshome.com> wrote:
> Bob Friesenhahn wrote:
>
>> On Thu, 31 Dec 2015, Martin Frost wrote:
>>
>>> Any thoughts about what could be causing the slowness?
>>>
>> Run 'iostat -xe' in order to see if a disk is reporting errors or is
>> abnormally slow.
>>
>
> I'd extend the options to get the pool summary and repeat every 30 seconds
> or so to see if any drive is holding up the pool:
>
> iostat -xetcMn 30
>
> --
> Ian.
>
>
>
> _______________________________________________
> openindiana-discuss mailing list
> openindiana-discuss at openindiana.org
> http://openindiana.org/mailman/listinfo/openindiana-discuss
>
More information about the openindiana-discuss
mailing list