[OpenIndiana-discuss] very slow zpool
Martin Frost
me at cs.Stanford.EDU
Fri Jan 1 01:17:36 UTC 2016
Oh, here's iostat output from after I stopped the slow scrub just now
(with more activity than I expected so maybe someone is working late
on New Year's Eve).
Martin
tty cpu
tin tout us sy wt id
0 83 0 4 0 96
extended device statistics ---- errors ---
r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b s/w h/w trn tot device
0.1 3.2 0.0 0.0 0.0 0.0 0.0 0.2 0 0 0 0 0 0 c3t0d0
0.3 3.0 0.0 0.0 0.0 0.0 0.0 0.3 0 0 0 0 0 0 c3t1d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 c3t2d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 c3t3d0
95.1 0.0 2.0 0.0 0.0 0.5 0.0 5.5 0 19 0 0 2 2 c1t5000C50035063BC3d0
84.4 0.0 1.9 0.0 0.0 0.5 0.0 5.7 0 18 0 0 0 0 c1t5000C5003505A723d0
85.3 0.0 2.0 0.0 0.0 0.5 0.0 6.0 0 19 0 0 0 0 c1t5000C5003504E1F3d0
94.6 0.0 2.0 0.0 0.0 0.5 0.0 5.5 0 19 0 0 2 2 c1t5000C50035070BE3d0
86.5 0.0 2.0 0.0 0.0 0.5 0.0 5.6 0 19 0 0 0 0 c1t5000C5003505BF17d0
83.3 0.0 1.9 0.0 0.0 0.5 0.0 5.7 0 18 0 0 0 0 c1t5000C5003503C847d0
93.3 0.0 2.0 0.0 0.0 0.5 0.0 5.6 0 19 0 0 2 2 c1t5000C50035015617d0
85.1 0.0 1.9 0.0 0.0 0.5 0.0 5.6 0 19 0 0 0 0 c1t5000C5003505EC87d0
82.1 0.0 1.8 0.0 0.0 0.5 0.0 6.0 0 17 0 0 0 0 c1t5000C50034FE2237d0
85.2 0.0 1.9 0.0 0.0 0.5 0.0 5.6 0 18 0 0 0 0 c1t5000C50035012E7Bd0
91.8 0.0 1.9 0.0 0.0 0.5 0.0 5.6 0 19 0 0 1 1 c1t5000C5003506371Bd0
93.2 0.0 1.9 0.0 0.0 0.5 0.0 5.5 0 20 0 0 0 0 c1t5000C5003505066Bd0
84.1 0.0 1.9 0.0 0.0 0.5 0.0 5.8 0 18 0 0 0 0 c1t5000C5003504BF8Bd0
94.6 0.0 1.9 0.0 0.0 0.5 0.0 5.6 0 20 0 0 1 1 c1t5000C5003506835Bd0
93.2 0.0 1.9 0.0 0.0 0.5 0.0 5.5 0 19 0 0 2 2 c1t5000C5003505954Fd0
93.1 0.0 2.0 0.0 0.0 0.5 0.0 5.6 0 19 0 0 2 2 c1t5000C50035043D4Fd0
84.4 0.0 1.9 0.0 0.0 0.5 0.0 5.4 0 19 0 0 0 0 c1t5000C50034FCD30Fd0
92.9 0.0 2.0 0.0 0.0 0.5 0.0 5.6 0 19 0 0 2 2 c1t5000C5002604233Fd0
tty cpu
tin tout us sy wt id
0 79 0 4 0 96
extended device statistics ---- errors ---
r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b s/w h/w trn tot device
0.0 5.3 0.0 0.0 0.0 0.0 0.0 0.5 0 0 0 0 0 0 c3t0d0
0.0 5.3 0.0 0.0 0.0 0.0 0.0 0.5 0 0 0 0 0 0 c3t1d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 c3t2d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 c3t3d0
97.4 0.0 2.3 0.0 0.0 0.4 0.0 4.0 0 18 0 0 2 2 c1t5000C50035063BC3d0
100.1 0.0 2.3 0.0 0.0 0.4 0.0 3.6 0 17 0 0 0 0 c1t5000C5003505A723d0
100.7 0.0 2.3 0.0 0.0 0.4 0.0 3.8 0 16 0 0 0 0 c1t5000C5003504E1F3d0
96.1 0.0 2.3 0.0 0.0 0.4 0.0 4.0 0 17 0 0 2 2 c1t5000C50035070BE3d0
100.6 0.0 2.3 0.0 0.0 0.4 0.0 3.6 0 17 0 0 0 0 c1t5000C5003505BF17d0
97.5 0.0 2.2 0.0 0.0 0.3 0.0 3.6 0 16 0 0 0 0 c1t5000C5003503C847d0
95.9 0.0 2.3 0.0 0.0 0.4 0.0 3.7 0 17 0 0 2 2 c1t5000C50035015617d0
100.4 0.0 2.3 0.0 0.0 0.3 0.0 3.5 0 16 0 0 0 0 c1t5000C5003505EC87d0
97.8 0.0 2.2 0.0 0.0 0.4 0.0 3.7 0 16 0 0 0 0 c1t5000C50034FE2237d0
100.8 0.0 2.3 0.0 0.0 0.4 0.0 3.7 0 17 0 0 0 0 c1t5000C50035012E7Bd0
94.2 0.0 2.2 0.0 0.0 0.4 0.0 4.2 0 17 0 0 1 1 c1t5000C5003506371Bd0
92.5 0.0 2.2 0.0 0.0 0.4 0.0 4.0 0 17 0 0 0 0 c1t5000C5003505066Bd0
99.4 0.0 2.3 0.0 0.0 0.4 0.0 4.2 0 18 0 0 0 0 c1t5000C5003504BF8Bd0
94.5 0.0 2.2 0.0 0.0 0.4 0.0 3.8 0 16 0 0 1 1 c1t5000C5003506835Bd0
95.6 0.0 2.3 0.0 0.0 0.4 0.0 4.1 0 17 0 0 2 2 c1t5000C5003505954Fd0
97.3 0.0 2.3 0.0 0.0 0.4 0.0 3.9 0 18 0 0 2 2 c1t5000C50035043D4Fd0
98.7 0.0 2.3 0.0 0.0 0.4 0.0 3.8 0 17 0 0 0 0 c1t5000C50034FCD30Fd0
94.2 0.0 2.3 0.0 0.0 0.4 0.0 4.0 0 17 0 0 2 2 c1t5000C5002604233Fd0
> Date: Thu, 31 Dec 2015 17:02:17 -0800
> From: Martin Frost <me at cs.stanford.edu>
>
> Thanks, Bob and Ian.
>
> First, dedup is off everywhere.
>
> Below is the output of the longer iostat command for a few intervals.
> There are some small error counts under trn (transport errors);
> these counts are for 604 days, so they seem pretty low.
> Otherwise nothing jumps out at me here. Let me know if
> you see anything interesting below. The scrub is still running,
> so it is probably generating almost all the activity below.
>
> Martin
>
> tty cpu
> tin tout us sy wt id
> 0 79 0 4 0 96
> extended device statistics ---- errors
> ---
> r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b s/w h/w trn
> tot device
> 0.1 17.4 0.0 0.1 0.0 0.0 0.0 0.5 0 0 0 0 0
> 0 c3t0d0
> 0.1 17.2 0.0 0.1 0.0 0.0 0.0 0.5 0 0 0 0 0
> 0 c3t1d0
> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0
> 0 c3t2d0
> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0
> 0 c3t3d0
> 94.9 10.3 3.5 0.0 0.0 1.0 0.0 9.8 0 35 0 0 2
> 2 c1t5000C50035063BC3d0
> 106.3 14.3 3.5 0.0 0.0 1.1 0.0 8.8 0 35 0 0 0
> 0 c1t5000C5003505A723d0
> 106.3 14.5 3.5 0.0 0.0 1.1 0.0 9.0 0 36 0 0 0
> 0 c1t5000C5003504E1F3d0
> 92.5 10.4 3.4 0.0 0.0 1.0 0.0 9.7 0 34 0 0 2
> 2 c1t5000C50035070BE3d0
> 107.5 15.0 3.6 0.0 0.0 1.1 0.0 8.9 0 37 0 0 0
> 0 c1t5000C5003505BF17d0
> 106.8 15.3 3.6 0.0 0.0 1.2 0.0 9.7 0 37 0 0 0
> 0 c1t5000C5003503C847d0
> 93.6 10.4 3.5 0.0 0.0 1.1 0.0 10.4 0 35 0 0 2
> 2 c1t5000C50035015617d0
> 105.8 15.0 3.5 0.0 0.0 1.1 0.0 9.3 0 36 0 0 0
> 0 c1t5000C5003505EC87d0
> 107.4 14.8 3.6 0.0 0.0 1.1 0.0 9.1 0 36 0 0 0
> 0 c1t5000C50034FE2237d0
> 107.8 15.0 3.6 0.0 0.0 1.2 0.0 9.4 0 36 0 0 0
> 0 c1t5000C50035012E7Bd0
> 92.8 10.6 3.5 0.0 0.0 1.1 0.0 10.5 0 35 0 0 1
> 1 c1t5000C5003506371Bd0
> 93.4 10.4 3.5 0.0 0.0 1.1 0.0 10.2 0 34 0 0 0
> 0 c1t5000C5003505066Bd0
> 105.4 14.8 3.5 0.0 0.0 1.1 0.0 9.4 0 36 0 0 0
> 0 c1t5000C5003504BF8Bd0
> 93.1 10.7 3.5 0.0 0.0 1.1 0.0 10.5 0 34 0 0 1
> 1 c1t5000C5003506835Bd0
> 94.0 10.5 3.6 0.0 0.0 1.1 0.0 10.9 0 36 0 0 2
> 2 c1t5000C5003505954Fd0
> 94.9 10.2 3.6 0.0 0.0 1.1 0.0 10.9 0 35 0 0 2
> 2 c1t5000C50035043D4Fd0
> 108.6 15.4 3.6 0.0 0.0 1.2 0.0 9.5 0 37 0 0 0
> 0 c1t5000C50034FCD30Fd0
> 93.5 10.8 3.6 0.0 0.0 1.1 0.0 10.6 0 36 0 0 2
> 2 c1t5000C5002604233Fd0
> tty cpu
> tin tout us sy wt id
> 0 79 0 4 0 96
> extended device statistics ---- errors
> ---
> r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b s/w h/w trn
> tot device
> 0.2 8.9 0.0 0.0 0.0 0.0 0.0 0.4 0 0 0 0 0
> 0 c3t0d0
> 0.1 9.0 0.0 0.0 0.0 0.0 0.0 0.5 0 0 0 0 0
> 0 c3t1d0
> 0.0 0.1 0.0 0.0 0.0 0.0 0.0 3.8 0 0 0 0 0
> 0 c3t2d0
> 0.0 0.1 0.0 0.0 0.0 0.0 0.0 3.8 0 0 0 0 0
> 0 c3t3d0
> 83.4 11.4 2.5 0.0 0.0 0.6 0.0 6.3 0 27 0 0 2
> 2 c1t5000C50035063BC3d0
> 99.6 12.8 2.7 0.0 0.0 0.7 0.0 6.5 0 30 0 0 0
> 0 c1t5000C5003505A723d0
> 100.0 12.6 2.8 0.0 0.0 0.7 0.0 6.2 0 29 0 0 0
> 0 c1t5000C5003504E1F3d0
> 84.4 11.1 2.5 0.0 0.0 0.6 0.0 6.3 0 27 0 0 2
> 2 c1t5000C50035070BE3d0
> 99.8 12.7 2.7 0.0 0.0 0.7 0.0 6.6 0 29 0 0 0
> 0 c1t5000C5003505BF17d0
> 99.2 12.5 2.7 0.0 0.0 0.7 0.0 6.5 0 30 0 0 0
> 0 c1t5000C5003503C847d0
> 83.7 11.5 2.6 0.0 0.0 0.7 0.0 6.9 0 28 0 0 2
> 2 c1t5000C50035015617d0
> 99.1 12.9 2.7 0.0 0.0 0.7 0.0 6.3 0 29 0 0 0
> 0 c1t5000C5003505EC87d0
> 97.7 12.8 2.6 0.0 0.0 0.7 0.0 6.1 0 28 0 0 0
> 0 c1t5000C50034FE2237d0
> 99.0 12.8 2.7 0.0 0.0 0.7 0.0 6.2 0 29 0 0 0
> 0 c1t5000C50035012E7Bd0
> 86.9 11.3 2.6 0.0 0.0 0.7 0.0 7.2 0 28 0 0 1
> 1 c1t5000C5003506371Bd0
> 88.8 11.5 2.7 0.0 0.0 0.7 0.0 6.7 0 28 0 0 0
> 0 c1t5000C5003505066Bd0
> 100.1 12.9 2.8 0.0 0.0 0.7 0.0 6.6 0 30 0 0 0
> 0 c1t5000C5003504BF8Bd0
> 84.8 11.4 2.6 0.0 0.0 0.6 0.0 6.4 0 27 0 0 1
> 1 c1t5000C5003506835Bd0
> 86.2 11.3 2.6 0.0 0.0 0.7 0.0 6.8 0 28 0 0 2
> 2 c1t5000C5003505954Fd0
> 84.3 11.7 2.6 0.0 0.0 0.6 0.0 6.7 0 28 0 0 2
> 2 c1t5000C50035043D4Fd0
> 98.4 12.5 2.7 0.0 0.0 0.7 0.0 6.6 0 29 0 0 0
> 0 c1t5000C50034FCD30Fd0
> 88.1 10.9 2.7 0.0 0.0 0.7 0.0 7.3 0 29 0 0 2
> 2 c1t5000C5002604233Fd0
> tty cpu
> tin tout us sy wt id
> 0 79 0 4 0 96
> extended device statistics ---- errors
> ---
> r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b s/w h/w trn
> tot device
> 0.0 2.6 0.0 0.0 0.0 0.0 0.0 0.5 0 0 0 0 0
> 0 c3t0d0
> 0.0 2.6 0.0 0.0 0.0 0.0 0.0 0.5 0 0 0 0 0
> 0 c3t1d0
> 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0
> 0 c3t2d0
> 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0
> 0 c3t3d0
> 102.8 13.8 3.2 0.0 0.0 0.8 0.0 6.9 0 31 0 0 2
> 2 c1t5000C50035063BC3d0
> 100.1 15.6 3.1 0.0 0.0 0.9 0.0 7.5 0 31 0 0 0
> 0 c1t5000C5003505A723d0
> 97.9 15.1 3.1 0.0 0.0 0.8 0.0 7.1 0 30 0 0 0
> 0 c1t5000C5003504E1F3d0
> 102.0 14.1 3.1 0.0 0.0 0.8 0.0 6.6 0 31 0 0 2
> 2 c1t5000C50035070BE3d0
> 103.1 15.1 3.2 0.0 0.0 0.8 0.0 7.0 0 31 0 0 0
> 0 c1t5000C5003505BF17d0
> 98.2 14.5 3.1 0.0 0.0 0.9 0.0 7.6 0 31 0 0 0
> 0 c1t5000C5003503C847d0
> 104.5 13.6 3.1 0.0 0.0 0.9 0.0 7.4 0 32 0 0 2
> 2 c1t5000C50035015617d0
> 99.8 15.3 3.1 0.0 0.0 0.8 0.0 7.4 0 31 0 0 0
> 0 c1t5000C5003505EC87d0
> 98.1 14.7 3.0 0.0 0.0 0.8 0.0 7.2 0 30 0 0 0
> 0 c1t5000C50034FE2237d0
> 98.6 14.8 3.1 0.0 0.0 0.8 0.0 7.2 0 30 0 0 0
> 0 c1t5000C50035012E7Bd0
> 103.5 13.5 3.1 0.0 0.0 0.9 0.0 7.4 0 32 0 0 1
> 1 c1t5000C5003506371Bd0
> 102.7 13.4 3.1 0.0 0.0 0.9 0.0 7.5 0 33 0 0 0
> 0 c1t5000C5003505066Bd0
> 95.4 14.4 3.1 0.0 0.0 0.8 0.0 7.2 0 30 0 0 0
> 0 c1t5000C5003504BF8Bd0
> 101.4 13.8 3.1 0.0 0.0 0.8 0.0 7.0 0 31 0 0 1
> 1 c1t5000C5003506835Bd0
> 103.3 13.4 3.1 0.0 0.0 0.9 0.0 7.5 0 32 0 0 2
> 2 c1t5000C5003505954Fd0
> 104.1 13.9 3.2 0.0 0.0 0.9 0.0 7.3 0 31 0 0 2
> 2 c1t5000C50035043D4Fd0
> 97.1 14.6 3.0 0.0 0.0 0.9 0.0 7.7 0 31 0 0 0
> 0 c1t5000C50034FCD30Fd0
> 103.4 13.0 3.2 0.0 0.0 0.9 0.0 7.8 0 33 0 0 2
> 2 c1t5000C5002604233Fd0
>
> On Thu, Dec 31, 2015 at 1:32 PM, Ian Collins <ian at ianshome.com> wrote:
>
> > Bob Friesenhahn wrote:
> >
> >> On Thu, 31 Dec 2015, Martin Frost wrote:
> >>
> >>> Any thoughts about what could be causing the slowness?
> >>>
> >> Run 'iostat -xe' in order to see if a disk is reporting errors or is
> >> abnormally slow.
> >>
> >
> > I'd extend the options to get the pool summary and repeat every 30 seconds
> > or so to see if any drive is holding up the pool:
> >
> > iostat -xetcMn 30
> >
> > --
> > Ian.
> >
> >
> >
> > _______________________________________________
> > openindiana-discuss mailing list
> > openindiana-discuss at openindiana.org
> > http://openindiana.org/mailman/listinfo/openindiana-discuss
> >
> _______________________________________________
> openindiana-discuss mailing list
> openindiana-discuss at openindiana.org
> http://openindiana.org/mailman/listinfo/openindiana-discuss
>
More information about the openindiana-discuss
mailing list