[OpenIndiana-discuss] very slow zpool
Jonathan Adams
t12nslookup at gmail.com
Fri Jan 1 01:22:16 UTC 2016
Hi Martin,
Sorry for to posting (reading to late at night on my tablet)
Just out of interest what percentage of the pool is being used at the
moment? What is the ashift on the pool? Do you know what make the drives
are?
Jon
On 1 Jan 2016 01:18, "Martin Frost" <me at cs.stanford.edu> wrote:
> Oh, here's iostat output from after I stopped the slow scrub just now
> (with more activity than I expected so maybe someone is working late
> on New Year's Eve).
>
> Martin
>
> tty cpu
> tin tout us sy wt id
> 0 83 0 4 0 96
> extended device statistics ---- errors
> ---
> r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b s/w h/w trn
> tot device
> 0.1 3.2 0.0 0.0 0.0 0.0 0.0 0.2 0 0 0 0 0
> 0 c3t0d0
> 0.3 3.0 0.0 0.0 0.0 0.0 0.0 0.3 0 0 0 0 0
> 0 c3t1d0
> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0
> 0 c3t2d0
> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0
> 0 c3t3d0
> 95.1 0.0 2.0 0.0 0.0 0.5 0.0 5.5 0 19 0 0 2
> 2 c1t5000C50035063BC3d0
> 84.4 0.0 1.9 0.0 0.0 0.5 0.0 5.7 0 18 0 0 0
> 0 c1t5000C5003505A723d0
> 85.3 0.0 2.0 0.0 0.0 0.5 0.0 6.0 0 19 0 0 0
> 0 c1t5000C5003504E1F3d0
> 94.6 0.0 2.0 0.0 0.0 0.5 0.0 5.5 0 19 0 0 2
> 2 c1t5000C50035070BE3d0
> 86.5 0.0 2.0 0.0 0.0 0.5 0.0 5.6 0 19 0 0 0
> 0 c1t5000C5003505BF17d0
> 83.3 0.0 1.9 0.0 0.0 0.5 0.0 5.7 0 18 0 0 0
> 0 c1t5000C5003503C847d0
> 93.3 0.0 2.0 0.0 0.0 0.5 0.0 5.6 0 19 0 0 2
> 2 c1t5000C50035015617d0
> 85.1 0.0 1.9 0.0 0.0 0.5 0.0 5.6 0 19 0 0 0
> 0 c1t5000C5003505EC87d0
> 82.1 0.0 1.8 0.0 0.0 0.5 0.0 6.0 0 17 0 0 0
> 0 c1t5000C50034FE2237d0
> 85.2 0.0 1.9 0.0 0.0 0.5 0.0 5.6 0 18 0 0 0
> 0 c1t5000C50035012E7Bd0
> 91.8 0.0 1.9 0.0 0.0 0.5 0.0 5.6 0 19 0 0 1
> 1 c1t5000C5003506371Bd0
> 93.2 0.0 1.9 0.0 0.0 0.5 0.0 5.5 0 20 0 0 0
> 0 c1t5000C5003505066Bd0
> 84.1 0.0 1.9 0.0 0.0 0.5 0.0 5.8 0 18 0 0 0
> 0 c1t5000C5003504BF8Bd0
> 94.6 0.0 1.9 0.0 0.0 0.5 0.0 5.6 0 20 0 0 1
> 1 c1t5000C5003506835Bd0
> 93.2 0.0 1.9 0.0 0.0 0.5 0.0 5.5 0 19 0 0 2
> 2 c1t5000C5003505954Fd0
> 93.1 0.0 2.0 0.0 0.0 0.5 0.0 5.6 0 19 0 0 2
> 2 c1t5000C50035043D4Fd0
> 84.4 0.0 1.9 0.0 0.0 0.5 0.0 5.4 0 19 0 0 0
> 0 c1t5000C50034FCD30Fd0
> 92.9 0.0 2.0 0.0 0.0 0.5 0.0 5.6 0 19 0 0 2
> 2 c1t5000C5002604233Fd0
> tty cpu
> tin tout us sy wt id
> 0 79 0 4 0 96
> extended device statistics ---- errors
> ---
> r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b s/w h/w trn
> tot device
> 0.0 5.3 0.0 0.0 0.0 0.0 0.0 0.5 0 0 0 0 0
> 0 c3t0d0
> 0.0 5.3 0.0 0.0 0.0 0.0 0.0 0.5 0 0 0 0 0
> 0 c3t1d0
> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0
> 0 c3t2d0
> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0
> 0 c3t3d0
> 97.4 0.0 2.3 0.0 0.0 0.4 0.0 4.0 0 18 0 0 2
> 2 c1t5000C50035063BC3d0
> 100.1 0.0 2.3 0.0 0.0 0.4 0.0 3.6 0 17 0 0 0
> 0 c1t5000C5003505A723d0
> 100.7 0.0 2.3 0.0 0.0 0.4 0.0 3.8 0 16 0 0 0
> 0 c1t5000C5003504E1F3d0
> 96.1 0.0 2.3 0.0 0.0 0.4 0.0 4.0 0 17 0 0 2
> 2 c1t5000C50035070BE3d0
> 100.6 0.0 2.3 0.0 0.0 0.4 0.0 3.6 0 17 0 0 0
> 0 c1t5000C5003505BF17d0
> 97.5 0.0 2.2 0.0 0.0 0.3 0.0 3.6 0 16 0 0 0
> 0 c1t5000C5003503C847d0
> 95.9 0.0 2.3 0.0 0.0 0.4 0.0 3.7 0 17 0 0 2
> 2 c1t5000C50035015617d0
> 100.4 0.0 2.3 0.0 0.0 0.3 0.0 3.5 0 16 0 0 0
> 0 c1t5000C5003505EC87d0
> 97.8 0.0 2.2 0.0 0.0 0.4 0.0 3.7 0 16 0 0 0
> 0 c1t5000C50034FE2237d0
> 100.8 0.0 2.3 0.0 0.0 0.4 0.0 3.7 0 17 0 0 0
> 0 c1t5000C50035012E7Bd0
> 94.2 0.0 2.2 0.0 0.0 0.4 0.0 4.2 0 17 0 0 1
> 1 c1t5000C5003506371Bd0
> 92.5 0.0 2.2 0.0 0.0 0.4 0.0 4.0 0 17 0 0 0
> 0 c1t5000C5003505066Bd0
> 99.4 0.0 2.3 0.0 0.0 0.4 0.0 4.2 0 18 0 0 0
> 0 c1t5000C5003504BF8Bd0
> 94.5 0.0 2.2 0.0 0.0 0.4 0.0 3.8 0 16 0 0 1
> 1 c1t5000C5003506835Bd0
> 95.6 0.0 2.3 0.0 0.0 0.4 0.0 4.1 0 17 0 0 2
> 2 c1t5000C5003505954Fd0
> 97.3 0.0 2.3 0.0 0.0 0.4 0.0 3.9 0 18 0 0 2
> 2 c1t5000C50035043D4Fd0
> 98.7 0.0 2.3 0.0 0.0 0.4 0.0 3.8 0 17 0 0 0
> 0 c1t5000C50034FCD30Fd0
> 94.2 0.0 2.3 0.0 0.0 0.4 0.0 4.0 0 17 0 0 2
> 2 c1t5000C5002604233Fd0
>
>
> > Date: Thu, 31 Dec 2015 17:02:17 -0800
> > From: Martin Frost <me at cs.stanford.edu>
> >
> > Thanks, Bob and Ian.
> >
> > First, dedup is off everywhere.
> >
> > Below is the output of the longer iostat command for a few intervals.
> > There are some small error counts under trn (transport errors);
> > these counts are for 604 days, so they seem pretty low.
> > Otherwise nothing jumps out at me here. Let me know if
> > you see anything interesting below. The scrub is still running,
> > so it is probably generating almost all the activity below.
> >
> > Martin
> >
> > tty cpu
> > tin tout us sy wt id
> > 0 79 0 4 0 96
> > extended device statistics ---- errors
> > ---
> > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b s/w h/w trn
> > tot device
> > 0.1 17.4 0.0 0.1 0.0 0.0 0.0 0.5 0 0 0 0 0
> > 0 c3t0d0
> > 0.1 17.2 0.0 0.1 0.0 0.0 0.0 0.5 0 0 0 0 0
> > 0 c3t1d0
> > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0
> > 0 c3t2d0
> > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0
> > 0 c3t3d0
> > 94.9 10.3 3.5 0.0 0.0 1.0 0.0 9.8 0 35 0 0 2
> > 2 c1t5000C50035063BC3d0
> > 106.3 14.3 3.5 0.0 0.0 1.1 0.0 8.8 0 35 0 0 0
> > 0 c1t5000C5003505A723d0
> > 106.3 14.5 3.5 0.0 0.0 1.1 0.0 9.0 0 36 0 0 0
> > 0 c1t5000C5003504E1F3d0
> > 92.5 10.4 3.4 0.0 0.0 1.0 0.0 9.7 0 34 0 0 2
> > 2 c1t5000C50035070BE3d0
> > 107.5 15.0 3.6 0.0 0.0 1.1 0.0 8.9 0 37 0 0 0
> > 0 c1t5000C5003505BF17d0
> > 106.8 15.3 3.6 0.0 0.0 1.2 0.0 9.7 0 37 0 0 0
> > 0 c1t5000C5003503C847d0
> > 93.6 10.4 3.5 0.0 0.0 1.1 0.0 10.4 0 35 0 0 2
> > 2 c1t5000C50035015617d0
> > 105.8 15.0 3.5 0.0 0.0 1.1 0.0 9.3 0 36 0 0 0
> > 0 c1t5000C5003505EC87d0
> > 107.4 14.8 3.6 0.0 0.0 1.1 0.0 9.1 0 36 0 0 0
> > 0 c1t5000C50034FE2237d0
> > 107.8 15.0 3.6 0.0 0.0 1.2 0.0 9.4 0 36 0 0 0
> > 0 c1t5000C50035012E7Bd0
> > 92.8 10.6 3.5 0.0 0.0 1.1 0.0 10.5 0 35 0 0 1
> > 1 c1t5000C5003506371Bd0
> > 93.4 10.4 3.5 0.0 0.0 1.1 0.0 10.2 0 34 0 0 0
> > 0 c1t5000C5003505066Bd0
> > 105.4 14.8 3.5 0.0 0.0 1.1 0.0 9.4 0 36 0 0 0
> > 0 c1t5000C5003504BF8Bd0
> > 93.1 10.7 3.5 0.0 0.0 1.1 0.0 10.5 0 34 0 0 1
> > 1 c1t5000C5003506835Bd0
> > 94.0 10.5 3.6 0.0 0.0 1.1 0.0 10.9 0 36 0 0 2
> > 2 c1t5000C5003505954Fd0
> > 94.9 10.2 3.6 0.0 0.0 1.1 0.0 10.9 0 35 0 0 2
> > 2 c1t5000C50035043D4Fd0
> > 108.6 15.4 3.6 0.0 0.0 1.2 0.0 9.5 0 37 0 0 0
> > 0 c1t5000C50034FCD30Fd0
> > 93.5 10.8 3.6 0.0 0.0 1.1 0.0 10.6 0 36 0 0 2
> > 2 c1t5000C5002604233Fd0
> > tty cpu
> > tin tout us sy wt id
> > 0 79 0 4 0 96
> > extended device statistics ---- errors
> > ---
> > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b s/w h/w trn
> > tot device
> > 0.2 8.9 0.0 0.0 0.0 0.0 0.0 0.4 0 0 0 0 0
> > 0 c3t0d0
> > 0.1 9.0 0.0 0.0 0.0 0.0 0.0 0.5 0 0 0 0 0
> > 0 c3t1d0
> > 0.0 0.1 0.0 0.0 0.0 0.0 0.0 3.8 0 0 0 0 0
> > 0 c3t2d0
> > 0.0 0.1 0.0 0.0 0.0 0.0 0.0 3.8 0 0 0 0 0
> > 0 c3t3d0
> > 83.4 11.4 2.5 0.0 0.0 0.6 0.0 6.3 0 27 0 0 2
> > 2 c1t5000C50035063BC3d0
> > 99.6 12.8 2.7 0.0 0.0 0.7 0.0 6.5 0 30 0 0 0
> > 0 c1t5000C5003505A723d0
> > 100.0 12.6 2.8 0.0 0.0 0.7 0.0 6.2 0 29 0 0 0
> > 0 c1t5000C5003504E1F3d0
> > 84.4 11.1 2.5 0.0 0.0 0.6 0.0 6.3 0 27 0 0 2
> > 2 c1t5000C50035070BE3d0
> > 99.8 12.7 2.7 0.0 0.0 0.7 0.0 6.6 0 29 0 0 0
> > 0 c1t5000C5003505BF17d0
> > 99.2 12.5 2.7 0.0 0.0 0.7 0.0 6.5 0 30 0 0 0
> > 0 c1t5000C5003503C847d0
> > 83.7 11.5 2.6 0.0 0.0 0.7 0.0 6.9 0 28 0 0 2
> > 2 c1t5000C50035015617d0
> > 99.1 12.9 2.7 0.0 0.0 0.7 0.0 6.3 0 29 0 0 0
> > 0 c1t5000C5003505EC87d0
> > 97.7 12.8 2.6 0.0 0.0 0.7 0.0 6.1 0 28 0 0 0
> > 0 c1t5000C50034FE2237d0
> > 99.0 12.8 2.7 0.0 0.0 0.7 0.0 6.2 0 29 0 0 0
> > 0 c1t5000C50035012E7Bd0
> > 86.9 11.3 2.6 0.0 0.0 0.7 0.0 7.2 0 28 0 0 1
> > 1 c1t5000C5003506371Bd0
> > 88.8 11.5 2.7 0.0 0.0 0.7 0.0 6.7 0 28 0 0 0
> > 0 c1t5000C5003505066Bd0
> > 100.1 12.9 2.8 0.0 0.0 0.7 0.0 6.6 0 30 0 0 0
> > 0 c1t5000C5003504BF8Bd0
> > 84.8 11.4 2.6 0.0 0.0 0.6 0.0 6.4 0 27 0 0 1
> > 1 c1t5000C5003506835Bd0
> > 86.2 11.3 2.6 0.0 0.0 0.7 0.0 6.8 0 28 0 0 2
> > 2 c1t5000C5003505954Fd0
> > 84.3 11.7 2.6 0.0 0.0 0.6 0.0 6.7 0 28 0 0 2
> > 2 c1t5000C50035043D4Fd0
> > 98.4 12.5 2.7 0.0 0.0 0.7 0.0 6.6 0 29 0 0 0
> > 0 c1t5000C50034FCD30Fd0
> > 88.1 10.9 2.7 0.0 0.0 0.7 0.0 7.3 0 29 0 0 2
> > 2 c1t5000C5002604233Fd0
> > tty cpu
> > tin tout us sy wt id
> > 0 79 0 4 0 96
> > extended device statistics ---- errors
> > ---
> > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b s/w h/w trn
> > tot device
> > 0.0 2.6 0.0 0.0 0.0 0.0 0.0 0.5 0 0 0 0 0
> > 0 c3t0d0
> > 0.0 2.6 0.0 0.0 0.0 0.0 0.0 0.5 0 0 0 0 0
> > 0 c3t1d0
> > 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0
> > 0 c3t2d0
> > 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0
> > 0 c3t3d0
> > 102.8 13.8 3.2 0.0 0.0 0.8 0.0 6.9 0 31 0 0 2
> > 2 c1t5000C50035063BC3d0
> > 100.1 15.6 3.1 0.0 0.0 0.9 0.0 7.5 0 31 0 0 0
> > 0 c1t5000C5003505A723d0
> > 97.9 15.1 3.1 0.0 0.0 0.8 0.0 7.1 0 30 0 0 0
> > 0 c1t5000C5003504E1F3d0
> > 102.0 14.1 3.1 0.0 0.0 0.8 0.0 6.6 0 31 0 0 2
> > 2 c1t5000C50035070BE3d0
> > 103.1 15.1 3.2 0.0 0.0 0.8 0.0 7.0 0 31 0 0 0
> > 0 c1t5000C5003505BF17d0
> > 98.2 14.5 3.1 0.0 0.0 0.9 0.0 7.6 0 31 0 0 0
> > 0 c1t5000C5003503C847d0
> > 104.5 13.6 3.1 0.0 0.0 0.9 0.0 7.4 0 32 0 0 2
> > 2 c1t5000C50035015617d0
> > 99.8 15.3 3.1 0.0 0.0 0.8 0.0 7.4 0 31 0 0 0
> > 0 c1t5000C5003505EC87d0
> > 98.1 14.7 3.0 0.0 0.0 0.8 0.0 7.2 0 30 0 0 0
> > 0 c1t5000C50034FE2237d0
> > 98.6 14.8 3.1 0.0 0.0 0.8 0.0 7.2 0 30 0 0 0
> > 0 c1t5000C50035012E7Bd0
> > 103.5 13.5 3.1 0.0 0.0 0.9 0.0 7.4 0 32 0 0 1
> > 1 c1t5000C5003506371Bd0
> > 102.7 13.4 3.1 0.0 0.0 0.9 0.0 7.5 0 33 0 0 0
> > 0 c1t5000C5003505066Bd0
> > 95.4 14.4 3.1 0.0 0.0 0.8 0.0 7.2 0 30 0 0 0
> > 0 c1t5000C5003504BF8Bd0
> > 101.4 13.8 3.1 0.0 0.0 0.8 0.0 7.0 0 31 0 0 1
> > 1 c1t5000C5003506835Bd0
> > 103.3 13.4 3.1 0.0 0.0 0.9 0.0 7.5 0 32 0 0 2
> > 2 c1t5000C5003505954Fd0
> > 104.1 13.9 3.2 0.0 0.0 0.9 0.0 7.3 0 31 0 0 2
> > 2 c1t5000C50035043D4Fd0
> > 97.1 14.6 3.0 0.0 0.0 0.9 0.0 7.7 0 31 0 0 0
> > 0 c1t5000C50034FCD30Fd0
> > 103.4 13.0 3.2 0.0 0.0 0.9 0.0 7.8 0 33 0 0 2
> > 2 c1t5000C5002604233Fd0
> >
> > On Thu, Dec 31, 2015 at 1:32 PM, Ian Collins <ian at ianshome.com> wrote:
> >
> > > Bob Friesenhahn wrote:
> > >
> > >> On Thu, 31 Dec 2015, Martin Frost wrote:
> > >>
> > >>> Any thoughts about what could be causing the slowness?
> > >>>
> > >> Run 'iostat -xe' in order to see if a disk is reporting errors or is
> > >> abnormally slow.
> > >>
> > >
> > > I'd extend the options to get the pool summary and repeat every 30
> seconds
> > > or so to see if any drive is holding up the pool:
> > >
> > > iostat -xetcMn 30
> > >
> > > --
> > > Ian.
> > >
> > >
> > >
> > > _______________________________________________
> > > openindiana-discuss mailing list
> > > openindiana-discuss at openindiana.org
> > > http://openindiana.org/mailman/listinfo/openindiana-discuss
> > >
> > _______________________________________________
> > openindiana-discuss mailing list
> > openindiana-discuss at openindiana.org
> > http://openindiana.org/mailman/listinfo/openindiana-discuss
> >
>
> _______________________________________________
> openindiana-discuss mailing list
> openindiana-discuss at openindiana.org
> http://openindiana.org/mailman/listinfo/openindiana-discuss
>
More information about the openindiana-discuss
mailing list