[OpenIndiana-discuss] very slow zpool

[Martin Frost]

I have a very slow zpool/ZFS filesystem, and I'm looking for the
cause of the extreme slowness.

The machine has been up 604 days, so when I get a chance I'm going
to reboot it to see if that helps.  But I'd appreciate any
suggestion of what else to do or what to look for.


For the last three months, rsyncing to ZFS on an oi_148 machine from
a nearby Linux machine has been much slower than it was previously,
for 4 of the 9 filesystems being rsync'd.

The full rsync of 9 filesystems that used to take 2 hours now takes
about 41 hours, even when it only transfers about 750MB over that
amount of time (over a 1 Gbps connection).  The last 5 of those 9
filesystems in the rsync still take a total of only about 30
minutes, but the first 4 go very slow.

Today I killed the rsync and started a scrub of the destination
pool, and the scrub too is progressing incredibly slowly.  The scrub
of this 32TB pool (about half full) used to take 20 hours.  At the
current rate, the scrub will take over 400 days to finish.


  # zfs list redact ; uptime ; date ; zpool status redact
  NAME    USED  AVAIL  REFER  MOUNTPOINT
  redact 15.1T  16.6T  1015M  /redact
   12:48pm  up 604 day(s), 21:07,  1 user,  load average: 0.30, 0.29, 0.29
  Thu Dec 31 12:48:48 PST 2015
    pool: redact
   state: ONLINE
   scan: scrub in progress since Thu Dec 31 09:12:23 2015
      7.23G scanned out of 23.0T at 584K/s, (scan is slow, no estimated time)
      0 repaired, 0.03% done
  config:

	  NAME                       STATE     READ WRITE CKSUM
	  redact                     ONLINE       0     0     0
	    raidz3-0                 ONLINE       0     0     0
	      c1t5000C50034FCD30Fd0  ONLINE       0     0     0
	      c1t5000C50034FE2237d0  ONLINE       0     0     0
	      c1t5000C5003503C847d0  ONLINE       0     0     0
	      c1t5000C5003504BF8Bd0  ONLINE       0     0     0
	      c1t5000C5003504E1F3d0  ONLINE       0     0     0
	      c1t5000C5003505A723d0  ONLINE       0     0     0
	      c1t5000C5003505BF17d0  ONLINE       0     0     0
	      c1t5000C5003505EC87d0  ONLINE       0     0     0
	      c1t5000C50035012E7Bd0  ONLINE       0     0     0
	    raidz3-1                 ONLINE       0     0     0
	      c1t5000C50035043D4Fd0  ONLINE       0     0     0
	      c1t5000C50035063BC3d0  ONLINE       0     0     0
	      c1t5000C50035070BE3d0  ONLINE       0     0     0
	      c1t5000C5003506835Bd0  ONLINE       0     0     0
	      c1t5000C5003506371Bd0  ONLINE       0     0     0
	      c1t5000C5003505066Bd0  ONLINE       0     0     0
	      c1t5000C5002604233Fd0  ONLINE       0     0     0
	      c1t5000C5003505954Fd0  ONLINE       0     0     0
	      c1t5000C50035015617d0  ONLINE       0     0     0
	  logs
	    mirror-2                 ONLINE       0     0     0
	      c3t2d0                 ONLINE       0     0     0
	      c3t3d0                 ONLINE       0     0     0

  errors: No known data errors


>From top (during the scrub):

  57 processes: 56 sleeping, 1 on cpu
  CPU states: 97.4% idle,  0.1% user,  2.5% kernel,  0.0% iowait,  0.0% swap
  Kernel: 7728 ctxsw, 6743 intr, 143 syscall
  Memory: 24G phys mem, 2909M free mem, 12G total swap, 12G free swap


Any thoughts about what could be causing the slowness?

Thanks for your help.

Martin