[OpenIndiana-discuss] Slow zfs list on oi_148 compared to snv_111b

[Shawn Moore]

I have several(5) ZFS machines that have been built to serve out over
NFS.  Most are OpenSolaris snv_111b, but we have two that are oi_148.
Between these we have ~ 200 volumes and 221000 snapshots (as of this
writing). On all of the oi_148's it takes a really really long time to
list the snapshots, but on the snv_111b's it's really quick.  It does
not appear to be related to the amount of data or hardware as the
oi_148's have 144GB ram, 2x6 CPU's and even mirror'd SSD ZIL's.  The
snv_111b's have 16GB ram, 2x4 CPU's and no SSD.  Any help or
troubleshooting advice is greatly appreciated as I would like to move
to OI for several reasons.  Below is my findings.


nas0 (oi_148):
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
nas0  3.97T  1.88T  2.09T    47%  1.02x  ONLINE  -
# time zfs list -H -o name -r nas0 | wc -l
22
real    0m0.926s
user    0m0.009s
sys     0m0.022s
# time zfs list -H -o name -t all -r nas0 | wc -l
8949
real    25m26.583s
user    0m1.475s
sys     0m14.229s


nas1 (oi_148):
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
nas1  1.98T  1.00T  1003G    50%  1.06x  ONLINE  -
# time zfs list -H -o name -r nas1 | wc -l
25
real    0m0.015s
user    0m0.005s
sys     0m0.013s
# time zfs list -H -o name -t all -r nas1 | wc -l
10710
real    4m7.599s
user    0m0.943s
sys     0m3.816s


nas3 (snv_111b):
NAME   SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
nas3  11.9T  9.51T  2.42T    79%  ONLINE  -
# time zfs list -H -o name -r nas3 | wc -l
67
real    0m0.347s
user    0m0.008s
sys     0m0.015s
# time zfs list -H -o name -t all -r nas3 | wc -l
98445
real    0m17.147s
user    0m6.160s
sys     0m10.835s


nas9 (snv_111b):
NAME   SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
nas9  5.97T  4.19T  1.77T    70%  ONLINE  -
# time zfs list -H -o name -r nas9 | wc -l
29
real    0m0.232s
user    0m0.005s
sys     0m0.012s
# time zfs list -H -o name -t all -r nas9 | wc -l
40846
real    0m7.138s
user    0m2.504s
sys     0m4.351s