[OpenIndiana-discuss] auto-scrub and its result

Fri Dec 28 21:47:09 UTC 2012

On 2012-12-28 21:45, Brogyányi József wrote:
> Hi Bob
>
> Thanks your answer. I'd like to know what does it mean clean shut down?
> I think there is a way what is the perfect.
> Could you write down that command?This before I used Ubuntu.

There are a few. The most proper one is "init 5" which brings down
the services, quiesces filesystems, syncs the disks and tells the
power source to unpower itself. Likewise, "init 0" does all this
except the unpowering, and "init 6" finishes by a full or fast
reboot (depending on config elsewhere).

Don't directly use shutdown, halt, reboot and such - as I did after
years with Linux - these commands do as advertized, with minimal
attempts to properly and slowly stop services (at least, was so
up till solaris 9, then I got out of the habit to use these often).
However, if the system is in a pinch - such as HDDs timing out so
a proper shutdown would likely hang, you might exactly want a rude
shutdown, such as "reboot -lnq", where:

      -l
          Suppress sending a message to  the  system  log  daemon,
          syslogd(1M) about who executed reboot.

      -n
          Avoid calling sync(2) and  do  not  log  the  reboot  to
          syslogd(1M)  or  to  /var/adm/wtmpx.  The  kernel  still
          attempts to sync filesystems prior to reboot, except  if
          the  -d  option  is also present. If -d is used with -n,
          the kernel does not attempt to sync filesystems.

      -q
          Quick. Reboot quickly and ungracefully, without shutting
          down running processes first.

For very difficult cases there is a lowlevel "uadmin 1 2" which is
basically a hook to the kernel's "shoot myself" exit point (detailed
in "man -s 2 uadmin").

> Brogyi
> I like your crontab command. :)

I think I've shared mine before, but here goes (part of our in-house
admin-script set COSas, but I don't think this wrapper has any
dependencies). It takes care to not issue a new scrub if another
is already in progress, and mails the results. Config files can
be used:

---
#!/bin/bash

# $Id: zpool-scrub.sh,v 1.6 2010/11/15 14:32:19 jim Exp $
# this script will go through all pools and scrub them one at a time
#
# Use like this in crontab:
# 0 22 * * * [ -x /opt/COSas/bin/zpool-scrub.sh ] && 
/opt/COSas/bin/zpool-scrub.sh
#
# (C) 2007 nickus at aspiringsysadmin.com and commenters
# (C) 2009 Jim Klimov, cosmetic mods and logging; 2010 - locking
# 
http://aspiringsysadmin.com/blog/2007/06/07/scrub-your-zfs-file-systems-regularly/
#
[ x"$MAILRECIPIENT" = x ] && MAILRECIPIENT=admin at mydomain.com

[ x"$ZPOOL" = x ] && ZPOOL=/usr/sbin/zpool
[ x"$TMPFILE" = x ] && TMPFILE=/tmp/scrub.sh.$$.$RANDOM
[ x"$LOCK" = x ] && LOCK="/tmp/`basename "$0"`.`dirname "$0" | sed 
's/\//_/g'`.lock"

COSAS_BINDIR=`dirname "$0"`
if [ x"$COSAS_BINDIR" = x./ -o x"$COSAS_BINDIR" = x. ]; then
         COSAS_BINDIR=`pwd`
fi

# Source optional config files
[ x"$COSAS_CFGDIR" = x ] && COSAS_CFGDIR="$COSAS_BINDIR/../etc"
if [ -d "$COSAS_CFGDIR" ]; then
     [  -f "$COSAS_CFGDIR/COSas.conf" ] && \
         . "$COSAS_CFGDIR/COSas.conf"
     [  -f "$COSAS_CFGDIR/`basename "$0"`.conf" ] && \
         . "$COSAS_CFGDIR/`basename "$0"`.conf"
fi

[ ! -x "$ZPOOL" ] && exit 1

### Include this after config files, in case of RUNLEVEL_NOKICK mask 
override
RUN_CHECKLEVEL=""
[ -s "$COSAS_BINDIR/runlevel_check.include" ] &&
     . "$COSAS_BINDIR/runlevel_check.include" &&
     block_runlevel

# Check LOCKfile
if [ -f "$LOCK" ]; then
     OLDPID=`head -n 1 "$LOCK"`
     BN="`basename $0`"
     TRYOLDPID=`ps -ef | grep "$BN" | grep -v grep | awk '{ print $2 }' 
| grep "$OLDPID"`
     if [ x"$TRYOLDPID" != x ]; then

         LF=`cat "$LOCK"`

         echo "= ZPoolScrub wrapper aborted because another copy is 
running - lockfile found:
$LF
Aborting..." | wall
         exit 1
     fi
fi
echo "$$" > "$LOCK"

scrub_in_progress() {
         ### Check that we're not yet shutting down
         if [ x"$RUN_CHECKLEVEL" != x ]; then
             if [ x"`check_runlevel`" != x ]; then
                 echo "INFO: System is shutting down. Aborting scrub of 
pool '$1'!" >&2
                 zpool scrub -s "$1"
                 return 1
             fi
         fi

         if $ZPOOL status "$1" | grep "scrub in progress" >/dev/null; then
                 return 0
         else
                 return 1
         fi
}

RESULT=0
for pool in `$ZPOOL list -H -o name`; do
         echo "=== `TZ=UTC date` @ `hostname`: $ZPOOL scrub $pool 
started..."
         $ZPOOL scrub "$pool"

         while scrub_in_progress "$pool"; do sleep 60; done

         echo "=== `TZ=UTC date` @ `hostname`: $ZPOOL scrub $pool completed"

         if ! $ZPOOL status $pool | grep "with 0 errors" >/dev/null; then
                 $ZPOOL status "$pool" | tee -a $TMPFILE
                 RESULT=$(($RESULT+1))
         fi
done

if [ -s $TMPFILE ]; then
         cat $TMPFILE | mailx -s "zpool scrub on `hostname` generated 
errors" $MAILRECIPIENT
fi

rm -f $TMPFILE

# Be nice, clean up
rm -f "$LOCK"

exit $RESULT
---

It mentions an include file with routines to detect the system being
shut down (avoid new work then):

---
# cat /opt/COSas/bin/runlevel_check.include
### Some utlities running from cron should choose to
### abort if the system is shutting down (i.e. check
### and restart some service). These routines should
### help...

### Variable can be used by includers to work around potential
### problems during include. Just call the routine like
### $RUN_CHECKLEVEL myprog param1 param2 ...
### Can also be used to check that the include succeeded.
RUN_CHECKLEVEL="run_checklevel"

### $Id: runlevel_check.include,v 1.4 2010/07/16 15:00:27 jim Exp $
### (C) 2005-2010 by Jim Klimov, JSC COS&HT

### Mask of "shutting down" run-levels for several OSes
### Can override via config file/environment
[ x"$RUNLEVEL_NOKICK" = x ] && RUNLEVEL_NOKICK="Linux  06
SunOS   056"

### Routine to check whether the system is shutting down
### Returns 0 or 1
### Echoes an empty string if ok to run,
###       not empty if shutting down
check_runlevel() {
     RUNLEVEL=`/usr/bin/who -r | sed 's/^\(.*\)run-level \([^ 
]\)\(.*\)$/\2/'`
     OS=`uname -s`

     # Check if current runlevel is OK to run:
     # If OS is not in list, always OK
     # If OS is in list, and RUNLEVEL is not in list, OK to run
     RLMASK=`echo "$RUNLEVEL_NOKICK" | egrep '^'"$OS " | head -1 | awk 
'{ print $2 }' | grep "$RUNLEVEL"`
     echo "$RLMASK"

     [ x"$RLMASK" = x ]
     return $?
}

### Check the runlevel and abort if shutting down.
### Expected to be the most used function ;)
block_runlevel() {
     if [ x"`check_runlevel`" != "x" ]; then
         echo "INFO: System '`hostname`.`domainname`' is shutting down.
INFO: Aborting agent $AGENTNAME ($AGENTDESC)..." >&2
         exit 0
     fi
     return 0
}

### Runs its specified parameters ($@) if the
### system is not shutting down
run_checklevel() {
     # Kick-Start service if not in given runlevels for given OS (i.e. 
restart)
     RLMASK="`check_runlevel`"
     if [ x"$RLMASK" = x ]; then
         "$@"
         return $?
     else
         echo "INFO: Not running '$@' for agent '$AGENTNAME' 
($AGENTDESC) up because '$OS' is in runlevel '$RUNLEVEL'" >&2
         return 0
     fi
}
---

HTH,
//Jim Klimov