[OpenIndiana-discuss] issue booting OpenIndiana 151.1.7

[Mika Anderson]

A couple of days ago, my server spontaneously? rebooted and now it is not
able to boot properly.

Some background:
I'm running OpenIndiana in a VM under ESXi.  The host runs on a Supermicro
board with a Xeon E3 and ECC ram.  There are two HBA controllers passed
through: IBM M1015 flashed to LSI2008 IT and IBM BR10i flashed to LSI1068
IT.

The issue happened while I was deleting ~2000 small files on a cifs share
from a Windows PC.  The delete operation was proceeding like normal until
about 75% when it was unable to complete any more deletes.  After 30
minutes or so, I logged into the ESXi to view the OI vm console.  It was at
the boot selection screen.  I selected the most recent option (which is
what it normally boots) and the system began to boot but shortly after, it
rebooted.  This behaviour is consistent and reproducible.  Here's the last
bit of info that the console was showing before this reboot:
ffffff000fc7cf40 zfs:arc_get_data_buf+173 ()
ffffff000fc7cf90 zfs:arc_buf_alloc+a2 ()
ffffff000fc7d030 zfs:arc_read_nolock+149 ()
ffffff000fc7d0d0 zfs:arc_read+79 ()
ffffff000fc7d130 zfs:dsl_read+33 ()
ffffff000fc7d1d0 zfs:traverse_prefetcher+115 ()
ffffff000fc7d2c0 zfs:traverse_visitbp+18c ()
ffffff000fc7d360 zfs.traverse_dnode+a3 ()
ffffff000fc7d440 zfs:traverse_visitbp+5bb ()
ffffff000fc7d520 zfs:traverse_visitbp+859 ()
ffffff000fc7d600 zfs:traverse_visitbp+859 ()
ffffff000fc7d6e0 zfs:traverse_visitbp+859 ()
ffffff000fc7d7c0 zfs:traverse_visitbp+859 ()
ffffff000fc7d8a0 zfs:traverse_visitbp+859 ()
ffffff000fc7d980 zfs:traverse_visitbp+859 ()
ffffff000fc7da20 zfs:traverse_dnode+a3 ()
ffffff000fc7db00 zfs:traverse_visitbp+2c6 ()
ffffff000fc7db90 zfs:traverse_prefetch_thread+ac ()
ffffff000fc7dc20 genunix:taskq_d_thread+b1 ()
ffffff000fc7dc30 unix:thread_start+8 ()
syncing file systems... done
skipping system dump - no dump device configured
rebooting...

Here's my troubleshooting steps so far:
-at the boot menu, select the second to last option and boot.  This is a
snapshot from several months ago of the system from before running a bunch
of OI upgrades and rebuilding pools.  The system boots to the desktop and
more or less works.  The problem is the snapshot is from before changes to
the pools (including upgrading from v28 pools to feature flag pools).
 After booting, it's looking for the old pool structure.
-remove the passthrough HBA and boot the most recent snapshot.  OI boots to
the desktop and zpool status reports the correct pool layout but of course
all the drives are missing.

So I'm not sure where to go next with troubleshooting.  If possible, I'd
like to recover the pools with permissions and shares intact.  If I lose
the permissions and shares, it's not too difficult to rebuild.  And worst
case if I lose the pools, well at least all the important stuff is backed
up.

Some thoughts I've had but am tentative to try without some direction:
-use the console that's available from the boot menu for troubleshooting.
 I have no idea where to start with this but maybe something can be done
here
-boot into the second to last snapshot which works, run through all the OI
upgrades, then try to re-import the pools
-build a new VM with one of OpenIndiana, OmniOS, Nexenta or even FreeNAS,
pass through the controllers and attempt to import the pools.

Any advice on things to try or warnings on places to beware are most
appreciated.

Best regards,

Mika