[OpenIndiana-discuss] System will not boot, boot-archive fails

[Nikola M]

On 09/21/15 07:10 AM, Richard Patterson wrote:
> Hi All,
>
> I have a server which will not boot... We had a power cut last week, but
> prior to this the system had been up for well over a year.
>
> Now, on bootup, I get messages saying
>
> svc:/system/boot-archive:default: Method or service exit timed out.
> Killing contract 16
> svc:/system/boot-archive:default: Method "/lib/svc/method/boot-archive"
> failed due to signal KILL
>
> It then drops me into maintenance mode.
I am in the dark here, but maybe this is your problem:
There is a bug in the GRUB that is shipped as a part of illumos, that 
breaks with STAGE2 if menu.lst is too big (e.g. you have many Boot 
environments , undetermined how much, maybe over 25 BEs?).
Problem is known by illumos but not addressed nor fixed and there is a 
word that GRUB2 implementation in S11 has same problem, just it uses 
bigger buffer and list of Be's can be a little bigger.
illumos people also are working on new boot loader etc.

It is reported that if that is the problem, you can delete some BE's 
inside of /rpool/boot/grub/menu.lst
and then it would boot right. (and later delete zfs dataset of BE's removed)
It is done when booting from LiveDVD/USB and importing zpool to access 
menu.lst.

>
> Once logged in, I can't run bootadm update-archive, as it says another
> instance of bootadm is already running. (then console hangs)
>
> Most commands hang the console, such as zpool list, or zfs list.
>
> I also get a very short amount of time to do anything, as it seems to
> run out of memory (or swap) after a few minutes
>
> WARNING: /etc/svc/volatile: File system full, swap space limit exceeded
>
> # swap -l
> No swap devices configured
> # swap -s
> 17112k bytes allocated + 1623k reserved = 18744k used, 10439268k available
>
> If I run swap -s again, the figures are the same, except the available
> is reducing.
>
> OpenIndiana Build oi_151a9 64-bit (illumos 52e13e00ba)
>
> 32GB Ram, Quad-core AMD CPU
>
> rpool is on a single 250GB SATA drive.

it is not healthy to be left with no room on disk, it kills ZFS 
performance, etc.
You can clear some space from rpool to some external drive, for starters.