[OpenIndiana-discuss] debugging service startup errors after recent pkg update

[Mooney, Tim]

All-

I applied updates on my hipster workstation at my workplace today, and since those updates I cannot get a login on the system, even when attempting a single-user boot.  I've tried selecting older boot environments (I have several) and even those exhibit svc.startd errors and will not give me single-user access.

I'm looking for advice and suggestions on how to potentially rescue the system.  In particular, if I  boot off a recent hipster live USB, is it possible to access the boot environment on my system's disks?  I'm familiar (in theory) with the idea of booting into an alternate boot environment and potentially adding/removing/verifying packages using pkg's -R option, as described here:

    https://docs.oracle.com/cd/E23824_01/html/E21801/mount.html

but those docs seem to assume you're booting off an older (or newer) boot env on the installed system, not some kind of a "rescue" live USB.  Is there a way to make a Live hipster image see the boot environments from outside the live image?

As far as the system, without being able to get a root prompt I don't have much info on what's going wrong or why it seems to impact multiple boot environments.

Prior to today's update, I had been running a boot environment created in mid April 2025, without any issues.

Today I decided I was overdue for updates, so I did a "pkg update -v".   Note that I did the "pkg update -v" in an ssh terminal window rather than from the console, and I did get disconnected at the very end of the process.  However, I logged back in and checked the command output that I had tee'd to a file and it indicated it had finished and was ready for a reboot.

When I boot the env created today, it says it's illumos-e98d23eb02.  It *does* boot, though it's really, really slow.  It never gets to multi-user mode, though, it just outputs messages about

    svc.startd[9]: svc:/network/location:default: Method or service exit timed out. Killing contract 33

If I force power off the system and then enable "verbose" and "single-user" mode via the boot options, the next boot does say that it is

    Booting to milestone "milestone/single-user:default".

but there's a lengthy pause (5-10 minutes) before it outputs

    Hostname: <workstation hostname>

and then another lengthy pause (several minutes) before it starts outputting the svc.startd messages again.  I tried waiting about 20-25 minutes and it never got to the point where I could get a prompt.

As I said above, even if I select the formerly working boot env from April or the previous one from December 2024, the behavior is similar.  It boots, but very slowly, but svc.startd just repeatedly outputs Method or service timeout errors.   The boot env from December 2024 has failure messages from svc:/system/identity:node instead of svc:/network/location:default , but the behavior is otherwise the same.

I do have system backups so the most data I would lose is a day, but I'm really not prepared (time-wise or procedure-wise) to rebuild the system from scratch, so rescuing it (if possible) is much preferred.

Any suggestions for debugging the issue and/or recovery would be greatly appreciated.

Thanks,

Tim
--
Tim Mooney                                              Tim.Mooney at ndsu.edu
Enterprise Computing & Infrastructure    701-231-1076 (Voice)
North Dakota State University, Fargo, ND 58105-5164