[OpenIndiana-discuss] Split-root installations

[Jim Klimov]

On 2013-11-27 15:15, Peter Tribble wrote:
> NIC drivers can't live in /usr at all. That part
> isn't an interesting problem to solve.

Good to know. Then we don't declare that networking should restart
after the devfsadm re-run for the possibly newly mounted /usr might
find new drivers.

We do it only due to the programs which appear with this /usr... :)

> That (testing for /usr/bin) is not a valid test. That, or any number
> of random things from /usr,  might exist on the root filesystem,
> minimally populated with just enough for the system to boot,
> waiting for the genuine /usr to be mounted over the top of
> it when ready. The check would have to be an explicit check of
> whether /usr is a separate mountpoint. (Parse vfstab, maybe.)

Not really. If the rootfs is monolithic and includes root and /usr
as it does today by default, there won't be a separate mountpoint,
but all the needed programs would be delivered - as they are today.
They would just be stored with no/low compression, that's all...

> But that really tells you what the proper way to solve this problem
> is - ensure that what you need from /usr is actually present before
> /usr gets mounted. Either by having the minimal subset of the files
> copied into a skeletal /usr that you mount over the top of, or move
> the files out of /usr into somewhere that will always be available
> early in boot (that's what we have /sbin and /lib for).

Thanks, I did consider some of these ideas as well :)

On one hand, if we're talking about getting just the physical/nwam
script to work, this is indeed a small subset of programs that should
be available as builtins in the shell or as binaries (in /sbin), the
latter including pkill (or ps for parsing its output - or revert to
looking into /proc?), sort (may be replaced with the snippet from
Lionel Cons); maybe ls; cut, cat, grep and some others are acceptably
implemented as ksh93 builtins.

It may be reasonable for a lot of other reasons, such as to simplify
the OS repairs in case only the minimal rootfs has got mounted, to
fully relocate svcs, svcadm, svcprop and svccfg and their libs into
/sbin+/lib as well, and make compatibility symlinks from their
original locations in /usr/{s}bin. Basically this would repeat what
I do for /sbin/sh (manually now, in-distro hopefully sometime in the
future). And some other programs like df, ls, less, vi, maybe bash...
The "minimal" root would not be so minimalistic as today, although
the full /usr filesystem remains a much larger monster anyway.

The specific set of programs that should be delivered this way
depends on what is used in those startup scripts that believe
they must be executed before /usr is fully mounted.

On the other hand, your suggestion about providing some /usr content
in the minimal root filesystem in case of a split-root is indeed
possible, but much harder to maintain than packaged relocation of
the files involved and a system of compatibility-symlinks. For one,
updating such files during package upgrades, when the /usr path is
in fact over-shadowed by a mounted FS, would be problematic. Also,
this would get us to pay the price for storing the files twice (in
the uncompressed or lz4 rootfs, and in the highly compressed usr).

However, for the sake of compatibility and least change to other
code, the rootfs-based skeleton of /usr/{s}bin/* could deliver the
symlinks to programs actually stored (once) in /sbin. Maintaining
this structure after initial installation of the split-root might
be as difficult, but it hopefully won't be needed as often as when
this directory would deliver real files that need upgrading.

Since the mounting of /usr in fs-root is explicitly overlaid (-O),
content of the skeleton /usr would not prevent the system from
starting and mounting well. And if the system is not split-root,
then its monolithic rootfs+usr would contain these skeleton symlinks
and a lot more under the /usr directory.

Would you argue that this is the proper solution, and things like
/sbin/vi or /sbin/ls are the acceptable "little blood" in this
operation?

And this may also be a bog - for example, the NWAM method can
call ldapclient whose work might require R/W access to /var/ldap or
something like that, and it also can use /var/yp for NIS/YP stuff
and the /var filesystem is not expected to be available until much
later in the boot.

Arguably, however, NWAM should not be used on network-booted systems
due to numerous dependencies on a fully-fledged rootfs hierarchy,
and can thus be remade to depend on filesystem/local for example,
by breaking the dependency cycle elsewhere... what do you think?

Possibly, it may be reasonable to create a service instance which
would depend on both nwam and filesystem/local, and reference the
same method script, so that if NWAM is enabled, it would complete
initialization after the root is fully available - and it can still
provide something based on the miniroot that the system starts with?

Thanks,
//Jim Klimov