[OpenIndiana-discuss] Split-root installations

[Peter Tribble]

On Wed, Nov 27, 2013 at 11:16 AM, Jim Klimov <jimklimov at cos.ru> wrote:

> First of all, thank you for sharing your advice.
> Coming from a distro maintainer, it is very valuable :)


Building a distro (largely from scratch, in my case) teaches you
an awful lot about how the system really works. Far more than I
ever wanted to know, to be honest.


> On 2013-11-27 11:03, Peter Tribble wrote:
>
>> On Wed, Nov 27, 2013 at 12:51 AM, Jim Klimov <jimklimov at cos.ru> wrote:
>>
>>> So far the best shot, both compact and effective, was fixing the
>>> filesystem/root service like in the patch below. Are there any
>>> fundamental objections to this sort of fix beside possible lack
>>> of style? Getting the hands wet in the guts of the system anyhow,
>>> can't keep them clean... ;)
>>>
>>
>> General comment - attempting to hack workarounds to broken
>> systems at the wrong place in the startup sequence seems like
>> a really bad idea.
>>
>
> Yes, maybe this is a workaround instead of a "proper solution"
> with a re-architecture of the system startup with unknown
> repercussions of the overhaul. I might not be the sort of
> specialist (or don't have enough spare time) needed to do
> the overhaul, but I can patch-in a working workaround which
> lets systems work. Even if it is ugly.
>
> Understanding the drawbacks and tradeoffs, which is what you
> help me do, is of course important. But one of the matters
> to consider is whether the stuff which may be broken after
> applying my workarounds did/could work well before them?
>
>
>  How can adding /usr change the validity of the network
>> configuration? There shouldn't be any new drivers (or
>> network booting wouldn't work at all), so there should be
>> nothing to do here.
>>
>
> Again, this is a theoretical construct which depends on things
> we can't really manage - delivery of drivers by their vendors.
> For good or bad, the /usr/kernel/drv is a valid location...
>
> Say, your small root delivers the illumos drivers like e1000g,
> and this is enough to mount the rootfs (if networked, or this
> doesn't matter at all for a local rootfs). Then your /usr comes
> up with, say, a vendor-provided package of VendorX NIC drivers.
>

That'll never work. NIC drivers can't live in /usr at all. That part
isn't an interesting problem to solve.

(In fact, splitting some minuscule fraction of the available drivers
off into /usr is an idea whose time is long past - you're adding
complexity for maybe 1-3% of the overall driver space usage.
Might have to keep it for compatibility, but it should be obvious
that you can't ever have critical drivers in /usr.)


> In fact, in case of a networked boot, this may be a different
> /usr filesystem image from what you had in the miniroot archive
> that the bootloader fetched from the network to load the kernel
> and initialize the system far enough to get networked filesystems.
> And in a split-root local filesystem you had no /usr at all.
>
> Your network/physical service, be it legacy-default or nwam,
> has already started, because it is earlier in SMF dependencies.
> Let's assume for now that the bugs discussed now have been fixed,
> and some NICs have been actually configured by this service.
> This service is expected to have configured all NICs on the system.
> Apparently, it could not plumb those NICs that it did not have
> drivers for at that time...
>

Again, a non-existent problem. You can't realistically load
network drivers from /usr and expect the system to work. I've
never seen one. If you did have one, then it's some whacko
oddball that probably needs to come with its own initialization
process and would be outside the regular system startup. Not
our problem.


> So adding /usr can change the validity of the network config.
>
>
>  If networking is broken at this stage (and it shouldn't be) then
>> that's a bug elsewhere - fixing fs-root is the wrong place.
>>
>
> At the very least, and as I wrote before, I found this problem
> *because* the "physical" startup methods fired before the separated
> /usr was mounted. And their SMF dependencies require that actually,
> which may be reasonable for network boots and is unreasonable for
> fully local-device boots.
>
>
>  restart = disable and enable. If your /usr is nfs mounted,
>> what happens when you disable networking?
>>
>
> I see a good point here, thanks :)
>
> So this trick with restarts should take place only in case that
> I am testing - with the locally mounted /usr filesystem (though
> how can we determine this well with i.e. iSCSI pools?), or perhaps
> a check that there was no /usr/bin at start of the script and
> there is one after it has done its work - actually this is what
> I am trying to counter here?..
>

That (testing for /usr/bin) is not a valid test. That, or any number
of random things from /usr,  might exist on the root filesystem,
minimally populated with just enough for the system to boot,
waiting for the genuine /usr to be mounted over the top of
it when ready. The check would have to be an explicit check of
whether /usr is a separate mountpoint. (Parse vfstab, maybe.)

But that really tells you what the proper way to solve this problem
is - ensure that what you need from /usr is actually present before
/usr gets mounted. Either by having the minimal subset of the files
copied into a skeletal /usr that you mount over the top of, or move
the files out of /usr into somewhere that will always be available
early in boot (that's what we have /sbin and /lib for).

-- 
-Peter Tribble
http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/