[OpenIndiana-discuss] Split-root installations

[James Carlson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/26/2013 5:51 PM, Peter Tribble wrote:
> The miniroot for any live or network boot already has a chunk of
> /usr. There's a real problem here in that there simply isn't a
> clean separation of root vs usr any more. If you really want to
> have a supportable separate /usr, then you're going to have to
> work through the entire system architecture working out which file 
> goes where and why.

Big +1 on this.  When I worked at Sun, supporting the hard split for
/usr was an architectural nightmare.  The *only* customer-related
reason it existed at all was to allow for NFS-mounted /usr in ancient
systems where a good hard drive was a few tens of megabytes, and
someone might have a cluster of IPCs being used as workstations.
Those days, along with hair bands, A.L.F., and John Hughes movies, are
long gone.

At some point along the line, we decided that the cost was out of line
with the benefit.  You can't buy disk drives small enough to make
sense out of a split /usr.  It's even tough to do if you go for exotic
non-rotating media, so there's little or no obvious benefit.  In terms
of cost, having a supported split /usr is a huge tax on every single
project that integrates: every project needs to consider what file
paths are available, when they are available, and then has to multiply
its test matrix by two (or more) to cover the extra options.

So, we started mostly ignoring it.  I believe the only support left in
there was a separate /usr mounted *locally* through normal disk
drivers so that / and /usr could have different mount options, but
even that wasn't really tested or thought too carefully about, so I
would not be shocked at all to find that an arbitrary collection of
system services are damaged if you attempt it.

I'm not trying to rain on someone's parade here, but I do suggest
considering the cost/benefit ratio when looking at restoring the bad
old days.  Reliability is in many ways driven by simplicity.

> (As an aside, I would really like to go back to the days where
> /lib was a symlink to /usr/lib, and things were much cleaner. But
> that ship sailed, and I fear it's not coming back.)

This is the one narrow spot where I disagree with you.  The reason the
symlink was removed is fairly deep and is a good thing, not bad.

(Speaking from a historical prospective here ...)  Solaris does not
have (and has never had) an ABI defined at the syscall interface.  The
syscall interface is a mere implementation detail.  It must be this
way in order to allow kernel interfaces to be improved in patches,
because the syscalls expose and depend on internal artifacts.  The key
architectural idea is that the ABI, which is what provides the
impressive application compatibility across releases for which Solaris
is known, is defined by the documented system-supplied libraries.

The old /sbin had no access to /lib, because / was mounted before
/usr.  As a consequence, the objects in /sbin had to be statically
linked executables.

These non-ABI-compliant things caused some important problems.  Chief
among them was patch bloating: any time you touch the main parts of
the kernel, you usually have to modify or at least recompile libc as
well.  But since those nasty static executables have embedded copies
of libc, you have to recompile (or at least relink) them as well, and
toss them into the same patch.  They cause at *least* bloating of the
patches, and sometimes even caused incorrect patches to be generated
because the humans who create patches don't always know what to
include(!).  It's a substantial cascade.

It much was worse when users had access to the execrable "libc.a"
turds.  They were able to create their own binaries with embedded
copies of libc and other system libraries, and thus a *HIDDEN* and
*UNFIXABLE* direct dependencies on the undocumented, unstable,
not-to-be-relied-on syscall interface.  If there was a bug in any
system library, it simply could not be fixed.  Ever.

When we finally got rid of the architectural wart that was static
linking, we realized we could do ourselves the same favor.  So, no
more statically linked crap in /sbin.  libc.so.1 moves to /lib, along
with the handful of things needed before /usr is mounted and /usr/lib
becomes available.

This promotes page sharing; simpler, smaller and more accurate
patches; and good architectural hygiene with no real pain.

(And, yes, we did internally have the discussion about what to do if
the single copy of /lib/libc.so.1 was somehow "broken."  There were
some who felt that statically linked executables might allow someone
to copy in a "correct" version of libc.so.1 from somewhere else and
thus fix a broken system.  But raf and the other good folks who
created /lib pointed out that if your system is broken, you shouldn't
be performing microsurgery on it like that.  You should boot from an
alternate medium -- a DVD, an old BE, a USB stick -- and mount up the
root for repair using real, supported tools; not just copied bits and
binary hackery.)

It's a bit of a shame that the extensive internal architectural
documentation for the system isn't available.

>> What would the esteemed community experts suggest?
>> 
>> 
> My worry here is that you're adding complexity to an already 
> complex system; instead, we need significant simplification.
> 

Another big +1 on that.

- -- 
James Carlson         42.703N 71.076W         <carlsonj at workingcode.com>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)

iEYEARECAAYFAlKWPYEACgkQ6TIHA1NkzvCV5gCfX0HJfLH4l5BVFGoJa8HQJh40
fcsAn3I8oQLSA3cJoDJuyoMgSnIOrlZZ
=kDbu
-----END PGP SIGNATURE-----