[OpenIndiana-discuss] Moving /var and/or /usr to own zfs filesystem

[Jim Klimov]

On 2013-09-04 21:24, Christopher X. Candreva wrote:
> No, I did not. However the machine that IS running doesn't have it set to
> legacy either. (And checking, it's just /var -- what I thought was /usr is
> an artifact of how I created /usr/local
>
> here is the output of df:
>
> chris at Jubal:~$ df -h
> Filesystem            Size  Used Avail Use% Mounted on
> rpool/ROOT/openindiana
>                        134G  2.1G  132G   2% /
> rpool/ROOT/openindiana/var
>                        132G  400M  132G   1% /var

okay, so this part seems to be a pretty standard split, probably helped
by the setup wizard? I think it supports making separate /var datasets.

> rpool/usr             132G   32K  132G   1% /rpool/usr
> rpool/usr/local       132G   31K  132G   1% /rpool/usr/local

And the latter should have had mountpoint=/usr/local set explicitly.
Or the rpool/usr might have "canmount=noauto mountpoint=/usr" to the
same effect, except that you might make more datasets under /usr,
which you usually don't need.

What sort of software do you have in /usr/local? If it is packaged
(home-brewn, or from collections like SunFreeware), this should really
be part of the root filesystem (fs-tree as of now) and ultimately it
is individual for each BE managed by packaging and beadm.

>
> chris at Jubal:~$ zfs get mountpoint rpool/ROOT/openindiana/var
> NAME                        PROPERTY    VALUE       SOURCE
> rpool/ROOT/openindiana/var  mountpoint  /var        local
>
>
> I'm wondering if the fact that I made it under ROOT/openindiana is why it
> "just works".

In my OI systems it usually "just works" even if the mountpoint is
inherited from the current root dataset (rpool/ROOT/openindiana in
your example), and if its mountpoint is in turn set to "/". Problems
arise when you boot another BE without changes via beadm (perhaps as
part of systems repairs), and the root dataset is "/a" (but hardcoded
to mount as "/") and its children are like "/a/var" and mount into
wrong positions or not at all. I am not sure I've ultimately published
my solution to this particular problem, but I did make related changes
to the SMF methods at least on one box, for certain :\

Also note that some filesystems are used before the mounting scripts
are run (actually, they are run in several layers, with other services
in-between), so I found that /var/tmp can be filled with some files
before it is mounted from another store. There are ways to work around
this, but it's easier to just not split this off.

Another problem is with /sbin/sh being a symlink to /usr/bin/ksh93.
When you just split-off the /usr filesystem, you can't boot anymore.
You have to carry over ksh93 (and a few libraries it may depend on)
into the root fs, and still be ready for adventures (i.e. have another
BE and/or Live Media ready for repairs of the main BE).

>
> (This is NOT a production system, just a test install I made and left
> running to see how stable it would be on this hardware).
>
>
> Thanks to both of you, I'll run a few more tests. I am guessing the problem
> with the way I did it is that it will still make different var volumes
> (paritions ? zfs's ?) for each BE ?

ZFS filesystem datasets, different - yes. And for many intents and
purposes, it is not a problem but rather a desired result ;)

Each BE is cloned from another usually, so the disk space consumed
is that of differing files between the two BE's.

>
> What is the dounside of using legacy mounts and /etc/vfstab ?

I'd say flexibility. In /etc/vfstab you spell out the full path to
the backing device of a filesystem (a ZFS dataset in this case),
and for each BE you'd have to edit this file and rename the parent
dataset (whose name coincides with BE name).

>
> I'm thinking now about just leaving /usr and only separating /var (and
> usr/local) to their own filesystems. My reasons for /var is simply to be
> able to separate out the space consumed by log files and everything else. In
> fact, I've thought about making var/spool it's own file system for a long
> time for similar reasons: Large mail run doesn't stop logging, and
> vice-versa.

If you read my posts in the Wiki and issue-tracker, you can find
proof (and steps to repeat) regarding more complex setups. Still...
You might find it more flexible to run such tasks in local zones.
You can delegate a ZFS hierarchy into a zone for its admin to manage,
and set it up in whatever branchiness you'd like (and use different
compression algorithms, quotas, etc.). Generally this zfs structure
is mounted before the zone daemons are running, and this avoids many
of the caveats related to similar setup for the global zone.

Think of the GZ as a hypervisor, running little more than SSH for
your administrative jobs (and maybe VNC for jobs which require X11
or just long-running so they don't abort on SSH hiccups), and maybe
local X11 GUI for interactive use. Active real life happens in local
zones, abstracted from hardware and many other low-level issues :)

HTH,
//Jim Klimov