[OpenIndiana-discuss] [developer] Re: Install OpenIndiana in an UFS root

[wessels]

That's quite some offtopic discussion my original question triggered.
Since the original question remains unanswered I'll add my 2cents to
"the zfs on a cloud instance" discussion as well.


The main complaints seem to be ram consumption by ZFS and performance
issues in a "cloud/vm environment"
First the "ZFS ram consumption" or "the inability the run an OS with ZFS on
a low memory system".
Apparently few seem to care about his/her data. Some bit rot here or there,
who cares. Many seem to have great trust in the entire (stable) storage
datapath of their (cloud) systems. "I've never have any storage problems"
"I've a very expensive enterprise grade san" That's great, but do you know
for sure that the block just read and put in ram is the same block written
earlier? No you don't. Only a fully checksummed FS like ZFS can give you
that guarantee. I know we've stored our data for decades without a
checksumming FS. But that doesn't make this a non issue. Don't forget the
tremendous increase in storage. The chance of getting corrupt data served
is much higher than we would like to. The need for a checksumming FS or
not, reminds me of the need for ECC ram. If don't have ECC and few memory
banks and preferably low density the change of being troubled by a bit flip
is very slim.  But if your dram module would experience a bit flip, your
system wouldn't know and happily continue if it doesn't panic. Uber
cautious? Not me, please read the famous ECC study:
http://www.cs.utoronto.ca/~bianca/papers/sigmetrics09.pdf
So before complaining about ZFS ram consumptions please take into account
some of the benefits of ZFS. And if you would to have more control on how
ZFS uses memory please submit a webrev with a fix.

Than the complaint of running (multiple) instances of Illumos based OS in
the cloud or hypervisor is more expensive than other OS's. I think Keith
already explained that nicely. Running multiple copies of any OS
concurrently on a hypervisor is always less efficient than running those
copies via a method of OS level virtualization (like zones, jails or par on
aix) Imo your using the tool the wrong way. Install the OS, preferably on
bare metal, and create a zone for each app you need. You'll see that you
get a mileage from your machines as well. At least I do, ESX and apps in
vm's deliver fewer performance than running those apps in zones.  That's
how you should run concurrent instances on an Illumos kernel, not by
creating multiple instances of the OS on a hypervisor. If you must run
Illumos on a hvm or cloud instance try to make use of it OS level
virtualization instead of installing more vm's.


Lastly I would like acknowledge that (bare metal) Illumos lacks certain
features that some hypervisors and cloud providers offers. Most relevant
the "live migration" of it's workload, whether in zones or kvm. For some
that may be a showstopper. I fully understand. It's very nice to migrate
the workload to another machine and be able to do some planned maintenance
without disrupting any service. However if you've for example a scale-out
architecture you could easily exclude future requests from a machine. After
the machine is quiesced it can be serviced. But not everybody has that and
could not run their machines without live migration.

So Illumos is certainly not the last word in operating systems like ZFS is
in file systems.