[OpenIndiana-discuss] ZFS on Openindiana vs ZFS on oracle linux

[Doug Hughes]

On 4/21/2017 2:06 PM, C. R. Oldham wrote:
> For Ubuntu, it is an effort to get ZFS on the root partition.  See
> these wiki entries:
>
> https://github.com/zfsonlinux/zfs/wiki/Ubuntu
>
> I have several machines installed with these instructions on 16.04,
> 16.10, and 17.04.  They do work, but it's a pain to get it installed.
>
> Note that ZoL does have a couple of limitations that can make it
> challenging to use it as the backing store for KVM.  We have an
> OpenNebula private cloud and some of qemu's caching options will cause
> the VMs not to start, claiming the FS underneath doesn't expose the
> right primitives.  I left the OS on ZFS on those machines and
> converted the partitions where the VMs keep their disks to XFS.
Not to Hijack, but:

I struggled with KVM on XFS for months before giving up. XFS is somewhat
fundamentally incompatible with qcow2. If you are using raw, I'm sure
it's great. Sine we use snapshots for backups and replication, the qcow2
functionality is a must-have. Even on a 96GB machine on a large (~40TB)
qcow2 image, the fragmentation in XFS got to be enough that it was
struggling to get 4MB allocations for XFS, and XFS would just fill the
logs. Even echo 3> /proc/sys/vm/drop_caches and tuning linux to have at
least 8MB available and use hugetlb and tuning scanning were not enough.
XFS would get irrevlocably stuck and require a hard reset of the entire
machine to free.

You *can* tune xfs allocation sizes, but only on an empty file. You
can't do it after the fact, which makes discovering that very annoying.
You can also completely preallocate a qcow2, but I'm not sure it's
helpful in the long run.

Running KVM on ZoL on the other hand, has been wonderful so far with
qcow2. (CentOS7.2)

Anyway, just saying, XFS is great until it isn't, and it's failure cases
will bite you months down the road and you may not have options. I'm
looking forward, soon, to having another zfs machine to replicate too,
then I'll use KVM+QCOW2 to take the snapshot, use zfs to replicate, then
blockcommit and done. Right now it's rsync and it takes way too long.