[OpenIndiana-discuss] ZFS; what the manuals don't say ...

[Robin Axelsson]

Hi,
I've been using zfs for a while but still there are some questions that 
have remained unanswered even after reading the documentation so I 
thought I would ask them here.

I have learned that zfs datasets can be expanded by adding vdevs. Say 
that you have created say a raidz3 pool named "mypool" with the command
# zpool create mypool raidz3 disk1 disk2 disk3 ... disk8

you can expand the capacity by adding vdevs to it through the command

# zpool add mypool raidz3 disk9 disk10 ... disk16

The vdev that is added doesn't need to have the same raid/mirror 
configuration or disk geometry, if I understand correctly. It will 
merely be dynamically concatenated with the old storage pool. The 
documentations says that it will be "striped" but it is not so clear 
what that means if data is already stored in the old vdevs of the pool.

Unanswered questions:

* What determines _where_ the data will be stored on a such a pool? Will 
it fill up the old vdev(s) before moving on to the new one or will the 
data be distributed evenly?
* If the old pool is almost full, an even distribution will be 
impossible, unless zpool rearranges/relocates data upon adding the vdev. 
Is that what will happen upon adding a vdev?
* Can the individual vdevs be read independently/separately? If say the 
newly added vdev faults, will the entire pool be unreadable or will I 
still be able to access the old data? What if I took a snapshot before 
adding the new vdev?

* Can several datasets be mounted to the same mount point, i.e. can 
multiple "file system"-datasets be mounted so that they (the root of 
them) are all accessed from exactly the same (POSIX) path and 
subdirectories with coinciding names will be merged? The purpose of this 
would be to seamlessly expand storage capacity this way just like when 
adding vdevs to a pool.
* If that's the case how will the data be distributed/allocated over the 
datasets if I copy a data file to that path?

Kind regards
Robin.