[OpenIndiana-discuss] ZFS storage best practices

[Gary]

On 5/27/11, Dan Swartzendruber wrote:
> It all depends, I guess.  Some people get better performance with iSCSI,
> others with NFS.  The big advantage of going NFS is that the OI server
> can back up user data by snapshotting the zfs filesystem on the OI
> server (this is not the only kind of backup you want, of course...)

This is largely dependent on your infrastructure, file sizes, usage
patterns, applications, etc. At the office I'm in the middle of an
extensive comparison benchmarking analysis using iozone and the Excel
spreadsheet that I gladly bought from the author (I'll try the free
gnuplot scripts eventually). The presentation is slightly different
under LibreOffice but the data is still the same and I'd be glad to
post my files on DropBox for others to peruse if there's any interest.
Here's what I'm testing with...

Cisco:
Nexus series (can't recall which)

NetApp:
ONTAP 8.0.1 p2 - soon to be p3 with 10G ethernet ports

VMware:
Linux (CentOS) and Solaris x86 under VMware
  (10G backend from the cluster to storage but the virtual devices
limit it to 1G)

Solaris:
Solaris 10 on M5000 with 10G
  (have to check which sol update it's using but it was fully patched
on our last quarterly cycle)

Exadata X2-2:
  (40G Infiniband to cell storage in the rack)


I started on the Linux side with VMware hosts and so far I've tested
iSCSI single disk and mirrored volumes using soft RAID and LVM2 under
CentOS. I also tested NFSv3 and ran one test on the Exadata cell
storage. But I plan on moving to Solaris and Linux variants of NFSv4,
Oracle DirectNFS, and possibly some 8G fibre channel LUNs as well.
Eventually, I'd like to throw in a recent release of OI but it may
have to be run under VMware unless I can find some spare hardware or
the SPARC release comes out by then. Also, each of my tests take about
a half a day to run because I'm scaling all the way up to 16G file
sizes. I'm using non-prod infrastructure so I can start most of them
during the day but even our dev Exadata nodes are starting to get some
use during business hours that necessitates running in the evenings
and weekends.

-Gary