[OpenIndiana-discuss] ZFS High-Availability and Sync Replication

[Doug Hughes]

On Tue, Nov 20, 2012 at 7:01 AM, Edward Ned Harvey (openindiana) <
openindiana at nedharvey.com> wrote:

> > From: Doug Hughes [mailto:doug at will.to]
> >
> > Well, to me, the most obvious is use another box with ZFS to mirror the
> > ISCSI devices on ZFS. I'm in the process of doing this myself with some
> > stuff. 2 head nodes and 2 storage nodes. The storage nodes are targets
> > and the head nodes are initators. You get performance and reliability,
> > and failover between head nodes is relatively easy with zpool import.
>
> This is precisely the thing I did with poor results, except that I didn't
> have 4 nodes - I had only 2.  So each node was both an initiator and a
> target.  I made the initiator service dependent on the target service (so
> the initiator would go down first and come up second).  I made zpool import
> of the iscsi pool dependent on the initiator being up, and I made vbox
> dependent on the iscsi zpool mounter.
>
> (see simplesmf, http://code.google.com/p/simplesmf/ for the zpool
> dependent on iscsi initiator)
>
> My word of advice, until such time (which may never come) that I might
> know more...  Never, ever let the iscsi targets go down unless the
> initiators are both down first.
>

strange. I tested failing the targets while the initiators are up and
failover worked fine. One half of the mirror went away and everything
worked as expected. I wonder if you might be running into some race or
other strange locking condition in the ISCSI kernel code being both target
and initiator for the same mirrored zpool? I tested it pretty thoroughly.
Aside from needing to run tcpkill to make Linux NFS clients really give up
on the current NFS/TCP connection and reestablish (or wait 5 minutes),
things worked as expected.