[OpenIndiana-discuss] best migration path from Solaris 10

[Paul B. Henson]

We've been running Solaris 10 for the past couple of years, primarily to 
leverage zfs to provide storage for about 40,000 faculty, staff, and 
students as well as about 1000 groups. Access is provided via NFSv4, 
CIFS (by samba), and http/https (including a local module allowing 
filesystem acl's to be respected via web access). This has worked 
reasonably well barring some ongoing issues with scalability 
(approximately a 2 hour reboot window on an x4500 with ~8000 zfs 
filesystems, complete breakage of live upgrade) and acl/chmod 
interaction madness.

We were just about to start working on a cutover to OpenSolaris (for the 
in-kernel CIFS server, and quicker access to new features/developments) 
when Oracle finished assimilating Sun and killed off the OpenSolaris 
distribution. We've been sitting pat for a while to see how things ended 
up shaking out, and at this point want to start reevaluating our best 
migration option to move forward from Solaris 10.

There's really nothing else available that is comparable to zfs (perhaps 
btrfs someday in the indefinite future, but who knows when that day 
might come), so our options would appear to be Solaris 11 Express, 
Nexenta (either NexentaStor or NexentaCore), and OpenIndiana (FreeBSD is 
occasionally mentioned as a possibility, but I don't really see that as 
suitable for our enterprise needs).

Solaris 11 is the official successor to OpenSolaris, has commercial 
support, and the backing of a huge corporation which historically has 
contributed the majority of Solaris forward development. However, that 
corporation is Oracle, and frankly, I don't like doing business with 
Oracle. With no offense intended to the no doubt numerous talented and 
goodhearted people that might work there, Oracle is simply evil. We've 
dealt with Oracle for a long time (in addition to their database itself, 
we're a PeopleSoft shop) and a positive interaction with them is quite 
rare. Since they took over Sun, costs on licensing, support contracts, 
and hardware have increased dramatically, at least in the cases where 
we've actually been able to get a quote. Arguably, we are not their 
target market, and they make that quite clear ;). There's also been 
significant brain drain of prior Sun employees since the takeover, so 
while they might still continue to contribute the most money into 
Solaris development, they might not be the future source of the most 
innovation. Given our needs, and our budget, I really don't consider 
this a viable option.

Nexenta, on the other hand, seems to be the kind of company I'd like to 
deal with. Relatively small, nimble, with a ton of former Sun zfs talent 
working for them, and what appears to be actual consideration for the 
needs of their customers. I think I'd more likely get my needs addressed 
through Nexenta, they've already started work on adding aclmode back and 
I've had some initial discussion with one of their engineers on the 
possibility of adding additional options such as denying or ignoring 
attempted chmod updates on objects with acls. It looks like they only 
offer commercial support for NexentaStor, not NexentaCore. Commercial 
support isn't a strict requirement, a sizable chunk of our 
infrastructure runs on a non-commercial linux distribution and open 
source software, but it can make management happier. NexentaStor seems 
positioned as a storage appliance, which isn't really what we need. I'm 
not particularly interested in a web gui or cli interface that hides the 
underlying complexity of the operating system and zfs, on the contrary, 
I want full access to the guts :). We have our zfs deployment integrated 
into our identity management system, which automatically provisions, 
destroys, and maintains filespace for our user/groups, as well as 
providing an API for end-users and administrators to manage quotas and 
other attributes. We also run apache with some custom modules. I still 
need to investigate further, but I'm not even sure if NexentaStor 
provides access into the underlying OS or encapsulates everything and 
only allows control through its own administrative functionality.

NexentaCore is more of the raw operating system we're probably looking 
for, but with only community-based support. Given that NexentaCore and 
OpenIndiana are now both going to be based off of the illumos core, I'm 
not quite certain what's going to distinguish them. NexentaCore will 
continue to leverage debian packaging as opposed to IPS, and currently 
defaults to a GNU userland rather than native Solaris userland. However, 
I've read that the next version is going to switch to the solaris 
userland, so will no longer be different from OpenIndiana in that respect.

OpenIndiana was initially positioned to be a binary compatible 
open-source equivalent to Solaris 11, such as CentOS is to Red Hat. 
However, given that Oracle has not released any of their internal 
development work on which Solaris 11 express is based, and there's 
really no certainty that it ever will, I'm not sure how that's going to 
be accomplished. Particularly in the case of zfs, Oracle has moved 
forward with new pool versions which are not supported. Assuming Oracle 
does not release the corresponding source code, is the intention to 
duplicate the functionality with separately written code? And what will 
happen if for example an innovative new zfs feature is developed on the 
open source side, requiring a new pool version? Oracle seems to own that 
namespace.

My first inclination is to prefer OpenIndiana, which would seem to be 
the most flexible option. I particularly like to be able to fix 
something in the code myself and immediately deploy it and then try to 
get it accepted upstream rather than spending ages trying to get some 
support person to agree it should be done at all, let alone having an 
engineer update proprietary source code to implement it. I've got Sun 
(Oracle) support tickets that have been open for over a year waiting to 
get things fixed that I could have fixed myself.

Nexenta on the other hand has the only product ready for short term 
production deployment (well, ruling out Oracle, which I mostly have), as 
I don't think there is a firm timeline yet on a production ready version 
of OpenIndiana. Of course, given the major changes in NexentaCore 4 
dropping a new deployment of NexentaCore 3 into place right now might 
not be the best idea either, it would probably be better to wait for 4 
to come out.

Sorry, I think I'm starting to ramble :). So, for those people with 
similar deployment needs as myself, with a large number of filesystems, 
a large population, and access via multiple protocols, what are you 
currently running? What do you plan to be running in the short to mid 
term? And why :)?


-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating Systems and Network Analyst  |  henson at csupomona.edu
California State Polytechnic University  |  Pomona CA 91768