[OpenIndiana-discuss] Enhancement to IPS (pkg)??

[Alasdair Lumsden]

On 25 Oct 2010, at 05:20, Sunay Tripathi wrote:

>> I also don't think that the only "commercial viability" of OpenIndiana
>> will come from being to upgrade existing Solaris 10 installs.  Surely
>> we want it to be commercially viable for free-standing new deployments
>> as well -- a goal which is not, as far as I can tell, insurmountably
>> distant.
> 
> I am all for new installed base. All the power to you. I am being
> sincere here and not sarcastic. But can you explain what you are
> doing to achieve that? New features, new deployment model, or
> something else? Would appreciate a meaningful response instead of
> more chest thumping.

Hi Sunay,

Sorry for joining this discussion late - I've been somewhat distracted due to emergency home decorating and the $dayjob.

The goal with OpenIndiana is to provide a community driven free open source alternative to Solaris 11, with free bug fixes and security updates. With the recent change in the Solaris 10 licensing terms and demise of OpenSolaris, Solaris has essentially now become a proprietary legacy unix platform for the elite that can afford it.

This puts OpenIndiana in the unique and fantastic position (from our perspective) of being the de-facto distribution people will turn to when looking to use Solaris for situations where they cannot justify the cost of Solaris 11 (or where they would prefer to spend their money on things like hardware).

Solaris has some amazing technologies that are light years ahead of Linux, and quite frankly IMHO if we get this right, there should be no argument on earth for people to continue to use Linux. Our goal is to remove all the barriers stopping widespread adoption of OpenIndiana.

OpenSolaris was never capable of doing this so long as it failed to provide free security/bug fixes, and this is something we are aiming to address in Q1 2010 with our first stable release. Obviously to maintain this we'll need a dedicated team of packagers, security officers, etc, so we will need to continue to grow our developer base.

We're very keen to get people on board and we're very positive about the future - OpenIndiana has a fantastic opportunity here and we'd all love for it to be a great success.

In terms of what we're doing here to achieve those goals, I can summarise it with:

1. Provide a stable branch with bug and security fixes

2. Massively increase the availability of free and open source software available to install out of the box (Why on earth are packages like Postfix, Exim, Proftpd etc still missing from the base OS?). This includes providing things like ffmpeg, vlc, mplayer, etc. End users should not have to learn how to use pkgbuild/SFE/blastwave/etc to install useful software.

3. Excellent easy to use documentation; something like the FreeBSD handbook springs to mind. I always found it to be well written and very concise.

4. Address as many "paper cuts" as possible (https://wiki.ubuntu.com/PaperCut) - this means fixing things like out of date termcaps, ensuring the backspace key works instead of printing out ^H on the console, making sure lots of things compile out of the box, providing sane inputrc/bashrc/vimrc files, etc. It's the stupid little niggley things that put people off switching. OpenSolaris has already gone a huge way towards addressing this, such as adding much needed libc functions like vasprintf, putting /usr/gnu/bin at the head of the path, adding flags to "ps", etc. But we must continue this effort to ensure existing and new users are not needlessly frustrated by stupid silly things.

5. Ensure the OS is slick, sexy, and build up an excellent reputation. We want our users to think "Wow, this is *amazing*", so much so they will evangelise the OS and become fanatical about it, tell their friends about it, so their friends start using it. And with that, will come the small businesses, then the SMEs, and eventually enterprises.

The goal is to have more installs of OpenIndiana than Solaris 11. I am almost certain CentOS have achieved this vs RHEL, so I'm sure we can too.


>>> I don't think you understand how mirrors work.
>> 
>> Respectfully I believe that I do understand how mirrors work.  I've
>> been creating them one-way-or-another since before Oracle bought Sun.
>> Before OI you could create a mirror by downloading all of the
>> manifests and package files (using ips-mirror.py or some other script)
>> and run pkg.depotd out of the resultant tree.  Now that OI is here and
>> is actually interested in folks being able to mirror there are
>> official tarball drops and an rsync service available serving the
>> various pkg repos that you need to install OI.
> 
> The mirror code that we wrote actually still depends on the origin
> server being available. It was a mirror for data only. Maybe something
> changed in last few months but quick look at the putbacks don't
> show any explicit putback addressing this. Is this something you
> fixed personally or is fixed in OI? I would love to try this out.

You're right - the "mirror" functionality is completely pointless, the person that designed it seemed to miss the point that latency is responsible for the slow pkg experience, not bandwidth. Metadata operations are the bit that are painfully slow.

We have "solved" this by simply providing people with a tarfile of our entire pkg.depotd repo for them to use. I have no idea why Sun never did this. This way you get a fully blown authoritative pkg server with metadata. Spinning up a pkg.depotd server is trivial, and no harder to do than setting up a mirror via the other method.

However you're absolutely right, it should be much easier to clone a pkg server, and hopefully the pkg5 team will introduce functionality like that. If they don't, then we have the option of doing so, and providing our additions to them to use if they wish. Such is the beauty of being free from Sun/Oracle - things become a meritocracy and the best idea wins (in theory!).

The problem, as ever, is finding the time and/or the people to do the work.


>>> And the kernel is small part of all you need to have a operating environment.
>> 
>> Absolutely, I was merely highlighting the fact that there is
>> absolutely a present consumer of the file:// URI style and it clearly
>> functions against a local file-based repository layout.
>> 
>>> Do an experiment
>>> for me at your home. Disconnect the internet, just try your local
>>> machines as publisher and see if you can bring up a new system which
>>> has all that you need (to start with, just build and compile the
>>> ON bits).
>> 
>> You can absolutely do this today.  OpenIndiana provides a full copy of
>> both "/dev" (current OI packages) and "/legacy" (a mirror of the
>> OpenSolaris "/dev" repo).  If you download these and run a local
>> pkg.depotd then you can install without the Internet.  It also turns
>> out that you can download both of these and merely run an Apache
>> server with a few ksh scripts (as I described in my blog entry) to
>> provide the same functionality.
> 
> So you would expect people to write their own script to achieve this.
> Why would you not allow the pkg system to do it for you with a simple
> command? Running pkg.depotd is for developer and not end users in my
> opinion.

I guess the answer here is that what you're asking for can be worked around, but a clean easy to use end-user implementation is missing. It would be good to have this, but until the IPS folk do the work, or until we can find a developer willing to do the work, it might not happen quite so quickly.

There are a lot of higher priority things on the todo list at the moment, so a recipe/documentation people can follow to achieve the goal should hopefully be sufficient until such functionality is added.


>>> I do appreciate your enthusiasm in using things and that is the right
>>> approach. Things are by far no where near complete to be palatable in
>>> enterprise but let people like me worry about that.
>> 
>> I'm glad that people like you are worrying about it, but I think I'll
>> continue to worry about it as well if that's perfectly alright.
> 
> Sorry about the last comment. I think every community member should
> worry about it. And seems like you are quite capable. I am not trying
> to imply that you don't know what you are doing. I was just trying
> to steer you towards making it easy for people who don't have your
> skill set to actually be able to run Solaris.

Absolutely - we should all be thinking about OpenIndiana from an end user experience. This the point behind the "papercuts" thing above. We should make no assumptions about our users, other than that they have high standards, and expect things to work, for things to be intuitive and easy to use, and for things to be sane.

At the end of the day, we want newcomers to feel that this operating system is slick, well built, easy to use, technically excellent, and going places. So much so, that if they normally deploy Solaris 10, Linux, or FreeBSD, that after spending time with OpenIndiana, they would consider deploying it instead.

Hopefully this is a goal we can all agree on, and hopefully all contribute towards.

The more organised we are, the more positive and hopeful we are, and the more work we put in, the more likely we are to achieve our goals :)

So I would invite you to get stuck in and join our development effort if you're interested, we're primarily communicating in #oi-dev on irc.freenode.net. We also have a development mailing list. You may also want to look into getting involved with Illumos if you're interested in working on the OS/Net side of things - they're ripping subsystems out left right and centre at the moment and things are moving forward; it's an exciting time to be involved.

Cheers,

Alasdair