[oi-dev] Distribution build system

[Deano]

Likely controversial view inline below...

-----Original Message-----
From: Alasdair Lumsden [mailto:alasdairrr at gmail.com] 
Sent: 17 March 2011 10:59
To: oi-dev at openindiana.org
Subject: Re: [oi-dev] Distribution build system

<snip>

>> Having such system will let us to release early, release often. It
>> should improve the development progress in general. Bugfixes and
>> security updates would get integrated in no time. New users would have
>> an easy start. We could point new contributors at the build system and
>> simply ask them to start hacking. Having all consolidations referenced
>> from a single build system would make it clear to them where things go.
>> Base system changes, etc. - core consolidations, new software - add-on
>> consolidation directory, and so on.
>
>*nods*
>
>That way we can have a standard for how things are laid out on the 
>filesystem, for how patches are applied, what environment is used, etc.
>
>This will also ensure everyone is building things in the same way, which 
>will reduce time wasted diagnosing build issues.

If we are going to this effort than we should IMHO do it probably and break
the direct link between OI and consolidations. I can hear the sharp intakes
of breath from here, from OI point of view consolidations are just groupings
of certain packages and files put together by an upstream provider, they
don't necessarily make any sense in term of useable packages down at the
users end. Example abound of odd inclusions that made sense to the original
consolidation author but not to the user (gruff is an known example).

>From a users, developers and installers point of view, I want to
get/update/remove X without knowing or caring about consolidations,
therefore they should become hidden behind our one stop build system.

If we whilst doing this work, we add a new OI thing (deliberately not giving
it a name) that provides OI useful chunks and mapping internally through
upstream consolidations and builds and grabs the bits of the version it
wants, we achieve this independence at the same time.

Effectively treating consolidation packages now as just another form of
source tree, to mine and use as we see fit.

Why go to this effect?

Because at the moment we can't control where things go, what versions we use
and even what our users can install/remove. By having a single build system
work in chunks of OI decided work, we can easily begin to control
where/what/how.

You press the build button and it knows about upstream consolidation and OI
chunks, out pops a distro with exactly in the state we set up, rather than
constant fighting upstream changes when and if they change.  We take pieces
as and when we require them and they meet our demands.

In some ways I'm not suggesting dropping consolidations but instead have a
system that mine's other consolidations. Adding a 'use file X in
consolidation Y' rather than 'depend on consolidation Y' operator.


>> Many people will say that this is not possible and that even Oracle is
>> not doing it. I say such system is possible but will require significant
>> amount of work and significant amount of time to prepare. It will be
>> worth the effort if done right. All major open source OS distributions
>> have the release process automated in one way or another. I think this
>> is the key to our success. Splitting the build and release engineering
>> process between consolidation maintainers/owners based on Oracle model
>> proved to not to work well for us.
>
>Anything is possible when it comes to code, and this is no exception. 
>Anyone who says it can't or shouldn't be done is wrong.
>
>If Oracle aren't doing it, its because they are behind the times. We 
>should work "smarter, not harder". Working smarter is definitely the key 
>to our success.
>
>We are "stuck in the mud" until we do this.

Yep I agree, I think however instead of digging ourselves out of the mud
piece by piece, we should look at the tires themselves.

>> I would like to start a dialog between the core contributors about such
>> build system. Discuss whether it is needed or not. And if the decision
>> is made that it is needed - discuss requirements, technical details and
>> then actually implement it!
>
>It is needed. The decision is a no brainer, so it gets my vote. The 
>question is how we implement it.
>
>Would anyone be interested in a Hackathon to get this work completed 
>over a weekend in the near future?

If the hackathon is about a month away I'd happily get involved (too busy
right now).

As a one stop build system is pretty big thing, I do think we should produce
a discussion document, with both short and long term goals. I'd rather us
spend longer and do it right, then quickly add another complicated layer
that makes sense now but will just become another burden down the line.

I wasn't involved during the OSol era so I can't comment on whether the
current overall distribution design was useful then, but at this moment, I'm
not sure I see its strengths. Its seems instead of taking the amazing source
and tech we have and pushing it forward we are largely fighting to keep our
head above water and I largely see that as a fault of the development and
distribution model which doesn't seem to fit a free open source OS.

I might be wrong and its nothing to do with consolidations, but OI being
able to take what it needs and requires without so many steps and obstacles
seem to be the start of making OI fast enough to respond and push forward.

Bye,
Deano