[oi-dev] Distribution build system

[Andrzej Szeszo]

On 17/03/2011 13:26, Deano wrote:
> If we are going to this effort than we should IMHO do it probably and break
> the direct link between OI and consolidations. I can hear the sharp intakes
> of breath from here, from OI point of view consolidations are just groupings
> of certain packages and files put together by an upstream provider, they
> don't necessarily make any sense in term of useable packages down at the
> users end. Example abound of odd inclusions that made sense to the original
> consolidation author but not to the user (gruff is an known example).
>
>  From a users, developers and installers point of view, I want to
> get/update/remove X without knowing or caring about consolidations,
> therefore they should become hidden behind our one stop build system.
>
> If we whilst doing this work, we add a new OI thing (deliberately not giving
> it a name) that provides OI useful chunks and mapping internally through
> upstream consolidations and builds and grabs the bits of the version it
> wants, we achieve this independence at the same time.
>
> Effectively treating consolidation packages now as just another form of
> source tree, to mine and use as we see fit.
>
> Why go to this effect?
>
> Because at the moment we can't control where things go, what versions we use
> and even what our users can install/remove. By having a single build system
> work in chunks of OI decided work, we can easily begin to control
> where/what/how.
>
> You press the build button and it knows about upstream consolidation and OI
> chunks, out pops a distro with exactly in the state we set up, rather than
> constant fighting upstream changes when and if they change.  We take pieces
> as and when we require them and they meet our demands.
>
> In some ways I'm not suggesting dropping consolidations but instead have a
> system that mine's other consolidations. Adding a 'use file X in
> consolidation Y' rather than 'depend on consolidation Y' operator.
>
>
Breaking the link with upstream consolidations may be the thing we will 
ultimately have to do. For now, while the development of the 
consolidations like sfw, jds, xnv, ips, caiman happens in the open we 
probably want to continue using them in their current form for as long 
as possible. We don't have manpower to re-create all build recipes and 
packages at the moment.

Creating a wrapper on top of existing consolidations and their build 
systems would be the simplest thing to do right now.
>>> >>  Many people will say that this is not possible and that even Oracle is
>>> >>  not doing it. I say such system is possible but will require significant
>>> >>  amount of work and significant amount of time to prepare. It will be
>>> >>  worth the effort if done right. All major open source OS distributions
>>> >>  have the release process automated in one way or another. I think this
>>> >>  is the key to our success. Splitting the build and release engineering
>>> >>  process between consolidation maintainers/owners based on Oracle model
>>> >>  proved to not to work well for us.
>> >
>> >Anything is possible when it comes to code, and this is no exception.
>> >Anyone who says it can't or shouldn't be done is wrong.
>> >
>> >If Oracle aren't doing it, its because they are behind the times. We
>> >should work "smarter, not harder". Working smarter is definitely the key
>> >to our success.
>> >
>> >We are "stuck in the mud" until we do this.
> Yep I agree, I think however instead of digging ourselves out of the mud
> piece by piece, we should look at the tires themselves.
>
>>> >>  I would like to start a dialog between the core contributors about such
>>> >>  build system. Discuss whether it is needed or not. And if the decision
>>> >>  is made that it is needed - discuss requirements, technical details and
>>> >>  then actually implement it!
>> >
>> >It is needed. The decision is a no brainer, so it gets my vote. The
>> >question is how we implement it.
>> >
>> >Would anyone be interested in a Hackathon to get this work completed
>> >over a weekend in the near future?
> If the hackathon is about a month away I'd happily get involved (too busy
> right now).
>
> As a one stop build system is pretty big thing, I do think we should produce
> a discussion document, with both short and long term goals. I'd rather us
> spend longer and do it right, then quickly add another complicated layer
> that makes sense now but will just become another burden down the line.
I reckon we quickly figure out short list of requirements and start 
implementing the system. From top to bottom. Starting with the 
distro-importer and distro-constructor, just like Guido is suggesting in 
another message. Then move on to adding support for the actual build 
systems.
> I wasn't involved during the OSol era so I can't comment on whether the
> current overall distribution design was useful then, but at this moment, I'm
> not sure I see its strengths. Its seems instead of taking the amazing source
> and tech we have and pushing it forward we are largely fighting to keep our
> head above water and I largely see that as a fault of the development and
> distribution model which doesn't seem to fit a free open source OS.
>
> I might be wrong and its nothing to do with consolidations, but OI being
> able to take what it needs and requires without so many steps and obstacles
> seem to be the start of making OI fast enough to respond and push forward.
>
> Bye,
> Deano