[oi-dev] OI project reboot required

Thu May 9 17:47:33 UTC 2013

Privet !

On Thu, May 9, 2013 at 1:45 PM, Jim Klimov <jimklimov at cos.ru> wrote:

> On 2013-05-09 13:06, Andrzej Szeszo wrote:
>
>> The process you have described sounds a lot like OI's original plan. It
>> didn't work out. There was too much baggage. No one was willing to spend
>> time learning it. It was just too ... ugly.
>>
>
> It's possible to try it differently this time :)
>

One of the main reasons why it is complicated to build OpenSolaris, is
definitely IPS itself.
Rather than just the fact, that almost every of the consolidations has
different ways in which they implemented their build, with different
locations for Makefile.master, a different directory hierarchy versus even
with pkgbuild and .spec files.
To resolve deps, that's cpu intensive like little else and also error prone.

>
> One way or another, in these consolidations we have tons of source
> code, which, one way or another, need a process to build them.
>
> Even if the system of build scripts, spec recipes, makefiles and so
> on would be changed to something else, just to develop this new build
> logic and to check validity and comparability of the resulting packages
> we (individual developers) need to be able to build it "the old way"
> and "the new way" (whatever that would be). AFAIK, very few people
> know what to do to create the binaries and packages in the old
> techniques, which makes it problematic for others (not "in the know")
> to start improvements or from-scratch rewrites.
>

Starting in 2007 I had the same dream. And I even started and made some
progress (still have it somewhere in my backup archives). My idea was, to
merge all consolidations (for MartUX) into one single Makefiles system,
namely into that of Sun's X11 group, as developed by Alan Coopersmith and
team on the basis of OS/Net's Makefiles. I think Moinak Ghosh has worked on
something similar even before me (as I learned later), and has come
further. So I stopped the effort. I recall that he released this on
belenix.org a long time ago. If I'm corerct he once called it
DistroGenerator (don't confuse this with DistroConstructor, which, btw, HE
also invented long before Sun took err stole it).The first so called
"Indiana" was in fact BeleniX in 2007!!! And is was completely unfair and
injust that Sun had hired that Ian Murdock destructor purely for marketing
reasons alone, and that he was permitted to give the stolen BeleniX his
name. This is the primary reason why those that remember all this
(including myself) could never be happy with the name "ind_IAN_a".

> Of course, some architectural overview or general framework for all
> consolidations of what we want to achieve as the new build system
> (i.e. "conforms to this list of technical requirements: A, B, C... Z")
> would be needed.
>
> I may be wrong to think that ability to build the whole beast in the
> old ways is useful to make the new building technology, but lack of
> it also hinders an ability to make new spins of the whole OI distro
> or just updated package repos, as one other commonly-requested useful
> result of the overall OI project. So far most users can rebuild the
> "kernel" part (illumos-gate) of their installations and third-party
> code like updated sendmail, apache or whatever they need to update,
> rebuilt and installed in traditional unpackaged ways into alternate
> paths or even overriding the system binaries, but that's about it.
> And this is wrong, especially for SOHO farms of several OI machines
> or zones which could otherwise reuse the in-house updated packages
> automatically and properly :)

I made bad experiences with IPS.
With SVR4 all this is a ton (1000kg!) easier, more transparent, and more
fine grained.
You can touch every package, even inside the repo, by hand.
It is not a database like monster, black box or even rather black hole,
that IPS appears to be, in my personal view (sorry about that).

Example: If you want to change a single package, maybe even a single file,
or an arbitrary number of packages, it is always easy, transparent and
doable: http://svr4.opensxce.org/sparc/5.11/

For example, if a new Firefox comes out, many users care about "the newest
version, yeah".
Here is the SVR4 way:

0.) Delete
http://svr4.opensxce.org/sparc/5.11/sunw_firefox-18.0%2cREV%3d110.0.4.2013.01.10.17.25-SunOS5.11-sparc-SUNW.pkg.gz

1.) Build FF or take the Oracle supplied redistributable bins from China
and create the package by just hitting make install in my prepared
SUNWfirefox subdir, this creates the SUNWfirefox package

2.) With a plain flat miniscript from another dir  convert it into the
pkgtool.net format

3.) Copy over or sftp upload the new file to
http://svr4.opensxce.org/sparc/5.11/

4.) ssh into http://svr4.opensxce.org/sparc/5.11/ and run bldcat
(if the server hosting http://svr4.opensxce.org/sparc/5.11/ is not a
Solaris machine, just do the previous steps but  take your local mirror of
http://svr4.opensxce.org/sparc/5.11/ and run bldcat there, then simply
upload the two catalog files to the real repo site.

To complete all the necessary steps as mentioned above takes a single
person just 30 minutes or less.
No mogrifying, and no problems.

>
>
>  Individual gates provide some semi-automated ways of building things. I
>> don't know anyone who managed to automate them all though. No one was
>> able to provide equivalent of "make world" to build complete OI to date.
>>
>
> There are occasional requests on the list from people who are willing
> to give it a try, and look at the same bits of knowledge from their
> perspective. If those are shared, including the info on what works
> and what doesn't and hypotheses on why it doesn't, it is possible
> that just by sheer increase of the number of eyes, hands and brains
> aimed at parts of the problem, it would budge and give in :)
>
> After all, most "manual" administrative tasks can be scripted with
> mega-scripts of logic based on practical observations (do this, check
> that, don't do this if...). If someone formulates the problems in
> English, others can translate them to bash or perl or make scripts...
>
>
>  There is no doubt Martin Bochnig has a lot of expertise in putting
>> operating systems together. I got impression that he was heavily
>> invested in SVR4 based systems though.
>
>

True.

> OI may not be an interesting
>> project for him as it will probably remain IPS based for the time being.
>>
>

Well, as said last winter, it would be interesting.
And I have very good feelings about OI.
The problem is simply, that I have all hands more than full with restarting
OpenSXCE.
And also I think that SVR4 in connection with Blastwave style pkgutil can
serve the users' wishes to have a repo in the cloud, while at the same time
being much faster and also more transparent during install time, and
certainly a 100 or 1000 times faster during development and repo
maintenance.

>
> I did lose track of what he implemented in the end,

I gave a short summary on my twitter account last night.

> but I think his
> builds did rely on existing IPS manifests, and he had fixed quite a
> few, and the SVR4 packages were built using automated backports of
> the IPS metadata. IF this assessment is correct,

Not correct.
Here is why:

http://openindiana.org/pipermail/openindiana-discuss/2012-December/010921.html

> then it is more a
> matter of taste - which packaging system the resulting OI distro
> would use, either format would come from same sources.
>
> //Jim Klimov
>
>

Jim, well: In the earlier JDS sources exactly this was indeed possible.
You build them like the newer ones, the *.spec files with pkgbuild.
And on the pkgbuild cmd line you specify if you want to generate IPS or
SVR4 packages.
So there already is one stable approach in existence and we with our few
men should not reinvent it. So if OI really wants to stick with IPS, we
could technically still build OI and OpenSXCE from the very same sources.
As long as we merge all other consolidations into JDS's pkgbuild framework.

Even if not: It is still possible for the two to co-exist in a much easier
way. Take OS/NEt aka Illumos: For IPS you have a subdir called pkg, for
SVR4 you (or at this time I) have the subdir pkgdefs.

poka,
   %martin

-
regards

%martin bochnig
  http://svr4.opensxce.org/sparc/5.11/
    http://opensxce.org/

http://wiki.openindiana.org/oi/MartUX_OpenIndiana+oi_151a+SPARC+LiveDVD
        http://www.youtube.com/user/MartUXopensolaris
            https://twitter.com/MartinBochnig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openindiana.org/pipermail/oi-dev/attachments/20130509/a679aa03/attachment-0005.html>