[OpenIndiana-discuss] Known Problem with IPS

[Till Wegmueller]

Hello all

I don't want to contribute to the heated thread but, I was looking into 
the performance Issues in IPS for quite some time and with help of 
others. Here is a list of known factoids in no particular order.

- The OpenIndiana Repository metadata is between 4MB and 200MB in size. 
In all my experiments with either Rust/Go/C++ simply reading those files 
requires deep understanding of the languages streaming and memory saving 
techniques. In Rust this is Zerocopy in C++ this is streaming parser, in 
Go this is Just in time parsing. All of the Techniques basically try to 
eliminate memory allocation. That is just for reading.

- Every attempt to merge these publisher created JSON files into the 
Overall JSON file will result in massive execution times. This is also 
true with the OmniOS IPS.

- Even though all three IPS have diverged a bit the OpenIndiana 
performance problem is mainly on the JSON metadata front. Neither OmniOS 
nor Solaris have to solve this particular problem. This issue with the 
size and complexity of the JSON files for the usecase of a rolling 
release Distrubution.

- Thus far the only way I could come up with based on my knowledge is to 
introduce code into the merging step that skips all "userland 
consolidation" packages (or maybe a configurable set of packages). 
However that would need to be done on the client and every once in while 
the old consolidations also need to be deleted from the main JSON again 
if they are not needed.

- Zypper dnf and other circumvent this problem by using Random access 
Databases as backing store for metadata. SQLite in most cases but any 
embedded key-value or NoSQL store will do. LLMDB, RocksDB or SQLite come 
to mind here. If we replace the main merged JSON with that i would 
expect drastic impreovments for search and other features that alsways 
need to skip the 3k entries for userland consolidation.

- SAT solver performence is the same over all three platforms. One can 
see this by developing for ARM which uses the Host IPS to generate an 
ARM ZFS pool. This is equally fast when using an OmniOS host as when 
using OpenIndiana.

- The amount of userland consolidation packages is based on the Build 
process of OpenIndiana and unfortunately not changeable as it turned out 
in the last months.

Hope this sheds some clarity on where the main problem lies. As 
mentioned I have some ideas how to address this but no initiative was 
successful thus far. But the problem is understood. It is this JSON 
based metadata and merging process. (Or any file that does not easily 
allow random access)

-Till