[OpenIndiana-discuss] Known Problem with IPS

[Bob Friesenhahn]

On Sun, 14 Jan 2024, Till Wegmueller wrote:
>
> - The OpenIndiana Repository metadata is between 4MB and 200MB in size. In 
> all my experiments with either Rust/Go/C++ simply reading those files 
> requires deep understanding of the languages streaming and memory saving 
> techniques. In Rust this is Zerocopy in C++ this is streaming parser, in Go 
> this is Just in time parsing. All of the Techniques basically try to 
> eliminate memory allocation. That is just for reading.

The fastest JSON parser I am aware of was developed by a previous 
work-mate. It is at 
"https://github.com/gdavidbutler/jsonTrivialCallbackParser".  There is 
a trade-off in that it places more responsibilities on the using code 
and thus any performance issues will be due to the using code.  If 
this was attached to Python (using a C-language glue layer), then 
Python would be the bottleneck. Perhaps the license (LGPL-3) is not 
good for OpenIndiana.

Using C code to read JSON and transform it to the database used for 
the rest of the work seems more feasible.

> - Zypper dnf and other circumvent this problem by using Random access 
> Databases as backing store for metadata. SQLite in most cases but any 
> embedded key-value or NoSQL store will do. LLMDB, RocksDB or SQLite come to 
> mind here. If we replace the main merged JSON with that i would expect 
> drastic impreovments for search and other features that alsways need to skip 
> the 3k entries for userland consolidation.

I definitely recommend SQLite for any intense usage involving a 
read-mostly situation (e.g. prepare/fill the database and then use 
it).  It is quite fast and SQLite has continued to advance by leaps 
and bounds for many years.

Bob
-- 
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
Public Key,     http://www.simplesystems.org/users/bfriesen/public-key.txt