[OpenIndiana-discuss] Questions about /var/pkg

Sat Jan 13 16:41:11 UTC 2024

as much as i value playful adventures, system stuff should always be c/c++. the last thing one would 
want is to add even more stuff that can't be uninstalled because of maybe only one single dependency.

the first and foremost problem of ips is speed. it's so slow that i had to write my own replacements for 
certain features like "list" which are used often. or for example the time "pkg update" needs before it 
even starts actually installing things, can easily be minutes on older systems. while on the same 
machine yum or zypper get that sorted within 5 or 10 seconds.
the second problem is the size of the repo. as i posted here some months ago, /var/pkg is significantly 
larger compared to most other package management systems.

On Sat, 13 Jan 2024 13:50:30 +0100, Till Wegmueller wrote:
> Glady. I made some experiments that went relatively far one being a repo
> format implementation in Golang[0] and one being some of that work being
> ported to rust [1].
> 
> -Till
> 
> [0] https://github.com/Toasterson/pkg6-go
> [1] https://github.com/openflowlabs/ips
> 
> On 13.01.2024 13:02, Matthew R. Trower wrote:
>> Ah okay, thanks for the clarification. There are a number of items I’d dearly like to improve about 
>> IPS, but none of them are something to tackle on a whim. I guess I’ll add metadata caching behavior 
>> to the list. Maybe once I get some more immediate matters off my plate, we can revisit this.
>> 
>> 
>> -- Matthew R. Trower
>> 
>>> On Jan 13, 2024, at 05:42, Till Wegmueller <toasterson at gmail.com> wrote:
>>> 
>>> Hi Mathew
>>> 
>>> It is a special statement for /var/pkg/publisher you will get stale data due to how the Metadata 
>>> download process works. This directory MUST always go forward in time from it's perspective. Meaning 
>>> it MUST be inside the Boot Environment. At least as long as nobody rewrites that :)
>>> 
>>> I can give you some details on the process if needed. But the gist is that there are pre split JSON 
>>> files pkg downloads and merged into the existing metadata under the publisher. Then it merged that 
>>> into the global cache. Nobody has ever tested if IPS has a detection mechanism if there are stale 
>>> files in this directory. This is fully undocumented territory as it was done in the last days of 
>>> OpenSolaris. OmniOS IPS might have such detections now but somebeody(Maybe you?) would need to merge 
>>> in that version of IPS. Solaris IPS is also open and wen could cherry-pick from there if we want 
>>> [0]. I do not know which IPS Helios is using but i would assume OmniOS's one. In any case you can 
>>> make a full cache refresh with `pkg refresh --full` and sometimes that is needed to recover from 
>>> situations where the cache is broken.
>>> 
>>> As side note there was a project to merge the IPS forks but it hit my ENOTIME. If you want to pick 
>>> that up and make improvements on the metadata process I am happy to help.
>>> 
>>> -Till
>>> 
>>> [0] https://github.com/oracle/solaris-ips
>>> 
>>>> On 13.01.2024 01:11, Matthew R. Trower wrote:
>>>> Hi Till,
>>>> To clarify, I only wish to share /var/pkg/publisher. /var/pkg clearly contains image-specific data, 
>>>> and bad bad things would happen if that were shared.
>>>> Does your statement still apply specifically to /var/pkg/publisher?
>>>> -- Matthew R. Trower
>>>>> On 1/12/24 12:25, Till Wegmueller wrote:
>>>>> Hi,
>>>>> 
>>>>> From personal experience, Yes this data is kind of a cache. It is later merged into the publisher 
>>>>> independant JSON. However you MUST NOT share this between BE's this directory is not safe to go 
>>>>> back in time due to how partial download and merging works. Do NOT share /var/pkg between BE's
>>>>> 
>>>>> -Till
>>>>> 
>>>>> On 12.01.2024 06:16, Matthew R. Trower wrote:
>>>>>> Hi,
>>>>>> 
>>>>>> Is the format of /var/pkg/publisher committed at this point? It looks to contain cached data and 
>>>>>> downloads from publishers, which ought not to be tied to any particular BE. If I create a ZFS 
>>>>>> dataset to share that among BEs, am I going to run into trouble?
>>>>>> 
>>>>>> What exactly are the contents of /var/pkg/lost+found? Can any files I discover in there be safely 
>>>>>> deleted?
>>>>>> 
>>>>>> 
>>>>>> -- Matthew R. Trower
>>>>>> 
>>>>>> _______________________________________________
>>>>>> openindiana-discuss mailing list
>>>>>> openindiana-discuss at openindiana.org
>>>>>> https://openindiana.org/mailman/listinfo/openindiana-discuss
>>>>> 
>>>>> _______________________________________________
>>>>> openindiana-discuss mailing list
>>>>> openindiana-discuss at openindiana.org
>>>>> https://openindiana.org/mailman/listinfo/openindiana-discuss
>>>> _______________________________________________
>>>> openindiana-discuss mailing list
>>>> openindiana-discuss at openindiana.org
>>>> https://openindiana.org/mailman/listinfo/openindiana-discuss
>>> 
>>> _______________________________________________
>>> openindiana-discuss mailing list
>>> openindiana-discuss at openindiana.org
>>> https://openindiana.org/mailman/listinfo/openindiana-discuss
>> 
>> _______________________________________________
>> openindiana-discuss mailing list
>> openindiana-discuss at openindiana.org
>> https://openindiana.org/mailman/listinfo/openindiana-discuss
> 
> _______________________________________________
> openindiana-discuss mailing list
> openindiana-discuss at openindiana.org
> https://openindiana.org/mailman/listinfo/openindiana-discuss