[OpenIndiana-discuss] Is it safe to remove /var/pkg/cache directory?
Predrag Zecevic
predrag.zecevic.1961 at googlemail.com
Mon Dec 18 13:07:24 UTC 2023
On 2023-12-15 11:40:21, Predrag Zečević wrote:
>
>
>> Strange is this:
>>
>> :; pkg list | grep golang
>> developer/golang 1.21.1-2023.0.0.0 i--
>> developer/golang-121 1.21.5-2023.0.0.0 i--
>>
>> But:
>> :; ls -dlh /var/pkg/publisher/openindiana.org/pkg/developer%2Fgolang*
>> drwxr-xr-x 2 root root 9 Dec 13 19:36
>> /var/pkg/publisher/openindiana.org/pkg/developer%2Fgolang
>> drwxr-xr-x 2 root root 4 Aug 23 2021
>> /var/pkg/publisher/openindiana.org/pkg/developer%2Fgolang-115
>> drwxr-xr-x 2 root root 4 Aug 25 2021
>> /var/pkg/publisher/openindiana.org/pkg/developer%2Fgolang-116
>> drwxr-xr-x 2 root root 3 Jun 19 15:21
>> /var/pkg/publisher/openindiana.org/pkg/developer%2Fgolang-119
>> drwxr-xr-x 2 root root 4 Oct 4 11:24
>> /var/pkg/publisher/openindiana.org/pkg/developer%2Fgolang-120
>> drwxr-xr-x 2 root root 5 Dec 14 12:47
>> /var/pkg/publisher/openindiana.org/pkg/developer%2Fgolang-121
>>
>> So, no directory was removed when package is removed! That explains
>> 15 GB in /var/pkg/publisher/ directory.
>>
>> If no one else have this problem, then it is not bug, but something else
>> Regards
>
> Yes, this is me (somehow):
>
> I was executing "pfexec pkg update -vn" every morning to get latest
> packages available (output was send to file, parsed for interesting
> packages, and sent to me per e-mail).
>
> Seems that command downloads packages?
> To prove, I did test:
>
> :; cd /var/pkg/publisher/openindiana.org/pkg
> :; find . -type f | sort > /tmp/pkg-1.txt
> :; pfexec pkg update -vn # executed at 2023-12-15 10:32:47 CET
> :; find . -type f | sort > /tmp/pkg-2.txt
> :; diff /tmp/pkg-2.txt /tmp/pkg-1.txt | wc -l
> 770
>
> So, I got 770 new entries under /var/pkg/publisher/openindiana.org/pkg
> !!!
>
> Example:
> :; ls -hld ./system%2Fsecurity%2Fkerberos-5/* | wc -l
> 891
>
> :; ls -hld ./system%2Fsecurity%2Fkerberos-5/* | tail -5
> -rw-r--r-- 1 root root 46K Dec 11 05:00
> ./system%2Fsecurity%2Fkerberos-5/0.5.11%2C5.11-2023.0.0.21923%3A20231211T012803Z
> -rw-r--r-- 1 root root 46K Dec 12 05:00
> ./system%2Fsecurity%2Fkerberos-5/0.5.11%2C5.11-2023.0.0.21925%3A20231212T013240Z
> -rw-r--r-- 1 root root 46K Dec 13 19:35
> ./system%2Fsecurity%2Fkerberos-5/0.5.11%2C5.11-2023.0.0.21925%3A20231213T011458Z
> -rw-r--r-- 1 root root 46K Dec 14 05:00
> ./system%2Fsecurity%2Fkerberos-5/0.5.11%2C5.11-2023.0.0.21926%3A20231214T011328Z
> -rw-r--r-- 1 root root 46K Dec 15 10:33
> ./system%2Fsecurity%2Fkerberos-5/0.5.11%2C5.11-2023.0.0.21928%3A20231215T071813Z
>
> :; du -sh ./system%2Fsecurity%2Fkerberos-5/
> 41M ./system%2Fsecurity%2Fkerberos-5/
>
> And so on...
>
> Am I right when assuming that 'pkg update -vn' should NOT _download_
> packages?
>
> From 'man pkg':
> ---<snip>-------
> -n
>
> Perform a trial run of the operation with no package
> changes made.
> ---<snip>-------
>
> If answer is yes, then I will open ticket...
>
> Also I assume that I am allowed to remove any package manifest file
> under /var/pkg/publisher/openindiana.org/pkg what is not installed?
>
> Regards.
>
> P.S: IMHO, seems *pkg* keeps installed package "registry" in this
> structure, under /var/pkg/publisher/openindiana.org:
>
> :; ls -hl catalog/
> total 2,0K
> lrwxrwxrwx 1 root root 65 Dec 15 10:33 catalog.attrs ->
> ../origins/12dad71d06df371e72f75c488f47c75a13e340d1/catalog.attrs
> lrwxrwxrwx 1 root root 66 Dec 15 10:33 catalog.base.C ->
> ../origins/12dad71d06df371e72f75c488f47c75a13e340d1/catalog.base.C
> lrwxrwxrwx 1 root root 72 Dec 15 10:33 catalog.dependency.C ->
> ../origins/12dad71d06df371e72f75c488f47c75a13e340d1/catalog.dependency.C
> lrwxrwxrwx 1 root root 69 Dec 15 10:33 catalog.summary.C ->
> ../origins/12dad71d06df371e72f75c488f47c75a13e340d1/catalog.summary.C
>
> :; ls -hl origins/12dad71d06df371e72f75c488f47c75a13e340d1/
> total 346M
> -rw-r--r-- 1 root root 40K Dec 15 09:16 catalog.attrs
> -rw-r--r-- 1 root root 7,5M Dec 15 08:29 catalog.base.C
> -rw-r--r-- 1 root root 310M Dec 15 08:29 catalog.dependency.C
> -rw-r--r-- 1 root root 29M Dec 15 08:29 catalog.summary.C
>
So, let me update here,
I have found hundred of thousands of "orphaned" manifest files (see
message above for examples, e.g. kerberos-5 package example).
Those files have occupied 15 GB of disk space (also in each BE created
during pkg update).
Space was eaten by 'pkg update -nv' command (package not installed but
manifest is downloaded and kept) which I was running last couple years,
once/day.
This is script (if someone else needs it), which might run for long time
(like it was on my installation: more than 48 hours) and at the end it
compiles list of commands to remove orphaned manifest files:
---8<---------
#!/bin/bash
# Clean-up /var/pkg/publisher/openindiana.org/pkg directory
TODAY=$(date '+%Y%m%dT%H%M%S')
PKGMFF="/var/tmp/PKGs.${TODAY}-manifests.txt"
PKGINP="/var/tmp/PKGs.${TODAY}-installed.txt"
PKGMRM="/var/tmp/PKGs.${TODAY}-manifests-to-remove.sh"
echo "Collecting manifest data from
/var/pkg/publisher/openindiana.org/pkg/ ..."
(
cd /var/pkg/publisher/openindiana.org/pkg/
find . -type f | cut -d "/" -f 2- | sort > ${PKGMFF}
)
echo "Collecting installed packages ..."
(
pkg info | grep FMRI: | cut -d "/" -f 4- | sort > ${PKGINP}
)
# Each row:
audio%2Faudio-utilities/0.5.11%2C5.11-2020.0.0.19486%3A20200221T011730Z
# < pkg_name > < datets >
# so, loop through manifest list, get pkg_name (replace %2F with /), get
datets value
# and try to find that pkg FMRI contains datets). In case of no
result, that manifest file may be removed
(
cat ${PKGMFF} | while read MANIFEST; do
PKG_URLN=${MANIFEST%/*} # package name, but needs fix (slashes)
PKG_NAME=${PKG_URLN//%2F/\/} # replace string with slash character
PKG_DATE=${MANIFEST: -16} # TS is last 16 characters (always)
# If pkg for this manifest file is not found in system, remove it:
PKG_INST=$(pkg info pkg://openindiana.org/${PKG_NAME} |& grep
${PKG_DATE})
echo "# Result of 'pkg info pkg://openindiana.org/${PKG_NAME} |&
grep ${PKG_DATE}': '${PKG_INST}'"
if [[ -z ${PKG_INST} ]]; then
echo "pfexec rm -rf
/var/pkg/publisher/openindiana.org/pkg/${MANIFEST}"
fi
done
echo
echo "pfexec pkg refresh --full"
) |& tee ${PKGMRM}
echo "Create backup BE and execute: 'bash ${PKGMRM}'"
---8<---------
Before running commands from script, backup BE HAS to be created, in
case you want roll-back.
Luckily for me, script (and set of commands did a job properly: I was
able to update OI and reboot it this morning w/o issues).
Right now:
:; du -sh /var/pkg/publisher/openindiana.org/*
3,5K /var/pkg/publisher/openindiana.org/catalog
1,5K /var/pkg/publisher/openindiana.org/certs
1,0K /var/pkg/publisher/openindiana.org/last_refreshed
366M /var/pkg/publisher/openindiana.org/origins
280M /var/pkg/publisher/openindiana.org/pkg
Which is acceptable (I have removed backup BE, but still have one
snapshot of old BE which will go away on next update)
Hope these finding can help someone.
Next week, at some quiet time, I will check my theory, and if I am right
will open ticket...
With best regards.
>>
>>
>>>
>>>> _______________________________________________
>>>> openindiana-discuss mailing list
>>>> openindiana-discuss at openindiana.org
>>>> https://openindiana.org/mailman/listinfo/openindiana-discuss
>>>
>>
>
--
Predrag Zečević
predrag.zecevic.1961 at googlemail.com
More information about the openindiana-discuss
mailing list