eclean-kernel has met the following issue: MissingDecompressorError('Kernel file /boot/initramfs-5.13.12-gentoo-x86_64.img is compressed with lzo, but the required decompressor is not installed')

[iq180]

However, emerge --info emerge app-arch/lzop shows:

...
=================================================================
                        Package Settings
=================================================================

app-arch/lzop-1.04::gentoo was built with the following:
USE="" ABI_X86="(64)"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs ccache clean-logs config-protect-if-modified distlocks ebuild-locks fail-clean fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"

[mgorny]

There's USE=lzo that pulls in the right deps.

[iq180]

What do you mean?

My kernel and initramfs are compiled using LZO compression.

I already have an installed package "app-arch/lzop".

I've added "app-admin/eclean-kernel lzo" to my "/etc/portage/package.use", re-emerged "app-admin/eclean-kernel", and now when I run "eclean-kernel -n 1", I get a message: "error('Header error - invalid compressed data')".

[iq180]

# modprobe configs && zgrep LZO /proc/config.gz
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_KERNEL_LZO=y
CONFIG_RD_LZO=y
# CONFIG_CRYPTO_LZO is not set
CONFIG_LZO_DECOMPRESS=y
CONFIG_DECOMPRESS_LZO=y

# emerge --info eclean-kernel
...
app-admin/eclean-kernel-2.99.3::gentoo was built with the following:
USE="lzo -lz4 -test -zstd" ABI_X86="(64)" PYTHON_TARGETS="python3_9 -python3_8"
FEATURES="userfetch protect-owned xattr assume-digests usersync strict config-protect-if-modified parallel-fetch ebuild-locks network-sandbox unknown-features-warn qa-unresolved-soname-deps sandbox unmerge-orphans binpkg-multi-instance sfperms userpriv pid-sandbox ipc-sandbox binpkg-logs ccache clean-logs unmerge-logs fixlafiles news binpkg-dostrip multilib-strict fail-clean binpkg-docompress merge-sync usersandbox distlocks preserve-libs"

# eclean-kernel -n 1
eclean-kernel has met the following issue:

  error('Header error - invalid compressed data')

If you believe that the mentioned issue is a bug, please report it
to https://github.com/mgorny/eclean-kernel/issues. If possible,
please attach the output of 'eclean-kernel --list-kernels' and your
regular eclean-kernel call with additional '--debug' argument.

[mgorny]

emerge -pv python-lzo

please.

[mgorny]

CC @Jannik2099

[iq180]

emerge -pv python-lzo

please.

It was emerged as a dependency of eclean-kernel.

-rw-rw---- 1 portage portage 4,1K 2021-08-20 01:38 /var/log/portage/dev-python:python-lzo-1.12-r2:20210819-223809.log
-rw-rw---- 1 portage portage 8,0K 2021-08-20 01:38 /var/log/portage/app-admin:eclean-kernel-2.99.3:20210819-223829.log
-rw-rw---- 1 portage portage  20K 2021-08-20 01:38 /var/log/portage/app-admin:eclean-kernel-2.99.3:20210819-223820.log

[Jannik2099]

well this is interesting

can you open a python shell (just run python) and then run import lzo ?
Does python-lzo have the same PYTHON_TARGETS as eclean-kernel?

@mgorny I wasn't aware we also do the kernel version parsing on initramfs files, is this intended?

[iq180]

can you open a python shell (just run python) and then run import lzo ?

# python
Python 3.9.6 (default, Aug  5 2021, 20:29:37) 
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lzo
>>> 

Does python-lzo have the same PYTHON_TARGETS as eclean-kernel?

# emerge --info|grep PYTHON
USE="...PYTHON_SINGLE_TARGET="python3_9" PYTHON_TARGETS="python3_9"..."

# emerge --info python-lzo|grep PYTHON
...
USE="-test" ABI_X86="(64)" PYTHON_TARGETS="python3_9 -python3_10 -python3_8"

# emerge --info eclean-kernel|grep PYTHON
...
USE="lzo -lz4 -test -zstd" ABI_X86="(64)" PYTHON_TARGETS="python3_9 -python3_8"

[mgorny]

@Jannik2099 see #22 (comment). Maybe the kernel uses a different stream format, similarly to lz4?

@iq180 would it be possible for you to attach the kernel image here? I don't think there are any secrets inside it ;-).

[Jannik2099]

ooooh, I didn't see that the error changed - whoops
Yeah, it's probably omitting some metadata then... why you do this kernel :(

[Jannik2099]

https://www.kernel.org/doc/Documentation/lzo.txt oh great, what a fustercluck

[mgorny]

If it's too hard to support, we can always declare it unsupported. Would be nice to get an lzo-compressed kernel anyway and see if lzop can deal with it.

[Jannik2099]

yeah, if this requires any custom logic I'd prefer to just drop it - lzo is obsoleted by zstd anyways in just about any metric (including being a sane format)
I'll try reproduce this the next few days

[iq180]

yeah, if this requires any custom logic I'd prefer to just drop it - lzo is obsoleted by zstd anyways in just about any metric (including being a sane format)
I'll try reproduce this the next few days

Hm. LZO is fast, it's approved to use in kernel/initramfs compression. But "it's too hard to support". Very strange.

Apparently, eclean is for those who use default kernel compilation settings only.

Very strange.

[iq180]

If it's too hard to support, we can always declare it unsupported

Great decision.

[Jannik2099]

it's approved to use in kernel/initramfs compression. But "it's too hard to support".
The kernel uses a custom encoding, not a widely standardized one

LZO has roughly the same compression ratio as lz4 at almost 5x lower decompression speed, and a roughly 30% lower compression ratio than zstd at less than half the decompression speed. It's outclassed by lz4 and zstd at both ends
https://facebook.github.io/zstd/

That being said if it turns out python-lzo supports the kernels compression encoding there's no reason not to support it, but I don't feel like writing a custom frame decompressor for an effectively deprecated algorithm

[mgorny]

@iq180, please stop being such an asshole. We're working on this in our free time, we don't get any money for it and apparently no gratitude either. We can't do much about the fact that LZO is poorly standardized and the kernel implementation apparently disagrees with the standard stream format.

If you want things to happen, either supply patches or talk to @Jannik2099 about the possibility of funding this work. Or at least stop being such a douchebag because that kind of attitude is literally killing volunteer work.

[mgorny]

@Jannik2099 BTW you may want to see if squashfs-tools has something useful to steal. If you really don't want to do it, then could you remove LZO support altogether and replace it with a clear error message?

[iq180]

@iq180, please stop being such an asshole. We're working on this in our free time, we don't get any money for it and apparently no gratitude either. We can't do much about the fact that LZO is poorly standardized and the kernel implementation apparently disagrees with the standard stream format.

If you want things to happen, either supply patches or talk to @Jannik2099 about the possibility of funding this work. Or at least stop being such a douchebag because that kind of attitude is literally killing volunteer work.

Asshole, as I consider, is the maintainer who fails to maintain.

[eternal-sorrow]

Why does it need to decompress initramfs images though? Can I somehow tell it to ignore them?