Ubuntu 22.04 VM broke

[evelikov]

Recently the CI broke in a somewhat odd manner:

• latest working job shows:

Using kernel 6.2.0-1019-azure/x86_64
Checking module compression ...
config: CONFIG_MODULE_COMPRESS_NONE=y
files: /lib/modules/6.2.0-1019-azure/kernel/crypto/af_alg.ko
Expected extension: (none)

• comparing to the first broken one:

Using kernel 6.5.0-1015-azure/x86_64
Checking module compression ...
config: CONFIG_MODULE_COMPRESS_ZSTD=y
files: /lib/modules/6.5.0-1015-azure/kernel/crypto/af_alg.ko
Expected extension: .zst

Comparing these with the container runs (22.04 and 23.10 respectively) which remain working fine.

Ubuntu 22.04

Found kernel 5.15.0-97-generic
Module search paths
/etc/depmod.d/ubuntu.conf:search updates ubuntu built-in
Using kernel 5.15.0-97-generic/x86_64
Checking module compression ...
config: CONFIG_MODULE_COMPRESS_NONE=y
files: /lib/modules/5.15.0-97-generic/kernel/fs/nfs/nfsv3.ko
Expected extension: (none)

Ubuntu 23.10

Found kernel 6.5.0-21-generic
Module search paths
/etc/depmod.d/ubuntu.conf:search updates ubuntu built-in
Using kernel 6.5.0-21-generic/x86_64
Checking module compression ...
config: CONFIG_MODULE_COMPRESS_ZSTD=y
files: /lib/modules/6.5.0-21-generic/kernel/fs/nfs/nfsv3.ko.zst
Expected extension: .zst

It seems like the azure flavour of the 6.5.0 kernel is using zstd kconfig yet the modules are actually uncompressed.
@xnox the current state seems broken. Any chance someone in the Canonical/Ubuntu team can fix this?

Thanks in advance

[evelikov]

@xnox is this of interest to the Canonical/Ubuntu team? I don't mind having a look myself, if anyone can point me to a the git repo/branch used to produce the jammy/azure variant.

Alternatively we can drop Ubuntu from the CI, although I'd rather have that as last resort.

[xuzhen]

The Makefile.modinst file was modified in Ubuntu.

--- linux-azure-6.5.0.orig/scripts/Makefile.modinst
+++ linux-azure-6.5.0/scripts/Makefile.modinst
@@ -24,7 +24,9 @@
 suffix-y				:=
 suffix-$(CONFIG_MODULE_COMPRESS_GZIP)	:= .gz
 suffix-$(CONFIG_MODULE_COMPRESS_XZ)	:= .xz
-suffix-$(CONFIG_MODULE_COMPRESS_ZSTD)	:= .zst
+## UBUNTU: Support, but do not require zstd compressed modules
+# Many external dkms and signing rely on uncompressed modules
+# suffix-$(CONFIG_MODULE_COMPRESS_ZSTD)	:= .zst
 
 modules := $(patsubst $(extmod_prefix)%.o, $(dst)/%.ko$(suffix-y), $(modules))

FYI:

1. https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/commit/scripts/Makefile.modinst?h=Ubuntu-hwe-6.5-6.5.0-14.14_22.04.1&id=3daebecd0259b9d0acf6267e9df261afa074bba8
2. https://bugs.launchpad.net/bugs/2045593

[xuzhen]

It looks like only Ubuntu 22.04 (Jammy) with kernel version 6.5 or later will have this issue.

[xuzhen]

After further investigation, I found that this problem was actually caused by the inconsistent judgment of module compression methods between dkms and run_test.sh. dkms also checks modules.dep

dkms/dkms.in

Lines 234 to 245 in 4d466bf

	# Figure out the correct module suffix for the kernel we are currently
	# dealing with, which may or may not be the currently installed kernel.
	set_module_suffix()
	{
	# $1 = the kernel to base the module_suffix on
	kernel_test="${1:-$(uname -r)}"
	module_uncompressed_suffix=".ko"
	grep -q '\.gz:' $install_tree/$kernel_test/modules.dep 2>/dev/null && module_compressed_suffix=".gz"
	grep -q '\.xz:' $install_tree/$kernel_test/modules.dep 2>/dev/null && module_compressed_suffix=".xz"
	grep -q '\.zst:' $install_tree/$kernel_test/modules.dep 2>/dev/null && module_compressed_suffix=".zst"
	module_suffix="$module_uncompressed_suffix$module_compressed_suffix"
	}

I created PR #410 for this issue.

[evelikov]

Thanks for the links @xuzhen

Looking through the patch and the issue at hand it doesn't seem clear to me, why devs opted for patching the kernel instead of changing the CONFIG toggle. It basically leaves the option enabled, only to patch out the functionality 🤔

@xnox @arighi any input?

[evelikov]

Inclined to merge #411 as a temporary workaround.