urpmi --no-verify is broken by rpm checking on its own with rpm-sequoia

[soig]

We are hitting issues on Mageia with 4.20.x.
The first issue is that signatures being refused

This one seems to be on Mageia :

error: Verifying a signature using certificate 00EDB89585B012A8916F0DF8B742FA8B80420F66 (Mageia Packages <[email protected]>):

1. Certificiate B742FA8B80420F66 invalid: certificate is not alive
   because: The primary key is not live
   because: Expired on 2012-03-13T12:10:11Z
2. Key B742FA8B80420F66 invalid: key is not alive
   because: The primary key is not live
   because: Expired on 2012-03-13T12:10:11Z

However this break "urpmi --no-verify" which used to behave like rpm --noverify
But now only rpm --nosignature can bypass this check, whereas rpm --no-verify used to do the same, which looks like regression ?

Previously, we used to check pkgs in urpmi but it looks like rpm does it on its own now too:
http://gitweb.mageia.org/software/rpm/urpmi/tree/urpm/main_loop.pm#n527
Which calls:
http://gitweb.mageia.org/software/rpm/urpmi/tree/urpm/main_loop.pm#n129
Which calls:
http://gitweb.mageia.org/software/rpm/urpmi/tree/urpm/signature.pm#n40
Which calls:
http://gitweb.mageia.org/software/rpm/perl-URPM/tree/URPM.xs#n3306
Which calls rpmReadPackageFile()

See attached logs showing urpmi output with both rpm-4.19 & 4.20
For me, there's 2 issues :

• rpmlib < 4.20 not doing some checks
• rpmlib redoing the checks performed by urpmi, and thus breaking urpmi when the user already says "yes, let's continue" after urpmi warned him/her
• same problem when using --no-verify which is the same as the user answering "y" above

Do we've to set RPMPROB_FILTER_VERIFY when calling rpmtsRun() ?
Would it impact urpmi running on older rpm versions (aka no more checks there) ?
Or should we kill that check (but then no more informative error in GUI?

Note that we all those upstream patches have been backported:
0001-Ensure-noarch-packages-don-t-get-debuginfo.patch
0001-Fix-noprep-regression-from-introducing-mkbuilddir.patch
0002-Drop-an-accidentally-added-duplicated-test.patch
0003-Make-build-in-place-much-less-of-a-hack-and-also-wor.patch
0001-Fix-incomplete-header-on-plain-src.rpm-build-modes-r.patch
0001-Hammer-in-no-debuginfo-for-noarch-packages-damn-it-r.patch
0001-Fix-regression-on-subpackage-debuginfo-RPMTAG_SOURCE.patch
0001-Fix-a-buildroot-regression-on-an-early-__spec_instal.patch

See you

LOG.urpmi-rpm-4.19.txt
LOG.urpmi-rpm-4.20.txt

[pmatilai]

The key management and all hasn't changed in rpm itself, this is rpm-sequoia being far, far stricter about what is accepted than the rpm internal pgp parser ever was. To go with rpm-sequoia, you'll need to keep your keys in shape, like not have keys expired 12 years ago.

The alternative is using the "traditional" PGP parser, it's still available only now split into separate repo (see INSTALL)

[soig]

Well the key is updated on media (=repos in urpmi).
So regular people who do upgrades with the installer will get the new key when removing old media, adding new media.
But I'm doing online updates for a decade using Cauldron (~ Rawhide for Fedora):

diff -u KEY-*
--- KEY-AVAIL2 2024-06-04 17:07:29.192787460 +0200
+++ KEY-INST2 2024-06-04 17:07:37.793998092 +0200
@@ -1,4 +1,4 @@
-pub rsa4096/0xB742FA8B80420F66 2011-02-07 [SCEA] [expires: 2025-12-31]
+pub rsa4096/0xB742FA8B80420F66 2011-02-07 [SCEA] [expired: 2012-03-13]
00EDB89585B012A8916F0DF8B742FA8B80420F66
uid Mageia Packages <[email protected]>
-sig 0xB742FA8B80420F66 2019-06-21 [selfsig]
+sig 0xB742FA8B80420F66 2011-02-07 [selfsig]

I guess I'm a special case 😅
But I may have to change urpmi.update so that it refresh the pubkey when it changes 😅

[soig]

Looks like even when adding new media, we don't re-import the key:
"pubkey 80420f66 already imported"
So I'll need to fix that.

Still @pmatilai, since urpmi checks pkgs first, should it set RPMPROB_FILTER_VERIFY when calling rpmtsRun() for real since at that point the user said yes to force ?

[pmatilai]

Yeah rpm doesn't have a meaningful way to update keys because back when I added the "is primary key id present", we didn't know anything about expiry, revokes, subkeys .. nothing.
#3083 promises to fix that, until then the best you can do is try importing, if it already exists erase the key and try again.

I don't know what urpmi --no-verify does, exactly. The story on rpm side is complicated as there are multiple related and partially overlapping mechanisms: the verify step during transactions is not a "signature check" per se, it's more of a payload validation step. Signatures can be used in there, but aren't by default. If urpmi's --no-verify is supposed to be a direct --nosignature counterpart then see 5ed632a.

[soig]

So there's two issues, so I'll use 2 parts :

Updating/reimporting a pubkey:
Stupid question but : how can one remove a key from rpmdb (through the API, not using rpm -e) ?
we've rpmtsImportPubkey() but no remove or refresh API.
There's also rpmKeyringAddKey() but no RemoveKey().

Currently the URPM C binding uses rpmtsImportPubkey() https://gitweb.mageia.org/software/rpm/perl-URPM/tree/URPM.xs#n3363
(called from the perl interface : https://gitweb.mageia.org/software/rpm/perl-URPM/tree/URPM/Signature.pm#n82)

➡️ I guess I'll have to run a small transaction removing the offending key before importing.
I don't see anything else ?
WDYT?

Urpmi --no-verify
By default (unless one uses --no-verify), urpmi first checks packages
http://gitweb.mageia.org/software/rpm/urpmi/tree/urpm/main_loop.pm#n527
before doing the actual installation:
http://gitweb.mageia.org/software/rpm/urpmi/tree/urpm/main_loop.pm#n544

The actual installation basically means we run a series of transactions here http://gitweb.mageia.org/software/rpm/urpmi/tree/urpm/install.pm#n352
my $trans = $db->create_transaction;
$trans->check
$trans->order
$trans->run
Which are defined in URPM binding : https://gitweb.mageia.org/software/rpm/perl-URPM/tree/URPM.xs#n2770 : respectively rpmtsCheck(), rpmtsOrder() and rpmtsRun()

But before installing, as said above, unless using --no-verify, we'll do some tests : http://gitweb.mageia.org/software/rpm/urpmi/tree/urpm/main_loop.pm#n129
which really happens here: http://gitweb.mageia.org/software/rpm/urpmi/tree/urpm/signature.pm#n40
For each package we call verify_signature() which is there in the URPM binding: https://gitweb.mageia.org/software/rpm/perl-URPM/tree/URPM.xs#n3306 where we use rpmReadPackageFile() to check each package file.

So at this point we already checked the pkgs for signatures.
If in interactive mode (either CLI or GUI) we ask the user if he wants to abort and force the installation.
Previously, this worked fine, but now rpmlib checks again the pkgs after we already did it and it doesn't know that the user actually said "frak this, install anyway".

So I think that in urpmi context, we want to prevent rpmlib to check again the pkgs since we already did it.
➡️ Does passing RPMPROB_FILTER_VERIFY would be enough to achieve this when calling rpmtsRun()?

[pmatilai]

I guess I'll have to run a small transaction removing the offending key before importing.

Yes, there's nothing better at the moment.

Does passing RPMPROB_FILTER_VERIFY would be enough to achieve this when calling rpmtsRun()?

Maybe, but again that verify step is NOT about signature verification as such, it could be package whose (header) signature is perfectly valid but truncated payload, or such. For "frak the signatures", IIRC you want to set rpmtsSetVfyFlags() to same as rpmtsSetVSFlags().

Note that letting verify do its job has more subtle side-effects too: installations only show as verified in the auditing log if rpm itself verified it.

[soig]

I guess I'll have to run a small transaction removing the offending key before importing.

Yes, there's nothing better at the moment.

I've a working patch for that.
Though that's not ideal b/c when adding a whole set of media/repos (core, nonfree, tainted -- each one existing as regular, updates, updates_testing), that mean adding/removing the same key for all.
Is there a way to get the expiry date with rpm API?

BTW other packagers are throwing stones at mea, saying there's a difference between an expired key and a revoked
key : a signature that was valid yesterday doesn't suddenly become invalid today,
just because the key expired!"

Does passing RPMPROB_FILTER_VERIFY would be enough to achieve this when calling rpmtsRun()?

Maybe, but again that verify step is NOT about signature verification as such, it could be package whose (header) signature is perfectly valid but truncated payload, or such. For "frak the signatures", IIRC you want to set rpmtsSetVfyFlags() to same as rpmtsSetVSFlags().

Note that letting verify do its job has more subtle side-effects too: installations only show as verified in the auditing log if rpm itself verified it.

That would only be set up if the urpmi verify pass failed and the user said "go on anyway" or if he sets up the no-verify option in /etc/urpmi/urpmi.cfg ot used the --no-verify option, so in regular cases that won't be an issue

[pmatilai]

BTW other packagers are throwing stones at mea, saying there's a difference between an expired key and a revoked
key : a signature that was valid yesterday doesn't suddenly become invalid today,
just because the key expired!"

It gets all a bit filosophical. If it doesn't come invalid at that point, then when does it? And if it doesn't, what's the point of expiry in the first place? It shouldn't be possible to create new signatures with an expired key for one, but I guess expiry should be considered wrt the time of signing instead of "now" I suppose. I kinda thought that's how Sequoia does it, but not sure (tagging @nwalfield for possible comments). To evaluate what actually happens one needs to look at both the key and the signature, which we don't have linked here AFAICS.

[nwalfield]

Sorry for the late reply. If a certificate expires (at say t2) that doesn't mean that signatures made prior to the expiry (e.g., at t1, where t1 < t2) are suddenly invalid. If that is happening, that is a bug, and I'd appreciate a report.

[soig]

Yeah but aborting on expired key when we don't have an API to refresh the key yet is an issue.
We should first add such an API, then later abort on expired keys.
As we say in French, "c'est mettre la charrue avant les boeufs" (=~ "it's putting the cart before the horse")😉
At least checking against signing date rather than now would be better in the current state of the rpms

[pmatilai]

Oh, no disagreement there. This is a case where basically everybody got caught with their pants down: in the first 20+ years of rpm's existence nobody gave much thought to expiry and all that, but then in recent years (even before sequoia) people started running into their ancient SHA1 based keys suddenly stopping to work, first due to FIPS and then more generally distros tried to push for disabling obsoleted crypto but disabling SHA1 got postponed and postponed and postponed for things like google-chrome, and then rpm-sequoia was introduced and pushing those people more strongly to update their signing systems. And when those parties finally updated their keys we realize there's no mechanism for them to update those keys at all. 🙈