Engine deployment on oVirt Node NG 4.5.3 (Stream 9) fails due to missing gpg key

[sjansen1]

Hello,

i try to deploy a new oVirt Cluster by using a fresh node installed with "ovirt-node-ng-installer-latest-el9.iso" (4.5.3, secureboot disabled). This works fine, but trying to deploy hosted engine with "hosted-engine --deploy" it fails after some time because the gpg key for the ovirt-engine-appliance is missing, this leads to a failed deployment.

[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Install ovirt-engine-appliance rpm] [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 10, "changed": false, "msg": "Failed to validate GPG signature for ovirt-engine-appliance-4.5-20221026100609.1.el9.x86_64: Public key for ovirt-engine-appliance-4.5-20221026100609.1.el9.x86_64.rpm is not installed"}

Anyone know a quick workaround for this issue?

[lveyde]

That is quite weird, as the key was supposed to be automatically installed.

In any case, you can manually import the public key as described here: rpms and gpg

[lveyde]

@sandrobonazzola

[sjansen1]

That is quite weird, as the key was supposed to be automatically installed.

In any case, you can manually import the public key as described here: rpms and gpg

Hi,

maybe i try to hack ansible to get it installed, i dont think i can fiddle with the engine when it is in local deployment phase.

[lveyde]

You need to import the key to the node, it's the one that supposed to check the signature of the appliance rpm.

[sjansen1]

Oh okay, i thought it is the engine vm that is created locally before it gets transfered to the target storage.

I try that, thank you.

[sjansen1]

Something is off with the gpg keys..

[root@ovnode01 packages]# rpm -qpi ovirt-engine-appliance-4.5-20221026100609.1.el9.x86_64.rpm
warning: ovirt-engine-appliance-4.5-20221026100609.1.el9.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID fe590cb7: NOKEY
Name : ovirt-engine-appliance
Version : 4.5
Release : 20221026100609.1.el9
Architecture: x86_64
Install Date: (not installed)
Group : Applications/System
Size : 1757431588
License : GPLv2
Signature : RSA/SHA256, Wed 26 Oct 2022 10:32:36 AM UTC, Key ID ab8c4f9dfe590cb7
Source RPM : ovirt-engine-appliance-4.5-20221026100609.1.el9.src.rpm
Build Date : Wed 26 Oct 2022 10:07:55 AM UTC
Build Host : 77f4425c96e4
URL : https://www.ovirt.org/
Summary : The oVirt Engine Appliance image (OVA)
Description :
This package contains the prebuild oVirt Engine appliance image. It is intended to
be used with hosted-engine setup.

[root@ovnode01 packages]# rpm -K ovirt-engine-appliance-4.5-20221026100609.1.el9.x86_64.rpm
ovirt-engine-appliance-4.5-20221026100609.1.el9.x86_64.rpm: digests SIGNATURES NOT OK

[root@ovnode01 packages]# ls -l /etc/pki/rpm-gpg/
total 40
-rw-r--r--. 1 root root 1683 Sep 6 14:47 RPM-GPG-KEY-centosofficial
-rw-r--r--. 1 root root 1037 Apr 12 2022 RPM-GPG-KEY-CentOS-SIG-Cloud
-rw-r--r--. 1 root root 2182 Sep 6 14:47 RPM-GPG-KEY-CentOS-SIG-Extras
-rw-r--r--. 1 root root 2182 Sep 6 14:47 RPM-GPG-KEY-CentOS-SIG-Extras-SHA512
-rw-r--r--. 1 root root 1809 Apr 12 2022 RPM-GPG-KEY-CentOS-SIG-Messaging
-rw-r--r--. 1 root root 1033 Feb 10 2022 RPM-GPG-KEY-CentOS-SIG-NFV
-rw-r--r--. 1 root root 1045 Feb 4 2022 RPM-GPG-KEY-CentOS-SIG-OpsTools
-rw-r--r--. 1 root root 1041 Jan 26 2022 RPM-GPG-KEY-CentOS-SIG-Storage
-rw-r--r--. 1 root root 1061 Mar 3 2022 RPM-GPG-KEY-CentOS-SIG-Virtualization
-rw-r--r--. 1 root root 2983 Jun 15 07:12 RPM-GPG-KEY-oVirt-4.5

[root@ovnode01 packages]# gpg --dry-run /etc/pki/rpm-gpg/RPM-GPG-KEY-oVirt-4.5
gpg: WARNING: no command supplied. Trying to guess what you mean ...
pub rsa2048 2014-03-30 [SC] [expires: 2028-04-06]
31A5D7837FAD7CB286CD3469AB8C4F9DFE590CB7
uid oVirt [email protected]
sub rsa2048 2014-03-30 [E] [expires: 2028-04-06]

Trying to import this key on another machine for testing fails..

[root@testnode03 rpm-gpg]# rpm --import /tmp/RPM-GPG-KEY-oVirt-4.5
error: /tmp/RPM-GPG-KEY-oVirt-4.5: key 1 import failed.

All other rpm gpg keys from the oVirt node can be imported on my testhost. For some reason, rpm does not like this key. I grabbed this key again from another oVirt Cluster (running Stream 8), same issue.

[lveyde]

Have you tried to re-pull the key, as in instructions?

I.e.

$ gpg --recv-keys FE590CB7
$ gpg --list-keys --with-fingerprint FE590CB7

pub 2048R/FE590CB7 2014-03-30 [expires: 2028-04-06]
Key fingerprint = 31A5 D783 7FAD 7CB2 86CD 3469 AB8C 4F9D FE59 0CB7
uid oVirt [email protected]
sub 2048R/004BC303 2014-03-30 [expires: 2028-04-06]

$ gpg --export --armor FE590CB7 > ovirt-infra.pub
# rpm --import ovirt-infra.pub

[sjansen1]

[root@ovnode01 ~]# gpg --recv-keys FE590CB7
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key AB8C4F9DFE590CB7: public key "oVirt [email protected]" imported
gpg: Total number processed: 1
gpg: imported: 1
[root@ovnode01 ~]# gpg --list-keys --with-fingerprint FE590CB7
pub rsa2048 2014-03-30 [SC] [expires: 2028-04-06]
31A5 D783 7FAD 7CB2 86CD 3469 AB8C 4F9D FE59 0CB7
uid [ unknown] oVirt [email protected]
sub rsa2048 2014-03-30 [E] [expires: 2028-04-06]

[root@ovnode01 ~]# gpg --export --armor FE590CB7 > ovirt-infra.pub
[root@ovnode01 ~]# rpm --import ovirt-infra.pub
warning: Signature not supported. Hash algorithm SHA1 not available.
error: ovirt-infra.pub: key 1 import failed.

just found this on the net : "RHEL 9 deprecating and no longer enabling SHA1 out of the box". Is it possible that CentOS Stream 9 has SHA 1 disabled?

[sjansen1]

[root@ovnode01 ~]# update-crypto-policies --set LEGACY
Setting system policy to LEGACY
Note: System-wide crypto policies are applied on application start-up.
It is recommended to restart the system for the change of policies
to fully take place.

[root@ovnode01 ~]# rpm --import ovirt-infra.pub

[root@ovnode01 ~]# rpm -K /var/cache/dnf/ovirt-45-upstream-6644f816c5ff2731/packages/ovirt-engine-appliance-4.5-20221026100609.1.el9.x86_64.rpm
/var/cache/dnf/ovirt-45-upstream-6644f816c5ff2731/packages/ovirt-engine-appliance-4.5-20221026100609.1.el9.x86_64.rpm: digests signatures OK

I try to continue for now, i hope the engine that gets created has legacy support enabled.

[lveyde]

Yes, looks like we may need to create new signing keys for EL9.

[sjansen1]

Good news, with "update-crypto-policies --set LEGACY" on the node, i was able to complete the hosted engine deployment. EL9 based node and engine is up and running on a new FC SAN.

I enable the policy on any addtional node to be sure.

[lveyde]

Just don't forget to switch back to the default after you're finished with the installation:

update-crypto-policies --set DEFAULT

[sandrobonazzola]

@lveyde is the new gpg key included in 4.5.4? Can we close this issue?