Boot fails with "vmlinuz has invalid signature" or "bad shim signature, you need to load the kernel first" #543

Mershl · 2024-03-28T20:10:45Z

Current workaround

Original issue text

Describe the bug
Trying to rebase an existing SB39 to SB40 fails to boot showing vmlinuz-6.8.1-300.fc40.x86.x64 has invalid signature. you need to load the kernel first on an Thinkpad T495. I was not able to reproduce the issue on my desktop systems.

coming from deployment:

fedora:fedora/39/x86_64/silverblue
                  Version: 39.20240328.0 (2024-03-28T00:39:32Z)
               BaseCommit: a1ac8885d05ca13728d92c2bcf1ded67a7ddb409d657e446a808397366a463b1
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C

going for:

fedora:fedora/40/x86_64/silverblue
                  Version: 40.20240328.n.0 (2024-03-28T08:09:28Z)
               BaseCommit: f726d0be3361a42a8ac175b08851de67a2d97c9c01ee130a3abcd32720120f9c
             GPGSignature: Valid signature by 115DF9AEF857853EE8445D0A0727707EA15B79CC

What I've tried so far

cleanup + rpm-ostree initramfs --enable + rebase
cleanup -r (removing all other deployments) + rebase
rebase to fedora:fedora/40/x86_64/testing/silverblue instead, same error except error message points to vmlinuz-6.8.2

To Reproduce

rpm-ostree rebase fedora:fedora/40/x86_64/silverblue
systemctl reboot

The text was updated successfully, but these errors were encountered:

travier · 2024-03-29T10:45:46Z

I've hit the same issue and I've disabled Secure Boot for now.

It's likely due to the bootloader being too old. bootupd should land in the image after the release and will then let people update their bootloader.

Or maybe we should include bootupd in Fedora 39 now so that people can update their bootloader before upgrading to Fedora 40.

Mershl · 2024-04-07T11:13:29Z

Oh Oh.

SB39 (stable) just updated (no rebase) to kernel 6.8.4 showing the very same issue.

  kernel 6.7.11-200.fc39.x86_64 -> 6.8.4-200.fc39.x86_64

breaks boot, showing

vmlinuz-6.8.4-200.fc39.x86_64 has invalide signature.
you need to load the kernel first.

(Thinkpad T495 using fedora:fedora/39/x86_64/silverblue)

HassoSigbjoernson · 2024-04-08T11:40:29Z

Or maybe we should include bootupd in Fedora 39 now so that people can update their bootloader before upgrading to Fedora 40.

And people have to be told what to do. I got a bad shim signature. you need to load the kernel first error while updating my SB39 system (no rebase to 40) and was able to fix it by copying files from /sysroot/ostree/deploy/fedora/deploy/[...]/usr/lib/ostree-boot/efi/EFI/fedora to /boot/efi/EFI/fedora but I was only able to do this because I had already encountered this problem when it affected arm64 systems a year ago.

travier · 2024-04-08T15:41:34Z

Pinging Universal Blue folks as it's likely going to impact them soon: @castrojo @noelmiller @EyeCantCU @KyleGospo

travier · 2024-04-09T16:38:41Z

Unfortunately, now that the kernel has landed in an update, we can not add bootupd to a future F39 build as you won't be able to boot it to get the fix.

Overlaying bootupd via rpm-ostree install is not enough as it needs to run a command during the commit build process.

So we have coreos/bootupd#635 as an option (that I have not tested yet) to create container images with bootupd based on an older commit with the previous kernel.

Or we document how to manually fix this until bootupd finally lands in Atomic Desktops. Unfortunately it will not fully be in Fedora 40 by default yet. See: https://gitlab.com/fedora/ostree/sig/-/issues/1

hferreiro · 2024-04-09T19:50:46Z

Can't the update to the kernel package be reverted in the same commit that bootupd is added?

travier · 2024-04-09T19:54:58Z

No unfortunately, for Fedora Atomic Desktops, we always take the latest RPMs from the repos.

jorgeml · 2024-04-10T15:59:12Z

@travier I'm watching this one with interest as my Silverblue image is now one week old. If I understand it correctly, the only solution at the moment is to disable Secure Boot? And not even upgrading to Fedora 40 will solve it in the near future?

travier · 2024-04-10T18:31:19Z

The workaround for this issue is going to be commands that do what bootupd does but manually unfortunately.

This is mostly doing a copy from /usr/lib/ostree-boot to /boot/efi on EFI systems, and a /usr/sbin/grub2-install call on BIOS ones, with the correct arguments.

I have not tested any of this so I'm not providing ready made commands and everything is at your own risk. Please make backups and make sure that you are confortable rescuing a broken system before trying thing out.

Help with testing in VMs or on real hardware is welcomed.

I recommend disabling Secure Boot in the meantime until this is fixed.

travier · 2024-04-10T20:06:06Z

Warning: Those instructions should be safe to follow, but still, do at your own risk, make backups

Here is the set of commands I've just used to update my (x86_64) EFI booted system successfully:

# Enter a root shell on the host (i.e. not in a toolbox)
$ sudo -i

# Make a backup of the content of the EFI partition
$ cd /boot/efi/
$ cp -a EFI EFI.bkp

# Copy updated bootloader versions
$ cp /usr/lib/ostree-boot/efi/EFI/BOOT/{BOOTIA32.EFI,BOOTX64.EFI,fbia32.efi,fbx64.efi} /boot/efi/EFI/BOOT/
$ cp /usr/lib/ostree-boot/efi/EFI/fedora/{BOOTIA32.CSV,BOOTX64.CSV,grubia32.efi,grubx64.efi,mmia32.efi,mmx64.efi,shim.efi,shimia32.efi,shimx64.efi} /boot/efi/EFI/fedora/

# Only needed if it exists already on your system
$ cp /usr/lib/ostree-boot/efi/EFI/fedora/shimx64.efi /boot/efi/EFI/fedora/shimx64-fedora.efi

# Sync changes to the disk
$ sync

# Reboot

Once reboot is successful, you can remove the backup copies:

# Enter a root shell on the host (i.e. not in a toolbox)
$ sudo -i

# Make a backup of the content of the EFI partition
$ cd /boot/efi/
$ rm -ri ./EFI.bkp

# Sync changes to the disk
$ sync

Edit: Updated to add 32bits EFI binaries as well.

For aarch64, update the filenames as needed.

jorgeml · 2024-04-10T20:23:58Z

Thanks @travier ! Just did this on my machine and it worked. I was also able to reenable Secure Boot. As a positive side effect, it also looks like the UEFI dbx update was applied. That had been blocked for a while.

cgwalters · 2024-04-11T13:14:32Z

OK sorry yes, this is an embarrassing problem.

@travier and I had a chat, a few things here:

We can safely add bootupd to f39
I think we can also document a flow of rpm-ostree upgrade but without restarting, then
Run bootupd from the new root

Alternatively, we can document how to run bootupd from a privileged container. (In fact, running it from the existing quay.io/fedora/fedora-coreos:stable container for example. One minor stumbling block here is that bootupd wants to act in a client/server model with systemd, but I think we can change it to not do that)

Longer term yes, the technical debt here is high and makes coreos/bootupd#454 important to do and just turn on automatic updates, or at least automatic updates when they're needed.

Mershl · 2024-04-11T18:56:04Z

Copy updated bootloader versions
$ cp /usr/lib/ostree-boot/efi/EFI/BOOT/{BOOTIA32.EFI,BOOTX64.EFI,fbia32.efi,fbx64.efi} /boot/efi/EFI/BOOT/
$ cp /usr/lib/ostree-boot/efi/EFI/fedora/{BOOTIA32.CSV,BOOTX64.CSV,grubia32.efi,grubx64.efi,mmia32.efi,mmx64.efi,shim.efi,shimia32.efi,shimx64.efi} /boot/efi/EFI/fedora/

Only needed if it exists already on your system
$ cp /usr/lib/ostree-boot/efi/EFI/fedora/shimx64.efi /boot/efi/EFI/fedora/shimx64-fedora.efi

@travier I've noticed that the affected Thinkpad T495 was missing boot/efi/EFI/BOOT completely. I skipped the line in your test steps and it worked out perfectly. SecureBoot reenabled, no regressions found so far.

Looking into other systems I see the mentioned boot/efi/EFI/BOOT. Does someone know the difference, does it depend on the birth date of the Fedora/Silverblue installation?

UPDATE: upgraded two other desktop systems successfully (SB39->SB40 with enabled SecureBoot) using the test steps.

A6GibKm · 2024-04-13T17:39:34Z

Warning: Do at your own risk, only lightly tested

Here is the set of commands I've just used to update my (x86_64) EFI booted system successfully:

# Enter a root shell on the host (i.e. not in a toolbox)
$ sudo -i

# Make a backup of the content of the EFI partition
$ cd /boot/efi/
$ cp -a EFI EFI.bkp

# Copy updated bootloader versions
$ cp /usr/lib/ostree-boot/efi/EFI/BOOT/{BOOTIA32.EFI,BOOTX64.EFI,fbia32.efi,fbx64.efi} /boot/efi/EFI/BOOT/
$ cp /usr/lib/ostree-boot/efi/EFI/fedora/{BOOTIA32.CSV,BOOTX64.CSV,grubia32.efi,grubx64.efi,mmia32.efi,mmx64.efi,shim.efi,shimia32.efi,shimx64.efi} /boot/efi/EFI/fedora/

# Only needed if it exists already on your system
$ cp /usr/lib/ostree-boot/efi/EFI/fedora/shimx64.efi /boot/efi/EFI/fedora/shimx64-fedora.efi

# Sync changes to the disk
$ sync

# Reboot

Once reboot is successful, you can remove the backup copies:

# Enter a root shell on the host (i.e. not in a toolbox)
$ sudo -i

# Make a backup of the content of the EFI partition
$ cd /boot/efi/
$ rm -ri ./EFI.bkp

# Sync changes to the disk
$ sync

Edit: Updated to add 32bits EFI binaries as well.

I tested these steeps on my workstation and my Thinkpad T480. I managed to get secure boot on both devices.

On the desktop I had to use fwupdmgr to update the UEFI dbx.

rugk · 2024-04-25T09:50:24Z

Has the same issue reported here (also includes a kernel bug report though I am unsure how helpful that is), so, but this question AFAIK remains unanswered:

And not even upgrading to Fedora 40 will solve it in the near future?

So does updating fix it? I mean I can boot into an old working Fedora 39 version.
If I upgrade from there to Fedora 40, does that work? (Edit: tested, does not work)

Alternatively, if I need to disable Secure Boot temporarily, can I just do so – upgrade to Fedora 40 and boot successfully, afterwards? (Edit: tested, does not work, I can only boot with SecureBoot disabled)

Also, would this possibly be a candidate for a common issue if it prevents upgrading to Fedora 40?

travier · 2024-04-25T11:44:22Z

It's a good idea to make it a common issue.

It's basically impossible to fix automatically right now for F39 users updating as the new kernel already landed in Fedora 39.

We'll try to get bootupd support to a sufficiently good shape to have a small set of commands to enter manually but in the meantime, the commands in #543 (comment) are the best that we have.

rugk · 2024-04-25T15:00:45Z

Okay then, I have created a draft for this here:
https://discussion.fedoraproject.org/t/booting-fails-with-vmlinuz-has-invalid-signature/114354

Unfortunately I miss the technical details etc. to properly document it. So feel free to edit it please.

rugk · 2024-04-29T22:16:27Z

FYI I can confirm the workaround posted worked:

# tree EFI
EFI
├── BOOT
│   ├── BOOTIA32.EFI
│   ├── BOOTX64.EFI
│   ├── fbia32.efi
│   └── fbx64.efi
└── fedora
    ├── BOOTIA32.CSV
    ├── BOOTX64.CSV
    ├── fonts
    ├── grub.cfg
    ├── grub.cfg.old
    ├── grubenv
    ├── grubenvLa7zjw
    ├── grubia32.efi
    ├── grubx64.efi
    ├── mmia32.efi
    ├── mmx64.efi
    ├── shim.efi
    ├── shimia32.efi
    ├── shimx64.efi
    └── shimx64-fedora.efi

4 directories, 18 files
# tree EFI.bkp/
EFI.bkp/
├── BOOT
│   ├── BOOTX64.EFI
│   └── fbx64.efi
└── fedora
    ├── BOOTX64.CSV
    ├── fonts
    ├── grub.cfg
    ├── grub.cfg.old
    ├── grubenv
    ├── grubenvLa7zjw
    ├── grubx64.efi
    ├── mmx64.efi
    ├── shim.efi
    ├── shimx64.efi
    └── shimx64-fedora.efi

fbruetting · 2024-05-02T12:28:28Z

Warning: Do at your own risk, only lightly tested

Here is the set of commands I've just used to update my (x86_64) EFI booted system successfully:

# Enter a root shell on the host (i.e. not in a toolbox)
$ sudo -i

# Make a backup of the content of the EFI partition
$ cd /boot/efi/
$ cp -a EFI EFI.bkp

# Copy updated bootloader versions
$ cp /usr/lib/ostree-boot/efi/EFI/BOOT/{BOOTIA32.EFI,BOOTX64.EFI,fbia32.efi,fbx64.efi} /boot/efi/EFI/BOOT/
$ cp /usr/lib/ostree-boot/efi/EFI/fedora/{BOOTIA32.CSV,BOOTX64.CSV,grubia32.efi,grubx64.efi,mmia32.efi,mmx64.efi,shim.efi,shimia32.efi,shimx64.efi} /boot/efi/EFI/fedora/

# Only needed if it exists already on your system
$ cp /usr/lib/ostree-boot/efi/EFI/fedora/shimx64.efi /boot/efi/EFI/fedora/shimx64-fedora.efi

# Sync changes to the disk
$ sync

# Reboot

Once reboot is successful, you can remove the backup copies:

# Enter a root shell on the host (i.e. not in a toolbox)
$ sudo -i

# Make a backup of the content of the EFI partition
$ cd /boot/efi/
$ rm -ri ./EFI.bkp

# Sync changes to the disk
$ sync

Edit: Updated to add 32bits EFI binaries as well.

Thereafter, I get the following:

After rpm-ostree update:

❯ rpm-ostree update
…
…
rpm-md repo 'updates-archive' (cached); generated: 2024-05-02T02:13:06Z solvables: 45817
Resolving dependencies... done
Applying 2 overrides and 201 overlays
Processing packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
error: Staging deployment: Cleaning deployments: Removing ostree/deploy/fedora/deploy/3fd29d12b510aabcc32bd5ab5a6262b4d1a4da43efc7a70788c11837f1afa140.0: unlinkat(classic-topbar-network-cellular-acquiring.svg): Das Dateisystem ist nur lesbar (engl.: „File system is read-only“)

At second try:

❯ LANG=en_US.utf8; rpm-ostree update
error: Remounting /sysroot read-write: Das Argument ist ungültig (engl.: „Argument not valid“)

rugk · 2024-05-02T13:56:32Z

@fbruetting seems like a different issue, OSTree updates should not have anything to do with the bootloader update. Maybe ask at the Fedora discussions forum for help?

kernel 6.9 won't boot on system installed prior to F39, as shim is too old. Shim 15.8-3 reached stable on 2023-03-21, so any system using secureboot installed before that won't be able to boot kernel 6.9 See coreos/fedora-coreos-tracker#1752 fedora-silverblue/issue-tracker#543

The 6.9 kernel won't boot on systems installed prior to F39, as the shim is too old. Add a systemd unit that updates the bootloader on those machines. Manually handle systems with mirrored ESPs. See also: coreos/fedora-coreos-tracker#1752 Fixes: fedora-silverblue/issue-tracker#543 Co-authored-by: Jonathan Lebon <[email protected]>

travier · 2024-06-28T15:43:18Z

For folks interested in helping us with this issue, I've written a draft for an article to be published on the Fedora Magazine: https://fedoramagazine.org/?p=40664&preview=1&_ppp=2f0e5b87ab

It includes a "simpler" workaround that would benefit from a little bit more testing. I have also not yet looked at the history to find which versions where impacted first so if folks can find that it would be helpful.

Thanks!

TimonLukas · 2024-06-28T21:15:35Z

@travier Just tried it on Kinoite (through Ublue), everything worked as expected! Thank you!

miabbott · 2024-06-30T18:13:31Z

It includes a "simpler" workaround that would benefit from a little bit more testing. I have also not yet looked at the history to find which versions where impacted first so if folks can find that it would be helpful.

I did some testing this weekend using a VM and the problem seems to be introduced with the 6.9 kernel in Fedora. I found this affects both Fedora Silverblue 39 (kernel-6.9.4-100.fc39) and Fedora Sliverblue 40 (kernel-6.9.4-200.fc40).

The problematic kernel is introduced in the following versions:

39.20240617.0 (2f87250a645076edab387fff73334d123297d22b6a8a47a5e08f2128d83426a5)
40.20240617.0 (c7bdc9ecab9df0609da0e04b685ca7608174e6cb81d49e1cf4b70b798557018e)

The preview link to the Fed Mag post has expired, but I used the workaround information in this issue successfully.

travier · 2024-07-01T11:35:18Z

Thanks @miabbott and @TimonLukas !

travier · 2024-07-01T14:05:15Z

Updated preview: https://fedoramagazine.org/?p=40664&preview=1&_ppp=8e94781824

miabbott · 2024-07-01T17:49:33Z

@travier if you want to update the article with IoT information, it looks like the fedora/stable/x86_64/iot ref received the affected kernel as part of 40.20240617.0 (62c8ff246886838c8b5df7ca5ff060fccee8705fa7114f3ec47dad0103ac3ba9)

miabbott · 2024-07-02T12:18:55Z

@travier See https://discussion.fedoraproject.org/t/update-originated-bad-shim-error/124397

Should we include instructions about bringing Kinoite/Silverlbue 39 up-to-date before performing the workaround?

travier · 2024-07-02T12:28:19Z

Ah, that's a good point, we indeed need that :/

travier · 2024-07-02T12:29:57Z

Hum, but why did this not happen when I tested on F39? I'll test again.

juhp · 2024-07-03T05:29:40Z

Do you want to update the preview link? (maybe it is only valid until updated or times out?)

travier · 2024-07-03T18:31:56Z

Updated again: https://fedoramagazine.org/?p=40664&preview=1&_ppp=71b3c6065a

miabbott · 2024-07-03T19:00:47Z

Updates looks good! Thank you for working on this @travier!

richiedaze · 2024-07-04T15:11:02Z

Ran into this issue trying to upgrade my old laptop on silverblue 39. Read and followed the above preview article, and now I am on 40. Ty

kronenpj · 2024-07-13T14:58:55Z

I’m unable to upgrade to the latest working version mentioned in the article. From a fresh Silverblue 39 installation, updated to 39.20240610.0:

root@fedora:~# rpm-ostree deploy 39.20240616.0 
Resolving version '39.20240616.0'
1 metadata, 0 content objects fetched; 592 B transferred in 2 seconds; 0 bytes content written
error: Version 39.20240616.0 not found in fedora:fedora/40/x86_64/silverblue

Is there another working version we should be using to test this solution?
Also, is there a way to find a list of all the {release}.{date}.x versions available?

travier · 2024-07-15T11:19:09Z

Please don't post the same things in multiple places without linking to it. I've answered in https://discussion.fedoraproject.org/t/talk-boot-fails-with-bad-shim-signature-in-atomic-desktops-and-iot/125412/6.

Mershl added the bug Something isn't working label Mar 28, 2024

travier added the f40 Related to Fedora 40 label Mar 29, 2024

travier mentioned this issue Mar 29, 2024

Is it safe to run /usr/bin/bootupctl backend generate-update-metadata in a Containerfile? coreos/bootupd#635

Open

travier added the f39 Related to Fedora 39 label Apr 8, 2024

castrojo mentioned this issue Apr 8, 2024

bootupd upgrade issue tracking (F40) ublue-os/main#551

Closed

travier changed the title ~~rebase to SB40 fails to boot on notebook ("vmlinuz has invalid signature")~~ F39/F40: Fails to boot on with "vmlinuz has invalid signature" Apr 9, 2024

travier mentioned this issue Apr 11, 2024

Can't update dbx via fwupdmgr #355

Closed

travier pinned this issue Apr 11, 2024

travier mentioned this issue Apr 11, 2024

Use bootupd and remove ostree-grub2 (was: All deployments are shown twice in grub) #120

Open

This comment was marked as off-topic.

Sign in to view

jlebon closed this as completed in coreos/fedora-coreos-config#3042 Jun 27, 2024

jlebon closed this as completed in coreos/fedora-coreos-config@8dd5b81 Jun 27, 2024

travier reopened this Jun 28, 2024

Boot fails with "vmlinuz has invalid signature" or "bad shim signature, you need to load the kernel first" #543

Boot fails with "vmlinuz has invalid signature" or "bad shim signature, you need to load the kernel first" #543

Comments

Mershl commented Mar 28, 2024 • edited by travier Loading

Current workaround

Original issue text

travier commented Mar 29, 2024

Mershl commented Apr 7, 2024 • edited Loading

HassoSigbjoernson commented Apr 8, 2024 • edited Loading

travier commented Apr 8, 2024

travier commented Apr 9, 2024

hferreiro commented Apr 9, 2024

travier commented Apr 9, 2024

jorgeml commented Apr 10, 2024

travier commented Apr 10, 2024 • edited Loading

travier commented Apr 10, 2024 • edited Loading

jorgeml commented Apr 10, 2024

cgwalters commented Apr 11, 2024

Mershl commented Apr 11, 2024 • edited Loading

A6GibKm commented Apr 13, 2024

rugk commented Apr 25, 2024 • edited Loading

travier commented Apr 25, 2024

rugk commented Apr 25, 2024

rugk commented Apr 29, 2024

fbruetting commented May 2, 2024 • edited Loading

rugk commented May 2, 2024

This comment was marked as off-topic.

travier commented Jun 28, 2024

TimonLukas commented Jun 28, 2024 • edited Loading

miabbott commented Jun 30, 2024 • edited Loading

travier commented Jul 1, 2024 • edited Loading

travier commented Jul 1, 2024 • edited Loading

miabbott commented Jul 1, 2024 • edited Loading

miabbott commented Jul 2, 2024

travier commented Jul 2, 2024

travier commented Jul 2, 2024

juhp commented Jul 3, 2024

travier commented Jul 3, 2024

miabbott commented Jul 3, 2024

richiedaze commented Jul 4, 2024

kronenpj commented Jul 13, 2024

travier commented Jul 15, 2024 • edited Loading

Mershl commented Mar 28, 2024 •

edited by travier

Loading

Mershl commented Apr 7, 2024 •

edited

Loading

HassoSigbjoernson commented Apr 8, 2024 •

edited

Loading

travier commented Apr 10, 2024 •

edited

Loading

travier commented Apr 10, 2024 •

edited

Loading

Mershl commented Apr 11, 2024 •

edited

Loading

rugk commented Apr 25, 2024 •

edited

Loading

fbruetting commented May 2, 2024 •

edited

Loading

TimonLukas commented Jun 28, 2024 •

edited

Loading

miabbott commented Jun 30, 2024 •

edited

Loading

travier commented Jul 1, 2024 •

edited

Loading

travier commented Jul 1, 2024 •

edited

Loading

miabbott commented Jul 1, 2024 •

edited

Loading

travier commented Jul 15, 2024 •

edited

Loading