Two partitions with the same COS_GRUB label causing issues during install time

[alihamzam]

Hi all,

We're trying to install Kairos onto bare-metal machines, and running into a few issues with the automated installation. We're using Kairos 3.5.0 with our own cloud-config file:

#cloud-config

install:
  device: /dev/disk/by-path/pci-0000:00:17.0-ata-3
  auto: true
  reboot: true

debug: true

k3s:
  enabled: true

users:
- name: "username"
  passwd: "password"
  groups: [ "admin" ]

bundles:
- targets:
  - run://quay.io/kairos/community-bundles:system-upgrade-controller_latest

stages:
  initramfs:
    - name: "Setup hostname"
      hostname: "my-custom-hostname-{{ trunc 6 .MachineID }}"

However, the installer fails at copying EFI files from the rootfs into the esp partition with the following error:

* error writing /run/cos/efi/EFI/boot/shimx64.efi.dualsigned: write /run/cos/efi/EFI/boot/shimx64.efi.dualsigned: no space left on device

We did some investigation and figured out that because of the physical machine we're using, udev has decided to mount the USB as the primary drive and the machine's installed HDD as second. In effect, this means the USB is /dev/sda and the HDD is /dev/sdb. We've tried on other machines, and we seem to get different results.

We think that by this stage of the installation, since there's a COS_GRUB partition on both the USB and HDD, the kairos-agent is mounting COS_GRUB, and the one on the USB gets mounted. If we mount COS_GRUB on the HDD, it's empty, but COS_GRUB on the USB is 100% full and it has a half-copied shimx64.efi.dualsigned, where it doesn't have one when we write the ISO to the USB.

If we try on a machine where the HDD is mounted as sda, then the installation seems to work fine. If we manually change the esp partition label on the USB (for example, to COS_GRUB2), the installation works too. However, we can't make this change through a config file or a command-line flag, since it looks like the COS_GRUB label for the USB installer is hardcoded in AuroraBoot: https://github.com/kairos-io/AuroraBoot/blob/77dfd9c540dd92f544a4505b50bae5c608f74e60/pkg/constants/constants.go#L46. Plus, given that it worked when we manually changed the label, we think it's safe to assume that the COS_GRUB label on the USB is not used anywhere.

We haven't hit this issue with Kairos 3.4.2, but it also seems that the HDD is consistently mounted as /dev/sda and not sdb under 3.4.2. We think there might be some change in logic in the kairos-agent between 3.4.2 and 3.5.0. We haven't tested 3.4.2 extensively so we're not entirely sure, however we do think 3.4.2 would have the same issue if the HDD was mounted as sdb.

Could we have some assistance here please? Thanks in advance!

[coneheed]

I wonder if the issue is with this function in kairos-agent?

At the time it is called, there will be 2 disks each containing a partition labelled "COS_GRUB".

And so it is arbitrary (or at least down to some OS/udev ordering) which disk will be first in the slice returned by GetDisks()

In the happy case, the target disk is returned in the slice first. When it fails, the install disk is returned first.

// GetEfiPartition returns the EFI partition by looking for the partition with the label "COS_GRUB"
func GetEfiPartition(logger *types.KairosLogger) (*types.Partition, error) {
	var efiPartition *types.Partition
	for _, d := range ghw.GetDisks(ghw.NewPaths(""), logger) {
		for _, part := range d.Partitions {
			if part.FilesystemLabel == constants.EfiLabel {
				efiPartition = part
				break
			}
		}
	}

	if efiPartition == nil {
		return efiPartition, fmt.Errorf("could not find EFI partition")
	}
	return efiPartition, nil

[Itxaka]

I dont think usb should be mounted as sdx devices. Maybe ita a usb nvme Drive? I seen those behave like normal disks but I think they should be treated differently and that may be an udev issue...

[alihamzam]

In our case, it looks like the USB is being mounted as /dev/sda. But even if it was mounted as a NVME device, the conflicting labels will still cause an issue, and depending on udev the wrong partition gets selected (i.e. the installer USB). It doesn't seem like we should be reliant on whether udev decides to prioritise the install target rather than the installer USB, since it seems that's not always the case?