Customization using additional OCI layers at boot time

[agracey]

One of the questions that we are continually asked is how to customize different portions of the stack. The options that we have currently pose a few issues:

• Building your own image
  • Not supportable
  • Causes maintenance burden for whoever needs to manage the custom images since they contain our components as well
• Plans
  • Not secure as they run as root
  • Imperative vs declarative

What I would like to suggest is that we add a mechanism that allows for mounting an additional set of OCI image layers on top of the root filesystem at boot time.

Basic process:

• Admin builds an OCI artifact (FROM scratch) that includes the files that they would like to be added/overridden
• Admin signs with cosign
• Admin publishes to registry
• Admin specifies add on image tag in CRD managed by Elemental operator
• Machines "upgrades" to new add-on
• When machine boots, grub adds additional overlays on top of Teal image before loading kernel
• System continues to boot normally with additive filesystem

I'm a little fuzzy on how grub treats filesystems and the handoff between grub and the kernel but from my understanding of the docs, this is doable?

I'm hoping to get some architecture drawings out shortly and can draw an extended version that includes this idea if needed.

Requirements:

• An operator needs to be able to install additional drivers or kernel modules into their systems
  • This should be at install time or on an already running system (including after the Kubernetes cluster is provisioned)
  • This can require a reboot if needed
• Secure boot should not be impacted
• A running system stays immutable
  • I would prefer not to deal with drift management systems
• Buildable and deployable by an end-user
• Declarative (as opposed to the one shot scripts that plans give us)

Nice to haves:

• The customization asset itself should be signable and verifiable at fetch and run time
• The asset should be relatively easy to build

[kkaempf]

We could just include luet in the base image 🤔

[agracey]

What would that look like to the admin setting it up?

[agracey]

Or the person building the customization?

[agracey]

I've edited the text above to add a list of requirements

[Itxaka]

Secure boot should not be impacted

Unless the kernel drivers are not signed! If you boot under secureboot both the kernel and the kernel modules need to be signed, otherwise the kernel will refuse to load those modules. Small issue but very important!