initrd-setup-root: Use systemd-confext instead of custom overlay mount #115
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pothos
force-pushed
the
kai/default-confext
branch
2 times, most recently
from
83d11c7 to
f7f20c1
Compare
3 tasks
pothos
added a commit
to flatcar/scripts
that referenced
this pull request
Dec 12, 2025
For A/B-updated /etc contents we used a custom overlay mount that provides the default files through a lowerdir loaded from /usr. Since then we upstreamed mutable systemd-confext support and now we can switch to it. This pulls in flatcar/init#138 and flatcar/bootengine#115 together with backported systemd patches that have opened or merged upstream PRs to fix --root= issues and add a refresh skip check to prevent boot disruptions due to the multiple daemon reloads and - more important - the missing atomic remount that would mean /etc is gone for a few milliseconds during boot. The skip logic works best with verity hashes and thus the default confext must be a verity extension image. User-provided confext don't work well yet unless they use verity due to the missing atomic remount and reliance on the skipping logic. We also need to look into stacking order and other mutabiliy settings. Signed-off-by: Kai Lueke <[email protected]>
chewi
approved these changes
Dec 12, 2025
Contributor
chewi
left a comment
chewi
left a comment
There was a problem hiding this comment.
I'm not very familiar with this, but it looks good. Please could you link the relevant unreleased systemd changes in this pull request for reference.
pothos
added a commit
to flatcar/scripts
that referenced
this pull request
Dec 15, 2025
For A/B-updated /etc contents we used a custom overlay mount that provides the default files through a lowerdir loaded from /usr. Since then we upstreamed mutable systemd-confext support and now we can switch to it. This pulls in flatcar/init#138 and flatcar/bootengine#115 together with backported systemd patches that have opened or merged upstream PRs to fix --root= issues and add a refresh skip check to prevent boot disruptions due to the multiple daemon reloads and - more important - the missing atomic remount that would mean /etc is gone for a few milliseconds during boot. The skip logic works best with verity hashes and thus the default confext must be a verity extension image. User-provided confext don't work well yet unless they use verity due to the missing atomic remount and reliance on the skipping logic. We also need to look into stacking order and other mutabiliy settings. TODO: List of PRs and patch files (inc dep commits) Signed-off-by: Kai Lueke <[email protected]>
pothos
added a commit
to flatcar/scripts
that referenced
this pull request
Dec 15, 2025
For A/B-updated /etc contents we used a custom overlay mount that provides the default files through a lowerdir loaded from /usr. Since then we upstreamed mutable systemd-confext support and now we can switch to it. This pulls in flatcar/init#138 and flatcar/bootengine#115 together with backported systemd patches that have opened or merged upstream PRs to fix --root= issues and add a refresh skip check to prevent boot disruptions due to the multiple daemon reloads and - more important - the missing atomic remount that would mean /etc is gone for a few milliseconds during boot. The skip logic works best with verity hashes and thus the default confext must be a verity extension image. User-provided confext don't work well yet unless they use verity due to the missing atomic remount and reliance on the skipping logic. We also need to look into stacking order and other mutabiliy settings. TODO: List of PRs and patch files (inc dep commits) Signed-off-by: Kai Lueke <[email protected]>
pothos
added a commit
to flatcar/scripts
that referenced
this pull request
Dec 16, 2025
For A/B-updated /etc contents we used a custom overlay mount that provides the default files through a lowerdir loaded from /usr. Since then we upstreamed mutable systemd-confext support and now we can switch to it. This pulls in flatcar/init#138 and flatcar/bootengine#115 together with backported systemd patches that have opened or merged upstream PRs to fix --root= issues and add a refresh skip check to prevent boot disruptions due to the multiple daemon reloads and - more important - the missing atomic remount that would mean /etc is gone for a few milliseconds during boot. The skip logic works best with verity hashes and thus the default confext must be a verity extension image. User-provided confext don't work well yet unless they use verity due to the missing atomic remount and reliance on the skipping logic. We also need to look into stacking order and other mutabiliy settings. The backported systemd patches relate to the following upstream PRs: systemd/systemd#39843 for vpick-Don-t-use-openat-directly-but-resolve-symlinks discover-image-Follow-symlinks-in-a-given-root sysext-Use-correct-image-name-for-extension-release test-Add-tests-for-handling-symlinks-with-systemd-sy Note that the patch in the PR relies on 0859fe3f32774f1e0c787974cc252ff922a1b868 but the backport patch not. systemd/systemd#39980 for sysext-Create-mutable-directory-with-the-right-mode sysext-Skip-refresh-if-no-changes-are-found systemd/systemd#39991 for sysext-Get-verity-user-certs-from-given-root systemd/systemd#40063 for sysext-Fix-config-file-support-with-root which relies on systemd/systemd#38250 for man-sysext.conf-add-systemd-sysext-config-files sysext-introduce-global-config-file sysext-support-ImagePolicy-global-config-option Signed-off-by: Kai Lueke <[email protected]>
Member
Author
Thanks, yes, I wanted to capture this somewhere and have now added it to the commit message in the scripts repo that ships the patches. |
So far we had a custom overlay mount for /etc that provided the A/B updated files from /usr in a lowerdir. Since then we upstreamed a mutable mode for sysext and confext. We can now switch over to it and provide a default confext by using the mutable mode. Because there is no atomic remount yet and also because we want to avoid daemon reloads during boot, this relies on a new skip logic in systemd-sysext/confext to only refresh in the final system boot when changes are found. Through only using verity images we know that no changes can be there because they get compared by hash and not mtime. When we would hit a refresh during boot then /etc contents are shortly gone and services sometimes fail during boot. A bit specific to Flatcar/Ignition is that we load confext twice in the initrd, once because we have a default confext that provides /etc contents for Ignition and a second time for loading user confexts for the final system (when users added new ones through Ignition). Signed-off-by: Kai Lueke <[email protected]>
pothos
force-pushed
the
kai/default-confext
branch
from
f7f20c1 to
d2365fe
Compare
Member
Author
|
Just noticed that we should not fail the boot when sysexts can't get merged (broken sysexts) but continue booting. Otherwise we would force people to use the serial/VGA console instead of SSH. For confext we can't really continue because then we lack /etc contents and failing properly is best - the errors one gets due to missing /etc files are very strange. I've also added detecting when the service is masked and then don't merge sysexts in the initrd. |
pothos
added a commit
to flatcar/scripts
that referenced
this pull request
Dec 17, 2025
For A/B-updated /etc contents we used a custom overlay mount that provides the default files through a lowerdir loaded from /usr. Since then we upstreamed mutable systemd-confext support and now we can switch to it. This pulls in flatcar/init#138 and flatcar/bootengine#115 together with backported systemd patches that have opened or merged upstream PRs to fix --root= issues and add a refresh skip check to prevent boot disruptions due to the multiple daemon reloads and - more important - the missing atomic remount that would mean /etc is gone for a few milliseconds during boot. The skip logic works best with verity hashes and thus the default confext must be a verity extension image. User-provided confext don't work well yet unless they use verity due to the missing atomic remount and reliance on the skipping logic. We also need to look into stacking order and other mutabiliy settings. The backported systemd patches relate to the following upstream PRs: systemd/systemd#39843 for vpick-Don-t-use-openat-directly-but-resolve-symlinks discover-image-Follow-symlinks-in-a-given-root sysext-Use-correct-image-name-for-extension-release test-Add-tests-for-handling-symlinks-with-systemd-sy Note that the patch in the PR relies on 0859fe3f32774f1e0c787974cc252ff922a1b868 but the backport patch not. systemd/systemd#39980 for sysext-Create-mutable-directory-with-the-right-mode sysext-Skip-refresh-if-no-changes-are-found systemd/systemd#39991 for sysext-Get-verity-user-certs-from-given-root systemd/systemd#40063 for sysext-Fix-config-file-support-with-root which relies on systemd/systemd#38250 for man-sysext.conf-add-systemd-sysext-config-files sysext-introduce-global-config-file sysext-support-ImagePolicy-global-config-option Signed-off-by: Kai Lueke <[email protected]>
pothos
added a commit
to flatcar/scripts
that referenced
this pull request
Dec 18, 2025
For A/B-updated /etc contents we used a custom overlay mount that provides the default files through a lowerdir loaded from /usr. Since then we upstreamed mutable systemd-confext support and now we can switch to it. This pulls in flatcar/init#138 and flatcar/bootengine#115 together with backported systemd patches that have opened or merged upstream PRs to fix --root= issues and add a refresh skip check to prevent boot disruptions due to the multiple daemon reloads and - more important - the missing atomic remount that would mean /etc is gone for a few milliseconds during boot. The skip logic works best with verity hashes and thus the default confext must be a verity extension image. User-provided confext don't work well yet unless they use verity due to the missing atomic remount and reliance on the skipping logic. We also need to look into stacking order and other mutabiliy settings. The backported systemd patches relate to the following upstream PRs: systemd/systemd#39843 for vpick-Don-t-use-openat-directly-but-resolve-symlinks discover-image-Follow-symlinks-in-a-given-root sysext-Use-correct-image-name-for-extension-release test-Add-tests-for-handling-symlinks-with-systemd-sy Note that the patch in the PR relies on 0859fe3f32774f1e0c787974cc252ff922a1b868 but the backport patch not. systemd/systemd#39980 for sysext-Create-mutable-directory-with-the-right-mode sysext-Skip-refresh-if-no-changes-are-found systemd/systemd#39991 for sysext-Get-verity-user-certs-from-given-root systemd/systemd#40063 for sysext-Fix-config-file-support-with-root which relies on systemd/systemd#38250 for man-sysext.conf-add-systemd-sysext-config-files sysext-introduce-global-config-file sysext-support-ImagePolicy-global-config-option Signed-off-by: Kai Lueke <[email protected]>
pothos
added a commit
to flatcar/scripts
that referenced
this pull request
Dec 18, 2025
For A/B-updated /etc contents we used a custom overlay mount that provides the default files through a lowerdir loaded from /usr. Since then we upstreamed mutable systemd-confext support and now we can switch to it. This pulls in flatcar/init#138 and flatcar/bootengine#115 together with backported systemd patches that have opened or merged upstream PRs to fix --root= issues and add a refresh skip check to prevent boot disruptions due to the multiple daemon reloads and - more important - the missing atomic remount that would mean /etc is gone for a few milliseconds during boot. The skip logic works best with verity hashes and thus the default confext must be a verity extension image. User-provided confext don't work well yet unless they use verity due to the missing atomic remount and reliance on the skipping logic. We also need to look into stacking order and other mutabiliy settings. The backported systemd patches relate to the following upstream PRs: systemd/systemd#39843 for vpick-Don-t-use-openat-directly-but-resolve-symlinks discover-image-Follow-symlinks-in-a-given-root sysext-Use-correct-image-name-for-extension-release test-Add-tests-for-handling-symlinks-with-systemd-sy Note that the patch in the PR relies on 0859fe3f32774f1e0c787974cc252ff922a1b868 but the backport patch not. systemd/systemd#39980 for sysext-Create-mutable-directory-with-the-right-mode sysext-Skip-refresh-if-no-changes-are-found systemd/systemd#39991 for sysext-Get-verity-user-certs-from-given-root systemd/systemd#40063 for sysext-Fix-config-file-support-with-root which relies on systemd/systemd#38250 for man-sysext.conf-add-systemd-sysext-config-files sysext-introduce-global-config-file sysext-support-ImagePolicy-global-config-option Signed-off-by: Kai Lueke <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
So far we had a custom overlay mount for /etc that provided the A/B
updated files from /usr in a lowerdir. Since then we upstreamed a
mutable mode for sysext and confext.
We can now switch over to it and provide a default confext by using the
mutable mode. Because there is no atomic remount yet and also because
we want to avoid daemon reloads during boot, this relies on a new skip
logic in systemd-sysext/confext to only refresh in the final system boot
when changes are found. Through only using verity images we know that no
changes can be there because they get compared by hash and not mtime.
When we would hit a refresh during boot then /etc contents are shortly
gone and services sometimes fail during boot.
A bit specific to Flatcar/Ignition is that we load confext twice in the
initrd, once because we have a default confext that provides /etc
contents for Ignition and a second time for loading user confexts for
the final system (when users added new ones through Ignition).
My upstream PR is merged so that SYSTEMD_IN_INITRD=0 is not needed in the future but setting this here spares us one more backport patch.
How to use
With the scripts PR that uses it
Testing done
See scripts PR