Container issues with Ubuntu Oracular systemd version 256-1ubuntu1 #13810

mihalicyn · 2024-07-23T16:13:42Z

Investigate and fix all the issues

TODO list:

fix unprivileged containers case (covered by Container: Allow apparmor nosymfollow mount flag in more cases #13820)
try to fix privileged containers case (we need canonical/lxd-imagebuilder@c928d22 )
land necessary fixes in LXC for both cases (LXC ships it's own AppArmor profiles) (covered by apparmor: allow nosymfollow remounts lxc/lxc#4466)
communicate with AppArmor team about possible bugs & land necessary fixes to upstream ( https://lore.kernel.org/all/[email protected]/ )

See also:
#13807
#12698
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2046486
#13844
#13860

The text was updated successfully, but these errors were encountered:

tomponline · 2024-07-24T09:34:51Z

For now the workaround is to do lxc config set <instance> security.nesting=true and restart container.

See also: canonical#12698 Closes canonical#12698 May close canonical#13810 Signed-off-by: Alexander Mikhalitsyn <[email protected]>

It turns out, that a ruleset: ``` {{- if .feature_mount_nosymfollow }} # see #12698 mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /[^spd]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /d[^e]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /de[^v]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /dev/.[^l]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /dev/.l[^x]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /dev/.lx[^c]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /dev/.lxc?*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /dev/[^.]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /dev?*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /p[^r]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /pr[^o]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /pro[^c]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /proc?*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /s[^y]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /sy[^s]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /sys?*{,/**}, {{- end }} ``` is not enough to allow nosymfollow. We still getting AppArmor denials like this: ``` [110841.647871] audit: type=1400 audit(1721910063.197:1611): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="lxd-secure-oriole_</var/snap/lxd/common/lxd>" name="/dev/shm/" pid=712867 comm="(sd-mkdcreds)" flags="ro, nosuid, nodev, noexec, remount, bind" ``` First of all, there is no "nosymfollow" in the kernel log. Which is a bug and should be fixed by: https://lore.kernel.org/all/[email protected]/ Secondly, it looks like these rules in the form of mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /some/path, just does not work at all. At least in AppArmor 4.0+ (have not yet tested with older ones). During my local experiments, I found that working variant of it might be: mount ``` options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) -> /some/path, ``` or wider: ``` mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow), ``` Let's just add a wider variant of the rule in addition to what we already have for unprivileged containers. But keep in mind that something is wrong with these rules in their more restrictive form (with path specifier). This is a matter of a futher investigation, because it's important for privileged containers case. See also: #12698 Closes #12698 May close #13810

tomponline · 2024-07-25T14:06:52Z

Reopens for privilege use case.

While investigating canonical#13810 I found that all ro+remount rules in the form: mount options=(ro,remount,bind,A,B,C) /some_pattern{,/**}, just does not work at all. This remount+bind case is a very special one, and we should rewrite all rules in this form: mount options=(ro,remount,bind,A,B,C) -> /some_pattern{,/**}, This syntax is not new. This change should be compatible with very old AppArmor versions including 2.11. Explanation why it was not noticed for years is that for unprivileged container case we have analogical rule but in a wider form: mount options=(ro,remount,bind,nodev,A,B,C), which masks the issue. But for privileged containers it's not. So, let's fix this for correctness. Signed-off-by: Alexander Mikhalitsyn <[email protected]>

While investigating #13810 I found that all ro+remount rules in the form: ``` mount options=(ro,remount,bind,A,B,C) /some_pattern{,/**}, ``` just does not work at all. This remount+bind case is a very special one, and we should rewrite all rules in this form: ``` mount options=(ro,remount,bind,A,B,C) -> /some_pattern{,/**}, ``` This syntax is not new. This change should be compatible with very old AppArmor versions including 2.11. Explanation why it was not noticed for years is that for unprivileged container case we have analogical rule but in a wider form: ``` mount options=(ro,remount,bind,nodev,A,B,C), ``` which masks the issue. But for privileged containers it's not. So, let's fix this for correctness.

mihalicyn · 2024-07-26T11:00:53Z

Ok, what we have about privileged containers:

jammy works
noble doesn't
oracular too

What we have in the audit logs:

[15669.318153] audit: type=1400 audit(1721991292.665:4813): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxd-no_</var/snap/lxd/common/lxd>" name="/run/systemd/mount-rootfs/" pid=145591 comm="(d-logind)" srcname="/" flags="rw, rbind"
[15669.327424] audit: type=1400 audit(1721991292.673:4814): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxd-no_</var/snap/lxd/common/lxd>" name="/run/systemd/mount-rootfs/" pid=145595 comm="(d-logind)" srcname="/" flags="rw, rbind"
[15669.336943] audit: type=1400 audit(1721991292.685:4815): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxd-no_</var/snap/lxd/common/lxd>" name="/run/systemd/mount-rootfs/" pid=145599 comm="(d-logind)" srcname="/" flags="rw, rbind"
[15669.346912] audit: type=1400 audit(1721991292.693:4816): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxd-no_</var/snap/lxd/common/lxd>" name="/run/systemd/mount-rootfs/" pid=145603 comm="(d-logind)" srcname="/" flags="rw, rbind"
[15669.357968] audit: type=1400 audit(1721991292.705:4817): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxd-no_</var/snap/lxd/common/lxd>" name="/run/systemd/mount-rootfs/" pid=145607 comm="(d-logind)" srcname="/" flags="rw, rbind"
[15671.673932] audit: type=1400 audit(1721991295.021:4818): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxd-or_</var/snap/lxd/common/lxd>" name="/run/systemd/mount-rootfs/" pid=145649 comm="(d-logind)" srcname="/" flags="rw, rbind"
[15671.674044] audit: type=1400 audit(1721991295.021:4819): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxd-or_</var/snap/lxd/common/lxd>" name="/dev/" pid=145651 comm="(sd-mkdcreds)" flags="rw, rslave"
[15671.683716] audit: type=1400 audit(1721991295.029:4820): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxd-or_</var/snap/lxd/common/lxd>" name="/run/systemd/mount-rootfs/" pid=145654 comm="(d-logind)" srcname="/" flags="rw, rbind"
[15671.686683] audit: type=1400 audit(1721991295.033:4821): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxd-or_</var/snap/lxd/common/lxd>" name="/dev/" pid=145658 comm="(sd-mkdcreds)" flags="rw, rslave"
[15671.692154] audit: type=1400 audit(1721991295.037:4822): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxd-or_</var/snap/lxd/common/lxd>" name="/run/systemd/mount-rootfs/" pid=145659 comm="(d-logind)" srcname="/" flags="rw, rbind"

This thing wants a recursive bind-mount of / to /run/systemd/mount-rootfs/. While it's perfectly safe for unprivileged containers, we can't allow it for privileged ones. It would be a security breach.

We have this commit in our imagebuilder canonical/lxd-imagebuilder@c928d22 and taking into account that images:debian/sid container works with security.privileged=true I would just apply the same quirk to the Ubuntu images.

tomponline · 2024-07-26T14:16:46Z

Have asked CPC about being able to apply a similar fix as canonical/lxd-imagebuilder@c928d22 to ubuntu: images.

https://chat.canonical.com/canonical/pl/dygqosfbp3gy3x7hcs5ssergoy

tomponline · 2024-07-26T17:59:27Z

@mihalicyn does this fix rely on the apparmor parser version in core24?

mihalicyn · 2024-07-26T18:21:56Z

@mihalicyn does this fix rely on the apparmor parser version in core24?

yes, but we don't need core24 itself.

Full list of changes we need to backport:

snapcraft: add apparmor part for core22 only (5.0-edge) lxd-pkg-snap#477
Instance: Allow nosymfollow mount flag for container apparmor profile #13681
Container: Allow apparmor nosymfollow mount flag in more cases #13820
(optional) Container: fix all apparmor ro+remount rules #13826

It turns out, that a ruleset: {{- if .feature_mount_nosymfollow }} # see canonical#12698 mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /[^spd]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /d[^e]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /de[^v]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /dev/.[^l]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /dev/.l[^x]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /dev/.lx[^c]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /dev/.lxc?*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /dev/[^.]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /dev?*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /p[^r]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /pr[^o]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /pro[^c]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /proc?*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /s[^y]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /sy[^s]*{,/**}, mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /sys?*{,/**}, {{- end }} is not enough to allow nosymfollow. We still getting AppArmor denials like this: [110841.647871] audit: type=1400 audit(1721910063.197:1611): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="lxd-secure-oriole_</var/snap/lxd/common/lxd>" name="/dev/shm/" pid=712867 comm="(sd-mkdcreds)" flags="ro, nosuid, nodev, noexec, remount, bind" First of all, there is no "nosymfollow" in the kernel log. Which is a bug and should be fixed by: https://lore.kernel.org/all/[email protected]/ Secondly, it looks like these rules in the form of mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) /some/path, just does not work at all. At least in AppArmor 4.0+ (have not yet tested with older ones). During my local experiments, I found that working variant of it might be: mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow) -> /some/path, or wider: mount options=(ro,remount,bind,nosuid,noexec,nodev,nosymfollow), Let's just add a wider variant of the rule in addition to what we already have for unprivileged containers. But keep in mind that something is wrong with these rules in their more restrictive form (with path specifier). This is a matter of a futher investigation, because it's important for privileged containers case. See also: canonical#12698 Closes canonical#12698 May close canonical#13810 Signed-off-by: Alexander Mikhalitsyn <[email protected]>

While investigating canonical#13810 I found that all ro+remount rules in the form: mount options=(ro,remount,bind,A,B,C) /some_pattern{,/**}, just does not work at all. This remount+bind case is a very special one, and we should rewrite all rules in this form: mount options=(ro,remount,bind,A,B,C) -> /some_pattern{,/**}, This syntax is not new. This change should be compatible with very old AppArmor versions including 2.11. Explanation why it was not noticed for years is that for unprivileged container case we have analogical rule but in a wider form: mount options=(ro,remount,bind,nodev,A,B,C), which masks the issue. But for privileged containers it's not. So, let's fix this for correctness. Signed-off-by: Alexander Mikhalitsyn <[email protected]>

tomponline · 2024-08-05T08:06:11Z

thanks @mihalicyn

I've added the vendored apparmor parser and your fixes as cherry-picks to latest/stable now.

tomponline · 2024-08-21T07:34:48Z

We are rolling out fixes to 5.21/stable for this now (snap refresh lxd --channel=5.21/stable --cohort="+" will get it by bypassing progressive rollout).