Add support for bpf token delegation #2109

gwenya · 2025-05-12T14:16:13Z

This adds support for bpf token delegation (closes #2099).

TODOs:

better documentation, right now it's a bit barebones, but I'm not really sure what to add either
tests if the CI can support them
validation if possible: the delegation options are kernel strings and which ones are available depend on the kernel version, hardcoding the current allowed set seems like a bad idea; if validation is impossible then document this fact
improve failure mode, right now if the delegation options are wrong or the feature is not supported by the kernel the forkbpf hook fails and prevents instance startup; this is recorded in lxc.log but the specific error message only shows up when running incus with the --debug flag
kernel support check? token delegation is a pretty new feature, it's not supported yet on standard ubuntu installations, so maybe we should check if it's available and issue according error messages, and document the minimum kernel version and config
clean up the forkbpf code
restrict to projects with low_level access

irhndt

Just took a look to learn and found a typo.

cmd/incusd/main_forkbpf.go

gwenya · 2025-05-13T08:50:21Z

@stgraber would it be possible / make sense for lxc to capture stderr from hooks and log it at a higher level than debug?

gwenya · 2025-05-13T09:16:07Z

cmd/incusd/main_forkbpf.go

+}
+
+func (c *cmdForkbpf) runChild(socket int, mountPath string) error {
+	fsfd, err := unix.Fsopen("bpf", 0)


I think if this call fails, and the parent has already reached the Recvmsg() call, it will hang forever.
I'm not quite sure how to best work around this.

Maybe the parent should call both waitpid and recvmsg in goroutines and send their results through channels so it can select on them?

cyphar · 2025-05-22T17:24:44Z

cmd/incusd/main_forkbpf.go

+		// Attach to the user namespace.
+		snprintf(path, sizeof(path), "/proc/%s/ns/user", pidstr);
+		if (dosetns_file(path) < 0) {
+			_exit(EXIT_FAILURE);
+		}
+
+		// Attach to the user namespace.
+		snprintf(path, sizeof(path), "/proc/%s/ns/mnt", pidstr);
+		if (dosetns_file(path) < 0) {
+			_exit(EXIT_FAILURE);
+		}


I guess you probably want to use pidfd_nsfd+change_namespaces, as well as pidfds if possible (just like forkproxy does)?

This runs as an lxc hook, so it gets the pid as LXC_PID environment variable.
Getting an fd for the pid just to then check if the pid is still valid right afterwards seems a bit pointless, I think it only makes sense if the pidfd gets passed into the process. If lxc hooks support that we can do it, but I couldn't find any information about that and the forkproxy which also runs as a hook does it like this as well.
I don't think it would be particularly useful either. From my experiments lxc seems to wait for the hook to finish before continuing container startup, so there should be no way for the pid to get recycled in the meantime.

gwenya · 2025-06-07T14:17:00Z

@stgraber do you know if we can test this in the github runners? I have a feeling that they might not get enough privileges. Apart from that, I think this could be ready, the TODO points I listed would be nice to have of course but not mandatory I think, and I currently don't have the time to figure them out.

stgraber · 2025-06-09T18:03:24Z

@stgraber do you know if we can test this in the github runners? I have a feeling that they might not get enough privileges. Apart from that, I think this could be ready, the TODO points I listed would be nice to have of course but not mandatory I think, and I currently don't have the time to figure them out.

The tests run as root inside of the Github runner VMs, so that should be fine as far as permissions.

gwenya · 2025-06-09T21:57:03Z

Alright then I'll try to come up with some tests for this.
Do they have unrestricted internet connectivity? I will probably need to use libbpf-bootstrap and download it from git within the runner or instance.

stgraber · 2025-06-09T22:13:45Z

Yep, no internet restrictions from the runners, so you can download and build a test then push it into a test container easily enough.

stgraber · 2025-06-21T06:58:10Z

@gwenya ping

gwenya · 2025-06-21T09:42:26Z

Sadly didn’t get around to this yet, was more busy with work recently. I hope I’ll get around to it soon, but it feels like a somewhat tricky thing to figure out so it might actually take a while.
I think I should first figure out how to run the tests locally, they don’t work on my system but I should be able to do it in a VM.

stgraber · 2025-07-28T18:12:29Z

@gwenya did you have some time to look at this one again?

gwenya · 2025-10-04T10:46:22Z

Sorry for not responding in so long, I will give it a try now

stgraber · 2025-10-06T15:56:39Z

@gwenya great to have you looking into this again!

The forkbpf helper creates a socket pair and forks, the child process then enters the user and mount namespaces of the container. The child process creates a filesystem configuration context and passes it to the parent process via the socketpair. The parent process configures the token delegation options and creates a mount file descriptor and passes that back to the child. The child process then attaches the mount to the desired path. Signed-off-by: Gwendolyn <[email protected]>

The token delegation is automatically enabled if any of the delegate_* options are set. It works by configuring forkbpf as a start-host lxc hook. The delegate_* options are not properly validated since they refer to internal kernel enums and therefore depend on the running kernel. Signed-off-by: Gwendolyn <[email protected]>

Signed-off-by: Gwendolyn <[email protected]>

gwenya · 2025-10-12T10:57:52Z

@stgraber do you have an idea what might cause this test failure? I have never seen that error message ("ERROR: Output buffer space exceeded") before and google only finds me an unconfirmed bug report in the file utility...

I'm not sure if it even makes sense to build that tool in the tests, we could instead build it once and host it somewhere to download, or just wait until bpftool tags a new release that includes the bpftool token subcommands.

I also appreciate input on the other open points in the task list, I'm not coming up with any good solutions for them and am tempted to just document those shortcomings and leave them be for now.

github-actions bot added Documentation Documentation needs updating API Changes to the REST API labels May 12, 2025

gwenya force-pushed the bpf-tokens branch 6 times, most recently from 1078454 to e31bb93 Compare May 12, 2025 18:49

irhndt reviewed May 12, 2025

View reviewed changes

cmd/incusd/main_forkbpf.go Outdated Show resolved Hide resolved

gwenya force-pushed the bpf-tokens branch from 163d648 to 41a161d Compare May 13, 2025 09:13

gwenya commented May 13, 2025

View reviewed changes

gwenya force-pushed the bpf-tokens branch from 41a161d to 13737c1 Compare May 13, 2025 09:42

cyphar reviewed May 22, 2025

View reviewed changes

gwenya force-pushed the bpf-tokens branch from 13737c1 to 6d122fa Compare June 7, 2025 13:26

gwenya marked this pull request as ready for review June 7, 2025 14:17

gwenya requested a review from stgraber as a code owner June 7, 2025 14:17

stgraber marked this pull request as draft July 28, 2025 18:12

gwenya force-pushed the bpf-tokens branch 4 times, most recently from ca9ada7 to ead8313 Compare October 5, 2025 14:42

gwenya added 10 commits October 12, 2025 02:45

api: Add bpf_token_delegation extension

93a0052

Signed-off-by: Gwendolyn <[email protected]>

doc: Update configs

cdaadfa

Signed-off-by: Gwendolyn <[email protected]>

doc: Add documentation for bpf token delegation

b7a4bb3

Signed-off-by: Gwendolyn <[email protected]>

doc: Update wordlist

21f6f04

Signed-off-by: Gwendolyn <[email protected]>

incusd/project: Require lowlevel access for bpffs options

6d4f3c4

Signed-off-by: Gwendolyn <[email protected]>

tests: Add test for bpf token delegation

d138127

Signed-off-by: Gwendolyn <[email protected]>

tests: Install libelf-dev as a dependency to build bpftool

d7c9743

Signed-off-by: Gwendolyn <[email protected]>

tests: Install llvm as a dependency to build bpftool

5c227f6

Signed-off-by: Gwendolyn <[email protected]>

gwenya force-pushed the bpf-tokens branch from d55d661 to 5c227f6 Compare October 12, 2025 00:46

Uh oh!

Add support for bpf token delegation #2109

Are you sure you want to change the base?

Add support for bpf token delegation #2109

Uh oh!

Conversation

gwenya commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

irhndt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gwenya commented May 13, 2025

Uh oh!

gwenya May 13, 2025

Choose a reason for hiding this comment

Uh oh!

cyphar May 22, 2025

Choose a reason for hiding this comment

Uh oh!

gwenya May 22, 2025

Choose a reason for hiding this comment

Uh oh!

gwenya commented Jun 7, 2025

Uh oh!

stgraber commented Jun 9, 2025

Uh oh!

gwenya commented Jun 9, 2025

Uh oh!

stgraber commented Jun 9, 2025

Uh oh!

stgraber commented Jun 21, 2025

Uh oh!

gwenya commented Jun 21, 2025

Uh oh!

stgraber commented Jul 28, 2025

Uh oh!

gwenya commented Oct 4, 2025

Uh oh!

stgraber commented Oct 6, 2025

Uh oh!

gwenya commented Oct 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

gwenya commented May 12, 2025 •

edited

Loading