Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split eBPF load and attach for Go programs #1169

Merged
merged 26 commits into from
Sep 19, 2024

Conversation

grcevski
Copy link
Contributor

@grcevski grcevski commented Sep 14, 2024

This PR changes how we load and attach BPF programs for a few reasons:

  1. Removing the need for the BPF file system, e.g. running a single tracer.
  2. Reducing kernel memory allocation. A single tracer can be attached to multiple executables.

Our previous approach was as follows:

  • Find unique executable to instrument based on the inode number.
  • Make a tracer for each executable
  • Load the tracer and attach the probes to the executable.
  • Repeat this for every new executable found.

The new approach does:

  • First time we load a Go program we load a single Go reusable tracer. We only patch constants in the eBPF which are independent of the executable, like ring buffer size.
  • Link the executable and add offsets table for every newly discovered Go program.
  • Remember a map of the closables per executable, close them instead of unloading the eBPF program when there are no instances of an executable left.

Further work is required to make the same thing happen for the user probe extensions for libssl and nodejs.

Closes #1158

@grcevski grcevski added do-not-merge WIP or something that musn't be merged wip labels Sep 14, 2024
@codecov-commenter
Copy link

codecov-commenter commented Sep 14, 2024

Codecov Report

Attention: Patch coverage is 83.41709% with 33 lines in your changes missing coverage. Please review.

Project coverage is 81.15%. Comparing base (008f4f2) to head (726c3a9).

Files with missing lines Patch % Lines
pkg/internal/discover/attacher.go 68.62% 10 Missing and 6 partials ⚠️
pkg/internal/ebpf/tracer_linux.go 69.04% 10 Missing and 3 partials ⚠️
pkg/internal/discover/attacher_linux.go 77.77% 1 Missing and 1 partial ⚠️
pkg/internal/ebpf/gotracer/gotracer.go 96.07% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1169      +/-   ##
==========================================
+ Coverage   76.83%   81.15%   +4.31%     
==========================================
  Files         136      136              
  Lines       11451    11497      +46     
==========================================
+ Hits         8798     9330     +532     
+ Misses       2121     1638     -483     
+ Partials      532      529       -3     
Flag Coverage Δ
integration-test 57.21% <79.39%> (+0.26%) ⬆️
k8s-integration-test 58.87% <81.40%> (?)
oats-test 35.82% <69.84%> (-0.05%) ⬇️
unittests 53.36% <2.01%> (-0.19%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@rafaelroquetto rafaelroquetto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

ta.log.Warn("can't unmount pinned root. Try unmounting and removing it manually", "error", err)
return
func UnmountBPFFS(pinPath string, log *slog.Logger) {
if err := unix.Unmount(pinPath, unix.MNT_FORCE); err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the use-case behind MNT_FORCE? I wonder if we should be using MNT_DETACH instead...

@grcevski grcevski changed the title WIP: Split eBPF load and attach Split eBPF load and attach for Go programs Sep 18, 2024
@grcevski grcevski removed do-not-merge WIP or something that musn't be merged wip labels Sep 18, 2024
Copy link
Contributor

@mariomac mariomac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@@ -9,5 +9,4 @@ import (
var (
pathRoot = tools.ProjectDir()
pathOutput = path.Join(pathRoot, "testoutput")
pathVarRun = path.Join(pathOutput, "run")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why has this become a problem? Not sure if related, but I noticed a few of my PRs are failing a few tests as well

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we'll be removing the BPF file system dependency, I removed all the tracking support for it. Timing is different with the new BPF cleanup and I think we don't always clean it up on time in tests.

@grcevski grcevski merged commit 9a23f73 into grafana:main Sep 19, 2024
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Attach executables to the same Go tracer.
4 participants