Can't get container info for PID

[jkruke]

Description

I am using the embedded Grafana Beyla of Grafana Alloy in a minikube kubernetes cluster.

After updating minikube, kubernetes and docker, Beyla cannot detect container info anymore for containers running in the same cluster.

Configuration

An excerpt of the alloy configuration file:

beyla.ebpf "default" {
    debug = true

    attributes {
        kubernetes {
            enable = "true"
        }
    }

    discovery {
        services {
            open_ports = "8080,8081,5432,9092"
            kubernetes {
                namespace = "kafka|cms"
            }
        }
    }

    output {
        traces = [otelcol.processor.filter.default.input]
    }
}
...

Log Extracts

Alloy log (beyla.ebpf.debug=true):

msg="processes matching selection criteria" component=discover.CriteriaMatcher len=0
msg="New port bind event" component=watcher.Tracer port=8080
msg="New port bound 8080" component=BPFLogger pid=11603 comm=java
msg="new process watching events" component=discover.ProcessWatcher interval=5s events="[{Type:0 Obj:{pid:3381 openPorts:[55296 12345] metadata:map[] podLab
msg="new process" component=discover.watcherKubeEnricher pid=3381
msg="Found namespace" component=httpfltr.Tracer nsPid=pid:[4026532560]
msg="can't get container info for PID" component=discover.watcherKubeEnricher pid=3381 error="/proc/3381/cgroup: couldn't find any docker entry for process

Relevant Versions

• Beyla 1.8.2 (through Grafana Alloy 1.4.3)
• Minikube 1.34.0 with Kubernetes 1.31.0 and Docker 27.2.0 (Comes through Docker Desktop 4.34.3)

Analysis

This part of Beyla code tries to get the docker container ID of a given PID by parsing the content of /proc/<PID>/cgroup:

beyla/pkg/internal/helpers/container/container.go

Line 30 in e2e9c3f

var dockerCgroup = regexp.MustCompile(`^\d+:.*:.*/.*-([\da-fA-F]+)\.scope`)

With Docker Engine 25.0.4, Beyla container ID detection still works. Here is a cgroup file content of a container ("kube-scheduler"):

0::/../../../kubepods-burstable.slice/kubepods-burstable-pode039200acb850c82bb901653cc38ff6e.slice/docker-864416f734742561b115581034682fd8555c000c7ea7a90c530b282a1b8d2e34.scope

With Docker Engine 27.3.1, Beyla cannot get container info anymore (see log extracts, msg="can't get container info for PID" component=discover.watcherKubeEnricher pid=3381 error="/proc/3381/cgroup: couldn't find any docker entry for process).
When I inspect the cgroup file of the kube-scheduler container again, it looks a bit different:

0::/../../pode039200acb850c82bb901653cc38ff6e/58452031ab6dcaa4fe3ff91f8a46fd41a4a2405586f3cf3d5cb9c93b5bcf62cc

This matches to none of the both regexes in container.go.

Suggested Solution

I assume, the first regex for dockerCgroup (^\d+:.*:.*/.*-([\da-fA-F]+)\.scope) should be modified because the cgroup does not necessarily end with .scope anymore.

[mariomac]

Hi @jkruke thank you for the bug report! We are fixing it in: #1287

When the linked PR is merged, the fix will be immediately available in our unstable development image: grafana/beyla:main, and will be part of our next stable release.

[jkruke]

I can verify that it works now with grafana/beyla:main.

Now, unfortunately, I got another issue.

time=2024-10-31T11:41:32.795Z level=DEBUG msg=Instrumentable component=discover.TraceAttacher created=0 type=java exec=/usr/lib/jvm/zulu17-ca-arm64/bin/java pid=270402
time=2024-10-31T11:41:32.795Z level=INFO msg="instrumenting process" component=discover.TraceAttacher cmd=/usr/lib/jvm/zulu17-ca-arm64/bin/java pid=270402 ino=1315976 type=java
time=2024-10-31T11:41:32.795Z level=DEBUG msg="loading eBPF program" component=ebpf.ProcessTracer program=*generictracer.Tracer type=1
time=2024-10-31T11:41:32.795Z level=ERROR msg="couldn't trace process. Stopping process tracer" component=discover.TraceAttacher error="loading and assigning BPF objects: loading eBPF program: can't load bpf:
 bad magic number '[118 101 114 115]' in record at byte 0x0"

But I gonna address this in a separate issue or at first just in the beyla slack channel, because this probably hasn't got anything to do with this fix.

Thanks for your quick action!

[mariomac]

Thanks for reporting @jkruke we are working on a fix.

[mariomac]

It's fixed now.