Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Beyla sometimes crashing on startup #1409

Open
mariomac opened this issue Nov 25, 2024 · 0 comments
Open

Beyla sometimes crashing on startup #1409

mariomac opened this issue Nov 25, 2024 · 0 comments

Comments

@mariomac
Copy link
Contributor

mariomac commented Nov 25, 2024 •
edited
Loading

There is a concurrent map/write somewhere. It only happens unfrequently (1%) during startup, in instances running in big clusters.

fatal error: concurrent map writes

goroutine 41 [running]:
maps.DeleteFunc[...](...)
	/usr/local/go/src/maps/maps.go:72
github.com/grafana/beyla/pkg/export/attributes.Selection.Normalize(0xc000018600)
	/opt/app-root/pkg/export/attributes/attr_select.go:70 +0x1f6
github.com/grafana/beyla/pkg/export/attributes.NewAttrSelector(0x7, 0xc000018600)
	/opt/app-root/pkg/export/attributes/attr_selector.go:38 +0x25
github.com/grafana/beyla/pkg/export/prom.newReporter({0x280c230, 0xc0005a4f00}, 0xc000068040, 0xc0004ef578, 0x0?)
	/opt/app-root/pkg/export/prom/prom.go:233 +0x51
github.com/grafana/beyla/pkg/internal/pipe.newGraphBuilder.PrometheusEndpoint.func6()
	/opt/app-root/pkg/export/prom/prom.go:212 +0x6b
reflect.Value.call({0x1fc9260?, 0xc002c82c00?, 0xc000d0a750?}, {0x2371c8f, 0x4}, {0x0, 0x0, 0x16?})
	/usr/local/go/src/reflect/value.go:581 +0xca6
reflect.Value.Call({0x1fc9260?, 0xc002c82c00?, 0x40f8b0?}, {0x0?, 0x0?, 0xc0000e61c0?})
	/usr/local/go/src/reflect/value.go:365 +0xb9
github.com/mariomac/pipes/pipe.(*reflectProvider).call(0xc002cb0300, {0x1fb5580, 0xc0026b8000})
	/opt/app-root/vendor/github.com/mariomac/pipes/pipe/builder.go:71 +0x53
github.com/mariomac/pipes/pipe.(*Builder[...]).Build(0x27f2fa0)
	/opt/app-root/vendor/github.com/mariomac/pipes/pipe/builder.go:132 +0x445
github.com/grafana/beyla/pkg/internal/pipe.(*graphFunctions).buildGraph(0xc001705b80)
	/opt/app-root/pkg/internal/pipe/instrumenter.go:137 +0x2a
github.com/grafana/beyla/pkg/internal/pipe.Build({0x280c230?, 0xc0005a4f00?}, 0x407f500?, 0xfffffffffffffffc?, 0x23b0cb7?)
	/opt/app-root/pkg/internal/pipe/instrumenter.go:86 +0x25
github.com/grafana/beyla/pkg/internal/appolly.(*Instrumenter).ReadAndForward(0xc001302db0)
	/opt/app-root/pkg/internal/appolly/appolly.go:107 +0x75
github.com/grafana/beyla/pkg/components.setupAppO11y({0x280c230, 0xc0005a4f00}, 0xc000068040, 0xc0004ef108)
	/opt/app-root/pkg/components/beyla.go:71 +0x16d
github.com/grafana/beyla/pkg/components.RunBeyla.func1()
	/opt/app-root/pkg/components/beyla.go:39 +0x65
created by github.com/grafana/beyla/pkg/components.RunBeyla in goroutine 1
	/opt/app-root/pkg/components/beyla.go:37 +0x146

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc0005a5040?)
	/usr/local/go/src/runtime/sema.go:71 +0x25
sync.(*WaitGroup).Wait(0x280c230?)
	/usr/local/go/src/sync/waitgroup.go:118 +0x48
github.com/grafana/beyla/pkg/components.RunBeyla({0x280c230, 0xc0005a4f00}, 0xc0004ef108)
	/opt/app-root/pkg/components/beyla.go:52 +0x1f0
main.main()
	/opt/app-root/cmd/beyla/main.go:77 +0x7c5

goroutine 133 [chan receive]:
github.com/grafana/beyla/pkg/internal/ebpf/common.(*ringBufForwarder).bgFlushOnTimeout(0xc001889050, 0x0)
	/opt/app-root/pkg/internal/ebpf/common/ringbuf.go:208 +0x38
created by github.com/grafana/beyla/pkg/internal/ebpf/common.(*ringBufForwarder).readAndForwardInner in goroutine 120
	/opt/app-root/pkg/internal/ebpf/common/ringbuf.go:136 +0xdd

goroutine 37 [select]:
github.com/hashicorp/golang-lru/v2/expirable.NewLRU[...].func1()
	/opt/app-root/vendor/github.com/hashicorp/golang-lru/v2/expirable/expirable_lru.go:86 +0xf5
created by github.com/hashicorp/golang-lru/v2/expirable.NewLRU[...] in goroutine 1
	/opt/app-root/vendor/github.com/hashicorp/golang-lru/v2/expirable/expirable_lru.go:82 +0x32a

goroutine 38 [IO wait]:
internal/poll.runtime_pollWait(0x7fa7dc5f3e50, 0x72)
	/usr/local/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc000228b00?, 0x2c?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000228b00)
	/usr/local/go/src/internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc000228b00)
	/usr/local/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc0001b7500)
	/usr/local/go/src/net/tcpsock_posix.go:159 +0x1e
net.(*TCPListener).Accept(0xc0001b7500)
	/usr/local/go/src/net/tcpsock.go:372 +0x30
net/http.(*Server).Serve(0xc000178000, {0x27fbcb0, 0xc0001b7500})
	/usr/local/go/src/net/http/server.go:3330 +0x30c
net/http.(*Server).ListenAndServe(0xc000178000)
	/usr/local/go/src/net/http/server.go:3259 +0x71
net/http.ListenAndServe(...)
	/usr/local/go/src/net/http/server.go:3514
main.main.func1()
	/opt/app-root/cmd/beyla/main.go:67 +0x14e
created by main.main in goroutine 1
	/opt/app-root/cmd/beyla/main.go:65 +0x746

goroutine 40 [syscall]:
os/signal.signal_recv()
	/usr/local/go/src/runtime/sigqueue.go:152 +0x29
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 1
	/usr/local/go/src/os/signal/signal.go:151 +0x1f

goroutine 49 [select]:
os/signal.NotifyContext.func1()
	/usr/local/go/src/os/signal/signal.go:288 +0x66
created by os/signal.NotifyContext in goroutine 1
	/usr/local/go/src/os/signal/signal.go:287 +0x156

goroutine 42 [runnable]:
github.com/prometheus/client_golang/prometheus.v2.NewDesc({}, {0x23a76d8, 0x1e}, {0x241b76e, 0x50}, {0x27f1620?, 0xc0017b64c8?}, 0x0)
	/opt/app-root/vendor/github.com/prometheus/client_golang/prometheus/desc.go:96 +0x5e
github.com/prometheus/client_golang/prometheus.v2.NewCounterVec({}, {{{0x0, 0x0}, {0x0, 0x0}, {0x23a76d8, 0x1e}, {0x241b76e, 0x50}, 0x0, ...}, ...})
	/opt/app-root/vendor/github.com/prometheus/client_golang/prometheus/counter.go:203 +0x72
github.com/prometheus/client_golang/prometheus.NewCounterVec(...)
	/opt/app-root/vendor/github.com/prometheus/client_golang/prometheus/counter.go:195
github.com/grafana/beyla/pkg/export/prom.newNetReporter({0x280c230, 0xc0005a4f00}, 0xc000068040, 0xc002413600)
	/opt/app-root/pkg/export/prom/prom_net.go:96 +0x3b8
github.com/grafana/beyla/pkg/export/prom.NetPrometheusEndpoint({0x280c230?, 0xc0005a4f00?}, 0xc000068040?, 0xc002413600)
	/opt/app-root/pkg/export/prom/prom_net.go:53 +0x78
github.com/grafana/beyla/pkg/internal/netolly/agent.(*Flows).pipelineBuilder.func7()
	/opt/app-root/pkg/internal/netolly/agent/pipeline.go:143 +0x5f
reflect.Value.call({0x1fcda80?, 0xc00133b260?, 0xc000d98270?}, {0x2371c8f, 0x4}, {0x0, 0x0, 0x1e?})
	/usr/local/go/src/reflect/value.go:581 +0xca6
reflect.Value.Call({0x1fcda80?, 0xc00133b260?, 0x40f8b0?}, {0x0?, 0xd0?, 0xc0000e6540?})
	/usr/local/go/src/reflect/value.go:365 +0xb9
github.com/mariomac/pipes/pipe.(*reflectProvider).call(0xc000f7c960, {0x1fb00c0, 0xc000e9ba40})
	/opt/app-root/vendor/github.com/mariomac/pipes/pipe/builder.go:71 +0x53
github.com/mariomac/pipes/pipe.(*Builder[...]).Build(0x27eff00)
	/opt/app-root/vendor/github.com/mariomac/pipes/pipe/builder.go:132 +0x445
github.com/grafana/beyla/pkg/internal/netolly/agent.(*Flows).buildPipeline(0xc001804480?, {0x280c230?, 0xc0005a4f00?})
	/opt/app-root/pkg/internal/netolly/agent/pipeline.go:75 +0x39
github.com/grafana/beyla/pkg/internal/netolly/agent.(*Flows).Run(0xc001737520, {0x280c230, 0xc0005a4f00})
	/opt/app-root/pkg/internal/netolly/agent/agent.go:227 +0x98
github.com/grafana/beyla/pkg/components.setupNetO11y({0x280c230, 0xc0005a4f00}, 0xc000068040, 0xc0004ef108)
	/opt/app-root/pkg/components/beyla.go:87 +0xee
github.com/grafana/beyla/pkg/components.RunBeyla.func2()
	/opt/app-root/pkg/components/beyla.go:47 +0x65
created by github.com/grafana/beyla/pkg/components.RunBeyla in goroutine 1
	/opt/app-root/pkg/components/beyla.go:45 +0x1e6

goroutine 70 [select]:
google.golang.org/grpc.newClientStreamWithParams.func4()
	/opt/app-root/vendor/google.golang.org/grpc/stream.go:393 +0x8c
created by google.golang.org/grpc.newClientStreamWithParams in goroutine 27
	/opt/app-root/vendor/google.golang.org/grpc/stream.go:392 +0xde9

goroutine 69 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0005a5340, 0x1)
	/opt/app-root/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x108
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0007d6500)
	/opt/app-root/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x7b
google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
	/opt/app-root/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0xd2
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 43
	/opt/app-root/vendor/google.golang.org/grpc/internal/transport/http2_client.go:468 +0x241b

goroutine 27 [select]:
google.golang.org/grpc/internal/transport.(*recvBufferReader).readHeaderClient(0xc00028a6e0, {0xc0000e0fd0, 0x5, 0x5})
	/opt/app-root/vendor/google.golang.org/grpc/internal/transport/transport.go:199 +0xa7
google.golang.org/grpc/internal/transport.(*recvBufferReader).ReadHeader(0xc00028a6e0, {0xc0000e0fd0?, 0xc0007d8248?, 0xc0004e7440?})
	/opt/app-root/vendor/google.golang.org/grpc/internal/transport/transport.go:145 +0x45
google.golang.org/grpc/internal/transport.(*transportReader).ReadHeader(0xc000288460, {0xc0000e0fd0?, 0x848?, 0x40?})
	/opt/app-root/vendor/google.golang.org/grpc/internal/transport/transport.go:614 +0x25
google.golang.org/grpc/internal/transport.(*Stream).ReadHeader(0xc0004e7440, {0xc0000e0fd0, 0x5, 0x5})
	/opt/app-root/vendor/google.golang.org/grpc/internal/transport/transport.go:557 +0xa6
google.golang.org/grpc.(*parser).recvMsg(0xc0000e0fc0, 0x400000)
	/opt/app-root/vendor/google.golang.org/grpc/rpc_util.go:659 +0x3e
google.golang.org/grpc.recvAndDecompress(0xc0000e0fc0, 0xc0004e7440, {0x0, 0x0}, 0x400000, 0x0, {0x0, 0x0}, 0x0)
	/opt/app-root/vendor/google.golang.org/grpc/rpc_util.go:823 +0x9a
google.golang.org/grpc.recv(0xc00139cc18?, {0x7fa7dc6a1800, 0x407f500}, 0x7?, {0x0?, 0x0?}, {0x2189e60, 0xc0021e6000}, 0xc0006802b8?, 0x0, ...)
	/opt/app-root/vendor/google.golang.org/grpc/rpc_util.go:916 +0x91
google.golang.org/grpc.(*csAttempt).recvMsg(0xc0002904e0, {0x2189e60, 0xc0021e6000}, 0x79d538?)
	/opt/app-root/vendor/google.golang.org/grpc/stream.go:1129 +0x2e5
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x10?)
	/opt/app-root/vendor/google.golang.org/grpc/stream.go:969 +0x1f
google.golang.org/grpc.(*clientStream).withRetry(0xc00069de60, 0xc0006bfdd8, 0xc0006bfdc8)
	/opt/app-root/vendor/google.golang.org/grpc/stream.go:773 +0x3ae
google.golang.org/grpc.(*clientStream).RecvMsg(0xc00069de60, {0x2189e60?, 0xc0021e6000?})
	/opt/app-root/vendor/google.golang.org/grpc/stream.go:968 +0x155
google.golang.org/grpc.(*GenericClientStream[...]).Recv(0x2819f20)
	/opt/app-root/vendor/google.golang.org/grpc/stream_interfaces.go:185 +0x4e
github.com/grafana/beyla/pkg/internal/kube.(*cacheSvcClient).connect(0xc0005a2bd0, {0x280c230, 0xc0005a4f00})
	/opt/app-root/pkg/internal/kube/cache_svc_client.go:84 +0x24b
github.com/grafana/beyla/pkg/internal/kube.(*cacheSvcClient).Start.func1()
	/opt/app-root/pkg/internal/kube/cache_svc_client.go:55 +0x18f
created by github.com/grafana/beyla/pkg/internal/kube.(*cacheSvcClient).Start in goroutine 41
	/opt/app-root/pkg/internal/kube/cache_svc_client.go:37 +0x133

goroutine 28 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0000506b0, {0x280bfe8, 0xc00021a410})
	/opt/app-root/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0x10c
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 27
	/opt/app-root/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x11a

goroutine 29 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0000506e0, {0x280bfe8, 0xc00021a460})
	/opt/app-root/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0x10c
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 27
	/opt/app-root/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x11a

goroutine 30 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000050710, {0x280bfe8, 0xc00021a4b0})
	/opt/app-root/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0x10c
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 27
	/opt/app-root/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x11a

goroutine 31 [select]:
google.golang.org/grpc/internal/resolver/dns.(*dnsResolver).watcher(0xc000228f00)
	/opt/app-root/vendor/google.golang.org/grpc/internal/resolver/dns/dns_resolver.go:220 +0x254
created by google.golang.org/grpc/internal/resolver/dns.(*dnsBuilder).Build in goroutine 29
	/opt/app-root/vendor/google.golang.org/grpc/internal/resolver/dns/dns_resolver.go:149 +0x3d9

goroutine 9 [chan receive]:
github.com/grafana/beyla/pkg/internal/netolly/ifaces.(*Watcher).sendUpdates(0xc000778960, {0x280c230, 0xc0005a4f00}, 0xc0005a2a10)
	/opt/app-root/pkg/internal/netolly/ifaces/watcher.go:80 +0x31d
created by github.com/grafana/beyla/pkg/internal/netolly/ifaces.(*Watcher).Subscribe in goroutine 42
	/opt/app-root/pkg/internal/netolly/ifaces/watcher.go:54 +0xa5

goroutine 45 [IO wait]:
internal/poll.runtime_pollWait(0x7fa7dc5f3c20, 0x72)
	/usr/local/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc000229200?, 0xc0007de000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000229200, {0xc0007de000, 0x8000, 0x8000})
	/usr/local/go/src/internal/poll/fd_unix.go:165 +0x27a
net.(*netFD).Read(0xc000229200, {0xc0007de000?, 0x1060100000000?, 0x8?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc00078f760, {0xc0007de000?, 0x800010601?, 0x0?})
	/usr/local/go/src/net/net.go:189 +0x45
bufio.(*Reader).Read(0xc0008009c0, {0xc000220ba0, 0x9, 0xc000090708?})
	/usr/local/go/src/bufio/bufio.go:241 +0x197
io.ReadAtLeast({0x27e5de0, 0xc0008009c0}, {0xc000220ba0, 0x9, 0x9}, 0x9)
	/usr/local/go/src/io/io.go:335 +0x90
io.ReadFull(...)
	/usr/local/go/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc000220ba0, 0x9, 0xc0005de9a8?}, {0x27e5de0?, 0xc0008009c0?})
	/opt/app-root/vendor/golang.org/x/net/http2/frame.go:237 +0x65
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000220b60)
	/opt/app-root/vendor/golang.org/x/net/http2/frame.go:501 +0x85
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0007d8248, 0xc0004f19d0)
	/opt/app-root/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1645 +0x1c6
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 43
	/opt/app-root/vendor/google.golang.org/grpc/internal/transport/http2_client.go:412 +0x1e39

goroutine 10 [chan receive]:
github.com/grafana/beyla/pkg/internal/netolly/ifaces.(*Registerer).Subscribe.func1()
	/opt/app-root/pkg/internal/netolly/ifaces/registerer.go:51 +0x78
created by github.com/grafana/beyla/pkg/internal/netolly/ifaces.(*Registerer).Subscribe in goroutine 42
	/opt/app-root/pkg/internal/netolly/ifaces/registerer.go:50 +0xba

goroutine 11 [select]:
github.com/grafana/beyla/pkg/internal/netolly/agent.(*Flows).interfacesManager.func1()
	/opt/app-root/pkg/internal/netolly/agent/agent.go:270 +0xcf
created by github.com/grafana/beyla/pkg/internal/netolly/agent.(*Flows).interfacesManager in goroutine 42
	/opt/app-root/pkg/internal/netolly/agent/agent.go:268 +0x1a8

goroutine 12 [chan receive]:
github.com/vishvananda/netlink.linkSubscribeAt.func1()
	/opt/app-root/vendor/github.com/vishvananda/netlink/link_linux.go:1868 +0x25
created by github.com/vishvananda/netlink.linkSubscribeAt in goroutine 9
	/opt/app-root/vendor/github.com/vishvananda/netlink/link_linux.go:1867 +0xcf

goroutine 13 [syscall]:
syscall.Syscall6(0x2d, 0x10, 0xc00238fea8, 0x10000, 0x0, 0xc000612150, 0xc00238fe5c)
	/usr/local/go/src/syscall/syscall_linux.go:95 +0x39
golang.org/x/sys/unix.recvfrom(0x0?, {0xc00238fea8?, 0x0?, 0x0?}, 0x0?, 0x0?, 0x0?)
	/opt/app-root/vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:527 +0x6f
golang.org/x/sys/unix.Recvfrom(0x10, {0xc00238fea8, 0x10000, 0x10000}, 0x0)
	/opt/app-root/vendor/golang.org/x/sys/unix/syscall_unix.go:341 +0x70
github.com/vishvananda/netlink/nl.(*NetlinkSocket).Receive(0xc001844240?)
	/opt/app-root/vendor/github.com/vishvananda/netlink/nl/nl_linux.go:632 +0x59
github.com/vishvananda/netlink.linkSubscribeAt.func2()
	/opt/app-root/vendor/github.com/vishvananda/netlink/link_linux.go:1884 +0x85
created by github.com/vishvananda/netlink.linkSubscribeAt in goroutine 9
	/opt/app-root/vendor/github.com/vishvananda/netlink/link_linux.go:1881 +0x2df

goroutine 83 [select]:
net/http/pprof.sleep(0xc00003c280, 0xc0016508c0?)
	/usr/local/go/src/net/http/pprof/pprof.go:117 +0x91
net/http/pprof.Profile({0x27fbbc0, 0xc0016508c0}, 0xc00003c280)
	/usr/local/go/src/net/http/pprof/pprof.go:163 +0x33d
net/http.HandlerFunc.ServeHTTP(0x40521c0?, {0x27fbbc0?, 0xc0016508c0?}, 0x755a16?)
	/usr/local/go/src/net/http/server.go:2220 +0x29
net/http.(*ServeMux).ServeHTTP(0x46f399?, {0x27fbbc0, 0xc0016508c0}, 0xc00003c280)
	/usr/local/go/src/net/http/server.go:2747 +0x1ca
net/http.serverHandler.ServeHTTP({0xc0007a3530?}, {0x27fbbc0?, 0xc0016508c0?}, 0x6?)
	/usr/local/go/src/net/http/server.go:3210 +0x8e
net/http.(*conn).serve(0xc001888fc0, {0x280bfb0, 0xc000122240})
	/usr/local/go/src/net/http/server.go:2092 +0x5d0
created by net/http.(*Server).Serve in goroutine 38
	/usr/local/go/src/net/http/server.go:3360 +0x485

goroutine 14 [IO wait]:
internal/poll.runtime_pollWait(0x7fa7dc5f39f0, 0x72)
	/usr/local/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc002704b80?, 0xc0007a3541?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc002704b80, {0xc0007a3541, 0x1, 0x1})
	/usr/local/go/src/internal/poll/fd_unix.go:165 +0x27a
net.(*netFD).Read(0xc002704b80, {0xc0007a3541?, 0xc00028a730?, 0x10000c0007c17d0?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc00078e020, {0xc0007a3541?, 0x964372?, 0xc0007d6500?})
	/usr/local/go/src/net/net.go:189 +0x45
net/http.(*connReader).backgroundRead(0xc0007a3530)
	/usr/local/go/src/net/http/server.go:690 +0x37
created by net/http.(*connReader).startBackgroundRead in goroutine 83
	/usr/local/go/src/net/http/server.go:686 +0xb6

goroutine 15 [syscall]:
runtime/pprof.readProfile()
	/usr/local/go/src/runtime/cpuprof.go:251 +0x4b
runtime/pprof.profileWriter({0x27e6240?, 0xc0016508c0?})
	/usr/local/go/src/runtime/pprof/pprof.go:869 +0x4f
created by runtime/pprof.StartCPUProfile in goroutine 83
	/usr/local/go/src/runtime/pprof/pprof.go:853 +0x136

goroutine 95 [IO wait]:
internal/poll.runtime_pollWait(0x7fa7dc5f3590, 0x72)
	/usr/local/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc002b4e480?, 0xc002e12000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc002b4e480, {0xc002e12000, 0x800, 0x800})
	/usr/local/go/src/internal/poll/fd_unix.go:165 +0x27a
net.(*netFD).Read(0xc002b4e480, {0xc002e12000?, 0x401fda0?, 0xc0027669a0?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc00078e0f8, {0xc002e12000?, 0xc0018c62a0?, 0xc002e12048?})
	/usr/local/go/src/net/net.go:189 +0x45
crypto/tls.(*atLeastReader).Read(0xc0001b46a8, {0xc002e12000?, 0x0?, 0xc0001b46a8?})
	/usr/local/go/src/crypto/tls/conn.go:809 +0x3b
bytes.(*Buffer).ReadFrom(0xc0027702b8, {0x27e6960, 0xc0001b46a8})
	/usr/local/go/src/bytes/buffer.go:211 +0x98
crypto/tls.(*Conn).readFromUntil(0xc002770008, {0x27e6be0, 0xc00078e0f8}, 0xc002766a10?)
	/usr/local/go/src/crypto/tls/conn.go:831 +0xde
crypto/tls.(*Conn).readRecordOrCCS(0xc002770008, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:629 +0x3cf
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:591
crypto/tls.(*Conn).Read(0xc002770008, {0xc002e1d000, 0x1000, 0xc00174fc00?})
	/usr/local/go/src/crypto/tls/conn.go:1385 +0x150
bufio.(*Reader).Read(0xc002e18ba0, {0xc0010a7700, 0x9, 0x4006d40?})
	/usr/local/go/src/bufio/bufio.go:241 +0x197
io.ReadAtLeast({0x27e5de0, 0xc002e18ba0}, {0xc0010a7700, 0x9, 0x9}, 0x9)
	/usr/local/go/src/io/io.go:335 +0x90
io.ReadFull(...)
	/usr/local/go/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc0010a7700, 0x9, 0x479fe5?}, {0x27e5de0?, 0xc002e18ba0?})
	/opt/app-root/vendor/golang.org/x/net/http2/frame.go:237 +0x65
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0010a76c0)
	/opt/app-root/vendor/golang.org/x/net/http2/frame.go:501 +0x85
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc002766fa8)
	/opt/app-root/vendor/golang.org/x/net/http2/transport.go:2319 +0xda
golang.org/x/net/http2.(*ClientConn).readLoop(0xc0017629c0)
	/opt/app-root/vendor/golang.org/x/net/http2/transport.go:2215 +0x7c
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 94
	/opt/app-root/vendor/golang.org/x/net/http2/transport.go:830 +0xd5b

goroutine 132 [chan receive (nil chan)]:
github.com/grafana/beyla/pkg/internal/ebpf/common.(*ringBufForwarder).bgListenContextCancelation(0xc001889050, {0x280bf78?, 0x407f500?}, {0x27f17d0, 0xc0001f0380})
	/opt/app-root/pkg/internal/ebpf/common/ringbuf.go:219 +0x3b
created by github.com/grafana/beyla/pkg/internal/ebpf/common.(*ringBufForwarder).readAndForward in goroutine 120
	/opt/app-root/pkg/internal/ebpf/common/ringbuf.go:128 +0x2fd

goroutine 46 [chan receive]:
github.com/grafana/beyla/pkg/internal/discover.(*TraceAttacher).attacherLoop.func1(0xc0005e23f0)
	/opt/app-root/pkg/internal/discover/attacher.go:67 +0x5a
github.com/mariomac/pipes/pipe.(*terminal[...]).start.func1()
	/opt/app-root/vendor/github.com/mariomac/pipes/pipe/node.go:206 +0x2a
created by github.com/mariomac/pipes/pipe.(*terminal[...]).start in goroutine 41
	/opt/app-root/vendor/github.com/mariomac/pipes/pipe/node.go:205 +0x71

goroutine 47 [chan receive]:
github.com/grafana/beyla/pkg/internal/discover.updateLoop.func1(0xc0005e22a0, 0xc0005e23f0)
	/opt/app-root/pkg/internal/discover/container_updater.go:33 +0x9a
github.com/mariomac/pipes/pipe.(*middle[...]).start.func1()
	/opt/app-root/vendor/github.com/mariomac/pipes/pipe/node.go:195 +0x3b
created by github.com/mariomac/pipes/pipe.(*middle[...]).start in goroutine 41
	/opt/app-root/vendor/github.com/mariomac/pipes/pipe/node.go:194 +0x276

goroutine 48 [chan receive]:
github.com/grafana/beyla/pkg/internal/discover.ExecTyperProvider.func1.1(0xc0005e2230, 0xc0005e22a0)
	/opt/app-root/pkg/internal/discover/typer.go:44 +0x8b
github.com/mariomac/pipes/pipe.(*middle[...]).start.func1()
	/opt/app-root/vendor/github.com/mariomac/pipes/pipe/node.go:195 +0x3b
created by github.com/mariomac/pipes/pipe.(*middle[...]).start in goroutine 41
	/opt/app-root/vendor/github.com/mariomac/pipes/pipe/node.go:194 +0x276

goroutine 113 [chan receive]:
github.com/grafana/beyla/pkg/internal/discover.(*matcher).run(0xc0022d3f00, 0xc0005e2380, 0xc0005e2230)
	/opt/app-root/pkg/internal/discover/matcher.go:49 +0x92
github.com/mariomac/pipes/pipe.(*middle[...]).start.func1()
	/opt/app-root/vendor/github.com/mariomac/pipes/pipe/node.go:195 +0x3b
created by github.com/mariomac/pipes/pipe.(*middle[...]).start in goroutine 41
	/opt/app-root/vendor/github.com/mariomac/pipes/pipe/node.go:194 +0x276

goroutine 114 [select]:
github.com/grafana/beyla/pkg/internal/discover.(*watcherKubeEnricher).enrich(0xc0022d3e80, 0xc0005e2310, 0xc0005e2380)
	/opt/app-root/pkg/internal/discover/watcher_kube.go:103 +0x152
github.com/mariomac/pipes/pipe.(*middle[...]).start.func1()
	/opt/app-root/vendor/github.com/mariomac/pipes/pipe/node.go:195 +0x3b
created by github.com/mariomac/pipes/pipe.(*middle[...]).start in goroutine 41
	/opt/app-root/vendor/github.com/mariomac/pipes/pipe/node.go:194 +0x276

goroutine 115 [select]:
github.com/grafana/beyla/pkg/internal/discover.(*pollAccounter).Run(0xc002645980, 0xc0005e2310)
	/opt/app-root/pkg/internal/discover/watcher_proc.go:134 +0x4db
github.com/mariomac/pipes/pipe.(*start[...]).Start.func1()
	/opt/app-root/vendor/github.com/mariomac/pipes/pipe/node.go:176 +0x37
created by github.com/mariomac/pipes/pipe.(*start[...]).Start in goroutine 41
	/opt/app-root/vendor/github.com/mariomac/pipes/pipe/node.go:175 +0x9c

goroutine 116 [select]:
github.com/grafana/beyla/pkg/internal/appolly.(*Instrumenter).FindAndInstrument.func1()
	/opt/app-root/pkg/internal/appolly/appolly.go:65 +0x19e
created by github.com/grafana/beyla/pkg/internal/appolly.(*Instrumenter).FindAndInstrument in goroutine 41
	/opt/app-root/pkg/internal/appolly/appolly.go:56 +0x131

goroutine 120 [syscall]:
syscall.Syscall6(0xe8, 0x17, 0xc003858430, 0x1, 0xffffffffffffffff, 0x0, 0x0)
	/usr/local/go/src/syscall/syscall_linux.go:95 +0x39
golang.org/x/sys/unix.EpollWait(0x4012e20?, {0xc003858430?, 0x415ba5?, 0x541?}, 0x70?)
	/opt/app-root/vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:55 +0x4f
github.com/cilium/ebpf/internal/unix.EpollWait(...)
	/opt/app-root/vendor/github.com/cilium/ebpf/internal/unix/types_linux.go:131
github.com/cilium/ebpf/internal/epoll.(*Poller).Wait(0xc002e16000, {0xc003858430, 0x1, 0x1}, {0x47055d?, 0x68?, 0x0?})
	/opt/app-root/vendor/github.com/cilium/ebpf/internal/epoll/poller.go:176 +0x298
github.com/cilium/ebpf/ringbuf.(*Reader).ReadInto(0xc0001f0380, 0xc000286cc0)
	/opt/app-root/vendor/github.com/cilium/ebpf/ringbuf/reader.go:160 +0x188
github.com/cilium/ebpf/ringbuf.(*Reader).Read(0xc002325620?)
	/opt/app-root/vendor/github.com/cilium/ebpf/ringbuf/reader.go:141 +0x25
github.com/grafana/beyla/pkg/internal/ebpf/common.(*ringBufForwarder).readAndForwardInner(0xc001889050, {0x27f17d0, 0xc0001f0380}, 0x0)
	/opt/app-root/pkg/internal/ebpf/common/ringbuf.go:150 +0x133
github.com/grafana/beyla/pkg/internal/ebpf/common.(*ringBufForwarder).readAndForward(0xc001889050, {0x280bf78, 0x407f500}, 0x0)
	/opt/app-root/pkg/internal/ebpf/common/ringbuf.go:129 +0x31c
github.com/grafana/beyla/pkg/internal/ebpf/watcher.(*Watcher).Run(0xc0018c3200, {0x280bf78, 0x407f500})
	/opt/app-root/pkg/internal/ebpf/watcher/watcher.go:98 +0x2b3
created by github.com/grafana/beyla/pkg/internal/ebpf.RunUtilityTracer in goroutine 115
	/opt/app-root/pkg/internal/ebpf/tracer_linux.go:328 +0x2c7

goroutine 121 [chan receive]:
github.com/grafana/beyla/pkg/internal/discover.(*pollAccounter).watchForProcessEvents(0xc002645980, 0xc002325610, 0xc000296e00)
	/opt/app-root/pkg/internal/discover/watcher_proc.go:171 +0x45
created by github.com/grafana/beyla/pkg/internal/discover.(*pollAccounter).Run in goroutine 115
	/opt/app-root/pkg/internal/discover/watcher_proc.go:122 +0x2b8

goroutine 266 [select]:
github.com/hashicorp/golang-lru/v2/expirable.NewLRU[...].func1()
	/opt/app-root/vendor/github.com/hashicorp/golang-lru/v2/expirable/expirable_lru.go:86 +0xf5
created by github.com/hashicorp/golang-lru/v2/expirable.NewLRU[...] in goroutine 41
	/opt/app-root/vendor/github.com/hashicorp/golang-lru/v2/expirable/expirable_lru.go:82 +0x32a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mariomac