ateom: export telemetry through an atelet unix-socket OTLP relay

[Chenyi Wang (chw120)]

ateom runs inside the worker pod that hosts the actor, and exported OTLP straight to the collector over the pod's network. This adds a node-local relay: ateom pushes OTLP/gRPC over a unix socket that atelet serves and forwards to the collector, so a worker pod needs no network path of its own to export spans and metrics.

Four things motivate it:

• Blast radius. The pod runs untrusted agent code, so allowing it egress to the collector makes the collector reachable to anything that escapes the sandbox. A unix socket cannot leave the node.
• Connection count. Worker pods are heavily oversubscribed; N ateoms per node each held their own collector connection. They collapse into atelet's single per-node one.
• Interference. ateom transparently redirects actor egress to its own atunnel listener, and its own outbound traffic has to stay clear of the rules it installs. A unix socket is not IP traffic.
• Shutdown loss. Teardown frees the actor's network and then the pod goes away, which is when the spans describing teardown are still queued in the batch processor. atelet outlives the worker pod.

The relay forwards the OTLP request verbatim rather than decoding and re-exporting, so each ateom's own resource (service.name, service.instance.id) survives instead of being absorbed into atelet's.

It is best-effort: an ateom that finds no socket at startup logs it and exports directly to OTEL_EXPORTER_OTLP_ENDPOINT as before, so this is a no-op for a cluster running an older atelet. atelet likewise declines to serve a relay when no collector is configured, since it would accept spans only to drop them. Both halves stay off with --otlp-relay-socket="".

The socket lives in ateompath.BasePath, the hostPath already mounted at the same path into atelet and into every ateom pod, so no new volume or controller change is needed.

Also includes:

• End-to-end tests covering the full serverboot-to-collector path.
• Observability documentation updates for Jaeger tracing.

It's a good idea to open an issue first for discussion.

• Tests pass
• Appropriate changes to documentation are included in the PR

[Chenyi Wang (chw120)]

Benjamin Elder (@BenTheElder) FYI, this is the PR for collecting ateom traces. Please review it when you have time. Thank you.

[Benjamin Elder (BenTheElder)]

Human vetted, AI first pass.

[Benjamin Elder (BenTheElder)]

🤖 should-fix 🟡 – RemoveAll is a lot more than this needs, and the blast radius on a misconfigured path is the node's whole ateom state. The comment explains the goal as clearing a stale socket so Listen does not hit EADDRINUSE — os.Remove does exactly that and fails on a directory, which is the right answer. RemoveAll instead succeeds: --otlp-relay-socket=/var/lib/ateom-gvisor (the BasePath itself, an easy slip since the socket lives under it) would have atelet silently delete the image cache, every actor bundle, the staged runsc binaries and the local checkpoints, at startup, before anything else runs.

Stop already uses os.Remove on the same path, so the two ends of the socket's lifecycle disagree about which call is appropriate.

[Benjamin Elder (BenTheElder)]

🤖 nit 🟢 – This only builds a valid target for an absolute sockPath. With a relative one, unix://otlp.sock parses otlp.sock as the authority with an empty path, so the dial fails — and it fails lazily, at first export, which is precisely the outcome the os.Stat check above exists to avoid. Stat would have passed, since a relative path resolves fine against the working directory, so the deterministic startup fallback is bypassed and the telemetry is quietly lost instead.

grpc.NewClient("unix:"+sockPath, ...) — the single-colon form — takes everything after the colon as the path and handles both shapes, or the flag could reject a non-absolute path up front.