feat(telemetry): forward request spans to Datadog APM with RUM correlation

[illegalprime]

Reviewable diff: +206/-13 across 8 files (excludes generated, test, and story files).

Summary

Adds a production tracing overlay so operators can forward fleet-api request spans to Datadog APM with one flag (run-fleet.sh --enable-tracing + DD_API_KEY in .env). Spans parent to the traceparent header the existing Datadog RUM integration already injects on API calls, so browser sessions link directly to backend traces in Datadog. The dev stack (Jaeger) is unchanged, and deployments without the overlay see no behavior change.

How it works

fleetd's tracing pipeline was already vendor-neutral OTLP (built for the dev Jaeger stack); this PR wires it to Datadog in production and closes the RUM↔APM gap:

1. Overlay activation — run-fleet.sh --enable-tracing (or ENABLE_TRACING=true in .env) validates prerequisites (overlay file present, DD_API_KEY set) before any compose invocation — the overlay's ${DD_API_KEY:?} interpolation would otherwise abort every compose command including down — then layers docker-compose.tracing.yaml onto the stack.
2. Span production — the overlay hard-codes FLEET_TELEMETRY_ENABLED=true on fleet-api. The existing otelhttp middleware creates a span per HTTP request and exports OTLP/HTTP to the collector. Host-networked fleet-api reaches it via extra_hosts: otel-collector=127.0.0.1 against a loopback-only published port.
3. RUM correlation — new FLEET_TELEMETRY_TRUST_INCOMING_TRACES (overlay default: on) disables otelhttp's public-endpoint mode, so server spans parent to the client's W3C traceparent instead of starting a fresh trace with a link. Datadog RUM already injects that header on same-origin API calls (allowedTracingUrls), giving session→trace navigation in Datadog. The sampler caps remote-sampled parents at FLEET_TELEMETRY_SAMPLE_RATE, so a client's sampled flag cannot force ingestion past the configured rate.
4. Export — the collector runs Datadog's documented two-hop pipeline: traces feed the datadog/connector (computes APM hit/error/latency stats, which the exporter itself stopped doing in contrib 0.95), and both the re-emitted traces and the stats metrics export to Datadog keyed by DD_API_KEY/DD_SITE, tagged with env from DD_ENV.

Diagrams

flowchart LR
    subgraph Browser
        RUM["Datadog RUM SDK"]
    end
    subgraph Host["Fleet host"]
        NGINX["fleet-client nginx"]
        API["fleet-api (otelhttp middleware)"]
        COLL["otel-collector sidecar"]
    end
    DD["Datadog APM"]
    RUM -- "fetch + traceparent" --> NGINX
    NGINX -- "/api-proxy strip" --> API
    API -- "OTLP HTTP (loopback 4318)" --> COLL
    COLL -- "traces + APM stats" --> DD
    RUM -. "RUM events" .-> DD

Loading

flowchart TD
    REQ["Incoming request with traceparent"] --> TRUST{"TRUST_INCOMING_TRACES?"}
    TRUST -- "false (default)" --> NEW["New trace, client context kept as span link"]
    TRUST -- "true (overlay default)" --> PARENT["Span parents to client trace"]
    PARENT --> CAP["Sampler caps remote-sampled parents at SAMPLE_RATE"]

Loading

Areas of the code involved

Area	What changed	Why it matters for review
server/internal/infrastructure/fleet-telemetry/telemetry.go	New TrustIncomingTraces config; sampler adds WithRemoteParentSampled(TraceIDRatioBased)	Security/cost boundary: decides how much authority clients get over tracing
server/internal/handlers/middleware/telemetry.go	Trust flag toggles otelhttp public-endpoint mode	The actual parent-vs-link semantics for RUM correlation
server/cmd/fleetd/main.go	Threads the flag into the middleware	One-line wiring
deployment-files/docker-compose.tracing.yaml	New overlay: collector sidecar, loopback OTLP, fleet-api tracing env	Networking (host-mode fleet-api ↔ bridged collector) and DD_API_KEY fail-fast
deployment-files/server/otel-collector-config.datadog.yaml	New collector config: datadog/connector two-hop pipeline	Without the connector, APM service pages show no stats
deployment-files/run-fleet.sh	--enable-tracing flag, .env toggle, early prerequisite validation	Validation must precede overlay layering; see decision below
deployment-files/README.md	New "Server Tracing (Datadog APM)" operator section	Documents the trust-by-default posture and how to disable it
.github/workflows/proto-fleet-artifact-build.yml	Packages the two new files into the deployment bundle	Missing this = overlay absent from installs

Key technical decisions & trade-offs

• Collector sidecar with the datadog exporter over the Datadog Agent or dd-trace-go — fleetd stays vendor-neutral OTLP and dev keeps Jaeger; the Agent would add a heavy container for traces alone, and dd-trace-go would fork the instrumentation path per backend.
• Trust incoming trace context by default in the overlay (off by default in fleetd) — required for RUM↔APM correlation; means any client on the operator LAN can supply trace IDs pre-auth. Blast radius is Datadog trace data only (no first-party code consumes span context); documented in the README with a per-site opt-out. Per-route scoping was rejected: nginx strips /api-proxy before fleet-api sees requests, so there is no clean server-side discriminator for RUM traffic.
• WithRemoteParentSampled(TraceIDRatioBased) instead of ParentBased defaults — the default honors any remote sampled flag, letting clients bypass SAMPLE_RATE (ingest cost). Capping is deterministic per trace ID, so correlation survives for traces that are recorded; an incoming not-sampled flag is still honored.
• Prerequisite validation before compose layering in run-fleet.sh — ${DD_API_KEY:?} in the overlay aborts every compose command once layered; validating late corrupted the data-volume reinit path (compose down fails, script proceeds).
• datadog/connector two-hop pipeline — since collector-contrib 0.95 the exporter no longer computes APM stats; without the connector, Trace Explorer works but service/resource pages stay empty.

Testing & validation

• just lint clean (buf, eslint, golangci-lint x3).
• go test green for both changed packages. New tests: middleware parent-vs-link semantics under both trust modes (span parents to client trace vs. new trace with link), and sampler regression tests proving a remote-sampled parent is dropped at rate 0 and kept at rate 1.
• Collector config validated with the pinned otelcol-contrib:0.150.1 image (validate exit 0).
• Compose merge validated in a simulated artifact-bundle layout: overlay alone and stacked with alerts + system-monitoring both render (docker compose config), extra_hosts/depends_on/env merges confirmed; missing DD_API_KEY fails with the intended message.
• bash -n clean; CRLF-tolerant .env toggle verified in a debian container.
• Not covered: no E2E run (no UI/API behavior change — spans only), and no live Datadog ingest test; first real deployment should confirm spans and APM stats arrive and that RUM session→trace links resolve.

🤖 Generated with Claude Code

[github-actions]

🔐 Codex Security Review

Note: This is an automated security-focused code review generated by Codex.
It should be used as a supplementary check alongside human review.
False positives are possible - use your judgment.

Scope summary

• Reviewed pull request diff only (7df92e1cb52681c1185d7132a55fb0d431857d06...f3412fe248e9ac056ca0777be2e7f2dc29c605d0, exact PR three-dot diff)
• Model: gpt-5.5

💡 Click "edited" above to see previous reviews for this PR.

---

Review Summary

Overall Risk: LOW

Findings

[LOW] Client-Controlled Trace Context Can Suppress Or Force Server APM Sampling

• Category: Reliability
• Location: deployment-files/docker-compose.tracing.yaml:12
• Description: The tracing overlay defaults FLEET_TELEMETRY_TRUST_INCOMING_TRACES to true, which makes fleet-api parent request spans to any incoming traceparent header. A requester that can reach the API can send an unsampled traceparent to suppress server spans, or a sampled client-chosen trace ID to influence which requests survive the configured sample rate.
• Impact: When operators enable tracing for debugging or incident investigation, malicious or buggy clients can blind APM for selected requests or increase exported trace volume beyond what the operator expects from server-side sampling.
• Recommendation: Default the deployment overlay to FLEET_TELEMETRY_TRUST_INCOMING_TRACES=false and keep incoming client context as a link unless the operator explicitly accepts this trust boundary. If parented RUM correlation is required, document it as an observability trade-off and consider a sampler strategy that does not let client trace flags decide whether server spans are recorded.

Notes

Reviewed only .git/codex-review.diff as requested. No auth, SQL injection, command injection, cryptostealing/pool hijack, protobuf wire-format, or secret-exposure issues were found in the changed hunks.

---

_{Generated by Codex Security Review |
Triggered by: @illegalprime |
Review workflow run}