RemoteMCPServer headersFrom secret is baked into agent config and never re-rendered on rotation — agents break permanently with short-lived tokens

[davidkarlsen]

Summary

When a RemoteMCPServer authenticates via headersFrom → Secret, the controller resolves that Secret at agent-render time and writes the literal header value into the per-agent config Secret (Secret/<agent>, config.json → http_tools[0].params.headers.Authorization). The agent pod mounts only its own config Secret, so it uses whatever value was current at the last render.

Nothing re-renders the agent config when the referenced Secret changes. With a short-lived credential the agent works for one token lifetime after each render, then every tool call fails — while a valid token sits unused in the Secret. Agent renders are event-driven and in our cluster occurred 4–17 hours apart, versus a 60-minute token TTL.

Environment

• controller 0.9.12
• RemoteMCPServer kagent.dev/v1alpha2, protocol: STREAMABLE_HTTP, remote MCP endpoint over HTTPS
• credential from the external-secrets GithubAccessToken generator (60 min token TTL), ExternalSecret with refreshInterval: 45m, target Secret key Authorization holding Bearer <token>

Reproduce

1. Create a RemoteMCPServer with headersFrom[0].valueFrom.type: Secret, pointing at a Secret that external-secrets refreshes from a short-lived token source.
2. Reference that RemoteMCPServer from an Agent's spec.tools.
3. Wait past the token TTL without making any change to the Agent spec.
4. All tool calls through that MCP server fail. Restarting the agent pod does not help — the config Secret itself is stale, so the new pod receives a byte-identical config (same kagent.dev/config-hash).

Expected vs actual

Expected: the agent's rendered config tracks the referenced Secret (watch + re-render), or the credential is resolved at connection time rather than inlined into the config.

Actual: the header value is a snapshot taken at render time. Recovery requires some unrelated event that happens to trigger an Agent re-render.

Evidence

• Three SHA-256-compared values: live auth Secret ≠ agent config Secret = pod-mounted /config/config.json. The pod matches its config Secret, so the kubelet mount sync is fine — the config Secret is the stale layer.
• Same pod, unchanged kagent.dev/config-hash: a read-only API call with the pod's token returned 200 at time T and 401 at T+16 min, while the freshly rotated token from the auth Secret returned 200 at that same moment. This rules out upstream rate limiting or server-side session state.
• The controller's own MCP loop does re-read the Secret — ~720 successfully registered remote MCP server log lines in a 3-hour window, with a stable tool catalog. Only the agent render is stale. Consequently RemoteMCPServer.status stays ACCEPTED and the UI looks healthy while every agent tool call gets a 401: the status reflects the controller's own connection, not the agent's.
• Agent-controller reconciles were rare in the same window (1 occurrence in 3 hours) versus continuous MCP reconciles, which is why the stale window lasts hours.
• The controller ServiceAccount already holds get,list,watch on core resources, so watching the referenced Secret needs no new RBAC.

Secondary finding (happy to split into its own issue)

A 401 from the remote MCP server is swallowed: the agent produces no function_response at all rather than surfacing the status. The model then invents an explanation — in our case "the repository does not exist / is not accessible" — and reports that to the user as if it were a finding. One session contained 11 consecutive tool calls with zero responses and zero warnings in the agent log, which made the actual cause (an expired credential) very hard to attribute.

Logging the upstream non-2xx status at WARN when an MCP tool call fails would make this diagnosable in seconds.

Suggested fix

Either:

1. Have the agent reconciler watch Secrets referenced via headersFrom and re-render the agent config on change (annotating the agent Deployment so the pod rolls), or
2. Stop inlining the resolved header and resolve it at connection time.

Option 1 keeps the existing "self-contained rendered config" model (the agent pod has no RBAC to read Secrets, which is a reasonable design) while making the implicit assumption — that everything inlined is static — actually hold.