[PaulMH]

Summary

Two bugs in Kubernetes service registration can leave the active node labeled vault-active=false. A Service that selects on vault-active=true then has no endpoints.

1. A successful patch does not replace an older failed patch for the same label.

 Notify(patch)
   if PatchPod(patch) fails
     patchesToRetry[patch.Path] = patch
+    return
+  delete(patchesToRetry, patch.Path)

Before this change, the sequence Notify(active=false) (fails), then Notify(active=true) (succeeds) left false queued. The 5-second periodicUpdateState loop then sent false and overwrote the correct label. The comment on patchesToRetry already says it should hold only "the last state we were aware of".

2. A retry after 401/403 sends the old token.

getCheckRetry reloads the token file on 401/403 and retries. But do() sets the Authorization header once, before client.Do, so the retry still sends the old token and fails the same way.

 retryablehttp.Client{
   CheckRetry: c.getCheckRetry(req),   // reloads c.config on 401/403
+  PrepareRetry: set Authorization from c.config.BearerToken
 }

How we hit this

Kubernetes service registration reads the service account token once at startup. It uses the token only when HA state changes. With bound service account tokens, that token expires (EKS caps it at 24h). As a result, a long-running node's first label update after a leadership change gets a 401.

In our case, a storage failover made the active node lose leadership and regain it about 3 seconds later. The API server audit log showed:

PATCH vault-active=false  401   (old token)
PATCH vault-active=false  401   (retry, still old token) -> queued
PATCH vault-active=true   200   (token reloaded)
PATCH vault-active=false  200   (stale retry from periodicUpdateState)

The active node stayed labeled false until we fixed it by hand. Either fix alone breaks this chain. Fix 1 also covers other transient failures, such as 5xx responses or network errors during a fast leadership flap.

Related: #32148 (a different failure mode with the same symptom: no pod labeled active).

Evidence

New test TestRetryHandlerNotifySupersedesQueuedPatch:

API returns 504
Notify(active=false)    -> fails, queued
API returns 200
Notify(active=true)     -> succeeds
updateState()           -> retry loop runs
assert label == "true"

• Before: retry_handler_test.go:485: expected "true" but received "false"
  After: ok

Token refresh on retry (a local check against a test server that returns 401 unless it gets the new token; not committed):

• Before: headers sent [Bearer old Bearer old], giving up after 2 attempt(s): ... resp statuscode: 401
  After: headers sent [Bearer old Bearer new], success

go test -race ./serviceregistration/kubernetes/... passes.

Merge Danger

Door: two-way

Stateless client-side change. A revert restores the old behavior.

Blast Radius: narrow

Only Kubernetes service registration label patching. A successful Notify now also clears the queued retry for the same label. Retries for other labels are not affected. Retries after 401/403 now send the reloaded token. Requests that did not fail are unchanged.

🤖 Generated with Claude Code

[hashicorp-cla-app]

Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement

Learn more about why HashiCorp requires a CLA and what the CLA includes

_{Have you signed the CLA already but the status is still pending? Recheck it.}