Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Karpenter doesn't seem to refresh access tokens #588

Open
logicfox opened this issue Nov 28, 2024 · 0 comments
Open

Karpenter doesn't seem to refresh access tokens #588

logicfox opened this issue Nov 28, 2024 · 0 comments

Comments

@logicfox
Copy link

Version

Karpenter Version: v0.7.0

Kubernetes Version: v1.30.5

Expected Behavior

Karpenter should refresh access tokens periodically to ensure that they do not expire.

Actual Behavior

Karpenter does not seem to refresh access tokens.

The GetToken function doesn't seem to be invoked anywhere in the codebase. I have noticed permission errors in the Karpenter pod logs when I have a node running for over 12 hours and the error message is typically about missing read permissions on the VM. The permissions are present and simply restarting the Karpenter pod fixes the problem, which leads me to suspect that the token is not being refreshed. My organisation has a 12 hour expiry policy for user login, but I do not know if that applies to identities. The start of the errors in the logs seems to correspond with the timeline.

Steps to Reproduce the Problem

  • Ensure that tokens expire after a period of time.
  • Deploy karpenter and a workload to trigger the creation of a pod on a Karpenter-managed node.
  • Wait till the token expires (12 hours in my case).
  • Notice that the pod logs mention read errors when trying to fetch VM details from ARM.

Resource Specs and Logs

TODO: I don't have the relevant logs currently, but I'll attach logs after reproducing the issue.

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant