Kubelet OpenTelemetry Tracing

[sallyom]

Enhancement Description

• One-line enhancement description (can be used as a release note): Instrument kubelet to generate and export OpenTelemetry trace data.
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-instrumentation/2831-kubelet-tracing/README.md
• Discussion Link:
• Primary contacts (assignee): @somalley, @dashpole
• Responsible SIGs: SIG-Instrumentation
• Enhancement target (which target equals to which milestone):
  • Alpha release target (1.25)
  • Beta release target (1.27)
  • Stable release target (1.33)
• Alpha
  • KEP (k/enhancements) update PR(s):
    • KEP-2831: Instrumenting Kubelet for OpenTelemetry Tracing #2832
  • Code (k/k) update PR(s):
    • kubelet tracing instrumentation kubernetes#105126
  • Docs (k/website) update PR(s):
    • Add docs for KubeletTracing feature gate 1.25 website#35137
• Beta
  • KEP (k/enhancements) update PR(s):
    • KEP-2831: adding beta graduation criteria #3714
  • Code (k/k) update PR(s):
    • kubelet: create top-level traces for pod sync and GC kubernetes#114504
    • Graduate KubeletTracing feature to beta kubernetes#115750
  • Docs (k/website) update(s):
    • [KEP-2731] Add docs for Kubelet OpenTelemetry tracing graduation website#39904
• Stable
  • KEP (k/enhancements) update PR(s): KEP-2831: update kubelet tracing KEP to target GA in 1.33 #5134
  • Code (k/k) update PR(s):
  • Docs (k/website) update(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[kikisdeliveryservice]

@sallyom can you please add the mandatory discussion link to this issue?

[ehashman]

/sig node instrumentation

[logicalhan]

/assign

[shivanshuraj1333]

/cc

[yangjunmyfm192085]

/cc

[249043822]

/cc

[hosseinsalahi]

Hello @sallyom

v1.24 Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00pm PT on Thursday Feb 3rd, 2022. This enhancement is targeting alpha for v1.24,

Here’s where this enhancement currently stands:

• Updated KEP file using the latest template has been merged into the k/enhancements repo.
  - KEP file PR is pending.
• KEP status is marked as implementable for this release
• KEP has a test plan section filled out.
• KEP has up to date graduation criteria.
• KEP has a production readiness review that has been completed and merged into k/enhancements.

The status of this enhancement is marked as tracked. Please keep the issue description up-to-date for release v1.24.
Thanks!

[sallyom]

@encodeflush thanks for the check-in - I'm not sure what the production readiness review is edit: found it, will add the questionnaire, I don't see it on the enhancements README can you point me to the steps to get that done? Thanks again. Also, am I meant to check those off as they are done or are they for someone else to decide?

[hosseinsalahi]

@sallyom Thanks for the update. Those criterion should be fulfilled form your side. From enhancements team point of view, we go through them and check if your KEP meets the requirements.

[chrisnegus]

Hi @sallyom 👋 1.24 Docs shadow here.

This enhancement is marked as 'Needs Docs' for the 1.24 release.

Please follow the steps detailed in the documentation to open a PR against the dev-1.24 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thu March 31, 11:59 PM PDT.

Also, if needed take a look at Documenting for a release to familiarize yourself with the docs requirement for the release.

Thanks!

[valaparthvi]

Hi @sallyom 👋 1.24 Release Comms team here.

We have an opt-in process for the feature blog delivery. If you would like to publish a feature blog for this issue in this cycle, then please opt in on this tracking sheet.

The deadline for submissions and the feature blog freeze is scheduled for 01:00 UTC Wednesday 23rd March 2022 / 18:00 PDT Tuesday 22nd March 2022. Other important dates for delivery and review are listed here: https://github.com/kubernetes/sig-release/tree/master/releases/release-1.24#timeline.

For reference, here is the blog for 1.23.

Please feel free to reach out any time to me or on the #release-comms channel with questions or comments.

Thanks!

[hosseinsalahi]

Hello @sallyom

I'm just checking in once more as we approach the 1.24 Code Freeze on 18:00 PDT, Tuesday, March 29th 2022

Please ensure the following items are completed:

• All PRs to the Kubernetes repo that are related to your enhancement are merged by the code freeze deadline.
• Have a documentation placeholder PR open by 18:00 PDT, Thursday, March 31, 2022.

For note, the status of this enhancement is currently marked as at risk.

Could you please confirm if following PR is part of the implementation for this enhancement?

• kubelet tracing instrumentation kubernetes#105126
• KEP-3085: Add KEP for Pod network ready condition #3087

Kindly please let me know if I'm missing any related PRs other than the ones I linked above. Thank you so much!

[ddebroy]

#3087 is not directly related to the overall OpenTelemetry enhancements.

[sallyom]

kubernetes/kubernetes#105126 is the initial implementation for this enhancement

[valaparthvi]

Hi @sallyom 👋 1.24 Release Comms team here.

We have an opt-in process for the feature blog delivery. If you would like to publish a feature blog for this issue in this cycle, then please opt in on this tracking sheet.

The deadline for submissions and the feature blog freeze is scheduled for 01:00 UTC Wednesday 23rd March 2022 / 18:00 PDT Tuesday 22nd March 2022. Other important dates for delivery and review are listed here: https://github.com/kubernetes/sig-release/tree/master/releases/release-1.24#timeline.

For reference, here is the blog for 1.23.

Please feel free to reach out any time to me or on the #release-comms channel with questions or comments.

Thanks!

Hey @sallyom, can you please add a placeholder PR for this feature in https://github.com/kubernetes/website by March 30, that is our extended deadline for feature blog freeze?

Example of a feature blog PR: kubernetes/website#30538
Example of a feature blog: https://github.com/kubernetes/website/blob/main/content/en/blog/_posts/2021-12-08-dual-stack-networking-ga.md

[gracenng]

Hi, 1.24 Enhancements Lead here 👋. With code freeze now in effect, this enhancement has not met the criteria for the freeze and has been removed from the milestone.

As a reminder, the criteria for code freeze is:

All PRs to the kubernetes/kubernetes repo have merged by the code freeze deadline
Feel free to file an exception to add this back to the release. If you plan to do so, please file this as early as possible.

Thanks!
/milestone clear

[dashpole]

I went through the kubelet codebase, and found the following places we could potentially add tracing:

• podresources grpc server
• CSI grpc client
• devicemanager client/server
• DRA client
• Pluginmanager client

I don't think any of those should block GA.

The only remaining step is to graduate the configuration resource (staging/src/k8s.io/apiserver/pkg/apis/apiserver/v1beta1/types.go).

[saschagrunert]

I don't think any of those should block GA.

I agree. Would that also mean that we graduate to GA in the first place and are free to add tracing later on per demand?

[dashpole]

I agree. Would that also mean that we graduate to GA in the first place and are free to add tracing later on per demand?

Yes, I think that is the right approach.

[ArkaSaha30]

Hello @somalley, @dashpole👋, v1.33 Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 14th February 2025 / 19:00 PDT Thursday 13th February 2025.

This enhancement is targeting stage stable for v1.33 (correct me, if otherwise)
/stage stable

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.33. KEPs targeting stable will need to be marked as implemented after code PRs are merged and the feature gates are removed.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on Thursday 6th February 2025 so that the PRR team has enough time to review your KEP.

For this KEP, we would just need to update the following:

• Since this KEP made into beta in v1.27, the KEP template has changed since then and it needs to be updated to the latest template mentioned above, PR needs to be mentioned in the description
• KEP status needs to be updated as implemented with latest-milestone: v1.33
• KEP readme needs to reflect the Graduation Criteria for stable
• PRR needs to be updated for stable

The status of this enhancement is marked as At risk for enhancements freeze. Please keep the issue description up-to-date with appropriate stages as well.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

[ffromani]

@dashpole howdy! do we want to move this forward in 1.33?

[dashpole]

Yes, I plan to graduate this to stable in 1.33

[dipesh-rawat]

Hi @somalley, @dashpole 👋, 1.33 Enhancements team here,

Just a quick friendly reminder as we approach the enhancements freeze later this week, at 02:00 UTC Friday 14th February 2025 / 19:00 PDT Thursday 13th February 2025.

The current status of this enhancement is marked as At risk for enhancement freeze. There are a few requirements mentioned in the comment #2831 (comment) that still need to be completed.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

[dipesh-rawat]

Hello @somalley, @dashpole 👋, 1.33 Enhancements team here,

It looks like PR #5134 covers most of the updates we need for inclusion of this KEP in v1.33. To ensure it’s tracked for the release, we also need to ensure KEP README.md readme using the latest template.

Currently I can see in the README the below question under Troubleshooting section is missing.

• What steps should be taken if SLOs are not being met to determine the problem?
  Refer KEP template here.

• The Risks and Mitigations section is also missing from the README.

The current status of this enhancement is still marked as At risk for enhancement freeze.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

[dipesh-rawat]

@somalley @dashpole Now that PRs #5170 #5134 (please do mention both PRs in issue description too) have been merged, all the KEP requirements are in place and merged into k/enhancements, this enhancement is all good for the upcoming enhancements freeze. 🚀

The status of this enhancement is marked as tracked for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

(cc: @ArkaSaha30)

/label tracked/yes

[michellengnx]

Hello @dashpole and @logicalhan 👋, v1.33 Docs Shadow here.
Does this enhancement work planned for v1.33 require any new docs or modification to existing docs?
If so, please follow the steps here to open a PR against dev-1.33 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday 27th February 2025 18:00 PDT.
Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.
Thank you!