gc: expose old GC barriers through metrics and warnings (#11259)

[wfxr]

What problem does this PR solve?

Old GC barriers can retain obsolete data and cause disk pressure. Operators need metrics and warnings identifying the retaining barriers.

Cherry-pick of #11259 to release-nextgen-20251011, replacing the bot-generated PR with resolved conflicts.

Issue Number: ref #11260, close #11268

What is changed and how does it work?

Report valid GC barriers whose timestamps lag the current time by more than 24 hours after successful transaction safe point advancement. Export per-barrier physical timestamps and throttle warnings to once every 10 minutes, excluding the reserved keyspace gc_worker barrier.

Reuse existing barrier reads and maintain observations on successful writes, expiry, force deletion, and leadership changes. Bind metric collection to the release branch's leader serving cycle.

The metric is pd_gc_barrier_timestamp_seconds{scope,keyspace_id,keyspace_name,barrier_id}. Writes alone do not discover barriers. Global barriers and NullKeyspace use an empty keyspace name. HTTP force deletion clears observations only after a successful storage commit.

Both release branches lack the source branch's GC cache/watch and keyspace metadata removal APIs. This backport preserves the existing release APIs and adds the required leadership hooks; it omits callbacks and tests for the unavailable metadata removal operation. Transaction test wrappers and storage-read assertions match the release interfaces. Tests use Go 1.23-compatible WaitGroup and benchmark APIs.

Check List

Tests

• Failpoint-enabled make basic-test for pkg/gc, pkg/keyspace, and server/api, with default and NextGen builds: passed.
• Targeted NextGen GC observation and force-deletion tests with -race: passed.
• HTTP TestGetGCSafePoint regression in both client discovery modes, with -race under default and NextGen builds: passed.
• Scoped make check passed for pkg/gc. The broader server/API check reports six existing lint findings in unchanged server/gc_service.go and server/api/{failpoint,plugin,plugin_disable}.go files.
• Final aggregate diff checked against gc: expose old GC barriers through metrics and warnings #11259; no conflict markers remain.

Code changes

• Adds GC barrier timestamp metrics and warning logs, with cleanup on expiry, deletion, leadership changes, and shutdown.

Side effects

• Retains state proportional to reported barriers and inspects already-loaded barrier collections during successful advancement; no additional storage reads or periodic scans.

Related changes

• Original PR: gc: expose old GC barriers through metrics and warnings #11259.
• Supersedes gc: expose old GC barriers through metrics and warnings (#11259) #11268.

Release note

Expose GC barriers whose timestamps lag the current time by more than 24 hours through per-barrier timestamp metrics and warning logs to help diagnose delayed garbage collection.

Summary by CodeRabbit

• New Features

  • Added Prometheus metrics for tracking GC barrier timestamps across keyspaces and global barriers.
  • Added warnings for barriers that remain older than 24 hours.
  • Added support for force-deleting service GC safe points through the HTTP API.

• Bug Fixes

  • GC barrier metrics now update immediately when barriers are changed or deleted.
  • Metrics are correctly cleared during leadership transitions and server shutdown.

[ti-chi-bot]

This cherry pick PR is for a release branch and has not yet been approved by triage owners.
Adding the do-not-merge/cherry-pick-not-approved label.

To merge this cherry pick:

1. It must be LGTMed and approved by the reviewers firstly.
2. For pull requests to TiDB-x branches, it must have no failed tests.
3. AFTER it has lgtm and approved labels, please wait for the cherry-pick merging approval from triage owners.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Advanced

Run ID: 1050d8c9-68e5-479b-8067-1db98ae154a8

📥 Commits

Reviewing files that changed from the base of the PR and between e999884 and c33479f.

📒 Files selected for processing (7)

• pkg/gc/gc_state_manager.go
• pkg/gc/gc_state_manager_test.go
• pkg/gc/metrics.go
• pkg/gc/metrics_test.go
• server/api/service_gc_safepoint.go
• server/server.go
• tests/integrations/client/http_client_test.go

Included review availability: Your plan provides up to 4 included reviews per hour; 3 remain after this review.

---

📝 Walkthrough

Walkthrough

The change adds Prometheus metrics and warnings for old GC barriers, integrates metric state with GC advancement and leadership, and routes legacy service safe-point deletion through serialized GC manager logic.

Changes

GC barrier metrics

Layer / File(s)	Summary
Barrier metrics collector pkg/gc/metrics.go, pkg/gc/metrics_test.go	Adds scoped barrier metrics, expiration filtering, 24-hour age filtering, warning throttling, generation fencing, concurrent collection, and lifecycle cleanup.
GC manager and leadership integration pkg/gc/gc_state_manager.go, pkg/gc/metrics_test.go, pkg/gc/gc_state_manager_test.go, server/server.go	GC advancement and compatibility operations observe barriers. Successful writes update or delete entries. Leadership callbacks publish and detach the active metrics.
Serialized force deletion pkg/gc/gc_state_manager.go, server/api/service_gc_safepoint.go, tests/integrations/client/http_client_test.go, pkg/gc/metrics_test.go	The legacy HTTP deletion path uses ForceDeleteServiceGCSafePoint. Successful deletion removes the matching metric, including for non-expiring barriers and absent services.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant PDServer
  participant GCStateManager
  participant barrierMetrics
  participant Prometheus
  PDServer->>GCStateManager: Become leader
  GCStateManager->>barrierMetrics: Register current metrics
  PDServer->>GCStateManager: Advance transaction safe point
  GCStateManager->>barrierMetrics: Observe loaded barriers
  Prometheus->>barrierMetrics: Scrape retained entries
  barrierMetrics-->>Prometheus: Return qualifying barrier timestamps

Loading

Merge Risk: ⚪ Minimal · up to c3347

The barrier observability and force-deletion changes appear mergeable with no identified actionable risk.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 11.76% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 17 functions across 7 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the primary change: exposing old GC barriers through metrics and warnings. It is concise and specific.
Description check	✅ Passed	The description covers the problem, implementation, issue references, tests, side effects, related changes, and release note. It also explains the release-branch limitations and compatibility behavior…
Linked Issues check	✅ Passed	The changes satisfy the coding requirements in [#11268]. They add pd_gc_barrier_timestamp_seconds with scope, keyspace, and barrier labels. They exclude gc_worker and use empty names for global an…
Out of Scope Changes check	✅ Passed	The changed files remain within [#11268]. GC manager and metric changes implement barrier observation, warning, lifecycle, and force-deletion behavior. Server leadership and shutdown hooks connect tha…

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[wfxr]

/retest

[codecov]

Codecov Report

❌ Patch coverage is 97.10983% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.65%. Comparing base (e999884) to head (c33479f).

Additional details and impacted files

@@                     Coverage Diff                      @@
##           release-nextgen-20251011   #11270      +/-   ##
============================================================
+ Coverage                     78.59%   78.65%   +0.06%     
============================================================
  Files                           491      491              
  Lines                         66337    66504     +167     
============================================================
+ Hits                          52139    52312     +173     
+ Misses                        10425    10414      -11     
- Partials                       3773     3778       +5     

Flag	Coverage Δ
unittests	78.65% <97.10%> (+0.06%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ti-chi-bot]

@coderabbitai[bot]: adding LGTM is restricted to approvers and reviewers in OWNERS files.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: coderabbitai[bot], JmPotato

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [JmPotato]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-09-17 03:41:04.194017087 +0000 UTC m=+262910.131674681: ☑️ agreed by JmPotato.

[wfxr]

/hold