Skip to content

fix: clarify benchmark regression report verdicts - #2394

Merged
zeitlinger merged 3 commits into
prometheus:mainfrom
zeitlinger:agent/benchmark-report-verdict
Aug 19, 2026
Merged

fix: clarify benchmark regression report verdicts#2394
zeitlinger merged 3 commits into
prometheus:mainfrom
zeitlinger:agent/benchmark-report-verdict

Conversation

@zeitlinger

Copy link
Copy Markdown
Member

Summary

  • Add an explicit base-vs-head regression verdict using comparable JMH metadata, confidence-interval overlap, and a practical 5% threshold.
  • Remove cross-method Nx slower rankings from the PR-head tables.
  • List head-only benchmarks separately as descriptive results with no regression verdict.

Motivation

The #2329 benchmark report described HistogramBenchmark.openTelemetryExponential as “20x slower” only because it was ranked against a different benchmark method in the same run. Its actual base/head delta was -5.7% with overlapping confidence intervals (within noise). This change makes that distinction explicit and prevents the within-run ranking from being mistaken for a regression.

Tests

  • python3 .mise/tasks/test_generate-benchmark-summary.py
  • python3 .mise/tasks/test_update-benchmarks.py
  • ruff check and ruff format --check on the changed Python files
  • mise run lint (full lint is blocked only by the pre-existing README Slack link returning HTTP 403; the pre-push scoped lint passed)

Related: #2329.

Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
@zeitlinger
zeitlinger marked this pull request as ready for review August 19, 2026 09:27
@zeitlinger
zeitlinger merged commit e5fa067 into prometheus:main Aug 19, 2026
18 checks passed
@zeitlinger
zeitlinger deleted the agent/benchmark-report-verdict branch August 19, 2026 11:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants