ci: add CI performance dashboard at /ci-dashboard

[jhinpan]

CI Performance Dashboard → rocm.github.io/FlyDSL/ci-dashboard

Every PR runs the benchmark matrix on three real GPUs (gfx950 / gfx942 / gfx1201), and the "Check benchmark performance (current vs main / vs tag)" step already computes per‑kernel regressions. The problem: those numbers only ever land in the job logs. Catching a regression today means knowing which run, opening the right job, and reading a wall of text.

This adds a static dashboard built around the one question a maintainer opens it for — did main regress, which kernel, how much — with the answer on the landing page.

Vanilla HTML/CSS/JS + a vendored Chart.js — no build step, no framework.

Four views

Health (landing)	A single count of real regressions on main, then each regressing kernel with a sparkline.
PR Check	Pick a PR → does its latest run slow any kernel vs main. The merge‑gating view.
Trends	Per‑kernel TB/s · TFLOPS across commits, one line per runner, with a noise band.
CI Board	Recent PRs/runs with per‑runner pass/fail/running chips and each run's worst Δ vs main.

PR Check

Trends

Noise‑aware, not just a fixed gate

The headline count isn't a naïve "−3% somewhere." A drop only counts as real when it exceeds the kernel's own run‑to‑run noise — main mean ± Kσ over recent main runs — falling back to the fixed −3% gate when there aren't enough samples (and the UI says which is in effect). On the current data that turns ~6 raw regressions into 2 real ones: the memory‑bound norms wobble into "watch", while tighter kernels like moe_gemm1 / hgemm are flagged. Trends draws the band; a noise‑aware toggle exposes the model.

How it works (no Pages‑setting change)

"Fly DSL test" run (3 runners, per PR/main) ──logs──┐
                                                     ▼
   .github/workflows/ci-dashboard-ingest.yml  (cron + after each run)
     gh api → parse logs → history.json / runs.json
                                                     │ push
                                                     ▼
                        branch  ci-dashboard-data   ←── fetched at runtime (raw, CORS *)
                                                     ▼
   docs.yml  ──copies ci-dashboard/ into the docs Pages artifact──►  /ci-dashboard
                                                + live PR status via the public Actions API

The page ships inside the existing docs Pages deploy (docs.yml gains one cp step), so it appears at /ci-dashboard with zero GitHub Pages configuration change. The data is regenerated by a scheduled workflow onto an orphan ci-dashboard-data branch the page fetches live — main history and the Sphinx build stay untouched.

Arch ↔ runner (the chips reflect the actual CI boxes): gfx950 = MI355X (CDNA4), gfx942 = MI325X (CDNA3), gfx1201 = R9700 (RDNA4).

What's here

• ci-dashboard/ — the SPA, a stdlib‑only parser + ingest (ingest/parse_bench.py, ingest/ingest.py) with a unit test, seed data, schema doc, banner asset
• .github/workflows/ci-dashboard-ingest.yml — cron + workflow_run ingestion (default GITHUB_TOKEN: actions:read for logs, contents:write for the data branch)
• docs.yml / docs/index.rst — bundle the page and link it
• README.md — a CI Performance Dashboard banner + badges entry

Verification

• Parser is exact against real logs. Against the actual logs of run 27196355973 for all three runners it reproduces every comparison row (e.g. layernorm −7.0%, moe_w4a16_s2_reduce +8.8%, gemm +1.9% TFLOPS), including the main~N fallback baselines. Covered by ci-dashboard/ingest/test_parse_bench.py (pure‑Python, no GPU/network); passes the repo's black + ruff gate.
• Seed data is real, ingested from recent runs so Trends has history on first paint; the live data branch carries the full window.
• Rendered end‑to‑end — the screenshots are the real build over real data.

Notes for review

• The ci-dashboard-data branch is the analogue of ATOM's gh-pages data store; bot‑written, never edited by hand. If you'd prefer the more ATOM‑faithful benchmark-action/github-action-benchmark + gh-pages‑branch model, that's a clean follow‑up (it needs the Pages source switched to a branch, which is why this avoids it).
• In‑progress runs show live status but no parsed numbers until they finish and the next ingest pass picks up the logs.
• Exact baseline commit SHA isn't captured yet — the UI surfaces which main (main / main~N); SHA‑level traceability is a small ingest follow‑up.
• Forks point CFG.repo / CFG.dataBranch (top of app.js) at their own org.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a static “CI Performance Dashboard” to the repo (served with docs) plus ingestion/parsing tooling to publish benchmark history and live run status to a dedicated data branch.

Changes:

• Introduces a new ci-dashboard/ static web app (HTML/CSS/JS) with CI board, regressions, trends, and local cross-check views.
• Adds stdlib-only benchmark log parsing (parse_bench.py) + GitHub Actions log ingestion (ingest.py) and a scheduled workflow to publish history.json / runs.json to ci-dashboard-data.
• Adds a local helper script to generate/push local.json, and links the dashboard from docs.

Reviewed changes

Copilot reviewed 13 out of 15 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
scripts/ci_dashboard_local.sh	Runs local benchmarks, parses into local.json, optionally pushes to ci-dashboard-data.
docs/index.rst	Adds a docs note linking to the dashboard.
ci-dashboard/styles.css	Adds dashboard styling.
ci-dashboard/ingest/test_parse_bench.py	Adds pure-Python tests for the benchmark parser.
ci-dashboard/ingest/parse_bench.py	Adds parser for benchmark tables + comparison blocks + AIter sweep.
ci-dashboard/ingest/ingest.py	Adds ingestion via gh CLI to build history.json and runs.json.
ci-dashboard/index.html	Adds the static dashboard page structure and UI elements.
ci-dashboard/data/schema.md	Documents the JSON data schema consumed by the dashboard.
ci-dashboard/data/runs.json	Seeds the dashboard with a bundled runs.json snapshot.
ci-dashboard/app.js	Implements data loading, rendering, and “live” overlay via GitHub API.
ci-dashboard/README.md	Documents dashboard architecture, local dev, and configuration.
.github/workflows/docs.yml	Bundles ci-dashboard/ into the docs Pages artifact.
.github/workflows/ci-dashboard-ingest.yml	Adds scheduled / triggered job to regenerate and publish dashboard data.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jhinpan]

All points before addressed in ac01cd13.

Correctness (your two catches)

• main~N fallback labels dropped (parse_bench.py:73): the comparison-row regex now allows ~ in the baseline label, and is_main treats any main* label (incl. main~1/main~2) as the main baseline rather than a tag. Added test_main_fallback_label_is_treated_as_main reproducing your synthetic current vs main~2 log.
• Empty pull_requests → pr: null (ingest.py:79): added resolve_pr(), which falls back to pulls?state=all&head=<head_owner>:<branch> and matches head.sha for pull_request events. Verified against a 40-run ingest: 35/35 PR runs now resolve (e.g. 27252270467→[Perf] flash_attn gfx950: dwordx4 O stores + flash-decoding split-K #670, this branch→ci: add CI performance dashboard at /ci-dashboard #675) where they were null before.

Copilot

• Regression badge now counts against the selected baseline (r[base].delta_pct <= regressionPct), not always vs_main.
• Board grouping key is br:${branch}:${commit} (no more run_id, which defeated grouping).
• Local cross-check matches CI by arch (not hard-coded gfx950); ci_dashboard_local.sh now derives --source from --arch and uses a per-run mktemp file.
• Tabs got real ARIA semantics (aria-selected toggled in JS, aria-controls → role="tabpanel" panels with aria-labelledby).
• docs.yml copies into a fresh dir (rm -rf + cp -r ci-dashboard/.) so a pre-existing target can't nest.
• Test is importable under pytest from repo root (prepends its dir to sys.path); corrected the AIter-sweep docstring (first table wins).

black+ruff clean, 6 parser tests pass.

[jhinpan]

git diff --check upstream/main...HEAD reports this as ci-dashboard/README.md:68: new blank line at EOF. Please drop the extra blank line after the final paragraph so the PR diff is whitespace-clean.

[jhinpan]

@coderfeli this looks solid now — I think it's ready for a review and merge.

One thing I'd like your eyes on, and that we can only fully validate after merge: the auto‑refresh. The dashboard's data is kept current by .github/workflows/ci-dashboard-ingest.yml, which is meant to keep pulling the latest CI/CD status on its own:

• workflow_run after every Fly DSL test run completes → ingests that run's results within a few minutes (covers main pushes and in‑flight PRs);
• schedule cron every 3h as a backstop;
• the live CI board also polls the public Actions API every 90s client‑side.

Because schedule and workflow_run only fire from the default branch, none of this actually runs while it's still a PR. So once merged, could you confirm the ingest workflow:

1. gets triggered on the next Fly DSL test completion,
2. creates/updates the orphan ci-dashboard-data branch with GITHUB_TOKEN (needs actions:read + contents:write — set in the workflow), and
3. the 3h cron actually fires.

Default max-runs is 120 (~10 days), so the first run also backfills history. If the org's Actions/permissions setup needs any tweak for this to trigger reliably, you'd know better than me.

Live preview meanwhile: https://jhinpan.github.io/FlyDSL-lab/

[coderfeli]

all histroy data here? use git artifact to store?

[jhinpan]

Good catch — and no, the full history isn't meant to live in the tree. What's committed under data/ is only a first-paint seed; the real store is already out-of-tree on the orphan ci-dashboard-data branch (our analog of ATOM's data.js on gh-pages) — machine-written by ci-dashboard-ingest.yml, never hand-edited. app.js fetches that branch at runtime and always prefers it; the seed only renders if the branch is unreachable (first load, or a fork before its first ingest).

You're right that the seed had drifted into being the whole dataset, which makes it read like the store. Fixed in 6107b2f1: trimmed it to a single most-recent snapshot per series — history.json 589 → 169 records, 233 KB → 62 KB, no history depth (runs.json 25 → 10) — and added a "note" field on each file spelling out that history lives on the branch. So the in-tree bytes are now a tiny fallback, and the "git artifact" you're pointing at is the ci-dashboard-data branch.

[coderfeli]

https://github.com/ROCm/ATOM/tree/main/.github/dashboard follow this folders and styles?

[jhinpan]

@coderfeli done — moved the dashboard to .github/dashboard/ to match ROCm/ATOM/.github/dashboard (6107b2f1). The served URL is unchanged (/ci-dashboard), so no external links break.

The layout now mirrors ATOM's model:

• source under .github/dashboard/ (like ATOM's .github/dashboard/index.html);
• history out-of-tree — ATOM keeps data.js on gh-pages; we keep history.json/runs.json on the ci-dashboard-data branch and fetch them live (see the thread on history.json). I kept the JSON contract our parser already emits rather than converting to github-action-benchmark's data.js.

One piece I can't finish without you — serving. ATOM's Pages serves the gh-pages branch directly (legacy build), so its dashboard deploys onto gh-pages. FlyDSL's Pages is build_type: workflow (the Sphinx artifact), so this page rides along inside the docs deploy instead. Matching ATOM exactly — a single self-contained index.html on gh-pages via a deploy-pages.yml — means flipping the Pages source to a branch, which is a repo-admin setting I can't touch. If you'd like that, I'll do the github-action-benchmark + gh-pages migration as a follow-up; just flip Pages → gh-pages and I'll wire the rest.

On styles — did you mean the visual look or the single-file packaging? We split index.html / app.js / styles.css for readability; ATOM inlines everything into one file. Happy to consolidate into a single index.html if you want structural parity there too.

[coderfeli]

@jhinpan Thanks. Does it mean we don't need to merge this pr but create a new one to main that points link to this branch?

[jhinpan]

@jhinpan Thanks. Does it mean we don't need to merge this pr but create a new one to main that points link to this branch?

Yeah, we do still need to merge it — the "just link to a branch" route can't actually serve this dashboard. Two reasons:

The page only goes live from main. Our Pages site is the Sphinx docs build, and the dashboard just gets copied into that same build (one cp step in docs.yml). That deploy only runs on a push to main, so on a PR branch the page never publishes anywhere. Serving it off a separate branch instead would mean switching the Pages source away from the docs site — that's a repo-admin change, and it'd break the docs deploy.

The auto-refresh only runs from main too. The ingest workflow fires on a schedule and after each Fly DSL test run, and GitHub only runs those triggers from the default branch — which is exactly why ci-dashboard-data doesn't exist yet. It also needs read access to the Fly DSL test logs, which only a workflow living inside this repo has.

One thing worth clearing up, since I think that's the worry: merging doesn't put any dashboard data into main. main only gets the static page (that one cp step, no Pages setting change) plus the two workflow files. All the actual data is written by the bot onto the separate ci-dashboard-data branch, which the workflow creates by itself on its first run.

So it's really just merging two small files. Once the first Fly DSL test finishes after merge, the data branch and the live page come up on their own — and that's the one part we can only validate once it's on main. The fork preview is great for demoing, but it can't be the permanent home.