Samples are filed under results/<tool>/<version>/<fixture>.yaml and reduced with min() over up to LIMIT_RUNS (30) of them. The version is the only thing in that path that distinguishes one sample from another, so every sample ever taken of a version pools into one number — no matter what else changed in between.
Plenty changes in between: the runner image, the fixture's dependency graph, the registry the install goes through, the emulated link in front of it. Two samples of pnpm@12.0.0 measured four months apart are treated as two measurements of the same thing, and the faster one wins.
We already know this and handle it by hand. From the README:
The numbers are recorded under their own fixture name (alotta-files-pnpr), and the runs recorded before the benchmark installed through pnpr are left where they are under alotta-files. They were measured against a different registry over a different network, so pooling the two would average unrelated things together. The same rule holds within the name: change the setup in a way that changes what a number means, and the runs recorded before the change have to be deleted, or min() pools them with the new ones and a number from the old setup stands in for the new one.
That is a real discipline enforced by prose. It worked for the pnpr change because that change was big enough to notice; a runner image bump is not, and nothing catches it.
Two things follow from having no date:
- Silent contamination. A setup change nobody thought to file under a new name quietly re-dates old numbers as current ones. The failure is invisible — the page just shows a number.
- A question we can't ask. "Did this version get slower since we last measured it?" has no answer in the current layout, because the samples that would answer it have already been collapsed. Regressions in the harness, the runner, or the registry are indistinguishable from noise.
What vltpkg does
vltpkg/benchmarks keys entirely by date: raw hyperfine output goes to Actions artifacts (7-day retention, never committed), and a processed snapshot per day is pushed to gh-pages with keep_files: true — ~400 dated directories going back to 2025-08-22.
Their trade is the opposite of ours and I don't think we want it wholesale: raw data that expires in a week can't be re-reported from a clone, and a single day's run is a worse estimator for "is this release faster than the last" than a min over 30 samples of that exact version. But the date is the part they have and we don't.
Direction
Keep version-keying and the min() — they answer the question the page asks. Add the date so that:
- a sample carries when it was measured, and by which harness commit;
min() is taken over a window rather than over all history, so an old sample can't stand in for a current one;- the samples behind a published number stay inspectable as a series instead of being collapsed on write.
Roughly: record the timestamp (and the harness SHA) per sample rather than only per file, and have the reporting run choose which samples are current instead of trusting that whatever is on disk belongs together.
Worth deciding before implementing: whether benchmarks.json should then carry a series rather than one number per tool, which is the point where this stops being an internal storage change and starts being a change to what https://pnpm.io/benchmarks can show.
Not blocking #67 — that PR changes what is published, not how samples are stored.
Samples are filed under
results/<tool>/<version>/<fixture>.yamland reduced withmin()over up toLIMIT_RUNS(30) of them. The version is the only thing in that path that distinguishes one sample from another, so every sample ever taken of a version pools into one number — no matter what else changed in between.Plenty changes in between: the runner image, the fixture's dependency graph, the registry the install goes through, the emulated link in front of it. Two samples of
pnpm@12.0.0measured four months apart are treated as two measurements of the same thing, and the faster one wins.We already know this and handle it by hand. From the README:
That is a real discipline enforced by prose. It worked for the pnpr change because that change was big enough to notice; a runner image bump is not, and nothing catches it.
Two things follow from having no date:
What vltpkg does
vltpkg/benchmarks keys entirely by date: raw hyperfine output goes to Actions artifacts (7-day retention, never committed), and a processed snapshot per day is pushed to
gh-pageswithkeep_files: true— ~400 dated directories going back to 2025-08-22.Their trade is the opposite of ours and I don't think we want it wholesale: raw data that expires in a week can't be re-reported from a clone, and a single day's run is a worse estimator for "is this release faster than the last" than a min over 30 samples of that exact version. But the date is the part they have and we don't.
Direction
Keep version-keying and the
min()— they answer the question the page asks. Add the date so that:min()is taken over a window rather than over all history, so an old sample can't stand in for a current one;Roughly: record the timestamp (and the harness SHA) per sample rather than only per file, and have the reporting run choose which samples are current instead of trusting that whatever is on disk belongs together.
Worth deciding before implementing: whether
benchmarks.jsonshould then carry a series rather than one number per tool, which is the point where this stops being an internal storage change and starts being a change to what https://pnpm.io/benchmarks can show.Not blocking #67 — that PR changes what is published, not how samples are stored.