The Benchmarks workflow runs on dispatch, uploads a JMH result file as an artifact and stops. The
latency table in benchmarks/README.md was pasted there by a person, and M68 is about to paste the
comparison against the official client next to it.
Nothing compares a run with the one before it. A regression is therefore visible only to somebody who
opens two artifacts side by side, which is nobody, and the claims it would eat are the ones this project
is chosen for: no boxing on the hot path, a native image that answers 37 ms after process start, a REST
engine that trades throughput for footprint and not much throughput. Any of those can be moved by a
dependency bump while every check in this repository stays green.
The work is a baseline and a comparison rather than a new harness. A results file committed to the
repository, a run that reads it, a report of what moved, and accepting a new baseline as a deliberate
step with the run id recorded rather than as an edit to a table.
The judgement is the band, and it is the whole difficulty. A JMH number from a shared GitHub runner is
noisy, and a threshold tight enough to catch a real regression will fire on a noisy neighbour, which
teaches everybody to ignore it within a month. So the band comes from measured run-to-run variance over
an unchanged tree rather than from a round number, and the first version reports rather than fails. A
gate that nobody trusts is worse than no gate, because it also costs the attention of the person who
still reads it.
This depends on M68 having run: a baseline of numbers that were never published is a file nobody can
check against anything.
Exit criterion: a benchmark run compares against a baseline committed in this repository and reports
every operation that moved outside a band derived from measured variance, the band and how it was
measured are written down, and replacing the baseline is an explicit step that records the run it came
from.
The
Benchmarksworkflow runs on dispatch, uploads a JMH result file as an artifact and stops. Thelatency table in
benchmarks/README.mdwas pasted there by a person, and M68 is about to paste thecomparison against the official client next to it.
Nothing compares a run with the one before it. A regression is therefore visible only to somebody who
opens two artifacts side by side, which is nobody, and the claims it would eat are the ones this project
is chosen for: no boxing on the hot path, a native image that answers 37 ms after process start, a REST
engine that trades throughput for footprint and not much throughput. Any of those can be moved by a
dependency bump while every check in this repository stays green.
The work is a baseline and a comparison rather than a new harness. A results file committed to the
repository, a run that reads it, a report of what moved, and accepting a new baseline as a deliberate
step with the run id recorded rather than as an edit to a table.
The judgement is the band, and it is the whole difficulty. A JMH number from a shared GitHub runner is
noisy, and a threshold tight enough to catch a real regression will fire on a noisy neighbour, which
teaches everybody to ignore it within a month. So the band comes from measured run-to-run variance over
an unchanged tree rather than from a round number, and the first version reports rather than fails. A
gate that nobody trusts is worse than no gate, because it also costs the attention of the person who
still reads it.
This depends on M68 having run: a baseline of numbers that were never published is a file nobody can
check against anything.
Exit criterion: a benchmark run compares against a baseline committed in this repository and reports
every operation that moved outside a band derived from measured variance, the band and how it was
measured are written down, and replacing the baseline is an explicit step that records the run it came
from.