GitHub PR: benchmark diff

[srid]

Produce reports like this for PRs in this repo, so that we can assess the impact of the PRs on benchmarks (code size, cpu & men). Take bench.csv produced by Hercules Effects for both commit A (commit where the PR branch branched off) and commit B (PR commit), and diff them.

---

#87 is an example of such PR which improved the types:builtin:intlist:plistEquals benchmark.

Old

types:builtin:intlist:plistEquals:==(n=3)            25843624(cpu)  59392(mem)  180(bytes)
types:builtin:intlist:plistEquals:/=(n=4)            12939651(cpu)  30928(mem)  181(bytes)
types:builtin:intlist:plistEquals:/=(empty;n=3)       7497147(cpu)  19064(mem)  177(bytes)

New

types:builtin:intlist:plistEquals:==(n=3)             9398080(cpu)  20846(mem)  95(bytes)
types:builtin:intlist:plistEquals:/=(n=4)             9398080(cpu)  20846(mem)  96(bytes)
types:builtin:intlist:plistEquals:/=(empty;n=3)       1639885(cpu)   4664(mem)  92(bytes)

Diff'ing these two tables, we would add a comment to the PR showing the number differences for each row/column much like the Plutus report. In the Plutus repo, benchmark reporting is manually triggered by adding a PR comment containing /benchmark.

[emiflake]

I was looking into implementing this. With Hercules, how would we create artifacts that we can use as means for comparison? It doesn't seem like Hercules publishes the bench.csv anywhere, or am I missing something?

[srid]

@nixinator @MatthewCroughan Could one of you help Emily here?

The nix flake check writes a file called bench.csv - we want Hercules to upload it somewhere. The check is run here: https://hercules-ci.com/accounts/github/Plutonomicon/derivations/%2Fnix%2Fstore%2F1nzsljyb7yv6fhjgg1p34kz1acrnm568-benchmark.drv/log?via-job=b6899b8f-ead0-43f6-90e0-5fba98df65c8

I vaguely recall seeing Hercules effects supporting artifact upload. Perhaps we can use that.

[MatthewCroughan]

We do not need to write the CSV in order to accomplish this. Doing so would not be a good benchmark either, as the benchmark might be effected by different processes running on the CI agent at a future time. Instead, we can simply perform two benchmarks in a single job, using two different revisions of the source code, which generates a new CSV, which is the diff of the two. None of this requires writing files to any location, it is just more stdout.

[srid]

the benchmark might be effected by different processes running on the CI agent at a future time.

Usually, yes - but for this project we don't care about that, because the specific benchmark metrics (code size, cpu/mem "budget") are calculated independent of the running machine's performance.

[srid]

Part of work: #144

[srid]

@emiflake @MatthewCroughan How do I get to the benchmark diff for a PR, say #173 ?

[emiflake]

How do I get to the benchmark diff for a PR

Hah, this is what I've been asking. But AFAICT we just need to drop the runIf. I'll quickly see if it works on a non-PR as well, if so I'll make a PR for it :)

[TotallyNotChase]

@srid is this still relevant?