Skip to content

Commit

Permalink
Fix benchmark that does not compare the exact result from the previou…
Browse files Browse the repository at this point in the history
…s commit (open-telemetry#5664)

Part of open-telemetry#4537

The benchmark action does not compare the exact result from the previous
commit. For instance, a recent action
https://github.com/open-telemetry/opentelemetry-go/actions/runs/10175392022
compares 2a454a7, which is a commit
that pushed a week ago.

This is because `actions/cache@v4` will not refresh the cache when the
key exists.

This is the logs from
https://github.com/open-telemetry/opentelemetry-go/actions/runs/10175392022/job/28170968531
on `Post Download previous benchmark data` step.

```
Cache hit occurred on the primary key Linux-benchmark, not saving cache.
```

Moreover, GitHub action only invalidates the key that has not been
accessed in over 7 days, which means forever in our case as long as we
push a new commit to the main branch.


https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#usage-limits-and-eviction-policy

---

To fix this, I use the current sha as the cache key and then use the
previous sha to fetch the cache. So, our benchmark can compare the exact
result from the previous commit.

Action result from my fork:
https://github.com/XSAM/opentelemetry-go/actions/runs/10189773887/job/28188416879
  • Loading branch information
XSAM authored Aug 1, 2024
1 parent 9e5b316 commit f7977e0
Showing 1 changed file with 9 additions and 1 deletion.
10 changes: 9 additions & 1 deletion .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,15 @@ jobs:
uses: actions/cache@v4
with:
path: ./benchmarks
key: ${{ runner.os }}-benchmark
# Use the current commit SHA as the cache key.
# This key won't exist on the first run, so the cache match falls back to restore-keys.
# Though, it won't be matched, the cache created will use this key as the cache key.
# So the next commit will be able to restore this cache (from the restore-keys).
# https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#matching-a-cache-key
key: ${{ runner.os }}-benchmark-${{ github.sha }}
# `github.event.before` means the commit before the push (i.e. the previous commit).
# So we can fetch the exact benchmark data from the previous commit.
restore-keys: ${{ runner.os }}-benchmark-${{ github.event.before }}
- name: Store benchmarks result
uses: benchmark-action/[email protected]
with:
Expand Down

0 comments on commit f7977e0

Please sign in to comment.