From f7977e064c9cfc862c24ac720036d5aad9911794 Mon Sep 17 00:00:00 2001 From: Sam Xie Date: Thu, 1 Aug 2024 16:39:05 -0700 Subject: [PATCH] Fix benchmark that does not compare the exact result from the previous commit (#5664) Part of #4537 The benchmark action does not compare the exact result from the previous commit. For instance, a recent action https://github.com/open-telemetry/opentelemetry-go/actions/runs/10175392022 compares 2a454a776a20dbe5fab008d2c4100c83ea572d97, which is a commit that pushed a week ago. This is because `actions/cache@v4` will not refresh the cache when the key exists. This is the logs from https://github.com/open-telemetry/opentelemetry-go/actions/runs/10175392022/job/28170968531 on `Post Download previous benchmark data` step. ``` Cache hit occurred on the primary key Linux-benchmark, not saving cache. ``` Moreover, GitHub action only invalidates the key that has not been accessed in over 7 days, which means forever in our case as long as we push a new commit to the main branch. https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#usage-limits-and-eviction-policy --- To fix this, I use the current sha as the cache key and then use the previous sha to fetch the cache. So, our benchmark can compare the exact result from the previous commit. Action result from my fork: https://github.com/XSAM/opentelemetry-go/actions/runs/10189773887/job/28188416879 --- .github/workflows/benchmark.yml | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/.github/workflows/benchmark.yml b/.github/workflows/benchmark.yml index 7a9bdecac2e..d314e06c7af 100644 --- a/.github/workflows/benchmark.yml +++ b/.github/workflows/benchmark.yml @@ -24,7 +24,15 @@ jobs: uses: actions/cache@v4 with: path: ./benchmarks - key: ${{ runner.os }}-benchmark + # Use the current commit SHA as the cache key. + # This key won't exist on the first run, so the cache match falls back to restore-keys. + # Though, it won't be matched, the cache created will use this key as the cache key. + # So the next commit will be able to restore this cache (from the restore-keys). + # https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#matching-a-cache-key + key: ${{ runner.os }}-benchmark-${{ github.sha }} + # `github.event.before` means the commit before the push (i.e. the previous commit). + # So we can fetch the exact benchmark data from the previous commit. + restore-keys: ${{ runner.os }}-benchmark-${{ github.event.before }} - name: Store benchmarks result uses: benchmark-action/github-action-benchmark@v1.20.3 with: