Skip to content

ci: cache benchmark gems for both revisions - #1627

Draft
mrousavy wants to merge 1 commit into
ci/ruby-3.3.12from
ci/cache-benchmark-ruby-gems
Draft

ci: cache benchmark gems for both revisions#1627
mrousavy wants to merge 1 commit into
ci/ruby-3.3.12from
ci/cache-benchmark-ruby-gems

Conversation

@mrousavy

@mrousavy mrousavy commented Sep 7, 2026

Copy link
Copy Markdown
Member

Performance CI currently installs benchmark gems from scratch in both checkouts. Enable ruby/setup-ruby's Bundler cache separately for head and base, letting the action key each cache by its checkout path, Ruby version and Gemfile.lock. Identical revisions skip the base setup because they reuse the head app.

The build script now checks the installed bundle before falling back to installation, so CI reuses the action's dependencies and standalone local builds still install missing gems.

Stacked on #1624 for the Ruby 3.3.12 upgrade.

Validation:

  • Built Ruby 3.3.12 locally and used the locked Bundler 2.3.22 to install both repository Gemfiles in deployment mode; bundle check and CocoaPods 1.16.2 passed without changing either lockfile.
  • Verified separate checkout bundle paths retain different locked gem versions simultaneously (benchmark 0.4.1 versus 0.4.0 in temporary fixtures).
  • Actionlint workflow syntax/expressions, ShellCheck, Bash syntax and git diff --check pass. Existing Blacksmith runner labels were excluded from actionlint; no new shellcheck diagnostics were introduced.

Hosted cache restore performance still needs a warm CI run; the first run can populate the caches.

Before the refresh onto the publisher rollout on main, commit 040b31c26 passed hosted validation: https://github.com/margelo/nitro/actions/runs/34153344045/job/101840132508. Both checkout caches uploaded successfully, both bundle check calls reused installed dependencies, and both simulator apps built successfully. These results validate the previous revision; the refreshed revision receives new hosted checks.

@vercel

vercel Bot commented Sep 7, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nitro-docs Ready Ready Preview Sep 7, 2026 7:03pm UTC

Request Review

@nitro-modules-bot

Copy link
Copy Markdown

Performance Report

⚠️ Advisory: Results do not fail this PR.

iOS

Benchmark Before After Difference
C++ copy(4 KiB) 821.9 ns 760.8 ns 🟢 -7.43% faster
C++ large array (1,024) 45.46 µs 43.01 µs 🟢 -5.37% faster
C++ small array (16) 934.0 ns 868.2 ns 🟢 -7.05% faster
Swift bounce native-owned buffer (1 MiB) 3.61 µs 3.43 µs 🟢 -5.01% faster
Swift copy(1 MiB) 60.31 µs 69.41 µs 🔴 +15.07% slower
Swift copy(4 KiB) 972.4 ns 1.05 µs 🔴 +8.13% slower
Swift deferred worker Promise (includes trigger call) 24.27 µs 26.00 µs 🔴 +7.15% slower
All Benchmarks
Benchmark Before After Difference
JavaScript addNumbers() 21.3 ns 21.3 ns 🟢 -0.02% faster
C++ bounce(1 MiB) 217.1 ns 214.2 ns 🟢 -1.35% faster
C++ bounce(4 KiB) 222.2 ns 217.9 ns 🟢 -1.96% faster
C++ bounce native-owned buffer (1 MiB) 3.42 µs 3.38 µs 🟢 -1.19% faster
C++ bounce native-owned buffer (4 KiB) 345.6 ns 351.2 ns 🔴 +1.61% slower
C++ copy(1 MiB) 45.80 µs 46.10 µs 🔴 +0.65% slower
C++ synchronousCallback() 181.4 ns 173.0 ns 🟢 -4.62% faster
C++ create() 388.6 ns 377.1 ns 🟢 -2.96% faster
C++ returnExisting() 192.2 ns 191.0 ns 🟢 -0.65% faster
C++ typed map (8 entries) 1.64 µs 1.65 µs 🔴 +0.87% slower
C++ optional trailing string 139.6 ns 139.1 ns 🟢 -0.34% faster
C++ addNumbers() 82.4 ns 85.7 ns 🔴 +3.92% slower
C++ simpleFunc() 61.8 ns 61.4 ns 🟢 -0.54% faster
C++ deferred worker Promise (includes trigger call) 17.78 µs 17.86 µs 🔴 +0.43% slower
C++ immediatePromise() 754.3 ns 731.1 ns 🟢 -3.07% faster
C++ number property get/set 130.7 ns 131.4 ns 🔴 +0.54% slower
C++ short ASCII string 129.2 ns 128.4 ns 🟢 -0.68% faster
C++ Unicode string 279.2 ns 273.4 ns 🟢 -2.11% faster
C++ nested Car struct 2.20 µs 2.18 µs 🟢 -0.82% faster
C++ number | string variant 121.3 ns 122.2 ns 🔴 +0.73% slower
Swift bounce(1 MiB) 238.7 ns 233.4 ns 🟢 -2.2% faster
Swift bounce(4 KiB) 239.8 ns 239.5 ns 🟢 -0.16% faster
Swift bounce native-owned buffer (4 KiB) 389.2 ns 389.4 ns 🔴 +0.06% slower
Swift large array (1,024) 46.71 µs 47.18 µs 🔴 +1.02% slower
Swift small array (16) 951.2 ns 939.5 ns 🟢 -1.23% faster
Swift synchronousCallback() 240.2 ns 232.4 ns 🟢 -3.26% faster
Swift create() 25.06 µs 26.17 µs 🔴 +4.4% slower
Swift returnExisting() 253.9 ns 242.0 ns 🟢 -4.69% faster
Swift typed map (8 entries) 2.34 µs 2.32 µs 🟢 -0.73% faster
Swift optional trailing string 152.1 ns 155.3 ns 🔴 +2.09% slower
Swift addNumbers() 68.8 ns 71.0 ns 🔴 +3.28% slower
Swift simpleFunc() 54.6 ns 54.4 ns 🟢 -0.39% faster
Swift immediatePromise() 1.25 µs 1.26 µs 🔴 +0.66% slower
Swift number property get/set 129.1 ns 130.2 ns 🔴 +0.8% slower
Swift short ASCII string 147.5 ns 149.5 ns 🔴 +1.38% slower
Swift Unicode string 327.5 ns 329.4 ns 🔴 +0.57% slower
Swift nested Car struct 2.45 µs 2.39 µs 🟢 -2.42% faster
Swift number | string variant 196.8 ns 199.4 ns 🔴 +1.31% slower
TurboModule addNumbers() 895.2 ns 900.3 ns 🔴 +0.57% slower

Android

Benchmark Before After Difference
C++ bounce native-owned buffer (1 MiB) 3.24 µs 3.06 µs 🟢 -5.74% faster
C++ create() 487.8 ns 455.5 ns 🟢 -6.63% faster
Kotlin copy(4 KiB) 4.65 µs 4.38 µs 🟢 -5.88% faster
Kotlin synchronousCallback() 1.02 µs 955.5 ns 🟢 -6.13% faster
Kotlin create() 1.39 µs 1.58 µs 🔴 +13.31% slower
Kotlin deferred worker Promise (includes trigger call) 37.98 µs 26.51 µs 🟢 -30.18% faster
All Benchmarks
Benchmark Before After Difference
JavaScript addNumbers() 20.6 ns 19.9 ns 🟢 -3.37% faster
C++ bounce(1 MiB) 385.4 ns 395.8 ns 🔴 +2.68% slower
C++ bounce(4 KiB) 399.8 ns 382.4 ns 🟢 -4.34% faster
C++ bounce native-owned buffer (4 KiB) 403.2 ns 400.7 ns 🟢 -0.62% faster
C++ copy(1 MiB) 23.79 µs 24.92 µs 🔴 +4.74% slower
C++ copy(4 KiB) 698.3 ns 670.1 ns 🟢 -4.04% faster
C++ large array (1,024) 34.66 µs 34.61 µs 🟢 -0.13% faster
C++ small array (16) 748.7 ns 720.8 ns 🟢 -3.73% faster
C++ synchronousCallback() 278.4 ns 265.0 ns 🟢 -4.82% faster
C++ returnExisting() 195.5 ns 202.0 ns 🔴 +3.31% slower
C++ typed map (8 entries) 1.52 µs 1.54 µs 🔴 +0.94% slower
C++ optional trailing string 138.4 ns 138.4 ns 🔴 +0.04% slower
C++ addNumbers() 83.6 ns 83.1 ns 🟢 -0.65% faster
C++ simpleFunc() 68.8 ns 68.2 ns 🟢 -0.86% faster
C++ deferred worker Promise (includes trigger call) 20.00 µs 20.58 µs 🔴 +2.93% slower
C++ immediatePromise() 778.2 ns 811.8 ns 🔴 +4.32% slower
C++ number property get/set 146.9 ns 141.3 ns 🟢 -3.81% faster
C++ short ASCII string 122.2 ns 126.9 ns 🔴 +3.88% slower
C++ Unicode string 231.8 ns 225.9 ns 🟢 -2.56% faster
C++ nested Car struct 1.87 µs 1.89 µs 🔴 +1.09% slower
C++ number | string variant 123.9 ns 119.9 ns 🟢 -3.2% faster
Kotlin bounce(1 MiB) 964.9 ns 966.3 ns 🔴 +0.14% slower
Kotlin bounce(4 KiB) 961.8 ns 929.7 ns 🟢 -3.34% faster
Kotlin bounce native-owned buffer (1 MiB) 3.81 µs 3.82 µs 🔴 +0.45% slower
Kotlin bounce native-owned buffer (4 KiB) 1.04 µs 1.01 µs 🟢 -2.64% faster
Kotlin copy(1 MiB) 219.00 µs 224.76 µs 🔴 +2.63% slower
Kotlin large array (1,024) 39.18 µs 40.73 µs 🔴 +3.94% slower
Kotlin small array (16) 987.3 ns 981.0 ns 🟢 -0.64% faster
Kotlin returnExisting() 530.6 ns 515.1 ns 🟢 -2.91% faster
Kotlin typed map (8 entries) 5.66 µs 5.66 µs 🟢 -0.02% faster
Kotlin optional trailing string 291.2 ns 293.4 ns 🔴 +0.74% slower
Kotlin addNumbers() 113.8 ns 115.4 ns 🔴 +1.35% slower
Kotlin simpleFunc() 88.5 ns 87.7 ns 🟢 -0.86% faster
Kotlin immediatePromise() 2.29 µs 2.34 µs 🔴 +1.86% slower
Kotlin number property get/set 206.9 ns 208.1 ns 🔴 +0.6% slower
Kotlin short ASCII string 404.1 ns 407.3 ns 🔴 +0.8% slower
Kotlin Unicode string 660.2 ns 666.8 ns 🔴 +1% slower
Kotlin nested Car struct 4.09 µs 4.07 µs 🟢 -0.32% faster
Kotlin number | string variant 365.3 ns 369.5 ns 🔴 +1.13% slower
TurboModule addNumbers() 195.9 ns 195.9 ns 🔴 +0.03% slower

Benchmarking Code Diff c0df74b5...921a978c (view raw output)

Raw measurements: performance-report-1 (JSON artifact). Run 34154015570, attempt 1. Download requires GitHub access.

Android: measurements, attempt 1, apps, attempt 1.
iOS: measurements, attempt 1, apps, attempt 1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant