Skip to content

ci: bound benchmark simulator setup and cleanup - #1630

Draft
mrousavy wants to merge 1 commit into
mainfrom
ci/bound-benchmark-simulator-lifecycle
Draft

ci: bound benchmark simulator setup and cleanup#1630
mrousavy wants to merge 1 commit into
mainfrom
ci/bound-benchmark-simulator-lifecycle

Conversation

@mrousavy

@mrousavy mrousavy commented Sep 7, 2026

Copy link
Copy Markdown
Member

Simulator creation and simctl boot can currently hang before the five-minute bootstatus timer starts. Put all three commands in one setup step with a five-minute Actions deadline, and give final simulator cleanup a one-minute deadline. Export the device ID immediately after creation so cleanup can run when boot fails or times out.

After successful setup, explicitly mark the newly created CI simulator as fresh. Skip preinstall terminate/uninstall calls for apps that cannot yet exist. Reused simulators keep that cleanup by default; each benchmark case still launches and terminates its own process.

Validation on c7576d288e314744bd3f52992924fe3a9cf74c2c: all GitHub checks passed, including TypeScript, lint, Nitrogen on both platforms, both native builds and both device measurement jobs. The iOS job completed all 92 measurements, created/booted the simulator in 38 seconds and deleted it in four seconds. The two expected missing-app termination errors seen in unchanged runs are absent.

Local tests cover fresh/reused installations, paired and same-SHA runs, invalid freshness flags and per-case process isolation. Workflow checks verify that create/boot/bootstatus share the setup deadline and final cleanup has its own deadline; GitHub Actions enforces the timeouts. TypeScript, actionlint, ShellCheck and diff checks pass. Rebase patch equivalence was verified with git range-diff.

Timing review:

Run Entire iOS measurement job Boot-ready to first app launch
Current PR 7m00s 6.61s
Unchanged iOS controller, #1628 7m18s 7.81s
Unchanged iOS controller, #1629 7m01s 8.46s
PR before rebase Cancelled by rebase 24.53s

The current run and controls share the same base and iOS runner/toolchain. These are separate hosted runs, and app extraction moved to after boot in this PR. The latest run is slightly shorter, but startup variation prevents attributing a reliable overall speedup to the change. The demonstrated benefits are bounded setup/cleanup and removing unnecessary commands/errors. Leave open under the requested works-and-speeds-up-CI merge condition.

@vercel

vercel Bot commented Sep 7, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nitro-docs Ready Ready Preview Sep 7, 2026 7:03pm UTC

Request Review

@nitro-modules-bot

Copy link
Copy Markdown

Performance Report

⚠️ Advisory: Results do not fail this PR.

iOS

Benchmark Before After Difference
C++ bounce(1 MiB) 232.7 ns 263.1 ns 🔴 +13.05% slower
C++ large array (1,024) 48.28 µs 45.62 µs 🟢 -5.51% faster
C++ deferred worker Promise (includes trigger call) 26.93 µs 24.80 µs 🟢 -7.9% faster
Swift copy(1 MiB) 59.03 µs 74.93 µs 🔴 +26.93% slower
Swift synchronousCallback() 242.6 ns 223.2 ns 🟢 -7.97% faster
Swift create() 26.41 µs 23.21 µs 🟢 -12.12% faster
Swift returnExisting() 232.1 ns 250.2 ns 🔴 +7.77% slower
Swift simpleFunc() 60.7 ns 67.8 ns 🔴 +11.77% slower
Swift deferred worker Promise (includes trigger call) 23.82 µs 28.37 µs 🔴 +19.14% slower
Swift nested Car struct 2.38 µs 2.52 µs 🔴 +5.85% slower
All Benchmarks
Benchmark Before After Difference
JavaScript addNumbers() 22.0 ns 22.7 ns 🔴 +2.74% slower
C++ bounce(4 KiB) 240.0 ns 239.6 ns 🟢 -0.19% faster
C++ bounce native-owned buffer (1 MiB) 3.74 µs 3.73 µs 🟢 -0.43% faster
C++ bounce native-owned buffer (4 KiB) 397.0 ns 388.4 ns 🟢 -2.16% faster
C++ copy(1 MiB) 83.40 µs 85.30 µs 🔴 +2.29% slower
C++ copy(4 KiB) 1.10 µs 1.15 µs 🔴 +3.8% slower
C++ small array (16) 906.6 ns 894.2 ns 🟢 -1.37% faster
C++ synchronousCallback() 214.1 ns 204.7 ns 🟢 -4.4% faster
C++ create() 431.4 ns 423.0 ns 🟢 -1.93% faster
C++ returnExisting() 202.8 ns 206.5 ns 🔴 +1.8% slower
C++ typed map (8 entries) 1.67 µs 1.66 µs 🟢 -0.18% faster
C++ optional trailing string 147.7 ns 146.9 ns 🟢 -0.53% faster
C++ addNumbers() 84.9 ns 87.4 ns 🔴 +2.99% slower
C++ simpleFunc() 63.8 ns 65.0 ns 🔴 +1.94% slower
C++ immediatePromise() 902.0 ns 902.8 ns 🔴 +0.09% slower
C++ number property get/set 133.2 ns 134.5 ns 🔴 +0.98% slower
C++ short ASCII string 137.4 ns 135.4 ns 🟢 -1.43% faster
C++ Unicode string 284.4 ns 284.6 ns 🔴 +0.05% slower
C++ nested Car struct 2.24 µs 2.30 µs 🔴 +2.64% slower
C++ number | string variant 127.7 ns 126.9 ns 🟢 -0.56% faster
Swift bounce(1 MiB) 240.0 ns 237.1 ns 🟢 -1.19% faster
Swift bounce(4 KiB) 242.8 ns 242.0 ns 🟢 -0.29% faster
Swift bounce native-owned buffer (1 MiB) 3.87 µs 3.84 µs 🟢 -0.73% faster
Swift bounce native-owned buffer (4 KiB) 384.9 ns 398.5 ns 🔴 +3.53% slower
Swift copy(4 KiB) 1.22 µs 1.22 µs 🟢 -0.39% faster
Swift large array (1,024) 47.91 µs 48.04 µs 🔴 +0.26% slower
Swift small array (16) 994.0 ns 1.03 µs 🔴 +3.78% slower
Swift typed map (8 entries) 2.40 µs 2.33 µs 🟢 -2.76% faster
Swift optional trailing string 156.0 ns 152.9 ns 🟢 -1.95% faster
Swift addNumbers() 78.2 ns 77.9 ns 🟢 -0.36% faster
Swift immediatePromise() 1.29 µs 1.26 µs 🟢 -2.72% faster
Swift number property get/set 137.2 ns 138.0 ns 🔴 +0.59% slower
Swift short ASCII string 165.6 ns 159.0 ns 🟢 -4% faster
Swift Unicode string 347.3 ns 345.0 ns 🟢 -0.66% faster
Swift number | string variant 193.2 ns 187.3 ns 🟢 -3.02% faster
TurboModule addNumbers() 867.7 ns 906.7 ns 🔴 +4.49% slower

Android

Benchmark Before After Difference
Kotlin bounce(1 MiB) 1.18 µs 1.03 µs 🟢 -12.64% faster
Kotlin copy(1 MiB) 303.66 µs 324.14 µs 🔴 +6.74% slower
Kotlin create() 1.55 µs 1.64 µs 🔴 +5.85% slower
All Benchmarks
Benchmark Before After Difference
JavaScript addNumbers() 22.1 ns 21.2 ns 🟢 -4.09% faster
C++ bounce(1 MiB) 435.6 ns 435.9 ns 🔴 +0.05% slower
C++ bounce(4 KiB) 435.9 ns 432.2 ns 🟢 -0.85% faster
C++ bounce native-owned buffer (1 MiB) 3.42 µs 3.41 µs 🟢 -0.1% faster
C++ bounce native-owned buffer (4 KiB) 451.5 ns 444.6 ns 🟢 -1.53% faster
C++ copy(1 MiB) 19.19 µs 19.01 µs 🟢 -0.92% faster
C++ copy(4 KiB) 795.1 ns 770.1 ns 🟢 -3.15% faster
C++ large array (1,024) 34.48 µs 34.95 µs 🔴 +1.36% slower
C++ small array (16) 740.0 ns 752.0 ns 🔴 +1.63% slower
C++ synchronousCallback() 324.6 ns 312.3 ns 🟢 -3.8% faster
C++ create() 541.1 ns 532.8 ns 🟢 -1.53% faster
C++ returnExisting() 214.4 ns 217.5 ns 🔴 +1.4% slower
C++ typed map (8 entries) 1.64 µs 1.70 µs 🔴 +3.6% slower
C++ optional trailing string 142.0 ns 141.1 ns 🟢 -0.63% faster
C++ addNumbers() 86.6 ns 86.7 ns 🔴 +0.09% slower
C++ simpleFunc() 71.9 ns 71.5 ns 🟢 -0.57% faster
C++ deferred worker Promise (includes trigger call) 23.29 µs 23.71 µs 🔴 +1.78% slower
C++ immediatePromise() 826.8 ns 825.5 ns 🟢 -0.15% faster
C++ number property get/set 154.1 ns 153.4 ns 🟢 -0.43% faster
C++ short ASCII string 138.7 ns 136.9 ns 🟢 -1.28% faster
C++ Unicode string 251.0 ns 246.4 ns 🟢 -1.83% faster
C++ nested Car struct 2.08 µs 2.06 µs 🟢 -0.73% faster
C++ number | string variant 128.8 ns 129.2 ns 🔴 +0.29% slower
Kotlin bounce(4 KiB) 1.02 µs 1.04 µs 🔴 +1.08% slower
Kotlin bounce native-owned buffer (1 MiB) 4.05 µs 4.06 µs 🔴 +0.38% slower
Kotlin bounce native-owned buffer (4 KiB) 1.13 µs 1.14 µs 🔴 +0.26% slower
Kotlin copy(4 KiB) 5.18 µs 4.98 µs 🟢 -3.89% faster
Kotlin large array (1,024) 38.82 µs 39.67 µs 🔴 +2.18% slower
Kotlin small array (16) 990.2 ns 983.4 ns 🟢 -0.69% faster
Kotlin synchronousCallback() 1.14 µs 1.13 µs 🟢 -1.05% faster
Kotlin returnExisting() 565.1 ns 569.2 ns 🔴 +0.73% slower
Kotlin typed map (8 entries) 6.20 µs 6.23 µs 🔴 +0.52% slower
Kotlin optional trailing string 320.1 ns 325.6 ns 🔴 +1.73% slower
Kotlin addNumbers() 113.6 ns 114.9 ns 🔴 +1.19% slower
Kotlin simpleFunc() 93.4 ns 93.0 ns 🟢 -0.5% faster
Kotlin deferred worker Promise (includes trigger call) 41.69 µs 43.20 µs 🔴 +3.62% slower
Kotlin immediatePromise() 2.40 µs 2.33 µs 🟢 -2.98% faster
Kotlin number property get/set 222.5 ns 221.8 ns 🟢 -0.32% faster
Kotlin short ASCII string 439.2 ns 438.5 ns 🟢 -0.17% faster
Kotlin Unicode string 722.0 ns 725.9 ns 🔴 +0.54% slower
Kotlin nested Car struct 4.51 µs 4.42 µs 🟢 -2.05% faster
Kotlin number | string variant 386.6 ns 387.3 ns 🔴 +0.16% slower
TurboModule addNumbers() 226.0 ns 224.0 ns 🟢 -0.87% faster

Benchmarking Code Diff 2e815f49...c7576d28 (view raw output)

Raw measurements: performance-report-1 (JSON artifact). Run 34154047323, attempt 1. Download requires GitHub access.

Android: measurements, attempt 1, apps, attempt 1.
iOS: measurements, attempt 1, apps, attempt 1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant