Skip to content

ci: benchmark clean and incremental builds#1022

Draft
jaagut wants to merge 1 commit into
mainfrom
agent/build-benchmark-baseline
Draft

ci: benchmark clean and incremental builds#1022
jaagut wants to merge 1 commit into
mainfrom
agent/build-benchmark-baseline

Conversation

@jaagut

@jaagut jaagut commented Jul 17, 2026

Copy link
Copy Markdown
Member

Summary

Add a reproducible build benchmark for comparing clean and incremental ROS workspace builds on both CI architectures.

Proposed changes

  • Add a script that cleans the workspace, measures a clean build, touches an existing C++ source without changing its contents, and measures the incremental rebuild.
  • Publish timings and exit codes as JSON artifacts and in the GitHub step summary.
  • Run the benchmark on pull requests and through manual dispatch on x86-64 and ARM64 runners.

Impact

This establishes the baseline used by the stacked build-time optimization PRs. It does not change build flags or robot runtime behavior.

Validation

  • python3 -m py_compile scripts/benchmark_build.py
  • python3 scripts/benchmark_build.py --help
  • pixi run -e default ruff check scripts/benchmark_build.py
  • pixi run -e default ruff format --check scripts/benchmark_build.py
  • git diff --check

A complete clean/incremental workspace build is intentionally delegated to this PR's benchmark workflow.

@jaagut

jaagut commented Jul 17, 2026

Copy link
Copy Markdown
Member Author

Build optimization overview

All CI checks pass on the complete PR stack (x64 and ARM64 builds, benchmarks, and pre-commit).

Stack stage ARM64 clean ARM64 incremental x64 clean x64 incremental
Baseline 384.46 s 32.15 s 574.64 s 52.45 s
Ninja 407.52 s 29.33 s 601.22 s 48.32 s
Ninja + ccache 334.84 s 23.35 s 480.56 s 33.22 s
Robot profile 414.39 s 24.68 s 642.18 s 39.88 s

Compared with the baseline:

  • Ninja improved incremental builds by about 8–9%, while clean-build results were about 5% slower in this run.
  • Adding ccache produced the strongest result: clean builds improved by about 13–16% and incremental builds by about 27–37%.
  • The robot production profile still improved incremental builds by about 23–24%, but clean builds were about 8–12% slower. This is consistent with the extra compiler work from RelWithDebInfo; that profile is primarily a runtime/deployment tradeoff rather than a clean-build optimization.
  • The vendored domain_bridge Ninja output collision is fixed, and the full stacked configuration now builds successfully on both architectures.

These are single CI measurements, so small differences should be treated as runner noise; the larger ccache gains are the clearest signal.

@jaagut

jaagut commented Jul 18, 2026

Copy link
Copy Markdown
Member Author

Repeated benchmark results (3x)

The consolidated benchmark workflow passed on both architectures. Each configuration ran three clean/incremental pairs on the same runner per architecture; all build return codes were zero.

Configuration ARM64 clean mean ARM64 incremental mean x64 clean mean x64 incremental mean
Unix Makefiles baseline 357.76 s 30.62 s 557.55 s 51.32 s
Ninja 395.57 s 28.64 s 584.51 s 47.19 s
Ninja + ccache 166.70 s 24.31 s 269.71 s 39.74 s
Robot profile 186.05 s 25.07 s 296.32 s 39.46 s

The ccache means include one cold population and two warm rebuilds. The clean-build split makes the effect clearer:

Configuration ARM64 cold ARM64 warm mean x64 cold x64 warm mean
Ninja + ccache 344.26 s 77.92 s 547.81 s 130.65 s
Robot profile 403.60 s 77.28 s 630.79 s 129.09 s

Compared with the repeated Make baseline:

  • Ninja made clean builds 10.6% slower on ARM64 and 4.8% slower on x64, but incremental builds improved by 6.5% and 8.0%.
  • Ninja + ccache improved the three-run clean mean by 53.4% on ARM64 and 51.6% on x64. Once warm, clean rebuilds improved by 78.2% and 76.6%. Incremental builds improved by 20.6% and 22.6%.
  • The robot profile improved the clean mean by 48.0% on ARM64 and 46.9% on x64, and incremental builds by 18.1% and 23.1%. Its cold build remained about 15–17% slower than cold Ninja + ccache because RelWithDebInfo performs more compiler work; with a warm cache, both profiles converged to nearly the same clean-build time.

Conclusion: Ninja alone is a modest incremental-build improvement, while ccache is the dominant optimization for repeated clean and incremental development builds. The production profile remains a runtime/deployment choice rather than an additional compilation-speed optimization.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: 🆕 New

Development

Successfully merging this pull request may close these issues.

1 participant