ci: rebase #3235 onto main + shard the compressed allow-list runs - #3479
Conversation
Rebase + extension of #3235 onto current main (post-#3462): - PRs: uncompressed full suite in 4 shards; zstd1/lz4 allow-lists split templates vs no-templates (8 jobs). Push to main unchanged (3 configs). - Setup cuts from #3235: DB startup overlaps builds (start-databases), goose on host, salted optimized Go cache, DEBUG_GCFLAGS opt-out. - TESTS_ONLY composes with package-pure shards; -run-carrying shards and unknown shard names are rejected loudly. - Docker Hub pre-pull retries (#3410) folded into start-databases. - envd build-debug keeps the musl-static link (#3444), gcflags parametrized. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PR SummaryMedium Risk Overview Reviewed by Cursor Bugbot for commit fa87dd9. Bugbot is set up for automated code reviews on this repo. Configure here. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Run 30634340619 (uncompressed): shard1 361s vs shard2 654s of summed test time - shard2 had inherited the distro-family suite (207s alone). Moving Cache/Fuse/Unsupported to shard1 lands 504s vs 511s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Measured resultsRun 1 (30634340619, cold salted cache): all green, wall 10.4 min — 3.5 min of that was one shard queuing behind the same run's arm64/lint jobs at the 24-slot runner-group ceiling. Junit verification across the four uncompressed shards: 227 distinct top-level tests, each exactly once. Run 2 (30635246032, warm cache + rebalanced split): all green, wall 6.8 min vs ~10–12 min on main.
Setup cuts confirmed: Build Packages 1.7–1.8m → 0.6–0.8m warm (salted cache), Start Services 1.9–2.2m → 0.7–0.8m (DB overlap + goose-on-host). The shard-1 regex was rebalanced from run 1's junits (summed 361s/654s → 504s/511s). Notes for review:
🤖 Generated with Claude Code |
The backgrounded otel pull-retry discarded its exit status, and unlike the DB containers (whose pull failures surface via bounded health waits) the fallback was an implicit un-retried pull inside the collector's 30s health window. start-services now re-runs pull-retry synchronously before starting the collector; the tag is derived from the Makefile that runs the image so a bump can't strand the pre-pulls. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit ecd1865. Configure here.
New dependency of start-databases and start-services; without the filter entry a PR touching only this script would skip the suite that exercises it, unlike the already-listed start-service.sh. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
No bugs found beyond what has already been resolved (both prior Cursor findings — the otel pre-pull swallow and the missing pull-retry.sh path filter — are fixed in the current commits). This is a large, complex CI/test-sharding rewrite (19 files, new composite action, custom TEST_SHARD partitioning logic with regex-based name splits) that the PR itself flags as still needing a human decision on how it lands (adopt into #3235 or take over), so I'm deferring rather than shadow-approving.
Extended reasoning...
Overview
This PR reworks the integration-test CI pipeline: it adds a start-databases composite action to overlap DB container pulls/boots with Go builds, replaces the ClickHouse migrator-image build with host-side goose (migrate-host), introduces a shared scripts/pull-retry.sh for Docker Hub pull retries, parametrizes DEBUG_GCFLAGS across five package Makefiles to allow optimized-but-race-enabled CI builds, and adds an 8-way TEST_SHARD matrix (tests/integration/Makefile) that splits the uncompressed suite by package/name-prefix and the compressed allow-list runs by package. A handful of integration tests are also switched from real template builds / fixed sleeps to RequestTemplateWithoutBuild / polling.
Security risks
None of consequence — this is CI/test infrastructure only. No production code paths (API, orchestrator, envd) change behaviorally; the Makefile edits only affect local/CI debug-build compiler flags.
Level of scrutiny
This warrants human review despite being CI-only, because the correctness property that matters here (every test runs exactly once across shards, none dropped or duplicated) is easy to silently violate and hard to verify from a diff read alone. The author has already run this in CI twice and reported junit-verified full coverage (227 distinct tests, each exactly once), which is good evidence, but the PR description explicitly states this is a draft pending a maintainer's decision (dobrac) on whether to adopt it into #3235 or take it over — i.e., the author already expects a human in the loop before merge.
Other factors
Both previously-flagged Cursor Bugbot issues (otel pre-pull discarding its exit status, and pull-retry.sh missing from the integration-inputs path filter) are addressed in the current commits (ecd1865, fa87dd9). The sharding partition math (name-prefix regex split, package complements via go list) checked out on manual read, but given the size of the diff and that it's explicitly awaiting a maintainer call on how to land, I'm deferring rather than approving.
…atrix (#3498) Adds the premade NixOS image to `TestTemplateBuildDistroFamilies`, so CI catches a base image that doesn't boot. The tar-only publish gate in `nixos-base-image.yml` provably can't: the pre-shim 26.05 image passed every tar assertion and still froze at boot with no units. The profile installs nothing, so reaching ready proves exactly the parts that are ours — the busybox `Bootstrap`, the `/sbin/e2b-nixos-init` shim (#3478), and the `rm -rf /etc/systemd/system` that lets `setup-etc` take over. **Immutable tag, never `:latest`** — `phases/base/hash.go` keys the base layer on `Config.FromImage` as written, so a republished `:latest` would keep building from the stale cached layer and a failure wouldn't reproduce. **Cost, measured on this run.** Lands in `templates-builds-2` (`Distro` misses `TEMPLATE_BUILDS_SHARD1_RE`). The subtest is **156s** — 515 MB compressed / 1.05 GiB unpacked, cold pull every time. Package wall 166s → 255s, job **5.7 → 7.2 min** against a 30 min cap. It grows the wall by more than 156/4 because the pull and unpack also slow its three concurrent neighbours (utilisation 3.7× → 3.0×). That makes shard 2 the long pole against shard 1's 4m52s; leaving `TEMPLATE_BUILDS_SHARD1_RE` alone — that 2.3 min gap wants its own measuring run, the way #3479 did it. **`compression-tests.tsv`: no change.** The test is already absent; the documented criterion is reading a snapshot back, and provisioning is compression-invariant like the other four cases. Entries are per top-level test anyway, so a subtest couldn't be listed on its own.
2 participants
What
Rebase + extension of #3235 (cc @dobrac), squashed onto current main after #3462 landed. Opened as a draft for a measuring run — the shard timings from this PR's CI decide the final shard balance.
Two things changed since #3235 was written, and this branch reconciles both:
PR-path matrix (8 jobs; push to main unchanged: 3 unsharded configs)
^TestTemplateBuildsplit by name prefix-skip '^TestTemplateBuild'Static partition verified: 227 top-level tests land exactly once across the uncompressed shards; templates/no-templates is an exact package complement.
TESTS_ONLYcomposes only with package-pure shards (the name-split shards' own-runwould collide — guarded, fails loudly). Unknown shard names fail loudly instead of silently running the whole suite.Carried from #3235 (unchanged in spirit)
start-databases);start-serviceswaits with bounded health loops. Docker Hub pre-pull retries (ci: pre-pull Docker Hub images with retries in start-services #3410) folded in, now covering the pinned tags + otel.migrate-host) — no migrator image build.DEBUG_GCFLAGSopt-out (race stays on,-N -loff in CI). envd keeps the musl-static link from test(integration): run the Alpine distro build with a musl-capable envd #3444, gcflags parametrized.RequestTemplateWithoutBuild; envd process tests poll instead of fixed sleeps.Expected (to be confirmed by this run)
Baseline on main (run 30626752076): integration jobs 8.4–9.9 min, PR wall ~10–12 min. Expected here: ~5–5.5 min wall, long pole likely uncompressed-templates-2 (it inherited the distro suite). Follow-ups after measurement: rebalance
TEMPLATE_BUILDS_SHARD1_RE, cache the built sandbox template keyed on inputs.Relationship to #3235: this supersedes the mechanical parts; adopt into the original PR or take this one over — @dobrac's call.
🤖 Generated with Claude Code