mechababs test-cluster: validate a cluster from the campaign

[lobennett]

mechababs test-cluster: validate a cluster from the campaign

What this is

mechababs test-cluster --cluster <config>, run from the campaign venv. Validating a cluster config was a repo-dev operation — checkout, pip install -e '.[test]', MECHABABS_E2E_WORKDIR, BABS_SPEC, pytest by hand. The campaign already carries every one of those, so validation moves to the operate side, next to iterate and status. The container shim stays the only prerequisite, and it is now checked before pytest starts instead of surfacing as a skip.

How it works

• It does not run in the campaign you point at. The scenario configures a campaign, adds a dataset, and retires a derivative, so it provisions its own throwaway campaign from the target campaign's pins and works there. The campaign supplies the environment, not the workspace — and because the pins come from it, a green run means those exact tools work on this cluster.
• One provisioning input, so dev-exercises-prod holds. Adding --campaign beside the existing checkout route would have left the user-facing path exercised only by users. I had it that way briefly and it drifted — a detached-HEAD fix landed on one route and not the other. So --campaign is required and the checkout route is gone: a dev run reaches the same code the way a user does, bootstrapping a dev campaign pinned at the checkout (87s of a 40-minute run). The two rungs still differ only in which cluster config and whether a container wraps them.
• Fails loudly. pytest exits 0 when everything skips, so the fixtures raise instead. Reporting success having validated nothing is the failure mode worth engineering against.
• Leaves the campaign clean. -p no:cacheprovider plus a redirected PYTHONPYCACHEPREFIX: a .pytest_cache/ inside code/mechababs reads as a dirty pin to guard.require_clean_pins, and one early run left a campaign refusing every later command.
• Takes the skeleton, not the ledger. configure writes the ledger, so gating on it made bootstrap → validate → configure impossible. Both commands now share one "campaign enough to run" check.

The suite moved into the package

Your plan had test-cluster read the vendored code/mechababs/tests/e2e. This ships it as package data instead, resolved through importlib.resources — option 1 from the issue thread. Under #101 the tools arrive via uvx --from git+… against a uv.lock, so there is no clone to reach into and tests/ is not in a wheel; packaging is the part that survives.

Two consequences. The SimBIDS phantom pipelines move out of examples/pipelines/ into the suite that consumes them — this reverses that bit of #100; they are fixtures, and a packaged suite cannot reach a fixture outside the package. And testpaths becomes load-bearing: without it a bare pytest also walks mechababs/, where the packaged conftest registers --cluster-config twice and collection fails. exclude-package-data keeps the dev wrappers out of the wheel, since package-data globs are additive rather than restrictive once setuptools_scm's file finder is involved.

Say the word if you would rather it stay in tests/ until #101 lands. The command only needs some way to find the scenario.

Changes

• mechababs/validate.py — the command body: cluster resolution, re-clonable ref resolution, the shim precheck, the pytest invocation.
• mechababs/cli.py — the test-cluster subcommand; the skeleton and campaign-venv guards extracted so configure and test-cluster share one copy.
• mechababs/testing/ — the e2e scenario, shipped as package data and located through importlib.
• requirements-campaign.txt — pytest in the campaign venv.
• docs — the tutorial leads with the campaign route (its ASPIRATIONAL block is gone), reference.md documents the subcommand, installation.md drops the driver venv.

Five commits, each green on its own.

Verified

• Real cluster (Sherlock login node, real slurm): 3 passed (38m54s) — bootstrap → configure → add-dataset → iterate: scaffold → submit → wait → merge, both pipelines, derivative zips landing in the output RIA.
• In-container (rootless podman): 2 passed, 1 failed. The failure is allow full BIDS-acceptable chars as derivative names PennLINC/babs#394 (grep -E reads the + in a zip name as a metacharacter). The cluster run's third test is green only because it pinned a patched babs, so 2/3 is the honest number on babs main.
• Unit: 58 passed.

The podman rung earned its keep: once the conftest imported mechababs.validate it could no longer load under the container's ambient pytest — invisible on a cluster, where the campaign venv is what runs. It now uses the dev campaign's own pytest.

run_in_podman.sh is the one change no full run covers. Its ${A[@]+"${A[@]}"} fix — an empty array aborts under set -u on bash before 4.4, so the script could not start on a RHEL 7 login node — is verified only in isolation and with bash -n.

What #101 will unlearn

The durable pieces are the command surface, the packaged suite, and having one provisioning input. What moves is lookup, each in a single place: _pins_from_campaign reads a vendored clone's branch and will read uv.lock; the skeleton check tests .datalad + code/<tool>, which a study-first campaign explicitly will not have. The campaign-venv guard is your open 💬 on environment enforcement — this reuses it rather than deciding it.

Closes #98

[asmacdo]

Without #101 the UX isn't as clean as I'd like it to end up, but this is a move in the right direction. I ran on-cluster and it worked well. Thanks @lobennett!!