feat(experimentalist): run in OpenShell with an external Harbor bridge

[ryana]

Summary

• make OpenShell the only host-side execution path for nemo experimentalist run and doctor
• run the Experimentalist control plane as a non-root OpenShell sandbox with no Docker CLI or Docker socket
• route Harbor evaluation through a narrow authenticated bridge that owns the host Docker client
• add scoped OpenShell provider profiles for inference, the Harbor bridge, GitHub, and GitLab
• package the git, gh, and glab clients that Experimentalist invokes directly
• separate read-only source access from branch-push and draft PR/MR publication authority
• keep the Dockerfile and packaged OpenShell runtime assets with the Experimentalist plugin

There is no --runtime option, host-execution mode, or automatic local fallback. The in-process CLI path is enabled only inside the marked Experimentalist container image.

Implemented trust boundary

1. The host CLI acquires a compatible Experimentalist image and launches OpenShell.
2. Experimentalist mutates and orchestrates candidates inside the marked container.
3. Model calls use gateway-managed inference.local; the inference credential is not attached to the sandbox.
4. OpenShell exposes randomized placeholders for the bridge and one selected source-control provider, then replaces them in allowed proxied requests.
5. Experimentalist submits a typed request plus candidate and dataset archives to the local bridge.
6. The bridge owns Harbor and the host Docker client. It never imports candidate Python into the Docker-owning process.
7. A fixed trusted Harbor adapter uploads the candidate into each task container and runs the fixed python -m main entrypoint.
8. Harbor verifiers are forced into separate containers.

The sandbox cannot select a Harbor import path, Docker command, image runtime flag, verifier mode, or arbitrary bridge command.

Default command and image behavior

nemo experimentalist run and nemo experimentalist doctor always dispatch through OpenShell on the host. Existing local inputs are translated into the uploaded sandbox workspace; paths outside the selected working directory fail closed. Experiment output stays host-side and is downloaded after the sandbox run.

The CLI uses local/nmp-experimentalist:local by default. It reuses the image only when the required runtime compatibility label is present; otherwise a source checkout builds the current host architecture through the existing Bake target. NEMO_EXPERIMENTALIST_IMAGE can select a published or differently tagged image.

Source-control clients and providers

Experimentalist directly calls:

• git for clone, fetch, checkout, commit, and push
• gh auth status and gh pr create
• glab auth status and glab mr create

The image includes all three clients. Git uses GIT_ASKPASS to receive only an OpenShell placeholder for HTTPS authentication.

The launcher supports none, github-read, github-publish, gitlab-read, and gitlab-publish. It attaches at most one source-control provider to a sandbox. Publish profiles add only HTTPS branch push and draft PR/MR creation to their read counterparts.

Plan forward

This MR is the research-preview implementation for AIRE-823. The local bridge is deliberately small and lives with the plugin so we can validate the boundary before committing to a platform service.

For production, preserve this API shape but move it to a dedicated NeMo Evaluator worker. Add approved task sources/images, per-job CPU/memory/storage/time quotas, scoped task-network policy, short-lived inference credentials, durable job state/cancellation, and audit records.

Remaining risk

The bridge process still has host-root-equivalent Docker authority. Harbor tasks may contain arbitrary Dockerfiles, and candidate code receives its inference credential inside the Harbor task container. This removes Docker authority from Experimentalist but does not defend against Docker/Harbor escapes, resource exhaustion, or credential exfiltration from the candidate container. Those are release-gating concerns for production, not claims this preview makes.

Docker Desktop also lacks Landlock in the tested environment. The explicit Docker Desktop policy retains process and network enforcement but marks filesystem isolation best-effort; the strict policy fails closed when Landlock is unavailable.

OpenShell's current static-provider placeholder resolution is sandbox-wide rather than cryptographically bound to a target hostname. Attaching only one source-control provider prevents GitHub-to-GitLab token substitution. A source-control placeholder can still be presented to the trusted bridge endpoint, and the bridge placeholder can be presented to the selected source host; neither endpoint reflects the resolved Authorization header. Target-bound placeholder resolution in OpenShell would strengthen this defense.

OpenShell applies .gitignore filtering to uploads inside Git repositories, but the agent can read every uploaded file. Secrets must remain ignored or provider-managed rather than tracked in the selected workspace.

Git LFS, SSH Git remotes, GitHub Enterprise, and registry download are outside the current policy.

Validation

• full Experimentalist test suite: 578 passed
• repository pre-commit suite: passed, including Ruff, formatting, ty, lock drift, licenses, policy checks, and UI lint
• built the plugin wheel and verified it contains the OpenShell launcher, policies, provider setup, askpass helper, and provider profiles with executable script modes
• verified the real host CLI has no --runtime option and fails closed when OpenShell is unavailable
• built, loaded, and ran the final runtime image for both linux/arm64 and linux/amd64
• on both architectures, verified the image marker, non-root execution, git/gh/glab availability, absence of Docker, and the container-only in-process CLI path
• verified inside OpenShell that the sandbox receives provider placeholders rather than the real credentials
• completed a real OpenShell -> bridge -> Harbor evaluation with the candidate in the task container and verifier in a separate container; reward 1.0
• verified gh auth status and authenticated HTTPS git ls-remote through the GitHub read provider
• verified a GitHub-attached sandbox is denied access to GitLab
• verified the GitLab read policy reaches the configured self-managed host; the available host credential is expired, so the upstream returned the expected 401 and authenticated GitLab behavior remains unverified
• inference provider configuration is implemented but was not exercised because no inference credential/model was configured locally
• OpenShell provider profile lint: passed

No branch push, pull request creation, or merge request creation was performed during source-provider validation.

Related

• Linear: https://linear.app/nvidia/issue/AIRE-823/prototype-containerized-experimentalist-execution-with-openshell
• Existing OpenShell deployment backend: feat(deployments): OpenShell sandboxed deployment backend (AIRCORE-872) #662

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	28185/36015	78.3%	62.6%
Integration Tests	16417/34733	47.3%	19.7%

[drazvan]

Should we do it the other way around? i.e., have nemo experimentalist run orchestrate the creation of the sandbox and trigger the agent in a safe way, by default.

[ryana]

yeah that's definitely better UX. would require a larger refactor. I'm trying to thread the needle.

[gaiadilorenzo]

This will reject valid overrides (e.g. retry configs). Not sure if inheriting from harbor job config is possible in this case

[gaiadilorenzo]

What would be the disadvantage of making HarborDataset backend-aware at construction time? For example, could HarborDataset.from_path() accept a local or bridge backend and construct the correct dependency runtime directly? We could then propagate the selected backend through DatasetFactory and the Eval Author dataset construction paths (stage(), promote_local(), and finalize()). Would that let us remove prepare_dataset(), avoid mutating datasets after construction, and ensure sandboxed paths never temporarily contain local Docker runtimes?

[gaiadilorenzo]

Is selecting RemoteHarborEvaluator implicitly from BRIDGE_URL_ENV intentional? Since this is Harbor-specific, a general backend abstraction may not be worthwhile, but could the Harbor config explicitly select local or bridge execution? That would make evaluator selection visible in effective configuration and avoid the same type: harbor config hanging behavior based on process environment.

[gaiadilorenzo]

Could we support Docker Compose tasks here instead of rejecting them? What additional security risk would accepting these tasks introduce? Rejecting them prevents existing Harbor suites such as Tau2 from running through the bridge.

[gaiadilorenzo]

Shouldn't we allow the Platform API calls the run makes here, and mirror the same rules in policy.docker-desktop.yaml? The experimenter needs POST /apis/intake/v2/workspaces/*/experiments, GET and PUT /apis/intake/v2/workspaces/*/experiments/*, and POST /apis/intake/v2/workspaces/*/ingest/otlp/v1/traces. With the readiness-only policy, experiment projection and OTLP trace ingestion fail with 403 policy_denied.

[ryana]

Superseded by #1008, rebuilt fresh from main under AIRE-823 with the task-envelope bridge contract and preserved Harbor verifier behavior.