docs(experimentalist): recommend Docker Sandboxes

[nicot]

Summary

• recommend Docker Sandboxes for laptop Experimentalist runs
• invoke the Tau3 example directly through sbx exec
• document host-service routing and the Apple Silicon Terminal-Bench limitation

Test plan

• Run the Tau3 benchmark and agent end to end inside a Docker Sandbox
• Run make docs-check

This is a draft because the sandbox workflow has not been fully validated yet.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-1034-docs-experimentalist-sandbox-ntonozzi.docs.buildwithfern.com/nemo-platform

[nicot]

Docker Sandbox smoke result

Validated on an Apple Silicon Mac with Docker Sandboxes (sbx), using the sandbox's private Docker daemon:

• nemo experimentalist loads and can be invoked directly through sbx exec.
• The sandbox can reach the host NeMo Platform at http://host.docker.internal:8080; /health/ready returned {"status":"ready"}.
• A real Tau3 Airline task ran with the checked-in nemo-experimentalist-tau3-nooa agent.
• Harbor completed 1/1 trials with 0 exceptions and reward 1.0 in 4m21s.

{
  "task_name": "sierra-research/tau3-bench__tau3-airline-2",
  "exception_info": null,
  "reward": 1.0,
  "started_at": "2026-07-31T22:58:10.951428Z",
  "finished_at": "2026-07-31T23:02:32.126926Z"
}

make docs-check also passes: all 211 MDX files parsed cleanly, with no gated-link errors.

Architecture caveat: the Terminal-Bench fix-git image is currently linux/amd64-only and does not run in an Apple Silicon linux/arm64 sandbox. Tau3 builds for the sandbox architecture and works.

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	29440/37448	78.6%	63.2%
Integration Tests	17381/36166	48.1%	20.5%

[nicot]

Full Docker Sandbox smoke result

Ran the complete canonical Tau3 Airline smoke benchmark inside a Docker Sandbox on Apple Silicon:

uv run --frozen --no-sync --python 3.13 \
  --package nemo-experimentalist-plugin \
  python benchmarks/run.py \
  --suite benchmarks/suites/tau3-airline.yaml \
  --config benchmarks/configs/tau3-smoke.yaml \
  --agent examples/tau3-nooa-agent \
  --output tmp/experimentalist-sandbox-smoke/full-tau3-airline-pr1034 \
  --test-concurrency 1

Validation and results:

• Canonical dataset validation: 50 tasks; fast split 6 train / 3 validation / 3 test.
• Process exit: 0.
• Total runtime: 5,490 seconds (91m38s).
• Held-out baseline: 3/3 observed, 0 missing, 0 harness errors, mean reward 0.333.
• Experimentalist: 1 round completed; 11/11 Harbor trials completed across 4 jobs with 0 errored trials.
• Candidate validation regressed from 0.333 to 0.000, so Experimentalist correctly retained agent-0 as the winner.
• Held-out winner: 3/3 observed, 0 missing, 0 harness errors.
• summary.json was generated successfully.
• make docs-check passes: 211 MDX files parsed cleanly, with no gated-link errors.

The Experimentalist/Harbor/Tau3 workflow itself completed end to end without infrastructure errors.

One post-run sandbox diagnostic exposed a Docker Engine 29.6.1 issue: docker system df can report rw layer snapshot not found after concurrent container cleanup. Docker documents this race as fixed in Engine 29.6.2. I upgraded sbx from 0.27.0 to 0.37.1, but the upgraded CLI requires a fresh Docker device login; two unattended login attempts expired, so verification of the upgraded daemon remains outstanding.