NVIDIA-NeMo · ryana · Jul 30, 2026 · Jul 30, 2026 · Jul 31, 2026 · Jul 31, 2026
@@ -3,7 +3,10 @@ title: "Get started with an example agent"
 description: "Run the optimization loop from end to end on τ-Bench"
 ---
 
-This provides a guide for how you can quickly load agent traces into your NeMo Platform, run the analyst agent to discover issues with that agent, and launch an experimentalist to fix those issues. This is meant as an example to show how the whole platform works. You can also jump straight to setting up your agent with NeMo Platform.
+This guide runs one Tau3 Airline agent through the full loop: Harbor evaluates
+the checked-in example agent, the Analyst discovers issues from its traces, and
+the Experimentalist improves the same agent. You can also jump straight to
+setting up your agent with NeMo Platform.
 
 ## Prerequisites
 
@@ -34,50 +37,47 @@ uv run nemo services start --config packages/nmp_platform/config/local.yaml
 
 You should now be able to navigate to `http://localhost:8080` and see the NeMo Platform web UI.
 
-## 3. Prepare Tau2 and run the airline agent
+## 3. Run the Tau3 Airline agent with Harbor
 
-Now that we have a running NeMo Platform, it's time to load up some data! This example uses the τ-Bench from Sierra. It's a great representation of a simplified agent that performs a business-critical task. Our first step will be to download the tau2 repository and install the dependencies to run it.
-
-Then, we'll run a script to go through 30 of the tasks from the Tau 2 Airline dataset. In order to make our example more realistic and representative of the kind of agent data we typically see in production, we won't record whether the tasks passed verification – we want to see if we can gain useful insights even without ground truth data. This could take up to 20 minutes, and will cost about a dollar.
+Evaluate the
+[checked-in Tau3 NOOA example agent](https://github.com/NVIDIA-NeMo/nemo-platform/tree/main/plugins/nemo-experimentalist/examples/tau3-nooa-agent)
+against the
+[`sierra-research/tau3-bench`](https://hub.harborframework.com/datasets/sierra-research/tau3-bench/latest)
+dataset on Harbor Hub. The Insights testbed selects six Airline tasks, runs the
+agent through Harbor, and uploads its traces and verifier rewards into NeMo
+Platform. This can take 20 minutes or longer because Harbor builds task
+containers and runs model-backed user simulation.
 
 ```bash
-git init tmp/tau2-bench
-git -C tmp/tau2-bench remote add origin https://github.com/sierra-research/tau2-bench.git
-git -C tmp/tau2-bench fetch --depth 1 origin 8ebb7499622fc2be9b9d510d6f7a7653461f4f29
-git -C tmp/tau2-bench checkout --detach FETCH_HEAD
-uv --directory tmp/tau2-bench sync --frozen
-uv --directory tmp/tau2-bench run --frozen tau2 check-data
-
 export NMP_BASE_URL=http://localhost:8080
+export INFERENCE_API_BASE=https://inference-api.nvidia.com/v1
 export INFERENCE_API_KEY=sk-...
-export OPENAI_API_KEY="$INFERENCE_API_KEY"
-export OPENAI_API_BASE=https://inference-api.nvidia.com/v1
 
 uv run --directory plugins/nemo-insights --frozen \
-  python -m testbed run tau2-airline \
-  --base "$NMP_BASE_URL" \
-  --set include_rewards=false \
-  --set tau2_repo="$PWD/tmp/tau2-bench"
+  python -m testbed run tau3-airline-harbor \
+  --base "$NMP_BASE_URL"
 ```
 
-The testbed runs the 30-task airline train split and ingests realistic traces into the `tau2-airline` workspace. At this point you can navigate to the traces view and see the traces from the agent.
+In Studio, open the `canonical-tau3-airline` workspace's Experiments area to
+inspect the task results, rewards, and traces.
 
 ## 4. Run the Analyst
 
 Now that we have data in our system, we can run the analyst agent to understand the ways that our agent is performing well, and where it is falling down. The goal of the analyst agent is to crawl production data, and determine where your agent is falling down or disappointing your customers.
 
 If the analyst discovers issues in your application, it will create what we call an 'insight'. An insight is a human-readable description of a problem in your agentic system. The closest analogy is a bug report. They don't try to describe why an issue happened or the code you should change to fix it, and they should be understandable to a user of your agent.
 
-Let's run it:
+Run the Analyst against the recorded Harbor evaluation:
 
 ```bash
-uv run --frozen nemo insights analyze \
-  --agent tau2-airline \
-  --workspace tau2-airline \
-  --base-url "$NMP_BASE_URL"
+uv run --directory plugins/nemo-insights --frozen \
+  python -m testbed analyze tau3-airline-harbor \
+  --live \
+  --base "$NMP_BASE_URL"
 ```
 
-This will take a few minutes to run. Once it's done, you can navigate to the insights page to see the issues the analyst discovered in the τ-Bench Airline Agent.
+This will take a few minutes. In Studio, open the
+`canonical-tau3-airline` workspace's Insights area to review the result.
 
 ## 5. Prepare a Tau3 Airline smoke dataset
 

@@ -0,0 +1,54 @@
+# syntax=docker/dockerfile:1
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+ARG NMP_PYTHON_BASE=nmp-python-base
+
+FROM ${NMP_PYTHON_BASE} AS dependencies
+COPY --from=nmp-workspace pyproject.toml uv.lock README.md ./
+RUN --mount=type=cache,target=/root/.cache/uv \
+    uv sync --frozen --only-group experimentalist --no-install-workspace --no-editable
+
+FROM dependencies AS builder
+COPY --from=nmp-workspace . .
+RUN --mount=type=cache,target=/root/.cache/uv \
+    for package in \
+        nemo-eval-author-plugin \
+        nemo-experimentalist-plugin \
+        nemo-insights-plugin \
+        nemo-platform \
+        nemo-platform-plugin \
+        nemo-platform-sdk \
+        nmp-common; do \
+        uv build --wheel --package "$package" --out-dir /wheels; \
+    done
+
+FROM ${NMP_PYTHON_BASE} AS runtime
+
+LABEL com.nvidia.nemo.experimentalist.openshell-runtime="1"
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    iproute2 \
+    nftables \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/* \
+    && groupadd --system sandbox \
+    && useradd --system --gid sandbox --create-home --home-dir /home/sandbox --shell /bin/bash sandbox \
+    && mkdir -p /sandbox \
+    && chown sandbox:sandbox /sandbox \
+    && touch /etc/nemo-experimentalist-container
+
+COPY --from=dependencies /app/.venv /app/.venv
+COPY --from=builder /wheels /wheels
+RUN UV_COMPILE_BYTECODE=0 uv pip install \
+    --python /app/.venv/bin/python \
+    --no-deps \
+    /wheels/*.whl \
+    && rm -rf /wheels
+
+ENV HOME=/home/sandbox \
+    PATH="/app/.venv/bin:$PATH" \
+    XDG_CACHE_HOME=/sandbox/.cache
+
+WORKDIR /sandbox
+USER sandbox
@@ -54,22 +54,52 @@ effective inputs:
 $NEMO experimentalist doctor
 ```
 
-## Run the Experimentalist locally
+## Run the Experimentalist in OpenShell
 
-`nemo experimentalist run` runs the local Experimentalist loop. It evaluates
-a baseline agent on Harbor-compatible train and validation datasets, proposes
-candidate mutations, and records its artifacts under the selected experiment
-directory.
+`nemo experimentalist run` resolves inputs on the trusted host, runs the
+optimization loop in an OpenShell sandbox, and submits Harbor work to an
+ephemeral authenticated host bridge. The sandbox receives a run-specific input
+snapshot, not the repository or home directory, and it receives no Docker
+socket or real bridge credential. OpenShell injects an opaque bridge placeholder
+and replaces it with the host token only at the permitted network boundary.
+There is no local-execution fallback.
 
-Configure the models before running an experiment:
+Trusted Python callers and tests can still invoke
+`nemo_experimentalist_plugin.experimentalist.run.run_experimentalist` directly
+without OpenShell. The public CLI deliberately does not expose that bypass.
+
+Install and configure OpenShell, then configure the optimizer model and the
+dedicated direct-inference credential used by candidates:
 
 ```bash
-export EXPERIMENTALIST_API_BASE=https://inference-api.nvidia.com/v1
-export EXPERIMENTALIST_API_KEY=sk-...
+export EXPERIMENTALIST_API_KEY=nvapi-optimizer-key
 export EXPERIMENTALIST_SMART_MODEL_NAME=openai/openai/openai/gpt-5.5
 export EXPERIMENTALIST_FAST_MODEL_NAME=openai/openai/openai/gpt-5-mini
+export INFERENCE_API_KEY=nvapi-dedicated-experimentalist-key
+export INFERENCE_API_BASE=https://inference-api.nvidia.com/v1
+export AUT_MODEL_NAME=openai/gpt-oss-120b
 ```
 
+The Experimentalist model values use NOOA's provider/routing/model identifier
+format. Provider setup removes NOOA's outer client prefix while preserving the
+gateway's provider/model route when deriving the OpenShell model ID. When the
+optimizer and candidate share a gateway key, `EXPERIMENTALIST_API_KEY` defaults
+to `INFERENCE_API_KEY`.
+
+Provider setup skips OpenShell 0.0.92's generic endpoint probe for the NVIDIA
+Inference Hub route because its GPT-5 proxy rejects that probe shape. This does
+not disable the sandbox's pinned inference route or network enforcement; the
+first Experimentalist model request performs the functional endpoint check.
+
+The candidate key is injected only into the candidate process inside Harbor.
+Use a spending-limited, easy-to-revoke credential. Source-control archival and
+winner publishing are intentionally unsupported by this runtime.
+
+The local bridge binds all host interfaces by default because the OpenShell
+Docker driver reaches it through the host-side gateway interface. On a trusted
+host, set `NEMO_EXPERIMENTALIST_HARBOR_BRIDGE_BIND` to that gateway's host IP
+to restrict the listener to the interface OpenShell uses.
+
 ### Insight-driven optimization
 
 A single local Insight in `.nemo-optimizer/insights.yaml` is selected by

@@ -0,0 +1,15 @@
+# Experimentalist control plane. Harbor and Docker remain host-side.
-# Experimentalist control plane. Harbor and Docker remain host-side.
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+# Experimentalist control plane. Harbor and Docker remain host-side.
-# Experimentalist control plane. Harbor and Docker remain host-side.
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+# Experimentalist control plane. Harbor and Docker remain host-side.
+target "nmp-experimentalist-docker" {
+  target     = "runtime"
+  context    = "."
+  dockerfile = "plugins/nemo-experimentalist/Dockerfile"
+  contexts = {
+    nmp-python-base = "target:nmp-python-base"
+    nmp-workspace   = "target:nmp-workspace"
+  }
+  cache-to   = maybe_registry_cache_to("nmp-experimentalist")
+  cache-from = maybe_registry_cache_from("nmp-experimentalist")
+  tags       = sha_and_maybe_latest_tags("nmp-experimentalist")
+  output     = image_output()
+  platforms  = get_platforms()
+}
@@ -4,6 +4,7 @@ version = "0.1.0"
 description = "NeMo Experimentalist plugin for NeMo Platform."
 requires-python = ">=3.12,<3.14"
 dependencies = [
+    "fastapi>=0.115.4",
     "pydantic>=2",
     "httpx",
     "harbor>=0.16",
@@ -15,9 +16,14 @@ dependencies = [
     "nemo-insights-plugin",
     "nemo-platform",
     "nemo-platform-plugin",
+    "python-multipart>=0.0.20",
     "tomlkit>=0.13.3",
+    "uvicorn>=0.34.0",
 ]
 
+[project.scripts]
+nemo-experimentalist-harbor-bridge = "nemo_experimentalist_plugin.harbor_bridge.service:main"
+
 [project.entry-points."nemo.cli"]
 experimentalist = "nemo_experimentalist_plugin.cli:ExperimentalistCLI"
 

@@ -15,6 +15,11 @@
 import typer
 import yaml
 from nemo_experimentalist_plugin.client import make_client
+from nemo_experimentalist_plugin.openshell.launcher import (
+    OpenShellLaunchError,
+    launch_openshell_run,
+)
+from nemo_experimentalist_plugin.openshell.preparation import prepare_openshell_run
 from nemo_experimentalist_plugin.preflight import (
     Probes,
     check_artifacts,
@@ -45,8 +50,11 @@
 from nemo_platform_plugin.cli import NemoCLI
 
 DEFAULT_WORKSPACE = "default"
+_CONTAINER_MARKER = Path("/etc/nemo-experimentalist-container")
 
 _PREFLIGHT_PROBES: Probes | None = None  # test seam; None → real probes
+_OPEN_SHELL_PREPARER = prepare_openshell_run
+_OPEN_SHELL_LAUNCHER = launch_openshell_run
 
 # Lazily imported in the experiment command: importing experimentalist.run reaches model
 # construction that requires EXPERIMENTALIST_API_* env at import time, and this module
@@ -57,6 +65,10 @@
 # TODO: Add remote train/validation dataset support when remote experiment mode is implemented.
 
 
+def _inside_experimentalist_container() -> bool:
+    return _CONTAINER_MARKER.is_file()
+
+
 def _default_experiment_dir(profile: AgentProfile | None) -> Path:
     if profile is None:
         experiments_root = Path("tmp")
@@ -94,9 +106,8 @@ def run(
                 help=(
                     "Baseline agent: a local directory or a git URL with an optional ref "
                     "(e.g. ssh://git@host/group/repo.git@main). Optional in Mode 1 (the insight "
-                    "supplies the agent) and overrides the insight's agent when given. A git "
-                    "source records provenance and enables --config storage.publish_winner to open a "
-                    "draft PR/MR for the winner against that ref."
+                    "supplies the agent) and overrides the insight's agent when given. Git sources "
+                    "are resolved on the trusted host before the credential-free snapshot enters OpenShell."
                 ),
             ),
             agent_spec: str | None = typer.Option(
@@ -203,7 +214,7 @@ def run(
                 readable=True,
             ),
         ) -> None:
-            """Run offline optimization for a baseline agent (local dir or git source)."""
+            """Run optimization inside OpenShell with Harbor execution on the trusted host."""
 
             if no_insight and (insight is not None or insight_id is not None):
                 typer.echo("--no-insight cannot be combined with --insight or --insight-id", err=True)
@@ -246,6 +257,7 @@ async def _flow() -> str:
                         insight_id=effective_insight.selector,
                         base_url=base_url_resolved,
                         enforce_insight_agent=agent is None,
+                        openshell=not _inside_experimentalist_container(),
                         probes=_PREFLIGHT_PROBES,
                     )
                 )
@@ -293,14 +305,29 @@ async def _flow() -> str:
                     scratch_dir=experiment_dir_resolved / "resolved",
                     plan=plan,
                 )
+                client = make_client(base_url_resolved)
+                if not _inside_experimentalist_container():
+                    try:
+                        prepared = await _OPEN_SHELL_PREPARER(
+                            inputs,
+                            experiment_dir=experiment_dir_resolved,
+                            client=client,
+                        )
+                        return await asyncio.to_thread(
+                            _OPEN_SHELL_LAUNCHER,
+                            prepared,
+                            experiment_dir=experiment_dir_resolved,
+                            platform_url=base_url_resolved,
+                        )
+                    finally:
+                        await client.close()
                 global run_experimentalist
                 if run_experimentalist is None:
                     from nemo_experimentalist_plugin.experimentalist.run import (
                         run_experimentalist as _run_experimentalist,  # noqa: PLC0415
                     )
 
                     run_experimentalist = _run_experimentalist
-                client = make_client(base_url_resolved)
                 try:
                     return await run_experimentalist(
                         agent=inputs.agent,
@@ -321,7 +348,7 @@ async def _flow() -> str:
 
             try:
                 output_text = asyncio.run(_flow())
-            except (OSError, ValueError, yaml.YAMLError) as exc:
+            except (OpenShellLaunchError, OSError, ValueError, yaml.YAMLError) as exc:
                 typer.echo(str(exc), err=True)
                 raise typer.Exit(code=1) from None
             typer.echo(output_text)
@@ -413,6 +440,7 @@ def doctor(
                 insight=effective_insight.ref if effective_insight is not None else None,
                 insight_id=effective_insight.selector if effective_insight is not None else None,
                 base_url=base_url_resolved,
+                openshell=not _inside_experimentalist_container(),
                 probes=_PREFLIGHT_PROBES,
             )
             if profile_obj is not None:

@@ -17,6 +17,7 @@
     Task,
     TrialResult,
 )
+from nemo_experimentalist_plugin.experimentalist.components.evaluator.models import DependencyRuntimeError
 from nemo_experimentalist_plugin.experimentalist.components.trace_analyzer import (  # noqa: F401
     Diagnostic,
     TraceAnalyzer,
@@ -638,6 +639,8 @@ async def run(
         for task_id, result in zip(unique_tasks, rationales_list, strict=True):
             if isinstance(result, asyncio.CancelledError):
                 raise result
+            if isinstance(result, DependencyRuntimeError):
+                raise result
             if isinstance(result, BaseException):
                 logging.getLogger(__name__).warning(f"Rationalizer failed for {agent_id}/{task_id}: {result}")
                 continue
@@ -665,6 +668,8 @@ async def run(
         for (trial, _), result in zip(trial_tasks, diagnoses_list, strict=True):
             if isinstance(result, asyncio.CancelledError):
                 raise result
+            if isinstance(result, DependencyRuntimeError):
+                raise result
             if isinstance(result, BaseException):
                 logging.getLogger(__name__).warning(
                     "TraceAnalyzer failed for %s/%s: %s",

@@ -39,6 +39,9 @@ def __init__(self, options: EvaluatorConfig, experiment_dir: Path | None = None)
         self.options = options
         self.experiment_dir = experiment_dir
 
+    def prepare_dataset(self, dataset: Dataset) -> None:
+        """Attach evaluator-specific task runtimes in place when required."""
+
     async def aggregate_results(self, results: Sequence[TrialResult]) -> dict[str, float | int]:
         """
         Aggregate evaluation results from multiple runs.