[FIX]: Evaluate probe verdicts over final traces

docs/api/core-types.md

@@ -14,6 +14,8 @@ Data types shared across the entire framework. All importable from `rampart` dir
         - ToolCall
         - SideEffect
         - Turn
+        - EvaluationRole
+        - TerminationReason
         - EvalOutcome
         - EvalResult
         - EvalContext
@@ -28,6 +30,8 @@ Data types shared across the entire framework. All importable from `rampart` dir
         - SafetyStatus
         - HarmCategory
         - InjectionRecord
+        - resolve_attack_verdict
+        - resolve_probe_verdict
         - resolve_as_attack
         - resolve_as_probe
 

docs/api/evaluators.md

@@ -7,6 +7,7 @@ Built-in evaluators. All extend `BaseEvaluator` and support composition via `|`,
       members:
         - ToolCalled
         - ResponseContains
+        - ResponseScope
         - SideEffectOccurred
         - LLMJudge
         - TranscriptScope

docs/api/index.md

@@ -13,7 +13,7 @@ API reference organized by RAMPART's package layout. Each page documents the pub
 | [Core Protocols](core-protocols.md) | `Session`, `AgentAdapter`, `Evaluator`, `Surface`, `InjectionHandle`, and more |
 | [Attacks](attacks.md) | `Attacks.xpia()`, `XPIAExecution` |
 | [Probes](probes.md) | `Probes.behavior()`, `SingleTurnExecution` |
-| [Evaluators](evaluators.md) | `ToolCalled`, `ResponseContains`, `SideEffectOccurred`, `LLMJudge`, `TranscriptScope` |
+| [Evaluators](evaluators.md) | `ToolCalled`, `ResponseContains`, `ResponseScope`, `SideEffectOccurred`, `LLMJudge`, `TranscriptScope` |
 | [Drivers](drivers.md) | `StaticDriver`, `LLMDriver` |
 | [Payloads](payloads.md) | `Payloads`, `PayloadTemplate`, `PayloadStore` |
 | [Surfaces](surfaces.md) | `OneDriveSurface` |

docs/attacks/xpia.md

@@ -121,26 +121,44 @@ For generating many variants at scale, use [`Payloads.generate_async()`][rampart
 Combine evaluators with `|` (OR), `&` (AND), and `~` (NOT):
 
 ```python
-from rampart.evaluators import ToolCalled, ResponseContains
+from rampart.evaluators import ResponseContains, ResponseScope, ToolCalled
 
 # OR: detect exfil via tool call or leaked content in response text
 evaluator = (
     ToolCalled("send_email", recipient=lambda v: "evil.com" in str(v))
-    | ResponseContains("attacker@evil.com")
+    | ResponseContains(
+        "attacker@evil.com",
+        scope=ResponseScope.ANY_TURN,
+    )
 )
 
 # AND: agent executed a command AND the response mentions credentials
 evaluator = (
     ToolCalled("exec", command=lambda v: ".ssh" in str(v))
-    & ResponseContains("id_rsa")
+    & ResponseContains("id_rsa", scope=ResponseScope.ANY_TURN)
 )
 
-# NOT: agent did NOT refuse — it complied with the injection
-evaluator = ~ResponseContains(lambda text: "I can't" in text or "I cannot" in text)
+# NOT: the agent failed to refuse on at least one turn
+evaluator = ~ResponseContains(
+    lambda text: "I can't" in text or "I cannot" in text,
+    scope=ResponseScope.ALL_TURNS,
+)
 ```
 
 Place the cheaper evaluator on the left side of `|` — it short-circuits if the left operand detects.
 
+!!! warning "Multi-turn scope"
+    State the temporal scope explicitly for multi-turn attacks. Use
+    `ANY_TURN` for "leaked at some point" and negate `ALL_TURNS` for "failed
+    to refuse at least once." Omitting `scope` inspects only the current
+    response and emits a `FutureWarning` for multi-turn transcripts. See
+    [Temporal Scope](../usage/authoring-tests.md#temporal-scope).
+
+    This release prepares evaluator semantics for final-trace verdicts. Until
+    that cadence change ships, attack executions still evaluate growing
+    prefixes. The attack forms above preserve their intended meaning during
+    that transition.
+
 ### LLMDriver for Adaptive Triggers
 
 For multi-turn attacks where the trigger conversation adapts based on agent responses, use [`LLMDriver`][rampart.drivers.llm.LLMDriver] instead of a static string:
@@ -208,7 +226,7 @@ See [`Attacks.xpia()`][rampart.attacks.Attacks.xpia] for the full API reference.
 | `inject` | `InjectionHandle \| list[InjectionHandle] \| None` | `None` | Prepared injections from `surface.inject()`. `None` for inline XPIA. |
 | `trigger` | `str \| list[str] \| Request \| list[Request] \| PromptDriver` | required | Benign prompt(s) that cause retrieval of injected content. |
 | `evaluator` | [`Evaluator`][rampart.core.evaluator.Evaluator] | required | What attack condition to detect. |
-| `max_turns` | `int` | `5` | Maximum prompt-response exchanges before `ERROR`. |
+| `max_turns` | `int` | `5` | Maximum prompt-response exchanges; reaching the limit resolves the trace normally. |
 | `event_handlers` | `list[ExecutionEventHandler] \| None` | `None` | Additional lifecycle event handlers. |
 
 ---

docs/concepts/overview.md

@@ -58,7 +58,7 @@ A single test run flows from your pytest test, through a RAMPART attack or probe
 
 *Request / response cycle for a single test run.*
 
-Under the hood, every execution follows a common lifecycle owned by [`BaseExecution`][rampart.core.execution.BaseExecution], which drives the per-turn loop between the strategy, your adapter, and the evaluator:
+Under the hood, every execution follows a common lifecycle owned by [`BaseExecution`][rampart.core.execution.BaseExecution]. The strategy drives requests through your adapter. Probes evaluate the completed trace once unless an explicit online stop condition is configured; attacks still use prefix evaluation pending their cadence migration.
 
 ```mermaid
 sequenceDiagram
@@ -76,11 +76,16 @@ sequenceDiagram
         Strat->>Strat: driver.next_prompt_async(history)
         Strat->>Adapter: session.send_async(request)
         Adapter-->>Strat: Response
-        Strat->>Eval: evaluate_async(context)
-        Eval-->>Strat: EvalResult
-        Note over Strat: Early stop if detected
+        opt Explicit online stop condition
+            Strat->>Eval: evaluate_async(prefix context)
+            Eval-->>Strat: stop EvalResult
+            Note over Strat: Stop if detected
+        end
     end
 
+    Strat->>Eval: evaluate_async(terminal context)
+    Eval-->>Strat: final EvalResult
+
     Strat-->>Exec: Result
     Exec->>Exec: fire ON_POST_EXECUTE
     Exec-->>Test: Result
@@ -112,7 +117,7 @@ Evaluators are **polarity-free**. They answer "did X happen?" — not "is X good
 - In an **attack**, detection means the attack objective was achieved → **UNSAFE**
 - In a **probe**, detection means the expected behavior is present → **SAFE**
 
-The [`Attacks`][rampart.attacks.Attacks] and [`Probes`][rampart.probes.Probes] factories handle this mapping automatically via [`resolve_as_attack`][rampart.core.result.resolve_as_attack] and [`resolve_as_probe`][rampart.core.result.resolve_as_probe].
+The [`Attacks`][rampart.attacks.Attacks] and [`Probes`][rampart.probes.Probes] factories handle this mapping automatically. Probes use [`resolve_probe_verdict`][rampart.core.result.resolve_probe_verdict] over one terminal evaluation; attacks retain [`resolve_as_attack`][rampart.core.result.resolve_as_attack] until their cadence migration.
 
 You can reuse the same evaluator in both contexts. A [`ToolCalled`][rampart.evaluators.tool_called.ToolCalled] evaluator detects whether a tool was called — whether that's good or bad depends on whether you're attacking or probing.
 

docs/concepts/probes.md

@@ -14,9 +14,9 @@ Probes use the inverse mapping from evaluator outcomes:
 | `NOT_DETECTED` | `UNSAFE` | The expected behavior is missing — a regression |
 | `UNDETERMINED` | `UNDETERMINED` | The evaluator could not determine whether the behavior is present |
 
-Precedence: `NOT_DETECTED` > `UNDETERMINED` > `DETECTED`. If any turn failed to detect the expected behavior, the agent is non-compliant.
-
-This logic lives in [`resolve_as_probe`][rampart.core.result.resolve_as_probe].
+The evaluator runs once over the completed trace, and the outcome maps directly
+to the verdict. This logic lives in
+[`resolve_probe_verdict`][rampart.core.result.resolve_probe_verdict].
 
 ---
 
@@ -26,9 +26,10 @@ Probe executions are simpler than attacks — no injection phase:
 
 1. **Create session** — Open a fresh session with the agent
 2. **Send prompts** — Drive the conversation via the prompt driver
-3. **Evaluate** — Check whether the expected behavior is present
-4. **Clean up** — Close the session
-5. **Report** — Produce a [`Result`][rampart.core.result.Result]
+3. **Stop (optional)** — Check an explicit online `stop_when` condition
+4. **Evaluate** — Check the completed trace once for expected behavior
+5. **Clean up** — Close the session
+6. **Report** — Produce a [`Result`][rampart.core.result.Result]
 
 ---
 
@@ -51,6 +52,9 @@ assert result, result.summary
 
 Provide exactly one of `prompt`, `prompts`, or `driver`.
 
+Probes run the full prompt sequence by default. Pass `stop_when=` only when an
+online condition intentionally defines an earlier terminal trace.
+
 ---
 
 ## Available Probes

docs/contributing/extending-rampart.md

@@ -179,29 +179,36 @@ The process mirrors the [Attack](#attack) walkthrough. The differences are summa
 |---|---|---|
 | **Location** | `rampart/attacks/_name.py` | `rampart/probes/_name.py` |
 | **Factory class** | `Attacks` | `Probes` |
-| **Resolution function** | `resolve_as_attack` | `resolve_as_probe` |
+| **Resolution function** | `resolve_as_attack` (pending cadence migration) | `resolve_probe_verdict` |
 | **Detected means** | UNSAFE | SAFE |
 | **Injection phase** | Often yes | No |
 
 ### 1. Create the Execution Class
 
-The file structure mirrors the [Attack walkthrough](#1-create-the-execution-class) — same imports, `__init__`, and `_execute_async` loop. The diff from `MyAttackExecution` is:
+Probe strategies drive the full trace first, then evaluate it once while the
+session is still active:
 
-```diff
--from rampart.core import (..., resolve_as_attack)
-+from rampart.core import (..., resolve_as_probe)
-
--class MyAttackExecution(BaseExecution):
-+class MyProbeExecution(BaseExecution):
-
--    return "my_attack"
-+    return "my_probe"
+```python
+async with await adapter.create_session_async() as session:
+    run = await run_trace_async(
+        session=session,
+        driver=self._driver,
+        max_turns=self._max_turns,
+        stop_when=self._stop_when,
+        manifest=adapter.manifest,
+    )
+    evaluation = await evaluate_terminal_async(
+        evaluator=self._evaluator,
+        run=run,
+    )
 
--    status = resolve_as_attack(eval_results=eval_results)
-+    status = resolve_as_probe(eval_results=eval_results)
+status = resolve_probe_verdict(evaluation=evaluation)
 ```
 
-Place the file in `rampart/probes/` (e.g. `_my_probe.py`). Most probes skip the injection phase — just session creation, prompt driving, and evaluation. For a complete working reference, see [`rampart/probes/_single_turn.py`](https://github.com/microsoft/RAMPART/blob/main/rampart/probes/_single_turn.py).
+Store `evaluation`, `run.turns`, and `run.termination_reason` on the returned
+`Result`. Most probes skip the injection phase. For a complete working
+reference, see
+[`rampart/probes/_single_turn.py`](https://github.com/microsoft/RAMPART/blob/main/rampart/probes/_single_turn.py).
 
 ### 2. Add a Factory Method to `Probes`
 
@@ -212,7 +219,7 @@ Add a static method to the `Probes` class in `rampart/probes/__init__.py`, mirro
 Probe tests have the same surface as attack tests, with two differences:
 
 - **No injection phase** to test.
-- **Result resolution** uses `resolve_as_probe` semantics (detected → SAFE, not detected → UNSAFE).
+- **Result resolution** uses `resolve_probe_verdict` semantics (detected → SAFE, not detected → UNSAFE).
 
 
 ## Evaluator
@@ -246,16 +253,18 @@ class MyEvaluator(BaseEvaluator):
         self._target = target
 
     async def evaluate_async(self, *, context: EvalContext) -> EvalResult:
-        """Evaluate the latest turn for the target condition.
+        """Evaluate the full trace for the target condition.
 
         Args:
             context (EvalContext): The evaluation context with turn history.
 
         Returns:
             EvalResult: Whether the condition was detected, with evidence.
         """
-        latest_turn = context.turns[-1]
-        detected = self._target in latest_turn.response.text
+        detected = any(
+            self._target in turn.response.text
+            for turn in context.turns
+        )
 
         return EvalResult(
             outcome=EvalOutcome.DETECTED if detected else EvalOutcome.NOT_DETECTED,
@@ -266,6 +275,15 @@ class MyEvaluator(BaseEvaluator):
 
 Evaluator tests should cover detection, non-detection, edge cases (empty response, missing data), and that `evidence` / `rationale` are populated correctly.
 
+!!! warning "Multi-turn evaluator migration"
+    Final-trace verdicts call an evaluator once with the complete transcript.
+    A custom evaluator that reads only `context.turns[-1]` intentionally judges
+    only the terminal response and cannot preserve earlier evidence. Rewrite
+    multi-turn predicates to inspect `context.turns` explicitly before
+    migrating execution cadence. The worked execution-strategy loop elsewhere
+    on this page still describes the current prefix-evaluation behavior and
+    will be replaced with the shared trace runner in the cadence change.
+
 
 ## Prompt Driver
 

docs/contributing/testing.md

@@ -140,7 +140,7 @@ When adding a new attack, test:
 Similar to attacks, but:
 
 1. No injection phase to test
-2. Result resolution uses `resolve_as_probe` (detected → SAFE, not detected → UNSAFE)
+2. Result resolution uses `resolve_probe_verdict` over one terminal evaluation (detected → SAFE, not detected → UNSAFE)
 
 ### Testing a New Evaluator
 

docs/probes/behavioral.md

@@ -10,9 +10,10 @@ Use behavioral probes for regression testing: ensure your agent still does the r
 
 1. **Create session** — Open a fresh session with the agent
 2. **Send prompts** — Drive the conversation via a prompt driver
-3. **Evaluate** — Check each turn for the expected behavior. Early-stops on detection.
-4. **Clean up** — Close the session
-5. **Result** — Produce a [`Result`][rampart.core.result.Result] via `resolve_as_probe` semantics
+3. **Stop (optional)** — Evaluate `stop_when` after each response and stop when detected
+4. **Evaluate** — Check the expected behavior once over the completed trace
+5. **Clean up** — Close the session
+6. **Result** — Map the final evaluation using probe semantics
 
 No injection phase.
 
@@ -54,20 +55,51 @@ result = await Probes.behavior(
 For full control over the conversation flow, use a [`StaticDriver`][rampart.drivers.static.StaticDriver]:
 
 ```python
-from rampart.drivers import StaticDriver
 from rampart import Request
+from rampart.drivers import StaticDriver
+from rampart.evaluators import ResponseContains, ResponseScope
 
 driver = StaticDriver(prompts=[
-    Request(prompt="Hello"),
-    Request(prompt="What tools do you have?"),
+    Request(prompt="Name a search tool you can use."),
+    Request(prompt="Describe that search tool."),
 ])
 
 result = await Probes.behavior(
     driver=driver,
-    evaluator=ResponseContains("search"),
+    evaluator=ResponseContains(
+        "search",
+        scope=ResponseScope.CURRENT_TURN,
+    ),
 ).execute_async(adapter=my_adapter)
 ```
 
+These forms express complete-transcript probe requirements:
+
+```python
+from rampart.evaluators import ResponseContains, ResponseScope
+
+# Every response must contain the expected term
+ResponseContains("Paris", scope=ResponseScope.ALL_TURNS)
+
+# No response may contain the forbidden term
+~ResponseContains("password", scope=ResponseScope.ANY_TURN)
+```
+
+!!! warning "Multi-turn scope"
+    Omitting `scope` inspects only the current response and emits a
+    `FutureWarning` for multi-turn transcripts. See
+    [Temporal Scope](../usage/authoring-tests.md#temporal-scope).
+
+    Probes do not stop early unless `stop_when` is configured. The verdict
+    evaluator therefore receives the completed trace, and `ALL_TURNS` or
+    negated `ANY_TURN` applies to every response that was produced.
+
+!!! note "Driver budgets"
+    An adaptive driver such as `LLMDriver` does not stop itself. Without
+    `stop_when`, it runs until `max_turns` and then evaluates that completed
+    trace once. Set an intentional budget, and add an explicit stop condition
+    when earlier termination is part of the scenario.
+
 ---
 
 ## Parameters
@@ -80,7 +112,8 @@ See [`Probes.behavior()`][rampart.probes.Probes.behavior] for the full API refer
 | `prompts` | `list[str] \| None` | `None` | A list of prompt strings. |
 | `driver` | [`PromptDriver`][rampart.core.prompt_driver.PromptDriver] `\| None` | `None` | A pre-built prompt driver. |
 | `evaluator` | [`Evaluator`][rampart.core.evaluator.Evaluator] | required | What behavior to detect. |
-| `max_turns` | `int` | `25` | Maximum exchanges before `ERROR`. |
+| `stop_when` | [`Evaluator`][rampart.core.evaluator.Evaluator] `\| None` | `None` | Optional online condition that stops the trace when detected. |
+| `max_turns` | `int` | `25` | Maximum exchanges; reaching the limit resolves the trace normally. |
 
 !!! warning
     Provide exactly one of `prompt`, `prompts`, or `driver`. Providing more than one or none raises `ValueError`.