feat(sdk): Evaluator v2 outbound worker runtime — managed + self-hosted pods, fork+exec sandbox, review hardening

CHANGELOG.md

@@ -10,6 +10,8 @@
 
 - `fp-cloud-cli`: typer 0.27.1 → 0.27.2, click 8.4.2 → 8.5.0, posthog 7.42.0 → 7.44.2 (#771)
 
+- `osv-scanner.toml` ignores GHSA-8mgp-746c-j5xp (nltk 3.10.3, CVSS 8.3) until 2026-11-20. Surfaced 2026-09-03 against an already-approved PR — nothing on the branch introduced it and nothing on the branch can resolve it, which is the case the allow-list exists for. nltk is transitive via `llama-index-core` and appears only in `sdk/python/uv.lock`, the dev/test lockfile that pins every extra so CI can exercise the adapters; the SDK's own runtime dependency list is empty and the `llamaindex` extra is imported lazily, so it reaches no shipped code path. OSV reports "0 vulnerabilities can be fixed" and an empty FIXED VERSION, so the only alternative is dropping the extra and its adapter tests. Dated to match the chromadb entries so the list is revisited in one pass
+
 - browserslist pinned to 4.28.8 in `overrides`, closing GHSA-73wf-gq98-2v4g and GHSA-c83g-rgw3-j3cx (both 7.5, both fixed in 4.28.7). They turned `main` red on its own scheduled Supply Chain run rather than on any PR's change — disclosed after this branch's first CI run, the same surface-late mechanism `osv-scanner.toml` documents for chromadb. browserslist is transitive-only (via `@babel/helper-compilation-targets`'s `^4.24.0`), so this is an override pin, not a dependency bump — and not `bun update browserslist`, which adds it to `dependencies` as a direct dep it is not and leaves 4.28.2 nested under `@babel/helper-compilation-targets`, keeping the gate red (#771)
 
 ## 1.0.3 — 2026-08-31

fp-cloud-cli/fp_cli/permissions.py

@@ -26,6 +26,7 @@
     "users:delete",
     "evaluations:read",
     "evaluations:trigger",
+    "evaluations:run",
     "dashboards:read",
     "dashboards:write",
     "dashboards:delete",

osv-scanner.toml

@@ -82,3 +82,27 @@ reason = "chromadb 1.1.1 (published 2026-06-12, surfaced 2026-08-24). No fixed v
 id = "GHSA-xph7-9rjv-w5fr"
 ignoreUntil = 2026-11-20
 reason = "chromadb 1.1.1 (published 2026-06-12, surfaced 2026-08-24). No fixed version. Transitive via crewai, dev/test lockfile only; see the shared justification above. Re-review by 2026-11-20."
+
+# ---------------------------------------------------------------------------
+# nltk 3.10.3 — one advisory, no fixed version.
+#
+# Surfaced 2026-09-03 against an already-approved PR, which is the case this
+# file exists for: nothing on the branch introduced it, and nothing on the
+# branch can resolve it.
+#
+# nltk is transitive via llama-index-core and appears only in
+# sdk/python/uv.lock, the dev/test lockfile that pins every extra so CI can
+# exercise the adapters. The SDK's own runtime dependency list is empty — the
+# `llamaindex` extra is opt-in and the integration module imports it lazily —
+# so nltk reaches no shipped code path. OSV reports "0 vulnerabilities can be
+# fixed" and an empty FIXED VERSION, so there is nothing to bump to; the only
+# alternative is dropping the llamaindex extra and the adapter tests with it.
+#
+# Same re-review date as the chromadb entries above, so the whole list is
+# revisited in one pass.
+# ---------------------------------------------------------------------------
+
+[[IgnoredVulns]]
+id = "GHSA-8mgp-746c-j5xp"
+ignoreUntil = 2026-11-20
+reason = "nltk 3.10.3 (surfaced 2026-09-03, CVSS 8.3). No fixed version. Transitive via llama-index-core, dev/test lockfile only; the SDK ships zero runtime dependencies. Re-review by 2026-11-20."

sdk/python/CHANGELOG.md

@@ -16,6 +16,154 @@ moved the version here automatically; nothing has landed against `0.0.1b2` yet.
 Add entries as changes merge — this section becomes the GitHub Release body when
 it ships.
 
+- Retire the old inbound evaluator boundary and add evaluator authoring plus the
+  outbound-only v2 worker runtime under the lazy `failproofai_sdk.evaluator`
+  namespace.
+- Harden the managed-evaluator source sandbox against a class of escapes an
+  adversarial review found: `str.format`/`format_map` C-level field traversal,
+  generator/frame introspection (`gi_frame.f_globals`) that reached the eval
+  globals and could poison a process-shared namespace across evaluations, and
+  `type.mro()` type-object reach. Attribute access is now **default-deny** (an
+  allowlist of the transcript data surface plus pure string/collection methods,
+  so every current and future introspection attribute is rejected), each eval
+  runs with **fresh per-call globals**, and a result whose text embeds a runtime
+  object repr (`<... at 0x...>`, a heap-pointer/ASLR disclosure that falls out of
+  any bound method's repr) is rejected at the output boundary. `enumerate` and
+  bare generator expressions are no longer permitted — both were gratuitous
+  pointer-repr sources; use `range(len(...))` and list/set/dict comprehensions.
+- Report **why** a server-authored definition was rejected. Every failure
+  collapsed to `evaluation raised <TypeName>`, so a hosted definition that can
+  never run reported only `evaluation raised UnsafeEvaluatorSource` — on every
+  session, forever, with nothing telling the author what was wrong. It matters
+  because the server accepts any source passing its size and key checks and does
+  not validate the sandbox's single-expression grammar, so a structurally
+  unrunnable definition is published successfully and then fails silently.
+  `UnsafeEvaluatorSource` now carries its detail (`evaluator_source must be one
+  expression`), bounded to `MAX_ERROR_MESSAGE_BYTES`. Deliberately narrower than
+  the generic handler, which still reports the type name only: this exception is
+  raised by our own validator before any customer source executes and describes
+  the source's shape, so it embeds no transcript content.
+- Scrub the sandbox child's environment. `subprocess.Popen` inherited
+  `os.environ`, so the process executing untrusted server-authored source ran
+  with `FAILPROOFAI_EVALUATOR_TOKEN` in its environment — on the managed pod,
+  the cross-tenant credential. Defence in depth rather than a live escape (the
+  AST allowlist and empty `__builtins__` already stop a managed expression
+  reaching `os.environ`): a future gap there can no longer be escalated into
+  credential theft. Only what the interpreter needs is forwarded, `PYTHONPATH`
+  included.
+- Contain a poison managed definition to its own run: source is now compiled
+  lazily inside the per-run executor, so a definition the sandbox rejects
+  dead-letters as one bounded `failed`/`eval_error` run instead of crashing the
+  assignment task and forcing it to be reclaimed until its attempt budget runs
+  out.
+- Run managed (server-authored) evaluations in a **killable fork+exec'd
+  subprocess** with hard `RLIMIT_CPU` + `RLIMIT_AS` + a parent wall-clock kill —
+  so a compute/memory bomb in a hosted definition (`sum(range(10**20))`) can no
+  longer exhaust the worker (SEC-001). Cancelling an in-process thread does not
+  stop it; a fresh subprocess the kernel bounds and the parent terminates does. A
+  plain `os.fork()` would deadlock — the worker is multi-threaded (asyncio loop,
+  executor, writer) and forking one hangs the child on an inherited lock — so the
+  sandbox execs a fresh `python -m ..._sandbox_runner` that sets its own limits;
+  the transcript crosses in via `to_wire`, only the result crosses back. The
+  effective budget is **clamped to a hard ceiling** (`MAX_SANDBOX_TIMEOUT_SECONDS`,
+  60s) so a large server-provided `timeout_seconds` cannot remove the bound.
+  Managed conditions, which previously ran with no timeout at all, are sandboxed
+  the same way. The result crossing back is **bounded on both sides** — the child
+  validates it (`result_items`, the 25-result limit) and refuses to serialize
+  anything over 1 MiB, and the parent reads at most that before killing the child
+  — so an oversized result (`metrics={str(x): 1 for x in range(100000)}`) cannot
+  OOM the worker either. The per-sandbox address space is capped (512 MiB) and the number of concurrent sandbox processes is bounded (a semaphore), so the AGGREGATE memory is bounded independent of the worker's `max_concurrency` — a fleet of concurrent runs can't OOM the host. Fails **closed** (`EvaluationSandboxUnavailable`) if the
+  sandbox cannot be spawned or the transcript cannot be serialized. Defense in depth at
+  compile time: reject `**` with a large/non-constant exponent and cap total AST
+  size. A managed condition the sandbox rejects now dead-letters as
+  `condition_error` instead of stranding the assignment. Only server-authored
+  source is isolated this way; customer evaluators still run in-process.
+- Require `execution_mode` on the wire instead of coercing a falsy/missing value
+  to `local` — a malformed value silently ran a `python` definition down the
+  customer path (or vice-versa); it is now a hard protocol error.
+- Count the sandbox-slot wait against the execution timeout (SEC-001). `_run_sandboxed`
+  acquired the `MAX_CONCURRENT_SANDBOXES` slot with an UNBOUNDED wait and only started
+  its wall-clock deadline afterward — so a run queued behind busy slots could, after the
+  runtime's `asyncio.wait_for` already reported it timed out (that cancels only the
+  awaiter, not the executor thread), still acquire a slot and launch a sandbox; 28
+  threads could pile up behind 4 long sandboxes and starve the worker (conditions have
+  no runtime-level wait at all). One wall-clock deadline now covers BOTH the slot wait
+  and execution: the slot is acquired with the remaining budget, and on timeout the run
+  raises `EvaluationTimeout` **without spawning a child**. Regression test: more
+  concurrent compute bombs than slots all resolve within ~one budget, not N serialized
+  budgets.
+- A managed (`python`) definition's applicability is now governed by the SERVER's
+  `condition_source`, never a colliding local condition (COR-001). `process_assignment`
+  keyed the local-definition lookup on `(eval_key, eval_version)` alone and selected
+  `local.condition` whenever a local definition with that key existed — so a managed
+  definition whose server condition was false could be forced to run anyway if the
+  worker had also registered a local definition under the same key whose condition was
+  true, executing server-managed source against the operator's intent. Condition
+  selection now branches on `execution_mode`, mirroring the evaluator branch: `LOCAL`
+  uses `local.condition`, `PYTHON` compiles and runs the server's `condition_source`
+  regardless of any key collision. Regression test: identical local+managed keys, local
+  condition true and managed false, asserts the definition is skipped and no managed run
+  is submitted.
+- Recognize the server's `incomplete_plan` terminal error (API-001). The server rejects a
+  plan that fails to cover every snapshotted definition with `422 incomplete_plan`; that
+  code is now in the SDK's `ERROR_SPECS` mirror and the shared `contract.json` fixture
+  (byte-identical with the server's), so a worker no longer treats a valid server-defined
+  failure as an unrecognized error. The fixture-equality test covers it.
+- Close a heap-address disclosure bypass in the managed-source sandbox
+  (adversarial-audit SEC). The output-boundary guard that rejects a `<obj at 0xADDR>`
+  repr in a result field was anchored on the literal `<`, so an allow-listed
+  `str(payload.get).replace("<", "")` — or an f-string / `%`-format of a bare bound
+  method — kept the live heap address while stripping the match, leaking an
+  ASLR/memory-layout primitive of the sandbox process into a persisted result. The fix
+  moves the defense to compile time: a bound method (the only reachable value with a
+  pointer repr — the transcript and result types are all frozen, pointer-free
+  dataclasses) may now only be **called**, never referenced as a bare value, so no
+  reachable value can carry a pointer repr through `str()`, an f-string, or `%`. The
+  output-boundary scan is kept and broadened (no longer requires the leading `<`) as
+  defense in depth. Legitimate evaluations — which call methods and read data
+  attributes — are unaffected; regression tests cover the `.replace("<","")`, f-string,
+  and `%` bypasses and confirm called-method/data-attribute stringification still works.
+- Switch the worker from long-polling to **normal (short) polling**, matching the
+  cadence of our other cloud surfaces. `claim` no longer sends `wait_seconds` and
+  the server returns immediately; when a claim comes back empty the worker sleeps
+  the server-advertised `poll_interval_seconds` (from the register response,
+  default 10 s) before polling again, instead of holding a request open for up to
+  25 s. Removes the `claim_wait_seconds` config knob and the
+  `request_timeout_seconds > claim_wait_seconds` constraint; the poll cadence is
+  now tuned centrally by the server, not per worker.
+- Bound the pre-plan condition phase by the assignment lease (hermes advisory).
+  Conditions were evaluated serially with no lease awareness before the plan was
+  created, and a managed condition could run its full sandbox budget — so a few
+  near-budget conditions could burn the whole lease before the plan request and
+  the server would fence the plan as `lease_lost`, reclaiming the assignment in a
+  loop instead of submitting a result. Each condition is now capped to the lease
+  time remaining before a plan-submission margin (using `lease_expires_at` when it
+  is in the future, else the negotiated lease duration), and once that budget is
+  gone the remaining conditions are skipped as `lease_exhausted` rather than run.
+  Local conditions, which previously had no timeout at all, are bounded the same
+  way. The complete fix — renewing the lease *during* the condition phase — needs
+  a server-side pre-plan heartbeat and is tracked separately.
+- Make a timed-out **synchronous** evaluator observable and stop it starving the
+  worker (hermes advisory). A synchronous evaluator that overruns its timeout runs
+  in the executor thread and cannot be cancelled (CPython cannot interrupt a
+  running thread), so its thread was permanently lost; with the pool sized to the
+  concurrency limit, one such orphan on a single-slot worker silently stopped all
+  further local evaluation. The eval executor now carries headroom over the
+  semaphore so an orphaned thread does not immediately starve live capacity — the
+  semaphore stays the real concurrency bound — and each orphan increments
+  `sync_evaluations_orphaned` and logs a warning naming the evaluator, so a hung
+  one is findable. This is a finite cushion, not a cure for a permanently-blocked
+  evaluator; prefer `async def` evaluators (cooperatively cancellable) or managed
+  `python` evaluators (subprocess-isolated, hard-killed) for long or untrusted work.
+- Let a managed source's list/set/dict comprehension read `session` on CPython
+  3.10/3.11 (hermes COR-001). The sandbox eval put `session` in the eval *locals*,
+  but a comprehension runs in its own scope and resolves a free name like
+  `session` from *globals* — so on 3.10 (a supported version) an allowed source
+  such as `all([session.event_count > 0 for i in range(1)])` raised `NameError`.
+  `session` now goes in a fresh per-call globals mapping and both eval paths use
+  empty locals, which keeps isolation and works across 3.10–3.14. Regression test
+  runs on the whole version matrix.
+
 ## 0.0.1b1 — 2026-08-24
 
 The first release under this name. Everything below describes the package as it

sdk/python/README.md

@@ -10,6 +10,14 @@ the platform.
 - **Dependencies:** none. Standard library only, so installing it constrains
   nothing else in your environment.
 
+## Evaluator v2 status
+
+The legacy inbound `agenteye-evaluator` package has been retired; do not build new
+evaluator services against its server-push HTTP contract. Evaluator v2 authoring
+and its customer-hosted, outbound-only worker runtime live under the lazy
+`failproofai_sdk.evaluator` namespace. Importing the top-level tracing SDK does not
+import or start the evaluator runtime.
+
 ## Installation
 
 ```bash

sdk/python/examples/evaluator_worker.py

@@ -0,0 +1,121 @@
+"""Customer evaluator with deterministic and optional async judge checks."""
+
+from __future__ import annotations
+
+import asyncio
+import ipaddress
+import json
+import os
+from urllib.parse import urlsplit
+from urllib.request import HTTPRedirectHandler, Request, build_opener
+
+from failproofai_sdk.evaluator import (
+    ConditionResult,
+    EvalResult,
+    Evaluator,
+    Metric,
+    Score,
+)
+
+app = Evaluator(name="customer-production", version="2026.08.1")
+
+
+class _RejectRedirects(HTTPRedirectHandler):
+    def redirect_request(self, request, file_pointer, code, message, headers, new_url):
+        return None
+
+
+@app.eval(
+    "tool_efficiency",
+    version="1.0.0",
+    labels=["tools", "deterministic"],
+    when=lambda session: ConditionResult(
+        session.count("tool_use") > 0, "no_tool_calls"
+    ),
+)
+def tool_efficiency(session):
+    calls = session.events_of_type("tool_use")
+    distinct = {
+        event.payload.get("tool_name")
+        for event in calls
+        if event.payload.get("tool_name")
+    }
+    value = len(distinct) / len(calls)
+    return EvalResult(
+        score=Score(value, passed=value >= 0.7),
+        metrics={
+            "tool_call_count": Metric(len(calls), unit="events"),
+            "distinct_tool_count": Metric(len(distinct), unit="tools"),
+        },
+        reasoning=f"{len(distinct)} distinct tools across {len(calls)} calls",
+    )
+
+
+def _judge_configured(session):
+    configured = bool(os.environ.get("EXAMPLE_JUDGE_URL"))
+    return ConditionResult(configured, "judge_not_configured")
+
+
+def _last_content(session, event_type):
+    events = session.events_of_type(event_type)
+    if not events:
+        return None
+    payload = events[-1].payload
+    fields = {
+        "human_input": ("response",),
+        "model_response": ("content",),
+        "agent_end": ("summary",),
+    }.get(event_type, ("content", "summary", "response"))
+    return next((payload.get(field) for field in fields if payload.get(field)), None)
+
+
+def _call_judge(question, answer):
+    url = os.environ["EXAMPLE_JUDGE_URL"]
+    parsed = urlsplit(url)
+    if parsed.scheme not in {"http", "https"} or not parsed.netloc:
+        raise ValueError("EXAMPLE_JUDGE_URL must be an absolute http(s) URL")
+    hostname = parsed.hostname
+    loopback = hostname == "localhost"
+    if hostname is not None and not loopback:
+        try:
+            loopback = ipaddress.ip_address(hostname).is_loopback
+        except ValueError:
+            loopback = False
+    if parsed.scheme != "https" and not loopback:
+        raise ValueError("EXAMPLE_JUDGE_URL must use https unless it targets loopback")
+    token = os.environ.get("EXAMPLE_JUDGE_TOKEN")
+    body = json.dumps({"question": question, "answer": answer}).encode("utf-8")
+    headers = {"Content-Type": "application/json", "Accept": "application/json"}
+    if token:
+        headers["Authorization"] = f"Bearer {token}"
+    request = Request(url, data=body, headers=headers, method="POST")
+    with build_opener(_RejectRedirects()).open(request, timeout=25) as response:  # nosec B310
+        result = json.loads(response.read(64 * 1024))
+    return float(result["score"]), str(
+        result.get("reasoning") or "Judge returned no reasoning"
+    )
+
+
+@app.eval(
+    "answer_relevance",
+    version="judge-api-v1",
+    labels=["llm_judge", "relevance"],
+    when=_judge_configured,
+    timeout_seconds=30,
+)
+async def answer_relevance(session):
+    question = _last_content(session, "human_input")
+    answer = _last_content(session, "model_response")
+    if question is None or answer is None:
+        raise ValueError("answer relevance requires human input and model output")
+    value, reasoning = await asyncio.to_thread(_call_judge, question, answer)
+    value = min(max(value, 0.0), 1.0)
+    return EvalResult(
+        score=Score(value, passed=value >= 0.7),
+        reasoning=reasoning,
+        labels=("llm_judge", "relevance"),
+    )
+
+
+if __name__ == "__main__":
+    app.run_from_env()