Skip to content

Stabilize live browser eval runtime#26

Merged
justsml merged 32 commits into
mainfrom
agent/eval-runtime-recovery
Jul 13, 2026
Merged

Stabilize live browser eval runtime#26
justsml merged 32 commits into
mainfrom
agent/eval-runtime-recovery

Conversation

@justsml

@justsml justsml commented Jul 13, 2026

Copy link
Copy Markdown
Owner

What changed

  • Stabilized the live browser-eval lifecycle: shared Juice Shop runs serialize safely, wait for target readiness, and browser /api/chat requests now cancel before the row deadline.
  • Made browser-side aborts report as timeout rather than a generic harness error.
  • Added strict Mastra inference-span polling so valid telemetry is not rejected before persistence completes.
  • Preserved the fresh smoke, repeat, and timeout artifacts, including LFM tuning evidence and the repaired DeepSeek/GLM/Qwen/GPT runs.

Why

Long-running live evaluations could leave a browser fetch pending after the Node-side timeout, while shared resettable Docker targets could race between rows. Those issues produced harness-only failures and prevented reliable repeat runs.

Validation

  • pnpm exec tsc --noEmit --pretty false
  • pnpm exec vitest run tests/security-chat/browser-e2e-failures.test.ts
  • Production /api/health check
  • Fresh strict-evidence GPT smoke
  • DeepSeek Flash repeats 2 and 3, plus GLM repeat 2, with persisted streams, usage, model-inference spans, and target cleanup.

@justsml
justsml marked this pull request as ready for review July 13, 2026 21:26
@justsml
justsml merged commit dad418d into main Jul 13, 2026
6 of 7 checks passed
@justsml
justsml deleted the agent/eval-runtime-recovery branch July 13, 2026 21:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant