bench(rdp): governed real-protocol vision-ladder qualification harness by abrichr · Pull Request #177 · OpenAdaptAI/openadapt-flow

abrichr · 2026-07-20T18:22:39Z

What

Adds a governed, repeatable qualification harness for OpenAdapt's vision-only
resolution ladder on a synthetic Tk workflow carried through a real FreeRDP
client/server round trip. The surface exposes no DOM, UIA, or other structural
backend, so healthy replay must resolve from recorded visual evidence.

Exact base: d952c363d1910f1699c1a4690002879b1990d743
Exact evidence head: 8a54a9003148f899867527802fc2d456ea2e5f87

Accepted evidence

GitHub Actions run
29978847851,
attempt 2, executed the exact 3-healthy/3-drift contract at mechanism commit
6031fde559b942a1d8b1a560d8b6cee8a6bfc800. Its sanitized artifact is committed
as benchmark/rdp_ladder/results.json with SHA-256
3b946dd098c4a4c2ca56c1ab531a2eaeeb37804aa10c440e576f7196a03ae37c.

Result:

3/3 healthy record → compile → governed replay trials passed.
Every armed target identity verified from live pixel evidence.
Policy admission, screen postconditions, runtime DocumentHashVerifier, and
the separate exact-content host oracle all confirmed.
Visual template rung used; structural rung use was zero.
Model calls, silent incorrect successes, and healthy over-halts were all zero.
3/3 injected-drift trials halted safely with zero writes, false completions,
and model calls.

The final authoritative run for the evidence-only head is
29979396980.
It passed the same 6/6 contract at exact head 8a54a900…; its immutable
artifact SHA-256 is
5247ac89ef004fb8873169b7a6ac62ad5eb765a37989c81f871f11fdc5f7f88d.

Safety and reproducibility contract

The harness:

binds evidence to a clean full candidate SHA and exact origin/main
merge-base;
runs exactly three healthy and three drift trials with trial-unique values
and acknowledged resets;
records and compiles the workflow through the production classes;
seals and encrypts the bundle before a strict policy/run gate;
verifies pixel identity before every pointer action, including Save;
binds the filesystem operation's idempotency key to its stable output path
and requires exactly one new document;
confirms the effect through both the runtime verifier and an independent
host read;
records per-step resolution, identity, postcondition, input, and effect
diagnostics; and
refuses qualification unless every healthy and drift invariant passes.

The narrowly path-filtered pull-request workflow runs when this harness or its
exact compiler/policy/runtime dependencies change. workflow_dispatch remains
available for explicit reruns. Action revisions and the fixture base image are
pinned.

Validation

Focused local suite: 83 passed.
Ruff lint and format: passed.
Release/public-artifact integrity: passed.
Claims/evidence gate: passed.
Publication artifact and license/source-boundary validation: passed on the
built wheel and sdist.
Final protected GitHub CI on 8a54a900…: all required checks passed,
including unit/coverage, browser E2E, wheel validation, lint/type, docs,
claims, PHI, Windows mock, and Linux AT-SPI.

Scope boundary

This is bounded evidence for a synthetic Tk task over real
FreeRDP-transported pixels and input. It exercises the production visual
resolver plus governed identity, policy, postcondition, and effect contracts.
It is not a Citrix ICA/HDX qualification, a Windows application qualification,
an aardwolf transport qualification, or WAN-captured drift. Those are separate
substrates and evidence sets.

…ted) (#181) * bench(canvas): no-DOM HTML5-canvas vision-ladder qualification (accepted) Adds `benchmark/canvas_ladder/` — a live proof of the vision-only resolution ladder over a genuine no-accessible-DOM HTML5 `<canvas>`, the class Citrix Workspace-*web* presents. Drives the UNMODIFIED Recorder -> compile_recording -> Replayer over a noVNC/TigerVNC canvas with NO structural backend, against a Dockerized clinic kiosk (reused from the rdp_ladder fixture, PR #177) + an independent document oracle. HONEST LABEL: no-DOM HTML5-canvas class — NOT Citrix ICA/HDX (no HDX codecs, no ICA compression, no Workspace-client input path). See README + the design note ~/oa/src/.private/rdp_citrix_validation_2026_07_20.md. results.json is `accepted: true` across three regimes: * healthy: record->compile->replay succeeds, ZERO model calls, VISUAL rungs only (template x3, structural never used), write independently confirmed; * moderate (legible) drift: resolves to the CORRECT target via ocr+geometry — never a silent WRONG write (robustness, not over-halt); * severe (illegible) drift: HALTS with no write and no model call — no blind coordinate replay, the anti-silent-wrong-action guarantee. Unlike PR #177 (partial: healthy effect unconfirmed), the healthy path here is clean — keyboard forwards through noVNC to the field and the oracle confirms the write. Adds an env-gated e2e (`OAFLOW_CANVAS_LADDER_E2E=1`) and a nightly `canvas-nodom-ladder` workflow (drafted, not a per-PR gate). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NyCHrzA1psrKMFfroYbzaM * style: ruff format the canvas-ladder e2e wrapper Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NyCHrzA1psrKMFfroYbzaM --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

abrichr · 2026-07-21T17:50:01Z

Reviewed/rebased code-only head: 9e5434d320d6683843032c22c4fb01b363fd6532 on exact base bd2a27caf300e58d2b19dcee3b1493c5ab6fa813.

Local non-injecting gates passed: 148 focused tests (2 opt-in Docker skips), Ruff 0.15.22 lint/format, strict safety mypy, claims, release consistency, shell/diff checks, and actual wheel+sdist license-boundary validation. No Docker fixture, VM/cloud resource, GUI, or input-injection run was started.

The committed v1 result remains honestly accepted: false. The remaining external evidence gate is the manual-only docker-rdp-vision-ladder workflow at this exact head: a v2 artifact with exactly 3 healthy + 3 drift trials and accepted: true, meeting the identity/policy/runtime+host effect and halt/no-write contract detailed in the updated PR body. The manual workflow was not dispatched. Keep draft; this is not general RDP, Windows-app, aardwolf, or Citrix evidence.

abrichr · 2026-07-21T23:40:25Z

Follow-up live-state refresh: main advanced after the first green CI run. The branch was rebased again and revalidated against the landed ambiguity-refusal resolver. Current reviewed code-only head: 092ffbc7b863633dc931b61819ae57950a3aa98b on exact base e740a7e4b68ed5c6461bdcd6fbd086e8ac75c8be.

Local non-injecting gates passed on the new base: 209 focused/resolver tests (2 opt-in Docker skips), Ruff 0.15.22 lint/format, strict safety mypy, claims, release consistency, shell/diff checks, and actual wheel+sdist license-boundary validation. No Docker fixture, VM/cloud resource, GUI, or input-injection run was started.

The committed v1 result remains honestly accepted: false. The remaining external evidence gate is the manual-only docker-rdp-vision-ladder workflow at this exact head: a v2 artifact with exactly 3 healthy + 3 drift trials and accepted: true, meeting the identity/policy/runtime+host effect and halt/no-write contract detailed in the updated PR body. The manual workflow was not dispatched. Keep draft; this is not general RDP, Windows-app, aardwolf, or Citrix evidence.

abrichr · 2026-07-23T00:18:06Z

Second live-state refresh: main advanced again after the prior green CI run. The branch was rebased and revalidated against the latest landed main. Current reviewed code-only head: 411cd37d3e8e6ae6d42c81f1050e2f1146536845 on exact base abe9f36b889d2939e389a07348bf4a68e5d54996.

Local non-injecting gates passed on the new base: 209 focused/resolver tests (2 opt-in Docker skips), Ruff 0.15.22 lint/format, strict safety mypy, claims, release consistency, shell/diff checks, and actual wheel+sdist license-boundary validation. No Docker fixture, VM/cloud resource, GUI, or input-injection run was started.

The committed v1 result remains honestly accepted: false. The remaining external evidence gate is the manual-only docker-rdp-vision-ladder workflow at this exact head: a v2 artifact with exactly 3 healthy + 3 drift trials and accepted: true, meeting the identity/policy/runtime+host effect and halt/no-write contract detailed in the updated PR body. The manual workflow was not dispatched. Keep draft; this is not general RDP, Windows-app, aardwolf, or Citrix evidence.

abrichr · 2026-07-23T00:24:39Z

Reproducibility follow-up on exact head 6f67366e59e626155629cc413b0e4ab69f89144a, still based on abe9f36b889d2939e389a07348bf4a68e5d54996:

pinned checkout, setup-python, and upload-artifact to the repository's currently approved full commit SHAs;
pinned readable ubuntu:24.04 to current multi-architecture manifest digest sha256:4fbb8e6a8395de5a7550b33509421a2bafbc0aab6c06ba2cef9ebffbc7092d90;
refreshed the public-artifact provenance hash.

Code-only revalidation passed: actionlint, 209 focused/resolver tests with 2 opt-in Docker skips, claims/release/diff/shell gates, fresh wheel+sdist build, and actual-archive license/source-boundary validation. No Docker build, fixture, VM/cloud resource, GUI, or input-injection run was started. The exact amd64 Docker build plus 3 healthy + 3 drift qualification remains the manually authorized external workflow gate; it was not dispatched. Keep draft.

abrichr · 2026-07-23T00:29:30Z

Qualification-trigger follow-up at exact head 0b46905a5900d003fced27417a5d6b5d82fbaa0c on base abe9f36b889d2939e389a07348bf4a68e5d54996:

GitHub cannot dispatch a new workflow_dispatch workflow before that workflow exists on the default branch. To make the qualification a real pre-merge gate, this head adds a pull_request trigger narrowly filtered to the workflow, benchmark/rdp_ladder/**, dependency manifests, and the exact compiler/policy/RDP/runtime identity-resolution-effect files that can change the contract. workflow_dispatch remains for later main-based reruns; there is still no schedule.

The checkout explicitly selects the PR head instead of GitHub's synthetic merge ref, and the harness records git rev-parse HEAD plus its origin/main merge-base, preserving exact candidate provenance. The 3 healthy + 3 drift acceptance contract is unchanged. Local actionlint, 209 focused tests with 2 Docker-gated skips, claims/release/shell/diff checks, and fresh actual-archive validation passed before the lease-safe push.

This push triggers both normal CI and the amd64 GitHub-hosted RDP qualification. Keep draft while they run; no PR number is hardcoded and no manual workflow was dispatched.

…l contract) Adds benchmark/rdp_ladder: the missing live proof for the vision-only resolution ladder (template -> template_global -> ocr -> geometry) on a real no-DOM RDP PIXEL surface — distinct from benchmark/rdp (transport+input only) and the UIA/AX structural proofs. Drives the unmodified Recorder -> compile_recording -> Replayer over a genuine RDP round-trip with no structural backend, plus a Dockerized kiosk fixture and an independent document oracle. First run (results.json) is HONEST/PARTIAL — accepted:false: proven: zero model calls on healthy replay; visual rungs only (structural never used); no silent write under injected DPI/theme/JPEG drift. NOT yet: healthy effect independently confirmed; clean safe-halt under drift. Caveats (in results.json): real RDP-transported pixels + input, but NOT Citrix ICA/HDX, NOT the aardwolf transport, and drift is simulated-on-a-real-session (not WAN capture). Landing as WIP so the harness isn't lost; follow-up to reach a clean acceptance + wire the docker CI job. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NyCHrzA1psrKMFfroYbzaM

This was referenced Jul 20, 2026

bench(canvas): no-DOM HTML5-canvas vision-ladder qualification (accepted) #181

Merged

fix(ci): unblock main — de-symlink workshop bib (sdist) + deterministic pixel font test #189

Merged

abrichr force-pushed the feat/rdp-vision-ladder-e2e-20260720 branch from 5c4529b to 9e5434d Compare July 21, 2026 17:48

abrichr changed the title ~~bench(rdp): real-RDP vision-ladder qualification harness (WIP, partial contract)~~ bench(rdp): governed real-protocol vision-ladder qualification harness Jul 21, 2026

abrichr force-pushed the feat/rdp-vision-ladder-e2e-20260720 branch from 9e5434d to 092ffbc Compare July 21, 2026 23:39

abrichr force-pushed the feat/rdp-vision-ladder-e2e-20260720 branch from 092ffbc to 411cd37 Compare July 23, 2026 00:16

abrichr and others added 6 commits July 22, 2026 20:37

fix(rdp): make real-wire qualification deterministic

017c8f4

fix(rdp): govern vision-ladder qualification evidence

a0e58d8

ci(rdp): pin qualification dependencies

a18daa9

ci(rdp): qualify pull request changes

4d6216a

fix(rdp): wait for selected record transition

8305a34

abrichr force-pushed the feat/rdp-vision-ladder-e2e-20260720 branch from 0b46905 to 8305a34 Compare July 23, 2026 03:40

abrichr added 6 commits July 22, 2026 20:46

fix(rdp): anchor the note field distinctly

7a520b0

test(rdp): prove compiled crop verifies healthy identity

bd8b109

test(rdp): retain per-step qualification diagnostics

73ef331

fix(rdp): encode punctuation as X11 keysyms

baa1e6b

fix(rdp): bind document idempotency to output path

6031fde

bench(rdp): record accepted 3+3 qualification evidence

8a54a90

abrichr marked this pull request as ready for review July 23, 2026 04:51

abrichr merged commit affedc5 into main Jul 23, 2026
18 checks passed

abrichr deleted the feat/rdp-vision-ladder-e2e-20260720 branch July 23, 2026 04:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

bench(rdp): governed real-protocol vision-ladder qualification harness#177

bench(rdp): governed real-protocol vision-ladder qualification harness#177
abrichr merged 12 commits into
mainfrom
feat/rdp-vision-ladder-e2e-20260720

abrichr commented Jul 20, 2026 •

edited

Loading

Uh oh!

abrichr commented Jul 21, 2026

Uh oh!

abrichr commented Jul 21, 2026

Uh oh!

abrichr commented Jul 23, 2026

Uh oh!

abrichr commented Jul 23, 2026

Uh oh!

abrichr commented Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

abrichr commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Accepted evidence

Safety and reproducibility contract

Validation

Scope boundary

Uh oh!

abrichr commented Jul 21, 2026

Uh oh!

abrichr commented Jul 21, 2026

Uh oh!

abrichr commented Jul 23, 2026

Uh oh!

abrichr commented Jul 23, 2026

Uh oh!

abrichr commented Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

abrichr commented Jul 20, 2026 •

edited

Loading