Follow through on external evaluation: installation, honest guarantees, and shared-learning evidence

[timeleft--]

Outcome

Turn external evaluation feedback into a smaller installation burden, accurate claims, and a measurable cross-session learning test. Preserve the positive evidence for rubric-based promotion and supersession without treating it as proof of factual truth or overall value.

First blockers

• JJ defaults to 0.28.0; current official stable is 0.45.1 as of 2026-09-09. Compatibility must be tested, and existing compatible installations must not be silently downgraded.
• Governed recall and MCP registration fixes are merged, but the latest public package remains 0.6.0. Prepare and verify the released artifact before telling evaluators to upgrade for these fixes.
• The evaluator exercised MCP directly, but native client registration and the two-session experiment did not complete. No elapsed trial time or net benefit is established.

Tasks

• P1 — Stop installing stale JJ by default; preserve compatible installed versions #98 — Stop installing stale JJ by default; preserve compatible installed versions
• P1 — Deliver merged recall isolation in a verified release and clarify runtime versions #99 — Deliver merged recall isolation in a verified release and clarify runtime versions
• P2 — Correct retrieval and Trust Gate claims; capture a reproducible search baseline #100 — Correct retrieval and Trust Gate claims; capture a reproducible search baseline
• P1 — Make Trust Gate data egress explicit and verify local-only configuration #101 — Make Trust Gate data egress explicit and verify local-only configuration
• P1 — Handle obvious secrets before draft persistence and remote review #102 — Handle obvious secrets before draft persistence and remote review
• P2 — Keep onboarding out of application .env files and verify native MCP registration #103 — Keep onboarding out of application .env files and verify native MCP registration
• P2 — Measure and reduce MCP context overhead without losing the learning workflow #104 — Measure and reduce MCP context overhead without losing the learning workflow
• P2 — Make local-only operation and missing-remote sync behavior clear #105 — Make local-only operation and missing-remote sync behavior clear
• P2 — Evaluate cross-session learning against a committed-docs baseline #106 — Evaluate cross-session learning against a committed-docs baseline

Existing delivery, not duplicate implementation

• Enforce governed FAVA recall visibility and atomic supersession #72 / feat: govern recall visibility and successor approval #93: governed reads, server identity, and approved supersession visibility are merged.
• Migrate FAVA Trails to MCP 2.0 compatibility #83 / fix: support MCP 2 server registration and protocol compatibility #94: current MCP compatibility and server registration changes are merged.
• Allow Trust Gate to use a local OpenAI-compatible model (Unsloth Studio) #85 / feat: allow Trust Gate local OpenAI-compatible providers #86 / feat: add per-machine Trust Gate config #87: local OpenAI-compatible Trust Gate and per-machine configuration already exist, including in 0.6.0.
• PRD skeleton: FAVA search and retrieval layer #59 remains the search discovery issue; the new retrieval task supplies an honest baseline and user evidence without preselecting a search architecture.

Order and dependencies

Start with JJ compatibility and release preparation. Documentation, safe onboarding, context measurement, local-only review validation, and local sync UX can progress independently. The two-session evaluation depends on a verified installed artifact, actual native registration, distinct identities, and an agreed data/review destination. Review candidate secret handling before inviting tests with sensitive data.

Decisions before an external evaluation

Agree who will participate, which synthetic or explicitly authorized data may be used, the local or cloud review destination, and the duration and success criteria. No new participation, provider consent, or formal pilot approval is implied by this issue. Historical data deletion or history rewriting is outside this follow-up.

Evidence limits

The reported test had seven LLM reviews: three approvals and four rejections. It supports the rubric and lineage behavior observed in that run. It does not prove prevention of initial secret storage or remote transmission, factual correctness of approvals, consistent agent adherence, or better outcomes than committed documentation.

Source baseline: main 6c5278a40a86246014901a88417f3455a46cdfcc, live GitHub/PyPI release metadata checked 2026-09-09, and summarized external feedback. The public issues exclude the evaluator’s private project details and original attachment. Creating this checklist does not claim its implementation is complete.