Forgekit — reliability infrastructure for AI coding agents

Your coding agent forgets. Forgekit remembers — outside the model, in plain git.

Forgekit is a Node.js CLI and MCP server (zero runtime dependencies) that gives your AI coding tools a reliability layer: agent memory that survives sessions and teammates, pre-push verification gates, and guardrails that run before and after every tool call. You author project rules once, and it emits native configuration for 12 agent tools. Claude Code is the most deeply exercised integration.

Inventory	Count
CLI commands (forge --help)	51
Lifecycle hook commands (SessionStart, UserPromptSubmit, PreToolUse, PostToolUse, Stop)	17
Config emitters (src/emit/)	13
Installable skill packs (skills/)	3
Per-event hook token budgets (hooks/budget.json)	5

Tool	Native config	MCP server entry	Automatic hooks
Claude Code	✅	✅	✅ (all 5 events, blocking)
Codex, Cursor, Gemini, Copilot, Zed, Continue	✅	✅	—
Aider, Windsurf, OpenClaw, Kimi Code	✅	—	—
Roo Code, VS Code	—	✅	—

Why it wins, measured:

• Hooks stay cheap. SessionStart injects a pointer index, not the full memory — measured 18,262 B of durable memory → 826 B index — and every hook event has a hard byte budget (hooks/budget.json: 12 KB SessionStart, 4–6 KB the rest), enforced by the launcher. No silent context bloat.
• Verification that can actually fail the push. forge verify runs the repo's real test suites and reports PASS / FAIL / INCOMPLETE / NOT_CONFIGURED against a tree fingerprint of the code it ran on — a red run blocks the push, not just the chat.
• Blast radius before you edit. forge impact lists the files a change probably touches from a cached code graph: 1.68 ms warm (the cache is built by forge init).

It does three jobs:

1. Share reliable project knowledge. Lessons, facts and decisions live in the repository as evidence-referenced memory: each claim carries references to its evidence, only tests, CI or a person raise its confidence, and teammates merge it over plain git.
2. Inspect likely change impact. Before an edit, a heuristic impact analysis lists the files the change probably touches, including coupled files you did not name. It reads a regex-derived code graph, so it can miss files as well as over-warn.
3. Verify the current change. forge verify runs the repository's detected test suites and reports PASS, FAIL, INCOMPLETE or NOT_CONFIGURED for the code state it ran on. A run that covers only part of the repository is partial verification, and says so.

Status: beta — read before you rely on it.

• Releases follow Semantic Versioning (CHANGELOG, release notes): since 1.0.0 a breaking change ships as a new major version. "Beta" describes maturity, not interface churn — the analyses are heuristic, the checks are advisory unless you enable enforcement, and most integrations have less real-world exercise than Claude Code. The core (init, sync, substrate, impact, ledger, guards) is tested and in daily use.
• Claude Code is the deepest-tested integration (full plugin, ambient UserPromptSubmit guards). The other eleven tools receive native config (most also an MCP server entry; Aider, Windsurf/Devin and Kimi Code do not) but no automatic hooks, and have had less real-world exercise. On OpenClaw specifically, rules arrive via AGENTS.md project context; the config-only path uses a one-command MCP registration, while installing the package as a compatible Codex bundle loads its skills and bundle-scoped MCP server. Neither path provides ambient hooks (see OpenClaw in ARCHITECTURE).
• Impact/blast-radius analysis is heuristic — a regex-approximate code graph, not a sound call graph. It is not conservative: it can miss affected files as well as flag unaffected ones, so treat its output as advisory and an empty result as "unknown", not "safe".
• "Proof-carrying memory" is a name, not a formal proof. Claims are content-addressed and carry evidence references; confidence moves only when independent oracles (tests, CI, a human) raise it. There is no theorem-prover in the loop.
• Some integrations shell out — forge harden, forge scan, and the git-native ledger assume Bash and Git are available (jq is not required — the guards read hook JSON with node). Claude hooks on Windows do not require bash on PATH: their Node launcher finds Git Bash and preserves guard exits. protect-paths needs no bash at all and fails closed: when it cannot reach a verdict, the tool call is blocked rather than let through.
• Guardrails reduce risk; they are not a security sandbox. What each integration actually does — config emission, MCP registration, automatic hooks, blocking — is in docs/INTEGRATIONS.md.

60-second quickstart

npm install -g @codewithjuber/forgekit   # or: npm install -g github:CodeWithJuber/forgekit
forge init                               # emit native config for the tools this repo uses
forge doctor                             # verify providers, hooks, and MCP wiring
forge demo                               # watch the gate catch a real defect (green → broken → green)

Then run a pre-action check inside a repository:

forge substrate "Change verifyToken in src/auth.js to require length > 20; update tests"

The result includes an assumption verdict, a model-tier recommendation, predicted impact, a context-coverage check (what was delivered, not a proof that it suffices), scope clusters, and a verification checklist. On Claude Code, the pre-action check can run automatically through a UserPromptSubmit hook. On other supported tools, Forgekit emits instructions and MCP configuration that the tool can invoke.

The three jobs, one command each:

forge ledger query "token validation"   # 1. what does the team already know about this area?
forge impact verifyToken                 # 2. which files will this edit probably touch?
forge verify                             # 3. did the change pass the suites that actually ran?

See it in action

Real terminal recordings of the actual CLI — nothing mocked:

forge verify on a change that imports a package that doesn't exist: it flags the hallucinated dependency (frobnicator — not in the codebase) and blocks the push.

A fake secret staged with git add: forge precommit spots the credential-like line and refuses the commit before it ever happens.

Cross-session memory: forge recall add saves a durable fact (deploy-window), forge recall list reads it back — memory that survives across agent sessions.

Contents

• 60-second quickstart
• See it in action
• Why Forgekit exists
• How the loop works
• Core capabilities
• Agent orchestration
• Advisor pairing
• Setup details
• Commands
• Team memory
• LLMs, retrieval, and embeddings
• Agent roles and MCP tools
• Honest limits
• Portfolio evidence
• Maturity boundary
• Measured evidence
• Structural comparison
• Python research prototypes
• White paper
• Public site
• Documentation
• Your next step
• Community and support

Why Forgekit exists

The pain it kills — and the mechanism that kills it:

• You fixed this bug last month. The agent just re-introduced it. Lessons die when a session compacts. Forgekit stores them in the repository as evidence-referenced memory: only tests, CI, or a human raise a claim's confidence, and wrong lessons decay out instead of ossifying. forge ledger query "token validation" finds what the team already knows — teammates get it by pulling. (Memory persists outside the model; it doesn't guarantee the agent reads it — retrieval is budget-bounded, and missing context becomes a question, not invented context.)
• The "safe" rename broke twelve files you never opened. forge impact <symbol> shows the predicted blast radius before you edit, including coupled files you never named. It is heuristic — a regex-derived graph, not a sound call graph — so treat the output as advisory and an empty result as "unknown", not "safe".
• "Done" with no test evidence. A completion gate blocks a session's "done" when code moved without test evidence or a doc/state artifact — and hands back the repair checklist (forge docs sync sweeps the diff for stale prose, forge handoff writes the bounded session snapshot, forge decide records choices so no session re-decides them). On Claude Code this runs automatically.
• Rules that die at context compaction. Guardrails — protected paths, cost budget, doom-loop detection, secret redaction — run as deterministic hooks, not prose in a prompt. They survive compaction the way CLAUDE.md does not. (Ambient hooks are the Claude Code integration; other tools get config and instructions. The substrate is advisory by default; set FORGE_ENFORCE=1 to block only its strongest signals.)
• One rulebook, eleven tools. Author your rules once; Forge emits each tool's native config, plus MCP for Roo and VS Code. Zero runtime dependencies — one Node CLI, plain files in git, no server to run.

How the loop works

Forgekit runs a deterministic substrate before work, lets the external coding agent act, and records evidence from tests, CI, or explicit human correction afterwards.

_{Open the interactive diagram: pan, zoom, search, and trace any node.}

Only independent oracles (tests, CI, a human accept/revert) move a memory's confidence — so a wrong lesson decays out instead of ossifying. Full design: ARCHITECTURE.md.

What you get

The day-to-day value first — what the substrate keeps outside the model, so it survives between sessions and tools:

• Memory that persists across sessions and teammates. [Implemented] Every lesson, fact, and verified reuse is proof-carrying memory (PCM) — our name for evidence-referenced, content-addressed memory: a claim that carries references to its own evidence and is only trusted once independent oracles raise its confidence above a floor (the "proof" is that evidence trail, not a formal proof). Wrong lessons decay out instead of ossifying.
• Heuristic impact before you break things. [Heuristic] Ask "what does changing verifyToken break?" and get the blast radius — the set of files an edit is predicted to impact, read from a regex-approximate code graph (not sound, and it can miss affected files), including coupled files you never named.
• Guardrails that can't be forgotten. [Implemented on Claude Code] Deterministic hooks check the rules a model shouldn't break (protected paths, cost budget, doom loops) — they survive a context compaction the way CLAUDE.md prose does not. They reduce risk as defence in depth; they are not a sandbox, and a sufficiently creative shell command can still bypass a regex guard.
• Work that finishes end to end. A completion gate blocks "done" once per session when code moved without test evidence or a doc/state artifact (a UI-only change — styling, class strings, JSX text — owes a design record or a UI check instead of a unit test, and in a shared checkout a file another agent's session trail claims is named, not weighed; anything no trail accounts for still counts) — with the repair checklist as the answer (forge docs sync sweeps the diff for stale prose, forge handoff writes the bounded session snapshot the next session resumes from, forge decide records choices so no session re-decides them).
• One config for 11 tools. Author your rules once; Forge emits each tool's native config, plus MCP for Roo and VS Code. Zero runtime dependencies — one Node CLI, plain files in git, no server.

The measured evidence

Every number is a median from npm run bench on this repo, recorded with its environment block in reports/benchmarks.md — the project rule is a number is an assumption until measured.

• Blast radius in 1.68 ms (warm code-graph). On 6 hand-labeled cases from this repo's real import graph, recall is 1.00 against 0.26 for looking at the edited file alone, and precision is 0.17 — impact walks reverse dependencies transitively by default, so it returns everything downstream while the labels name only the direct referencers (restricted to one hop the same cases return their labeled sets). The precision 0.90 this line used to quote does not reproduce. On nine real Python repositories the research prototype's impact oracle reached recall 0.022 (refutation).
• A full pre-action gate in 851 ms (median on this repo, warm, on a 4-core Linux VM — this row is machine-bound; see the environment block) — assumption check, routing, reuse lookup, context assembly, blast radius, scope, and goal anchor in one deterministic pass, no LLM call. On Claude Code it runs on every prompt, automatically.
• The white paper's 62.1% routing saving is refuted. It was measured on the 30 tasks the prototype's thresholds were tuned on. On 80 held-out tasks from real issues, counting every escalation, the pipeline spent 20.2% more than always using the premium tier; per output a judge accepted it cost $1.06 against $1.76, but only 6 and 3 of 64 outputs were accepted (refutation). forge cost --stages reports only your measured stages.
• Conflict-free team memory — merging two 500-claim ledger replicas takes 191 ms on that same VM (I/O-bound, so it varies with the disk); the merge is order-independent and property-tested, so teammate ledgers converge to the same state no matter who syncs first, over plain git. The substrate is advisory by default. Set FORGE_ENFORCE=1 to block only its strongest signals: a vacuous task, required context that cannot be assembled, or a large impact set from a fresh repository graph.

Core capabilities

• Evidence-weighted memory. Claims are content-addressed, keep provenance and outcome references, and earn or lose confidence through a closed set of oracles. Time decay moves unreviewed knowledge toward uncertainty rather than silently treating it as permanent fact.
• Git-native team merge. Claims and append-only logs merge by set union. The join is property-tested for commutativity, associativity, and idempotence.
• Heuristic impact prediction. Forgekit builds a regex-derived code graph and walks reverse dependencies to estimate affected files and tests; the pre-action check and the Stop gate's repair checklist also walk the empirical refutation's sibling and forward relations and tag each file with the relation that reached it. It is not conservative: it can miss affected files (including constructs its parser does not recognize) as well as produce false positives, and the sibling/forward files are lower-precision co-change candidates.
• Budgeted context assembly. Definitions, direct dependants, sibling tests, and trusted lessons are selected under a token budget. Missing required context becomes a question rather than invented context. Coverage is syntactic — delivered, not proven sufficient — token counts are estimates, a file that only fits as a "read this" pointer stays a pending read, and a definition counts as delivered only when its whole body is.
• Model-tier recommendation. A deterministic rubric combines task text and repository signals. An optional LLM proposal can only lower the tier, confidence-gated and bounded; a vote for a higher tier is never applied automatically — it is recorded as an advisory escalateTo recommendation, which names the tier only once an external check has actually failed (the doom-loop diagnosis at its thrash threshold). Forgekit advises which tier to request; it does not itself proxy or fail over model traffic. The opt-in cross-provider router (route universal) minimises expected routing cost for a success target; its budgets are expectations, not spend caps, and its benchmark headline is repository-reported. Its shipped prior refits exactly from pinned public data in this repository, and a new in-repo held-out replay finds it no better than a fixed cascade chosen on the same dev tasks (docs/UNIVERSAL_ROUTING.md). For agents nobody watches, --mode unattended adds a confidence-gated vote raise, a writes-code floor, a risk floor and a gate on the top tier (Agent orchestration).
• Proof-gated reuse. Cached code is served only after evidence clears a confidence floor and declared dependencies still resolve in the current repository graph.
• Lifecycle guardrails. Claude Code hooks cover prompt preflight, protected paths, cost budget, repeated failures, format-on-edit, secret redaction, completion checks, and session learning. These controls reduce risk; they do not create a secure execution boundary.
• Verification, partial or full. forge verify runs the repository's detected test suites, reports PASS, FAIL, INCOMPLETE, or NOT_CONFIGURED, checks unknown symbols, and binds provenance to the code state it ran on. A verdict covers only the suites that ran: partial coverage is INCOMPLETE, not PASS. --deep adds structural, security, spec-drift, impact, and optional model-review lenses.

Agent orchestration

A lead agent running several sub-agents mostly pays for context re-reads, not new work. In one measured session, 7 parallel agents on one shared e2e environment used ~857M tokens in ~4.5 h, ~98% of them cache re-reads: agents polled logs every ~10 minutes against a 250–300k-token context, more agents ran than the environment could serve, and the top model tier was hand-picked for routine work (~140M tokens).

forge sync now emits an orchestration rule pack into every tool's config: no polling, a parallel-agent cap (default 2), heavy suites in CI, fresh small sub-agents with short reports, model choice via forge route, tests plus lead review for money/auth/secrets/migrations/security changes, isolated e2e environments, and a lean lead. Each rule has a stable id (orch.no-polling, …) and can be switched per repo or globally:

forge orchestration                        # rules, routing settings, and where each value came from
forge orchestration off orch.parallel-cap  # this repo; add --global for every repo
forge orchestration set route.mode unattended
forge route "add a refund endpoint" --mode unattended --json   # tier key + policy steps

In unattended mode, forge route raises the tier on a confident proposer vote, starts code-writing tasks at mid, keeps risky tasks on the premium tier, and allows the top tier only with --allow-top and a top deterministic score. The default conservative mode is unchanged. Details, defaults and the evidence: docs/ORCHESTRATION.md.

Advisor pairing

Claude Code's advisor tool lets the main model consult a second, typically stronger model at decision points: before committing to an approach, when an error keeps recurring, and before declaring a task done. The advisor reads the whole conversation and is billed at its own rate per call, and Claude Code accepts it only when it ranks at or above the session's main model. forge advisor configures it once, per repo or globally, emits it into Claude Code's settings, and checks the pairing before Claude Code has to refuse it:

forge advisor set opus                     # this repo: .forge/forge.config.json → .claude/settings.json + an Advisor section in AGENTS.md
forge advisor set fable --global           # every repo: the user-level config → ~/.claude/settings.json (what /advisor saves to)
forge advisor check sonnet opus            # ok — exit 1 when the pairing would be refused (for scripts and CI)
forge advisor pairings claude-opus-4-8     # which advisors this main model accepts, and the suggested one
forge advisor off                          # remove advisorModel again

The Advisor section tells every agent that reads AGENTS.md when to consult it (approach, recurring error, done check, risky code), when not to (short, well-specified tasks), and that the evidence wins over its guidance; each rule has a stable id (advisor.before-approach, …) and can be switched with forge advisor rule off <id>. The prompt-time advisory adds one line at a decision point, the doom-loop advisory names the advisor when a failure recurs, and forge doctor reports the pairing and the environment variables that keep the advisor off (CLAUDE_CODE_DISABLE_ADVISOR_TOOL, DISABLE_TELEMETRY, CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC). The pairing table ships as data (source/advisor.json, with the date it was verified against the docs); a model newer than it is accepted with a note, since Claude Code validates the pairing at launch.

Setup details

Install using one path:

Use case	Command
Claude Code 2.1.275 or later, one step	/plugin install forgekit --marketplace CodeWithJuber/forgekit
Claude Code or another plugin-capable supported host	/plugin marketplace add CodeWithJuber/forgekit then /plugin install forgekit
Global CLI from npm	npm install -g @codewithjuber/forgekit
Directly from GitHub	npm install -g github:CodeWithJuber/forgekit
Contributor checkout	git clone https://github.com/CodeWithJuber/forgekit.git && cd forgekit && npm link

The plugin runs its guards and its forge-cortex MCP server from the plugin directory, so it needs only node on PATH. It does not put the forge command on your shell's PATH; to run the forge commands below yourself, also install the npm package.

Initialize inside a project:

forge init
forge doctor
forge doctor --fix

forge init can merge Forgekit hooks and permissions into ~/.claude/settings.json. That file is global and affects all repositories. Use forge init --no-settings to skip the merge or forge init --remove-settings to reverse Forgekit-managed entries. The implementation preserves unrelated entries and creates a timestamped backup before changing the file. When the Forgekit Claude Code plugin is enabled (in user, project or local settings), its hooks/hooks.json already runs every guard, so forge init and forge doctor --fix merge permissions only and never register the guards a second time.

forge init emits configuration only for the agent tools the repository already uses: Claude Code plus any tool with a sign on disk (.cursor/, .codex/, .github/copilot-instructions.md, and so on). Choose explicitly with forge init --tools claude,cursor or --tools all; the choice is recorded in .forge/forge.config.json, so later forge sync runs emit the same set. In AGENTS.md, Forgekit owns only the block between <!-- forge:begin --> and <!-- forge:end -->. A hand-written AGENTS.md keeps every line and gets the block appended; forge sync and the Stop-hook auto-sync rewrite that block and nothing else.

For an explicit model provider, inspect or update configuration with forge config. API keys remain environment variables; Forgekit's provider file stores the environment-variable name, not the secret value.

Commands

Commands are advisory unless their documented enforcement flag is enabled. Full worked examples and output live in docs/GUIDE.md.

Group	Command	Does
Core	forge init	scaffold this repo's config — emits the tools it uses (Claude + detected, or --tools) from one shared source
	forge sync	recompile the canonical source into each tool's native config files
	forge doctor	health-check installed tools, guards, MCP auth, and config drift
	forge tools	primary-tool config — gitignore secondary-tool artifacts (.cursor/.gemini/…) for tools this repo doesn't use (forge tools <name> sets it, --reset clears)
	forge catalog	Start Here — list every tool, crew, and guard with a one-line why
	forge docs	docs↔code drift — check (registry reconcile) / render (regenerate machine-owned tables + diagrams) / sync (diff-driven stale-docs sweep) / impact (reusable doc-reference graph: which docs mention what THIS diff changed)
	forge update	self-update — --check reports if a newer version is available, bare applies it, --to <version> pins/downgrades
	forge config	provider setup — show / switch / add providers, set default model
Substrate	forge substrate	one pre-action gate: assumptions, route, impact, scope, memory, verify
	forge preflight	assumption check — what a task names that the repo doesn't define
	forge impact	hazard-aware blast radius — SCC-aware propagation + data-driven threshold from PageRank centrality and ledger incident history
	forge scope	decompose files into independent clusters (+ coupled files you didn't name)
	forge context	budgeted context assembly + completeness gate — what an edit NEEDS known, delivered or owed
	forge route	recommend the cheapest capable model for a task (+ gateway config); route universal: any provider's models, lowest expected cost for the success asked for, learned from outcomes
	forge orchestration	agent-orchestration rule pack (no polling, parallel cap, CI for heavy checks, fresh sub-agents, model routing, risk review, isolated envs, lean lead) and the unattended routing policy — list, switch rules on/off, set routing mode/gates, per repo or --global
	forge advisor	Claude Code's advisor tool — a stronger second model consulted at decision points: set the advisor per repo (emitted into .claude/settings.json by sync, with an Advisor rule section in AGENTS.md) or --global, check a main/advisor pairing before Claude Code refuses it, list accepted pairings
	forge verify	independent verification gate — tests + hallucinated-symbol + provenance (--deep: multi-lens consensus)
	forge demo	scripted scenario — a throwaway repo goes green → broken → green while forge verify watches (the README hero, runnable)
	forge precommit	commit-level gate — staged code w/o docs + secret scan (FORGE_COMMIT_GATE=block
	forge test-select	smart test selection — the tier contract as a command: which tests this change owes (selection only, never runs them)
Memory	forge cortex	self-correcting project memory — status / why
	forge recall	manage cross-session memory (list / add / consolidate)
	forge remember	add a durable fact to this repo's portable memory (forge brain)
	forge brain	show / rebuild the portable project memory index
	forge ledger	evidence-referenced memory — stats / verify / show / blame / query / compact / at / diff / root / ratify / retract / merge / sync / import
	forge claim	claim interop — export the ledger's claims as open JSON any tool can read (step 1 toward claims as a standard)
	forge handoff	bounded session snapshot — rewrite .forge/state.md, re-injected each session start
	forge decide	append-only decision log — D-#### ADR-lite entries in .forge/decisions.md
	forge know	route any fact to its storage home (decision / ledger / recall / …) — total, never dropped
Quality	forge scan	vet a skill/MCP for injection/RCE/exfil before install (skill-gate)
	forge spec	spec-as-contract — init (OpenSpec) / lock / check drift
	forge harden	wire security controls — pre-commit gate (gitleaks + commit gate) + sandbox settings
	forge radar	dependency-currency rings — staleness/major-lag/advisories from live registry evidence, cached 24h
Config	forge brand	print the active brand token map
	forge atlas	build / query the code-graph (where-is-Y, has-symbol)
	forge stack	detect this repo's real stack (languages, frameworks, test commands) from its manifests
	forge integrations	opt-in third-party MCP servers (e.g. context7) — add records the managed set and writes only with --yes (--adopt claims a same-name entry); remove reverses it
	forge cost	real per-day spend via ccusage + measured stage factors (--stages)
	forge budget	cost governance — per-day and per-task spend budgets, alert threshold, and an opt-in circuit breaker the cost guard enforces
	forge models	each tier's model family resolved to a concrete model — newest in the provider's live catalog (else the shipped snapshot), with its price and where both came from
Labs (experimental)	forge taste	enable one UI-taste tool for this repo (no arg = list)
	forge uicheck	deterministic UI checks — contrast · fingerprint <file...> · design <file...> · visual
	forge imagine	consequence simulation — predicted breaks + the minimal dry-run test suite for a task
	forge lean	scope-minimality (M5) — measure the diff's footprint vs what the task asked for
	forge anchor	goal-drift check — are your actual (git) changes still on the stated goal?
	forge diagnose	doom-loop check — record a failure; 3× the same signature mints a diagnosis + escalation
	forge dash	live dashboard: ledger, metrics trends, radar, memory browser, timeline, blast radius
	forge report	emit a static, self-contained HTML snapshot of .forge/ — opens offline, no server
	forge deja	anti-repetition — have you done this task before? ranks prior solved/verified sessions
	forge reuse	proof-carrying code cache — query / mint --file / stats
	forge rank	load-bearing code — PageRank centrality × past-incident history, circular-dependency clusters, chokepoint files
	forge collide	parallel-session conflict radar — who else recently touched the files (or their import neighbors) you are editing

Terminal output is plain when piped. On an interactive terminal it can use color and confidence meters; NO_COLOR disables color and FORCE_COLOR=1 enables it explicitly.

Team memory

Lessons, durable facts, and verified reuse artifacts land as content-addressed claims under .forge/ledger/:

forge init

# Work normally; hooks and explicit memory commands add claims and evidence.
git pull
forge ledger merge <path-to-another-ledger>
forge ledger sync

Identical claim content converges to one identifier while provenance records preserve authors. forge ledger blame <id> shows the mint history, oracle outcomes, and derived trust. The default storage is files in git, not a hosted database or synchronization service.

With no flags, forge ledger sync serializes ledger state under refs/forge/ledger on the repository's git remote. A non-fast-forward race triggers a re-merge and bounded retry. A shared directory can be selected with --dir <path> or FORGE_SYNC_DIR; --personal includes the per-user ledger.

LLMs, retrieval, and embeddings

Direct LLM adapters

src/llm.js supports two wire formats:

• Anthropic Messages API;
• OpenAI-compatible chat completions, used for OpenAI, Gemini, OpenRouter, and LiteLLM.

Credentials are read from environment variables and are not put in command-line arguments. The implementation performs a bounded single-prompt request. It does not implement streaming, conversation persistence, or a model-driven tool-call execution loop.

Repository-local retrieval

Forgekit retrieves from its code graph and evidence ledger, then assembles a context bundle for an external coding agent. This is retrieval and augmentation for source-code work, not a general enterprise RAG pipeline. There is no document connector layer, chunking service, citation generator, managed index, or vector database in this repository.

Optional embeddings

MinHash remains the zero-dependency default. To opt into semantic similarity, configure an external provider with one of the formats the current implementation accepts:

# OpenAI-compatible embedding endpoint
export FORGE_EMBED="https://api.example.com/v1/embeddings"
export FORGE_EMBED_MODEL="your-embedding-model"
export FORGE_EMBED_KEY="your-provider-key"

# Or a local/external command that implements Forgekit's stdin/stdout vector protocol
export FORGE_EMBED="cmd:./my-embedding-provider"

Vectors are cached in .forge/embed-cache.jsonl. Provider failure, timeout, malformed output, or missing vectors falls back to MinHash. The repository test uses a deterministic fake provider; it verifies adapter and fallback behavior, not live performance of a hosted model.

The performance snapshot in reports/benchmarks.md was generated at commit eb68ea9 and does not measure the optional embedding path. Current embedding behavior is defined by src/embed.js and test/embed.test.js.

Agent roles and MCP tools

Forgekit registers five Claude Code role definitions:

• scout — read-only repository investigation;
• verifier — fresh-context correctness review;
• independent-reviewer — diff/spec/test-only merge gate;
• frontend-verifier — visual and accessibility review;
• doc-sync — documentation consistency after code changes.

These are concrete role prompts supplied to the host tool. Forgekit does not contain a runtime that spawns these roles, exchanges inter-agent messages, or executes a LangGraph-style state machine.

The built-in MCP server exposes 21 tools, including:

• pre-action checks: substrate_check, preflight_check, assumption_gate, route_task;
• repository analysis: predict_impact, scope_files, rank_code, collide_check;
• memory: cortex_lessons, forge_remember, forge_ledger_query, forge_ledger_ratify, forge_ledger_retract;
• operations: forge_doctor, forge_provider_status, forge_cost, dashboard data.

The MCP server is the tool-provider side of function calling. The compatible host remains responsible for deciding when to call a tool and feeding the result back to its model.

Honest limits

• Beta software. The CLI is released, CI-tested and semantically versioned (a breaking change is a major version); "beta" is about maturity and evidence, not about interface churn. Support is maintainer-led and best-effort; there is no SLA.
• Claude Code is the deepest-tested integration. Other emitters and MCP configuration are implemented, but the repository does not provide equivalent real-world exercise evidence for every supported host.
• The code graph is heuristic. Regex extraction is not a sound call graph and can both over-predict and miss dependencies.
• Routing is advisory. Forgekit recommends a tier. LiteLLM or another gateway must be run separately to move traffic, handle failover, enforce quotas, or manage credentials. Routing costs are expected values, not caps, and no cost saving has been measured end to end: the ~90 % figure in the plans is a target (a hypothesis), and the white paper's 62.1 % routing saving was refuted on held-out tasks.
• Memory is external state, not model training. Claims live in files and are retrieved into context; Forgekit does not update model weights. It supplies persistence and external checks that a model does not guarantee on its own — one tested way to do that, not the only one.
• Embeddings are an adapter, not an included model or vector store. A user supplies the external command or HTTP service. MinHash remains the default and fallback.
• Guardrails are defence in depth. Shell-regex checks can be bypassed and post-tool redaction runs after the external tool has executed. Forgekit is not a sandbox.
• Human ratification is a workflow convention. Ratify and retract operations record the git author, but this repository does not authenticate a human or enforce enterprise approval roles.
• Evaluations are deliberately scoped. The JavaScript impact set has six hand-labelled cases; the Python router set has 30 hand-labelled tasks tuned to its rubric; the Python impact study has five mutations. None is a production field study.
• No enterprise platform claim. There is no Azure OpenAI/AI Foundry integration, vector DB, business-system/RPA connector layer, multi-tenant service, Kubernetes deployment, IaC stack, or public production-usage case study in this repository.

Portfolio evidence

The project is best read as agent reliability and developer tooling. It is not presented as an enterprise multi-agent application, a general-purpose RAG platform, or an Azure AI deployment. Per-claim status (implemented, measured, refuted, hypothesis, partial, reported) is kept in docs/status/.

For reviewers evaluating hands-on Agentic AI or GenAI work, each claim below links to the implementation and its closest test or build proof. The evidence snapshot used for this table is default-branch commit 3d9be37.

Area	Implementation evidence	Test or delivery evidence	Evidence-safe claim
MCP tools	src/mcp_tools.js defines 21 tool schemas; src/cortex_mcp.js implements JSON-RPC initialize, tools/list, and tools/call handlers	test/mcp.test.js, test/cortex_mcp.test.js, recent CI runs	Implemented an MCP server that exposes memory, preflight, routing, impact, verification, and health operations to compatible clients
Agent memory	src/ledger.js implements content-addressed claims, an oracle taxonomy, time-decayed validity, ranked retrieval, and a semilattice merge; src/ledger_store.js adds persistence, hash verification, and quarantine; src/ledger_sync.js adds directory and git-ref sync	test/ledger.test.js, test/ledger_store.test.js, test/ledger_sync.test.js	Implemented durable, evidence-weighted, mergeable memory for coding-agent workflows
LLM integration	src/llm.js implements Anthropic Messages and OpenAI-compatible chat-completions calls; src/providers.js configures Anthropic, OpenRouter, LiteLLM, OpenAI, Gemini, and custom endpoints	test/llm.test.js, test/providers.test.js	Implemented direct, bounded single-prompt LLM adapters and provider configuration; this is not a streaming or autonomous tool-call client loop
Retrieval and embeddings	src/context.js assembles code definitions, dependants, tests, and trusted lessons under a token budget; src/embed.js supports an optional command or OpenAI-compatible embedding endpoint, cosine similarity, and a disk cache; src/reuse.js falls back to MinHash and gates reuse on evidence	test/context.test.js, test/embed.test.js, test/reuse.test.js	Implemented repository-local retrieval/context augmentation and an optional embedding adapter; no vector database or enterprise-document ingestion pipeline is claimed
Guardrails and verification	hooks/hooks.json wires lifecycle hooks; global/guards/protect-paths.sh and global/guards/secret-redact.sh add path and secret controls; src/skillgate.js, src/verify.js, and src/consensus.js implement scanning and multi-lens checks	test/secrets.test.js, test/skillgate.test.js, test/verify.test.js, test/consensus.test.js; Security workflow and CodeQL	Implemented deterministic defence-in-depth controls and evidence-producing verification; the regex guards are not a security sandbox
Human review affordances	src/ledger.js defines human accept/revert oracles; src/ledger_store.js implements ratify and retract records	test/ledger.test.js, test/ledger_store.test.js	Implemented auditable human correction and ratification paths; no identity-enforced RBAC or enterprise approval workflow is claimed
Agent roles	.claude-plugin/plugin.json registers scout, verifier, independent-reviewer, frontend-verifier, and doc-sync	test/channels.test.js checks plugin-channel wiring	Authored five concrete Claude Code role definitions; they are declarative roles, not a multi-agent orchestration runtime
Evaluation	src/eval.js calculates precision, recall, and F1; bench/bench.mjs provides a seeded benchmark harness; reports/benchmarks.md records methodology and limitations	test/eval.test.js, test/bench.test.js	Implemented reproducible evaluation for the repository's impact predictor and local performance; the datasets are small and are not field benchmarks
Python research	research/python-prototypes/router_gate/ implements assumption gating, model routing, execution, verification, escalation, CLI, and MCP; research/python-prototypes/impact_oracle/ implements Python AST parsing and a persistent NetworkX dependency graph	router_gate tests, router_gate live demonstration results, impact_oracle tests	Built working Python research prototypes; the shipped Forgekit runtime is Node and the Python packages are not presented as production services
Delivery engineering	package.json defines a Node 20+ CLI with no runtime dependencies; .github/workflows/release.yml gates releases and configures npm provenance; .github/workflows/smoke.yml exercises clean install and uninstall	latest release; successful audited runs for CI, Smoke, Security, CodeQL, and Scorecard	Demonstrates packaging, cross-platform CI, security checks, and repeatable OSS release engineering; it does not establish enterprise production operation

Maturity boundary

Evidence level	What belongs here
Implemented and tested in the Node runtime	CLI and config emitters; 21 MCP tools; agent memory and sync; code-context assembly; optional embedding adapter; LLM provider adapters; heuristic impact analysis; lifecycle guardrails; verification; benchmark harness; release automation
Research or integration demonstration	Five declarative Claude Code agent roles; Python router/gate and impact-oracle packages; a 30-task live routing demonstration; support for external embedding providers; configuration emitted for integrations other than the deeply tested Claude Code path
Not claimed by this repository	A collaborating multi-agent runtime; LangGraph, LangChain, Semantic Kernel, AutoGen, CrewAI, or Copilot Studio; Azure OpenAI or Azure AI Foundry; a vector database; enterprise-document RAG; business-system or RPA connectors; production Python deployment; multi-tenant cloud operation, SLA/SLO, Kubernetes, or infrastructure as code

Personal maintainer use is intentionally not used as proof of organizational adoption. No customer count, enterprise deployment, production traffic, or service-level claim is made without corresponding public evidence.

Measured evidence

Numbers below are reported only with their test boundary. See reports/benchmarks.md and the linked evaluation artifacts for full methodology.

Parser-stable snapshot labels used by the generated project pages are:

• A full pre-action gate in 851 ms median — deterministic, warm repository graph, LLM disabled;
• Blast radius in 1.68 ms median — warm impact query; and
• 20.2% more cost than always-premium — the held-out routing result. The 62.1% saving the white paper reported came from a 30-task demonstration with thresholds tuned on those same tasks; on 80 pre-registered held-out tasks the same router spent 20.2% more (table below). Per judged-correct output the pipeline cost $1.06 against always-premium's $1.76.

The boundaries in the table below are part of each result.

Measurement	Recorded result	Boundary
Warm impact query	1.68 ms median	30 runs on one JavaScript repository with a memoized adjacency index; not model latency
Deterministic substrate check	851 ms median	3 runs on one repository, warm graph, LLM disabled, on a 4-core Linux VM (Node v22.22.2, recorded 2026-09-26 in the environment block) — wall-clock rows are machine-bound; re-run npm run bench on your own hardware
Impact quality	precision 0.17, recall 1.00, F1 0.28 (the precision 0.90 / F1 0.92 reported before 2026-09-21 do not reproduce)	6 hand-labelled symbols in this repository (which imports only by relative path, so it does not exercise tsconfig path aliases), scored by evalImpact against labels re-derived by git grep; impact walks reverse dependencies transitively by default, so precision measures the transitive closure against direct-only labels; edited-file-only baseline recall 0.26
Ledger replica merge	191 ms median	3 runs merging two synthetic 500-claim replicas with 250 claims shared, on the same 4-core Linux VM (I/O-bound: varies with the disk more than the CPU-bound rows do)
Python router live demonstration	62.1% calculated cost reduction versus always-premium (repriced tokens; refuted by the next row)	30 hand-labelled tasks, thresholds tuned to the set, real measured LLM tokens, approximate public prices; demonstration, not field benchmark
Python router, held-out evaluation	total spend 20.2% higher than always-premium; gate F1 0.37; success = judge-accepted (6/64 vs 3/64), not test-verified	80 tasks from real GitHub issues and PRs, thresholds frozen, pre-registered; refutes the row above
Python impact oracle	precision 0.633, recall 1.000, F1 0.753	5 mutations in the bundled demo package; mutation-derived test failures as ground truth
Python impact oracle, real repositories	precision 0.398, recall 0.022, F1 0.042 (grep baseline F1 0.437)	759 files in 9 open-source repositories, co-change ground truth, pre-registered; refutes the row above

Recent audited CI runs cover Node 20, Node 22, and Windows Git Bash, plus the reusable quality gate (CI workflow). The quality gate runs the Node unit suite, Biome checks, TypeScript type checking, critical-level npm audit, ShellCheck, zero-runtime-dependency assertion, version and documentation checks, and npm pack --dry-run. The Python prototype pytest suites were not part of CI at that commit; since aedddf5 a separate CI job runs both prototype suites, the Theorem D sanity checks (research/recompute_corrections.py --theorem-checks) and the recomputation of every corrected number from the replication package.

Structural comparison

This table describes architecture, not a claim of superiority or equivalent product scope.

Concern	Forgekit implements	Boundary
Memory	Content-addressed claims, provenance, oracle-weighted validity, time decay, and git-native union merge	No hosted synchronization, managed database, enterprise tenancy, or RBAC
Retrieval	MinHash retrieval by default; optional external embeddings; proof-gated code reuse	No vector database or general enterprise corpus pipeline
Routing	A visible deterministic recommendation with optional bounded LLM input; LiteLLM alias config emission	Does not proxy traffic, manage quotas, perform failover, or hold provider keys
Tool use	A JSON-RPC MCP server with 21 executable tool handlers	Does not implement the model/client loop that chooses and executes tool calls autonomously
Agent roles	Five host-consumable Claude Code role definitions	No multi-agent graph, scheduler, inter-agent messaging layer, or named orchestration framework
Verification	Repository tests, provenance, structural and security lenses, limited benchmark suites	No managed GenAI evaluation service, production telemetry, online evaluation, or broad red-team certification

Python research prototypes

The Python packages under research/python-prototypes/ are the original research baselines preserved for auditability and result reproduction. They are useful evidence of Python implementation work, but they are not the shipped Forgekit runtime.

Router Gate

router_gate implements:

1. deterministic assumption assessment;
2. transparent cheap/mid/premium routing;
3. a pluggable executor;
4. caller-supplied verification;
5. bounded escalation after failure;
6. token and cost records;
7. CLI and MCP surfaces.

Its checked-in evaluation used live model calls for a 30-task hand-labelled demonstration. The set was used to tune the rubric, so perfect separation is evidence that the mechanism works on that set, not a general accuracy estimate. Despite legacy package metadata using the phrase “production-ready,” the repository-level maturity classification is research prototype.

Impact Oracle

impact_oracle parses Python ASTs, stores a NetworkX dependency graph as JSON, and predicts change impact through reverse-dependency traversal plus the repaired sibling and forward relations (the in-tree package is the repaired v2, 49 tests). The bundled evaluation mutates five symbols and uses pytest failures as independent ground truth. On nine real repositories the as-shipped traversal reached recall 0.022; the repaired version's held-out point estimate is above grep's, but that it beats grep is not established (refutation). The Node forge impact graph is a different, regex-derived implementation and inherits none of these numbers.

The production Forgekit equivalents are the Node implementations behind forge preflight, forge route, forge atlas, forge impact, and the MCP substrate tools.

White paper

The cognitive-substrate white paper explains the motivation, formal model, evidence map, adjacent work, and the relationship between the Python research prototypes and the current Node implementation. Treat the paper's measurements according to their stated methodology; do not blend results from different codebases or evaluation sets. The "faculty" and theorem language lives there and in research/, with dated corrections: the papers now state what a frozen model does not guarantee (durable state across calls, context beyond the window, weight updates from outcomes, reliable self-verification) rather than claiming it cannot adapt at all, and they name the prior art (CoALA, Reflexion) the architecture builds on. Their PDFs are historical, pre-correction editions; read the HTML.

Public site

Forgekit includes a hand-authored landing page at landing/index.html and generates a status page at public/index.html. The generator reads repository data from package.json, README.md, CHANGELOG.md, and the benchmark report.

npm run pages:build
BUILD_PAGES_LIVE=1 npm run pages:build

The default build is offline and deterministic. Optional live mode reads public GitHub repository metadata with bounded retries and caching. GitHub Pages deployment is defined in .github/workflows/static.yml; this static documentation site is not an AI application deployment.

Documentation

Document	Contents
ONBOARDING.md	Five-minute setup and design principles
docs/GUIDE.md	Full command reference, worked examples, MCP schemas, and honest limits
examples/	Runnable pain-point solution demos (cost governance, adversarial doctor) — each with copy-paste steps, a demo script, and a GIF
docs/INTEGRATIONS.md	Per-tool behaviour: config emission, MCP registration, automatic hooks, blocking — and how each is tested
docs/UNIVERSAL_ROUTING.md	The cross-provider router: model, shipped prior, evidence status and modeling limits
docs/ORCHESTRATION.md	Agent-orchestration rule pack, on/off controls, and routing for unattended agents
docs/status/	Machine-readable status of every load-bearing claim (generated table)
ARCHITECTURE.md	Runtime layers, emitters, state, and design decisions
reports/benchmarks.md	Reproducible benchmark snapshot, methodology, environment, and limitations
research/python-prototypes/README.md	Explicit maturity boundary for the Python research packages
docs/cognitive-substrate/	White paper, evidence map, ecosystem map, and prototype sources
docs/RELEASING.md	Tag, npm, provenance, and GitHub Release flow
CHANGELOG.md	Changes by release

Current implementation truth for optional embeddings is src/embed.js: use an http:// or https:// endpoint, or cmd:<command>. A bare FORGE_EMBED=1 is not a valid provider configuration. Benchmark comparison prose that describes MinHash-only behavior is historical and does not supersede the current source and tests.

Your next step

Sixty seconds, three commands, and your agent tooling gets a memory:

npm install -g @codewithjuber/forgekit
forge init      # emit native config for the tools this repo uses
forge doctor    # verify providers, hooks, and MCP wiring

Then try it on your own repo:

forge impact <symbol>     # what will this edit probably touch?
forge ledger query "..."  # what does the team already know about this area?
forge verify              # did the change pass the suites that actually ran?

• Found a gap or a wrong answer? Open an issue — the project's own rule is a number is an assumption until measured.
• Using it in your workflow? ⭐ Star the repo and tell us what it caught.

Community and support

• Get help: SUPPORT.md and Discussions
• Contribute: CONTRIBUTING.md and Code of Conduct
• Direction: ROADMAP.md and GOVERNANCE.md
• Security: SECURITY.md for private vulnerability reporting
• Accessibility: ACCESSIBILITY.md

---

MIT licensed. Built by CodeWithJuber.