True incremental indexing: clean-rebuild parity, 6x post-edit indexing speedup, explicit freshness, and graph-derived search ranking - #1245
Open
ahundt wants to merge 918 commits into
Open
Conversation
Add self_dogfood workload expansion in scripts/run-benchmark-campaign.py with exact repository revision/tree identities, candidate-scoped profiles, and result rejection on background mismatches. Make create_self_dogfood_worktree check out the declared commit, retain pre/post mutation SHA-256 values, and add c_new_leaf for a real indexed create-file route alongside high-fanout fallback scenarios. Document the JSON contract in docs/BENCHMARK_CAMPAIGN.md. Verify with 125 benchmark/report unit tests, Ruff, source-safety, and an optimized c_new_leaf smoke: incremental_exact 367 ms, fresh rebuild 4456 ms, canonical equality 1/1, cleanup complete. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
scripts/benchmark-incremental-speed.py previously wrote rank_refresh=stale_on_exact for every matrix cell, including profiles labeled default. That changed candidate behavior and invalidated comparisons against binaries with different compiled policies. Add RANK_REFRESH_CANDIDATE_DEFAULT and apply_rank_refresh_override() so default runs make no rank_refresh config write. Record the requested policy and whether an override was applied in synthetic, matrix, self-dogfood, and capability-quality reports; retain explicit eager, stale_on_exact, and stale_on_incremental experiments. Document the comparison contract in docs/BENCHMARK_CAMPAIGN.md and cover CLI default, no-write, and explicit-write behavior in tests/test_benchmark_incremental_speed.py. Verification: uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign (90 tests); uv run ruff check scripts/benchmark-incremental-speed.py tests/test_benchmark_incremental_speed.py; bash scripts/check-source-safety.sh; git diff --check. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Large c_new_leaf runs with the default stale_on_incremental policy passed every mutation oracle but differed from fresh rebuilds by 99 SEMANTICALLY_RELATED rows. summarize-benchmark-results.py labeled those disclosed stale views as REJECT: graph correctness, hiding the intended latency/freshness policy. benchmark-incremental-speed.py now unions structured stale_with_warning views and reruns canonical comparison after excluding only SEMANTICALLY_RELATED rows when semantic_edges is explicitly stale. The declared-stale gate passes only when every remaining node, edge, property, and file hash equals the fresh rebuild; non-semantic or undeclared differences still fail. summarize-benchmark-results.py reports Core graph separately from Full graph freshness and emits PASS: DECLARED STALE VIEWS for a verified deferred view. docs/BENCHMARK_CAMPAIGN.md documents the gate and tests cover stale-view collection, exact exclusion scope, core mismatches, decision text, and table categories. Verification: uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign tests.test_summarize_benchmark_results (133 tests); uv run ruff check on changed Python files; bash scripts/check-source-safety.sh; git diff --check. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
evaluate_pair_incremental_policy() treated every unconfigured binary as stale_on_incremental. The pinned upstream binary produced immediate similarity pair freshness without a stale warning, so the evaluator rejected correct observed behavior against a policy that upstream does not expose. Record candidate_default when incremental_derived_refresh is not explicitly configured. Classify the observed lifecycle as immediate_pair_freshness, immediate_full_freshness, deferred_with_warning, or unreported_stale; retain strict contract checks for explicit eager and deferred overrides. Separate pair freshness from whole-canonical-graph equality so each category remains visible. Document cross-version default handling in docs/BENCHMARK_CAMPAIGN.md and extend tests/test_benchmark_incremental_speed.py with deferred, immediate-pair, unreported-stale, and explicit-eager cases. Verification: uv run python -m unittest tests.test_benchmark_incremental_speed tests.test_benchmark_campaign tests.test_summarize_benchmark_results (133 tests); uv run ruff check on benchmark/report files; bash scripts/check-source-safety.sh; git diff --check. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Semantic quality cases store incremental-versus-fresh graph comparisons under pair_lifecycle.canonical_graph. correctness_findings() only read the case-level field, so rejected upstream semantic and similarity rows incorrectly reported that no stage-level witness was recorded even though retained results contained exact counts and first differences. Add canonical_mismatch_finding() in scripts/summarize-benchmark-results.py and use it for both case-level and pair-lifecycle comparisons. Preserve declared-stale-view context and place the intentional deferred-freshness explanation before its expected mismatch witness. Add test_pair_lifecycle_canonical_rejection_reports_exact_graph_witness() in tests/test_summarize_benchmark_results.py. Verification: 134 benchmark/campaign/report unittests passed; Ruff, scripts/check-source-safety.sh, and git diff --check passed. Audit-only regeneration completed 39/39 capability cells plus 9/9 semantic and 9/9 similarity cells with zero missing, corrupt, duplicate, or unplanned results. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
scan_campaign() validated completion and result hashes but trusted the artifact inventory in attempt.json. Changed bytes, a missing worker log, or an unlisted extra file therefore remained status=complete even though phase timing and lifecycle evidence no longer matched the recorded attempt. Add validate_attempt_artifacts() in scripts/run-benchmark-campaign.py. Completed attempts now require matching cell identity and status, then rebuild the recursive artifact inventory and compare every relative path, byte count, SHA-256, file count, and total byte count. Completions without an attempt field retain historical hand-authored-plan compatibility. Add changed, missing, and unlisted artifact regressions in tests/test_benchmark_campaign.py and document audit-only re-inventory in docs/BENCHMARK_CAMPAIGN.md. Verification: 135 benchmark/campaign/report unittests passed; Ruff, scripts/check-source-safety.sh, and git diff --check passed. Strict audit-only scans rehashed 39 capability cells, 9 semantic cells, and 9 similarity cells with zero corrupt, missing, duplicate, or unplanned results. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
mutation_reindex_details() interpreted candidate-default stale-warning metadata without considering an explicit similarity_enabled=false or semantic_edges_enabled=false override. The semantic-pair table correctly said capability disabled while the incremental mutation table contradicted it with deferred with warning. Pass disabled_pair_capabilities from summarize_group() into mutation_reindex_details() and give the explicit off state precedence over candidate freshness policy metadata. Extend test_disabled_semantic_pair_control_is_not_described_as_freshness_deferral() to cover the mutation table. Verification: 135 benchmark/campaign/report unittests passed; Ruff, scripts/check-source-safety.sh, and git diff --check passed. Audit-only regeneration preserved 3/3 semantic-off and 3/3 similarity-off large-corpus cells with zero corrupt, missing, duplicate, or unplanned results, and both reports now render capability disabled in the pair and mutation tables. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
scripts/benchmark-incremental-speed.py previously measured Git-history and HTTP-link runtime profiles without task-quality oracles, leaving enabled/disabled cost cells unable to demonstrate user-visible graph evidence. Add deterministic, network-free fixtures that reuse FILE_CHANGES_WITH and HTTP_CALLS from the production pipeline. The Git fixture records four fixed-date co-change commits. The HTTP fixture places a Python client and source-discovered Ktor route in distinct service namespaces so only the optional HTTP linker produces the direct caller-to-handler edge. Query existing relationship schemas with bounded results and require the declared paths, counterpart, count, caller, handler, and method/confidence evidence in the quality judgments. Register both cases in --capability-quality and cover fixture identity plus oracle queries in tests/test_benchmark_incremental_speed.py. Optimized binary SHA-256 2f2ceeb052f78bcdfc2b64b6175cf016315f3c3b9e28af4ec1487624e2523806 passed Git and HTTP enabled smokes; the HTTP disabled control missed the direct fetch_order-to-configureRouting edge as required. Every isolated work root was removed. Verification: 139 benchmark/campaign/report unittests passed; Ruff, scripts/check-source-safety.sh, git diff --check, and optimized MCP enabled/disabled smoke cases passed. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
inject_context_once used srv->session_project and srv->session_root even when search_graph resolved an explicit project in another store. That produced correct result rows with zero node counts, the wrong schema, and the session CWD ecosystem in the same response. Pass the resolved project through the JSON and TOON context paths, retain session_project as the server CWD identity, resolve ecosystem metadata from the registered target root, and suppress stale PageRank statistics and key functions. Free cbm_project_t fields after the one-shot metadata lookup. Add search_graph_explicit_project_context_uses_resolved_store_project in tests/test_tool_consolidation.c. Verified with ASan/UBSan suites: tool_consolidation 102/102, mcp 225/225, rust_lsp 506/506, registry 60/60, lang_contract 35/35; scripts/check-source-safety.sh passed. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
handle_search_graph returned completed BM25 JSON before the regular yyjson response builder. Requests with query plus format=json therefore omitted both session_project and the one-shot _context header, while graph-mode JSON and compact TOON included them. Add json_payload_with_context_once in src/mcp/mcp.c to copy the bounded BM25 object, invoke the shared inject_context_once path, and serialize it before the early return. Extend search_graph_explicit_project_context_uses_resolved_store_project with the exact BM25 JSON branch. TDD evidence: the new assertion failed with 101 existing consolidation tests passing because session_project was absent. Verified after the fix with ASan/UBSan: tool_consolidation 102/102 and mcp 225/225; scripts/check-source-safety.sh and git diff --check passed. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
A supervised watcher reindex could replace a project database while the request thread retained a read-only handle to the unlinked prior generation. The watcher also treated any non-null worker response as success, including status=error, and shutdown freed g_server before an in-flight watcher callback had joined. Pass the server through cbm_mcp_index_run_supervised_path, defer cached-store retirement through store_stale, classify indexed/degraded responses before committing watcher baselines, and join the watcher before freeing the server while keeping the watcher object alive for autoindex teardown. Normalize glob-compatible wildcards before POSIX regex compilation so valid-but-ambiguous inputs such as alpha?orker and alternations return the intended symbols across name_pattern, qn_pattern, and pattern. Preserve explicit group, class, escaped, and dot-regex constructs. Place validation fixtures under the isolated test cache, clean runner-owned caches after both passing and failing runs by default, and reuse CBM_TEST_ARTIFACT_DIR as the explicit failed-run retention path. Verification: CBM_ONLY_SUITE=input_validation build/c/test-runner (48 passed); CBM_ONLY_SUITE=mcp build/c/test-runner (227 passed); make -f Makefile.cbm cbm (O2 build and ad-hoc signature); MCP initialize plus watcher-active EOF shutdown exited 0; git clang-format reported no changed-line edits; git diff --check passed. The repository-wide lint-format target still reports pre-existing formatting debt outside this diff. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Incremental behavior previously lacked one table-driven oracle covering every supported parser/LSP language family. That left exact-upsert and scoped-LSP fallback routing vulnerable to language-specific drift even when individual fixtures passed. Add run_incremental_language_oracle_case in tests/test_pipeline.c. Each synthetic fixture performs a full fast index, edits source to add a callable symbol, runs incremental indexing, asserts the publication kind and fallback reason, and compares the resulting canonical graph with a fresh fast rebuild. Cover Go, C, C++, CUDA, and Python exact publication plus JavaScript, TypeScript, TSX, PHP, C#, Java, Kotlin, and Rust scoped_lsp_gap full fallback. The helper releases pipelines, configuration, and project storage through one cleanup path; ordinary passes and failures remove fixtures, while CBM_TEST_ARTIFACT_DIR retains failed fixtures inside the runner-owned cache. Verification: the ASan/UBSan test runner passed incremental_cross_lsp_language_matrix_matches_fresh_rebuild (13 cases) and eight adjacent exact, C-header fallback, Python/JavaScript scoped-LSP, delete, rename, overlay publish-failure, and overlay extraction-failure tests. git clang-format reported no changed-line edits and git diff --cached --check passed. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The pipeline had separate coverage for TypeScript re-export IMPORTS edges and direct cross-file calls, but no end-to-end guard proved that their composition resolves a caller through a barrel file to the implementation symbol. Add pipeline_typescript_barrel_reexport_call_resolves_implementation in tests/test_pipeline.c. Its three synthetic files use nested hyphenated directories, a multiline named import, an async awaited call, and an export-from barrel; cross_file_call_exists must find callerOperation -> targetOperation in the persisted database. Verification: the focused test passed with the ASan/UBSan runner and the current-source TSan runner. Current-source TSan also passed watcher unwatch, replacement, deferred-free, publication-generation, supervised-response, and pipeline-lock contention guards. git clang-format reported no changed-line edits and git diff --cached --check passed. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
tests/test_pipeline.c lacked persisted CALLS coverage for a Python entry point importing PyO3 functions and lacked incremental-versus-fresh oracles for mixed Python/Rust and Rust/TypeScript/JavaScript repositories. Add pipeline_python_pyo3_import_resolves_rust_function_calls plus two mixed-language incremental tests. The tests require Python edits to publish as CBM_PIPELINE_PUBLISH_INCREMENTAL_EXACT, require Rust and TypeScript scoped-LSP gaps to publish a full rebuild with reason scoped_lsp_gap, and compare each resulting graph with a fresh fast rebuild. Route setup_lang_repo and setup_incremental_repo through cbm_resolve_cache_dir, reject truncated paths, and remove partially created fixture trees on setup errors. Assertion-aborted fixtures now remain inside the test runner cache lifecycle and are retained only through CBM_TEST_ARTIFACT_DIR. Verification: all three focused tests passed with the ASan/UBSan runner and the current-source TSan runner. Apple leaks reported 0 leaks for 0 total leaked bytes for each focused test. A real MCP 2025-11-25 trace_path request against a mixed Python/Rust repository returned cli_main -> _run_cli_command and cli_main -> serve_mcp with isError=false and clean EOF shutdown. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
summarize_group now derives a Pareto workload identity from scenario, report mode, index mode, corpus metadata, and task-set hashes. mark_pareto_frontier compares candidates only when that identity matches, preventing small semantic canaries from dominating full-repository measurements. Add test_pareto_does_not_compare_different_workloads in tests/test_summarize_benchmark_results.py. Verified with 140 benchmark campaign/measurement/report tests, ruff check, and regenerated retained composition output. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
search_graph previously ran an unfiltered graph search before a JSON semantic-only lookup, so an empty vector result returned unrelated nodes and paid an unnecessary graph-result-page scan. Detect semantic-only requests, skip cbm_store_search and cbm_store_search_overlay_view, and return explicit empty semantic_results with recovery guidance. Define query_graph as an early compositional tool for effective, computationally efficient custom Cypher. Mark examples, labels, properties, and LIMIT as optional guidance; retain server caps and dependency-ranking instructions; make truncation hints offer narrowing or cap changes without prescribing LIMIT. Install portable sandbox recovery guidance that distinguishes MCP approval from shell sandbox authorization. Verification: ASan/UBSan MCP 228 passed; tool_consolidation 103 passed; token_reduction 50 passed; CLI 150 passed; source-safety passed. The repository-wide clang-format target remains red on pre-existing files; changed-line formatting issues were corrected.
docs/CONFIGURATION.md:82 previously listed auto_index=false even though CBM_CONFIG_REGISTRY in src/cli/cli.c defaults the key to true. This could lead users to assume streamlined first-use indexing was disabled. tests/test_cli.c:3837 now derives the expected Markdown row from CBM_CONFIG_REGISTRY, so future default changes must keep the configuration table synchronized. Verification: CBM_ONLY_SUITE=cli build/c/test-runner (151 passed); bash scripts/check-source-safety.sh; git clang-format --diff HEAD -- tests/test_cli.c. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
cbm_parse previously returned success when parse_return_item stopped before postfix list indexing or an arbitrary trailing token. A query containing labels(n)[0] could therefore ignore its alias, aggregate, ordering, and limit and execute as an unbounded projection. Require TOK_EOF after parse_post_where in src/cypher/cypher.c and transfer the parent cursor after a recursively validated UNION tail. Route both list-index rejection paths through one diagnostic that gives supported labels(n) and n.label/count(*) rewrites. Add parser regressions in tests/test_cypher.c. Verified 155/155 Cypher tests under ASan/UBSan, all three UNION tests, source-safety, an optimized CLI rejection, and the supported five-row node-label aggregation. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
execute_single previously evaluated the complete WHERE tree while only the first MATCH alias was bound. A target-alias predicate such as NOT callee.file_path CONTAINS "tests/" therefore rejected every seed before CALLS expansion and returned zero rows. Add allocation-free three-valued partial evaluation in src/cypher/cypher.c: bound leaves prune early, unbound node or edge aliases remain unknown, and AND/OR/NOT/XOR preserve boolean semantics and safe short-circuiting. Register eval_expr_partial in src/foundation/recursion_whitelist.h. Add four relationship-target and mixed-alias regressions in tests/test_cypher.c. Verified 159/159 Cypher tests under ASan/UBSan, source-safety and whitelist checks, an -O2 production build, and the formerly empty hotspot query returning 20 bounded rows. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
…key registry
get_schema/get_schema_overlay_view previously returned relationship
patterns as preformatted "(A)-[T]->(B) [Nx]" strings via
schema_format_rel_pattern, forcing every consumer to re-parse text to get
at the source label, edge type, target label, or observed count
individually. Replace it with a structured cbm_schema_relationship_t
{source_label, edge_type, target_label, observed_count} array
(schema_collect_rel_patterns_from_stmt, store.h/store.c), and centralize
the previously-duplicated SCHEMA_MAX_JSON_KEYS/SCHEMA_REL_PATTERN_LIMIT
enum constants as CBM_STORE_SCHEMA_PROPERTY_KEY_LIMIT and
CBM_STORE_SCHEMA_RELATIONSHIP_PATTERN_LIMIT beside the existing schema
caps in store.h. Add const-correct cbm_store_schema_node_base_properties/
cbm_store_schema_edge_base_properties accessors so base-column lists have
one owner instead of being redeclared per caller.
Add a declared node/edge property-key registry
(schema_declared_node_property_keys/schema_declared_edge_property_keys,
~107 entries, sorted and duplicate-free) enumerating every JSON property
key any pipeline pass or git-context writer can emit on a node or edge,
exposed via cbm_store_schema_declared_node_property_keys/
cbm_store_schema_declared_edge_property_keys. Each row carries a trailing
comment naming its writer for provenance. The registry is a maintenance
contract, not a runtime dependency: nothing outside the new contract test
reads it. Document the four places that must stay in sync (writer,
store.c table, store.h accessor comment, contract test) in both store.h
and CLAUDE.md.
Add tests/test_schema_declared_property_keys.c: indexes mixed-language
fixtures (Python/TypeScript/Rust) plus a best-effort real git repo
(git init/commit via the portable cbm_git_drain_command, matching
test_git_context.c's tolerance for "git not available") to also
forward-verify the ~10 git_context registry rows, asserts every
discovered non-base property key is in the declared registry for its
entity kind, asserts both tables are sorted/duplicate-free, and reverse-
pins a sample of registry keys to catch dead rows. Cap-blindness guarded:
a label/type that hits the discovery limit fails loudly instead of
passing on unprovable completeness.
Register the new suite in Makefile.cbm (TEST_STORE_SRCS) and
tests/test_main.c.
Verification: make -f Makefile.cbm test — 6850 passed, 1 skipped, 0
failed, ASan/UBSan clean. make -f Makefile.cbm test-leak — 0 leaks for 0
total leaked bytes.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
…and key
query_graph and related tools previously had no config-controlled
default output encoding, and the TOON/JSON format enum lived nowhere
canonical, risking drift between the wire strings different call sites
might use. Add cbm_mcp_output_format_t {CBM_MCP_OUTPUT_TOON,
CBM_MCP_OUTPUT_JSON, CBM_MCP_OUTPUT_INVALID} plus the "toon"/"json" wire
string constants to mcp.h with a drift-prevention comment, as the single
source of truth for every response-format call site.
Add the default_response_format config key to CBM_CONFIG_REGISTRY in
cli.c (default "toon", validated to "toon" or "json" only via
cbm_config_set), so `codebase-memory-mcp config set default_response_format
json` persists a default and CBM_MCP_OUTPUT_INVALID.
Verification: make -f Makefile.cbm test — 6850 passed, 1 skipped, 0
failed, ASan/UBSan clean.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
…, live tools/list schema
Response-format consolidation: add cbm_mcp_response_format/
cbm_mcp_invalid_response_format so every response-generating handler
reads the format argument or default_response_format config through one
function instead of duplicating the enum-mapping logic, and migrate
query_graph's legacy-JSON detection to the canonical
cbm_mcp_output_format_t enum. Fix the regression this surfaced: TOON-path
_context/session_project delivery (toon_append_context_once) was
serializing the one-shot context object as a raw escaped JSON string
inside the TOON stream instead of native TOON fields, silently dropping
it for any TOON-consuming client. Add toon_append_context_model plus its
scalar/table helpers to map the same format-neutral context model
inject_context_once already builds into native TOON output.
Self-healing zero-row hints: query_graph_no_rows_hint reparses the
already-executed query on the zero-row path only, walks MATCH/OPTIONAL
MATCH patterns and EXISTS predicates in WHERE and post-WITH-WHERE
clauses (hint_walk_expr_exists_types/hint_walk_where_exists_types), and
probes each referenced label/edge type against the store via
cbm_store_schema_label_observed/cbm_store_schema_type_observed
(schema_probe_one_row: fail-open on any non-SQLITE_DONE outcome, so a
probe error never produces a false accusation). Unobserved names are
named in the response; a bounded hint_seen_t/hint_seen_add set (capped
at CBM_STORE_SCHEMA_HINT_VOCAB_LIMIT) dedups names repeated across
patterns/UNION branches so "Klass, Klass." can't happen and avoids
re-probing a name already resolved. The vocabulary summary and the
generic zero-row fallback no longer claim WHERE-clause predicates were
verified, since only labels/edge types are ever probed — only the
labels/edge types claim is made.
One invalidation authority: every graph-mutation path now calls
the single cbm_mcp_server_notify_index_published (flags-only,
any-thread-safe) instead of ad hoc local staleness bookkeeping —
sync auto-index, in-process index_repository, in-process
autoindex_thread, overlay compaction (gated on compacted>0),
index_dependencies. handle_delete_project's notify is gated on the
mutation actually succeeding (!is_error): a delete of a nonexistent
project is a no-op and must not stale the tools/list description cache
for nothing. The request-thread-only drain
(mcp_drain_tools_list_changed) is factored once and shared by both
transports (line-delimited and Content-Length-framed), gated so a
notification published before any tools/list has ever been served is
suppressed rather than sent for a client that never asked.
Shared overlay-aware schema-view selector (mcp_get_current_schema/
mcp_overlay_view_ready) ends the schema/overlay-readiness dance
previously triplicated across handle_get_graph_schema,
build_resource_schema, and the query path, so query_graph's advertised
vocabulary can never diverge from the view a query actually runs
against. The selector takes an mcp_cypher_vocabulary_t level instead of
a bool: MCP_CYPHER_MATCH_VOCABULARY (labels/edge types/observed
patterns with counts — enough to compose executable MATCH clauses;
never parses property JSON) vs MCP_CYPHER_FULL_QUERY_VOCABULARY (adds
per-label/type property keys for WHERE/RETURN authorship; O(total
property rows) json_each discovery, so only the docstring cache and
get_graph_schema request it).
Route the two remaining canonical-only cbm_store_get_schema callers
through the selector at MATCH_VOCABULARY level: inject_context_once
(first-response _context previously paid the full property-key scan for
counts it never rendered, and could advertise canonical labels a ready
overlay had tombstoned) and build_resource_architecture (rendered only
rel_patterns yet fetched full property keys, canonical-only). The
architecture resource's mixed-read-model disclosure moves
relationship_patterns into the overlay-aware list accordingly.
build_query_graph_tool_description resolves the project's real
store via resolve_store(srv, project) instead of trusting srv->store
as-is: on the very first tools/list of a fresh session for an
already-indexed project, srv->store was still the empty default
in-memory store (maybe_auto_index detects "already indexed" and skips
reindexing without ever opening it), so the schema section rendered
silently empty until some other tool call happened to resolve it first.
The project name is snapshotted into a local buffer before the
resolve_store call rather than passed as a live alias to
srv->current_project, because resolve_store can call
reap_stale_store internally, which frees srv->current_project when a
deferred invalidation is pending — aliasing it produced a
heap-use-after-free (caught by the ASan full-suite run, confirmed fixed
by rerun).
Docstring clarity for newcomers: the live schema section now embeds its
notation legend in the section header ("Labels name{extra property
keys}[count]:") so the compact schema is self-explanatory in every
tools/list response, and the query_graph/get_architecture/search_code
format parameters gain the brief description their three siblings
already had, all naming default_response_format.
Tests: response-format and TOON-context regressions in test_mcp.c/
test_tool_consolidation.c/test_input_validation.c/test_token_reduction.c;
zero-row hint coverage including no-hidden-tool-name, missed-graph shadow-
project probing, empty-project no-crash, TOON/JSON format parity,
UNION/repeated-name dedup, and the previously-untested EXISTS-predicate
walk; notification coverage for every notify call site including the
delete_project no-op negative direction and the "notify before first
tools/list is suppressed" negative direction; a cold-start regression
test seeding a real on-disk project store and calling tools/list as the
literal first request of a fresh session; overlay-consistency tests for
the two rerouted callers (first_response_context_uses_ready_overlay_schema,
resource_arch_rel_patterns_use_ready_overlay) proving _context and the
architecture resource report the active-overlay vocabulary, not
tombstoned canonical labels/types.
Docs: README.md documents the zero-row hint behavior and the
default_response_format config example; docs/CONFIGURATION.md adds the
default_response_format row to the important-keys table.
Verification: make -f Makefile.cbm test (full ASan/UBSan suite) run on
this exact tree after the final revisions. Earlier iterations of this
content additionally passed make -f Makefile.cbm test-leak (0 leaks for
0 total leaked bytes) and manual checks against the live MCP binary
(zero-row hint names real schema on two repositories; the query_graph
tools/list docstring shows the live schema on the very first request of
a fresh session); revisions since those checks are docstring text, the
vocabulary-enum rename, and the two selector reroutes, all covered by
the full-suite rerun.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previous behavior: - src/cypher/cypher.c parsed MATCH after WITH without executing it against projected bindings. - OPTIONAL MATCH predicates were applied after null extension, multi-key ORDER BY was not represented, and relationship expansion silently capped each seed at ten paths. Changes: - Add next_stage ownership and iterative execution/free traversal for MATCH and OPTIONAL MATCH after WITH, including UNION transfer and active-overlay checks. - Apply OPTIONAL predicates to candidate paths before null extension. - Represent up to 32 projected ORDER BY keys and use stable O(rows log rows) merge sorts with O(rows) pointer/index scratch. - Replace fixed-factor node and relationship buffers with ceiling-bounded geometric growth shared by canonical and active-overlay paths. - Keep classic and streamlined query_graph descriptions aligned with the implemented grammar. Tests: - make -f Makefile.cbm test: 6865 passed, 1 skipped under ASan/UBSan. - build/c/test-runner cypher: 519 passed. - build/c/test-runner mcp tool_consolidation: 588 passed. - scripts/check-source-safety.sh and scripts/check-nolint-whitelist.sh pass. Semantics checked against Neo4j Cypher documentation: https://neo4j.com/docs/cypher-manual/current/clauses/with/ https://neo4j.com/docs/cypher-manual/current/clauses/match/ https://neo4j.com/docs/cypher-manual/current/clauses/order-by/ Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previous behavior: - with_add_vbinding_var stored a WITH alias in cbm_node_t.qualified_name to share allocation ownership. - RETURN g.qualified_name after WITH g, COUNT(*) therefore returned the alias literal g instead of the indexed qualified name. Changes: - Track heap-owned variable aliases with binding_t.var_name_owned. - Copy and free aliases independently from cbm_node_t fields. - Leave aggregate node stubs eligible for the existing id-based indexed property fetch. Tests: - cypher_exec_with_node_groupvar_prop reproduces g versus test.ValidateOrder and now passes. - build/c/test-runner cypher: 519 passed. - make -f Makefile.cbm test: 6865 passed, 1 skipped under ASan/UBSan. - macOS leaks-at-exit exact regression: 0 leaks, 0 leaked bytes. - Optimized -O2 query_graph returns qualified names in both TOON and JSON. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
tests/test_tool_consolidation.c adds query_graph_description_uses_ready_overlay_view, which proves tools/list describes the ready overlay that query_graph executes instead of canonical rows replaced for the same file. tests/test_mcp.c adds mcp_published_schema_refreshes_description_once and extends mcp_delete_project_sends_list_changed. These protocol fixtures prove publication reopens the store, repeated publication emits one notifications/tools/list_changed event, and project deletion removes cached vocabulary from the next tools/list response. Verification: 590 MCP/consolidation tests; full ASan/UBSan suite (6867 passed, 1 skipped); TSan suite (449 passed) plus all three exact freshness cases; macOS leaks reports 0 leaks for 0 bytes for each exact case. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
tests/test_tool_consolidation.c:cross_project_search_not_confused_by_prefix now describes the path-derived project-name collision without embedding a local username, repository path, or private graph size. The executable myapp/myapp-other-project fixture and prefix-boundary assertions are unchanged. Verification: cross_project_search_not_confused_by_prefix passed (1 selected, 349 filtered); scripts/check-source-safety.sh passed; git diff --check passed; tracked-source scan found no athundt, justtalk, processtree-rs, or private feedback filenames. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Merge upstream/main 7d6cdb2 into api-consolidation 0fcb7e1 as a tested capability superset. Retain the branch's streamlined, classic, and hidden-reveal MCP surfaces; JSON/TOON output; read-only Cypher execution; semantic search; dependency indexing; PageRank; incremental indexing; and modular harness installer. Integrate upstream ObjectScript routine/UDL extraction, macro and return-type call resolution, Python alias and C++ header resolution, atomic SQLite publication, SCC/cycle reporting, Cypher deadlines, and bounded multi-source impact traversal. Fix merge-specific defects in handle_detect_changes project validation, cbm_pipeline_release_objectscript_tables ownership, no-sanitizer test defines, focused-suite registration, portable config publication, and installed streamlined guidance. The guidance keeps auto-index/get_code/query_graph discovery while adding coverage tiers and removes the stale fixed row ceiling. Verification: ASan/UBSan full runner 7,155 passed with 27 sandbox-loopback cases separately passing in the escalated HTTP suite (44 passed, 1 Windows-only skip); TSan 449 passed; incremental leak gate 514 passed with 0 leaks; MCP 261; CLI 232; extraction 313; tool consolidation 113; Cypher 179; SQLite writer 16; configuration editors 127; integration 28; language contracts 38; PageRank 60. scripts/check-source-safety.sh and git diff --cached --check pass. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Replace scripts/autotune.py's global config mutation, mutable local-repository lookup, architecture resource request, and source-tree result file with build_matrix_spec() over scripts/run-benchmark-campaign.py. Bind each plan to the full revision, binary SHA-256, compiler, flags, harness hash, candidate-default/rank-disabled controls, and PageRank parameter profiles. Store content-addressed manifests under the ignored durable campaign root and transfer execution with os.execv so the shared runner owns isolation, resumption, logs, and cleanup. Add tests/test_autotune.py for shared-runner plan validation, paired profile expansion, and absence of the removed resources/read, atexit, Path.home, and database-deletion paths. Verification: uv run python -W error::ResourceWarning -m unittest tests.test_autotune (3 tests); ruff check scripts/autotune.py tests/test_autotune.py; git diff --cached --check. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add run_mcp_surface_parity() coverage for classic discovery, streamlined discovery before _hidden_tools, and relisting in the same process after reveal. Compare complete tools/list contracts, validation-equivalent get_code aliases, capability outcomes, bounded hidden-handler recognition, and notifications/tools/list_changed without presenting dispatch recognition as behavioral proof. Make McpClient cleanup record subprocess return codes and require both reader threads and server processes to be reaped. Close SQLite snapshot and cloned-fixture connections explicitly so ResourceWarning-as-error runs detect ownership regressions. Render capability outcomes before tool counts in scripts/summarize-benchmark-results.py, retain response bytes and single-observation latency with explicit evidence limits, and document the parity mode in docs/BENCHMARK_CAMPAIGN.md. Verification: 141 warning-clean benchmark/campaign/summarizer tests; optimized build/c/codebase-memory-mcp parity smoke passed for 12 capability groups with full post-reveal classic contract parity and reaped lifecycle; ruff check; source-safety; git diff --cached --check. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previous behavior: self-dogfood graph validation inferred every stale derived view from task-specific MCP warnings. A changed-file query does not mention semantic_edges, so exact incremental runs with intentionally deferred semantic edges were rejected as canonical graph failures despite derived_view_state recording semantic_edges as stale. Add persisted_stale_views() in scripts/benchmark-incremental-speed.py to read the canonical SQLite freshness ledger through a read-only connection and union it with response-local warnings. Missing legacy derived_view_state tables remain compatible, while core node, edge, and file-hash mismatches are still rejected after excluding only declared SEMANTICALLY_RELATED rows. Add tests/test_benchmark_incremental_speed.py coverage for project-scoped stale/complete rows and retain tests/test_mcp.c coverage proving moderate incremental_exact publication persists semantic_edges=stale and query_graph reports it for semantic queries. Verification: 76 benchmark harness tests passed; 262 ASan/UBSan MCP tests passed; focused exact-moderate MCP test passed; git diff --check passed. The repository-wide lint-format target still reports pre-existing formatting violations outside these changes. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Centralize FileRenameInfoEx publication in compat_fs.c, retain long-path handling, and bound transient namespace retries without adding sleeps. Preserve successful publication when CloseHandle cannot roll it back. Release file-backed MCP stores and cancellation scopes on every JSON-RPC exit, key the query_graph schema cache to the opened database identity, and keep scanner paths project-relative across Windows Git and PowerShell modes. Validate YAML lock sidecars through an opened locked descriptor, gate ELF linker flags out of MinGW builds, normalize CRLF source reads, and make watcher/zombie fixtures fail at their actual setup boundary. Verified by make -f Makefile.cbm test (7823 passed, 2 platform skips), make -f Makefile.cbm test-tsan (1111 passed, 2 platform skips), make -f Makefile.cbm lint-ci, Clang analyzer, and the affected MinGW/Wine cohort (11 tests passed). Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
…erns The tracked patterns matched only the two literal names soak-results/ and soak-results-query-leak/, so hand-named run output such as soak30-linux/, soak-mac2/, and memlab-linux.log could still be committed by accident; that gap is how 86 run-log files reached upstream main (issue DeusData#1330, removed there by 66326c2). Take the identical seven-line hunk upstream main added in that fix so the next upstream merge resolves .gitignore without conflict. Root-anchored patterns leave scripts/soak-legs.sh and scripts/soak-test.sh tracked; git ls-files confirms no tracked file matches the new patterns. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Merge upstream/main at 08f6d1d into api-consolidation-merge. Retain the working-row-bounded cross_join_nodes executor and dynamic yyjson UNWIND parser. Add cbm_cypher_cross_join_alloc at src/cypher/cypher.c:5757 as an O(1)-time, O(1)-memory compatibility seam for upstream overflow tests; execution never allocates the full Cartesian product. Generate CLI tool help from the MCP registries, declare index_repository project/repo_path alternatives, reject removed manage_adr sections writes before mutation, and retain fork project resolution and mutation guards. Carry upstream extraction, Windows IPC, release-contract, workflow, and graph-UI i18n changes. Verification: full ASan/UBSan exit 0; focused cypher 205, extraction 343, MCP 332; TSan exit 0; lint-ci and lint-source-safety pass; Clang analyzer exit 0 with no warning on merge-added production lines; macOS leaks 0 bytes; graph-UI i18n 6/6. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Owner
|
Reviewed as far as a PR this size can be reviewed, and I want to be straight with you about both halves: the evidence quality here is well above what we normally see, and the PR cannot merge in this shape. Those are separate facts and I do not want the second to erase the first. What is genuinely impressive. A 39-cell benchmark matrix with an immutable audit manifest and a published SHA. Honest counter-metrics — you report that your own query latency is 3–4× worse than upstream's and that your full index is slower by default. And real parity testing: Why it cannot merge as-is. 896 commits, 395 files, and roughly nine distinct features — incremental rework, freshness surfacing, PageRank ranking, dependency indexing, a streamlined tool surface, Cypher extensions, git snapshot/history, HTTP-call linking, and a containerised Python/uv/Docker benchmark framework — plus flipped defaults and four shipped agent skills. Against a main that is moving fast, this is in a permanent merge-chase by construction: the last CI run was on a SHA that is no longer the head, the benchmark ran on a third SHA, and your "final verified head" is a fourth. Three things need saying plainly:
And one number needs a correction. The headline 3.2× is measured against upstream The direction problem is the real blocker, though. This PR decides several one-way doors unilaterally: a 73-key config registry with presets and persistent config writing, a new always-on What I would like to salvage, as atomic reproduce-first PRs. These are worth having regardless of how the direction call goes:
Each of those would land quickly on its own. Sent as one PR, they cannot land at all. I would rather have five merged PRs from you than one that stays open indefinitely — and if you would prefer, we can distill any of them with |
Replace staged variable-length expansion with one whole-pattern trail matcher using stable logical edge IDs and a reusable used-edge bitmap. Preserve exact and zero-hop bounds, unbounded read-only traversal, arbitrary relationship-type filters, active-overlay identity, directional specialization, and loud working-budget/deadline errors without a semantic depth cap. Add cbm_store_trail_graph_load/cbm_store_trail_graph_arcs as the shared typed graph-view path. Publish PageRank, LinkRank, node-degree rows, and completeness metadata in one savepoint; retain the prior generation on allocation, insertion, outer-transaction rollback, or non-convergence, with the named 100-iteration default shared by runtime and CLI. Launch copied daemon test images with posix_spawn on Darwin so the cumulative ASan parent is not copied before exec. Preserve the named watchdog and report spawn, wait, and terminating-signal failures through runtime_test_wait_image_probe. Verification: ASan/UBSan 7,861 passed and 2 skipped; TSan 1,116 passed and 2 platform skips with no race report; macOS leaks 0 bytes; lint-ci and source-safety passed; Clang analyzer exit 0 with no diagnostic on a changed line; cumulative copied-image reproducer 1,550 passed and 1 platform skip. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
runtime_application_worker now waits on a per-connection condition and processes serialized requests until disconnect, reducing application-thread create/join events from O(requests) to O(application-using connections). Cancellation recognizes pending and active tokens; disconnect cancels the session, signals stop, joins the worker, and only then closes session-owned storage. compat_thread adds an indefinite pthread/Win32 condition wait so the worker parks without polling or a timeout constant. Service startup and teardown track partial mutex/condition initialization. tests/test_daemon_runtime.c verifies two sequential tagged requests use one application worker. Verification: daemon_runtime ASan/UBSan 46/46; changed-path TSan cases pass with no race report; four application-worker leak probes each report 0 leaks for 0 bytes; source-safety, clang-format, and git diff --check pass. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Commit be69a80 added the foreign-SQLite schema guard but resolve_store_internal then reopened every valid cache for service. Return the read-only handle from open_validated_cbm_query_store so validation and request execution share one ownership lifecycle; close partial handles on every failure and retain the inventory/quarantine close path. Add the CBM_ENABLE_TEST_SEAMS query-open counter and resolve_store_validates_and_serves_with_one_query_open regression. Extend benchmarks/run_benchmark.py with opt-in post-index project probes so file-backed resolution is measured separately from pre-index dispatch. Verification: 107 benchmark unit tests; 6 focused MCP ASan/UBSan lifecycle tests; make -f Makefile.cbm lint-format; ruff format --check; git diff --check. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
run_self_dogfood_case previously returned without executing the opt-in indexed_query_probes path, repeating the parsed-but-unused overhead-probe defect. Resolve the indexed project immediately after the initial build and measure named project calls before mutation for both MCP and CLI transports. Reuse measure_mcp_overhead_probes and measure_cli_overhead_probes through measure_indexed_query_probes_for_transport; retain the trial summary in each case and the probe configuration in report parameters. Verification: RED missing-helper test; GREEN named-project MCP argument test; 108 benchmark unit tests; ruff format --check; git diff --check. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add a CBM_ENABLE_TEST_SEAMS-only CBM_TEST_SKIP_REQUEST_MEM_COLLECT switch in src/mcp/mcp.c. The switch bypasses only cbm_mem_collect() after release_request_store() has closed the file-backed SQLite handle and cleared server ownership, so benchmark variants retain publication and cleanup semantics. Expose a per-server collection counter through src/mcp/mcp_internal.h and add request_store_release_collection_can_be_isolated_for_measurement in tests/test_mcp.c. The test verifies the query result, released store, and zero collection calls under the ablation. Verification: RED undefined symbol cbm_mcp_server_request_mem_collect_count_for_testing; GREEN ASan/UBSan 1 passed, 7864 filtered; make -f Makefile.cbm lint-format; git diff --check. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Reuse one daemon-log window helper for run_index_mcp() and post-index MCP probes. Stream msg=mem.census at=mcp.request records into first/last/min/max/delta summaries for rss_kb, mi_area_kb, and mi_live_kb. The census scan costs O(B) time for B new log bytes and O(L + F) memory for the longest line and fixed field set; retained memory does not grow with probe count. Verification: RED KeyError daemon_mem_census; GREEN 109 tests and 25 subtests in tests/test_run_benchmark.py; ruff format check; git diff --check. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add CBM_TEST_RETAIN_REQUEST_STORE under CBM_ENABLE_TEST_SEAMS in src/mcp/mcp.c so benchmarks can separate repeated open, validation, integrity, and close work from query execution. Shipping builds retain the existing request-end close and allocator collection behavior. Add request_store_retention_can_be_isolated_for_measurement in tests/test_mcp.c. The RED run failed at ASSERT(retained_store); the GREEN ASan/UBSan run passed 1/1, the neighboring request_store tests passed 2/2, clang-format passed, and scripts/check-source-safety.sh passed. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Change run_index_mcp in benchmarks/run_benchmark.py to read the raw MCP result, capture the current daemon log window, and archive a referenced .worker-log before JSON decode failures remove the isolated cache. BenchmarkCommandError now records response, stderr, daemon tails, log path, and content-addressed archive metadata. Add path-with-spaces coverage in tests/test_run_benchmark.py. The focused RED failed because FakeClient.call_tool_text was unreachable through the old call_tool path; all 110 benchmark tests and ruff format checks pass. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
src/store/store.c now records whether sealing failed during SQLITE_CHECKPOINT_TRUNCATE, PRAGMA journal_mode=DELETE preparation, or journal-mode stepping, including SQLite return codes and frame counts. src/pipeline/pipeline.c distinguishes destination sealing, sidecar removal, and cbm_rename_replace failures instead of collapsing them into pipeline.err phase=publish. tests/test_store_checkpoint.c proves an active WAL reader blocks journal detachment and that sealing succeeds after COMMIT. tests/test_pipeline.c pins destination_prepare and rename_replace diagnostics. Verified: CBM_ONLY_SUITE=store_checkpoint make -f Makefile.cbm test (4 passed); two targeted pipeline tests passed under ASan/UBSan; make -f Makefile.cbm lint-format; make -f Makefile.cbm lint-source-safety. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
benchmarks/run_benchmark.py fact_artifact_rows now accepts the content-addressed artifact_name, artifact_sha256, and artifact_bytes fields returned by archive_measurement_log, including when they occur under error_detail. tests/test_run_benchmark.py reproduces the prior zero-row artifacts.json result and pins the retained measurement_log path, digest, and size. Normalization remains O(N + A) time and O(A) output/deduplication memory. Verified: uv run python -m unittest tests.test_run_benchmark (111 passed); ruff format --check benchmarks/run_benchmark.py tests/test_run_benchmark.py; applying the normalizer to runset 6212ff60481c78cd0b4f3d38 emits one 2,306-byte artifact row with SHA-256 aae551fce0dcc3602070131639f903afda98cee1df3924e2f35ad742698763ef. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
benchmarks/run_benchmark.py:summarize_daemon_profiles_since streams only selected msg=prof records from the request log window and records count, total_us, min_us, max_us, and mean_us in O(B + R) time and O(L + P) memory. measure_indexed_query_probes_for_transport now separates mcp_tool_execute/<tool> and mcp_request_total/tools/call from client-observed latency without retaining per-request profile samples. tests/test_run_benchmark.py verifies offset isolation, tool-name filtering, malformed-duration rejection, exact aggregation, and output keys. Verification: uv run python -m unittest tests.test_run_benchmark (112 passed); ruff format --check benchmarks/run_benchmark.py tests/test_run_benchmark.py (passed); git diff --check (passed). Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
src/mcp/mcp.c records resolve_store open_validate, integrity, project_lookup, and session_sync spans plus request_release_store store_close and mem_collect spans under the existing CBM_PROFILE gate. Production behavior and request-scoped SQLite ownership are unchanged. benchmarks/run_benchmark.py selects those six fixed phase keys alongside mcp_tool_execute and mcp_request_total; summarize_daemon_profiles_since retains O(L + P) memory rather than per-request samples. tests/test_run_benchmark.py pins component selection and aggregation. Verification: make -f Makefile.cbm cbm (Clang production build passed); uv run python -m unittest tests.test_run_benchmark (112 passed); ruff format --check benchmarks/run_benchmark.py tests/test_run_benchmark.py (passed); make -f Makefile.cbm lint-format (passed); make -f Makefile.cbm lint-source-safety (passed); git diff --check (passed). Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
src/mcp/mcp.c records resolve_project, graph_counts, dependency_inventory, project_coverage, pagerank, freshness_overlay, and serialize spans around the existing index_status operations under CBM_PROFILE. No response fields, SQL, ownership, or production-profile defaults change. benchmarks/run_benchmark.py adds the seven fixed keys to the bounded indexed-query profile selection. tests/test_run_benchmark.py pins graph_counts collection. Verification: uv run python -m unittest tests.test_run_benchmark (112 passed); make -f Makefile.cbm cbm (passed); make -f Makefile.cbm lint-format (passed); make -f Makefile.cbm lint-source-safety (passed); git diff --check (passed). Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Replace repeated node, edge, and PageRank aggregate scans in src/mcp/mcp.c with cbm_store_get_project_graph_stats(). The project_graph_stats table is keyed by project and store generation; publication refresh is O(P+N+E+R), current reads are O(log P), and legacy or invalidated stores retain an exact O(N+E+R) fallback with O(1) result memory. Invalidate summaries transactionally across canonical graph CRUD, file-delta publication, coverage shadow graph rebuilds, and raw PageRank writes. Preserve cursor mutation_gen ownership, restore invalidation state on rollback, finalize all SQLite statements, and retain request-scoped store close semantics. Make cbm_pagerank_compute() clear stale rank rows for empty graphs and finalize statistics for compute, disabled, empty, unchanged, and deferred policies. Keep authoritative graph publication successful when summary refresh fails; readers then use exact scans and the server logs pagerank.graph_stats_refresh_failed. Tests: store_nodes 123 passed; pagerank 64 passed; mcp 336 passed under ASan/UBSan. make -f Makefile.cbm lint-format and lint-no-suppress passed; lint-ci cppcheck completed before reporting only the subsequently corrected format violations. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
tests/test_cli.c read_test_file and read_test_file_alloc used narrow fopen while production writes through cbm_fopen. test_path_exists used narrow stat, so a valid Windows extended-length path could be reported absent after a successful write. Route all three observers through cbm_fopen/cbm_file_exists. Windows conversion is O(path bytes) runtime work, latency, and transient memory; POSIX remains direct libc. This preserves the long-path capability test without changing product behavior or adding limits. Verified: make -f Makefile.cbm test (7,868 passed, 2 skips); make -f Makefile.cbm lint-ci; MinGW link; Wine exact long-path test (1 passed, 329 filtered). Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Merge upstream/main d6be58e into fork head 70eaa3e (merge base 08f6d1d). Retain upstream's OPTIONAL MATCH saturation correction while keeping the fork's separate bounded-output and configurable working-row contracts. Cypher and MCP schema: - src/cypher/cypher.c and src/cypher/cypher.h use one whole-pattern DFS with stable query-local relationship IDs, relationship uniqueness across every pattern segment, parallel-edge identity, variable-length reachability, query-sized bindings, and typed scalar values. Exhausted working budgets return actionable errors instead of plausible prefix answers. - The read-only language contract is owned by a stable typed capability registry. src/mcp/mcp.c generates the query_graph tools/list description from that registry and appends project vocabulary separately, preventing parser, documentation, and MCP-schema drift. PageRank and configuration: - src/pagerank/pagerank.c makes node/edge SQLite scans and multi-table rank publication failure-atomic, rejects unrepresentable ID-map capacities, requires convergence before publication, preserves outer-transaction rollback, and removes unused node-label allocations. - src/pagerank/pagerank.h, src/cli/cli.c, and src/cli/cli.h source algorithm defaults, accepted extents, recommended minima/maxima, edge-type mappings, and tuning levels from shared typed definitions. The default 100-iteration budget matches the tested convergence fixture; invalid raw values fall back without publishing partial ranks. Discovery, MCP, and resource ownership: - src/pipeline/walk_path.h provides checked geometric path storage shared by pass_envscan.c, pass_pkgmap.c, and path_alias.c. The iterative walkers remove fixed path/depth/file-count truncation, retain exclusion and symlink-cycle handling, and release complete path-alias collections after fatal allocation failures. - src/mcp/mcp.c replaces the fixed semantic keyword array with query-sized pointer storage, distinguishes input/resource/store failures, and frees vector results on every terminal path so combined searches never return partial semantic data as complete. Complexity and capability bounds: - PageRank remains O(I*(V+E)) runtime and O(V+E) memory for I iterations. Whole-pattern matching is output-sensitive and retains the unavoidable worst-case cost of enumerating valid graph matches; working memory is bounded by pattern/DFS state, the used-relationship bitmap, and the configured working-row budget rather than a silent answer-changing prefix cap. - Exact filesystem traversal is linear in visited entries plus resolved path bytes, with O(P+D) traversal workspace for maximum path length P and directory depth D; retained aliases/package entries remain proportional to useful output. Semantic keyword preparation is O(Q) time and O(Q) transient pointers for Q inputs. Verification: - make -f Makefile.cbm -j16 cbm build/c/test-runner built the complete production source set, linked build/c/codebase-memory-mcp, and applied the required macOS ad-hoc signature. - ASan/UBSan focused suites passed: pagerank 68/68, cypher 262/262, and path_alias 21/21. - git diff --check, git diff --cached --check, and both-parent post-commit diff checks passed. The committed tree contains 47 changed files, including the required new src/pipeline/walk_path.h.
Previous behavior: - src/store/store.c:19481 cbm_store_vector_search treated optional node_vectors absence as "vector_search prepare: no such table: node_vectors", so search_graph smoke B3 failed on FAST and legacy indexes without semantic materialization. - cppcheck analyzed production-only constant-false failpoint stubs, took 9m14s locally, and stopped CI before it could finish. Changes: - src/store/store.c:19481-19567 prepares the optional node scan before keyword allocation, reuses store_table_unavailable, validates a present token_vectors table, and closes, finalizes, or frees every early path. - tests/test_store_nodes.c:181 and tests/test_mcp.c:4041 pin empty store results plus explicit JSON semantic_results and TOON semantic[0] responses while malformed vector data remains an error. - src/cypher/cypher.c:4727 splits scalar grouping from binding-aware entity identity so rb_apply_distinct no longer passes nullable binding arguments; exact length prefixes, canonical entity ids, expected-linear hashing, and allocation atomicity remain. - Makefile.cbm:1409 analyzes CBM_ENABLE_TEST_SEAMS so allocation and cleanup failpoint paths are checked without production-stub false positives. - LLVM 20 formatting is applied to the five source files rejected by the exact CI formatter. Complexity and ownership: - Both semantic tables absent returns in O(1) time and memory. - Token vectors present with node vectors absent validates requested rows in O(Q * D) time. - Full exact search remains O(N * (D + Q * D + log K)) runtime and O(Q * D + K + copied result bytes) memory. - DISTINCT remains expected O(total encoded bytes) runtime with O(total unique key bytes) retained storage. - SQLite statements, token readers, keyword vectors, results, and yyjson documents retain single-owner cleanup on success and failure. Verification: - make -f Makefile.cbm -j16 cbm build/c/test-runner - 262 Cypher tests passed under ASan/UBSan - 6 store_vector_search_ tests passed under ASan/UBSan - Explicit JSON/TOON missing-vector-table MCP test passed under ASan/UBSan - make -f Makefile.cbm -j2 lint-ci with cppcheck 2.21 and clang-format 20.1.8 passed in 33.0s - Normalized diffs against merge a0f4380 and parents 70eaa3e/d6be58e found no formatter-induced semantic change Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Resolve the post-merge cross-platform failures without reducing the fork's read-only Cypher or overlay capabilities. - add cbm_stat in src/foundation/compat_fs.c, use _wstat64 for UTF-8 Windows paths, and open directory identities with FILE_FLAG_BACKUP_SEMANTICS; migrate Windows-compiled metadata call sites while preserving POSIX lstat behavior - route Windows-compiled file reads, copies, shell configuration appends, update archives, and MCP SQLite validation through cbm_fopen; keep only POSIX/Linux raw fopen implementations - make cbm_platform_process_group_state distinguish ENOENT/ESRCH disappearance from EACCES/EIO/EOF ambiguity, retaining O(processes) runtime and O(1) auxiliary memory with balanced FILE/DIR cleanup - trust successful Darwin POSIX_SPAWN_SETPGROUP setup for short-lived children while retaining setpgid/getpgid proof for the fork fallback and its existing kill/reap lifecycle - keep query_graph response selection centralized for JSON and TOON, share freshness metadata across serializers, and resolve omitted projects from session/current state before store replacement - use Get-ChildItem -LiteralPath for unpatterned recursive Windows search_code and preserve the separate PowerShell 5.1 Include/Exclude branch - cover CJK cache database listing/querying, Unicode stat and directory identity, Darwin zombie timing, Linux procfs errors, JSON/TOON dirty-overlay defaults, PageRank numeric boundaries, uninstall .exe resolution, and long-root envscan setup Verification: - exact macOS production build: build/final/codebase-memory-mcp - exact Linux production build: build/linux-final/codebase-memory-mcp - Windows production cross-build plus two Wine --version smokes; complete Windows test-runner cross-compile (not treated as native Windows semantics) - ASan/UBSan affected suites: 1,208 passed (platform, subprocess, MCP, pipeline, CLI, input_validation) - make -j3 -f Makefile.cbm lint-ci; scripts/check-source-safety.sh; git diff --check - make -f Makefile.cbm test-analyze BUILD_DIR=build/analyze completed with the repository's existing analyzer warning baseline Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The Ubuntu TSan test subprocess_zombie_only_group_is_quiesced_without_extending_settle observed tree_quiesced=false because cbm_platform_process_group_state treated races from every unrelated /proc entry as an unknown owned-group snapshot. Parse each numeric /proc entry as a checked pid_t, call getpgid() before opening its stat file, skip vanished or mismatched processes, and retain fail-closed UNKNOWN behavior for errors and classification failures involving selected group members. Document the ownership and failure semantics in src/foundation/platform_internal.h. The scan remains O(P) runtime and O(1) auxiliary memory for P processes while reducing procfs file opens and stat parsing from O(P) to O(G) for G target-group members. Verified with 34 focused Linux Docker TSan subprocess tests plus one expected timing skip; five churn-stressed repetitions totaling 170 passes plus five expected timing skips; 61 macOS ASan/UBSan platform and subprocess tests; the Linux production build; lint-ci; formatting; source-safety checks; and git diff --check. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Route cbm_daemon_ipc_transport_probe through metadata-only socket and named-pipe presence checks. POSIX reuses the owner/mode/type validation from the legacy probe; Windows reuses the secure rendezvous snapshot and zero-time WaitNamedPipeW classification. The following runtime connection remains authenticated, while cold-start polling no longer consumes an accept slot or creates a probe-only worker lifecycle. For P endpoint/rendezvous bytes the probe remains O(P) runtime and O(1) auxiliary memory. Extend daemon_ipc_transport_probe_distinguishes_reservation_from_listener to require an empty accept queue after a positive probe. This directly detects the six-client cold-start regression without increasing startup bounds. Size path_alias_tree_fixture_create and its lateral cbm_mkdtemp buffers with CBM_SZ_256, as required by src/foundation/compat.h when Windows expands /tmp through %TEMP%. Reuse cbm_fopen for Unicode temporary paths. This matches the stack-buffer-overflow root cause documented by commit 3bcbc59 and leaves the path-alias algorithm unchanged. Verification: signed macOS production build; ASan/UBSan path_alias 21/21, daemon_ipc 47/47, daemon_bootstrap 24/24; TSan same 92 tests with no report; complete x86-64 Windows ASan/UBSan runner cross-link; clang-format, source-safety, and git diff --check. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Windows path_alias tests reported config_skipped reason=short_read because load_tsconfig_file opened tsconfig.json in text mode: ftell measured physical CRLF bytes while fread translated them to LF. Open the UTF-8 path with cbm_fopen(..., "rb") so sizing, allocation, parsing, and the file-limit check operate on the same exact byte stream. path_alias_tree_fixture_create also failed before the >500-byte path assertion because cbm_mkdir maps to ANSI _mkdir on Windows. Reuse cbm_mkdir_p, the existing UTF-8/extended-length directory owner, and name the fixture mode instead of adding another path helper or literal. For N config bytes, loading remains O(N) runtime and O(N) peak buffer memory. For S fixture segments and final path length P, setup is O(S*P) time and O(P) live path memory; cleanup ownership is unchanged. Verification: macOS ASan/UBSan path_alias 21/21; complete x86-64 Windows ASan/UBSan runner cross-link; clang-format; source-safety; git diff --check. Wine sanitizer startup remains an auxiliary tool limitation (interception_win real_memcpy == 0), so native Windows CI is the runtime authority. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
ahundt
changed the title
Contributor
Author
|
Thanks for the careful review. My preference is still for this PR to be merged, and the current head addresses the concrete blockers you listed. If distillation is the only option, I request and would appreciate Co-Authored-By credit. Status: the blockers from your review are closed, at one SHA. CI, the benchmark, and the head now run on the same commit, which also ends the merge-chase you called out: the
On the numbers: 3.2x was correct against the main of its day, and the gap has now improved to 6x against the main of today! A fresh 39-cell matrix at this branch head measures against a build of current main directly (at the time of writing, 2026-08-02,
None of the remaining deltas is overhead. They are the itemized costs of capabilities and of fixing upstream bugs correctly, each named with what it buys: the semantic-edges pass ( On #879/#880, the direction question: this PR already contains #879's capability in stronger form. #879 proposes weighted degree as a persisted per-node importance score. This branch already computes, persists, indexes, and serves weighted degree, and two graph-derived scores besides. All three tables are written in one publication by Which score to rank by is the question your review actually poses, and it has now been measured. #879 has some strong evidence: its real-corpus top-40 audit caught the test-penalty under-firing, 33 of 40 top-ranked symbols were test scaffolding and 0 after the fix. That audit validates scaffolding suppression, a guard against one failure mode; it does not measure whether #879's score surfaces the right answer, and #879's own text defers exactly that question as "a measured decision" on PageRank. This PR contains the deferred measurement, currently the only retrieval-quality measurement published for either ranking design: with ranking on, the known correct answer ( Tuning the constants cannot close that gap, because the formula's inputs are lexical. #879's writer multiplies degree by name-shape constants, declared at Since you raised the direction question, my answer is that #879's writer should carry the same retrieval-quality measurement before it becomes the ranking anyone depends on. If either PR's approach measures better than what is here, I would readily adopt that approach, and I hold this PR to the same standard. #879's writer is also not required for #880: Executive summary. What I am asking for, and what I would suggest on the ranking question:
|
ahundt
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
1. What does this PR do?
Hey, thanks again for the cool project! This PR continues my earlier PR #151. This Aug 2 addresses your requests: the
api-consolidationbranch is now merged with upstream main. Two true merge commits bring in every upstream commit throughf0c65fb2a997(2026-07-29):d0f00de6bd6dcombines upstream560ad40d19b2with the previously verified headb42801f3f765and resolves all 14 textual conflicts, and41fd25cdd86bintegrates thef0c65fb2a997release-hardening wave. No history was rewritten; both parents are preserved in each merge. At that merge the branch was 0 commits behind upstream main, and every upstream commit throughf0c65fb2a997remains contained here, so CI can run. A third merge,a0f4380b6f18(2026-08-01), integrates every upstream commit throughd6be58ef9d43(2026-07-31). At the time of writing (2026-08-02 16:39 UTC,d6be58ef9d43) that is still the upstream main head, so the branch has caught main and is 0 commits behind. Every defect claim in section 4 was re-verified at that revision. Beyond the three merges, 37 follow-up commits fix defects found while validating the merged results (listed with their verification in each commit message). The merge also removed the 86 committed soak/memlab run-log files (1.3 MiB) reported separately as issue #1330; upstream has since removed them on main in66326c2c.The core claim is unchanged: upstream main's post-edit indexing is labeled incremental but returns a graph that differs from an independent clean rebuild in 3 of 3 repetitions on the measured build, that is, the graph it serves after an edit is incorrect, and upstream's own source comment documents that
FILE_CHANGES_WITHandDATA_FLOWSedges are not reproduced incrementally (pipeline_incremental.c:351-360atd6be58ef9d43). This PR makes incremental source-structure indexing exact, and in its default configuration indexes the same large-codebase edit 3.2× faster than the measured upstream build (569 vs 1,828 ms). After an edit the server names every deferred derived view (rank, routes, semantic edges, architecture), omits stale rank values where returning them would mislead, and withincremental_derived_results_refresh=at_publishproduces a graph identical to the clean rebuild in 3 of 3 repetitions while still beating its own full rebuild (8,568 vs 9,768 ms).The PR adds optional indexing of installed dependency sources with package identity, optional graph-derived search ranking, and a shorter default MCP tool list, and it extends the read-only Cypher subset, Git-snapshot change detection, and semantic analysis. One attribution correction to my previous description: the shared daemon is upstream's subsystem (
src/daemon/, merged here on 2026-07-23); this PR retains it and contributes fixes and hardening on it, listed in section 2. Since findings like these are easy to misread from descriptions alone, I would appreciate review against the code, tests, and published run reports rather than summaries.The full classic API, request schemas, and upstream 3D visualization are retained. Every major capability has a one-command switch and defaults can be trivially revised if preferred.
2. Capabilities and controls
Every setting is declared in one registry with its default, accepted values, and environment override:
CBM_CONFIG_REGISTRYinsrc/cli/cli.c:12997. Two commands control everything:The six preset bundles pair a tool surface (
streamlined-orclassic-) withautomatic-dependency-source-indexing-disabledor-enabled, plus therank-disabledandminimal-indexingbenchmark ablations (codebase-memory-mcp config preset listshows each name with its description; the apply transaction iscbm_config_apply_preset()incli.c:12925). When an environment override pins one of a preset's keys, apply returns status 1 rather than claiming an ineffective change.File names below are unique in the tree and live under
src/; each switch is the exact argument list forcodebase-memory-mcp, with the default in parentheses. Line numbers are at the branch head7433fee6bed3.cbm_pipeline_run_incremental()pipeline_incremental.c:2826config set incremental_reindex always(default
always;full_rebuildrebuilds atomically from scratch andfast_mode_indexes_onlyrestricts incremental reindexing to fast-mode indexes)at_publishrecomputes everything immediately and matches the clean rebuild exactlycbm_pipeline_run_incremental()pipeline_incremental.c:2826config set incremental_derived_results_refresh at_publishconfig set rank_refresh at_publish(default:
defer_all_incremental_reindexesfor both)auto_indexdefault at session start, suppress release notices for dev builds, initialize the active-image fingerprint before comparison, validate fingerprint-cache epochs and admit cross-volume caches, and initialize capped memory once in the daemon instead of in every CLI frontendmain_run_daemon_ctl()main.c:1728daemon start/daemon stop/daemon status(default: session-managed, started automatically)
MEMBER_OFreverse edge and a cross-language normalization pass that repairs extraction-miss cases, so classes participate in traversals and inherit rank from their members instead of staying disconnectedprocess_def()pass_definitions.c:279repair:
pass_normalize.czz_order_core, moved from ninth to firstcbm_pagerank_compute()pagerank.c:407config set rank_enabled false(default
true)index_dependenciesindexes named packages on demand, and when installed packages exceedauto_dep_limit(default 20, 0 = unlimited, max 10,000) the most-imported packages are selected, ranked by project import referencescbm_dep_auto_index_effective()depindex.c:984ranking:
depindex.c:583-616config set auto_index_deps true(default
false)_hidden_tools; calling_hidden_toolsreveals the full classic tool list live, no restart (upstream advertises all tools and curates only statically via startup--profileflags)cbm_mcp_tools_list_range()mcp.c:3429config set tool_mode classic(default
streamlined)formatoverride and an actionable invalid-format error; the merged head keeps upstream's UTF-8 output guard so quoting or truncation cannot split a multibyte charactercbm_mcp_response_format()mcp.c:2685config set default_response_format json(default
toon)auto_indexin
CBM_CONFIG_REGISTRYconfig set auto_index false(default
true; upstream defaults tofalseand skips with a log hint)WITH, and optional-match semantics; adds multi-key ordering, output caps, and schema-aware rejection of unsupported syntax; the tool description, parser, and executor share one definition, and the query schema is stable against incidental storage changescbm_cypher_execute_impl()cypher.c:9091cbm_pipeline_pass_semantic_edges()pass_semantic_edges.c:1783config set semantic_edges_enabled falseconfig set similarity_enabled false(default
truefor both)cbm_git_snapshot_read()git_snapshot.c:319config set githistory_enabled false(default
true)pass_route_nodes.c, run in both the full and incremental pipelines) and adds a dedicated pass that links each HTTP call site to its handler, fixing missing or misclassified linkscbm_pipeline_pass_httplinks()pass_httplinks.c:1375config set httplinks_enabled false(default
true)build_fingerprint_modein
CBM_CONFIG_REGISTRYconfig set build_fingerprint_mode always_rehash(default
cached_exact)3. Measured results
The headline numbers come from a 39-cell benchmark experiment: 13 configurations, each run 3 times, paired and interleaved across configurations to reduce timing drift, on production builds in isolated Linux Docker containers (clang-18,
-O2, 16 build jobs, 16 container CPUs, container--init, fresh case-owned repositories and caches, arm64). Every cell applies the same real C source edit to a large codebase, runs incremental indexing, and compares the resulting graph with a separate clean rebuild after the same edit; every configuration also passes 12 retrieval-evidence checks (4 post-edit retrieval probes across the 3 repetitions: the changed file, its source text, its graph node, and its architecture evidence). All 39 planned cells completed with zero missing, corrupt, duplicate, or unplanned cells. Docker Desktop exposed 7.75 GiB despite the requested 16 GiB limit; the largest observed candidate peak was about 1.6 GiB, so no run was memory-capacity limited.Two scope statements, up front. First, the measured upstream build is
aedb979f5570(upstream main as of 2026-06-28), pinned when the benchmark cohort was frozen; the upstream main now merged into this branch isd6be58ef9d43(2026-07-31). The graph-correctness defect reproduced on the measured comparator, and at current upstream maind6be58ef9d43upstream's source still documents non-reproduced incremental edges (pipeline_incremental.c:351-360, the file is unchanged sincef0c65fb2a997) with no parity verification mechanism. Second, this matrix ran at PR revision2c2de761a24e, so its timing ratios apply to the named revisions, edit, and machine; a second full 39-cell matrix has since re-measured the merged branch head7433fee6bed3against an upstream build atd6be58ef9d43(below). What the merged head does run is the full test suite, including the incremental-parity tests, at the final verified head listed in section 7.All timings are medians (p50) over the 3 repetitions. Repeated query p50 is the median latency of the repeated read-only graph queries the retrieval probes issue within each candidate's persistent MCP session, so it measures a warm service, not process startup. Response payload p50 is the exact default tool-response size (tokens are the recorded deterministic bytes/4 estimate). Peak RSS is the highest peak resident memory observed across the 3 repetitions. "Deferred views named" means the server explicitly names each optional derived view awaiting recomputation after the fast source-structure refresh, and the freshness-scoped comparison still checks every remaining canonical node, edge, property, and file hash against the independent rebuild. A clean rebuild of the same post-edit tree is the ground truth: a post-edit graph that differs from it is incorrect, and queries against it can return wrong answers.
aedb979f5570, defaultminimal-indexingpresetUpstream main's post-edit reindex is labeled incremental but returns an incorrect graph, one that differs from the ground-truth rebuild (speed bought with wrong answers), while this PR's default is both 3.2× faster on the edit (569 vs 1,828 ms) and exact on non-stale source structure. The minimal preset outputs a graph identical to the clean rebuild while indexing the edit 3.6× faster (508 vs 1,828 ms) and full-indexing 1.37× faster (5.136 vs 7.055 s) than the measured upstream build. Two metrics go the other way and I state them plainly: upstream's repeated read-only query latency is lower (6.2 vs 16.7 to 23.2 ms; this PR's replies carry first-response freshness and provenance context and are 29% smaller, 429 vs 604 bytes), and upstream's full index is faster than this PR's default configuration (7.055 vs 10.029 s) because the default enables optional capabilities upstream does not have: disabling semantic edges alone brings the full index to 6.659 s, and the rank, similarity, git-history, and route ablations measure 9.269 to 10.140 s. Upstream peak RSS was not recorded in this run, so no memory comparison against upstream is claimed.
Refresh-at-publish recomputes every global derived result (rank, semantic edges, architecture, routes) immediately after the edit; at 8,568 ms that is still faster than its own 9,768 ms full rebuild, which is the only other way to get them all fresh. It is opt-in; the default defers exactly that graph-wide work and names the deferred views until refreshed.
Automatic dependency indexing stays off by default so its cost stays out of the default first index: enabling it indexed this workload's six installed packages in about 27 s of dependency work at first index, which is work upstream cannot do, and after an edit, checking unchanged dependencies cost 13 ms.
Two earlier development checkpoints (
c9726c7fd9b7,0fcb7e109518) ran in the same matrix and quantify what the post-merge work fixed: their native configurations auto-indexed dependencies (about 20.5 to 20.8 s of the 43.8 s full index) and carried a since-removed startup regression that pushed repeated queries to 187 to 206 ms; the current default full-indexes in 10.0 s with 23.2 ms repeated queries, and both checkpoints also failed the canonical-graph gate that the current head passes.A second 39-cell matrix re-measured the merged result: same container design (13 configurations, 3 repetitions each, paired-interleaved, isolated Linux Docker containers, arm64, 16 container CPUs), run at branch head
7433fee6bed3(2026-08-02) with the upstream comparator freshly built atd6be58ef9d43, the upstream main head this branch merges. All 39 planned cells completed with zero missing, corrupt, duplicate, or unplanned cells. The mutation differs from the first matrix (a new isolated C source file is added instead of editing an existing one), so timings are not comparable between the two matrices. Upstream's default again served an incorrect post-mutation graph in 3 of 3 repetitions: it differs from the ground-truth rebuild of the same tree, so queries after the edit can return wrong answers, and upstream ships no mechanism that detects this. Every configuration of this PR passed its graph gate.d6be58ef9d43, defaultminimal-indexingpresetminimal-indexingis the measured Pareto frontier of this matrix: no passing, fully measured candidate in the same workload is at least as good on overall quality and every cost axis while strictly better on any. All speed comparisons below are how much faster this PR is than current main (d6be58ef9d43) in this matrix; a negative value means slower, and every negative is bought by a named capability gain. Post-mutation indexing: +578% (473 vs 3,208 ms) at upstream-equivalent capabilities and +513% (523 ms) at the default, roughly 6x either way, and correct where upstream's result is incorrect. Clean rebuild: -6% (8.025 vs 7.508 s) at upstream-equivalent capabilities (parity verification runs in the test and benchmark harness, so it adds nothing here; the difference is graph-building work this branch always does, including theMEMBER_OFclass-connectivity repair and exact statistics publication, with no single component isolated by an ablation, and at this price the post-mutation graph is verified correct where upstream's is incorrect); -13% (8.674 s) at the default, which adds ranking, similarity, git history, and route linking with the measured benefits in the table below; +81% (4.141 s) withminimal-indexing, fully correct. Repeated query: -2% (17.6 vs 17.3 ms) at upstream-equivalent capabilities, closing the first matrix's 3 to 4x gap, -14% (20.0 ms) at the default, +4% (16.6 ms) withminimal-indexing, at effectively equal peak memory (1,532 vs 1,536 MB). In this workload upstream's default reply is smaller (306 vs 407 bytes); the PR's replies carry the first-response freshness and provenance context described above. Refresh-at-publish again produced the fully identical graph while beating its own clean rebuild (7,805 ms vs 8,437 ms).The default's remaining costs are itemized capability prices, not overhead, and each is refundable with one switch. Overhead, cost that returns no value, is what this head (
7433fee6bed3) removed: the first matrix's repeated-query deficit does not reproduce (17.6 ms with upstream-equivalent capabilities vs upstream's 17.3 ms), and the like-for-like clean rebuild is -6% (0.52 s, 8.025 vs 7.508 s) at equal peak memory. Parity verification runs in the test and benchmark harness, not in production indexing, so that 0.52 s is not a verification tax: it is graph-building work this branch always does, including theMEMBER_OFclass-connectivity repair and exact statistics publication, with no single component isolated by an ablation. What the build delivers at that price is the guarantee upstream does not offer at any price: the post-mutation result is verified correct. Every cost above that baseline is priced below against the same build with exactly that capability toggled (all other defaults unchanged); the default configuration's own rebuild spread is [8.437, 9.599 s] over 3 repetitions, so prices under 0.6 s are smaller than that spread and should be read as approximate:SEMANTICALLY_RELATEDpair matches the clean rebuild, none missing, none spuriousconfig set semantic_edges_enabled falseconfig set rank_enabled falseSIMILAR_TOedges for near-duplicate discoveryconfig set similarity_enabled falseconfig set httplinks_enabled falseFILE_CHANGES_WITHcoupling edges and Git-scoped incremental change detectionconfig set githistory_enabled falseconfig set auto_index_deps trueSummed, the default's entire repeated-query premium over upstream at the merged head (
7433fee6bed3) is 2.7 ms (20.0 vs 17.3 ms). The one axis where upstream is faster like-for-like is the clean rebuild, where this PR is -6%; what stands on the other side of that price is the post-mutation result: +578% (473 vs 3,208 ms) with a verified-correct graph, where upstream's faster-looking incremental serves an incorrect graph, one that differs from its own ground-truth rebuild in 3 of 3 repetitions, speed bought with wrong answers and no switch that refunds it.Result-quality contribution of the two new optional capabilities, isolated on one PR build (
13a4679b27f6) and test project by repeating the same index-then-query measurement 5 times per state, alternating the capability between enabled and disabled. MRR (Mean Reciprocal Rank) rewards placing the known correct answer near the top, Hit@1 and Hit@5 report whether it appears first or within the first five, nDCG@5 (normalized Discounted Cumulative Gain) rewards correct ordering within the first five, and task success means the task's named expected answer was returned:zz_order_core) ahead of eight lexical-only decoyszz_order_corereturned 1st of 9; MRR, Hit@1, Hit@5, nDCG@5, task success all 1.000zz_order_core9th of 9; MRR 0.111, the other four metrics 0canonicalDependencyAPI) with its dependency, package, and read-only provenance on the same resultcanonicalDependencyAPIreturned 1st, with package identitycanonicalDependencyAPIabsentSemantic refresh quality was re-verified in the same matrix: with refresh at publish, every judged
SEMANTICALLY_RELATEDpair matches the clean rebuild, with none missing and none spurious.4. Upstream main defects this PR corrects
Every claim in this table was re-verified by direct code reading at upstream main
d6be58ef9d43(2026-07-31, the revision merged into this branch) unless a different revision is named; upstream file:line references are at that revision.d6be58ef9d43cbm_pipeline_run_incremental()pipeline_incremental.cFILE_CHANGES_WITHandDATA_FLOWSedges are not reproduced incrementally (pipeline_incremental.c:351-360); no parity guarantee or verification mechanism exists; measured divergence 3 of 3 onaedb979f5570resolve_store()mcp.cresolve_store_internal,mcp.c:2081-2083); upstream's daemon single-ownership makes cross-process replacement rarer but the check is still absentWITHstages, optional matches, and multi-keyORDER BYcompute correctlycbm_cypher_execute_impl()cypher.cparse_order_by_clauseparses exactly one ordering key, so secondaryORDER BYkeys are silently dropped (cypher.c:1699-1709); the aggregate, post-WITH, and optional-match failures were verified on the measured comparatoraedb979f5570; upstream has since fixed aggregate-column classification and two optional-match defects (PRs #1221, #1177, #1385, contained here through thed6be58ef9d43merge), while multi-keyORDER BYremains absent upstreamLIMITlowers but cannot bypass the configured row cap, and truncation is reported with the limiting condition instead of being silentscan_pattern_nodes()cypher.cmax_rowstimes a fixed growth factor before inline property filters andWHERErun (cypher.c:3027-3052, cap applied atcypher.c:3037), so matching rows past the prefix are discardedauto_indexin
CBM_CONFIG_REGISTRYfalseand an unset value skips indexing with only a log hint (mcp.c:11156-11169)MEMBER_OFreverse edge and a cross-language repair pass for extraction-miss casesprocess_def()pass_definitions.c:279repair:
pass_normalize.cDEFINES_METHODonly when extraction supplies the parent; there is no reverse edge and no repair pass (pass_normalize.cdoes not exist upstream), so extraction-miss classes stay disconnectedindex_statusreports whether the working tree is dirty and whether the index matches itadd_git_context_json()mcp.cadd_git_context_json(mcp.c:2242) emits branch and commit identity but no working-tree-state field; upstream's watcher tracks a dirty signature internally but does not report it as provenancecbm_pipeline_githistory_compute_with_limits()pass_githistory.c:646FILE_CHANGES_WITHcandidates at a hard-coded 8,192 (MAX_COUPLINGS,pass_githistory.c:346) collected in hash-table iteration order with no sort, no configuration, and no log (pass_githistory.c:215-217, 331), so on a history with more qualifying pairs the emitted coupling edges are an arbitrary subset rather than the strongest couplings; per-file temporal facts stop at the first 16,384 files encountered (MAX_FILE_TEMPORAL,pass_githistory.c:406)Three defect claims from my previous description are retired because re-auditing upstream showed them already fixed there: the half-opened SQLite handle on failed open (closed upstream in
e6e9c58e, before the measured comparator), the config-linking pass's large stack arrays (the pass was rewritten upstream and now holds only small static tables), and the dropped route-discovery pass (rebuilt upstream aspass_route_nodes.c; the merged branch keeps upstream's pass and adds its own, section 2). Shutdown lifecycle also converged: upstream's daemon now joins worker threads and plumbs cancellation, and the merged branch carries both sides' shutdown work; this PR's remaining distinct contribution there is the verification discipline (leak gate, cancellation and reaping tests) rather than a defect claim.One further upstream hard-limit defect is fixed here but deliberately kept out of the table, because neither tree currently wires the affected scanner into a production pass: upstream's env-URL walker holds a fixed 512-byte-per-frame path stack and silently skips directories nested deeper than 256 levels (
pass_envscan.c:514, 532atd6be58ef9d43). This branch's walker uses dynamically grown path storage (walk_path.h), logs unrepresentable bindings instead of dropping them, and pins both behaviors with the regression testsenvscan_walks_more_than_256_pending_directoriesandenvscan_accepts_root_path_longer_than_512_bytes.Every remaining row is regression-tested. Defects found while merging upstream and validating the result were fixed before this update; each is documented in its commit message. Two behaviors are boundaries rather than defects: the Cypher subset does not claim full openCypher conformance, and optional derived results refresh on the documented deferred schedule. Production
src/contains nosystem()calls, and the heap-leak gate reports 0 leaked bytes at the final verified head.5. Compatibility, breaking changes, and migration
config set tool_mode classicrestores the classic list on a running installation. Streamlined and classic surfaces answer from the same graph and Cypher contract.daemon stoprefusal while committed clients remain, and orderly SIGINT/SIGTERM shutdown (cancel and reap work, join background threads, close transports, release claims and leases, remove the endpoint). This PR's daemon changes are the fixes listed in section 2.--reset-indexesis passed; the system-wide upgrade verified at an earlier head preserved all 462 existing indexes across 17 detected clients. The installer recognizes configuration written by earlier development builds and replaces it canonically instead of rejecting it, keeps user-owned TOML child tables when replacing a managed parent table, and refuses to overwrite genuinely user-edited entries.d6be58ef9d43, the repository's CI has run its full matrix on this pull request at7433fee6bed3: the test shards (Linux gcc, macOS 14 and 15-intel, Windows CLANG64), ThreadSanitizer on three platforms, lint, smoke on Linux, macOS, and Windows, CodeQL, and the security gates all pass; the DCO sign-off check is the only failing check.6. How to run the tests
The full suite passed with zero failures at the final verified head (7,823 passed, 2 platform-specific skips), and the CI lint gates (
lint-ci) exit 0 there. The suite includes generalized extraction and incremental tests across C, C++, Go, Python, JavaScript, TypeScript, Java, Kotlin, and Rust, mixed-language call resolution, the daemon lifecycle, bootstrap, and IPC suites, and the tests that arrived with the merged upstream commits. The benchmark runner records exact candidate commits, binary SHA-256 values, compiler and optimization flags, CPU and memory allocation, environment overrides, case cleanup, and invalid-run status in an immutable audit manifest.7. Evidence and revisions
f0c65fb2a997d6be58ef9d43d0f00de6bd6d560ad40d19b2and the previous verified headb42801f3f765; all 14 textual conflicts resolved, no history rewrite41fd25cdd86bf0c65fb2a997a0f4380b6f18d6be58ef9d43; no history rewrite24aa8a415f5c7433fee6bed32c2de761a24eaedb979f5570c9726c7fd9b7,0fcb7e10951813a4679b27f6All experiments ran on production builds. The first 39-cell run set's immutable audit manifest has SHA-256
1becd57c00edbe965073f4d7c9a79cca8812947f207ed51851c856369aa2c002, and the second run set's (branch head7433fee6bed3, 2026-08-02) has SHA-2561e8f0d96128bd352107a0fb1633bcb0bcbd696a5ea8cb8226569d62e580a65e0; each experiment root retains the input specification, per-run manifests, container environment records, logs, build identities, and cleanup status, so the numbers are independently reproducible with the command above. The generated reports are published in a gist; its current tree contains the 2026-07-28 run set and the earlier 39-cell runbe2097bdbf14, and the focused ranking, dependency, and semantic reports remain in the gist revision history.Ranking follows PageRank (Brin and Page, 1998) and LinkRank (Kim, Son, and Jeong, 2010); nDCG (Järvelin and Kekäläinen, 2002) defines the ranking metric; RepoGraph (ICLR 2025) motivates repository graphs. The repository experiments, not the citations, are the evidence that this implementation works.
8. Checklist
make -f Makefile.cbm testat the final verified head: 7,823 passed, 2 platform-specific skips)make -f Makefile.cbm lint-ciexit 0 at the final verified head: cppcheck 2.20, LLVM 20 clang-format, suppression-policy and source-safety checks; the previous body's 49 open findings and 20 formatting violations are cleared)