fix(extract): retain provenance for legacy and data nodes

[GmailTedam]

Summary

• retain one file-level graph node for deliberately unexpanded and oversized JSON artefacts
• backfill legacy sourceless semantic nodes only from real incident-edge provenance
• keep provenance selection deterministic and continue warning when no evidence exists

Evidence

• 94 focused tests passed, 0 skipped
• Ruff passed on all changed source and tests
• BulletTrain regeneration: 88,458 nodes / 159,519 edges, 0 schema errors
• all 3,442 JSON files in the manifest have graph source coverage
• multigraph diagnostic: 0 missing endpoints, 0 dangling endpoints, 0 duplicate/collapsed edge groups

[Copilot]

Pull request overview

This PR updates JSON extraction and graph rebuild behavior to preserve file-level provenance for JSON artefacts that are intentionally not structurally expanded (oversized/data-shaped JSON) and to deterministically backfill missing source_file on legacy semantic nodes using incident-edge provenance.

Changes:

• JSON extractor now always emits a single file-identity node for skipped/oversized/data JSON instead of emitting zero nodes.
• Graph builder backfills source_file for legacy nodes that lack it, selecting deterministically from incident edges’ source_file.
• Tests updated/added to lock in the new “retain file identity + deterministic provenance” behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
tests/test_extract.py	Updates expectations so skipped/data/oversized JSON still yields one file-identity node.
tests/test_build.py	Adds coverage for deterministic source_file backfill from incident-edge evidence (and warning behavior when none exists).
graphify/extractors/json_config.py	Emits a file-level node even when JSON is skipped or too large, preserving provenance visibility.
graphify/build.py	Implements pre-validation provenance backfill for legacy nodes missing source_file.

Suppressed comments (1)

graphify/build.py:895

• This loop skips nodes whenever the source_file key exists, even if it is empty/whitespace or non-string. Elsewhere in build_from_json (e.g., _fold_node_aliases) emptiness is treated as missing; keeping the current check can leave nodes effectively sourceless and prevent deterministic backfill.

        if not isinstance(node, dict) or "source_file" in node:
            continue

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[graphify-labs]

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 2 advisory finding(s) below merit a look before merge.

Formal verification. No changes could be formally verified in this run.

---

Graphify review — findings

Makes extract_json always emit a file-identity node so data JSON (fixtures, datasets, oversized files, unparseable content, missing grammar) stays visible to freshness/affected checks, while still skipping per-key expansion for non-config JSON to avoid orphan key-node blowup. Backfills source_file on legacy semantic nodes during build_from_json by deriving it deterministically from the incident edge with the lowest case-folded path, and leaves the node flagged by the validator when no incident edge proves a path rather than inventing one.

Worth a look

• incident_sources keyed by raw endpoint but node id may be coerced later, causing missed backfill — graphify/build.py:895 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
• Oversized JSON no longer reports an error — graphify/extractors/json_config.py:88 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 1566 functions depend on the 617 functions this change touches.

Health — this change adds coupling hotspots:

• new: _rebuild_code() — 98 callers, 50 callees
• new: build_from_json() — 192 callers, 18 callees
• new: build_merge() — 53 callers, 13 callees
• new: to_obsidian() — 36 callers, 13 callees
• new: extract_files_direct() — 17 callers, 20 callees
• new: to_wiki() — 41 callers, 7 callees
• new: extract_corpus_parallel() — 26 callers, 11 callees
• new: _call_claude_cli() — 31 callers, 9 callees
• …and 40 more — each is listed as a finding

Verification — 1566 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

• verification_scope: 1179 function(s) in the blast radius were not formally verified this run

Formal verification

Could not verify: Could not verify build\_from\_json.

The verifier did not have enough to check build\_from\_json, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: not verifiable: all 6 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly NameError — names the real obstacle, not a sampling gap)

Could not verify: Could not verify extract\_json.

The verifier did not have enough to check extract\_json, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set

· 48 more finding(s) on lines outside this diff (see the check run).