fix(sql): recover T-SQL bracket-named, CREATE OR ALTER, and PROC routines (#3164)

[egarcia74]

Fixes #3164. Replaces #2960 — same change, squashed to a single commit and rebased onto current v8 so it cherry-picks cleanly; #2960 carries the review history (four rounds of Graphify-review findings, each reproduced and either fixed or refuted in the thread) and is being closed with a pointer here.

Problem

T-SQL stored procedures and functions are silently dropped from the graph. On a real 391-file SQL Server DACPAC corpus ([Schema].[Object] naming throughout), 0 of 26 stored procedures produced a node while tables and views extracted fine, so the gap is invisible unless you count.

Two independent defects stack:

1. The grammar cannot parse the T-SQL routine shape. CREATE PROCEDURE ... AS BEGIN ... END has no create_procedure production in tree-sitter-sql, so ERROR-node regex recovery is these objects' only path into the graph.
2. The recovery regex does not speak T-SQL. It accepted only bare or double-quoted names — no bracket-delimited names, no CREATE OR ALTER, no PROC shorthand. On this corpus every routine uses brackets, so recovery recovered nothing.

What this PR does

All changes are in graphify/extractors/sql.py:

• One shared _ROUTINE_RECOVERY_RX, hoisted to module level. The two recovery sites (walk-time ERROR-node scan, whole-file has_error fallback) had drifted; the second minted a phantom dbo() node for a mixed-delimiter name (CREATE PROCEDURE dbo.[usp_Mixed]). Sharing the pattern fixes that and prevents re-drift.
• Bracket-delimited identifiers, including T-SQL's ]] escape for a literal ] ([dbo].[a]]b] recovers as one routine, not truncated at the escape).
• CREATE OR ALTER alongside OR REPLACE; PROC alongside PROCEDURE; \bCREATE so a bare word ending in CREATE (AUTOCREATE PROCEDURE x) cannot match.
• The per-ERROR-node scan is removed. Any ERROR node sets root.has_error, so the whole-file scan already covers everything a per-node scan could, from the same pattern with _add_node deduping by id. The per-node scan was also unsound: an ERROR fragment can begin mid-comment (tree-sitter's lexer does not nest /* */), which let commented-out DDL fabricate routine nodes.
• Comment/string masking for the whole-file scan (_scan_sql, a linear offset-preserving scanner): single-quoted strings (dynamic SQL) and line/block comments (nesting-aware; an unclosed block comment blanks to EOF) are blanked; double-quoted and bracket-delimited identifiers are preserved because they carry the recoverable names, and a match whose CREATE starts inside one is skipped as identifier data. An unterminated or comment-swallowing delimiter span is irreducibly ambiguous, so the union of every reading is blanked — over-blanking can lose a routine on an already-broken line, never fabricate one.

Known, documented holes: multi-line single-quoted dynamic SQL, and string-quoting dialects the mask does not model (MySQL "..." strings, PostgreSQL dollar quoting) still reach the scan when the has_error gate is armed.

Tests

tests/test_multilang.py: bracket names, ]] escapes, OR ALTER, PROC, mixed-delimiter no-phantom, keyword-inside-identifier no-phantom, commented-out and string-embedded DDL not fabricated, nested and unclosed comments, plus a differential fuzz over the mask (a 1M-case fuzz baseline was run offline against the final scanner).

Validation

• Repro from fix(sql): T-SQL stored procedures and functions are silently dropped — 0 of 26 recovered on a 391-file SQL Server corpus #3164 on this branch: ['[dbo].[usp_Load]()', '[dbo].[usp_Refresh]()', 'routines.sql'] (was ['routines.sql'] on v8).
• Full suite on this branch over current v8 (0.9.51): 5041 passed, 49 skipped (test_ollama*.py excluded — local Ollama config, unrelated).
• ruff check graphify/ tests/: all checks passed.
• Merges cleanly alongside the companion core-dependency change (Should tree-sitter-sql be a core dependency? Default installs still extract 0 nodes from .sql (follow-up to #1745, #3047) #3165 / formerly feat(sql): make tree-sitter-sql a core dependency so default installs parse .sql (#1745) #2959); both touch tests/test_multilang.py.

Out of scope, noted in #3164: a bracket-named CREATE FUNCTION that does parse structurally gets the label dbo].[fn_Total( — a separate delimiter-stripping defect in the structural path.

[graphify-labs]

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 3 advisory finding(s) below merit a look before merge.

Formal verification. No changes could be formally verified in this run.

---

Graphify review — findings

Rewrites SQL routine recovery so the ERROR-node scan and the whole-file has_error fallback share a single _ROUTINE_RECOVERY_RX, which now handles double-quoted and bracket-delimited (""/]]-escaped) names, PROC shorthand, CREATE OR ALTER, and requires a word boundary on CREATE — fixing quoted PL/pgSQL and bracket-named T-SQL routines being dropped or split into mismatched duplicate nodes. Adds _scan_sql, a linear character scanner that blanks comments and single-quoted strings (so dynamic SQL and commented-out DDL can't fabricate routines) while preserving delimited identifiers verbatim and returning their spans for the recovery scan to skip; block-comment blanking is nesting-aware and distrusted/ambiguous delimiter spans blank the union of all readings to end-of-line, carrying comment state forward. The trade is documented over-blanking on non-nesting dialects (MySQL/Oracle) and multi-line dynamic SQL as known false-negative holes, chosen so a recovery-only path never fabricates.

Worth a look

• Dollar-quoted PostgreSQL strings are left visible to routine recovery — graphify/extractors/sql.py:77 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
• PostgreSQL E'' strings are not masked when they use backslash-escaped quotes — graphify/extractors/sql.py:165 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
• Recovery scan can fabricate routines from PostgreSQL dollar-quoted strings — graphify/extractors/sql.py:659 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 318 functions depend on the 146 functions this change touches.

Health — this change adds coupling hotspots:

• new: dispatch_command() — 2 callers, 122 callees
• new: extract_sql() — 20 callers, 9 callees
• new: walk() — 1 callers, 8 callees
• new: test_poisoned_manifest_is_healed() — 0 callers, 6 callees

Verification — 318 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

• verification_scope: 158 function(s) in the blast radius were not formally verified this run

Formal verification

Could not verify: Could not verify extract\_sql.

The verifier did not have enough to check extract\_sql, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set

· 4 more finding(s) on lines outside this diff (see the check run).

[safishamsi]

Shipped in v0.9.52 via authorship-preserving cherry-pick so you keep contributor-graph credit. Thanks @egarcia74! Release: https://github.com/Graphify-Labs/graphify/releases/tag/v0.9.52