Skip to content

fix(pipeline): escape dynamic edge properties as JSON - #1065

Open
tmonestudio wants to merge 1 commit into
DeusData:mainfrom
tmonestudio:codex/fix-json-edge-properties
Open

fix(pipeline): escape dynamic edge properties as JSON#1065
tmonestudio wants to merge 1 commit into
DeusData:mainfrom
tmonestudio:codex/fix-json-edge-properties

Conversation

@tmonestudio

@tmonestudio tmonestudio commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Prevents malformed JSON in edge properties when parser-derived values contain JSON-significant characters.

Two edge builders serialized dynamic strings directly:

  • sequential IMPORTS edges wrote local_name without escaping (defence-in-depth; import aliases are identifier-shaped);
  • parallel GRAPHQL_CALLS edges escaped the callee but not the operation text.

The reachable GraphQL reproduction is a single-line, single-quoted Python string whose operation contains a bare double quote.

Fix

Use the existing cbm_json_escape helper for both dynamic values before formatting the properties object.

The regression fixture keeps the whole-table json_valid(properties) invariant and now passes an embedded quote through extraction to emit_graphql_edge.

Validation

Checklist

  • Every commit is signed off (git commit -s)
  • Regression fixture is demonstrably RED without the fix and GREEN with it
  • Whole-table JSON-validity invariant retained

@tmonestudio
tmonestudio requested a review from DeusData as a code owner July 12, 2026 22:47
@tmonestudio
tmonestudio force-pushed the codex/fix-json-edge-properties branch from a88ccc4 to 67433fc Compare July 12, 2026 23:47
@DeusData DeusData added the bug Something isn't working label Jul 14, 2026
@DeusData DeusData added this to the 0.9.1-rc milestone Jul 14, 2026
@DeusData DeusData added parsing/quality Graph extraction bugs, false positives, missing edges priority/high Needs near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker. labels Jul 14, 2026
@DeusData

Copy link
Copy Markdown
Owner

Thanks for the focused JSON escaping fix. This is a high-priority 0.9.1-rc graph-correctness PR. Review will verify that dynamic edge properties are escaped identically in sequential and parallel paths and that quotes, backslashes, control characters, and non-ASCII values remain valid round-trip JSON.

isc-tdyar added a commit to isc-tdyar/codebase-memory-mcp that referenced this pull request Jul 14, 2026
Add cbm_pipeline_set_version() to stamp a version label (e.g. '28.0',
'30.0') on every node emitted during indexing. Versioned runs bypass
the incremental path so both versions accumulate side-by-side in the
graph.

Add diff_versions MCP tool: given a project, from_version, and
to_version, returns added and removed node names (default label: Class,
limit 500 per side) using OPTIONAL MATCH anti-join queries.

Also auto-derives version from trailing numeric path segment in
index_repository (e.g. /exports/MyApp/28.0 → version='28.0') when
the version param is omitted.

Refs DeusData#1065

Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
isc-tdyar added a commit to isc-tdyar/codebase-memory-mcp that referenced this pull request Jul 14, 2026
Add cbm_pipeline_set_version() to stamp a version label (e.g. '28.0',
'30.0') on every node emitted during indexing. Versioned runs bypass
the incremental path so both versions accumulate side-by-side in the
graph.

Add diff_versions MCP tool: given a project, from_version, and
to_version, returns added and removed node names (default label: Class,
limit 500 per side) using OPTIONAL MATCH anti-join queries.

Also auto-derives version from trailing numeric path segment in
index_repository (e.g. /exports/MyApp/28.0 → version='28.0') when
the version param is omitted.

Refs DeusData#1065

Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
isc-tdyar added a commit to isc-tdyar/codebase-memory-mcp that referenced this pull request Jul 15, 2026
Add cbm_pipeline_set_version() to stamp a version label (e.g. '28.0',
'30.0') on every node emitted during indexing. Versioned runs bypass
the incremental path so both versions accumulate side-by-side in the
graph.

Add diff_versions MCP tool: given a project, from_version, and
to_version, returns added and removed node names (default label: Class,
limit 500 per side) using OPTIONAL MATCH anti-join queries.

Also auto-derives version from trailing numeric path segment in
index_repository (e.g. /exports/MyApp/28.0 → version='28.0') when
the version param is omitted.

Refs DeusData#1065

Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
@DeusData

Copy link
Copy Markdown
Owner

The fix is correct and I want to land it — escaping parser-derived strings through cbm_json_escape before embedding them in properties JSON is exactly right (same class as the service-edge JSON-validity work in #898 and the UTF-8 fix in #1017). One thing to fix first, though:

The test as written is a vacuous guard. I verified it: reverting both production changes (the esc_ln/esc_op escaping in pass_definitions.c and pass_parallel.c) and re-running contract_edge_parallel_service_edges still passes — the AND json_valid(properties)=0 → invalid_props==0 assertion never fires because the fixture contains no string with a JSON-breaking character. So the test passes with and without your fix, which means it can't catch a regression.

To make it a genuine reproduce-first guard, the fixture needs a value that actually breaks JSON when unescaped — e.g. an import whose local_name or a service-edge URL/topic string contains a double-quote, backslash, or a CR/LF. With that, json_valid(properties) would be 0 on unfixed code (test RED) and 1 with your escaping (GREEN). A URL argument with an embedded quote is probably the easiest to construct in the fixture.

Could you add that? Once the test is RED-without-the-fix I'll merge immediately — the production change is already good. Thanks for closing a real JSON-corruption gap.

@tmonestudio
tmonestudio force-pushed the codex/fix-json-edge-properties branch 2 times, most recently from 6392ffb to f809e98 Compare July 19, 2026 17:34
@tmonestudio

Copy link
Copy Markdown
Contributor Author

Thanks — addressed. The fixture now includes a GraphQL operation with a literal double quote (user(name: "quoted")), so reverting the esc_op escaping makes json_valid(properties) fail, while the patched code keeps the properties valid JSON.

I also cleaned the branch onto current main as a single commit (f809e98). The previous Ubuntu smoke failure was the unrelated EOF shutdown timeout (binary did not exit within 5 seconds after EOF); the fresh pr-smoke (ubuntu-latest) run is now green, as is macOS. The remaining long-running checks are still in progress. Ready for re-review — thank you!

@tmonestudio
tmonestudio force-pushed the codex/fix-json-edge-properties branch from f809e98 to 4811126 Compare July 19, 2026 18:58
@DeusData

Copy link
Copy Markdown
Owner

Thanks for addressing the vacuous regression and for cleaning the branch. Current head 4811126908d77c274850956a8cc9c19e099a41e6 has a green check set. The exact-head re-review will verify that the literal-quote fixture is genuinely RED when the production escaping is reverted and GREEN with the fix. No additional contributor action is requested in the meantime.

isc-tdyar added a commit to isc-tdyar/codebase-memory-mcp that referenced this pull request Jul 25, 2026
Add cbm_pipeline_set_version() to stamp a version label (e.g. '28.0',
'30.0') on every node emitted during indexing. Versioned runs bypass
the incremental path so both versions accumulate side-by-side in the
graph.

Add diff_versions MCP tool: given a project, from_version, and
to_version, returns added and removed node names (default label: Class,
limit 500 per side) using OPTIONAL MATCH anti-join queries.

Also auto-derives version from trailing numeric path segment in
index_repository (e.g. /exports/MyApp/28.0 → version='28.0') when
the version param is omitted.

Refs DeusData#1065

Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
@tmonestudio
tmonestudio force-pushed the codex/fix-json-edge-properties branch from 542bb6c to 0ab6f1d Compare July 25, 2026 23:55
isc-tdyar added a commit to isc-tdyar/codebase-memory-mcp that referenced this pull request Jul 28, 2026
Add cbm_pipeline_set_version() to stamp a version label (e.g. '28.0',
'30.0') on every node emitted during indexing. Versioned runs bypass
the incremental path so both versions accumulate side-by-side in the
graph.

Add diff_versions MCP tool: given a project, from_version, and
to_version, returns added and removed node names (default label: Class,
limit 500 per side) using OPTIONAL MATCH anti-join queries.

Also auto-derives version from trailing numeric path segment in
index_repository (e.g. /exports/MyApp/28.0 → version='28.0') when
the version param is omitted.

Refs DeusData#1065

Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
@DeusData

Copy link
Copy Markdown
Owner

Thank you for this — the production fix is exactly right and we want it. One problem with the regression test, and then it merges.

What is correct. Both sites interpolated parser-derived text straight into JSON properties without escaping: local_name on IMPORTS (pass_definitions.c:474) and operation on GRAPHQL_CALLS (pass_parallel.c:1997). Nice catch that callee was already escaped on that second one and operation was not — that asymmetry is easy to read past.

You reused cbm_json_escape rather than hand-rolling a second escaper, which is the right instinct. I checked the helper's truncation behaviour too: each escape tests remaining space before writing, so output can never end mid-sequence with a dangling backslash. And I did the sizing math on the wrapper — worst case for the graphql props is about 812 bytes against a 1 KB buffer, imports 144 against 256, so it never truncates the closing "}. Incidentally your fix also closes a pre-existing malformation vector there: a raw local_name over ~238 bytes previously could truncate that closing brace.

I tried to break it and could not. Bare quote, trailing backslash, control characters, embedded NUL, and a crafted-length truncation attack are all contained post-fix. The round trip is clean too: encoded once, bound verbatim through the parameterised bind_text, decoded once by readers — no double-escape anywhere.

The problem: the regression test passes with and without your fix.

The fixture uses a multiline triple-quoted Python string. That operation never reaches emit_graphql_edge at all — strip_and_validate_string_arg (extract_calls.c:1450) rejects any string argument containing \n, since only \t is exempt below MIN_PRINTABLE. So first_string_arg comes back NULL and the emitter falls back to the callee name "gql", which is perfectly valid JSON either way. Your ASSERT_EQ(invalid_props, 0) therefore holds on unpatched main.

The same applies to the mechanism described in the PR body: a multiline operation producing malformed JSON is end-to-end unreachable. The genuinely reachable vector is a bare double quote on a single line — which is a better bug than the one you described, and your fix already handles it.

The change I would like, and then this goes in:

return gql('query GetUser { user(name: "quoted") { id name } }')

A single-quoted Python string, so the inner double quote survives extraction, and every character is ≥ 0x20 so validation passes it through. Pre-fix that emits "operation":"GetUser { user(name: "quoted")…"json_valid returns 0 → your assertion goes red.

Please confirm it fails with the production hunks reverted before pushing. That revert-check is the whole point of the exercise here — a green test that would also be green without the fix tells us nothing later.

Two smaller notes, neither blocking:

  • The IMPORTS local_name path has no discriminating fixture (the alias in the test is the benign "gql"), and the "multiline Rust import alias" scenario in the description does not really exist since Rust aliases are identifiers. That is completely fine as defence-in-depth — just worth labelling it as such rather than as a reproduction.
  • The whole-table json_valid(properties) = 0 sweep is a genuinely good invariant-style assertion. Please keep it; it becomes valuable the moment the fixture actually exercises the escape path.

One thing I noticed while reviewing that is not yours: emit_grpc_edge interpolates service and method raw into properties using the same pattern. Both derive from identifier-shaped text so the risk is low, but we will track it on our side.

Escape import aliases and GraphQL operation strings before serializing edge properties, preventing raw control characters and quotes from producing malformed JSON.

Add a reachable single-line GraphQL regression fixture and assert that emitted edge properties remain valid JSON.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Signed-off-by: tmonestudio <tmonestudio@users.noreply.github.com>
@tmonestudio
tmonestudio force-pushed the codex/fix-json-edge-properties branch from 0ab6f1d to 2d6766f Compare July 31, 2026 23:14
@tmonestudio

Copy link
Copy Markdown
Contributor Author

Updated as requested. The fixture now uses the reachable single-line, single-quoted Python form, so the embedded double quote reaches emit_graphql_edge.

I also ran the explicit sensitivity check before pushing the final branch:

The PR is now rebased on current main as one signed-off commit (2d6766f6). I kept the whole-table json_valid(properties) invariant and corrected the PR description to distinguish the reachable GraphQL reproduction from the defensive IMPORTS hardening. Full CI is running on the final head.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working parsing/quality Graph extraction bugs, false positives, missing edges priority/high Needs near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants