Extraction is structured output, so format drift gets frozen into the graph

[behrnt-slatgng]

Disclosure first: I build Grunz, a hosted chat + coding agent on abliterated open-weight models. Nothing to sell — graph construction has a failure mode that other RAG systems do not, and it is worth naming.

Extraction is structured output, so the graph inherits format drift

The indexing pipeline asks a model to emit entities and relationships in a fixed shape. That is a structured-output contract. Everything downstream — the graph, the communities, the summaries, every query — is built on it.

Which means graphrag is more exposed to output-format degradation than ordinary RAG is, because in ordinary RAG a malformed generation is one bad answer, and here it is a permanent defect baked into the index.

Why this bites harder now

A lot of people index with open-weight models, and a growing share of those are abliterated / decensored checkpoints. On those, the first thing that degrades is not knowledge but instruction-following and output-format adherence. The model still knows what an entity is. It gets worse at emitting it in the shape you specified.

So you do not get loud parse failures. You get a slightly worse graph:

• entities silently dropped because the delimiter drifted mid-record
• relationship types free-texted instead of taken from your allowed set, fragmenting the schema
• an extra preamble on one chunk in forty, so that chunk contributes nothing
• descriptions that run long and blur the entity boundary

None of this errors. The pipeline completes. You get a graph that is quietly 8% thinner in places that correlate with nothing you can inspect.

Why it is worse than a bad answer

It is frozen. A bad generation at query time is one bad answer. A bad extraction is in the index until you re-run the whole thing.

It is invisible at evaluation. You evaluate query quality, but the defect is in construction. Low recall on a topic looks like a retrieval problem, and the actual cause is that three chunks about that topic failed extraction two hours earlier.

Community summarisation compounds it. Summaries are built from the extracted graph, so a gap in extraction becomes a confident summary of a partial neighbourhood.

Questions

1. Does indexing report extraction health — records attempted vs parsed, malformed rate per chunk? If that exists I would like to be pointed at it, because it is the number that would catch this. If it does not, it seems like the highest-value observability you could add: a per-chunk parse-success rate would immediately surface a model that is quietly underperforming.

2. Is there interest in constrained decoding for the extraction step? Grammar-level constraint removes the model's willingness from the equation entirely. For a step whose output is frozen into an index, that seems worth the throughput cost, and it is much more defensible than retrying.

3. Has anyone compared graph density for the same corpus across extraction models? My expectation is that the spread is wider than people assume, and that it correlates with format compliance rather than with model quality scores.

Happy to be told extraction is already validated strictly and this is a non-issue. I would rather find that out than keep worrying about it.

[Dimitri Pletschette (dmxplx)]

My advice: treat extraction quality as a first-class metric. Track parse success, schema compliance, and graph density, and consider constrained decoding to prevent silent format drift. This could help? https://github.com/DimYiannis/Constrained-GraphRag

[behrnt-slatgng]

constrained decoding is prob the right fix when u own the endpoint. we dont always, so we end up validating after and retrying. does urs work against a hosted openai compatible endpoint or only local

[Dimitri Pletschette (dmxplx)]

From what I found, and unless someone as a better idea, for hosted endpoints that don't support constrained decoding, validation plus retrying sounds like the more portable approach. Hope this helps. If this solved your issue, please mark this comment, or another relevant one in the future, as the "solution" so that other users can see what worked for you. It would be highly appreciated. Thank you very much. https://microsoft.github.io/graphrag

[behrnt-slatgng]

tried to mark urs as the solution but this category doesnt have that option. validate + retry it is for the hosted ones, thx

[Tom Jones (tjonesit)]

One addition to question 1, from tool-calling tests we ran this week, so treat it as adjacent evidence for extraction. When you track parse success per model, check whether the failures belong to the model or to the endpoint serving it.

Two cases from the same night:

• Two open-weight models on one provider looked like they were failing on 90 items each. Every one of those was an item where the right answer was to call no tool, and the model had answered correctly in text. That provider omits the tool_calls field entirely in that case, and our scorer read the missing field as a failure. A third model on the same items returned an empty list and scored fine.
• gpt-oss-120b, asked to retry after a correction, returned 31 tool calls as plain text in its own native format, which the provider did not parse into structured calls. Read by hand, 30 of the 31 were the correct call. On the first turn of the same conversations, the same provider parsed them normally.

Neither raised an error and both looked like model quality. So for your question 3, I would add a second axis: the same model served by two providers. Our guess is that some of the spread you expect comes from the serving layer's parser, and a per-chunk parse rate with no record of which endpoint produced the chunk would not tell those apart.