Repository navigation
Extract rendered text from SVG (and drawio) so diagrams can be used as references #104
Description
Summary
Allow a local file: reference to point at an .svg (later .drawio) and validate supporting_text against the text the diagram actually renders.
Current behaviour
FileSource reads files as raw text, and acquire.py has no format entry for SVG or drawio. Quotes are therefore matched against raw XML after normalization, which:
- fails on labels split across
<tspan>elements (a quote that crosses a line break becomesfoo tspan x 10 y 24 bar); - matches against attribute and style noise (
fill,id,transform); - for drawio, often sees nothing at all, because the
<diagram>payload is deflated and base64-encoded.
Proposal
Add an SVGExtractor to etl/extract/, following the HTML, PDF and XML extractors:
- Emit one line per
<text>element, joining its<tspan>children. Also read HTML inside<foreignObject>, which is where drawio's SVG export puts labels. - Use only text that is rendered. Skip
<desc>,<title>,<metadata>, comments,<defs>/<symbol>content not referenced by<use>, elements withdisplay:noneorvisibility:hidden, and elements with zero opacity. This is an integrity rule, not just a cleanup step: a quote must not be able to pass against text a reader of the figure cannot see. - Optionally emit edge lines (
<source label> <edge label> <target label>) when connectors follow a simple convention, e.g.data-source="node-id"anddata-target="node-id", with the edge label as the connector's text. This makes simple relational claims ("PTEN inhibits AKT") matchable. Edge lines should only be emitted when the connector's path endpoints actually fall on the referenced nodes' bounding boxes. Otherwise the attributes become a hidden channel again. - Route it: add
.svgandimage/svg+xmlto the format maps inacquire.py, and haveFileSourcepass recognised formats through their extractor instead of returning raw text.
drawio support (decompress, walk mxCell, strip HTML from value, edges from source/target) can be part of this issue or a follow-up.
Out of scope
- Text converted to outlines/paths (would need OCR).
- Inferring edges from the geometry of arbitrary SVGs with no convention.
- Producing the SVG; see Allow image snippet checking #63 for the raster → SVG pipeline this enables.
Tests (written first)
- Multi-
<tspan>label is quotable across the line break. - Text inside
<desc>,<metadata>, hidden or zero-opacity elements is not quotable. - Attribute values (
id,fill) are not quotable. - drawio-exported SVG with
<foreignObject>labels is quotable. - With the edge convention: a valid connector yields a quotable edge line; a connector whose endpoints don't touch the named nodes yields none.
Activity
github-actions commentedon Oct 11, 2026
Summary and codebase notes
Ask: add an
SVGExtractor(drawio later) so afile:reference to a diagram is matched against the text it renders, not against raw XML. Only rendered text counts, so a quote can't pass against hidden text. Edge lines (A <label> B) are optional and are emitted only when the connector geometry confirms thedata-source/data-targetattributes.Related:
What I found in the code (checked against
main):etl/extract/hasbase.py(ExtractorABC withformats()andextract(data, content_type=...), plusExtractorRegistry),html.py,xml.pyandpdf.py. Extractors register via the@ExtractorRegistry.registerdecorator and are imported inextract/__init__.py. An SVG extractor is a small addition.etl/acquire.pyhas_CONTENT_TYPE_FORMATSand_SUFFIX_FORMATS..svgandimage/svg+xmlwould map to"svg"there.sniff_formatchecks magic bytes, so<svgand<?xml … <svgneed handling. An XML-declared SVG would otherwise sniff asxmland go to the JATS extractor..drawioand.drawio.xmlalso begin with<mxfile.FileSource.fetchinetl/sources/file.pycurrently doesread_textand returns raw content. Its docstring says "content is read as-is without parsing". Routing known formats through extractors changes that for.htmlfiles too. Decide whether to apply the new routing to SVG only or to every recognised format. Applying it to all formats would change existing behaviour for.htmlreferences.etl/reference_fetcher.pyhas per-format cache stamps (EXTRACTOR_CACHE_VERSION,XML_EXTRACTION_CACHE_VERSION, and so on). A new extractor starts with no cached entries, so it needs no bump. Any later change to its output that makes cached text wrong should get its own stamp, following the convention documented inextract/__init__.py.Design suggestions
defusedxmlorlxmlwithresolve_entities=Falseandno_network=Trueavoids entity-expansion (billion laughs) and XXE on untrusted SVGs. Don't use BeautifulSoup'shtml.parserfor SVG, which lowercasesforeignObjectand mishandles namespaces.xml.pyalready uses bs4, so check whether itslxml-xmlmode is enough.<g display="none">), and thestyle=attribute, the presentation attribute and CSS<style>blocks can all set it. Reject or warn on<style>rules that target text, or implement a minimal class-selector pass. Silently ignoring CSS is a hole in the integrity rule.opacity,fill-opacityandfill="none"orfill="transparent"on text. Also consider text coloured the same as an opaque background, or placed outside theviewBoxor clipped byclip-path. Full rendering fidelity is out of scope. I'd suggest handling the cheap, explicit cases and documenting the rest as known limits. If "a quote can't pass against invisible text" is the stated guarantee, say which cases are not covered.<switch>/systemLanguagealternatives,<tspan display="none">, and<a>wrappers.<use>references: resolvehrefandxlink:href, and guard against reference cycles.font-size.<text>works. For a<tspan>withx/yordyit is a visual line break, so join with a space and let the existing normalizer collapse whitespace. ForforeignObjectHTML, reuseHTMLExtractorlogic for the fragment.M/Lpoints), applytransformwhere present, and test against each node's bounding box with a small tolerance. For<path>curves, only the endpoints are needed. Phasing it as a follow-up PR would keep the first PR small. Phase 1 is the text extractor plus routing. Phase 2 is edges. Phase 3 is drawio.<diagram>payload is base64 → raw deflate → URL-encoded XML. That is straightforward withzlib.decompress(data, -15). Newer exports may be uncompressed. Strip HTML fromvaluewhenhtml=1, and skip cells withvisible="0". Hidden layers are a drawio-specific version of the same integrity problem.source_figure, raster checksum and transcription method somewhere. That belongs to the cache-format discussion in The cache-file format has no owner: consumers must mirror the emitter, so every new frontmatter field is a downstream break #89 and is not needed for this PR. A cheap hook is putting the SVG's own sha256 inReferenceContent.metadataso reports can say which file version passed.Tests. The five listed in the issue are good. I'd add: visibility inherited from an ancestor
<g>, a<style>-driven hide, a<use>cycle, and an XXE payload that must not resolve.Status: it needs no design decision to start, and it is TDD-ready as written.
If you want an implementation, here is a starting prompt:
@dragon-ai-agent please implement phase 1 of #104: add SVGExtractor in etl/extract/svg.py (secure XML parser; one line per <text>, tspans joined, foreignObject HTML via HTMLExtractor; skip desc/title/metadata/comments/unreferenced defs, display:none, visibility:hidden, zero opacity, including inherited and style-attribute cases), register it, add .svg / image/svg+xml to the format maps in etl/acquire.py with sniffing for <svg, route FileSource through extractors for SVG, write tests first including the cases in the issue, and run just test. Leave edges and drawio for follow-ups.I haven't run any of this or verified how the existing extractors behave beyond reading their structure.
Add SVGExtractor for the text a figure renders (#104)
Recognise SVG by suffix, content type and leading bytes (#104)
Read file: SVG references through SVGExtractor (#104)
Re-read file: SVG entries cached as raw markup (#104)
Document SVG figures as file: references (#104)
Never serve a stale raw-markup SVG entry as evidence (#104)
Merge pull request #105 from linkml/ccr-8db10bbe-nmwf03
Read native .drawio files as the labels they draw (#104)
Merge pull request #107 from linkml/ccr-8db10bbe-nmwf03
caufieldjh commentedon Oct 13, 2026
Everything proposed here is on
main:etl/extract/svg.py): one line per<text>with tspans joined,<foreignObject>HTML read, and only rendered text counted. Stylesheets are evaluated, and anything that can't be evaluated is reportedunavailableinstead of guessed at. Starts at fd0f40d..svg,image/svg+xmland leading-byte sniffing inacquire.py(50a20a1).FileSourcepasses figures through their extractor (ea9af1f). Other formats,.htmlincluded, are unchanged.svg_geometry.py).The five tests listed in the issue are all in
tests/test_svg_extractor.pyandtests/test_svg_edges.py. The extra cases from the triage comment (inherited hiding, stylesheet hiding,<use>cycles, XXE, entity expansion) are covered too. Known limits, including text converted to outlines, are indocs/how-to/use-local-files-and-urls.md. #111 fixes the last two places in the docs that named SVG without drawio.Provenance (source figure, raster checksum) stays with #63 / #89.