Allow image snippet checking #63

Activity

1. github-actions

   added

   enhancementNew feature or requestNew feature or request

   on Sep 1, 2026
2. github-actions commented on Sep 1, 2026

   

   github-actionsbot

   on Sep 1, 2026 – with GitHub Actions

   Contributor

   More actions

   Some notes to help frame the design space for this, based on how the rest of LRV currently works (I didn't find any other open issue that overlaps this one — closed #12 "Create generic URL checking" is the closest precedent, since it establishes the pattern of validating a reference by URL rather than by fetched full text).

   Tension with the core design principle

   The README is explicit that LRV's value proposition is deterministic substring matching of supporting_text against fetched full text/abstract (src/linkml_reference_validator/validation/supporting_text_validator.py, with fuzzy-repair fallback in fuzzy_text_utils.py / repairer.py). Both routes floated in the issue trade away some of that determinism:

   • Third-party annotation service — as you note, this imports nondeterminism. It's arguably fine if it's scoped as a new, distinct capability rather than folded into the existing ValidationSeverity.ERROR semantics — e.g. surfaced only as INFO/WARNING with the annotation service's identity + a content hash recorded in the result, so a human can audit why it passed rather than trusting it blindly. This keeps LRV itself deterministic (same inputs → same LRV output) even though the upstream annotation isn't.
   • Embedding image data in the record (hex/base64) — I'd push back on this even harder than text: LinkML data files are typically YAML/JSON, and base64-blob-in-YAML is unpleasant to diff, review in a PR, or store in git at any scale. If image data has to travel with the record at all, base64 is not it.

   A third option that stays closer to LRV's existing model

   There's a middle path that preserves determinism and mirrors how text validation already works: instead of "quoting" a crop by embedding pixels, treat the crop as a claim that region R of source figure F is what's shown, and verify that claim deterministically:

   • Reference = (PMID/PMCID/DOI, figure_id_or_url) — for PMC-derived papers you likely already have a path to the figure list via the OA/figures endpoints, reusable from the existing etl/fulltext/pmc.py / epmc_preprint.py providers.
   • Excerpt = either a bounding box [x, y, w, h] (resolution-normalized) into that source figure, or a cropped image + checksum.
   • Validation = deterministic image containment check: template/feature matching (OpenCV matchTemplate or ORB/SIFT keypoint matching) to confirm the crop is genuinely a sub-region of the cited figure, with a perceptual hash (imagehash, pHash/dHash) to tolerate lossless re-encoding/recompression the way fuzzy_text_utils.py tolerates whitespace/unicode noise in text. No AI judgment call involved — it's the image analogue of "is this substring in that string," just with a similarity threshold instead of exact match.
   • This would need optional deps (opencv-python-headless, Pillow, imagehash) — pyproject.toml doesn't currently define an extras group, so this would be a good opportunity to introduce linkml-reference-validator[images] rather than making these hard dependencies for all users.

   Data model note

   Whatever the excerpt representation, a checksum field (algorithm-tagged, e.g. sha256:...) on the crop is worth having regardless of which validation strategy is used — it's cheap, it lets you detect "this crop changed since it was checked" independent of re-running image matching, and it composes with the URL+checksum approach you floated.

   Suggested scoping

   Given the open questions you've flagged, this reads like it needs a design decision before implementation (not yet at the "write tests first" stage per the repo's TDD convention). If it's helpful, a reasonable phase 1 would be just the deterministic bounding-box/checksum containment check against a PMC-sourced figure (no third-party service, no new severity semantics) — that's a self-contained, testable slice that doesn't foreclose adding the annotation-service route later as an opt-in, lower-trust check.
3. cmungall commented on Oct 11, 2026

   

   cmungall

   on Oct 11, 2026 – with Claude

   MemberAuthor

   More actions

   A concrete version of the "trusted third-party annotation" route, which keeps LRV itself exact:

   1. Outside LRV: an LLM (or a person) transcribes the raster figure into an SVG.
   2. Outside LRV: the SVG is checked against the original. This can be partly mechanical: render the SVG and compare it with the raster (overlay or difference image, structural similarity score, perceptual hash). An agent or person only reviews the cases where they differ.
   3. In LRV: supporting_text is validated against the SVG's rendered text, like any other local reference.

   The uncertain step (reading the figure) stays separate from the exact one (matching quotes). The SVG in between can be inspected, so a pass means "supported by this checked transcription of figure F".

   Two rules make this sound:

   • Only rendered text counts. An SVG can hold text that never appears on screen (<desc>, <metadata>, hidden or zero-opacity elements). If LRV matched against it, a transcription could slip in unsupported claims that pass any visual check.
   • Provenance is recorded. The cached reference should carry the source figure (PMID/PMCID + figure ID), a checksum of the raster, the transcription method or model, and the verification result. Then a report can't present "matched the SVG" as "matched the paper". This touches the cache-format question in The cache-file format has no owner: consumers must mirror the emitter, so every new frontmatter field is a downstream break #89.

   Relational claims ("A inhibits B") need edges encoded in the SVG, e.g. connectors with data-source/data-target. Those attributes are invisible, but they can be checked mechanically: the connector's path endpoints must fall on the named nodes.

   The LRV-side piece is tracked in #104 (SVG extractor). It is small, can be tested on its own, and is also useful for diagrams people draw by hand. The pixel-crop containment approach discussed above stays the option for figures that can't be transcribed (photos, micrographs, blots).

   ---

   Generated by Claude Code
4. caufieldjh

   added sub-issues on Oct 13, 2026

   • Record figure provenance in a YAML file beside each file: figure #112
   • Check that an image crop is a region of a cited figure #113