url: pages: a quote that runs through a link or bold text fails; cache readable text instead of markup #102

Activity

1. github-actions

   added

   bugSomething isn't workingSomething isn't working

   enhancementNew feature or requestNew feature or request

   on Oct 10, 2026
2. github-actions commented on Oct 10, 2026

   

   github-actionsbot

   on Oct 10, 2026 – with GitHub Actions

   Contributor

   More actions

   Summary

   Since #92, URLSource caches sanitized body markup (sanitize_html in etl/extract/html.py). A quote that spans inline markup (<a>, <b>, <em>) fails against that markup. The proposal is to cache readable text instead, and bump URL_SOURCE_CACHE_VERSION.

   I read the current code on main. I haven't run the repro URL.

   Related issues

   • Create generic URL checking #12 (closed): the original generic URL source. It asked for "markdown-converted contents", so flattened text was the original intent.
   • URLSource caches raw page HTML: scripts, comments and attributes reach a committed cache #92 (closed): introduced sanitize_html. This issue is its follow-up. It was deliberately "markup kept, not flattened", so this reverses that decision. The reason to do it is that the cache is read by people and diffs, and matched by quotes.
   • A PDF URL cached by URLSource gets the URL as its title #93 (closed): the title recovery in URLSource.fetch reads citation_title from <meta> before sanitizing. Flattening must stay after _extract_title, as it is now.
   • PDF extraction writes NUL bytes for unmapped glyphs (usually fi/fl ligatures) #100: another cache-fidelity problem, where PDF NUL bytes break matching. Together with this one it shows the cache text needs to be what a quote is matched against.
   • Landing-page fallback is stamped full_text_html: a cache entry can claim full text while holding only the abstract #65: same theme. The cached body has to be real article text, because a quote validates against anything in it.

   Check on "HTMLExtractor may serve as is"

   It probably can't, as written. In HTMLExtractor._extract_scope (etl/extract/html.py, around lines 240–290):

   1. <p>-only branch. If the page has any <p>, only paragraph text is returned. Headings, <li> items outside a <p>, table cells and <pre> are dropped. The issue asks for list items and table rows to be kept, so a page with a few <p> tags would silently lose text that matches today.
   2. extract() narrows to <article> or <main>. For a generic url: page, text that matches today in a sidebar or footer would stop matching. This is arguably a feature (Landing-page fallback is stamped full_text_html: a cache entry can claim full text while holding only the abstract #65), but it is a behaviour change that isn't mentioned.
   3. No table-row handling. The issue wants one line per row with separated cells. The extractor only inserts a "\n" after each block-level tag, so <td> cells would be on separate lines, not joined on one row.
   4. <pre> whitespace is not distinguished, though sanitize_html handles it.

   Probably the cleanest option is a new whole-document mode or function, e.g. html_to_text(content). It would run after the existing sanitize step, or fold into it. It would keep the "block tags get a break, inline tags are unwrapped" logic and skip the <p>-only and <article> shortcuts. It would also add the row-per-line rule, and it should be idempotent like sanitize_html. Getting this right matters because the invariant "every quote that matches today still matches" holds only if no text node is dropped.

   Alternative: match against extracted text

   The issue mentions matching quotes against extracted text while keeping markup in the cache. I'd stay with the primary proposal. Matching against derived text means the validator and the cache file can disagree about what was matched, which is a bad property for an audit trail. A cache that reads like the page is also the simpler design.

   Suggested tests, TDD per CLAUDE.md

   • Doctests for the new function covering <a>, <b>, <br>, <li>, a table row, entities, and <pre>.
   • A regression test with the Flint sentence: "This press release was produced by <a>the City of Flint</a>."
   • A property check on fixtures: any quote that matches the old sanitized HTML text-node-wise also matches the flattened text.
   • A test that a page with both <p> and <li> or <td> text keeps all of it.
   • A test that old cache entries (url_source_version = 1) are re-fetched after the bump.
   • A test that title still comes from <meta>.

   Open questions

   • Should table cells be separated by |, a tab, or a space? This affects quotes copied from a rendered table, since normalization collapses whitespace but not pipes.
   • Do cache files from earlier versions need a migration note? Bumping the version re-fetches them, which needs network access and can fail on pages that have since changed.

   I can't tell from the repo whether mechmaker#82 or the DaTMech JSON-source workaround have changed, so I haven't relied on them.

   If you want a PR for this:

   @dragon-ai-agent please implement #102: add a whole-document HTML-to-text function (not reusing HTMLExtractor's <p>-only/<article> shortcuts), use it in URLSource.fetch after title extraction, bump URL_SOURCE_CACHE_VERSION to 2, and add doctests and tests per the list in this comment.
3. caufieldjh

   closed this as completedin #103on Oct 12, 2026