Cache url: HTML pages as readable text, so quotes through links and bold text match

docs/how-to/validate-urls.md

@@ -16,7 +16,7 @@ When a reference field contains a URL, the validator:
 
 1. Fetches the web page content
 2. Extracts the page title (see [Titles](#titles))
-3. Sanitizes HTML and caches the content for future validations
+3. Converts HTML to readable text and caches it for future validations
 4. Validates your supporting text against the page content
 
 ## URL Format
@@ -80,20 +80,29 @@ When the validator encounters a URL reference, it:
 The fetcher stores:
 
 - **Title**: See [Titles](#titles)
-- **Content**: Sanitized HTML for HTML pages; plain text and XML as received;
+- **Content**: Readable text for HTML pages; plain text and XML as received;
   extracted text for PDFs
 - **Content type**: `url` for pages, `full_text_pdf` for PDFs with text
 
-HTML is sanitized before it is cached, because caches are often committed to
-public repositories and page scripts routinely carry signed asset URLs and API
-keys. `<script>`, `<style>`, `<noscript>` and `<template>` elements and HTML
-comments are removed. So are `<meta>`, `<link>` and `<base>`, which hold nothing
-but attributes. Every other tag attribute is removed except `rowspan`,
-`colspan` and `scope`, which carry table structure. Body markup and text are
-kept. There is no HTML-to-text conversion, so tag names can still surface
-during normalization. If validation fails because tags interrupt the text,
-consider extracting plain text and validating against a local `file:`
-reference instead.
+HTML is converted to readable text before it is cached. A quote copied from
+the page then matches even when it runs through a link, bold or italic text,
+and a cached page can be read, and diffed, line by line:
+
+- Inline tags (`<a>`, `<b>`, `<em>`, `<span>`, ...) are removed and their
+  text is kept in place.
+- Block-level tags (`<p>`, `<li>`, `<h1>`-`<h6>`, `<div>`, `<br>`, ...) start
+  a new line.
+- A table row is one line, its cells separated by ` | `. Normalization drops
+  the `|`, so a quote copied from a rendered row with its cells separated by
+  spaces still matches.
+- `<pre>` keeps its whitespace. Elsewhere whitespace collapses to one space.
+- Entities such as `&amp;` are unescaped.
+
+The whole page is kept, including navigation, sidebars and footers; it is not
+narrowed to the article body. Because caches are often committed to public
+repositories and page scripts routinely carry signed asset URLs and API keys,
+`<script>`, `<style>`, `<noscript>` and `<template>` elements and HTML comments
+are removed with their text, and no tag attribute reaches the cache.
 
 #### Titles
 
@@ -176,11 +185,11 @@ URL validation is designed for static web pages. It may not work well with:
 
 ### Raw Content
 
-The validator stores sanitized page markup, not extracted text. For HTML pages:
+The validator stores the page's text, not its markup. For HTML pages:
 
-- HTML tags are preserved in the cache, without attributes other than
-  `rowspan`, `colspan` and `scope`
-- The text normalization during validation handles most cases
+- Text that only appears in an attribute, such as an image's `alt` text, is
+  not kept
+- Text inside `<noscript>` or `<template>` is not kept
 - Complex HTML layouts may require careful text extraction
 
 ### No Rendering

docs/troubleshooting.md

@@ -902,11 +902,14 @@ rather than as `unavailable`, so it falls outside this stamp and is not
 re-tested. Clear such entries by hand if you were running that setting before
 this version.
 
-`url:` entries carry their own stamp, `url_source_version: 1`. Version 1 is
+`url:` entries carry their own stamp, `url_source_version`. Version 1 is
 two URLSource changes: HTML is sanitized before caching, so page scripts,
 comments and attributes stop reaching the cache (issue #92), and a PDF's title
 is recovered from a publisher landing page or its embedded metadata rather than
-set to its URL (issue #93). A `url:` entry with a missing or older stamp is
+set to its URL (issue #93). Version 2 caches an HTML page as readable text
+rather than markup, so a quote that runs through a link or bold text matches
+(issue #102). Every quote that matched the version 1 markup still matches the
+text, provided the page itself has not changed. A `url:` entry with a missing or older stamp is
 re-fetched once on the next validation fetch. If the page cannot be reached,
 the old entry is still served and is not rewritten, so a later run retries.
 Other sources are unaffected. With `trust_cached_entries` set, old entries are
@@ -920,6 +923,18 @@ stop matching once the entry is re-fetched. If the same text appears elsewhere
 on the page, nothing changes. If it does not, cite a source that carries it,
 such as the `PMID:` or `DOI:` of the article.
 
+**The version 2 refresh rewrites every cached HTML page.** Each `url:` HTML
+entry changes from one line of markup to many lines of text, so expect a large
+diff in a committed cache. The refresh fetches each page again, so a page that
+has changed or gone since it was cached can stop supporting a quote; an
+unreachable page keeps its old entry, as above. To keep that diff reviewable,
+refresh in a commit of its own: right after upgrading, run your usual
+validation on an otherwise unchanged checkout and commit only the rewritten
+cache files, before any curation changes. Entries are refreshed only as a
+validation uses them (see
+[Cached references re-fetched after upgrading](#cached-references-re-fetched-after-upgrading)),
+so a page no validation cites keeps its old entry until one does.
+
 **A cache-wide refresh adds one line to every enriched entry.** Both index
 providers now set `access_type` where they previously left it unset, so a
 refreshed public entry gains a `full_text_access_type: open` line. It means the

src/linkml_reference_validator/etl/extract/html.py

@@ -10,7 +10,14 @@
 import re
 from typing import Optional, Union
 
-from bs4 import BeautifulSoup, Comment, Tag  # type: ignore
+from bs4 import (  # type: ignore
+    BeautifulSoup,
+    CData,
+    Comment,
+    NavigableString,
+    Tag,
+)
+from bs4.element import PreformattedString  # type: ignore
 
 from linkml_reference_validator.etl.extract.base import Extractor, ExtractorRegistry
 from linkml_reference_validator.etl.rules import ARTICLE_BODY_SELECTORS
@@ -43,12 +50,46 @@
 KEPT_ATTRIBUTES = frozenset({"rowspan", "colspan", "scope"})
 
 
+#: Tags that start a new line in :func:`html_to_text`: the extractor's block
+#: tags, plus the document-level and form containers a whole page also has.
+#: ``td`` and ``th`` stay in it although a row's cells are joined on one line
+#: by :func:`_row_cells`: that path is taken only inside a ``<tr>``, so a cell
+#: met anywhere else still gets a line of its own.
+TEXT_BREAK_TAGS = frozenset(BLOCK_LEVEL_TAGS) | {
+    "body", "caption", "center", "details", "dialog", "fieldset", "form",
+    "head", "html", "legend", "menu", "option", "summary", "tbody", "textarea",
+    "tfoot", "thead", "title",
+}
+
+#: Between the cells of a table row. Normalization drops punctuation, so a
+#: quote copied from a rendered row, with the cells separated by spaces, still
+#: matches. Empty cells are left out, so a row's columns need not line up with
+#: the row above; this is text for matching and reading, not a table.
+CELL_SEPARATOR = " | "
+
+
+#: Marks, on the walk's stack, the end of a block-level element.
+_BLOCK_END = object()
+
+
+def _without_non_content(content: str) -> BeautifulSoup:
+    """Parse a page, dropping ``NON_CONTENT_TAGS`` and comments."""
+    soup = BeautifulSoup(content, "html.parser")
+    for tag in soup.find_all(NON_CONTENT_TAGS):
+        if not tag.decomposed:  # already gone with an enclosing non-content tag
+            tag.decompose()
+    for comment in soup.find_all(string=lambda text: isinstance(text, Comment)):
+        comment.extract()
+    return soup
+
+
 def sanitize_html(content: str) -> str:
     """Strip an HTML page to its body markup and text, for caching.
 
     Drops ``NON_CONTENT_TAGS`` and comments, and every attribute not in
-    ``KEPT_ATTRIBUTES``. Markup is kept, unlike :class:`HTMLExtractor`, which
-    flattens to plain text; this is for ``url:`` pages stored as HTML.
+    ``KEPT_ATTRIBUTES``. Markup is kept, unlike :class:`HTMLExtractor` and
+    :func:`html_to_text`, which flatten to plain text. ``url:`` pages are
+    cached through :func:`html_to_text` since #102.
 
     Examples:
         >>> sanitize_html('<p class="x" onclick="f()">Hi<script>k=1</script><!-- c --></p>')
@@ -60,12 +101,7 @@ def sanitize_html(content: str) -> str:
         >>> sanitize_html("<pre><b>x</b>    <b>y</b></pre>")
         '<pre><b>x</b>    <b>y</b></pre>'
     """
-    soup = BeautifulSoup(content, "html.parser")
-    for tag in soup.find_all(NON_CONTENT_TAGS):
-        if not tag.decomposed:  # already gone with an enclosing non-content tag
-            tag.decompose()
-    for comment in soup.find_all(string=lambda text: isinstance(text, Comment)):
-        comment.extract()
+    soup = _without_non_content(content)
     for tag in soup.find_all(True):
         tag.attrs = {k: v for k, v in tag.attrs.items() if k in KEPT_ATTRIBUTES}
     # Removed elements leave runs of indentation behind. Collapse each
@@ -77,6 +113,146 @@ def sanitize_html(content: str) -> str:
         text.replace_with("\n" if "\n" in text else " ")
     return str(soup)
 
+
+def html_to_text(content: str) -> str:
+    r"""Flatten a whole HTML page to readable text, for caching ``url:`` pages.
+
+    Drops what :func:`sanitize_html` drops, then unwraps every inline tag in
+    place and starts a new line at each block-level tag and ``<br>``. A table
+    row becomes one line, its cells joined by ``CELL_SEPARATOR``. ``<pre>``
+    keeps its whitespace; elsewhere whitespace runs collapse to one space, as
+    a browser would render them. Entities are unescaped.
+
+    Unlike :class:`HTMLExtractor`, which serves article text, nothing is
+    narrowed to ``<article>`` or ``<main>`` and no ``<p>``-only shortcut is
+    taken. Every text node of the sanitized markup appears, contiguous, in
+    the result, so a quote that matched the markup still matches, and a quote
+    through a link or bold text now matches too.
+
+    Examples:
+        >>> html_to_text('<p>Produced by <a href="/x">the City of Flint</a>.</p>')
+        'Produced by the City of Flint.'
+        >>> html_to_text("<p>The <b>Flint water crisis</b> was</p><ul><li>one</li><li>two</li></ul>")
+        'The Flint water crisis was\none\ntwo'
+        >>> html_to_text("<p>Line one<br>Line two</p>")
+        'Line one\nLine two'
+        >>> html_to_text("<table><tr><th>Gene</th><th>Finding</th></tr><tr><td>POLG</td><td>A &amp; B</td></tr></table>")
+        'Gene | Finding\nPOLG | A & B'
+        >>> html_to_text("<pre>x    y\n  z</pre>")
+        'x    y\n  z'
+        >>> html_to_text("<p>near (<i>GUSB</i>, <i>GRN</i>)\n   here</p><script>k=1</script>")
+        'near (GUSB, GRN) here'
+    """
+    return "\n".join(_lines(_without_non_content(content)))
+
+
+def _lines(node: Tag) -> list[str]:
+    """The lines of ``node``'s text, the last one included.
+
+    Lines outside ``<pre>`` are never blank and carry no leading or trailing
+    whitespace. A ``<pre>`` block's lines are kept as written, blank ones and
+    indentation included, since its whitespace is part of its text.
+
+    Walks the tree with a stack of its own rather than by recursion, so a page
+    nested deeper than Python's recursion limit is flattened like any other.
+    Only a table nested in a table cell recurses, through :func:`_row_cells`.
+    """
+    lines: list[str] = []
+    line: list[str] = []
+    # Children are pushed in reverse so they pop in document order. A block's
+    # closing break is pushed beneath its children, to end its last line.
+    stack: list[object] = list(reversed(list(node.children)))
+    while stack:
+        child = stack.pop()
+        if child is _BLOCK_END:
+            _end_line(lines, line)
+        elif isinstance(child, PreformattedString) and not isinstance(child, CData):
+            continue  # a doctype, declaration or processing instruction: not text
+        elif isinstance(child, NavigableString):
+            line.append(re.sub(r"\s+", " ", str(child)))
+        elif not isinstance(child, Tag):
+            continue
+        elif child.name == "br":
+            _end_line(lines, line)
+        elif child.name == "pre":
+            _end_line(lines, line)
+            lines.extend(_pre_text(child).strip("\n").splitlines())
+        elif child.name == "tr":
+            _end_line(lines, line)
+            row = CELL_SEPARATOR.join(cell for cell in _row_cells(child) if cell)
+            if row:
+                lines.append(row)
+        elif child.name in TEXT_BREAK_TAGS:
+            _end_line(lines, line)
+            stack.append(_BLOCK_END)
+            stack.extend(reversed(list(child.children)))
+        else:
+            stack.extend(reversed(list(child.children)))
+    _end_line(lines, line)
+    return lines
+
+
+def _end_line(lines: list[str], line: list[str]) -> None:
+    """Move the line being built onto ``lines``, unless it holds only whitespace."""
+    text = re.sub(r" +", " ", "".join(line)).strip()
+    if text:
+        lines.append(text)
+    line.clear()
+
+
+def _pre_text(pre: Tag) -> str:
+    r"""A ``<pre>`` block's text, whitespace kept and each ``<br>`` a line break.
+
+    ``get_text()`` would ignore ``<br>`` and weld the words either side.
+
+    Examples:
+        >>> _pre_text(BeautifulSoup("<pre>a<br>b  <i>c</i></pre>", "html.parser").pre)
+        'a\nb  c'
+    """
+    return "".join(
+        "\n" if isinstance(node, Tag) else str(node)
+        for node in pre.descendants
+        if (isinstance(node, Tag) and node.name == "br")
+        or type(node) is NavigableString
+        or isinstance(node, CData)
+    )
+
+
+def _row_cells(row: Tag) -> list[str]:
+    """The text of each cell in a table row, and of anything else it holds.
+
+    A cell may sit inside a wrapper, as in the ``<tr><form><td>`` of older
+    layouts, and a row may hold stray text. Neither is dropped: the sanitized
+    markup kept that text, so a quote may already match it.
+
+    Examples:
+        >>> row = BeautifulSoup("<tr>x<form><td>a</td><td>b</td></form></tr>", "html.parser")
+        >>> _row_cells(row.tr)
+        ['x', 'a', 'b']
+    """
+    cells: list[str] = []
+    for child in row.children:
+        if isinstance(child, Tag) and child.name in ("td", "th"):
+            cells.append(_cell_text(child))
+        elif isinstance(child, Tag) and child.find(["td", "th"]) is not None:
+            cells.extend(_row_cells(child))
+        elif isinstance(child, Tag):
+            cells.append(_cell_text(child))
+        elif type(child) is NavigableString or isinstance(child, CData):
+            cells.append(" ".join(child.split()))
+    return cells
+
+
+def _cell_text(cell: Tag) -> str:
+    """One table cell's text on one line, whatever blocks it holds.
+
+    Examples:
+        >>> _cell_text(BeautifulSoup("<td><p>a</p><p>b <i>c</i></p></td>", "html.parser").td)
+        'a b c'
+    """
+    return " ".join(_lines(cell))
+
+
 @ExtractorRegistry.register
 class HTMLExtractor(Extractor):
     """Extract readable text from HTML markup, as bytes or as a string.

src/linkml_reference_validator/etl/reference_fetcher.py

@@ -132,8 +132,10 @@
 #: URL (#93). Neither fix rewrites an entry already written, and both kinds of
 #: entry carry a current ``extractor_version``, so without this they would be
 #: served as they are forever. Stamped from ``reference.metadata``, like the
-#: HTML stamp, and only on a fresh URLSource fetch.
-URL_SOURCE_CACHE_VERSION = 1
+#: HTML stamp, and only on a fresh URLSource fetch. Version 2: HTML is cached as
+#: readable text rather than markup, so a quote through a link or bold text
+#: matches (#102).
+URL_SOURCE_CACHE_VERSION = 2
 
 _FORMAT_TO_CONTENT_TYPE = {
     "pdf": "full_text_pdf",

src/linkml_reference_validator/etl/sources/url.py

@@ -20,7 +20,7 @@
 from linkml_reference_validator.models import ReferenceContent, ReferenceValidationConfig
 from linkml_reference_validator.etl.sources.base import ReferenceSource, ReferenceSourceRegistry
 from linkml_reference_validator.etl.acquire import ContentAcquirer, sniff_format
-from linkml_reference_validator.etl.extract.html import sanitize_html
+from linkml_reference_validator.etl.extract.html import html_to_text
 from linkml_reference_validator.etl.extract.pdf import PDFExtractor
 from linkml_reference_validator.etl.rules import LANDING_PAGE_RULES
 
@@ -37,10 +37,10 @@
 class URLSource(ReferenceSource):
     """Fetch reference content from web URLs.
 
-    Fetches HTML and plain text content. HTML keeps its markup but is
-    sanitized (see :func:`sanitize_html`) so page scripts and attributes do not
-    reach the cache. Plain text and XML are stored as fetched. Content is cached
-    to disk like other sources.
+    Fetches HTML and plain text content. HTML is flattened to readable text
+    (see :func:`html_to_text`), so page scripts and attributes do not reach the
+    cache and a quote can run through a link or bold text. Plain text and XML
+    are stored as fetched. Content is cached to disk like other sources.
 
     Examples:
         >>> source = URLSource()
@@ -109,10 +109,10 @@ def fetch(
             )
 
         content = self._decode(data, content_type_header)
-        # Before sanitizing: citation_title lives in a <meta> attribute.
+        # Before flattening: citation_title lives in a <meta> attribute.
         title = self._extract_title(content, url)
         if "html" in content_type_header or self._looks_like_html(data):
-            content = sanitize_html(content)
+            content = html_to_text(content)
 
         return ReferenceContent(
             reference_id=f"url:{url}",