Skip to content
scrapyPublic

About

Fast HTML extraction for Python and Rust, without a DOM

Resources

Security policy

Stars

17 stars

Watchers

4 watching

Forks

Repository files navigation

Frostwork

A fast streaming HTML parser for Python and Rust: every selector is answered in one pass over the page.

Frostwork compiles a set of CSS or XPath selectors and extracts their values in one scan of an HTML response. Reuse the schema across Scrapy callbacks, web-poet pages or standalone code. Working memory tracks parser state, pending matches and returned values, not a document tree. Frostwork builds a minimal tree only when needed.

Frostwork supports a focused selector subset. By default it answers against the tree a browser builds, checked against Chrome; parser="lxml" answers against lxml's, checked for value parity with Parsel. Python rejects unsupported selectors before scanning; strict=False requests empty columns. There is no parser fallback. Check the compatibility contract before migrating code that needs arbitrary tree traversal.

The benchmarks compare throughput, memory and selector coverage with Parsel, lxml and other engines, including workloads where Frostwork loses. Performance depends on page shape, selector count and output volume.

Install

Frostwork requires Python ≥ 3.10. Install the core package, or include the web-poet integration:

pip install frostwork
pip install "frostwork[webpoet]"

Published wheels contain the Rust extension, so installing from PyPI does not need a Rust toolchain. Building an editable checkout does; see the Python development guide.

Extract a named item

Build a Page once, then reuse it for each response:

from frostwork import Page

PRODUCT = (Page()
           .field("name", "h1::text")              # first match, or None
           .field("price", ".price::text")
           .field_all("images", "img::attr(src)")) # every match, or []

html = b"<h1>Widget</h1><span class=price>$9</span><img src=/a.png>"
assert PRODUCT.extract(html).to_dict() == {
    "name": "Widget", "price": "$9", "images": ["/a.png"],
}

In an ordinary Scrapy callback, hand over the response itself. extract_response passes the original bytes with the Content-Type header and the URL, and Frostwork makes the encoding decision a browser would:

def parse_product(self, response):
    yield PRODUCT.extract_response(response).to_dict()

Use field_all whenever you need every match, and field_join to join text nodes. Do not pass response.encoding yourself. It is Scrapy's own decision, and it would override the page's declarations; see encoding inputs. For positional columns without field names, use the primitive extract API.

The same engine is a Rust library — frostwork::extract(html, &queries, None), with Page/Plan for named fields and compile-once reuse. See the runnable example.

Scrapy and web-poet

For a runnable Scrapy project, start with frostwork-demo. Its local storefront examples cover ordinary callbacks, ItemLoader, web-poet, nested products and browser rendering, with tests that run real crawls and compare exported items against Parsel. For existing spiders, follow the migration workflow to audit complete schemas, verify saved responses and monitor extracted items.

A page object declares its selectors; Frostwork fills every field from one scan of the response:

from frostwork.webpoet import FrostPage, field

class ProductPage(FrostPage):
    name = field("h1::text")
    price = field(".price::text")
    images = field("img::attr(src)", all=True)
    specs = field(".spec ::text", join=" ")
    brand = field("//meta[@itemprop='brand']/@content")

to_item() returns a dict here. Add Returns[YourItem] for a typed item, as shown in the Python guide.

For injection, install and configure scrapy-poet using its setup guide, or copy the working settings and explicitly assigned callbacks from the demo. Outside Scrapy, construct the page directly: item = await ProductPage(response=http_response).to_item().

Before migrating a spider

Run frostwork-audit --scan myproject/spiders/ to inspect selector literals without importing the spider. Once you have a schema, PRODUCT.check().raise_for_status() checks its selectors and shared budgets. Then compare complete items on saved responses before switching a callback.

The main compatibility boundaries are:

  • Parse mode: the default browser mode and parser="lxml" can build different trees from the same markup. A port that must reproduce Parsel's items exactly uses parser="lxml".
  • Selector context: grouped fields support fewer shapes than flat fields. Audit the actual schema.
  • HTML values: bare-element selectors return source HTML, which can differ from lxml serialization.
  • Encoding: decoding follows browser behavior, including documented differences from Parsel/w3lib.

The compatibility contract contains the exact limits and cases where Frostwork returns values Parsel cannot. The Python guide covers groups, processors, response adapters and runtime item validation.

Build, test, benchmark

make bootstrap     # create .venv and install the pinned Python test toolchain
make ci            # Rust, Python, lxml and Chrome differentials, encoding, and fuzz gates
make bench         # benchmark matrix against Parsel
make soak          # multi-seed differential and fuzz soak

make help lists the individual targets. TESTING.md explains what each gate proves and what remains outside it.

More

License

Apache-2.0. See LICENSE.

About

Fast HTML extraction for Python and Rust, without a DOM

Resources

Security policy

Stars

17 stars

Watchers

4 watching

Forks

Releases

Used by

Contributors

Languages