feat(crawl): add the Crawl client, exposed as client.crawl

[StefanoGuerrini]

What this adds

A client for the Crawl API (one start URL in, the URLs behind it out), as client.crawl on ZenRows and standalone as ZenRowsCrawlClient(apiKey, { baseURL?, retries? }) (default base https://api.zenrows.com/v1). Exported from the package entry.

Method	Call
create({ url, depth, maxItems?, maxPages?, includePatterns?, excludePatterns?, outputFormat?: "html" }, { idempotencyKey? })	POST /crawls, sends only the fields set
get(id, { cursor?, limit? })	GET /crawls/{id}
list({ cursor?, limit? }) / iterCrawls({ limit? })	GET /crawls, following next_cursor
iterResults(id, { limit? })	follows next_cursor until null; on a running crawl it yields what is kept so far and returns instead of spinning
stop(id)	POST /crawls/{id}/stop
getContent(id, contentIdOrResult) → string	GET /crawls/{id}/contents/{content_id}, taking the id from a result's content_url
download(id) → async iterator of parsed NDJSON lines	GET /crawls/{id}/download, streamed
wait(id, { timeout?, pollInterval?, maxPollInterval? })	polls get(id, { limit: 1 }) until the status is not running; starts at 2s, ×1.5, capped at 15s, 300s timeout; WaiterTimeoutError on timeout, the crawl keeps running

Inputs are camelCase and responses keep the API's snake_case. Response enums are Extensible<...>; unknown fields are tolerated. A failed crawl carries error.code and error.detail.

Errors throw ZenRowsCrawlError (status, code, problem, extras, plus retryAfter from Retry-After). A 403 REQS008 reads "403 Crawl is not enabled for this account (REQS008): …"; CRAWL_NOT_ENABLED_CODE is exported to branch on. Requests send X-API-Key; idempotent requests retry on 429/502/503/504 and network errors. A create without an idempotencyKey is not retried, so a 429 too_many_crawls (the account has too many crawls running) reaches the caller with retryAfter.

The shared HTTP transport gains an optional error-factory argument so the Crawl client throws its own error type; existing callers keep their default and their public API is unchanged.

Tests

• tests/crawl-client.test.ts (msw): request shape (path, method, X-API-Key, Idempotency-Key, body omits unset fields), parsing, unknown fields and enum values, paging stops on a null cursor, 404 / 422 / 429 / 403 REQS008, the waiter ending, resolving on failed, and timing out.
• tests/e2e/crawl.e2e.test.ts: a real crawl, skipped unless ZENROWS_API_KEY, ZENROWS_CRAWL_BASE_URL and ZENROWS_E2E_CRAWL_URL are set, so pnpm test -- --run stays hermetic. It bypasses the shared msw server. Its target comes from ZENROWS_E2E_CRAWL_URL (and an optional ZENROWS_E2E_CRAWL_INCLUDE pattern). It crawls at depth 1 (3 items, 5 pages, HTML), waits, checks the results match the include pattern, reads one page's HTML, checks the download's line count, finds the crawl in the list, stops the ended crawl, and expects crawl_not_found on an unknown id. On 429 too_many_crawls it waits retryAfter and retries, for up to 5 minutes. The README's Contributing section says how to run it (pnpm test:e2e).

pnpm check: clean. pnpm test -- --run: 25 files passed, 1 skipped (the e2e); 304 tests passed, 6 skipped; no type errors. pnpm build emits the new exports.

E2E evidence

Ran pnpm test:e2e against a live ZenRows API deployment, with the target set through ZENROWS_E2E_CRAWL_URL and ZENROWS_E2E_CRAWL_INCLUDE: 6 of 6 tests passed, no type errors.

🤖 Generated with Claude Code

[StefanoGuerrini]

Crawl Beta public-surface review

This review checks the public API against what production serves to a crawl-beta customer today: Crawl 475016a (docs/openapi/api.yaml) and gateway e382348. crawl-beta is at 100%. crawl-beta-extract (output_format: json, discovery: pagination) is staff-only. Reviewed at 54f85f1: pnpm build is clean, and pnpm test -- --run passes 304 tests with no type errors.

Verdict: changes needed. Items 1 to 5 become breaking changes if they ship as they are.

Fix before merge: breaking if shipped

1. No Beta marking. The README labels Extract and Batch "(beta)", but not Crawl. Add "Crawl (beta)" to the README section and @beta to each Crawl export.
2. download() drops X-Crawl-Status (src/crawl/client.ts:169). It returns a bare async generator, so a caller cannot tell a partial download (crawl still running) from a full one. Return an object that carries the status and the lines. Adding the status later changes the return type.
3. Crawl has no discovery and no extract_calls (src/crawl/types.ts:26). Production always returns discovery. Add discovery: Extensible<"links" | "pagination">[] and extract_calls?: number. Adding a required field later breaks every caller that builds a Crawl, such as test mocks.
4. CrawlExportLine.content is string (src/crawl/types.ts:78). Production sends an object for json crawls, and GET /crawls lists staff crawls in the same org. Type it string | Record<string, unknown> now; widening it later breaks callers.
5. code falls back to "internal" (src/crawl/errors.ts:38). Production uses internal_error, so callers can branch on a value the API never sends. Leave code undefined when the body has no code.

Fix before merge: behaviour and docs

6. A keyed create retries 429 too_many_crawls. The shared transport retries 429 up to 3 times and sleeps for each Retry-After. The README says the create throws with retryAfter. too_many_crawls is a capacity limit, so do not retry it on create. Keep the retries on 5xx and network errors.
7. wait() defaults to 300 s (src/crawl/client.ts:200), but MCP and CLI use 600 s. All surfaces will use one value; it is not decided yet.
8. CrawlErrorCode lacks domain_not_allowed and internal_error (src/crawl/types.ts:8).
9. The README's 429 sentence is not correct. The limit is the account's active jobs: 3 by default, shared with its Batch jobs. The error list also leaves out 400 unknown_parameter, 409 idempotency_request_in_flight and 422 idempotency_key_reused.

Matches production

• create() takes url and depth as required, outputFormat: "html" only, and no discovery. It sends only the fields the caller set.
• Response enums are open, unknown fields pass through, and iterCrawls() ends on an absent next_cursor.
• wait() returns a failed crawl and does not throw. A timeout leaves the crawl running.
• ErrorFactory, crawlError and contentIdOf stay out of the public types.
• The README says a crawl stays on the start URL's domain. Production enforces this by registrable domain, so subdomains count (urlx.InScope).

Can wait

These are additive and need no change now:

• client.crawl on ZenRows takes no baseURL or retries.
• getContent() has no download=true option.
• Errors do not expose the RateLimit headers.

Release: minor bump (2.2.0) in the release commit.

[StefanoGuerrini]

Replaced by #35: the review fixes, with a history that drops the AI co-author trailer. The review on this PR still applies.