feat(crawl): add the Crawl client, exposed as client.crawl

README.md

@@ -19,10 +19,12 @@ SDK to access [Zenrows](https://www.zenrows.com/) API directly from Node.js. Zen
   - [Batch](#batch)
     - [Failed runs and extensible values](#failed-runs-and-extensible-values)
     - [Extract in a batch](#extract-in-a-batch)
+  - [Crawl](#crawl)
   - [Concurrency](#concurrency)
     - [An important note about Promise.allSettled() on TypeScript](#an-important-note-about-promiseallsettled-on-typescript)
 - [Examples](#examples)
 - [Contributing](#contributing)
+  - [Running the Crawl e2e test](#running-the-crawl-e2e-test)
 - [License](#license)
 
 ## Installation
@@ -265,6 +267,63 @@ const { ZenRowsBatchClient } = require("zenrows");
 const batch = new ZenRowsBatchClient(apiKey, { baseURL: "https://async.api.zenrows.com/v1" }); // baseURL is optional
 ```
 
+### Crawl
+
+[Crawl](https://docs.zenrows.com) takes one start URL and returns the URLs behind it, and is reachable via `client.crawl`. It follows the links on each page up to `depth` hops, staying on the start URL's domain. A crawl runs asynchronously: `create()` returns at once, `wait()` polls until it ends, and `iterResults()` reads the URLs it kept. With `outputFormat: "html"` it also fetches each kept page, which you read with `getContent()` or all at once with `download()`.
+
+```javascript
+const { ZenRows } = require("zenrows");
+
+const apiKey = "YOUR-API-KEY";
+
+(async () => {
+  const client = new ZenRows(apiKey);
+
+  const crawl = await client.crawl.create({
+    url: "https://example.com/products/",
+    depth: 1, // link hops from the start URL
+    maxItems: 10, // stop once this many URLs are kept
+    maxPages: 20, // stop once this many pages are fetched; bounds the cost
+    includePatterns: ["/product/"], // keep only URLs containing one of these
+    outputFormat: "html", // omit for URLs only
+  });
+
+  const ended = await client.crawl.wait(crawl.crawl_id); // completed, stopped or failed
+  if (ended.status === "failed") throw new Error(ended.error?.detail);
+
+  for await (const result of client.crawl.iterResults(crawl.crawl_id)) {
+    console.log(result.url);
+    if (result.content_status === "fetched") {
+      const html = await client.crawl.getContent(crawl.crawl_id, result);
+    }
+  }
+
+  // Or every result, with its page, as parsed NDJSON lines.
+  for await (const line of client.crawl.download(crawl.crawl_id)) {
+    console.log(line.url, line.content?.length);
+  }
+})();
+```
+
+`client.crawl` also exposes `get()` (status, coverage and one page of results), `list()` and `iterCrawls()` (the account's crawls, newest first), and `stop()` (idempotent; a crawl that already ended answers as it ended). `create()` takes an `idempotencyKey` option, so a retried create does not start a second crawl. Read results after `wait()`: while a crawl runs, `iterResults()` yields only what it has kept so far. `wait()` throws `WaiterTimeoutError` after `timeout` seconds (default 300) and leaves the crawl running.
+
+Errors throw a `ZenRowsCrawlError` with `status`, `code` and `problem`. Branch on `status` and `code`:
+
+- `403` `REQS008`: Crawl is not enabled for this account.
+- `404` `crawl_not_found` / `content_not_found`.
+- `422` `invalid_parameter` / `invalid_start_url`: `problem.detail` names the field.
+- `429` `too_many_crawls`: the account has too many crawls running. Nothing was created; retry after `error.retryAfter` seconds.
+
+`outputFormat` takes `"html"`; without it a crawl returns URLs only. Response enum fields (`status`, `stop_reason`, `error.code`, `content_status`) are `Extensible<...>`: the known values plus any `string`. A failed crawl carries `error.code` and `error.detail`.
+
+The crawl client (`ZenRowsCrawlClient`) also works standalone:
+
+```javascript
+const { ZenRowsCrawlClient } = require("zenrows");
+
+const crawl = new ZenRowsCrawlClient(apiKey, { baseURL: "https://api.zenrows.com/v1" }); // baseURL is optional
+```
+
 ### Concurrency
 
 To limit the concurrency, it uses [fastq](https://github.com/mcollina/fastq), which will simultaneously send a maximum of requests. The concurrency is determined by the plan you are in, so take a look at the [pricing](https://www.zenrows.com/pricing) and set it accordingly. Take into account that each client instance will have its own limit, meaning that two different scripts will not share it, and 429 (Too Many Requests) errors might arise.
@@ -340,6 +399,27 @@ npx tsx index.ts # TS example
 
 Pull requests are welcome. For significant changes, please open an issue first to discuss what you would like to change.
 
+Before opening a pull request, run `pnpm check` and `pnpm test -- --run`. Neither touches the network.
+
+### Running the Crawl e2e test
+
+`tests/e2e/crawl.e2e.test.ts` runs a small real crawl (depth 1, up to 3 items and 5 pages, HTML output) and checks every Crawl method against the live API. It is skipped unless the first three variables are set:
+
+- `ZENROWS_API_KEY`: a key with Crawl access. Keep it out of your shell history, e.g. read it from a file.
+- `ZENROWS_CRAWL_BASE_URL`: the API base, e.g. `https://api.zenrows.com/v1`. Point it at a local or staging deployment to test that instead.
+- `ZENROWS_E2E_CRAWL_URL`: the start URL to crawl, a site you may crawl whose start page links to at least one page.
+- `ZENROWS_E2E_CRAWL_INCLUDE` (optional): an include pattern, e.g. `/product/`. When set, the crawl keeps only matching URLs and the test asserts every result contains it.
+
+```bash
+export ZENROWS_API_KEY="$(cat path/to/key-file)"
+export ZENROWS_CRAWL_BASE_URL=https://api.zenrows.com/v1
+export ZENROWS_E2E_CRAWL_URL=https://example.com/products/
+export ZENROWS_E2E_CRAWL_INCLUDE=/product/
+pnpm test:e2e
+```
+
+The run creates one crawl on the account and uses up to a few pages of credits. If the account has too many crawls running, the test waits and retries (up to 5 minutes).
+
 ## License
 
 [MIT](./LICENSE)

package.json

@@ -19,6 +19,7 @@
   "scripts": {
     "build": "tsup src/index.ts --dts --format cjs,esm --clean",
     "test": "vitest",
+    "test:e2e": "vitest run tests/e2e",
     "format": "biome format --write .",
     "check": "biome check ."
   },

src/batch/transport.ts

@@ -1,5 +1,6 @@
 import packageJson from "../../package.json" with { type: "json" };
 import { ZenRowsBatchError, parseProblem } from "./errors.js";
+import type { ProblemJson } from "./types.js";
 
 const RETRYABLE_STATUSES = new Set([429, 502, 503, 504]);
 const IDEMPOTENT_METHODS = new Set(["GET", "PUT", "DELETE", "HEAD", "OPTIONS"]);
@@ -33,6 +34,16 @@ function sleep(ms: number): Promise<void> {
   return new Promise((resolve) => setTimeout(resolve, ms));
 }
 
+/** Builds the error thrown for a non-2xx response. Defaults to `ZenRowsBatchError`. */
+export type ErrorFactory = (
+  response: Response,
+  problem: ProblemJson | undefined,
+  extras: Record<string, unknown> | undefined,
+) => Error;
+
+const batchError: ErrorFactory = (response, problem, extras) =>
+  new ZenRowsBatchError(response.status, problem, extras);
+
 export interface RequestOptions {
   query?: object;
   body?: unknown;
@@ -51,14 +62,15 @@ export class BatchTransport {
     private readonly baseURL: string,
     private readonly apiKey: string,
     private readonly retries: number = DEFAULT_RETRIES,
+    private readonly errorFactory: ErrorFactory = batchError,
   ) {}
 
-  /** Send a request, parse the response, throw `ZenRowsBatchError` on non-2xx. */
+  /** Send a request, parse the response, throw the factory's error on non-2xx. */
   async requestJson<T>(method: string, path: string, options: RequestOptions = {}): Promise<T> {
     const response = await this.send(method, path, options);
     if (!response.ok) {
       const { problem, extras } = await parseProblem(response);
-      throw new ZenRowsBatchError(response.status, problem, extras);
+      throw this.errorFactory(response, problem, extras);
     }
     const text = await response.text();
     if (!text) {
@@ -72,7 +84,7 @@ export class BatchTransport {
     const response = await this.send(method, path, options);
     if (!response.ok) {
       const { problem, extras } = await parseProblem(response);
-      throw new ZenRowsBatchError(response.status, problem, extras);
+      throw this.errorFactory(response, problem, extras);
     }
     return response;
   }

src/crawl/client.ts

@@ -0,0 +1,207 @@
+import { BatchTransport } from "../batch/transport.js";
+import { pollUntil } from "../batch/waiters.js";
+import { crawlError } from "./errors.js";
+import type {
+  Crawl,
+  CrawlExportLine,
+  CrawlList,
+  CrawlOutputFormat,
+  CrawlResult,
+  CrawlStop,
+  CrawlWithResults,
+} from "./types.js";
+
+const DEFAULT_CRAWL_API_URL = "https://api.zenrows.com/v1";
+
+export interface CrawlClientConfig {
+  /** Override the Crawl API base URL. Default `https://api.zenrows.com/v1`. */
+  baseURL?: string;
+  /** Retries for transient failures (429/502/503/504 + network errors) on idempotent requests. Default 3. */
+  retries?: number;
+}
+
+export interface CreateCrawlOptions {
+  /** The page the crawl starts from: an absolute public `http`/`https` URL. */
+  url: string;
+  /** Link hops to follow from the start URL, 1–100000. */
+  depth: number;
+  /** Stop once this many URLs are kept. Server default 10. */
+  maxItems?: number;
+  /** Stop once this many pages are fetched; bounds the cost. Server default 10. */
+  maxPages?: number;
+  /** Keep a URL only if it contains at least one of these substrings. */
+  includePatterns?: string[];
+  /** Drop a URL that contains any of these substrings, even if it matches an include pattern. */
+  excludePatterns?: string[];
+  /** Also fetch each kept URL's page. Absent means URLs only. */
+  outputFormat?: CrawlOutputFormat;
+}
+
+export interface WaitForCrawlOptions {
+  /** Seconds before `WaiterTimeoutError`. Default 300. The crawl keeps running on timeout. */
+  timeout?: number;
+  /** Seconds before the first re-poll. Default 2; grows ×1.5 per poll. */
+  pollInterval?: number;
+  /** Cap on the poll interval in seconds. Default 15. */
+  maxPollInterval?: number;
+}
+
+function contentIdOf(content: string | CrawlResult): string {
+  if (typeof content === "string") return content;
+  if (!content.content_url) {
+    throw new Error(
+      `getContent: result for ${content.url} has no content_url (content_status: ${content.content_status ?? "absent"})`,
+    );
+  }
+  const id = content.content_url.split("/").pop();
+  if (!id) throw new Error(`getContent: cannot read a content id from ${content.content_url}`);
+  return id;
+}
+
+/**
+ * Client for the Zenrows Crawl API: give it one start URL, read back the URLs behind it.
+ * Usable standalone (`new ZenRowsCrawlClient(apiKey)`) or via `client.crawl` on a `ZenRows`
+ * instance. A crawl runs asynchronously: `create()` returns at once, `wait()` polls until it
+ * ends, and `iterResults()` / `download()` read what it kept.
+ */
+export class ZenRowsCrawlClient {
+  readonly apiKey: string;
+  private readonly transport: BatchTransport;
+
+  constructor(apiKey: string, config: CrawlClientConfig = {}) {
+    this.apiKey = apiKey;
+    this.transport = new BatchTransport(
+      (config.baseURL ?? DEFAULT_CRAWL_API_URL).replace(/\/+$/, ""),
+      apiKey,
+      config.retries,
+      crawlError,
+    );
+  }
+
+  /**
+   * Start a crawl. Sends only the fields you set. With an `idempotencyKey`, a retry returns the
+   * crawl the first request created instead of starting another, and transient failures are
+   * retried. A 429 `too_many_crawls` (the account has too many crawls running) throws
+   * `ZenRowsCrawlError` with `retryAfter` set.
+   */
+  create(options: CreateCrawlOptions, opts: { idempotencyKey?: string } = {}): Promise<Crawl> {
+    const body = {
+      url: options.url,
+      depth: options.depth,
+      max_items: options.maxItems,
+      max_pages: options.maxPages,
+      include_patterns: options.includePatterns,
+      exclude_patterns: options.excludePatterns,
+      output_format: options.outputFormat,
+    };
+    return this.transport.requestJson("POST", "/crawls", {
+      body,
+      idempotencyKey: opts.idempotencyKey,
+    });
+  }
+
+  /** Read a crawl's status, coverage and one page of results (`limit` 1–10000, default 1000). */
+  get(
+    crawlId: string,
+    options: { cursor?: string; limit?: number } = {},
+  ): Promise<CrawlWithResults> {
+    return this.transport.requestJson("GET", `/crawls/${encodeURIComponent(crawlId)}`, {
+      query: options,
+    });
+  }
+
+  /** One page of the account's crawls, newest first (`limit` 1–100, default 20). */
+  list(options: { cursor?: string; limit?: number } = {}): Promise<CrawlList> {
+    return this.transport.requestJson("GET", "/crawls", { query: options });
+  }
+
+  /** Every crawl of the account, newest first, following `next_cursor`. */
+  async *iterCrawls(options: { limit?: number } = {}): AsyncGenerator<Crawl> {
+    let cursor: string | undefined;
+    while (true) {
+      const page = await this.list({ ...options, cursor });
+      yield* page.crawls;
+      cursor = page.next_cursor;
+      if (!cursor) return;
+    }
+  }
+
+  /**
+   * Every URL the crawl kept, following `next_cursor` until it is null. Call it once the crawl
+   * has ended (see `wait()`): on a running crawl it yields the URLs kept so far and returns.
+   */
+  async *iterResults(
+    crawlId: string,
+    options: { limit?: number } = {},
+  ): AsyncGenerator<CrawlResult> {
+    let cursor: string | undefined;
+    while (true) {
+      const page = await this.get(crawlId, { ...options, cursor });
+      yield* page.results;
+      // A running crawl's cursor is never null; an empty page means nothing more is kept yet.
+      if (page.next_cursor === null || page.results.length === 0) return;
+      cursor = page.next_cursor;
+    }
+  }
+
+  /** Stop a running crawl. Idempotent: a crawl that already ended answers as it ended. */
+  stop(crawlId: string): Promise<CrawlStop> {
+    return this.transport.requestJson("POST", `/crawls/${encodeURIComponent(crawlId)}/stop`);
+  }
+
+  /**
+   * One kept URL's page as fetched (HTML). Pass the content id, or the `CrawlResult` itself
+   * (its `content_url` names the content; present once `content_status` is `fetched`).
+   */
+  async getContent(crawlId: string, content: string | CrawlResult): Promise<string> {
+    const contentId = contentIdOf(content);
+    const response = await this.transport.requestRaw(
+      "GET",
+      `/crawls/${encodeURIComponent(crawlId)}/contents/${encodeURIComponent(contentId)}`,
+    );
+    return response.text();
+  }
+
+  /**
+   * Every result of the crawl in one download, one parsed line at a time (with the page when
+   * the crawl has an `output_format`). On a running crawl it holds what was kept so far.
+   */
+  async *download(crawlId: string): AsyncGenerator<CrawlExportLine> {
+    const response = await this.transport.requestRaw(
+      "GET",
+      `/crawls/${encodeURIComponent(crawlId)}/download`,
+    );
+    if (!response.body) return;
+    const decoder = new TextDecoder();
+    let buffered = "";
+    for await (const chunk of response.body as AsyncIterable<Uint8Array>) {
+      buffered += decoder.decode(chunk, { stream: true });
+      let newline = buffered.indexOf("\n");
+      while (newline !== -1) {
+        const line = buffered.slice(0, newline).trim();
+        buffered = buffered.slice(newline + 1);
+        if (line) yield JSON.parse(line) as CrawlExportLine;
+        newline = buffered.indexOf("\n");
+      }
+    }
+    const last = (buffered + decoder.decode()).trim();
+    if (last) yield JSON.parse(last) as CrawlExportLine;
+  }
+
+  /**
+   * Poll until the crawl ends (any status but `running`) and return it, without results.
+   * A `failed` crawl resolves too: read `error`. Throws `WaiterTimeoutError` after `timeout`
+   * seconds; the crawl is not stopped.
+   */
+  async wait(crawlId: string, options: WaitForCrawlOptions = {}): Promise<Crawl> {
+    // `limit: 1` keeps each poll cheap; the one result it reads is dropped.
+    const ended = await pollUntil(() => this.get(crawlId, { limit: 1 }), {
+      isDone: (c) => c.status !== "running",
+      timeout: options.timeout ?? 300,
+      initialInterval: options.pollInterval ?? 2,
+      maxInterval: options.maxPollInterval ?? 15,
+    });
+    const { results: _results, next_cursor: _cursor, ...crawl } = ended;
+    return crawl;
+  }
+}