Skip to content

fix(sdk): follow batch pagination and stop on cancel in both SDKs - #411

Open
us wants to merge 1 commit into
mainfrom
fix/sdk-batch-pagination
Open

fix(sdk): follow batch pagination and stop on cancel in both SDKs#411
us wants to merge 1 commit into
mainfrom
fix/sdk-batch-pagination

Conversation

@us

@us us commented Aug 7, 2026

Copy link
Copy Markdown
Owner

The bug (batch-scrape-plan A4)

batch_scrape / batchScrape polled the async batch job and, on completed, returned only the first status page's data. The batch status endpoint returns at most ~100 documents per page with a next cursor, so any batch over 100 URLs silently lost every page past the first — the highest-harm item in the batch plan. A cancelled job was also not a terminal branch, so the poll span until timeout.

The fix

  • Poll page 1 to a terminal state; a new cancelled branch raises instead of spinning. Then follow next, accumulating every page's data.
  • Each next-page request is rebuilt from the SDK's own base path plus only the skip/limit cursor values, never the server's absolute next URL. Reusing that URL double-prefixes when api_url carries a path (/api) and trusts a host that may not match. The seen set is seeded with page 1 so a cursorless/repeat next stops cleanly rather than re-appending or looping.
  • Tolerate a null data page (list(None) would crash the Python path; TS was already guarded with ?? []).
  • Crawl polls (Python + TS) also gain the cancelled terminal branch (same infinite-hang class).

Verified against the real server contract: the SaaS /v2/batch/scrape/{id} cursor is {origin}{prefix}/{id}?skip=N, engine page size 100, next null on the last page, cancelled a genuine engine terminal state. The SDK deliberately stays on /v2/batch/scrape (the paginated surface); native /v1/batch/scrape returns a full CrawlState with no pagination.

Tests

Python 41 pass, TypeScript 19 pass. New: pagination collects all pages, cancel raises without spinning, null-data page yields [].

batch_scrape/batchScrape returned only the first status page's data, so any
batch over ~100 documents silently lost every page past the first; a cancelled
job was also not a terminal branch, so the poll spun until timeout.

- poll page 1 to a terminal state (new cancelled branch raises instead of
  spinning), then follow the next cursor accumulating every page's data
- rebuild each page request from the SDK's own base path plus only the
  skip/limit cursor values, never the server's absolute next URL: reusing it
  double-prefixes when api_url carries a path (/api) and trusts a host that may
  not match. seed the seen-set with page 1 so a cursorless/repeat next stops
  cleanly rather than re-appending or looping
- tolerate a null data page (list(None) would crash the Python path; TS already
  guarded with ?? [])
- crawl polls (Python + TS) also gain the cancelled terminal branch
- tests: pagination collects all pages, cancel raises without spinning, null
  data yields []
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant