Repository navigation
fix(extract): fall back to Autoparse on REQS007, and drop stale beta copy - #41
Merged
Merged
Conversation
The PyPI summary still read "synchronous scraping + Batch (async job) API" and the keywords left Extract out, so the package page undersold what 1.6.0 ships. Name Fetch, Extract, Batch and Crawl in the summary and add extract, structured-data, web-scraping and crawl to the keywords. Drop "domain-gated open beta" from the extract() docstring and its tests: Extract has worked on any prepared domain with any active key since the gate was removed. Point the README's Extract link at the Extract docs. The new metadata reaches PyPI on the next release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…copy extract(mode="auto") only fell back to Autoparse on a 402 AUTH010, the code Extract answered while it was gated by domain. The gate is gone: a domain that isn't prepared yet now answers 403 REQS007, so in practice the fallback never fired and callers got the error. The CLI, the MCP server and the Dify plugin already fall back on both codes. Treat 403 REQS007 like the legacy 402 AUTH010, in the sync and async variants, keyed on status + code so an auth 403 or a REQS007 on another status is never mistaken for it. Bring the rest of the shipped copy in line with the docs: - docs/openapi.yaml: Extract is no longer "private beta, domains not enrolled fail with the upgrade-required error"; it works on any prepared domain and an unprepared one fails with 403 REQS007. Only that paragraph changes: the public zenrows.com/openapi.json is a summarised spec, not the Batch source this copy tracks, so a full refresh would drop most schemas. - README: Batch is available to every account with the same key and is billed per successful request, not a private beta that is free to use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Extract is generally available, but the README and the extract() docstring still labelled it beta. Drop the label. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… README Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AnderRV
approved these changes
Oct 13, 2026
…allback # Conflicts: # README.md
…lback Resolve the extract() docstring and test conflicts in favour of the REQS007 fallback wording, so this branch merges cleanly whichever of the two lands first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With retries enabled, every 422 was retried by urllib3. Extract answers 422 with code no_data_extracted when it knows the kind of page but finds no data on it, and that answer is final: retrying returns the same result after another full Extract call. Retry 422s in the client instead, where the body can be read: a 422 no_data_extracted is returned at once, any other 422 is still retried up to `retries` times on the same backoff schedule as before. Export is_no_data_extracted(response) so callers can branch on it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oxy is set extract() and extract_async() added the wire `mode=auto` (Adaptive Stealth Mode) on top of whatever the caller passed. The API rejects mode=auto combined with js_render or premium_proxy (400 REQ_INVALID_PARAMS), so a caller forcing either one, like the Extract docs' js_render + premium_proxy example, always failed unless they also knew to pass adaptive_stealth=False. Leave mode=auto out when params sets js_render or premium_proxy (true or "true"), or already carries its own wire mode, on both the Extract request and the Autoparse fallback. Document it in the docstring and README. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…allback # Conflicts: # src/zenrows/client.py # tests/test_fetch_extract.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
extract(mode="auto")andextract_async()now fall back to Autoparse on 403REQS007(domain not prepared yet), as well as on the legacy 402AUTH010. The check matches status and code together, so an auth 403 (AUTH002…) orREQS007on another status never triggers it._is_auth010becomes_is_extract_domain_unavailable.extract()docstring documents both codes and how to prepare a domain (POST /v1/extract/prepared-domains).docs/openapi.yaml: thezenrows_paramsdescription no longer says Extract is in private beta and fails on domains not enrolled.no_data_extracted(Extract knows the kind of page but found no data on it) is never retried: the client now retries 422s itself, where it can read the body, and any other 422 is still retried up toretriestimes on the same backoff. Newis_no_data_extracted(response)export.extract()no longer adds Adaptive Stealth Mode (wiremode=auto) whenparamssetsjs_renderorpremium_proxy, or its ownmode: the API rejectsmode=autocombined with either (400REQ_INVALID_PARAMS, see https://docs.zenrows.com/fetch/features/adaptive-stealth-mode). Same on the Autoparse fallback request.Why
Since the domain gate was removed, Extract answers
403 REQS007for a domain that isn't prepared yet (https://docs.zenrows.com/extract/setup#domain-coverage). The SDK only knewAUTH010, so its Autoparse fallback never fired and callers got the error. The CLI, MCP server and Dify plugin already fall back on both codes. The README and spec copy described access rules that no longer exist (https://docs.zenrows.com/batch/introduction).On the spec: the DEVELOPMENT.md procedure is to copy the backend spec in. The public
zenrows.com/openapi.jsonis a summarised document (3 schemas, no run/history paths), not the Batch spec this copy tracks, so I changed only the stale paragraph. No model regeneration was needed, because that text doesn't reachmodels.py. Conveyor'sdocs/openapi.yamlhas the same stale paragraph upstream, so the next sync from it would bring it back unless it's fixed there too.Relation to #40
#40 edits the same
extract()docstring and says REQS007 is "returned as is". This PR supersedes that sentence: whichever merges second needs a trivial rebase on that paragraph.Testing
make test: 234 passed (5 new tests: sync REQS007 fallback, fallback disabled, auth 403 doesn't fall back, REQS007 on a non-403 doesn't, async REQS007 fallback; the non-auto-mode test now covers both codes)make check(format + typecheck) andmake lintcleanAfter merge
Release a patch (
make release PART=patch) so PyPI users get the fallback.🤖 Generated with Claude Code