refactor(fetch): make web fetch a client-owned core command by ankitranjan7 · Pull Request #295 · agentrhq/webcmd

ankitranjan7 · 2026-08-12T12:16:36Z

What this does

Moves webcmd web fetch into the core command surface and removes the old adapter split:

web fetch is now an inbuilt/client-owned command, not a site adapter.
web fetch-browser and clis/web are removed.
Fetch never opens a real browser and never escalates into browser automation.
Fetch uses only the approved HTTP ladder: plain HTTP → impit Chrome TLS impersonation → impit Firefox TLS impersonation.
If all fetch tiers fail, the command returns a structured failure/hint; agents should explicitly create/use a browser Session and run the existing webcmd browser run workflow.

Example browser fallback after fetch fails:

webcmd session create -f json
webcmd --session <session_id> browser run --url https://example.com --code "return await page.locator('body').innerText()"

Why the old bugs kept coming back

web fetch and web fetch-browser lived in two different places with two different lifecycles:

	Lived in	Shipped?	Discoverable?
`web fetch`	hardcoded fast path in `main.ts`	yes	no
`web fetch-browser`	adapter in `clis/web/`	no — `clis/` is not in `package.json` `files`	yes, in the manifest

This PR removes the split: web fetch is the one core command, and browser viewing is an explicit browser command workflow.

Issues fixed

Issue	Was	Now
#247	`fetch-browser` was advertised/hinted but not shipped	`fetch-browser` is gone; fetch fallback points to explicit browser Session usage
#252	`web fetch` was invisible to help/list/completions	core manifest/list/help/completions expose `web/fetch`
#246	argument/error paths could show raw stack traces	standard error envelopes and exit codes
#264	CSP/header text caused false `FETCH_BLOCKED`	challenge classifier only treats relevant challenge evidence as blocking
#283	healthy `server: cloudflare` responses could be misclassified	healthy 200s from Cloudflare-backed sites are allowed

Command surface

webcmd web fetch --url <url>

Supported fetch options stay intentionally small:

--timeout <seconds>
--max-chars <n>
--allow-private <true|false>
normal output flags such as -f json|md|yaml

Removed/unsupported browser-ish fetch flags include --browser, --wait, and the retired web fetch-browser command.

Hosted/cloud behavior

web fetch is client-owned. In hosted mode it still runs locally in the user's CLI process, including when root hosted flags like --workspace are present.

The cloud server should not execute web/fetch as a hosted adapter. Existing cloud behavior filters client-owned/local-only commands out of hosted execution, and browser commands continue to call Browser Use as usual.

No cloud runtime change is required for this PR beyond the normal post-release package pin bump/contract verification.

Verification

Local OpenCLI checks:

npm run build
npm run typecheck
npm run check:codex-plugin
npm run check:package-bin
npm run check:hosted-contract
npm run check:plugin-parity
npm test — 403 files, 5065 passed, 1 skipped
node dist/src/main.js --workspace test-workspace web fetch --url https://example.com --timeout 3 -f json

Cloud-focused read-only checks:

npm run typecheck
npm run test -- tests/default-adapters.test.ts tests/adapter-manifest-builder.test.ts

GitHub Actions on this PR are green for build, unit, plugin, Bun, audit, and docs sync; scheduled/live Cloak/Mintlify jobs are skipped as expected.

github-actions · 2026-08-12T12:17:00Z

🟠 Maintainer review suggested — low confidence

The automated review could not reach a fully supported conclusion.

This review is advisory and does not block merging.

…tier `webcmd web` lived in two places: `web fetch` was a hardcoded fast path in main.ts, and `web fetch-browser` was an adapter in clis/web that the published tarball never shipped (`clis/` is not in package.json `files`). The split produced a recurring class of bug rather than isolated ones. The whole ladder now lives in src/fetch and ships in dist/: - `web fetch` walks plain HTTP -> impit -> browser in one command. A blocked page is rendered and returned instead of raising an error that names a second command. `--browser false` opts out; escalation is local-mode only, since hosted mode executes adapters server-side. - `web fetch-browser` keeps the article-export pipeline (--output, --download-images, --wait-for, --diagnose) for callers who want files. - Both are registered in the core registry, so help, `list`, completions and the manifests carry them with no plugin installed. - The fast path stays for plain fetches but hands `-h`/`-f`/`--trace` to the registered command, and renders the standard error envelope instead of leaking a raw Node stack trace. clis/web is deleted; build-manifest now emits core-registered commands with no modulePath, since there is no adapter file under clis/ to resolve. Also fixes the challenge classifier, which flattened every response header into one string and grepped it. A CSP naming cdnjs.cloudflare.com, or `server: cloudflare` on a healthy 200, was read as a bot challenge — so example.com and news.ycombinator.com both burned two retries and failed with FETCH_BLOCKED. Header evidence is now limited to headers describing the response itself, and markers are split into decisive (cf-mitigated, "just a moment") vs corroborating (cloudflare, captcha), the latter requiring a 403/429/503. Fixes #246, #247, #252, #264 Fixes #283 (classifier half; the safe-proxy EPIPE half landed in #265) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…s paths Follow-ups found while verifying the move end to end: - `web fetch -f md` rendered a nine-column table of the result object while the fast path printed a document — the same command with two shapes. Adds a `renderMarkdown` hook on CliCommand so a command whose payload is prose can own its markdown, and points `web fetch` at the formatter the fast path already used. Both paths now emit byte-identical output. - Hosted mode kept falling through to the hosted runner once flag handling moved to the registered command, which would cloud-route a command the service cannot execute yet. Hosted mode now always takes the fast path, so its behavior is unchanged from before this refactor. - `--format=json` (equals form) slipped past the fast-path flag guard and was silently ignored; the guard now splits on `=`. - Drops the dead duplicate of the flag guard from fetch/command.ts. - The article-download e2e still invoked `web read`, renamed to `web fetch-browser` back in 0.5.x. It swallowed the resulting CLI failure and passed vacuously, so it had been testing nothing since. All six real sites now exercise the pipeline for real. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Core-registered commands carry no modulePath, since there is no adapter file under clis/ to resolve. That leaves a consumer which loads adapters by path — webcmd-cloud's hosted executor — with nothing to import. Rather than have the cloud hardcode that `web` lives in dist/src/fetch/, the manifest now states it: `packageExport: './fetch/command'`, a subpath export the package already declares. The cloud resolves it through the pinned package's own export map, so this keeps working if the internal layout moves. A new test asserts every manifest entry is resolvable — a clis/ path or a package export that exists and points at a real source file. Without it a rename would only surface as a hosted runtime failure after publish. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ng runtime The hosted cloud executor calls adapter command functions directly, in a worker that has no local browser. There, a blocked `web fetch` escalated by calling webcmd's own executeCommand, reached for a local Chromium, and hung on a CDP connect for ~12s before failing with an opaque BrowserCommandError instead of a structured FETCH_BLOCKED. The existing hosted-mode guard did not cover this: shouldUseHostedMode() reads the local ~/.webcmd config, which on a cloud worker is absent, so it reported local mode and escalation proceeded. An embedder now sets WEBCMD_EMBEDDED_EXECUTOR=1 to declare that it drives these functions and owns browser execution itself. Escalation then declines with a hint naming the browser-backed command. The CLI never sets it, so local behaviour is unchanged. Reproduced against a packed build with no local config: 11.8s BrowserCommandError before, 1.6s structured FETCH_BLOCKED after. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

beubax changed the title ~~refactor(fetch): move web into core, one auto-escalating fetch command~~ refactor(fetch): make web fetch a client-owned core command Aug 12, 2026

ankitranjan7 and others added 21 commits August 13, 2026 01:34

chore: configure .png files as binary in .gitattributes

dfffa0a

refactor(fetch): route client-owned fetch through core parser

3fae8c3

test(fetch): cover structured help and argument envelope

a9dea21

fix(fetch): stop after three non-browser transports

c6bb3f6

fix(fetch): bound body reads by deadline

087ea16

feat(fetch): publish client-owned core manifest entry

5a10773

test(fetch): avoid ignored contract artifact

5ba365b

fix(hosted): present client-owned web fetch locally

234e3e2

refactor(fetch): remove browser-backed fetch command

ae91969

docs(fetch): require explicit browser sessions after fetch

d137a6e

test(fetch): verify packed core command surface

de0149f

test(plugin): allow removed browser fetch command

0bcce72

test: include browser command catalog routing

6d39f64

fix: isolate client-owned web fetch

b02d8e7

chore: sync plugin metadata version

629647a

fix(fetch): tear down proxy peers on reset

b2ae00f

chore: untrack final fix report

2bbbcc6

beubax force-pushed the worktree-web-fetch-core branch from cfbe50f to 2bbbcc6 Compare August 12, 2026 20:07

beubax added 3 commits August 13, 2026 01:42

chore: keep codex plugin prompt check in sync

bcdfec7

test: skip browser runner suite without playwright browser

848910e

test: make session tests portable on windows

86404fc

beubax merged commit feec557 into main Aug 12, 2026
33 checks passed

This was referenced Aug 12, 2026

[Bug]: webcmd web fetch prints a raw Node stack trace instead of a formatted CLI error #246

Closed

[Bug]: web fetch-browser is not installed by default, so the FETCH_BLOCKED escalation path is a dead end #247

Closed

beubax deleted the worktree-web-fetch-core branch August 12, 2026 20:27

Navigation Menu

Uh oh!

refactor(fetch): make web fetch a client-owned core command - #295

refactor(fetch): make web fetch a client-owned core command#295
beubax merged 24 commits into
mainfrom
worktree-web-fetch-core

ankitranjan7 commented Aug 12, 2026 •

edited by beubax

Loading

Uh oh!

github-actions Bot commented Aug 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ankitranjan7 commented Aug 12, 2026 • edited by beubax Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this does

Why the old bugs kept coming back

Issues fixed

Command surface

Hosted/cloud behavior

Verification

Uh oh!

github-actions Bot commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟠 Maintainer review suggested — low confidence

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ankitranjan7 commented Aug 12, 2026 •

edited by beubax

Loading

github-actions Bot commented Aug 12, 2026 •

edited

Loading