Skip to content

feat!: allow passing a custom HTTP client to ApifyClient - #1058

Draft
vdusek wants to merge 15 commits into
v3from
feat/pluggable-http-client
Draft

feat!: allow passing a custom HTTP client to ApifyClient#1058
vdusek wants to merge 15 commits into
v3from
feat/pluggable-http-client

Conversation

@vdusek

@vdusek vdusek commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

What

ApifyClient sends its requests through an HTTP client you can replace. The design follows apify-client-python (apify/apify-client-python#641), with axios as the only built-in transport.

  • HttpClient (src/http_clients/base.ts) is the abstract base holding the shared pipeline: header merging, JSON and form serialization, compression, query encoding, the retry loop with exponential backoff, timeout growth, body parsing, statistics and the ApifyApiError conversion. A transport implements sendRequest() and can override isTimeoutError(), isRetryableTransportError() and close().
  • AxiosHttpClient (src/http_clients/axios.ts) is the built-in transport. Axios only moves bytes now, since the pipeline serializes and parses the bodies. The keep-alive agents, the proxy handling and requestInterceptors live here.
  • ApifyClient.withCustomHttpClient({ token, baseUrl, publicBaseUrl, httpClient }) mirrors Python's with_custom_http_client. The token is applied through setDefaultAuthorization() unless the client already has an Authorization header. httpClient is a lazy getter with a setter, and client.stats aliases the custom client's stats.
  • HttpRequest, HttpResponse, ApifyRequestConfig, ApifyResponse and the option types are exported from the package root. http_client.ts and interceptors.ts are gone, InvalidResponseBodyError has its own file, and the base class has its own retry loop, so async-retry is no longer a dependency.
  • Docs: an "HTTP clients" concept page, a "Build a custom HTTP client" guide over fetch, a section in the v3 upgrading guide.
  • Tests: test/pluggable_http_client.test.ts drives a call()-only client and a hooks-only client over node:http through the pipeline, plus classification tests for AxiosHttpClient.

CLIENT_METHOD_REGEX in apify_api_error.ts skips *HttpClient. stack frames, since the error is created inside AxiosHttpClient._makeRequest() and clientMethod read that frame.

This rewrites http_client.ts, so #1046, #1051, #1052 and #1055 will need a rebase onto it. The error-response stream drain from #1055 is deliberately not folded in.

Open calls

  1. Static factory withCustomHttpClient() as in Python, or an httpClient option on the constructor?
  2. requestInterceptors moved to AxiosHttpClientOptions. Nothing in the SDK or Crawlee uses it.
  3. ApifyApiError.httpMethod is uppercase now; keeping the lowercase axios form would take one toLowerCase().

BREAKING CHANGE: requestInterceptors moved from ApifyClientOptions to AxiosHttpClientOptions; pass an AxiosHttpClient to ApifyClient.withCustomHttpClient() to keep using them, and note that an interceptor now sees the body already serialized. ApifyRequestConfig and ApifyResponse no longer extend the axios types: forceBuffer became responseType: 'buffer', and responseType takes 'parsed' | 'buffer' | 'stream'. InvalidResponseBodyError.response is the transport's HttpResponse with the raw body. ApifyApiError.httpMethod is uppercase (GET). The axios, httpAgent and httpsAgent properties moved from HttpClient to AxiosHttpClient.

Closes #855

✍️ Drafted by Claude Code

@vdusek vdusek added the t-tooling Issues with this label are in the ownership of the tooling team. label Sep 10, 2026
@vdusek vdusek self-assigned this Sep 10, 2026
@github-actions

github-actions Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

⚠️ There are broken links in the documentation.

See more at https://github.com/apify/apify-client-js/actions/runs/34868181636#summary-104057190803

# Conflicts:
#	docs/02_concepts/02_error-handling.md
#	docs/public-api/apify-client.api.md
#	src/apify_api_error.ts
#	src/apify_client.ts
#	src/base/resource_client.ts
#	src/http_client.ts
#	src/resource_clients/actor.ts
#	src/resource_clients/dataset.ts
#	src/resource_clients/key_value_store.ts
#	src/resource_clients/log.ts
#	src/resource_clients/request_queue.ts
#	src/resource_clients/run.ts
#	src/resource_clients/task.ts
#	test/client_timeouts.test.ts
#	test/http_client.test.ts
# Conflicts:
#	docs/04_upgrading/upgrading_v3.md
#	src/apify_api_error.ts
#	src/http_client.ts
#	src/resource_clients/dataset.ts
# Conflicts:
#	docs/04_upgrading/upgrading_v3.md
#	src/apify_api_error.ts
#	src/http_client.ts
#	src/interceptors.ts
#	src/utils.ts
#	test/http_client.test.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants