feat!: allow passing a custom HTTP client to ApifyClient - #1058
Draft
vdusek wants to merge 15 commits into
Draft
Conversation
Contributor
|
See more at https://github.com/apify/apify-client-js/actions/runs/34868181636#summary-104057190803 |
# Conflicts: # src/resource_clients/dataset.ts
# Conflicts: # docs/02_concepts/02_error-handling.md # docs/public-api/apify-client.api.md # src/apify_api_error.ts # src/apify_client.ts # src/base/resource_client.ts # src/http_client.ts # src/resource_clients/actor.ts # src/resource_clients/dataset.ts # src/resource_clients/key_value_store.ts # src/resource_clients/log.ts # src/resource_clients/request_queue.ts # src/resource_clients/run.ts # src/resource_clients/task.ts # test/client_timeouts.test.ts # test/http_client.test.ts
# Conflicts: # docs/04_upgrading/upgrading_v3.md # src/apify_api_error.ts # src/http_client.ts # src/resource_clients/dataset.ts
# Conflicts: # docs/04_upgrading/upgrading_v3.md # src/apify_api_error.ts # src/http_client.ts # src/interceptors.ts # src/utils.ts # test/http_client.test.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
ApifyClientsends its requests through an HTTP client you can replace. The design follows apify-client-python (apify/apify-client-python#641), with axios as the only built-in transport.HttpClient(src/http_clients/base.ts) is the abstract base holding the shared pipeline: header merging, JSON and form serialization, compression, query encoding, the retry loop with exponential backoff, timeout growth, body parsing, statistics and theApifyApiErrorconversion. A transport implementssendRequest()and can overrideisTimeoutError(),isRetryableTransportError()andclose().AxiosHttpClient(src/http_clients/axios.ts) is the built-in transport. Axios only moves bytes now, since the pipeline serializes and parses the bodies. The keep-alive agents, the proxy handling andrequestInterceptorslive here.ApifyClient.withCustomHttpClient({ token, baseUrl, publicBaseUrl, httpClient })mirrors Python'swith_custom_http_client. The token is applied throughsetDefaultAuthorization()unless the client already has anAuthorizationheader.httpClientis a lazy getter with a setter, andclient.statsaliases the custom client's stats.HttpRequest,HttpResponse,ApifyRequestConfig,ApifyResponseand the option types are exported from the package root.http_client.tsandinterceptors.tsare gone,InvalidResponseBodyErrorhas its own file, and the base class has its own retry loop, soasync-retryis no longer a dependency.fetch, a section in the v3 upgrading guide.test/pluggable_http_client.test.tsdrives acall()-only client and a hooks-only client overnode:httpthrough the pipeline, plus classification tests forAxiosHttpClient.CLIENT_METHOD_REGEXinapify_api_error.tsskips*HttpClient.stack frames, since the error is created insideAxiosHttpClient._makeRequest()andclientMethodread that frame.This rewrites
http_client.ts, so #1046, #1051, #1052 and #1055 will need a rebase onto it. The error-response stream drain from #1055 is deliberately not folded in.Open calls
withCustomHttpClient()as in Python, or anhttpClientoption on the constructor?requestInterceptorsmoved toAxiosHttpClientOptions. Nothing in the SDK or Crawlee uses it.ApifyApiError.httpMethodis uppercase now; keeping the lowercase axios form would take onetoLowerCase().BREAKING CHANGE:
requestInterceptorsmoved fromApifyClientOptionstoAxiosHttpClientOptions; pass anAxiosHttpClienttoApifyClient.withCustomHttpClient()to keep using them, and note that an interceptor now sees the body already serialized.ApifyRequestConfigandApifyResponseno longer extend the axios types:forceBufferbecameresponseType: 'buffer', andresponseTypetakes'parsed' | 'buffer' | 'stream'.InvalidResponseBodyError.responseis the transport'sHttpResponsewith the rawbody.ApifyApiError.httpMethodis uppercase (GET). Theaxios,httpAgentandhttpsAgentproperties moved fromHttpClienttoAxiosHttpClient.Closes #855
✍️ Drafted by Claude Code