httpclient: add composable HTTP client with layered RoundTripper chain#4081
httpclient: add composable HTTP client with layered RoundTripper chain#4081
Conversation
|
|
||
| body, err := io.ReadAll(resp.Body) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("reading response body: %v", err) |
There was a problem hiding this comment.
Use %w instead of %v to preserve the error chain for errors.Is/errors.As upstream. Per project error handling patterns: "Use %w for wrapping (allows errors.Is/errors.As upstream). Use %v only when you intentionally want to break the error chain."
|
Commits
Review
|
|
Commits Review LGTM |
710ddb2 to
e6a792a
Compare
Introduce a new internal/httpclient package that provides a production-ready
HTTP client built as a chain of http.RoundTripper decorators. The client is
configured via benthos service.ConfigField specs and assembled from parsed
YAML configuration.
RoundTripper chain (outermost to innermost):
- Tracing: OpenTelemetry spans per request with method, URL, status attributes
- Max response body: io.LimitReader cap on response bodies
- Retry: exponential backoff with jitter, Retry-After header support,
configurable status code classification (retry/drop/success), adaptive
429-only mode as default, body replay via GetBody
- TPS rate limit: token bucket via golang.org/x/time/rate
- Metrics: benthos metrics (request duration, count, errors, active gauge)
with method and status class labels
- Logging: structured JSON access logs with configurable level and
request/response body capture with prefix sniffing
- Auth: bearer token and benthos signer (OAuth1, basic_auth, JWT)
- Base transport: net/http.Transport with full HTTP/2 config, TLS,
proxy, connection pooling, and dialer settings
Key design decisions:
- Each layer is a no-op passthrough when not configured (zero allocation)
- RetryConfig is a Go-only API (not YAML-exposed) for consumer control
over retry/drop/success status codes; YAML exposes only 429 backoff
- H2TransportConfig maps directly to net/http.HTTP2Config with validation
- MaxIdleConnsPerHost defaults to GOMAXPROCS+1 instead of Go's default of 2
- Response body drain on retry capped at 1MB to prevent stalling
- Backoff jitter uses full-range [-delay/2, +delay/2] distribution
e6a792a to
e5df6d6
Compare
Introduce a new internal/httpclient package that provides a production-ready
HTTP client built as a chain of http.RoundTripper decorators. The client is
configured via benthos service.ConfigField specs and assembled from parsed
YAML configuration.
RoundTripper chain (outermost to innermost):
- Tracing: OpenTelemetry spans per request with method, URL, status attributes
- Max response body: io.LimitReader cap on response bodies
- Retry: exponential backoff with jitter, Retry-After header support,
configurable status code classification (retry/drop/success), adaptive
429-only mode as default, body replay via GetBody
- TPS rate limit: token bucket via golang.org/x/time/rate
- Metrics: benthos metrics (request duration, count, errors, active gauge)
with method and status class labels
- Logging: structured JSON access logs with configurable level and
request/response body capture with prefix sniffing
- Auth: bearer token and benthos signer (OAuth1, basic_auth, JWT)
- Base transport: net/http.Transport with full HTTP/2 config, TLS,
proxy, connection pooling, and dialer settings
Key design decisions:
- Each layer is a no-op passthrough when not configured (zero allocation)
- RetryConfig is a Go-only API (not YAML-exposed) for consumer control
over retry/drop/success status codes; YAML exposes only 429 backoff
- H2TransportConfig maps directly to net/http.HTTP2Config with validation
- MaxIdleConnsPerHost defaults to GOMAXPROCS+1 instead of Go's default of 2
- Response body drain on retry capped at 1MB to prevent stalling
- Backoff jitter uses full-range [-delay/2, +delay/2] distribution