Skip to content

httpclient: add composable HTTP client with layered RoundTripper chain#4081

Open
mmatczuk wants to merge 3 commits intomainfrom
mmt/httpclient
Open

httpclient: add composable HTTP client with layered RoundTripper chain#4081
mmatczuk wants to merge 3 commits intomainfrom
mmt/httpclient

Conversation

@mmatczuk
Copy link
Contributor

@mmatczuk mmatczuk commented Mar 9, 2026

Introduce a new internal/httpclient package that provides a production-ready
HTTP client built as a chain of http.RoundTripper decorators. The client is
configured via benthos service.ConfigField specs and assembled from parsed
YAML configuration.

RoundTripper chain (outermost to innermost):
  - Tracing: OpenTelemetry spans per request with method, URL, status attributes
  - Max response body: io.LimitReader cap on response bodies
  - Retry: exponential backoff with jitter, Retry-After header support,
    configurable status code classification (retry/drop/success), adaptive
    429-only mode as default, body replay via GetBody
  - TPS rate limit: token bucket via golang.org/x/time/rate
  - Metrics: benthos metrics (request duration, count, errors, active gauge)
    with method and status class labels
  - Logging: structured JSON access logs with configurable level and
    request/response body capture with prefix sniffing
  - Auth: bearer token and benthos signer (OAuth1, basic_auth, JWT)
  - Base transport: net/http.Transport with full HTTP/2 config, TLS,
    proxy, connection pooling, and dialer settings

Key design decisions:
  - Each layer is a no-op passthrough when not configured (zero allocation)
  - RetryConfig is a Go-only API (not YAML-exposed) for consumer control
    over retry/drop/success status codes; YAML exposes only 429 backoff
  - H2TransportConfig maps directly to net/http.HTTP2Config with validation
  - MaxIdleConnsPerHost defaults to GOMAXPROCS+1 instead of Go's default of 2
  - Response body drain on retry capped at 1MB to prevent stalling
  - Backoff jitter uses full-range [-delay/2, +delay/2] distribution

@mmatczuk mmatczuk changed the title httpclient: add composable HTTP client with layered RoundTripper chain Michał Matczuk 05/03/2026, 11:18 httpclient: add composable HTTP client with layered RoundTripper chain Mar 9, 2026

body, err := io.ReadAll(resp.Body)
if err != nil {
return nil, fmt.Errorf("reading response body: %v", err)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use %w instead of %v to preserve the error chain for errors.Is/errors.As upstream. Per project error handling patterns: "Use %w for wrapping (allows errors.Is/errors.As upstream). Use %v only when you intentionally want to break the error chain."

@claude
Copy link

claude bot commented Mar 9, 2026

Commits

  1. The jira: switch to httpclient commit mixes documentation changes (docs/modules/components/pages/processors/jira.adoc) with code changes. Per commit policy, documentation changes must be in a separate commit from code changes in multi-commit PRs.

Review
The PR introduces a well-structured internal/httpclient package with layered RoundTripper chain (tracing, retry, rate limiting, metrics, logging, auth, base transport) and migrates the Jira processor to use it, removing the old jirahttp/http_metrics package and DoRequestWithRetries helper.

  1. New code in internal/impl/jira/jirahttp/client.go uses %v instead of %w in fmt.Errorf, breaking the error chain. See inline comment.

@claude
Copy link

claude bot commented Mar 9, 2026

Commits
LGTM

Review
This PR introduces a well-designed internal/httpclient package with a composable RoundTripper chain (tracing, max body, retry, TPS rate limit, metrics, logging, auth, base transport), then migrates the Jira processor to use it — removing the old http_metrics and jira_helper packages.

LGTM

Introduce a new internal/httpclient package that provides a production-ready
HTTP client built as a chain of http.RoundTripper decorators. The client is
configured via benthos service.ConfigField specs and assembled from parsed
YAML configuration.

RoundTripper chain (outermost to innermost):
  - Tracing: OpenTelemetry spans per request with method, URL, status attributes
  - Max response body: io.LimitReader cap on response bodies
  - Retry: exponential backoff with jitter, Retry-After header support,
    configurable status code classification (retry/drop/success), adaptive
    429-only mode as default, body replay via GetBody
  - TPS rate limit: token bucket via golang.org/x/time/rate
  - Metrics: benthos metrics (request duration, count, errors, active gauge)
    with method and status class labels
  - Logging: structured JSON access logs with configurable level and
    request/response body capture with prefix sniffing
  - Auth: bearer token and benthos signer (OAuth1, basic_auth, JWT)
  - Base transport: net/http.Transport with full HTTP/2 config, TLS,
    proxy, connection pooling, and dialer settings

Key design decisions:
  - Each layer is a no-op passthrough when not configured (zero allocation)
  - RetryConfig is a Go-only API (not YAML-exposed) for consumer control
    over retry/drop/success status codes; YAML exposes only 429 backoff
  - H2TransportConfig maps directly to net/http.HTTP2Config with validation
  - MaxIdleConnsPerHost defaults to GOMAXPROCS+1 instead of Go's default of 2
  - Response body drain on retry capped at 1MB to prevent stalling
  - Backoff jitter uses full-range [-delay/2, +delay/2] distribution
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant