Expose http transport options

[jem-davies]

Go's DefaultTransport var is currently referenced in:

• http_client input & output
• http processor
• schema_registry_decode & schema_registry_encode processors

It is possible that when using Bento as a library (StreamBuilder for instance), that this DefaultTransport var has been altered, and when a Bento stream that uses the above components is created, it will use the modified DefaultTransport.

This may not be desired.

---

This PR adds a FieldSpec for a http transport & func (pConf *ParsedConfig) FieldHTTPTransport() which allows users to create custom http transports from config - based on Go's DefaultTranpsort:

var DefaultTransport RoundTripper = &Transport{
	Proxy: ProxyFromEnvironment,
	DialContext: defaultTransportDialContext(&net.Dialer{
		Timeout:   30 * time.Second,
		KeepAlive: 30 * time.Second,
	}),
	ForceAttemptHTTP2:     true,
	MaxIdleConns:          100,
	IdleConnTimeout:       90 * time.Second,
	TLSHandshakeTimeout:   10 * time.Second,
	ExpectContinueTimeout: 1 * time.Second,
}

And then adds those Config Options to the http_client, http, schema_registry_decode & schema_registry_encode components.

Also as an additional option to deal with potential altered default transports - we have a BENTO_OVERRIDE_DEFAULT_HTTP_TRANSPORT Env Var that when set to "true" will ensure that the DefaultTransport is not referenced but will create a Transport from the defaults in the config options.

TODO:

• Add tests
• Add new transport options to schema_registry_decode
• Add new transport options to schema_registry_encode

---

Update last commit removes Env Var and removes references to http.DefaultTransport

[gregfurman]

Why do we not rather extend the public/service/config_tls.go to support these new fields?

[gregfurman]

Where the implementation is in internal/tls/type.go instead of within the httpClient. That way, any component using our default TLS can leverage the additional fields instead of just relying on those components using httpclient

[jem-davies]

hmm but most of the fields aren't TLS related? - Isn't the issue more that wherever we use http.DefaultTransport in Bento - we need to provide the option to override - not just in the http_client input, output & http processor?

[jem-davies]

prolly would want to move it to a separate package for the concern of building http transports - which I think is the point you are making here?

[gregfurman]

hmm but most of the fields aren't TLS related? - Isn't the issue more that wherever we use http.DefaultTransport in Bento - we need to provide the option to override - not just in the http_client input, output & http processor?

What I'm trying to say is that if someone wants to use a custom transport and specify their TLS, we handle this seperately instead of together. For example:

tls:
   field_a: A
transport:
   field_b: B

where tls (transport layer security) should be a subset of transport. Like, in the httpclient.ConfigFromParsed we retrieve the TLS config

bento/internal/httpclient/config.go

Lines 155 to 157 in 51abef7

	if conf.TLSConf, conf.TLSEnabled, err = pConf.FieldTLSToggled(hcFieldTLS); err != nil {
	return
	}

And then retrieve the transport

bento/internal/httpclient/config.go

Lines 165 to 167 in 51abef7

	if conf.transport, err = pConf.FieldHTTPTransport(hcFieldTransport); err != nil {
	return
	}

Where now it's up to the caller to set the transport.TLSClientConfig = tlsConf instead of just returning a single transport with this set.

I suppose it's not a blocker or anything, am more concerned that we're treating two fields that should otherwise be tighly coupled as seperate.

[jem-davies]

Yeah reading back this convo, I think we aren't really understanding what we are saying - but you are right:

we're treating two fields that should otherwise be tightly coupled as separate

We probably should move the tls & proxyURL stuff into the FieldSpec for a transport that would be cleaner.

Then it's just get the transport and go - no need to clone it to add tls & proxyURL to it...

I just had a thought that to do that you would have to deprecate the http_client.tls then copy it to the http_client.transport - and then is some future time remove the http_client.tls?

[jem-davies]

https://warpstreamlab-uwt2030.slack.com/archives/C0768UE3NEM/p1763648436717639

[gregfurman]

LGTM. I think there's a leftover struct that should be removed. I also think it's woth re-thinking the loose coupling of TLS and Transport but we can explore this in a follow-up.

[gregfurman]

Is this used anywhere?

Suggested change

	type CustomTransport struct {
	CustomTransportEnabled bool
	DialContextTimeout time.Duration
	DialContextKeepAlive time.Duration
	ForceAttemptHTTP2 bool
	MaxIdleConns int
	IdleConnTimeout time.Duration
	TlsHandshakeTimeout time.Duration
	ExpectContinueTimeout time.Duration
	}

[jem-davies]

taken that out 👍

[gregfurman]

hmm but most of the fields aren't TLS related? - Isn't the issue more that wherever we use http.DefaultTransport in Bento - we need to provide the option to override - not just in the http_client input, output & http processor?

What I'm trying to say is that if someone wants to use a custom transport and specify their TLS, we handle this seperately instead of together. For example:

tls:
   field_a: A
transport:
   field_b: B

where tls (transport layer security) should be a subset of transport. Like, in the httpclient.ConfigFromParsed we retrieve the TLS config

bento/internal/httpclient/config.go

Lines 155 to 157 in 51abef7

	if conf.TLSConf, conf.TLSEnabled, err = pConf.FieldTLSToggled(hcFieldTLS); err != nil {
	return
	}

And then retrieve the transport

bento/internal/httpclient/config.go

Lines 165 to 167 in 51abef7

	if conf.transport, err = pConf.FieldHTTPTransport(hcFieldTransport); err != nil {
	return
	}

Where now it's up to the caller to set the transport.TLSClientConfig = tlsConf instead of just returning a single transport with this set.

I suppose it's not a blocker or anything, am more concerned that we're treating two fields that should otherwise be tighly coupled as seperate.