Graceful shutdown with HTTP2 CONNECT

[howardjohn]

Title: Graceful shutdown with HTTP2 CONNECT

Description:
We use Envoy to tunnel HTTP traffic over HTTP2 CONNECT.

When the server is shutting down, it will send us a GOAWAY. However, Envoy will still treat the connection as live.

What we want is for the inner-HTTP to stop using the HTTP2 CONNECT transport when it gets a GOAWAY, and only send new inner-http requests on a new CONNECT tunnel.

[mattklein123]

cc @alyssawilk

[alyssawilk]

Is this one of those "two passes through Envoy" issues?

The HTTP connection pool does properly handle goaways by marking the connection as draining and not creating new streams on that connection
https://github.com/envoyproxy/envoy/blob/main/source/common/http/conn_pool_base.cc#L100

That said that HTTP/2 connection has no idea it's handling HTTP/1.1 as the inner payload so can't send a connection: close header. I think that'd have to be done by whoever is sending the HTTP/1.1. If you want to handle it in Envoy you could potentially advise open streams that a connection is draining and propagate that through the internal listener code but that's a feature I suspect you'd have to pick up yourselves.

[howardjohn]

I don't think I necessarily need connection:close onto the HTTP 1/1 connections.

So we have

app -- HTTP/1.1 ---> Envoy (Handling HTTP with an HCM) ---Internal Listener--> Same Envoy process ---> HTTP/2 CONNECT ---> Backend Proxy server (Terminates CONNECT) ---HTTP/1.1 --> backend app

When the "Backend Proxy server" is shutting down, it will send a GOAWAY over the outer HTTP/2 CONNECT, wait N seconds, then close the connection.

During this time between the GOAWAY and connection close ("draining"), I would want Envoy to not use the draining CONNECT connection for new HTTP/1.1 requests, and instead open up a new CONNECT request (or use an existing one, if there is another one already).

[alyssawilk]

yeah so you'd want to tweak the codec client to warn all streams when the connection is draining, propogate that through the internal listener, and use that as a drain signal for the HTTP/1 "connection"