Graceful shutdown: Await all data on the stream being consumed

[bwoebi]

According to the HTTP/3 spec and also WebTransport:

Once all accepted requests and pushes have been processed, the endpoint can permit the connection to become idle, or it MAY initiate an immediate closure of the connection.
~ https://www.rfc-editor.org/rfc/rfc9114.html#name-connection-shutdown

If the endpoint
were to do both of those simultaneously, the peer could potentially
receive the CONNECTION_CLOSE before receiving the
CLOSE_WEBTRANSPORT_SESSION, thus never receiving the application
error data contained in the latter. To avoid this, the endpoint
SHOULD wait until all of the data on the CONNECT stream is
acknowledged before sending the CONNECTION_CLOSE
~ https://datatracker.ietf.org/doc/draft-ietf-webtrans-http3/

How is this achievable with quiche?

I assume that the only possibility is to poll stream_capacity() on every single open stream each time some data arrives from the peer, comparing against the bidirectional or unidirectional send buffer sizes? But not really sure what to compare the returned value against - what is the capacity if everything has been acknowleged? Or is stream_capacity() the max capacity and not related to the acknowledged bytes? I'm not sure from documentation.

Apart from that I cannot find a possibility to await "has all data been acknowledged?" on one or all streams of a connection.

Is this missing functionality or am I just overlooking some API?

[LPardue]

Hi thanks for flagging this.

There is currently no API to determine if stream or datagram data was acknowledged by the peer. That's a feature request to consider but it might be low on the priority list. The reason being that transport-level acknowledgment is no guarantee that the peer has read the application data. Consider the case where an application reads data slower than the RTT of a link, even if the closing side waits for the stream data to be acked before closing, a CONNECTION_CLOSE implicitly resets all streams and so the other side, if too slow, wouldn't be able to do anything anyway.

I think this is a flaw in the WebTransport spec and I'll raise an issue

[bwoebi]

Okay, thanks for raising it on the WebTransport spec then.

Do you by chance know what the proper mechanism of winding down for HTTP/3 connections on the server side is?
Let's say I send a GOAWAY. I finish responding to the current in-flight requests. I .... immediately close the connection.

Then retransmissions of the final response will be ignored (as per RFC 9000) - "An endpoint enters the closing state after initiating an immediate close. In the closing state, an endpoint retains only enough information to generate a packet containing a CONNECTION_CLOSE frame and to identify packets as belonging to the connection", i.e. retransmission state and buffers are considered dropped immediately.

So, I'm not seeing any option, but basically doing a sleep for a fixed amount of time to increase the likeliness of data having been received by the other end, before actually closing the connection?
Or am I misunderstanding the RFCs or is this something implementations should just not worry about?

[LPardue]

You're asking all the right questions.

Unfortunately GOWAY is quite racy, even back in the days of HTTP/2 and TCP. At the point a server sends GOAWAY, the client might have already sent new requests.

So the general recommendation is to give a time period that is some multiple of the RTT for the client to receive and process the GOAWAY. In that period any new requests could be accepted or rejected. After that initial period, have another time period where you either complete all the outstanding work, or it expires and you forcibly CONNECTION_CLOSE.

The one wrinkle is that GOAWAY sending could be blocked by flow or congestion control. So you'd want to ensure it was emitted before starting timers.

[bwoebi]

Thanks for responding.

I wished the HTTP/3 RFC just had one sentence mandating the client to immediately close the connection himself once a GOAWAY frame was received and all requests have been processed.

But it just says:

Once all accepted requests and pushes have been processed, the endpoint can permit the connection to become idle, or it MAY initiate an immediate closure of the connection.

So, I'll just do a timeout after the final response of 3-4x the RTT I guess.