Add graceful shutdown #10701

yawkat · 2024-04-11T11:05:01Z

This PR adds a new interface, GracefulShutdownLifecycle, that parts of the framework can implement to provide graceful shutdown functionality. It is designed to complement LifeCycle.stop(), though the implementations are unrelated.

The netty HTTP server, on graceful shutdown, will stop accepting new connections, and wind down existing ones. HTTP/1 connections are closed as soon as in-progress requests have been served, while HTTP/2 and /3 connections send a GOAWAY packet to indicate to the client that no further requests will be processed.

There is also a readiness health indicator that turns DOWN when a graceful shutdown begins. To keep the health endpoint accessible, there is a new micronaut.server.netty.listeners.*.support-graceful-shutdown config option to disable graceful shutdown for a management port.

Graceful shutdown is triggered by application code through a new GracefulShutdownManager API.

Graceful shutdown is best-effort. In-progress requests may take too long, or HTTP/2 clients may refuse to shut down their connection. For this reason, GracefulShutdownLifecycle returns a CompletionStage. The stage will complete when a graceful shutdown has been achieved, but this may be never, so a user should always add a timeout. After a graceful shutdown (or timeout), a normal LifeCycle.stop should be performed.

Docs are still pending, I want Graeme to review the API before I write it up.

This PR adds a new interface, GracefulShutdownLifecycle, that parts of the framework can implement to provide graceful shutdown functionality. It is designed to complement LifeCycle.stop(), though the implementations are unrelated. The netty HTTP server, on graceful shutdown, will stop accepting new connections, and wind down existing ones. HTTP/1 connections are closed as soon as in-progress requests have been served, while HTTP/2 and /3 connections send a GOAWAY packet to indicate to the client that no further requests will be processed. There is also a readiness health indicator that turns DOWN when a graceful shutdown begins. To keep the health endpoint accessible, there is a new `micronaut.server.netty.listeners.*.support-graceful-shutdown` config option to disable graceful shutdown for a management port. Graceful shutdown is triggered by application code through a new GracefulShutdownManager API. Graceful shutdown is best-effort. In-progress requests may take too long, or HTTP/2 clients may refuse to shut down their connection. For this reason, GracefulShutdownLifecycle returns a CompletionStage. The stage will complete when a graceful shutdown has been achieved, but this may be never, so a user should always add a timeout. After a graceful shutdown (or timeout), a normal LifeCycle.stop should be performed. Docs are still pending, I want Graeme to review the API before I write it up.

sonarcloud · 2024-04-11T12:12:50Z

Quality Gate failed

Failed conditions
1 New Bugs (required ≤ 0)
1 New Critical Issues (required ≤ 0)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

graemerocher · 2024-04-16T17:22:56Z

context/src/main/java/io/micronaut/runtime/server/GracefulShutdownLifecycle.java

+ * @author Jonas Konrad
+ * @since 4.5.0
+ */
+public interface GracefulShutdownLifecycle {


not sure lifecycle is the right name here, since this is only graceful shutdown and not startup. Rename to GracefulShutdownCapable

context/src/main/java/io/micronaut/runtime/server/GracefulShutdownLifecycle.java

graemerocher · 2024-04-16T17:25:09Z

context/src/main/java/io/micronaut/runtime/server/GracefulShutdownLifecycle.java

+     * @param stages The input lifecycles
+     * @return A future that completes when all inputs have completed shutdown
+     */
+    static CompletionStage<?> shutdownAll(Stream<? extends GracefulShutdownLifecycle> stages) {


what about timeouts? The idea is a window where graceful shutdown is allowed but then it still has to shutdown. I would expect some duration parameter here

The idea is that the caller adds a timeout, and after the timeout calls normal Lifecycle.stop.

context/src/main/java/io/micronaut/runtime/server/GracefulShutdownManager.java

...src/main/java/io/micronaut/http/server/netty/configuration/NettyHttpServerConfiguration.java

graemerocher · 2024-04-16T17:28:54Z

.../src/main/java/io/micronaut/management/health/indicator/GracefulShutdownHealthIndicator.java

+
+    @Override
+    public Publisher<HealthResult> getResult() {
+        return Mono.just(HealthResult.builder("gracefulShutdown")


move to constant rename to graceful-shutdown

Other health indicators are camel case, eg diskSpace and compositeDiscoveryClient()

graemerocher · 2024-04-16T17:31:09Z

context/src/main/java/io/micronaut/runtime/server/GracefulShutdownManager.java

+     * down
+     */
+    public CompletionStage<?> shutdownGracefully() {
+        return GracefulShutdownLifecycle.shutdownAll(delegates.stream());


I think we need better error handling here. What happens when there are multiple implementations and one of them fails to shutdown? How do we know and log which one failed?

Say other modules like the JMS, Kafka etc. modules implement this interface I would want to know which one failed and why.

The interface is not really designed to fail. If an implementation cannot gracefully shut down for whatever reason, it should just signal completion and Lifecycle.stop will shut it down hard.

graemerocher · 2024-04-16T17:33:43Z

.../src/main/java/io/micronaut/management/health/indicator/GracefulShutdownHealthIndicator.java

+    @Override
+    public Publisher<HealthResult> getResult() {
+        return Mono.just(HealthResult.builder("gracefulShutdown")
+            .status(shuttingDown ? HealthStatus.DOWN : HealthStatus.UP)


it would be nice to return some detail showing the state of what GracefulShutdownLifecycle is currently shutting down and hence we can know what is blocking the shutdown process

ive added a shutdown state mechanism

graemerocher · 2024-04-16T17:35:33Z

context/src/main/java/io/micronaut/runtime/server/GracefulShutdownManager.java

+     * @return A future that completes when all {@link GracefulShutdownLifecycle} beans have shut
+     * down
+     */
+    public CompletionStage<?> shutdownGracefully() {


what calls GracefulShutdownManager.shutdownGracefully()?

In this PR nothing. could make it a management endpoint, or just leave it to application code.

there is already a shutdown endpoint so if enabled it should be called from that, but also we would likely need to integrate the shutdown hooks we add already to handle SIGTERM

See https://dratler-shay.medium.com/how-to-optimize-graceful-shutdown-in-kubernetes-and-avoid-customer-impact-8c93ee3a6483

We need to be able to specify a termination grace period as well, in Spring this spring.lifecycle.timeout-per-shutdown-phase=20s

graemerocher · 2024-04-16T20:29:09Z

Perhaps we should try and address #6493 in this PR as well

sdelamo

we need to include documentation about this new API in the main documentation.

andrebrov · 2024-09-11T13:32:38Z

@yawkat Hello!

Wanted to see if there are any blockers around this issue since it's a quite important feature.

yawkat · 2024-09-12T08:45:33Z

No blockers, it's just on backburner

altro3 · 2024-10-11T15:52:15Z

Any news about this feature?

graemerocher · 2024-10-14T07:33:43Z

@yawkat can you prioritise this next again after the client work

yawkat added the type: enhancement New feature or request label Apr 11, 2024

yawkat added this to the 4.5.0 milestone Apr 11, 2024

yawkat requested review from graemerocher and sdelamo April 11, 2024 11:05

graemerocher requested changes Apr 16, 2024

View reviewed changes

sdelamo requested changes Apr 17, 2024

View reviewed changes

shutdown state reporting

8c36718

sdelamo linked an issue Apr 19, 2024 that may be closed by this pull request

Micronaut doesn't give the application code control of the order in which resources are gracefully shutdown/closed #6493

Open

rename

0b3ec41

sdelamo mentioned this pull request May 3, 2024

Micronaut doesn't give the application code control of the order in which resources are gracefully shutdown/closed #6493

Open

sdelamo removed this from the 4.5.0 milestone Oct 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add graceful shutdown #10701

Add graceful shutdown #10701

yawkat commented Apr 11, 2024

sonarcloud bot commented Apr 11, 2024

graemerocher Apr 16, 2024

graemerocher Apr 16, 2024

yawkat Apr 16, 2024

graemerocher Apr 16, 2024

yawkat Apr 18, 2024

graemerocher Apr 16, 2024

yawkat Apr 16, 2024

graemerocher Apr 16, 2024

yawkat Apr 19, 2024

graemerocher Apr 16, 2024

yawkat Apr 16, 2024

graemerocher Apr 16, 2024

graemerocher commented Apr 16, 2024

sdelamo left a comment

andrebrov commented Sep 11, 2024

yawkat commented Sep 12, 2024

altro3 commented Oct 11, 2024

graemerocher commented Oct 14, 2024

Add graceful shutdown #10701

Are you sure you want to change the base?

Add graceful shutdown #10701

Conversation

yawkat commented Apr 11, 2024

sonarcloud bot commented Apr 11, 2024

Quality Gate failed

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

graemerocher commented Apr 16, 2024

sdelamo left a comment

Choose a reason for hiding this comment

andrebrov commented Sep 11, 2024

yawkat commented Sep 12, 2024

altro3 commented Oct 11, 2024

graemerocher commented Oct 14, 2024