feat(run): provides RunWithGracefulShutdown

[liufuyang]

So to allow users easily wire up a clean-up function that can be called BEFORE the root context is done.

closes #563

---

So with typical cloudrunner code initiated like this:

Before:

        err := cloudrunner.Run(func(ctx context.Context) error {
		grpcServer := cloudrunner.NewGRPCServer(ctx)
		cleanup := func() {
			logger := cloudrunner.Logger(ctx)
			select {
			case <-ctx.Done():
				logger.Warn("Root context is already done, cannot use client anymore.")
			default:
				logger.Info("Root context is still live.")
			}
		}

		defer cleanup()
		cloudrunner.Logger(ctx).Info("hello world")

		return cloudrunner.ListenGRPC(ctx, grpcServer)
	})

Output when cancel the process:

2023-11-16T16:35:50.491+0100	info	cloudrunner-go/run.go:173	up and running	{"config": {"cloudrunner": {"PORT": 8080, "K_SERVICE": "___Test_RunWithGracefulShutdown_helloWorld_ctx_cancel_should_before_clean_up_function_call2_in_go_einride_tech_cloudrunner.test", "K_REVISION": "", "K_CONFIGURATION": "", "CLOUD_RUN_JOB": "", "CLOUD_RUN_EXECUTION": "", "CLOUD_RUN_TASK_INDEX": 0, "CLOUD_RUN_TASK_ATTEMPT": 0, "CLOUD_RUN_TASK_COUNT": 0, "GOOGLE_CLOUD_PROJECT": "", "RUNTIME_SERVICEACCOUNT": "", "SERVICE_VERSION": "", "LOGGER_DEVELOPMENT": true, "LOGGER_LEVEL": "debug", "LOGGER_REPORTERRORS": false, "PROFILER_ENABLED": false, "PROFILER_MUTEXPROFILING": false, "PROFILER_ALLOCFORCEGC": true, "TRACEEXPORTER_ENABLED": false, "TRACEEXPORTER_TIMEOUT": "10s", "TRACEEXPORTER_SAMPLEPROBABILITY": 0.01, "METRICEXPORTER_ENABLED": false, "METRICEXPORTER_INTERVAL": "1m0s", "METRICEXPORTER_RUNTIMEINSTRUMENTATION": false, "METRICEXPORTER_HOSTINSTRUMENTATION": false, "METRICEXPORTER_OPENCENSUSPRODUCER": false, "SERVER_TIMEOUT": "4m50s", "CLIENT_TIMEOUT": "10s", "CLIENT_RETRY_ENABLED": true, "CLIENT_RETRY_INITIALBACKOFF": "200ms", "CLIENT_RETRY_MAXBACKOFF": "1m0s", "CLIENT_RETRY_MAXATTEMPTS": 5, "CLIENT_RETRY_BACKOFFMULTIPLIER": 2, "CLIENT_RETRY_RETRYABLESTATUSCODES": ["Unavailable", "Unknown"], "REQUESTLOGGER_MESSAGESIZELIMIT": 0, "REQUESTLOGGER_STATUSTOLEVEL": null}}, "resource": {"telemetry.sdk.language": "go", "telemetry.sdk.name": "opentelemetry", "telemetry.sdk.version": "1.20.0"}, "buildInfo": {"mainPath": "", "goVersion": "go1.21.3", "buildSettings": {}}}
2023-11-16T16:35:50.491+0100	info	cloudrunner-go/run_test.go:117	hello world
2023-11-16T16:35:50.491+0100	info	cloudrunner-go/run.go:220	watching for termination signals
2023-11-16T16:35:50.491+0100	info	cloudrunner-go/grpcserver.go:74	gRPCServer listening	{"address": ":8080"}
2023-11-16T16:35:54.624+0100	info	cloudrunner-go/run.go:227	got signal:	{"signal": "interrupt"}
2023-11-16T16:35:54.624+0100	info	cloudrunner-go/run.go:229	cloudrunner.Shutdown is not used. Canceling root context directly. Call RunWithGracefulShutdown(...) to enable graceful shutdown if preferred.
2023-11-16T16:35:54.624+0100	info	cloudrunner-go/grpcserver.go:71	gRPCServer shutting down
2023-11-16T16:35:54.624+0100	warn	cloudrunner-go/run_test.go:110	Root context is already done, cannot use client anymore.
2023-11-16T16:35:54.624+0100	info	cloudrunner-go/run.go:202	goodbye

Note

Noticing it says Root context is already done, cannot use client anymore.

---

After:

        err := cloudrunner.RunWithGracefulShutdown(func(ctx context.Context, shutdown *cloudrunner.Shutdown) error {
		grpcServer := cloudrunner.NewGRPCServer(ctx)
		cleanup := func() {
			logger := cloudrunner.Logger(ctx)
			select {
			case <-ctx.Done():
				logger.Warn("Root context is already done, cannot use client anymore.")
			default:
				logger.Info("Root context is still live.")
			}
		}

		shutdown.RegisterCancelFunc(cleanup)
		cloudrunner.Logger(ctx).Info("hello world")

		return cloudrunner.ListenGRPC(ctx, grpcServer)
	})

Output when cancel the process:

2023-11-16T16:33:04.269+0100	info	cloudrunner-go/run.go:173	up and running	{"config": {"cloudrunner": {"PORT": 8080, "K_SERVICE": "___Test_RunWithGracefulShutdown_helloWorld_ctx_cancel_should_before_clean_up_function_call2_in_go_einride_tech_cloudrunner.test", "K_REVISION": "", "K_CONFIGURATION": "", "CLOUD_RUN_JOB": "", "CLOUD_RUN_EXECUTION": "", "CLOUD_RUN_TASK_INDEX": 0, "CLOUD_RUN_TASK_ATTEMPT": 0, "CLOUD_RUN_TASK_COUNT": 0, "GOOGLE_CLOUD_PROJECT": "", "RUNTIME_SERVICEACCOUNT": "", "SERVICE_VERSION": "", "LOGGER_DEVELOPMENT": true, "LOGGER_LEVEL": "debug", "LOGGER_REPORTERRORS": false, "PROFILER_ENABLED": false, "PROFILER_MUTEXPROFILING": false, "PROFILER_ALLOCFORCEGC": true, "TRACEEXPORTER_ENABLED": false, "TRACEEXPORTER_TIMEOUT": "10s", "TRACEEXPORTER_SAMPLEPROBABILITY": 0.01, "METRICEXPORTER_ENABLED": false, "METRICEXPORTER_INTERVAL": "1m0s", "METRICEXPORTER_RUNTIMEINSTRUMENTATION": false, "METRICEXPORTER_HOSTINSTRUMENTATION": false, "METRICEXPORTER_OPENCENSUSPRODUCER": false, "SERVER_TIMEOUT": "4m50s", "CLIENT_TIMEOUT": "10s", "CLIENT_RETRY_ENABLED": true, "CLIENT_RETRY_INITIALBACKOFF": "200ms", "CLIENT_RETRY_MAXBACKOFF": "1m0s", "CLIENT_RETRY_MAXATTEMPTS": 5, "CLIENT_RETRY_BACKOFFMULTIPLIER": 2, "CLIENT_RETRY_RETRYABLESTATUSCODES": ["Unavailable", "Unknown"], "REQUESTLOGGER_MESSAGESIZELIMIT": 0, "REQUESTLOGGER_STATUSTOLEVEL": null}}, "resource": {"telemetry.sdk.language": "go", "telemetry.sdk.name": "opentelemetry", "telemetry.sdk.version": "1.20.0"}, "buildInfo": {"mainPath": "", "goVersion": "go1.21.3", "buildSettings": {}}}
2023-11-16T16:33:04.269+0100	info	cloudrunner-go/run.go:220	watching for termination signals
2023-11-16T16:33:04.269+0100	info	cloudrunner-go/run_test.go:117	hello world
2023-11-16T16:33:04.269+0100	info	cloudrunner-go/grpcserver.go:74	gRPCServer listening	{"address": ":8080"}
2023-11-16T16:33:43.245+0100	info	cloudrunner-go/run.go:227	got signal:	{"signal": "interrupt"}
2023-11-16T16:33:43.247+0100	info	cloudrunner-go/run.go:237	graceful shutdown has begun
2023-11-16T16:33:43.247+0100	info	cloudrunner-go/run_test.go:112	Root context is still live.
2023-11-16T16:33:43.247+0100	info	cloudrunner-go/run.go:241	Shutdown.shutdownFunc() has finished, meaning we will shutdown cleanly
2023-11-16T16:33:43.247+0100	info	cloudrunner-go/run.go:247	exiting by canceling root context due to shutdown signal
2023-11-16T16:33:43.247+0100	info	cloudrunner-go/grpcserver.go:71	gRPCServer shutting down
2023-11-16T16:33:43.247+0100	info	cloudrunner-go/run.go:202	goodbye

Note

Noticing it now says Root context is still live.

[liufuyang]

Old implementation makes a hard bond between the root context and SIGTERM, which means when a SIGTERM is received, the ctx will be canceled directly and user has no control of it.

New implementation breaks the bond, see trapShutdownSignal() for details. Taking some ideas from this post. https://medium.com/over-engineering/graceful-shutdown-with-go-http-servers-and-kubernetes-rolling-updates-6697e7db17cf

[liufuyang]

Using 10 seconds here as in our preferred CloudRun environment the terminationGracePeriodSeconds seems not configurable and a fixed 10s is specified in the CloudRun doc.

https://cloud.google.com/run/docs/container-contract#instance-shutdown

Perhaps I should put this in the comment as well.

[quoral]

Hey @liufuyang!

We've recently run into similar issues that this PR attempts to solve, and we've merged a very preliminary solution in #585. It does not solve the use-case for some things that you are aiming to solve in this PR however, like pull-based PubSub cloudrun instances.

There's work planned to implement those fixes down the line as well, so I'll keep you in the loop on how that goes with an upcoming ADR where we can discuss if that solution approaches a solution for your usecase. For now as I think we're gonna take that implementation in a different direction than these proposed changes, I'll close this PR and we can move the future discussions over to #563.

Thanks again!