Getting "closed pool" error under high load testing

[EmadBinAbid]

I was performing some DOS load testing for my application. When I increased the load high enough, the overall CPU usage increased to the level where Kubernetes decided to restart the main server pod (let's call it A). This started the application shutdown procedure, which resulted in "closed pool" error entries from DB operations in the logs.

The code to gracefully shutdown the server in my application looks something like below (simplified version):

// ... `pgxpool.Pool` instance is created above and stored in the `p` variable

defer p.Close()
srv := NewServer()
var wg sync.WaitGroup
CloseServerHandler(&wg, srv)

fmt.Println("Starting server")
StartServer(srv)
fmt.Println("Waiting for shutdown ...")
wg.Wait()
fmt.Println("Exiting main thread")

CloseServerHandler is a helper function that catches application termination signals and broadcasts termination over the terminate channel by closing it. It also runs the shutdown operation on the server and waits for it to finish. It calls Shutdown() function from go/src/net/http package, which guarantees that the shutdown routine waits indefinitely for connections to return to idle and then shut down.

I have p.Close() in the defer block, so it will only run once the shutdown process is completed gracefully and the program control returns to the main thread. So, at this point, I don't really feel like doubting the positioning of p.Close().

Curious if there is something that might be causing the pool to get closed prematurely because of high load testing?

I am using v5.4.3 of pgx.

[EmadBinAbid]

@jackc

[jackc]

I don't see how that could happen. Calling Close should be the only way for the pool to close.

I would suggest either running it locally with a debugger and breakpoint on Pool.Close or using a forked copy of pgx with a panic in Pool.Close to see when it is being triggered.