2.x.x - Shutdown hangs forever if there are stalled TLS connections

[andreasley]

Commit: 2.0-rc.1
Environment: macOS (issue happens for both BSD sockets and Transport Services)

Issue: Graceful shutdown never completes if the server is using TLS and a connection is still open.

The following code demonstrates the issue by doing the following:

1. Starts a HTTP server
2. After one second, connects a socket (without sending any data)
3. After another two seconds, triggers graceful shutdown

import Foundation
import Hummingbird
import ServiceLifecycle
import HummingbirdTLS
import NIOSSL

@main
struct HummingbirdShutdownTesterApp {
    
    static let hostname = "127.0.0.1"
    static let port = 8080

    static func main() async throws {
        
        // use any identity for testing; in this example,`server.p12` from Hummingbird's testing assets is being used
        guard let p12Path = Bundle.main.path(forResource: "server", ofType: "p12") else {
            print("Failed to find certificate")
            exit(1)
        }

        let p12Passphrase = "HBTests"
        let p12Bundle = try NIOSSLPKCS12Bundle(file: p12Path, passphrase: p12Passphrase.utf8)
        
        var tlsConfiguration = TLSConfiguration.makeServerConfiguration(
            certificateChain: p12Bundle.certificateChain.map({ .certificate($0) }),
            privateKey: .privateKey(p12Bundle.privateKey)
        )
        
        var app = Application(
            router: Router(),
            server: try .tls(tlsConfiguration: tlsConfiguration),
            configuration: .init(address: .hostname(hostname, port: port))
        )
        
        app.logger.logLevel = .trace
        
        var serviceGroup: ServiceGroup?

        Task {
            // trigger shutdown after the specified number of seconds
            try await Task.sleep(for: .seconds(3), tolerance: .seconds(1))
            await serviceGroup?.triggerGracefulShutdown()
        }

        Task {
            // wait one second for the server to start
            try await Task.sleep(for: .seconds(1), tolerance: .seconds(1))
            // connect a single socket and keep it open
            openSocket()
        }

        serviceGroup = ServiceGroup(services: [app], logger: app.logger)
        try await serviceGroup?.run()
    }
    
    static func openSocket() {
        
        let socketDescriptor = socket(AF_INET, SOCK_STREAM, IPPROTO_IP)
        guard socketDescriptor != -1 else {
            print("Failed to create socket")
            return
        }
        
        var serverAddress = sockaddr_in()
        serverAddress.sin_family = sa_family_t(AF_INET)
        serverAddress.sin_port = in_port_t(port).bigEndian
        serverAddress.sin_addr.s_addr = inet_addr(hostname)
        
        let connectResult = withUnsafePointer(to: &serverAddress) {
            $0.withMemoryRebound(to: sockaddr.self, capacity: 1) {
                connect(socketDescriptor, $0, socklen_t(MemoryLayout<sockaddr_in>.size))
            }
        }
        
        guard connectResult != -1 else {
            print("Socket failed to connect (server not running or sandbox restriction?)")
            return
        }
        
        print("Connected socket with descriptor \(socketDescriptor); server shutdown will now hang")
    }
}

[andreasley]

The problem originates here:

hummingbird/Sources/HummingbirdCore/Server/HTTP/HTTPChannelHandler.swift

Lines 99 to 105 in d29b0de

	} onGracefulShutdown: {
	// set to cancelled
	if processingRequest.exchange(.cancelled) == .idle {
	// only close the channel input if it is idle
	asyncChannel.channel.close(mode: .input, promise: nil)
	}
	}

When closing the channel with mode .input, the function doShutdown() is never called on the NIOSSL.SSLConnection.

Setting mode to .all fixes this.

The same may apply to onCancel.

I still find SwiftNIO fairly confusing, so I'm not sure if that's a suitable fix. ;)

[adam-fowler]

Can I ask you to try two different things, separately and together

1. When you create your Application add an additional channel handler

let app = Application(
    router: router,
    server: http1(additionalChannelHandlers: [IdleStateHandler(readTimeout: .seconds(15))])
)

2. Can you try using the quiescing-helper branch of Hummingbird and see if that helps with the shutdown

[andreasley]

1. Adding the IdleStateHelper like in the following snippet (to still use TLS) didn't help:

var app = Application(
    router: Router(),
    server: try .tls(.http1(additionalChannelHandlers: [IdleStateHandler(readTimeout: .seconds(15))]), tlsConfiguration: tlsConfiguration),
    configuration: .init(address: .hostname(hostname, port: port))
)

 
2. Using the quiescing-helper branch worked and the app shut down properly.

Using both changes, the app also shut down properly.

[adam-fowler]

#453 has now been merged and will be in the next release so I am going to close this now.