tyrus-client 1.16: Failed SSL handshake leads to memory leak

[D-ADY]

I'm using the tyrus client with jdk container under JDK8.
When establishing a connection via wss, in case there is a problem with the presented certificate (in my case CertificateNotYetValidException), after some time the application throws an OutOfMemory error.

Digging deeper in the heap dump, there are a lot of instances to java.security.cert.TrustAnchor and of org.glassfish.tyrus.core.coder.CoderWrapper.
Actually, for each invocation to ClientManager#connectToServer() a whole object graph remains behind and is never garbage collected once the CertificateNotYetValidException error is thrown.

After some investigations, the problem seems to be from org.glassfish.tyrus.container.jdk.client.SslFilter.
Once the error is thrown, handleSslError() will bubble up the error to ClientFilter, but on it's way down, when hitting TaskQueueFilter::close(), it stops there.
The TaskQueueFilter will queue a close() task for SslFilter but as the taskLock is already taken (doing the handshake), the task itself is never run.
Thus, SslFilter:close() is never called and all it's resources together with the underlying TransportFilter resources are never released.

One way to fix this would be to simply add to SslFilter::handleSslError() to close() after bubbling the error:
private void handleSslError(Throwable t) {
onError(t);
close();
}
After applying this hotfix, the memory leak was solved.
Must be checked if there are no other code paths that would have other undesired side-effects.