JSSE: fix SSLEngine.unwrap() data delivery and close_notify handling after closeOutbound()

[cconlon]

This PR fixes a hang in the SunJSSE interop test SSLEngine/NoAuthClientAuth.java when wolfJSSE is run against wolfSSL master containing 07d4174 (wolfSSL/wolfssl#10863), along with silent discarding of application data received after closeOutbound().

Problem

When an application calls SSLEngine.closeOutbound() while inbound is still open, unwrap() routed all processing through ClosingConnection() / wolfSSL_shutdown(). This caused two issues:

1. Application data records arriving after closeOutbound() were decrypted into native wolfSSL's internal buffer but never delivered to the application (silently discarded).
2. As of Fix use-after-free in some TLS shutdown/ReceiveData sequences wolfssl#10863, wolfSSL_shutdown() returns WOLFSSL_SHUTDOWN_NOT_DONE without processing incoming records while decrypted application data is still buffered. With unread data pending, the peer's close_notify was never processed, isInboundDone() never returned true, and applications looping on engine close status spun forever.

The NoAuthClientAuth.java close sequence (server closeOutbound() while the client still sends application data followed by close_notify) hits exactly this state.

Fix

Only route unwrap() into ClosingConnection() when inbound is already closed or the handshake is not finished. Otherwise fall through to the normal receive path, which delivers pending application data and processes the peer's close_notify via RecvAppData().

The WolfSSLSocket path is unaffected: it makes a single wolfSSL_shutdown() call per connection (unidirectional close), never reaching the bidirectional branch where the native pending-data check lives.

Testing

Added testCloseOutboundWithPendingAppData to WolfSSLEngineTest: drives two engines through the NoAuthClientAuth close sequence with a bounded wrap/unwrap loop, asserting app data sent during close is delivered and both engines fully close.

[Copilot]

Pull request overview

This PR adjusts WolfSSLEngine.unwrap() behavior after closeOutbound() so that, when inbound is still open and the handshake is complete, unwrap continues through the normal receive path to deliver any pending peer application data and process the peer’s close_notify (preventing hangs in close loops). It also adds a regression test that reproduces the SunJSSE interop close sequence that previously could spin indefinitely.

Changes:

• Update WolfSSLEngine.unwrap() to only route into ClosingConnection() when inbound is already closed or the handshake is not finished, avoiding discarding buffered application data after closeOutbound().
• Add testCloseOutboundWithPendingAppData to validate app-data delivery during close and ensure both engines fully close within a bounded wrap/unwrap loop.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
src/java/com/wolfssl/provider/jsse/WolfSSLEngine.java	Refines unwrap shutdown routing so inbound app-data and peer close_notify are still processed after closeOutbound() when appropriate.
src/test/com/wolfssl/provider/jsse/test/WolfSSLEngineTest.java	Adds a regression test covering the close sequence with pending app data to prevent hangs and silent data loss.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.