Lingering connections with many ApnsClient instances

[JungdoKim]

Our app send about 6 million pushes per month.
Among them, About 70% are android FCM and 30% IOS APNs
While running our service, every 20~30 days, we encounter network problem and server down problem.

below log is network log of network problem.

------- network error log -
2021-12-30 09:13:41.841 [com.rationalowl.minerva.msg.pushsend.CustomPushSender.lambda$0(CustomPushSender.java:1139)] [ERROR] ex
java.net.UnknownHostException: failed to resolve 'api.push.apple.com' after 2 queries 
	at io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1046) ~[channel_211105_1_en.jar:?]
	at io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:999) ~[channel_211105_1_en.jar:?]
	........
2021-12-30 09:13:32.888 [com.rationalowl.minerva.msg.pushsend.CustomPushSender$DefaultFcmCallback.onFailure(CustomPushSender.java:112)] [ERROR] ex
java.net.UnknownHostException: fcm.googleapis.com
	at java.net.InetAddress.getAllByName0(InetAddress.java:1281) ~[?:1.8.0-vlinx]
	at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[?:1.8.0-vlinx]
	at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[?:1.8.0-vlinx]
       .......
-----------------------------

It takes time to detect the cause of problem and finally got to know that Pushy library makes it.
Attached 21.12.30.txt file is active connection list by 'netstat -an' command at the windows server.
Attached file show that about 9000 connections are active.
Among them, about 8000 connections are what Pushy library grabs.
In the file, remote address starts with '17.188.xxx.xxx' is Apple Apns server.

If connections are inbound connections, server will be not shut down.
Unfortunately pushy library makes outbound connection when request push to the api.push.apple.com:443.
Every OS limit outbound port.
In Windows server, outbound port limit is set by MaxUserPort(default 5000, max 65000)
In linux, outbound port limit is set by net.ipv4.ip_local_port_range.

And time goes by APNs outbound connection increase day by day.
Although we request Android FCM push more than APNs, Android FCM outbound connection keeps dozens of connection.

If Pushy library grabs outbound connection without releasing.
At some point, grabed outbound connections reach to the OS limit outbound port(connection).
At that point, If server request connection to the outbound server, OS interrupt the outbound connections.
It seems DNS error will be presented at that time. And above '== network error log ===' is that.

21.12.30.txt

[jchambers]

Thanks for the detailed report!

Can you say more about how you're using Pushy? Do you have multiple ApnsClient instances? How many concurrent connections do you allow per client? If you have multiple clients, how and when do you shut them down?

[JungdoKim]

Thanks for fast response!

It councluded that it's not bug but my mis-using library.
Your questione makes me look into sources and find out cause and fix it.
Being as a result, the problem is that there's no shut down ApnsClient instance routine at my code.

To say more detail...
I have only one ApnsClient instance and set 400 concurrent connections to ApnsClient.
At first, ApnsClient instance only created at server start time and no shutdown routine. So just ApnsClient shutdown only when server shutdown. So that doesn't make problem.

But some requirement came into and ApnsClient instance should need to re-construct at server running time.
Before re-constructing ApnsClient instance, there need to clean connection associated with ApnsClient instance.
But I missed it and that makes problem.

Thanks again !

[jchambers]

Great! Glad it's resolved.

One other thing I'll offer: 400 concurrent connections for a single client is a lot. There are only ~10 APNs servers in play at any given time (you can check with dig +short api.push.apple.com). Having 400 concurrent connections means that you have 40 connections open to each of those servers, and that's unlikely to give you any performance benefits. My strong recommendation would be to limit your concurrent connection count to somewhere between 1x and 2x the size of your event loop group, which should in turn be no more than 1x to 2x the number of physical cores on your processor. Please see the "best practices" wiki page for more discussion and details.

[jchambers]

Also: I've edited the title of this discussion to more accurately reflect the problem so it's easier for other readers to find and learn from in the future. Cheers!