Socket closed when pushing 3k+ submissions to Central

[lognaturel]

Software versions

Briefcase v1.17.4, macOS 10.15.4, Java 11

Problem description

2020-10-02 10:37:18,044 [ForkJoinPool-5-worker-3] INFO  o.a.http.impl.execchain.RetryExec - I/O exception (java.net.SocketException) caught when processing request to {s}->https://test.central.getodk.org:443: Socket closed
2020-10-02 10:37:18,044 [ForkJoinPool-5-worker-3] INFO  o.a.http.impl.execchain.RetryExec - Retrying request to {s}->https://test.central.getodk.org:443
2020-10-02 10:37:18,044 [ForkJoinPool-5-worker-1] ERROR o.o.briefcase.reused.job.JobsRunner - Error running Job
java.io.UncheckedIOException: java.net.SocketException: Socket closed
	at org.opendatakit.briefcase.reused.http.CommonsHttp.uncheckedExecute(CommonsHttp.java:149)
	at org.opendatakit.briefcase.reused.http.CommonsHttp.execute(CommonsHttp.java:96)
	at org.opendatakit.briefcase.push.central.PushToCentral.pushSubmission(PushToCentral.java:270)
	at org.opendatakit.briefcase.push.central.PushToCentral.lambda$null$13(PushToCentral.java:160)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
	at java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:373)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
	at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1040)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.net.SocketException: Socket closed
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
Caused by: java.net.SocketException: Socket closed

	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.http.client.fluent.Request.internalExecute(Request.java:173)
	at org.apache.http.client.fluent.Executor.execute(Executor.java:262)
	at org.opendatakit.briefcase.reused.http.CommonsHttp.uncheckedExecute(CommonsHttp.java:142)
	... 16 common frames omitted

Steps to reproduce the problem

Try to push 3k+ submissions to a Central server. This has happened to me several times and I have to close Briefcase and restart the transfer.

Expected behavior

No crash.

Other information

I haven't started doing any exploration. Maybe it's a concurrency issue? Or could the server be taking too long to respond at some point? It may very well just be intermittent and have nothing to do with the number of submissions sent.

It seems that with the same data set it failed twice at 4113 submissions sent. 4105 with the same form definition but different set of submissions. Sent several batches of ~500 forms and then an attempt to push 3000 failed at 1114. It's starting to feel like a resource management issue on the Briefcase end of things.

[lognaturel]

@getodk/testers here's a form and submissions that I was getting the failure for. It would be great to try on Windows and get a sense of whether it might be Mac-only. I'm on macOS 10.15.4, Java 11.

[kkrawczyk123]

I was able to push to Central without any problems, none crash was visible on Windows 10.

[lognaturel]

Confirming that on master as of right now (dd75f13) I'm still getting the same crash. Latest attempt was at 4402 submissions.