Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix google cloud keep alive settings #876

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

mdedetrich
Copy link
Contributor

@mdedetrich mdedetrich commented Oct 24, 2024

So when trying to upload a file to google cloud storage I received the following error

org.apache.pekko.http.impl.engine.client.OutgoingConnectionBlueprint$UnexpectedConnectionClosureException: The http server closed the connection unexpectedly before delivering responses for 1 outstanding requests
    at org.apache.pekko.http.impl.engine.client.OutgoingConnectionBlueprint$.$anonfun$apply$7(OutgoingConnectionBlueprint.scala:135)
    at org.apache.pekko.http.impl.engine.client.OutgoingConnectionBlueprint$.$anonfun$apply$7$adapted(OutgoingConnectionBlueprint.scala:135)
    at org.apache.pekko.http.impl.util.One2OneBidiFlow$One2OneBidi$$anon$1$$anon$4.onUpstreamFinish(One2OneBidiFlow.scala:112)
    at org.apache.pekko.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:537)
    at org.apache.pekko.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:401)
    at org.apache.pekko.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:662)
    at org.apache.pekko.stream.impl.fusing.ActorGraphInterpreter$SimpleBoundaryEvent.execute(ActorGraphInterpreter.scala:71)
    at org.apache.pekko.stream.impl.fusing.ActorGraphInterpreter$SimpleBoundaryEvent.execute$(ActorGraphInterpreter.scala:67)
    at org.apache.pekko.stream.impl.fusing.ActorGraphInterpreter$BatchingActorInputBoundary$OnComplete.execute(ActorGraphInterpreter.scala:103)
    at org.apache.pekko.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:637)
    at org.apache.pekko.stream.impl.fusing.ActorGraphInterpreter.org$apache$pekko$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:813)
    at org.apache.pekko.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:831)
    at org.apache.pekko.actor.Actor.aroundReceive(Actor.scala:547)
    at org.apache.pekko.actor.Actor.aroundReceive$(Actor.scala:545)
    at org.apache.pekko.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:729)
    at org.apache.pekko.actor.ActorCell.receiveMessage(ActorCell.scala:590)
    at org.apache.pekko.actor.ActorCell.invoke(ActorCell.scala:557)
    at org.apache.pekko.dispatch.Mailbox.processMailbox(Mailbox.scala:272)
    at org.apache.pekko.dispatch.Mailbox.run(Mailbox.scala:233)
    at org.apache.pekko.dispatch.Mailbox.exec(Mailbox.scala:245)
    at java.util.concurrent.ForkJoinTask.doExec
    at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec
    at java.util.concurrent.ForkJoinPool.scan
    at java.util.concurrent.ForkJoinPool.runWorker
    at java.util.concurrent.ForkJoinWorkerThread.run

I did some reading at googleapis/nodejs-storage#101 (comment) and https://pekko.apache.org/docs/pekko-http/current/configuration.html, my current theory is that we need to make sure that both the connection pool has its keep alive timeout set to infinity and any requests using the Connection: Keep Alive header explicitly

@raboof @jrudolph Would be good if you manage to look into this, I am not 100% sure that this will actually solve the underlying issue, its just based off of what I could find on the internet.

@mdedetrich mdedetrich force-pushed the fix-google-cloud-keep-alive-settings branch 3 times, most recently from 8a36d3d to dd2bd5e Compare October 24, 2024 11:47
@@ -83,7 +83,7 @@ class FcmSenderSpec
val request: HttpRequest = captor.getValue
Unmarshal(request.entity).to[FcmSend].futureValue shouldBe FcmSend(false, FcmNotification.empty)
request.uri.toString should startWith("https://fcm.googleapis.com/v1/projects/projectId/messages:send")
request.headers.size shouldBe 1
request.headers.size shouldBe 2
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we have added an extra keep alive header we know return 2 headers, but we still only check the first header for what we expect

@mdedetrich mdedetrich force-pushed the fix-google-cloud-keep-alive-settings branch from dd2bd5e to f6e594f Compare October 24, 2024 12:08
@mdedetrich mdedetrich force-pushed the fix-google-cloud-keep-alive-settings branch from f6e594f to 19fa066 Compare October 24, 2024 12:56
@pjfanning pjfanning added this to the 1.1.0 milestone Oct 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants