Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom logic for uploading indexes to S3 #12

Merged
merged 3 commits into from
Nov 12, 2024
Merged

Custom logic for uploading indexes to S3 #12

merged 3 commits into from
Nov 12, 2024

Conversation

odisseus
Copy link
Collaborator

@odisseus odisseus commented Nov 6, 2024

The logic of S3Upload mostly replicates s3-upload.kt, but the code is based on Scala futures instead of Kotlin coroutines.

I have tested this implementation manually by uploading indexes for 9 JDKs (61 files total). I have also tested two failure modes (request timeout and incorrect bucket name).

@odisseus odisseus self-assigned this Nov 6, 2024
@lukaszwawrzyk
Copy link
Collaborator

Make sure that the code is robust:

  • we fail fast for obvious error like bucket name is wrong or whatever. Likely all parallel jobs will return quickly with error and that should do it. The error should be clearly visible. We don't want it to hang without output like the original solution.
  • introduce some degree of resiliency from infrastructure problems, i.e. if connection drops or we have some timeout on connecting in random file upload or whatever, we should retry configurable number of times, maybe even with some backoff, to ensure that at least we try a few times before failing the whole job.

@odisseus odisseus force-pushed the custom-s3 branch 2 times, most recently from ba5515c to 5b67887 Compare November 12, 2024 06:21
@odisseus
Copy link
Collaborator Author

It turns out that our S3 client already has some kind of retry functionality (see software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage).

@lukaszwawrzyk lukaszwawrzyk merged commit 946c326 into main Nov 12, 2024
2 checks passed
@odisseus odisseus deleted the custom-s3 branch November 13, 2024 02:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants