Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modified the GCS Backend to use transferManager package for Downloads in Parallel #367

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

gkeesh7
Copy link
Collaborator

@gkeesh7 gkeesh7 commented Oct 1, 2024

  • There was an issue with Large Blob Downloads failing because they were exceeding the buffer Guard config in the GCSBackend.
  • This change addressed that by doing the following modifications

Modifications

  1. Added the dependency of Transfer Manager package.
  2. Updated the GCSImpl to have a *transfermanager.Downloaderfield
  3. Modified the GCSImpl download function to make use of the Downloader.

@gkeesh7 gkeesh7 added the bug Something isn't working label Oct 1, 2024
@gkeesh7 gkeesh7 self-assigned this Oct 1, 2024
@gkeesh7 gkeesh7 requested a review from zachcheu October 1, 2024 11:29
@Anton-Kalpakchiev
Copy link
Collaborator

Should we wait until we have properly tested this internally before making the change to the external repo? We can then make all our changes with confidence.

On another note, this CICD is extremely flaky, we should fix this soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants