Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Try streaming tiffs to s3 #347

Open
EmileSonneveld opened this issue Nov 7, 2024 · 1 comment
Open

Try streaming tiffs to s3 #347

EmileSonneveld opened this issue Nov 7, 2024 · 1 comment
Assignees
@EmileSonneveld

Comments

@EmileSonneveld
Copy link
Contributor

This has been tried here: #199
But was disabled: 01b67c8#diff-eb0b0fb3836fe17a6e2feaf1496210faf468bfd995d245163cf3876af89626a9L229

We can try again for tiffs with some extra precaution.
@EmileSonneveld EmileSonneveld self-assigned this Nov 7, 2024
EmileSonneveld added a commit that referenced this issue Nov 7, 2024
@EmileSonneveld
Try uploadToS3TryFirstWithStreaming. Only on CDSE-dev and CDSE-stagin…
554590c 
…g with extra logging and a try-catch. #347
EmileSonneveld added a commit that referenced this issue Nov 8, 2024
@EmileSonneveld
Use TRY_SWIFT_STREAMING instead of SparkContext. By default on false. #…
b7442e2 
…347
EmileSonneveld added a commit that referenced this issue Nov 8, 2024
@EmileSonneveld
Align S3AsyncClient more with S3Client to test if error persists. #347
23399f9
@EmileSonneveld
Copy link
Contributor Author

@pvbouwel Would this add TRY_SWIFT_STREAMING to the environment variables on the executors?
https://git.vito.be/projects/TPT/repos/os_creodias_openeo_k8s/browse/kube_resources/applications/openeo/values_cdse-dev.yaml#23

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@EmileSonneveld EmileSonneveld

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@EmileSonneveld