Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing ETag header when downloading files from s3proxy #615

Open
bthdimension opened this issue Mar 12, 2024 · 5 comments
Open

Missing ETag header when downloading files from s3proxy #615

bthdimension opened this issue Mar 12, 2024 · 5 comments

Comments

@bthdimension
Copy link

When I do a streaming download using the aws adk, I get the following error:
software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Response missing required ETag header.

The streaming is done as follows:

s3AsyncClient.getObject(
                r -> r.bucket(bucket).key(key),
                AsyncResponseTransformer.toBlockingInputStream());
InputStream inputStream = responseFuture.join();
IOUtils.copy(inputStream, fileOutputStream);

I'm running s3proxy within a JUnit test and set it up as follows:

        Properties properties = new Properties();
        properties.setProperty(S3ProxyConstants.PROPERTY_AUTHORIZATION, "aws-v2-or-v4");
        properties.setProperty(S3ProxyConstants.PROPERTY_IDENTITY, accessKeyId);
        properties.setProperty(S3ProxyConstants.PROPERTY_CREDENTIAL, secretAccessKey);
        properties.setProperty(S3ProxyConstants.PROPERTY_IGNORE_UNKNOWN_HEADERS, "true");
        properties.setProperty(S3ProxyConstants.PROPERTY_ENDPOINT, endpoint);
        properties.setProperty(FilesystemConstants.PROPERTY_BASEDIR, TestFileManagement.S3_BASE_DIRECTORY);

        context = ContextBuilder
                .newBuilder("filesystem")
                .credentials("identity", "credential")
                .overrides(properties)
                .build(BlobStoreContext.class);

        s3Proxy = S3Proxy.Builder.fromProperties(properties)
                .blobStore(context.getBlobStore())
                .build();

There are some randomly generated files in the S3_BASE_DIRECTORY.

My s3proxy version is 1.9.0 due to the issue mentioned here (see my last comment): #535
My aws sdk version is:

  • software.amazon.awssdk:s3:2.24.3
  • software.amazon.awssdk.crt:aws-crt:0.29.10
@gaul
Copy link
Owner

gaul commented Mar 12, 2024

S3Proxy includes the ETag if it is exists. For the filesystem backend, this is only present if S3Proxy created the object which stores the tag in the extended attributes if they are supported (I believe all platforms with modern Java versions). If you are calling getBlob on an objected created outside S3Proxy, e.g., exposing an existing directory via S3, this attribute will not be present and thus S3Proxy will not include the header. I am curious about Response missing required ETag header. -- this was not always a requirement so perhaps there is some regression in the AWS library or configuration you need to use?

@bthdimension
Copy link
Author

Thank you very much for the quick answer. If I use the S3 client to upload the files, they indeed have the ETag. Is there a way to set these attributes manually? Otherwise I'll just use this workaround.
Thanks!

@gaul
Copy link
Owner

gaul commented Mar 13, 2024

S3Proxy (via Apache jclouds) previously tried computing ETag on demand when it was not available in apache/jclouds@496e27f. This was reverted in apache/jclouds@86e947d since it was computationally expensive for large objects, particularly when listing objects.

Could you share why your library is enforcing the presence of ETag? This should have a configuration knob to disable the behavior.

@bthdimension
Copy link
Author

The libraries I'm using are 'software.amazon.awssdk:s3:2.24.3' and 'software.amazon.awssdk.crt:aws-crt:0.29.10'. The client that seems to be enforcing it is "software.amazon.awssdk.services.s3.S3AsyncClient" and it is specifically the CRT client.

I create the client as follows:

S3AsyncClient.crtBuilder()
                .endpointOverride(URI.create(url))
                .forcePathStyle(true)
                .credentialsProvider(StaticCredentialsProvider.create(credentials))
                .region(Region.US_EAST_1)
                .targetThroughputInGbps(targetThroughputInGbps)
                .minimumPartSizeInBytes(minimumPartSizeInBytes)
                .checksumValidationEnabled(false)
                .build()

When creating the CRT client, I' can't really find an option to disable the ETag check. I was only able to disable the checksum validation.

@zjl578254155
Copy link

.forcePathStyle(true)

Hello, may I ask if this issue has been resolved?
I am creating the download file in this way, but there is still an error message saying that Etag is missing. I don't know where the problem lies?
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants