Max Content Length

[djchapm]

Hello - in my never ending pursuit for a way to stream files to s3 when we don't know the content length at the time we start the request, I've refreshed some old code and started playing with TransferManager again using AsyncRequestBody.fromPublisher and Flux...

This is sort of related to the recent #2908 and the more long term request aws/aws-sdk-java#474

What is really great - is that I can do this if I incorrectly set the size to something arbitrarily high. For example, this works fine:

  S3AsyncClient s3AsyncClient =
                S3AsyncClient.crtBuilder()
                        .credentialsProvider(DefaultCredentialsProvider.create())
                        .region(Region.US_EAST_1)
                        .targetThroughputInGbps(20.0)
                        .minimumPartSizeInBytes(100000L)
                        .build();
        S3TransferManager transferManager = S3TransferManager.builder()
                .s3Client(s3AsyncClient)
                .build();
        Flux<ByteBuffer> flux = Flux.just("one", "two", "three")
                .map(val -> ByteBuffer.wrap(val.getBytes()));
        //verify flux:
        //flux.subscribe(System.out::println);

        UploadRequest.Builder req = UploadRequest.builder();
        req.requestBody(AsyncRequestBody.fromPublisher(flux));
        req.addTransferListener(LoggingTransferListener.create());
        req.putObjectRequest(b -> b.bucket(BUCKET).key(FILE).contentLength(5000000L));
        Upload upload = transferManager.upload(req.build());
        CompletedUpload completedUpload = upload.completionFuture().join();
        System.out.println(completedUpload.response());

So even though I'm setting the contentLength to ~5M bytes and uploading content of "onetwothree", TransferManager takes care of it and the file looks fine in S3. When I pull it down the actual content is 11 bytes as expected, not like it's padded with spaces or anything. So it seems that if my async publisher completes, the s3Client correctly stops and closes the stream at that point. Let me know if you see any issues with this.

So this is really great but limited - if I try to go higher than 5M - I start getting exceptions that my upload is smaller than the minimum size - the limit is somewhat random - which makes me think underlying native libs are doing something with respect to available runtime memory?

software.amazon.awssdk.services.s3.model.S3Exception: Your proposed upload is smaller than the minimum allowed size (Service: S3, Status Code: 400

So my question is - how is this calculated? And is there any chance we could take this a step further so I can just set the size to something like a terrabyte while maintaining the publisher aspect and minimum part size?

Thanks!

[djchapm]

This is working now with integration a couple days ago of awslabs/aws-c-s3#285. No longer need to specify content size.

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.