OutOfMemoryError on S3 put of large file when max heap is limited #5850

lread · 2025-02-01T06:16:06Z

Describe the bug

Starting with v2.30.9, when I attempt to s3 put a large file, and my maximum heap is limited, I suffer an OutOfMemoryException. I have not witnessed this with previous versions.

Regression Issue

Select this option if this issue appears to be a regression.

Expected Behavior

Large file puts successfully when max heap is relatively small.

Current Behavior

When heap is small, put of large file will result in OutOfMemoryException.
When file is large (2gb) will always result in OutOfMemoryException.

See reproduction below.

Reproduction Steps

I'm a clojure developer so forgive my rusty Java skills.
And apologies if I'm doing something goofy and this "just happened to work" in previous versions.

Summary

For versions 2.30.9 to 2.30.11 (latest as of this writing)

A put of a 1gb file will pass if the heap is big enough but will fail if the heap is smaller (tried 200mb and 1gb) with an OutOfMemoryError: Java heap space
A put of a 2gb file seems to always fail regardless of heap size with an OutOfMemoryError: Requested array size exceeds VM limit
I don't see problems with gets.

For versions < 2.30.9

All seems good

Test Source Code

Given src/main/java/S3Test.java:

import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.core.ResponseInputStream;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.core.sync.ResponseTransformer;
import software.amazon.awssdk.core.util.VersionInfo;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.*;
import java.io.File;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.net.URI;
import java.nio.file.Paths;
import java.util.HashMap;
import java.util.Map;

public class S3Test {

    public static void main(String[] args) throws IOException {
        System.out.printf("aws sdk version: %s%n", VersionInfo.SDK_VERSION);
        System.out.printf("max heap %dmb%n", Runtime.getRuntime().maxMemory() / 1024 / 1024);

        Map<String, String> opts = parseArgs(args);
        int fileMb = Integer.parseInt(opts.getOrDefault("file-mb", "512"));
        String bucket = "foobucket";
        String op = opts.getOrDefault("op", "put");

        S3Client s3 = createS3Client("http://127.0.0.1:9000", "foouser", "foosecret");

        try {
            // Create test bucket if it does not exist
            try {
                s3.headBucket(HeadBucketRequest.builder().bucket(bucket).build());
            } catch (S3Exception e) {
                if (e.statusCode() == 404) {
                    s3.createBucket(CreateBucketRequest.builder().bucket(bucket).build());
                } else {
                    throw e;
                }
            }

            switch (op) {
                case "put":
                    System.out.printf("Putting %dmb file%n", fileMb);
                    createTestFile("testfile", fileMb);
                    s3.putObject(PutObjectRequest.builder()
                                    .bucket(bucket)
                                    .key("testfile")
                                    .build(),
                            RequestBody.fromFile(new File("testfile")));
                    break;

                case "get":
                    System.out.println("Getting file");
                    File destFile = new File("testfile.down");
                    if (destFile.exists()) {
                        destFile.delete();
                    }
                    s3.getObject(GetObjectRequest.builder()
                                    .bucket(bucket)
                                    .key("testfile")
                                    .build(),
                            ResponseTransformer.toFile(Paths.get("testfile.down")));
                    System.out.printf("Downloaded file: %.2fmb%n", destFile.length() / 1024.0 / 1024.0);
                    break;

                default:
                    throw new IllegalArgumentException("Invalid operation: " + op);
            }
        } finally {
            s3.close();
        }
    }

    private static Map<String, String> parseArgs(String[] args) {
        Map<String, String> opts = new HashMap<>();
        for (String arg : args) {
            String[] parts = arg.split("=", 2);
            if (parts.length == 2) {
                opts.put(parts[0], parts[1]);
            }
        }
        return opts;
    }

    private static void createTestFile(String filePath, int sizeMb) throws IOException {
        long sizeBytes = (long) sizeMb * 1024 * 1024;
        try (RandomAccessFile file = new RandomAccessFile(filePath, "rw")) {
            file.setLength(sizeBytes);
        }
    }

    private static S3Client createS3Client(String endpoint, String accessKey, String secretKey) {
        return S3Client.builder()
                .region(Region.AWS_GLOBAL)
                .endpointOverride(URI.create(endpoint))
                .credentialsProvider(StaticCredentialsProvider.create(
                        AwsBasicCredentials.create(accessKey, secretKey)))
                .build();
    }
}

And ./pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.example</groupId>
  <artifactId>s3-test</artifactId>
  <version>1.0-SNAPSHOT</version>
  <properties>
    <maven.compiler.release>11</maven.compiler.release>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <sdk.version>2.30.8</sdk.version>
    <max.heap>4096m</max.heap>
    <test.op>put</test.op>
    <file.mb>1</file.mb>
  </properties>
  <dependencies>
    <dependency>
      <groupId>software.amazon.awssdk</groupId>
      <artifactId>s3</artifactId>
      <version>${sdk.version}</version>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-simple</artifactId>
      <version>2.0.16</version>
    </dependency>
  </dependencies>
  <build>
    <plugins>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>exec-maven-plugin</artifactId>
        <version>3.5.0</version>
        <configuration>
          <executable>java</executable>
          <arguments >
            <argument>-classpath</argument>
            <classpath/>
            <argument>-Xmx${max.heap}</argument>
            <argument>S3Test</argument>
            <argument>op=${test.op}</argument>
            <argument>file-mb=${file.mb}</argument>
          </arguments>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

Environment Setup

I went with a local minio server as my S3 target launched like so:

docker run -p 9000:9000 -p 9001:9001 --name minio \
  -e "MINIO_ROOT_USER=foouser" \
  -e "MINIO_ROOT_PASSWORD=foosecret" \
  minio/minio server /data --console-address ":9001"

Java version is:

$ java --version
openjdk 23.0.1 2024-10-15
OpenJDK Runtime Environment Temurin-23.0.1+11 (build 23.0.1+11)
OpenJDK 64-Bit Server VM Temurin-23.0.1+11 (build 23.0.1+11, mixed mode, sharing)

Sanity Test

(I'll leave out the copious mvn INFO output)

Let's put a 1mb file with a max heap of 4gb using v2.30.11

$ mvn clean compile exec:exec -Dtest.op=put -Dfile.mb=1 -Dmax.heap=4096m -Dsdk.version=2.30.11
aws sdk version: 2.30.11
max heap 4096mb
Putting 1mb file

Ok now let's fetch that same file

$ mvn clean compile exec:exec -Dtest.op=get -Dmax.heap=4096m -Dsdk.version=2.30.11
aws sdk version: 2.30.11
max heap 4096mb
Getting file
Downloaded file: 1.00mb

All looks fine

Reproduction

[fail] v2.30.11 put 2g file 200mb heap

Ok, now let's try to put a 2g file with a max heap of 200mb using v2.30.11

$ mvn clean compile exec:exec -Dtest.op=put -Dfile.mb=2048 -Dmax.heap=200m -Dsdk.version=2.30.11
aws sdk version: 2.30.11
max heap 200mb
Putting 2048mb file
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:277)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)
	at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:405)
	at software.amazon.awssdk.core.io.SdkFilterInputStream.read(SdkFilterInputStream.java:66)
	at software.amazon.awssdk.http.auth.aws.internal.signer.io.ChecksumInputStream.read(ChecksumInputStream.java:52)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:136)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.getChunk(ChunkedEncodedInputStream.java:112)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.currentChunk(ChunkedEncodedInputStream.java:95)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:90)
	at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:140)
	at software.amazon.awssdk.http.apache.internal.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:157)
	at org.apache.http.impl.execchain.RequestEntityProxy.writeTo(RequestEntityProxy.java:121)
	at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
	at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152)
	at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:252)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:103)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:229)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:226)
	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:102)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:79)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:57)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:355)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:253)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:903)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:863)
    at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:454)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke (Method.java:580)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)

[pass] v2.30.8 put 2g file 200mb heap

Let's try the same again but with v2.30.8

$ mvn clean compile exec:exec -Dtest.op=put -Dfile.mb=2048 -Dmax.heap=200m -Dsdk.version=2.30.8
aws sdk version: 2.30.8
max heap 200mb
Putting 2048mb file

It works.

[pass] v2.30.8 get 2gb file 200mb heap

And with the same limited heap, lets get the file with v2.30.8

$ mvn clean compile exec:exec -Dtest.op=get -Dmax.heap=200m -Dsdk.version=2.30.8
aws sdk version: 2.30.8
max heap 200mb
Getting file
Downloaded file: 2048.00mb

works fine.

[pass] v2.30.11 get 2gb file 200mb heap

We don't seem to have issues with get on sdk v2.30.11

$ mvn clean compile exec:exec -Dtest.op=get -Dmax.heap=200m -Dsdk.version=2.30.11
aws sdk version: 2.30.11
max heap 200mb
Getting file
Downloaded file: 2048.00mb

all is good.

[fail] v2.30.10 put 2gb file 200mb heap

$ mvn clean compile exec:exec -Dtest.op=put -Dfile.mb=2048 -Dmax.heap=200m -Dsdk.version=2.30.10
aws sdk version: 2.30.10
max heap 200mb
Putting 2048mb file
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:277)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)
	at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:405)
	at software.amazon.awssdk.core.io.SdkFilterInputStream.read(SdkFilterInputStream.java:66)
	at software.amazon.awssdk.http.auth.aws.internal.signer.io.ChecksumInputStream.read(ChecksumInputStream.java:52)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:136)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.getChunk(ChunkedEncodedInputStream.java:112)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.currentChunk(ChunkedEncodedInputStream.java:95)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:90)
	at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:140)
	at software.amazon.awssdk.http.apache.internal.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:157)
	at org.apache.http.impl.execchain.RequestEntityProxy.writeTo(RequestEntityProxy.java:121)
	at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
	at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152)
	at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:252)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:103)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:229)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:226)
	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:102)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:79)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:57)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:355)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:253)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:903)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:863)
    at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:454)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke (Method.java:580)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)

[fail] v2.30.9 put 2gb file 200mb heap

$ mvn clean compile exec:exec -Dtest.op=put -Dfile.mb=2048 -Dmax.heap=200m -Dsdk.version=2.30.9
aws sdk version: 2.30.9
max heap 200mb
Putting 2048mb file
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:277)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)
	at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:405)
	at software.amazon.awssdk.core.io.SdkFilterInputStream.read(SdkFilterInputStream.java:66)
	at software.amazon.awssdk.http.auth.aws.internal.signer.io.ChecksumInputStream.read(ChecksumInputStream.java:52)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:136)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.getChunk(ChunkedEncodedInputStream.java:112)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.currentChunk(ChunkedEncodedInputStream.java:95)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:90)
	at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:140)
	at software.amazon.awssdk.http.apache.internal.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:157)
	at org.apache.http.impl.execchain.RequestEntityProxy.writeTo(RequestEntityProxy.java:121)
	at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
	at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152)
	at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:252)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:103)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:229)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:226)
	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:102)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:79)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:57)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:355)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:253)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:903)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:863)
    at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:454)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke (Method.java:580)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)

[fail] v2.30.11 put 2gb file 2gb heap

Maybe 200mb is silly small, let's try a 2gb heap.

$ mvn clean compile exec:exec -Dtest.op=put -Dfile.mb=2048 -Dmax.heap=2048m -Dsdk.version=2.30.9
aws sdk version: 2.30.11
max heap 2048mb
Putting 2048mb file
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:277)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)
	at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:405)
	at software.amazon.awssdk.core.io.SdkFilterInputStream.read(SdkFilterInputStream.java:66)
	at software.amazon.awssdk.http.auth.aws.internal.signer.io.ChecksumInputStream.read(ChecksumInputStream.java:52)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:136)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.getChunk(ChunkedEncodedInputStream.java:112)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.currentChunk(ChunkedEncodedInputStream.java:95)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:90)
	at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:140)
	at software.amazon.awssdk.http.apache.internal.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:157)
	at org.apache.http.impl.execchain.RequestEntityProxy.writeTo(RequestEntityProxy.java:121)
	at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
	at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152)
	at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:252)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:103)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:229)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:226)
	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:102)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:79)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:57)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:355)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:253)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:903)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:863)
    at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:454)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke (Method.java:580)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)

[fail] v2.30.11 put 2gb file 8gb heap

Let's give ourselves some elbow room.

$ mvn clean compile exec:exec -Dtest.op=put -Dfile.mb=2048 -Dmax.heap=8192m -Dsdk.version=2.30.11
aws sdk version: 2.30.11
max heap 8192mb
Putting 2048mb file
Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:277)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)
	at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:405)
	at software.amazon.awssdk.core.io.SdkFilterInputStream.read(SdkFilterInputStream.java:66)
	at software.amazon.awssdk.http.auth.aws.internal.signer.io.ChecksumInputStream.read(ChecksumInputStream.java:52)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:136)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.getChunk(ChunkedEncodedInputStream.java:112)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.currentChunk(ChunkedEncodedInputStream.java:95)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:90)
	at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:140)
	at software.amazon.awssdk.http.apache.internal.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:157)
	at org.apache.http.impl.execchain.RequestEntityProxy.writeTo(RequestEntityProxy.java:121)
	at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
	at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152)
	at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:252)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:103)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:229)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:226)
	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:102)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:79)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:57)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:355)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:253)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:903)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:863)
    at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:454)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke (Method.java:580)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)

Huh! Notice this time though: Requested array size exceeds VM limit, a different issue maybe.

[pass] v2.30.11 put 1gb file 8gb heap

Let's try a smaller file then:

$ mvn clean compile exec:exec -Dtest.op=put -Dfile.mb=1024 -Dmax.heap=8192m -Dsdk.version=2.30.11 
aws sdk version: 2.30.11
max heap 8192mb
Putting 1024mb file

Ok that worked.

[fail] v2.30.11 put 1gb file 1gb heap

But if we reduce the heap to 1gb, we fail.

$ mvn clean compile exec:exec -Dtest.op=put -Dfile.mb=1024 -Dmax.heap=1024m -Dsdk.version=2.30.11
aws sdk version: 2.30.11
max heap 1024mb
Putting 1024mb file
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:277)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)
	at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:405)
	at software.amazon.awssdk.core.io.SdkFilterInputStream.read(SdkFilterInputStream.java:66)
	at software.amazon.awssdk.http.auth.aws.internal.signer.io.ChecksumInputStream.read(ChecksumInputStream.java:52)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:136)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.getChunk(ChunkedEncodedInputStream.java:112)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.currentChunk(ChunkedEncodedInputStream.java:95)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:90)
	at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:140)
	at software.amazon.awssdk.http.apache.internal.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:157)
	at org.apache.http.impl.execchain.RequestEntityProxy.writeTo(RequestEntityProxy.java:121)
	at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
	at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152)
	at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:252)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:103)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:229)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:226)
	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:102)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:79)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:57)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:355)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:253)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:903)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:863)
    at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:454)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke (Method.java:580)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)

Possible Solution

In the 2.30.9 changelog I see:

AWS SDK for Java v2

Features
Buffer input data from ContentStreamProvider in cases where content length is known.

Feels like the new behaviour I am witnessing might be related to this change.

Additional Information/Context

I've been using the aws sdk for a long while without any issues.
Thanks for all the hard work!

AWS Java SDK version used

2.30.8,2.30.9,2.30.10,2.30.11

JDK version used

openjdk 23.0.1 2024-10-15

Operating System and version

Linux pop-os 6.9.3-76060903-generic #202405300957~~1736980680~~22.04~44ea8a9 SMP PREEMPT_DYNAMIC Thu J x86_64 x86_64 x86_64 GNU/Linux

The text was updated successfully, but these errors were encountered:

scarytom · 2025-02-03T09:41:36Z

Just to add some context here: we have been experiencing multiple out-of-memory crashes of our systems since updating to the latest AWS SDK version, and this bug seems to be the cause.

Rolling back to v2.30.8 has got things working again.

bhoradc · 2025-02-03T16:51:51Z

Hello @lread,

Thank you for providing detailed information about the issue you encountered and the steps you took to reproduce it.

I tested below cases to confirm the OutOfMemoryError pre and post ContentStreamProvider changes released in v2.30.9.

With maximum heap size of 4096MB, the results were as follows for a PUT S3 operation.

Java SDK version 2.30.8 and 2.30.9:

With a 512MB file, the operation succeeded.

Java SDK version 2.30.9:

With a 1024MB file, the operation failed with a "Java heap space" OutOfMemoryError.

Java SDK version 2.30.8:

With a 1024MB file, the operation succeeded.

While the above mentioned buffering changes on v2.30.9 seems related to this issue, we'll investigate this further and provide an update.

Regards,
Chaitanya

lread · 2025-02-03T17:56:05Z

Thanks for sharing @scarytom!

Thanks for the follow up @bhoradc! You might also want to try with a 2gb file.
For versions >= 2.30.9 it seems to always throw regardless of heap size with the slightly different OutOfMemoryError: Requested array size exceeds VM limit

lread added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Feb 1, 2025

lread mentioned this issue Feb 1, 2025

ops: cljdoc docker container is sporadically dying cljdoc/cljdoc#996

Open

bhoradc added p1 This is a high priority issue potential-regression Marking this issue as a potential regression to be checked by team member and removed needs-triage This issue or PR still needs to be triaged. labels Feb 3, 2025

zoewangg mentioned this issue Feb 4, 2025

Fixed an issue that the SDK unnecessarily bufferred the entire conten… #5855

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OutOfMemoryError on S3 put of large file when max heap is limited #5850

OutOfMemoryError on S3 put of large file when max heap is limited #5850

lread commented Feb 1, 2025 •

edited

Loading

scarytom commented Feb 3, 2025

bhoradc commented Feb 3, 2025

lread commented Feb 3, 2025

OutOfMemoryError on S3 put of large file when max heap is limited #5850

OutOfMemoryError on S3 put of large file when max heap is limited #5850

Comments

lread commented Feb 1, 2025 • edited Loading

Describe the bug

Regression Issue

Expected Behavior

Current Behavior

Reproduction Steps

Summary

Test Source Code

Environment Setup

Sanity Test

Reproduction

[fail] v2.30.11 put 2g file 200mb heap

[pass] v2.30.8 put 2g file 200mb heap

[pass] v2.30.8 get 2gb file 200mb heap

[pass] v2.30.11 get 2gb file 200mb heap

[fail] v2.30.10 put 2gb file 200mb heap

[fail] v2.30.9 put 2gb file 200mb heap

[fail] v2.30.11 put 2gb file 2gb heap

[fail] v2.30.11 put 2gb file 8gb heap

[pass] v2.30.11 put 1gb file 8gb heap

[fail] v2.30.11 put 1gb file 1gb heap

Possible Solution

Additional Information/Context

AWS Java SDK version used

JDK version used

Operating System and version

scarytom commented Feb 3, 2025

bhoradc commented Feb 3, 2025

lread commented Feb 3, 2025

lread commented Feb 1, 2025 •

edited

Loading