S3CrtAsyncClient does not work with AnonymousCredentialsProvider

[cgoina]

Describe the bug

When the S3CrtAsyncClientBuilder is configured only with AnonymousCredentialsProvider it generates a NullPointerException.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

CrtAsyncClient should work with an open bucket when only AnonymousCredentials are set

Current Behavior

The problem is in the CrtCredentialsProviderAdapter

        this.crtCredentials = new DelegateCredentialsProvider.DelegateCredentialsProviderBuilder()
            .withHandler(() -> {

                if (credentialsProvider instanceof AnonymousCredentialsProvider) {
                    return Credentials.createAnonymousCredentials();
                }

                AwsCredentialsIdentity sdkCredentials =
                    CompletableFutureUtils.joinLikeSync(credentialsProvider.resolveIdentity());
                // next will generate the NPE - 
                // if the credentials are null it should just return new Credentials() - this worked when I ran my program in debugger
                byte[] accessKey = sdkCredentials.accessKeyId().getBytes(StandardCharsets.UTF_8);
                byte[] secreteKey = sdkCredentials.secretAccessKey().getBytes(StandardCharsets.UTF_8);

                byte[] sessionTokens = null;
                if (sdkCredentials instanceof AwsSessionCredentialsIdentity) {
                    sessionTokens =
                        ((AwsSessionCredentialsIdentity) sdkCredentials).sessionToken().getBytes(StandardCharsets.UTF_8);
                }
                return new Credentials(accessKey, secreteKey, sessionTokens);

            }).build();

Reproduction Steps

create a crt async client then run a list or getobject request against an open bucket that does not require any credentials for reading.
{
S3CrtAsyncClientBuilder asyncS3ClientBuilder = S3AsyncClient.crtBuilder();
asyncS3ClientBuilder.credentialsProvider(AnonymousCredentialsProvider.create());
S3AsyncClient asyncClient = asyncS3ClientBuilder.build();

    ListObjectsV2Request listRequest = ListObjectsV2Request.builder()
            .bucket(s3Adapter.getBucket())
            .prefix(prefix)
            .delimiter("/")
            .build();

    ListObjectsV2Publisher listObjectsV2Publisher = s3Adapter.getAsyncS3Client().listObjectsV2Paginator(listRequest);
    ......

}

Possible Solution

Change the internal.CrtCredentialsProvider's constructor

    public CrtCredentialsProviderAdapter(IdentityProvider<? extends AwsCredentialsIdentity> credentialsProvider) {
        this.credentialsProvider = credentialsProvider;
        this.crtCredentials = new DelegateCredentialsProvider.DelegateCredentialsProviderBuilder()
            .withHandler(() -> {

                if (credentialsProvider instanceof AnonymousCredentialsProvider) {
                    return Credentials.createAnonymousCredentials();
                }

                AwsCredentialsIdentity sdkCredentials =
                    CompletableFutureUtils.joinLikeSync(credentialsProvider.resolveIdentity());
                // I don't know if this is the best but when I changed the values in my debugger to avoid the NPE this worked.
                if (sdkCredentials.accessKeyId() == null || sdkCredentials.secretAccessKey() == null) return new Credentials()

                byte[] accessKey = sdkCredentials.accessKeyId().getBytes(StandardCharsets.UTF_8);
                byte[] secreteKey = sdkCredentials.secretAccessKey().getBytes(StandardCharsets.UTF_8);

                byte[] sessionTokens = null;
                if (sdkCredentials instanceof AwsSessionCredentialsIdentity) {
                    sessionTokens =
                        ((AwsSessionCredentialsIdentity) sdkCredentials).sessionToken().getBytes(StandardCharsets.UTF_8);
                }
                return new Credentials(accessKey, secreteKey, sessionTokens);

            }).build();
    }

Additional Information/Context

No response

AWS Java SDK version used

2.30.1

JDK version used

"1.8.0_422" - OpenJDK 64-Bit Server VM (Zulu 8.80.0.17-CA-macos-aarch64) (build 25.422-b05, mixed mode)

Operating System and version

Mac OS Sequoia 15.2

[debora-ito]

@cgoina can you share the full stacktrace of the NPE?

[cgoina]

This stack I got with jdk 23 but it fails at the same spot with jdk 8 as I specified in the ticket

2025-01-17_18:01:17.454 [main] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=GET, protocol=https, host=aind-msma-morphology-data.s3.us-west-2.amazonaws.com, encodedPath=, headers=[amz-sdk-invocation-id, User-Agent], queryParameters=[list-type, delimiter, prefix])
Exception in thread "AwsEventLoop 4" java.lang.NullPointerException: Cannot invoke "String.getBytes(java.nio.charset.Charset)" because the return value of "software.amazon.awssdk.identity.spi.AwsCredentialsIdentity.accessKeyId()" is null
	at software.amazon.awssdk.services.s3.internal.crt.CrtCredentialsProviderAdapter.lambda$new$0(CrtCredentialsProviderAdapter.java:51)
2025-01-17_18:01:17.464 [sdk-async-response-0-0] DEBUG s.a.a.r.i.DefaultStandardRetryStrategy - Request attempt 1 encountered non-retryable failure
software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: A callback has reported failure.
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
	at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
	at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleIoError(S3CrtResponseHandlerAdapter.java:192)
	at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:184)
	at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:154)
	at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:25)
2025-01-17_18:01:17.467 [main] DEBUG o.j.j.s.i.AbstractS3StorageService - List content segmentation/exaSPIM_653159_zarr with org.janelia.jacsstorage.service.ContentAccessParams@4ed9f7b1[filterType=<null>,selectedEntries=[],entryNamePattern=<null>,maxDepth=1,filterTypeSpecificParams={}] - 0.074 secs

java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: A callback has reported failure.

	at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:64)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:978)
	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:955)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:554)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2238)
	at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:907)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:885)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:554)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2238)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:135)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:152)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:113)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:907)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:885)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:554)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2238)
	at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:907)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:885)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:554)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2238)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:108)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:907)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:885)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:554)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2238)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167)
	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:978)
	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:955)
	at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:526)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1575)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: A callback has reported failure.
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
	at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
	at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleIoError(S3CrtResponseHandlerAdapter.java:192)
	at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:184)
	at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:154)
	at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:25)

[cgoina]

The current implementation that checks for an AnonympusCredentialsProvider instance is incomplete because.

1. I can write my own anonymous provider just like () -> MyAnonymousAWSCredentials and this will still fail
2. If the current AnonymousCredentialsProvider is first one in a AwsCredentialsProviderChain it still fails with the same error

[debora-ito]

I can't reproduce the NullPointerException.

In theory, you would not need to add the null checks in CrtCredentialsProviderAdapter because if it's using AnonymousCredentialsProvider then the code would hit return before it reaches that point.

In you code example a s3Adapter is used, does it have any custom implementation, like a custom AnonymousCredentialsProvider?

    ListObjectsV2Publisher listObjectsV2Publisher = 
        s3Adapter.getAsyncS3Client().listObjectsV2Paginator(listRequest);

[debora-ito]

Another possible cause is you're using an old version of software.amazon.awssdk.crt:aws-crt, it should be 0.33.6 with SDK version 2.30.1.

[cgoina]

I am using 'software.amazon.awssdk.crt:aws-crt:0.33.9' but the example I provided will not fail because it uses AnonymousCredentials provider and the implementation already checks that. As I mentioned use a credentials provider defined as a lambda or a chain provider that has the anonymousprovider first in the chain:

            AwsCredentialsProviderChain.Builder credentialsProviderBuilder = AwsCredentialsProviderChain.builder();
            credentialsProviderBuilder
                     .addCredentialsProvider(AnonymousCredentialsProvider.create())
                    .addCredentialsProvider(ProfileCredentialsProvider.create())
                    .addCredentialsProvider(SystemPropertyCredentialsProvider.create())
                    .addCredentialsProvider(EnvironmentVariableCredentialsProvider.create())
                    .addCredentialsProvider(InstanceProfileCredentialsProvider.create());
            credentialsProvider = credentialsProviderBuilder.build();

If you put the AnonymousCredentialsProvider last in the chain it will work. If you put it first it doesn't. In some sense this is a regresssion because the async client created by S3AsyncClient.builder() works with the above chain - the one created by crtBuilder doesn't

[debora-ito]

Okay, I see the issue now: when the AnonymousCredentialsProvider is the first in the credential provider chain, the default s3AsyncClient will pick and use it, while the S3CrtAsyncClient will throw the error. We'll look into this.

In my local tests, putting the AnonymousCredentialsProvider as last throws the error too if none of the previous providers was picked, so I would double-check in your tests if another credential provider is being picked up before the chain reaches the AnonymousCredentialsProvider.

And as a side note, since having AnonymousCredentialsProvider as first works for s3AsyncClient, this means that the other providers in the chain will never be used. Just want to be sure this is the behavior you expected to see.