AWS CRT-based S3 client 400 bad request during upload files to ceph.

[IgorekSergeevich]

Hello everyone, I encountered a problem when upload files to ceph, I use aws-crt async client. For some reason I get 400 bad request during upload (it's doesn't matter multipart upload or just upload). Upload to aws s3 and minio works fine. Maybe you have any ideas what it could be?

Dependencies which i use:

implementation "software.amazon.awssdk:s3:2.28.6"
implementation "software.amazon.awssdk.crt:aws-crt:0.33.9"

What i see in console:

software.amazon.awssdk.services.s3.model.S3Exception: (Service: S3, Status Code: 400, Request ID: tx00000e6ef5fccbf3b15a0-0067921b65-513bf62-ceph-objectstore)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 

Original Stack Trace:
		at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
		at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
		at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
		at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
		at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:279)
		at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:92)
		at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
		at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
		at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
		at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:135)
		at software.amazon.awssdk.core.internal.metrics.BytesReadTrackingPublisher$BytesReadTracker.onComplete(BytesReadTrackingPublisher.java:74)
		at software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:275)
		at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
		at software.amazon.awssdk.utils.async.SimplePublisher.complete(SimplePublisher.java:157)
		at java.base/java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:819)
		at java.base/java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:803)
		at java.base/java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2195)
		at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onErrorResponseComplete(S3CrtResponseHandlerAdapter.java:200)
		at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:179)
		at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:148)
		at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:25)

What i see in ceph log file:

"POST /customers-bucket/76/94/559476/bosch-annual-report-2023%20%281%29.pdf?uploads HTTP/1.1" 200 307 - "aws-sdk-java/2.28.6 md/io#async md/http#s3crt ua/2.0 os/Linux#5.15.0-60-generic lang/java#17.0.13 md/OpenJDK_64-Bit_Server_VM#17.0.13+12-LTS md/vendor#BellSoft md/en_US md/kotlin/1.9.21-release-633 cfg/retry-mode#standard cfg/auth-source#stat CRTS3NativeClient/0.1.x platform/unknown" - latency=0.010000149s
"PUT /customers-bucket/76/94/559476/bosch-annual-report-2023%20%281%29.pdf?partNumber=1&uploadId=2~WEWFO-TZ_90u9DJz8y7hXibTWR5n79o HTTP/1.1" 400 8388941 - "aws-sdk-java/2.28.6 md/io#async md/http#s3crt ua/2.0 os/Linux#5.15.0-60-generic lang/java#17.0.13 md/OpenJDK_64-Bit_Server_VM#17.0.13+12-LTS md/vendor#BellSoft md/en_US md/kotlin/1.9.21-release-633 cfg/retry-mode#standard cfg/auth-source#stat CRTS3NativeClient/0.1.x platform/unknown" - latency=0.134002000s
"PUT /customers-bucket/76/94/559476/bosch-annual-report-2023%20%281%29.pdf?partNumber=2&uploadId=2~WEWFO-TZ_90u9DJz8y7hXibTWR5n79o HTTP/1.1" 400 1700859 - "aws-sdk-java/2.28.6 md/io#async md/http#s3crt ua/2.0 os/Linux#5.15.0-60-generic lang/java#17.0.13 md/OpenJDK_64-Bit_Server_VM#17.0.13+12-LTS md/vendor#BellSoft md/en_US md/kotlin/1.9.21-release-633 cfg/retry-mode#standard cfg/auth-source#stat CRTS3NativeClient/0.1.x platform/unknown" - latency=0.018000269s

My kotlin spring config for client:

import java.net.URI
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider
import software.amazon.awssdk.awscore.retry.AwsRetryStrategy
import software.amazon.awssdk.http.crt.AwsCrtAsyncHttpClient
import software.amazon.awssdk.services.s3.S3AsyncClient

@Bean
fun s3Client(properties: S3Properties): S3AsyncClient {
    val credentials = AwsBasicCredentials.create(properties.accessKey, properties.secretKey)
    
    val httpClient = AwsCrtAsyncHttpClient.builder()
        .proxyConfiguration { proxyConfig ->
            proxyConfig
                .useEnvironmentVariableValues(false)
                .useSystemPropertyValues(false)
        }
        .build()
    
    val retryStrategy = AwsRetryStrategy.standardRetryStrategy()
        .toBuilder()
        .maxAttempts(properties.retryMaxAttempts)
        .build()

    return S3AsyncClient.builder()
        .endpointOverride(URI(properties.endpoint))
        .credentialsProvider(StaticCredentialsProvider.create(credentials))
        .forcePathStyle(properties.forcePathStyle)
        .httpClient(httpClient)
        .region(properties.region)
        .overrideConfiguration { configuration ->
            configuration.retryStrategy(retryStrategy)
        }
        .build()
}

[debora-ito]

Hi @IgorekSergeevich you need to reach out to the Ceph support team for help with troubleshooting. We don't guarantee that the AWS SDKs will work with third-party s3-compatible solutions.