AWS CRT .so libs are not copied to docker image

[luneo7]

Ever since upgrade to an AWS CRT version >= v0.31.0 which contains changes adding a GraalVM feature and changes how the library is loaded in native image, now we get exceptions with the following stack trace:

org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:896)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:872)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class software.amazon.awssdk.crt.io.ClientBootstrap
	at software.amazon.awssdk.http.crt.AwsCrtHttpClientBase.<init>(AwsCrtHttpClientBase.java:76)
	at software.amazon.awssdk.http.crt.AwsCrtAsyncHttpClient.<init>(AwsCrtAsyncHttpClient.java:52)
	at software.amazon.awssdk.http.crt.AwsCrtAsyncHttpClient.<init>(AwsCrtAsyncHttpClient.java:49)
	at software.amazon.awssdk.http.crt.AwsCrtAsyncHttpClient$DefaultAsyncBuilder.buildWithDefaults(AwsCrtAsyncHttpClient.java:248)
	at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.lambda$resolveAsyncHttpClient$18(SdkDefaultClientBuilder.java:467)
	at [email protected]/java.util.Optional.map(Optional.java:260)
	at software.amazon.awssdk.utils.Either.lambda$map$0(Either.java:51)
	at [email protected]/java.util.Optional.orElseGet(Optional.java:364)
	at software.amazon.awssdk.utils.Either.map(Either.java:51)
	at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.lambda$resolveAsyncHttpClient$19(SdkDefaultClientBuilder.java:467)
	at [email protected]/java.util.Optional.map(Optional.java:260)
	at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.resolveAsyncHttpClient(SdkDefaultClientBuilder.java:467)
	at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.lambda$finalizeAsyncConfiguration$6(SdkDefaultClientBuilder.java:322)
	at software.amazon.awssdk.utils.AttributeMap$DerivedValue.primeCache(AttributeMap.java:604)
	at software.amazon.awssdk.utils.AttributeMap$DerivedValue.get(AttributeMap.java:593)
	at software.amazon.awssdk.utils.AttributeMap$Builder.resolveValue(AttributeMap.java:400)
	at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1596)
	at software.amazon.awssdk.utils.AttributeMap$Builder.build(AttributeMap.java:362)
	at software.amazon.awssdk.core.client.config.SdkClientConfiguration$Builder.build(SdkClientConfiguration.java:224)
	at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.finalizeAsyncConfiguration(SdkDefaultClientBuilder.java:324)
	at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.asyncClientConfiguration(SdkDefaultClientBuilder.java:234)
	at software.amazon.awssdk.services.dynamodb.DefaultDynamoDbAsyncClientBuilder.buildClient(DefaultDynamoDbAsyncClientBuilder.java:53)
	at software.amazon.awssdk.services.dynamodb.DefaultDynamoDbAsyncClientBuilder.buildClient(DefaultDynamoDbAsyncClientBuilder.java:25)
	at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.build(SdkDefaultClientBuilder.java:169)
	at software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClientBuilder_Xj2Z4vzdyMCKYlz7wkx2aT3iMQ4_Synthetic_ClientProxy.build(Unknown Source)
	at io.quarkiverse.amazon.common.runtime.AmazonClientCommonRecorder$1.apply(AmazonClientCommonRecorder.java:174)
	at io.quarkiverse.amazon.common.runtime.AmazonClientCommonRecorder$1.apply(AmazonClientCommonRecorder.java:163)
	at software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient_cpT1vk1x-E9vxkBhgfSZXHC2Q1E_Synthetic_Bean.createSynthetic(Unknown Source)
	at software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient_cpT1vk1x-E9vxkBhgfSZXHC2Q1E_Synthetic_Bean.doCreate(Unknown Source)
	at software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient_cpT1vk1x-E9vxkBhgfSZXHC2Q1E_Synthetic_Bean.create(Unknown Source)
	at software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient_cpT1vk1x-E9vxkBhgfSZXHC2Q1E_Synthetic_Bean.create(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
	at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.c48(Unknown Source)
	at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.computeIfAbsent(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
	at io.quarkus.arc.impl.ClientProxies.getApplicationScopedDelegate(ClientProxies.java:21)
	at software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient_cpT1vk1x-E9vxkBhgfSZXHC2Q1E_Synthetic_ClientProxy.arc$delegate(Unknown Source)
	at software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient_cpT1vk1x-E9vxkBhgfSZXHC2Q1E_Synthetic_ClientProxy.batchGetItemPaginator(Unknown Source)

[luneo7]

The workaround that I've found out is to copy the so file following the pattern target/PROJECT_NAME-native-image-source-jar/*.so to the same directory as the final binary, and when building the docker file making sure that the crt lib so file is copied alongside with the native binary... it seems that we don't need anymore the inclusion of the lib as a resource in the native image, as the loader now detects it is running as native image and doesn't copy the lib file to a temp dir anymore, it just tries to load the lib and if it doesn't load it, it throws one exception... only in JVM mode that it copies to a temp dir... we still need to set the classes to runtime init though...

Example:

FROM debian:bookworm-slim

RUN mkdir -p /opt/app 

COPY ./target/myproject-runner ./target/myproject-native-image-source-jar/*.so /opt/app/
RUN mv /opt/app/myproject-runner /opt/app/myproject && \
    if ls /opt/app/*.so 1> /dev/null 2>&1; then \
    chmod 755 /opt/app/*.so ; \
    fi;

^ with this if it has the .so file or not it builds the docker image and CRT works as expected

It would be good though if there was a way to fix that in the amazon services extension

[scrocquesel]

Interesting, shouldn't our native integration tests fail ?

[luneo7]

Didn't check the integration tests, but we just get that error during runtime, so native build passes, a request during runtime fails. And it should fail as the code for the lib loader changed and when running in native image it doesn't load the lib from resources anymore

[scrocquesel]

Got it, this is related to this upstream issue quarkusio/quarkus#41250.

[luneo7]

Yeah, maybe this is just adding to the docs that we need to copy the lib files when building the docker image, and we can also drop the inclusion of the lib as a resource during build time, with that we will decrease the final binary size