How to build and link a custom static library

[its-felix]

I've been trying for the past 3 days to build a DuckDB Static Library to use with this library, but nothing has worked so far.

My latest approach is to build DuckDB mostly similar to https://github.com/duckdb/duckdb/blob/v1.2.2/.github/workflows/BundleStaticLibs.yml .

The build script I'm using (after many other attempts) looks like this:

#!/bin/bash
export PWD=`pwd`
docker run                                                                                                   \
-v$PWD:$PWD                                                                                                  \
-e CMAKE_BUILD_PARALLEL_LEVEL=2                                                                              \
-e EXTENSION_CONFIGS="$PWD/extension_config.cmake"                                                           \
-e ENABLE_EXTENSION_AUTOLOADING=1                                                                            \
-e ENABLE_EXTENSION_AUTOINSTALL=1                                                                            \
-e BUILD_BENCHMARK=1                                                                                         \
-e FORCE_WARN_UNUSED=1                                                                                       \
-e VCPKG_TARGET_TRIPLET=arm64-linux                                                                          \
-e VCPKG_ROOT=$PWD/duckdb/vcpkg                                                                              \
-e VCPKG_TOOLCHAIN_PATH=$PWD/duckdb/vcpkg/scripts/buildsystems/vcpkg.cmake                                   \
-e USE_MERGED_VCPKG_MANIFEST=1                                                                               \
-e STATIC_OPENSSL=1                                                                                          \
-e STATIC_LIBCPP=1                                                                                           \
-e EXTENSION_STATIC_BUILD=1                                                                                  \
quay.io/pypa/manylinux_2_28_aarch64                                                                          \
bash -c "yum install -y perl-IPC-Cmd curl zip unzip tar gcc gcc-c++ ninja-build && (cd $PWD/duckdb && bash $PWD/install_vcpkg.sh) && git config --global --add safe.directory $PWD/duckdb && make bundle-library -C $PWD/duckdb"

install_vcpkg.sh:

#!/bin/bash
set -eux
rm -rf vcpkg
mkdir vcpkg
cd vcpkg
git init
git remote add origin "https://github.com/microsoft/vcpkg.git"
git fetch origin "ce613c41372b23b1f51333815feb3edd87ef8a8b"
git checkout "ce613c41372b23b1f51333815feb3edd87ef8a8b"
./bootstrap-vcpkg.sh

Where duckdb is DuckDB at v1.2.2 and extension_config.cmake is:

duckdb_extension_load(core_functions)
duckdb_extension_load(parquet)
duckdb_extension_load(json)
duckdb_extension_load(icu)
duckdb_extension_load(httpfs
    GIT_URL https://github.com/duckdb/duckdb-httpfs
    GIT_TAG c22532453e9fab8404f91729708d9f35e23d323d
    INCLUDE_DIR extension/httpfs/include
)
duckdb_extension_load(aws
    GIT_URL https://github.com/duckdb/duckdb-aws
    GIT_TAG e92e45b30ba17594b1101db22699a2244adfaeb1
)

The build seems to work and results in a libduckdb_bundle.a with a size of 118.4 MB

When I'm trying to build my go-app using the following Dockerfile:

FROM --platform=linux/arm64 golang:1.24

RUN mkdir /build
WORKDIR /build

ENV GOOS=linux
ENV GOARCH=arm64
ENV CGO_ENABLED=1
ENV CPPFLAGS="-DDUCKDB_STATIC_BUILD"
ENV CGO_LDFLAGS="-lduckdb_bundle -lstdc++ -lm -ldl -L/build/cron/libs"

COPY ../.. .

WORKDIR /build/cron
RUN go build -o bootstrap -tags "lambda,lambda.norpc,duckdb_use_static_lib"

(/build/cron/libs is a copy of the build/release folder of my duckdb build)

The build of the go app fails with a lot of errors:

[...] more above
12.48 /usr/bin/ld: aws_secret.cpp:(.text._ZN6duckdb39DuckDBCustomAWSCredentialsProviderChainC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_[_ZN6duckdb39DuckDBCustomAWSCredentialsProviderChainC5ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_]+0x5bc): undefined reference to `Aws::Auth::ProfileConfigFileAWSCredentialsProvider::ProfileConfigFileAWSCredentialsProvider(long)'
12.48 /usr/bin/ld: aws_secret.cpp:(.text._ZN6duckdb39DuckDBCustomAWSCredentialsProviderChainC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_[_ZN6duckdb39DuckDBCustomAWSCredentialsProviderChainC5ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_]+0x614): undefined reference to `Aws::Auth::SSOCredentialsProvider::SSOCredentialsProvider()'
12.48 /usr/bin/ld: aws_secret.cpp:(.text._ZN6duckdb39DuckDBCustomAWSCredentialsProviderChainC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_[_ZN6duckdb39DuckDBCustomAWSCredentialsProviderChainC5ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_]+0x6a4): undefined reference to `Aws::Auth::InstanceProfileCredentialsProvider::InstanceProfileCredentialsProvider(long)'
12.48 /usr/bin/ld: aws_secret.cpp:(.text._ZN6duckdb39DuckDBCustomAWSCredentialsProviderChainC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_[_ZN6duckdb39DuckDBCustomAWSCredentialsProviderChainC5ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_]+0x710): undefined reference to `Aws::Auth::STSAssumeRoleWebIdentityCredentialsProvider::STSAssumeRoleWebIdentityCredentialsProvider()'
12.48 /usr/bin/ld: aws_secret.cpp:(.text._ZN6duckdb39DuckDBCustomAWSCredentialsProviderChainC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_[_ZN6duckdb39DuckDBCustomAWSCredentialsProviderChainC5ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_]+0x7c8): undefined reference to `Aws::Auth::STSProfileCredentialsProvider::STSProfileCredentialsProvider(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::chrono::duration<long, std::ratio<60l, 1l> >)'
12.48 /usr/bin/ld: aws_secret.cpp:(.text._ZN6duckdb39DuckDBCustomAWSCredentialsProviderChainC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_[_ZN6duckdb39DuckDBCustomAWSCredentialsProviderChainC5ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_]+0x8b8): undefined reference to `Aws::Utils::Threading::ReaderWriterLock::ReaderWriterLock()'
12.48 /usr/bin/ld: aws_secret.cpp:(.text._ZN6duckdb39DuckDBCustomAWSCredentialsProviderChainC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_[_ZN6duckdb39DuckDBCustomAWSCredentialsProviderChainC5ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_]+0xa34): undefined reference to `Aws::Auth::ProfileConfigFileAWSCredentialsProvider::ProfileConfigFileAWSCredentialsProvider(char const*, long)'
12.48 /usr/bin/ld: /build/cron/libs/libduckdb_bundle.a(aws_secret.cpp.o): in function `duckdb::CreateAWSSecretFromCredentialChain(duckdb::ClientContext&, duckdb::CreateSecretInput&)':
12.48 aws_secret.cpp:(.text._ZN6duckdbL34CreateAWSSecretFromCredentialChainERNS_13ClientContextERNS_17CreateSecretInputE+0x74): undefined reference to `Aws::Utils::DateTime::DateTime(std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&)'
12.48 /usr/bin/ld: aws_secret.cpp:(.text._ZN6duckdbL34CreateAWSSecretFromCredentialChainERNS_13ClientContextERNS_17CreateSecretInputE+0x240): undefined reference to `Aws::Auth::AWSCredentialsProviderChain::GetAWSCredentials()'
12.48 /usr/bin/ld: aws_secret.cpp:(.text._ZN6duckdbL34CreateAWSSecretFromCredentialChainERNS_13ClientContextERNS_17CreateSecretInputE+0x380): undefined reference to `vtable for Aws::Auth::AWSCredentialsProviderChain'
12.48 /usr/bin/ld: aws_secret.cpp:(.text._ZN6duckdbL34CreateAWSSecretFromCredentialChainERNS_13ClientContextERNS_17CreateSecretInputE+0x384): undefined reference to `vtable for Aws::Auth::AWSCredentialsProviderChain'
12.48 /usr/bin/ld: aws_secret.cpp:(.text._ZN6duckdbL34CreateAWSSecretFromCredentialChainERNS_13ClientContextERNS_17CreateSecretInputE+0x470): undefined reference to `vtable for Aws::Auth::AWSCredentialsProvider'
12.48 /usr/bin/ld: aws_secret.cpp:(.text._ZN6duckdbL34CreateAWSSecretFromCredentialChainERNS_13ClientContextERNS_17CreateSecretInputE+0x474): undefined reference to `vtable for Aws::Auth::AWSCredentialsProvider'
12.48 /usr/bin/ld: aws_secret.cpp:(.text._ZN6duckdbL34CreateAWSSecretFromCredentialChainERNS_13ClientContextERNS_17CreateSecretInputE+0x4a0): undefined reference to `Aws::Client::ClientConfiguration::ClientConfiguration(char const*, bool)'
12.48 /usr/bin/ld: aws_secret.cpp:(.text._ZN6duckdbL34CreateAWSSecretFromCredentialChainERNS_13ClientContextERNS_17CreateSecretInputE+0x730): undefined reference to `Aws::Utils::DateTime::Now()'
12.48 /usr/bin/ld: aws_secret.cpp:(.text._ZN6duckdbL34CreateAWSSecretFromCredentialChainERNS_13ClientContextERNS_17CreateSecretInputE+0x750): undefined reference to `Aws::Utils::DateTime::operator<=(Aws::Utils::DateTime const&) const'
12.48 /usr/bin/ld: aws_secret.cpp:(.text._ZN6duckdbL34CreateAWSSecretFromCredentialChainERNS_13ClientContextERNS_17CreateSecretInputE+0x1328): undefined reference to `Aws::Auth::ProfileConfigFileAWSCredentialsProvider::ProfileConfigFileAWSCredentialsProvider(char const*, long)'
12.48 /usr/bin/ld: aws_secret.cpp:(.text._ZN6duckdbL34CreateAWSSecretFromCredentialChainERNS_13ClientContextERNS_17CreateSecretInputE+0x133c): undefined reference to `Aws::Auth::ProfileConfigFileAWSCredentialsProvider::GetAWSCredentials()'
12.48 /usr/bin/ld: aws_secret.cpp:(.text._ZN6duckdbL34CreateAWSSecretFromCredentialChainERNS_13ClientContextERNS_17CreateSecretInputE+0x1480): undefined reference to `vtable for Aws::Auth::ProfileConfigFileAWSCredentialsProvider'
12.48 /usr/bin/ld: aws_secret.cpp:(.text._ZN6duckdbL34CreateAWSSecretFromCredentialChainERNS_13ClientContextERNS_17CreateSecretInputE+0x1484): undefined reference to `vtable for Aws::Auth::ProfileConfigFileAWSCredentialsProvider'
12.48 /usr/bin/ld: aws_secret.cpp:(.text._ZN6duckdbL34CreateAWSSecretFromCredentialChainERNS_13ClientContextERNS_17CreateSecretInputE+0x148c): undefined reference to `vtable for Aws::Config::AWSConfigFileProfileConfigLoader'
12.48 /usr/bin/ld: aws_secret.cpp:(.text._ZN6duckdbL34CreateAWSSecretFromCredentialChainERNS_13ClientContextERNS_17CreateSecretInputE+0x1490): undefined reference to `vtable for Aws::Config::AWSConfigFileProfileConfigLoader'
12.48 /usr/bin/ld: aws_secret.cpp:(.text._ZN6duckdbL34CreateAWSSecretFromCredentialChainERNS_13ClientContextERNS_17CreateSecretInputE+0x1bbc): undefined reference to `Aws::Auth::DefaultAWSCredentialsProviderChain::DefaultAWSCredentialsProviderChain()'
12.48 /usr/bin/ld: aws_secret.cpp:(.text._ZN6duckdbL34CreateAWSSecretFromCredentialChainERNS_13ClientContextERNS_17CreateSecretInputE+0x1bd0): undefined reference to `Aws::Auth::AWSCredentialsProviderChain::GetAWSCredentials()'
12.48 /usr/bin/ld: /build/cron/libs/libduckdb_bundle.a(aws_secret.cpp.o):(.data.rel.ro._ZTIN6duckdb39DuckDBCustomAWSCredentialsProviderChainE[_ZTIN6duckdb39DuckDBCustomAWSCredentialsProviderChainE]+0x10): undefined reference to `typeinfo for Aws::Auth::AWSCredentialsProviderChain'
12.48 /usr/bin/ld: /build/cron/libs/libduckdb_bundle.a(aws_secret.cpp.o):(.data.rel.ro._ZTVN6duckdb39DuckDBCustomAWSCredentialsProviderChainE[_ZTVN6duckdb39DuckDBCustomAWSCredentialsProviderChainE]+0x20): undefined reference to `Aws::Auth::AWSCredentialsProviderChain::GetAWSCredentials()'
12.48 /usr/bin/ld: /build/cron/libs/libduckdb_bundle.a(aws_secret.cpp.o):(.data.rel.ro._ZTVN6duckdb39DuckDBCustomAWSCredentialsProviderChainE[_ZTVN6duckdb39DuckDBCustomAWSCredentialsProviderChainE]+0x28): undefined reference to `Aws::Auth::AWSCredentialsProvider::IsTimeToRefresh(long)'
12.48 /usr/bin/ld: /build/cron/libs/libduckdb_bundle.a(aws_secret.cpp.o):(.data.rel.ro._ZTVN6duckdb39DuckDBCustomAWSCredentialsProviderChainE[_ZTVN6duckdb39DuckDBCustomAWSCredentialsProviderChainE]+0x30): undefined reference to `Aws::Auth::AWSCredentialsProvider::Reload()'
12.48 /usr/bin/ld: /build/cron/libs/libduckdb_bundle.a(create_secret_functions.cpp.o): in function `_GLOBAL__sub_I_create_secret_functions.cpp':
12.48 create_secret_functions.cpp:(.text.startup._GLOBAL__sub_I_create_secret_functions.cpp+0xc): undefined reference to `OPENSSL_init_ssl'
12.48 /usr/bin/ld: /build/cron/libs/libduckdb_bundle.a(crypto.cpp.o): in function `duckdb::AESGCMStateSSL::~AESGCMStateSSL() [clone .localalias]':
12.48 crypto.cpp:(.text._ZN6duckdb14AESGCMStateSSLD2Ev+0x24): undefined reference to `EVP_CIPHER_CTX_free'
12.48 /usr/bin/ld: /build/cron/libs/libduckdb_bundle.a(crypto.cpp.o): in function `duckdb::AESGCMStateSSL::GenerateRandomData(unsigned char*, unsigned long)':
12.48 crypto.cpp:(.text._ZN6duckdb14AESGCMStateSSL18GenerateRandomDataEPhm+0x8): undefined reference to `RAND_bytes'
12.48 /usr/bin/ld: /build/cron/libs/libduckdb_bundle.a(crypto.cpp.o): in function `duckdb::AESGCMStateSSL::Process(unsigned char const*, unsigned long, unsigned char*, unsigned long)':
12.48 crypto.cpp:(.text._ZN6duckdb14AESGCMStateSSL7ProcessEPKhmPhm+0x58): undefined reference to `EVP_DecryptUpdate'
12.48 /usr/bin/ld: crypto.cpp:(.text._ZN6duckdb14AESGCMStateSSL7ProcessEPKhmPhm+0xc0): undefined reference to `EVP_EncryptUpdate'
12.48 /usr/bin/ld: /build/cron/libs/libduckdb_bundle.a(crypto.cpp.o): in function `duckdb::GetCipher(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
12.48 crypto.cpp:(.text._ZN6duckdb9GetCipherERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x14): undefined reference to `EVP_aes_256_gcm'
12.48 /usr/bin/ld: crypto.cpp:(.text._ZN6duckdb9GetCipherERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x20): undefined reference to `EVP_aes_128_gcm'
12.48 /usr/bin/ld: crypto.cpp:(.text._ZN6duckdb9GetCipherERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x24): undefined reference to `EVP_aes_192_gcm'
12.48 /usr/bin/ld: /build/cron/libs/libduckdb_bundle.a(crypto.cpp.o): in function `duckdb::AESGCMStateSSL::InitializeEncryption(unsigned char const*, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*)':
12.48 crypto.cpp:(.text._ZN6duckdb14AESGCMStateSSL20InitializeEncryptionEPKhmPKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x40): undefined reference to `EVP_EncryptInit_ex'
12.48 /usr/bin/ld: /build/cron/libs/libduckdb_bundle.a(crypto.cpp.o): in function `duckdb::AESGCMStateSSL::InitializeDecryption(unsigned char const*, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*)':
12.48 crypto.cpp:(.text._ZN6duckdb14AESGCMStateSSL20InitializeDecryptionEPKhmPKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x44): undefined reference to `EVP_DecryptInit_ex'
12.48 /usr/bin/ld: /build/cron/libs/libduckdb_bundle.a(crypto.cpp.o): in function `duckdb::AESGCMStateSSL::AESGCMStateSSL()':
12.48 crypto.cpp:(.text._ZN6duckdb14AESGCMStateSSLC2Ev+0x30): undefined reference to `EVP_CIPHER_CTX_new'
12.48 /usr/bin/ld: /build/cron/libs/libduckdb_bundle.a(crypto.cpp.o): in function `duckdb::AESGCMStateSSL::Finalize(unsigned char*, unsigned long, unsigned char*, unsigned long)':
12.48 crypto.cpp:(.text._ZN6duckdb14AESGCMStateSSL8FinalizeEPhmS1_m+0x40): undefined reference to `EVP_EncryptFinal_ex'
12.48 /usr/bin/ld: crypto.cpp:(.text._ZN6duckdb14AESGCMStateSSL8FinalizeEPhmS1_m+0x64): undefined reference to `EVP_CIPHER_CTX_ctrl'
12.48 /usr/bin/ld: crypto.cpp:(.text._ZN6duckdb14AESGCMStateSSL8FinalizeEPhmS1_m+0x9c): undefined reference to `EVP_CIPHER_CTX_ctrl'
12.48 /usr/bin/ld: crypto.cpp:(.text._ZN6duckdb14AESGCMStateSSL8FinalizeEPhmS1_m+0xb4): undefined reference to `EVP_DecryptFinal_ex'
12.48 /usr/bin/ld: /build/cron/libs/libduckdb_bundle.a(crypto.cpp.o): in function `_GLOBAL__sub_I_crypto.cpp':
12.48 crypto.cpp:(.text.startup._GLOBAL__sub_I_crypto.cpp+0xc): undefined reference to `OPENSSL_init_ssl'
12.48 /usr/bin/ld: /build/cron/libs/libduckdb_bundle.a(regexp.cc.o): in function `duckdb_re2::Regexp::~Regexp()':
12.48 regexp.cc:(.text._ZN10duckdb_re26RegexpD2Ev+0x244): undefined reference to `__cxa_call_terminate'
12.48 /usr/bin/ld: /build/cron/libs/libduckdb_bundle.a(regexp.cc.o): in function `duckdb_re2::Regexp::Ref()':
12.48 regexp.cc:(.text._ZN10duckdb_re26Regexp3RefEv+0x190): undefined reference to `__cxa_call_terminate'
12.48 /usr/bin/ld: /build/cron/libs/libduckdb_bundle.a(regexp.cc.o): in function `duckdb_re2::Regexp::Incref()':
12.48 regexp.cc:(.text._ZN10duckdb_re26Regexp6IncrefEv+0x280): undefined reference to `__cxa_call_terminate'
12.48 /usr/bin/ld: /build/cron/libs/libduckdb_bundle.a(regexp.cc.o): in function `duckdb_re2::Regexp::Decref()':
12.48 regexp.cc:(.text._ZN10duckdb_re26Regexp6DecrefEv+0x3b4): undefined reference to `__cxa_call_terminate'
12.48 collect2: error: ld returned 1 exit status

I'm at a point where I'm completely lost what else I should try, also because I'm not at all familiar with the C/C++ Ecosystem und buildsystems

[taniabogatsch]

Hi @its-felix, I can try to give you some pointers to hopefully fix this, based on what I can tell from your description.

The build script I'm using (after many other attempts) looks like this: + make bundle-library

bundle-library uses the installed archive tool (ar) to 1. find all libraries, and then 2. bundle them into one. You can try to build with make gather-libs instead, which is what DuckDB does since 1.2.2. The advantage here is that you should be able to inspect the resulting build/release/libs/ folder and exactly know if all the libraries are there, main, third parties, and your extensions. That's the first thing I would make sure - and double-check that there is (a) library for the AWS extension, since that one seems problematic.

After make gather-libs, you can also launch DuckDB's CLI like so: build/release/duckdb. There, I'd make sure that everything works as expected, especially the AWS extension.

Once you've confirmed that, the next step is indeed to build you Go app.
I would recommend that instead of using make bundle-library, you just archive (read up on the ar tool usage, it should be straightforward) everything inside build/release/libs/ yourself - into a new archive (libcustom.a or so).

Then, change your docker file like so

ENV CGO_LDFLAGS="-lcustom -lstdc++ -lm -ldl -L/path/to/your/new/archive"

I am suggesting these steps because the linker error suggests that somehow the AWS library is missing (i.e., it is not part of the bundled library). And figuring out why that is the case should (might) hopefully resolve this for you. 🤞

[its-felix]

Thanks for the pointers, this helped me a bit figuring out what exactly happens.

I think I got a little further now by also pulling duckdb/build/release/vcpkg_installed/arm64-linux/lib/*.a.

Currently, I'm using the following Dockerfile to build the archive:

FROM golang:1.24 AS build

WORKDIR /build
RUN mkdir libs
COPY duckdb/build/release/libs/*.a ./libs
COPY duckdb/build/release/vcpkg_installed/arm64-linux/lib/*.a ./libs
WORKDIR /build/libs
RUN echo *.a | xargs -n1 ar x
RUN echo *.o | xargs ar rcs ../libduckdb_bundle.a

FROM scratch
COPY --from=build /build/libduckdb_bundle.a .
COPY --from=build /build/libs/* ./libs/

The duckdb/build/release/vcpkg_installed/arm64-linux/lib/*.a includes things like libssl.a among some AWS related things, so it looks like I'm somewhat on the right track, just missing some bits.

For reference, this is (part of) my current go build output:

11.28 /usr/bin/ld: channel.c:(.text+0x19f0): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(channel.c.o): in function `aws_channel_set_statistics_handler':
11.28 channel.c:(.text+0x1a84): undefined reference to `aws_crt_statistics_handler_destroy'
11.28 /usr/bin/ld: channel.c:(.text+0x1adc): undefined reference to `aws_crt_statistics_handler_get_report_interval_ms'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(connection_manager.c.o): in function `s_schedule_culling':
11.28 connection_manager.c:(.text+0x750): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: connection_manager.c:(.text+0x798): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: connection_manager.c:(.text+0x8b4): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(connection_manager.c.o): in function `s_aws_http_connection_manager_finish_destroy':
11.28 connection_manager.c:(.text+0xafc): undefined reference to `aws_mutex_clean_up'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(connection_manager.c.o): in function `s_aws_http_connection_manager_execute_transaction':
11.28 connection_manager.c:(.text+0x1388): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: connection_manager.c:(.text+0x1584): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(connection_manager.c.o): in function `s_cull_task':
11.28 connection_manager.c:(.text+0x1afc): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: connection_manager.c:(.text+0x1b38): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(connection_manager.c.o): in function `s_aws_http_connection_manager_h2_on_goaway_received':
11.28 connection_manager.c:(.text+0x1d90): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: connection_manager.c:(.text+0x1e00): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(connection_manager.c.o): in function `s_aws_http_connection_manager_h2_on_initial_settings_completed':
11.28 connection_manager.c:(.text+0x1efc): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: connection_manager.c:(.text+0x1f34): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(connection_manager.c.o): in function `s_aws_http_connection_manager_on_connection_shutdown':
11.28 connection_manager.c:(.text+0x2038): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: connection_manager.c:(.text+0x20c0): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(connection_manager.c.o): in function `s_aws_http_connection_manager_on_connection_setup':
11.28 connection_manager.c:(.text+0x21ec): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: connection_manager.c:(.text+0x224c): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(connection_manager.c.o): in function `aws_http_connection_manager_new':
11.28 connection_manager.c:(.text+0x2400): undefined reference to `aws_http_fatal_assert_library_initialized'
11.28 /usr/bin/ld: connection_manager.c:(.text+0x2454): undefined reference to `aws_mutex_init'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(connection_manager.c.o): in function `aws_http_connection_manager_acquire':
11.28 connection_manager.c:(.text+0x2a78): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: connection_manager.c:(.text+0x2a98): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(connection_manager.c.o): in function `aws_http_connection_manager_release':
11.28 connection_manager.c:(.text+0x2b54): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: connection_manager.c:(.text+0x2b74): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: connection_manager.c:(.text+0x2bbc): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: connection_manager.c:(.text+0x2c18): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: connection_manager.c:(.text+0x2cec): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(connection_manager.c.o): in function `aws_http_connection_manager_acquire_connection':
11.28 connection_manager.c:(.text+0x2e64): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: connection_manager.c:(.text+0x2eac): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(connection_manager.c.o): in function `aws_http_connection_manager_release_connection':
11.28 connection_manager.c:(.text+0x3050): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: connection_manager.c:(.text+0x3088): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(connection_manager.c.o): in function `aws_http_connection_manager_fetch_metrics':
11.28 connection_manager.c:(.text+0x3270): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: connection_manager.c:(.text+0x3294): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(epoll_event_loop.c.o): in function `s_schedule_task_common':
11.28 epoll_event_loop.c:(.text+0x5b0): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: epoll_event_loop.c:(.text+0x5e0): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: epoll_event_loop.c:(.text+0x72c): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(epoll_event_loop.c.o): in function `aws_event_loop_thread':
11.28 epoll_event_loop.c:(.text+0xc78): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: epoll_event_loop.c:(.text+0xce0): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(epoll_event_loop.c.o): in function `s_schedule_task_now':
11.28 epoll_event_loop.c:(.text+0x1088): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: epoll_event_loop.c:(.text+0x10b8): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: epoll_event_loop.c:(.text+0x11d8): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(exponential_backoff_retry_strategy.c.o): in function `s_default_gen_rand':
11.28 exponential_backoff_retry_strategy.c:(.text+0x18): undefined reference to `aws_device_random_u64'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(exponential_backoff_retry_strategy.c.o): in function `s_exponential_retry_task':
11.28 exponential_backoff_retry_strategy.c:(.text+0x54): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: exponential_backoff_retry_strategy.c:(.text+0x70): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(exponential_backoff_retry_strategy.c.o): in function `s_exponential_backoff_release_token':
11.28 exponential_backoff_retry_strategy.c:(.text+0x1d4): undefined reference to `aws_mutex_clean_up'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(exponential_backoff_retry_strategy.c.o): in function `s_exponential_retry_acquire_token':
11.28 exponential_backoff_retry_strategy.c:(.text+0x2f8): undefined reference to `aws_mutex_init'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(exponential_backoff_retry_strategy.c.o): in function `s_exponential_retry_schedule_retry':
11.28 exponential_backoff_retry_strategy.c:(.text+0x538): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: exponential_backoff_retry_strategy.c:(.text+0x54c): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: exponential_backoff_retry_strategy.c:(.text+0x61c): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(future.c.o): in function `aws_future_impl_new_by_value':
11.28 future.c:(.text+0x1fc): undefined reference to `aws_mutex_init'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(future.c.o): in function `aws_future_impl_new_by_value_with_clean_up':
11.28 future.c:(.text+0x27c): undefined reference to `aws_mutex_init'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(future.c.o): in function `aws_future_impl_new_pointer':
11.28 future.c:(.text+0x2fc): undefined reference to `aws_mutex_init'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(future.c.o): in function `aws_future_impl_new_pointer_with_destroy':
11.28 future.c:(.text+0x37c): undefined reference to `aws_mutex_init'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(future.c.o): in function `aws_future_impl_new_pointer_with_release':
11.28 future.c:(.text+0x404): undefined reference to `aws_mutex_init'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(future.c.o): in function `aws_future_impl_is_done':
11.28 future.c:(.text+0x4b8): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: future.c:(.text+0x4c4): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(future.c.o): in function `s_future_impl_destroy':
11.28 future.c:(.text+0x5b0): undefined reference to `aws_mutex_clean_up'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(future.c.o): in function `aws_future_impl_set_error':
11.28 future.c:(.text+0x6b0): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: future.c:(.text+0x6e0): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: future.c:(.text+0x70c): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(future.c.o): in function `aws_future_impl_set_result_by_move':
11.28 future.c:(.text+0x764): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: future.c:(.text+0x784): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: future.c:(.text+0x818): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(future.c.o): in function `aws_future_impl_register_callback':
11.28 future.c:(.text+0x8c8): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: future.c:(.text+0x8f8): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: future.c:(.text+0x900): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(future.c.o): in function `aws_future_impl_register_callback_if_not_done':
11.28 future.c:(.text+0x954): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: future.c:(.text+0x97c): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(future.c.o): in function `aws_future_impl_register_event_loop_callback':
11.28 future.c:(.text+0x9cc): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: future.c:(.text+0xa0c): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: future.c:(.text+0xa14): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(future.c.o): in function `aws_future_impl_register_channel_callback':
11.28 future.c:(.text+0xa80): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: future.c:(.text+0xac0): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: future.c:(.text+0xac8): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(future.c.o): in function `aws_future_impl_wait':
11.28 future.c:(.text+0xb20): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: future.c:(.text+0xb44): undefined reference to `aws_condition_variable_wait_for_pred'
11.28 /usr/bin/ld: future.c:(.text+0xb50): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(log_channel.c.o): in function `s_foreground_channel_clean_up':
11.28 log_channel.c:(.text+0x18): undefined reference to `aws_mutex_clean_up'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(log_channel.c.o): in function `s_foreground_channel_send':
11.28 log_channel.c:(.text+0x50): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: log_channel.c:(.text+0x6c): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(log_channel.c.o): in function `aws_background_logger_listen_for_messages':
11.28 log_channel.c:(.text+0xa0): undefined reference to `aws_condition_variable_wait_pred'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(log_channel.c.o): in function `s_background_channel_send':
11.28 log_channel.c:(.text+0x120): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: log_channel.c:(.text+0x18c): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(log_channel.c.o): in function `s_background_channel_clean_up':
11.28 log_channel.c:(.text+0x204): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: log_channel.c:(.text+0x224): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: log_channel.c:(.text+0x268): undefined reference to `aws_mutex_clean_up'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(log_channel.c.o): in function `aws_background_logger_thread':
11.28 log_channel.c:(.text+0x2dc): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: log_channel.c:(.text+0x348): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: log_channel.c:(.text+0x37c): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: log_channel.c:(.text+0x39c): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(log_channel.c.o): in function `aws_log_channel_init_foreground':
11.28 log_channel.c:(.text+0x498): undefined reference to `aws_mutex_init'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(log_channel.c.o): in function `aws_log_channel_init_background':
11.28 log_channel.c:(.text+0x51c): undefined reference to `aws_mutex_init'
11.28 /usr/bin/ld: log_channel.c:(.text+0x62c): undefined reference to `aws_mutex_clean_up'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(log_writer.c.o): in function `aws_log_writer_init_file':
11.28 log_writer.c:(.text+0x238): undefined reference to `aws_fopen'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(pem.c.o): in function `aws_pem_objects_init_from_file_contents':
11.28 pem.c:(.text+0x7c8): undefined reference to `aws_base64_compute_decoded_len'
11.28 /usr/bin/ld: pem.c:(.text+0x7e8): undefined reference to `aws_base64_decode'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(pem.c.o): in function `aws_pem_objects_init_from_file_path':
11.28 pem.c:(.text+0xd04): undefined reference to `aws_byte_buf_init_from_file'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(pkcs11_tls_op_handler.c.o): in function `s_aws_pkcs11_tls_op_handler_do_operation':
11.28 pkcs11_tls_op_handler.c:(.text+0x38): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: pkcs11_tls_op_handler.c:(.text+0x98): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: pkcs11_tls_op_handler.c:(.text+0x108): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: pkcs11_tls_op_handler.c:(.text+0x1b0): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: pkcs11_tls_op_handler.c:(.text+0x1f4): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: pkcs11_tls_op_handler.c:(.text+0x204): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(pkcs11_tls_op_handler.c.o): in function `s_aws_custom_key_op_handler_destroy':
11.28 pkcs11_tls_op_handler.c:(.text+0x23c): undefined reference to `aws_mutex_clean_up'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(pkcs11_tls_op_handler.c.o): in function `aws_pkcs11_tls_op_handler_new':
11.28 pkcs11_tls_op_handler.c:(.text+0x2e0): undefined reference to `aws_mutex_init'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(s2n_connection.c.o): in function `s2n_connection_get_peer_cert_chain':
11.28 s2n_connection.c:(.text+0x5068): undefined reference to `s2n_crypto_free'
11.28 /usr/bin/ld: s2n_connection.c:(.text+0x5190): undefined reference to `s2n_crypto_free'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(pkcs11_lib.c.o): in function `s_pkcs11_unlock_mutex':
11.28 pkcs11_lib.c:(.text+0x23c): undefined reference to `aws_mutex_unlock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(pkcs11_lib.c.o): in function `s_pkcs11_lock_mutex':
11.28 pkcs11_lib.c:(.text+0x2d0): undefined reference to `aws_mutex_lock'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(pkcs11_lib.c.o): in function `s_pkcs11_destroy_mutex':
11.28 pkcs11_lib.c:(.text+0x374): undefined reference to `aws_mutex_clean_up'
11.28 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(pkcs11_lib.c.o): in function `s_pkcs11_create_mutex':
11.28 pkcs11_lib.c:(.text+0x3d0): undefined reference to `aws_mutex_init'

Most of the missing things should apparently be part of libaws-c-common.a which is already part of the archive I currently unpack (into .o) and then repack (into libduckdb_bundle.a)

[its-felix]

I was able to confirm that my libaws-c-common.a contains the symbols in question:

nm bundle/libs/libaws-c-common.a | grep "aws_mutex_lock"
                 U aws_mutex_lock
                 U aws_mutex_lock
                 U aws_mutex_lock
                 U aws_mutex_lock
00000000000001e0 T aws_mutex_lock
                 U aws_mutex_lock
                 U aws_mutex_lock

but the unpacked mutex.c.o does not:

nm bundle/libs/mutex.c.o | grep "aws_mutex_lock"
<empty>

I assume some other archive contains a different mutex.c.o and the last one wins when I unpack everything. Digging further.

[its-felix]

I was able to resolve above issue by unpacking each archive into its own subdirectory:

FROM quay.io/pypa/manylinux_2_28_aarch64 AS build

WORKDIR /build
RUN mkdir libs
COPY duckdb/build/release/libs/*.a ./libs
COPY duckdb/build/release/vcpkg_installed/arm64-linux/lib/*.a ./libs
WORKDIR /build/libs
RUN ls *.a | xargs -n1 -I{} sh -c 'mkdir -p "objs-{}" && (cd "objs-{}" && ar x "../{}")'
RUN echo objs-*/*.o | xargs ar rcs ../libduckdb_bundle.a

FROM scratch
COPY --from=build /build/ ./

It turns out that jemalloc contains a mutex.c.o too, so that's where the clash came from:

find bundle/libs -name 'mutex.c.o'
> bundle/libs/objs-libjemalloc_extension.a/mutex.c.o
> bundle/libs/objs-libaws-c-common.a/mutex.c.o

And the unpacked AWS version of mutex.c.o contains the symbol:

nm bundle/libs/objs-libaws-c-common.a/mutex.c.o | grep "aws_mutex_lock"
> 00000000000001e0 T aws_mutex_lock

Some other symbols are missing now:

11.93 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(compression.c.o): in function `aws_compression_library_init':
11.93 compression.c:(.text+0x20): undefined reference to `aws_common_library_init'
11.93 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(compression.c.o): in function `aws_compression_library_clean_up':
11.93 compression.c:(.text+0x6c): undefined reference to `aws_common_library_clean_up'
11.93 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(pem.c.o): in function `aws_pem_objects_init_from_file_contents':
11.93 pem.c:(.text+0x7c8): undefined reference to `aws_base64_compute_decoded_len'
11.93 /usr/bin/ld: pem.c:(.text+0x7e8): undefined reference to `aws_base64_decode'
11.93 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(pem.c.o): in function `aws_pem_objects_init_from_file_path':
11.93 pem.c:(.text+0xd04): undefined reference to `aws_byte_buf_init_from_file'
11.93 collect2: error: ld returned 1 exit status

However, this time I can find the symbol in the final bundle already:

nm bundle/libduckdb_bundle.a | grep "aws_byte_buf_init_from_file"
                 U aws_byte_buf_init_from_file
                 U aws_byte_buf_init_from_file
                 U aws_byte_buf_init_from_file
                 U aws_byte_buf_init_from_file
                 U aws_byte_buf_init_from_file
                 U aws_byte_buf_init_from_file

[its-felix]

(Documenting all this in hope anyone with the same error messages might find this helpful in the future)

After coming across some more missing symbols, I figured out that even within the same archive there are duplicate names, i.e. in libaws-c-common.a there's 2 different file.c.o.

I adjusted the build a bit accordingly:

RUN ls *.a | xargs -n1 sh /usr/bin/lib/ar-extract.sh
RUN ls objs-*/*/*.o | xargs ar rcs ../libduckdb_bundle.a

#!/bin/sh
set -eux
mkdir -p "objs-$1"
cd "objs-$1"

i=0
ar t "../$1" | while read member
do
  (mkdir -p "$i" && cd "$i" && ar x "../../$1" "$member")
  i=$((i+1))
done

Now down to some more undefined symbols:

11.62 uuid.c:(.text+0x20): undefined reference to `aws_device_random_buffer'
11.62 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(socket.c.o): in function `s_stop_accept_task':
11.62 socket.c:(.text+0x19fc): undefined reference to `aws_condition_variable_notify_one'
11.62 /usr/bin/ld: socket.c:(.text+0x1a24): undefined reference to `aws_condition_variable_notify_one'
11.62 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(socket.c.o): in function `s_close_task':
11.62 socket.c:(.text+0x3b84): undefined reference to `aws_condition_variable_notify_one'
11.62 /usr/bin/ld: socket.c:(.text+0x3bac): undefined reference to `aws_condition_variable_notify_one'
11.62 collect2: error: ld returned 1 exit statu

[its-felix]

After expanding my ar-extract.sh a bit more (previously, I looped through all members, but would still only extract the first named-member if there are multiple with the same name):

#!/bin/sh
set -eux
mkdir -p "objs-$1"
cd "objs-$1"

ftemp=$(mktemp)
cp "../$1" "$ftemp"

i=0
while true
do
  member=$(ar t "$ftemp" | head -n 1)
  if [ -z "$member" ]; then
    break
  fi

  (mkdir -p "$i" && cd "$i" && ar x "$ftemp" "$member")
  ar d "$ftemp" "$member"
  i=$((i+1))
done

rm "$ftemp"

During go build:

12.43 ub_duckdb_transaction.cpp:(.text._ZN6duckdb15DuckTransaction6CommitERNS_16AttachedDatabaseEmNS_10unique_ptrINS_18StorageCommitStateESt14default_deleteIS4_ELb1EEE+0x188): undefined reference to `__cxa_call_terminate'
12.43 /usr/bin/ld: /build/database/libs/libduckdb_bundle.a(ub_duckdb_transaction.cpp.o):ub_duckdb_transaction.cpp:(.text._ZN6duckdb15DuckTransaction10WriteToWALERNS_16AttachedDatabaseERNS_10unique_ptrINS_18StorageCommitStateESt14default_deleteIS4_ELb1EEE+0x300): more undefined references to `__cxa_call_terminate' follow

[taniabogatsch]

One more thing, maybe instead of touching and (re)bundling the archives (libs), you could try to leave them as they are, and just provide all of them in the linking command (it'll look ugly, but maybe you can avoid duplicate symbol issues)?

See the LDFLAGS here: https://github.com/duckdb/duckdb-go-bindings/blob/main/linux-amd64/cgo_static.go.

Another pit that I fell into before is that on Linux (not sure what system you are on) linking order matters.

The linker searches from left to right, and notes unresolved symbols as it goes. If a library resolves the symbol, it takes the object files of that library to resolve the symbol (b.o out of libb.a in this case).

https://stackoverflow.com/questions/45135/why-does-the-order-in-which-libraries-are-linked-sometimes-cause-errors-in-gcc

[its-felix]

Thanks for all your help @taniabogatsch !

I turns out that the issue was simply a incompatability between gcc/g++ used in the DuckDB Build, where I used quay.io/pypa/manylinux_2_28_aarch64 and the one present in golang:1.24.

After switching the base image for my Go-Build to quay.io/pypa/manylinux_2_28_aarch64 the build succeeds with the fat-bundled DuckDB as described above (unpack and repack everything from duckdb/build/release/libs and duckdb/build/release/vcpkg_installed/arm64-linux/lib).

For future readers, my working Dockerfile for the GO BUILD looks like this:

FROM quay.io/pypa/manylinux_2_28_aarch64 AS build

ADD "https://go.dev/dl/go1.24.2.linux-arm64.tar.gz" ./go.tar.gz
RUN tar -C /usr/local -xzf go.tar.gz && rm go.tar.gz
ENV PATH="$PATH:/usr/local/go/bin"

ENV GOOS=linux
ENV GOARCH=arm64
ENV CGO_ENABLED=1
ENV CPPFLAGS="-DDUCKDB_STATIC_BUILD"
# ENV CGO_LDFLAGS="-lduckdb_static -lcore_functions_extension -licu_extension -ljson_extension -lparquet_extension -ljemalloc_extension -lhttpfs_extension -laws_extension -laws-cpp-sdk-core -lcurl -lcrypto -lssl -lz -laws-crt-cpp -laws-c-common -laws-c-io -ls2n -laws-c-cal -laws-c-http -laws-c-compression -lduckdb_fastpforlib -lduckdb_fmt -lduckdb_fsst -lduckdb_hyperloglog -lduckdb_mbedtls -lduckdb_miniz -lduckdb_pg_query -lduckdb_re2 -lduckdb_skiplistlib -lduckdb_utf8proc -lduckdb_yyjson -lduckdb_zstd -lstdc++ -lm -ldl -L/build/database/libs"
ENV CGO_LDFLAGS="-lduckdb_bundle -lstdc++ -lm -ldl -L/build/database/libs"

WORKDIR /build
COPY ../.. .

WORKDIR /build/database
RUN go build -o database -tags "lambda,duckdb_use_static_lib"

FROM gcr.io/distroless/cc:nonroot

COPY --from=build --chown=nonroot:nonroot /build/database/database /usr/local/bin/database
USER nonroot

ENTRYPOINT ["/usr/local/bin/database"]

And the "fat-build" of libduckdb_bundle.a remains at the state of this comment: #426 (comment)

Should the DuckDB gather-libs target maybe be expanded to also include duckdb/build/release/vcpkg_installed/<vcpkg-target-triplet>/lib/*.a ?

Also, another question out of curiosity ... How/Why does a regular go build seem to work on pretty much any host I've tried out of the box when not trying to bundle a custom static lib of duckdb but instead using the provided ones by this lib (go-duckdb)? And in extension to that, is there any way how I could make my DuckDB Build more robust so that I can build my go application on more or less any linux/arm64 host just like I seem to be able to do when using this library with the provided ones?

[its-felix]

I've now merged the steps into a single multi-stage Dockerfile which can be used to build the "fat-bundle" in a single step:

ARG IMAGE=$TARGETARCH
FROM quay.io/pypa/manylinux_2_28_$IMAGE AS build_duckdb

ARG EXTENSION_CONFIG
ARG VCPKG_TARGET_TRIPLET
ARG DUCKDB_GIT_REF
ARG VCPKG_GIT_REF

WORKDIR /build

# install dependencies
RUN yum install -y perl-IPC-Cmd curl zip unzip tar gcc gcc-c++ ninja-build
# checkout duckdb
RUN (mkdir duckdb && cd duckdb && git init && git remote add origin "https://github.com/duckdb/duckdb.git" && git fetch --all && git checkout "$DUCKDB_GIT_REF")
# checkout vcpkg
RUN (mkdir duckdb/vcpkg && cd duckdb/vcpkg && git init && git remote add origin "https://github.com/microsoft/vcpkg.git" && git fetch origin "$VCPKG_GIT_REF" && git checkout "$VCPKG_GIT_REF")

# setup vcpkg
RUN (cd duckdb/vcpkg && ./bootstrap-vcpkg.sh)

# copy extension config
COPY $EXTENSION_CONFIG ./extension_config.cmake

ENV CMAKE_BUILD_PARALLEL_LEVEL=2
ENV EXTENSION_CONFIGS="/build/extension_config.cmake"
ENV ENABLE_EXTENSION_AUTOLOADING=1
ENV ENABLE_EXTENSION_AUTOINSTALL=1
ENV FORCE_WARN_UNUSED=1
ENV VCPKG_TARGET_TRIPLET="$VCPKG_TARGET_TRIPLET"
ENV VCPKG_ROOT="/build/duckdb/vcpkg"
ENV VCPKG_TOOLCHAIN_PATH="/build/duckdb/vcpkg/scripts/buildsystems/vcpkg.cmake"
ENV USE_MERGED_VCPKG_MANIFEST=1
ENV STATIC_OPENSSL=1
ENV STATIC_LIBCPP=1
ENV EXTENSION_STATIC_BUILD=1

# gather libs
RUN make gather-libs -C ./duckdb

FROM quay.io/pypa/manylinux_2_28_$IMAGE AS build_bundle

ARG VCPKG_TARGET_TRIPLET

WORKDIR /build
COPY --from=build_duckdb "/build/duckdb/build/release/libs/*.a" ./
COPY --from=build_duckdb "/build/duckdb/build/release/vcpkg_installed/$VCPKG_TARGET_TRIPLET/lib/*.a" ./

# copy ar-extract helper
COPY ar-extract.sh /usr/local/bin/ar-extract.sh

# extract archives
RUN ls *.a | xargs -n1 sh /usr/local/bin/ar-extract.sh
# bundle fat archive
RUN ls objs-*/*/*.o | xargs ar rcs libduckdb_bundle.a

FROM scratch
COPY --from=build_bundle /build/libduckdb_bundle.a ./

Usage:

docker build --platform linux/arm64 --build-arg IMAGE=aarch64 --build-arg EXTENSION_CONFIG=extension_config.cmake --build-arg VCPKG_TARGET_TRIPLET=arm64-linux --build-arg DUCKDB_GIT_REF=v1.2.2 --build-arg VCPKG_GIT_REF=ce613c41372b23b1f51333815feb3edd87ef8a8b . --output ./bundle

GitHub Action Usage:
https://github.com/explore-flights/libduckdb/blob/31a312d9d9eda668b9a727759339c128192ca26e/.github/workflows/build.yml

[its-felix]

I have now also created reusable composite actions in https://github.com/explore-flights/libduckdb/tree/main/.github/actions

I have tested them with my own setup (linux/arm64), but they should work for x86 too. Not so sure about macOS/Windows.

An example can be found here: https://github.com/explore-flights/libduckdb-example

[taniabogatsch]

Also, another question out of curiosity

As far as I know, we're building our libraries with that manylinux image (quay.io/pypa/manylinux_2_28_${{ matrix.config.image }}) to achieve 'as high as possible' compatibility out of the box. There might be more to it, but that's what I know. 😅

[prochac]

This is what we do since the v1.2.2 release of DuckDB
duckdb/duckdb#17312 (comment)