invalid peer certificate: BadSignature when connecting to s3 from arm64/aarch64

[geekodour]

Just trying to connect to the delta lake on s3 fails when attempted from arm64/aarch64 machine, the same exact code and credentials work from amd64 machines(both host and docker images).

Environment

Delta-rs version:

> poetry show|rg "delta|polars|s3|fs|idna"
deltalake           0.17.4      Native Delta Lake Python binding based on d...
fsspec              2024.3.1    File-system specification
idna                3.7         Internationalized Domain Names in Applicati...
polars              0.20.25     Blazingly fast DataFrame library
s3fs                2024.3.1    Convenient Filesystem interface over S3
s3transfer          0.10.1      An Amazon S3 Transfer Manager

Binding:

Environment:

• Cloud provider: Reproduced on ARM machines on AWS and Hetzner
• OS: Linux 6.1.90 #1-NixOS SMP Thu May 2 14:29:32 UTC 2024 aarch64 GNU/Linux
• Other:
  • This issue was reproduced on host ARM machines aswell as ARM docker images

---

Bug

What happened:

OSError: Generic S3 error: Error after 10 retries in 3.594594837s, max_retries:10, retry_timeout:180s, source:error sending request for url (https://s3.ap-south-1.amazonaws.com/<bucket_name>/<path1>/<path2>/_delta_log/_last_checkpoint): error trying to connect: invalid peer certificate: BadSignature

What you expected to happen:
I'd expect things to just work as it does in amd64. The AWS credentials are correct and are specified via env vars.

How to reproduce it:

from deltalake import DeltaTable
dt_info = {'type': 's3', 'path': 's3a://<bucket_name>/<path1>/<path2>', 'options': {'AWS_S3_LOCKING_PROVIDER': 'dynamodb', 'DELTA_DYNAMO_TABLE_NAME': '<table_name>'}}
dt = DeltaTable(dt_info["path"], storage_options=dt_info["options"])

More details:

• Think this issue might be similar to Azure AD Auth fails on ARM64 #2475 and invalid peer certificate: BadSignature with Cloudflare WARP denoland/deno#21169
• I've done the following

# docker
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates
RUN update-ca-certificates

# nixos host (installed the following package)
cacert

None of this is needed on amd64 images/hosts, things just work. To clarify, even after explicitly installing the certificate bundles like this things do not seem to be working. I am not sure what exactly is causing the issue here.

[rtyler]

Thank you for the thorough bug report @geekodour! Nothing springs to mind as the culprit just yet, so let me see if I can reproduce this 🤔

[ion-elgreco]

Related issue #2511, will close so we keep one open