Podman 5.4 regression [aarch64]: SIGSEGV: invalid memory reference

[Razican]

Issue Description

I was using the quay.io/podman/stable:latest image on aarch64 (Raspberry Pi K3S cluster, as build image in a GitLab CI runner), and since the upgrade to 5.4, building a normal Rust package in an image fails with a segmentation fault.

I have tried to use the quay.io/podman/stable:v5.3 version, and works fine. Even, using the v5.3 version, if I do dnf upgrade -y podman and upgrades to 5.4, the build fails again.

Steps to reproduce the issue

Steps to reproduce the issue

1. Use the quay.io/podman/stable:v5.4 image or quay.io/podman/stable:v5.3 and upgrade to podman 5.4 in a GitLab runner in aarch64 (example: Raspberry Pi). You can create a new image from it. Example dockerfile:

   FROM quay.io/podman/stable:v5.3
   RUN dnf upgrade -y podman

2. Use it to build a Rust image based on docker.io/rust:alpine3.21. In this image, run cargo to build a package. Example dockerfile:

   FROM docker.io/rust:alpine3.21
   RUN apk add --no-cache build-base
   ENV RUSTFLAGS="-C target-feature=-crt-static"
   RUN cargo install mdbook-mermaid

   Example GitLab configuration:

   build-container:
     image: quay.io/podman/stable:v5.4
     script:
       - podman build -f ./rust_image.Dockerfile .

3. At some point, after part of the build process, you'll get the error:

    error: could not compile `pest` (lib)
    Caused by:
      process didn't exit successfully: `/usr/local/rustup/toolchains/1.85.0-aarch64-unknown-linux-musl/bin/rustc --crate-name pest --edition=2021 /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/pest-2.7.15/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off --cfg 'feature="default"' --cfg 'feature="memchr"' --cfg 'feature="std"' --check-cfg 'cfg(docsrs,test)' --check-cfg 'cfg(feature, values("const_prec_climber", "default", "memchr", "miette-error", "pretty-print", "std"))' -C metadata=ad1c6c26bc004204 -C extra-filename=-0e9f56b08d1b76c7 --out-dir /tmp/cargo-installJTny6T/release/deps -C strip=debuginfo -L dependency=/tmp/cargo-installJTny6T/release/deps --extern memchr=/tmp/cargo-installJTny6T/release/deps/libmemchr-9f3e61bb84415a1a.rmeta --extern thiserror=/tmp/cargo-installJTny6T/release/deps/libthiserror-12cc08f65ec944a5.rmeta --extern ucd_trie=/tmp/cargo-installJTny6T/release/deps/libucd_trie-ef6ec1f78939dd4f.rmeta --cap-lints allow -C target-feature=-crt-static` (signal: 11, SIGSEGV: invalid memory reference)
    warning: build failed, waiting for other jobs to finish...
    error: failed to compile `mdbook-mermaid v0.14.1`, intermediate artifacts can be found at `/tmp/cargo-installJTny6T`.
    To reuse those artifacts with a future compilation, set the environment variable `CARGO_TARGET_DIR` to that path.
    subprocess exited with status 101
    subprocess exited with status 101
    Error: building at STEP "RUN cargo install mdbook-mermaid": exit status 101
   

Describe the results you received

I receive a segmentation fault

Describe the results you expected

I expected the build to complete successfully, as with v5.3

podman info output

host:
  arch: arm64
  buildahVersion: 1.39.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-3.fc41.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 91.66
    systemPercent: 1.76
    userPercent: 6.58
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: container
    version: "41"
  eventLogger: file
  freeLocks: 2048
  hostname: runner-t3ctd6mf-project-38114189-concurrent-6-s9nn3vg9
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.6.74+rpt-rpi-2712
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 601128960
  memTotal: 8455880704
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.14.0-1.fc41.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.14.0
    package: netavark-1.14.0-1.fc41.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.14.0
  ociRuntime:
    name: crun
    package: crun-1.20-2.fc41.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.20
      commit: 9c9a76ac11994701dd666c4f0b869ceffb599a66
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250217.ga1e48a0-2.fc41.aarch64
    version: ""
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 144h 50m 54.00s (Approximately 6.00 days)
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.additionalImageStores:
    - /var/lib/shared
    - /usr/lib/containers/storage
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.13-2.fc41.aarch64
      Version: |-
        fusermount3 version: 3.16.2
        fuse-overlayfs: version 1.13-dev
        FUSE library version 3.16.2
        using FUSE kernel interface version 7.38
    overlay.mountopt: nodev,fsync=0
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 984720285696
  graphRootUsed: 109250195456
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.4.0
  BuildOrigin: Fedora Project
  Built: 1739232000
  BuiltTime: Tue Feb 11 00:00:00 2025
  GitCommit: ""
  GoVersion: go1.23.5
  Os: linux
  OsArch: linux/arm64
  Version: 5.4.0

Podman in a container

Yes

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Running in a GitLab runner in a K8S cluster (K3S) in a Raspberry Pi 5.

Additional information

Worked on Podman 5.3, fails on Podman 5.4 (everything else being equal)

[giuseppe]

I don't see any commit except https://github.com/containers/podman/pull/24547/files that could have affected it.

Can you check what is the output of cat /proc/self/limits on the same environment with the two versions of podman?

[Luap99]

My only guess would be that it is a duplicate of #25460, are thes epaths symlinks?

Also the SIGSEGV is from rustc? No matter what that seems like a bug as it should never produce a SIGSEGV but rather produce an actual error?!