Error running rootless podman image

[GarbageYard]

We have been using DIND so far for our GitLab pipeline and want to switch to podman in order to achieve the following:

1. Use non-root image (using USER in Containerfile) for our GitLab pipeline
2. Use rootless podman for our GitLab runner (Nice to have)

I have a Ubuntu 24.04 host where I have installed gitlab-runner and podman (v4.9.3) using instructions from this section.

On gitlab-runner host:

$ cat /etc/subuid
gitlab-runner:100000:65536
$ cat /etc/subgid
gitlab-runner:100000:65536

$ sudo cat /etc/gitlab-runner/config.toml
concurrent = 1
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0
[session_server]
  session_timeout = 1800
[[runners]]
  name = "A Working Man"
  url = "https://gitlab.company.com/"
  id = 3238
  token = "xxxxxxxxxxxx"
  executor = "docker"
  environment = ["FF_NETWORK_PER_BUILD=1"]
  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.docker]
    tls_verify = false
    image = "quay.io/podman/stable"
    privileged = true
    host = "unix:///run/user/986/podman/podman.sock"
    security_opt = ["apparmor:unconfined"]
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    network_mtu = 0

I also created a Containerfile based on instructions from this page.

# cat Containerfile 
FROM opensuse/leap:15.6

ARG USERNAME=gitlab-runner
ARG USER_UID=1010
ARG USER_GID=$USER_UID

RUN zypper install -y \
        podman \
        crun \
        fuse-overlayfs

RUN groupadd --gid $USER_GID $USERNAME && \
    useradd --uid $USER_UID --gid $USER_GID -m $USERNAME && \
    mkdir -p /home/$USERNAME

#VOLUME /home/$USERNAME/.local/share/containers
COPY podman-containers.conf /home/$USERNAME/.config/containers/containers.conf
COPY containers.conf /etc/containers/containers.conf
RUN mkdir -p /home/$USERNAME/.local/share/containers/storage/libpod
RUN chown $USERNAME:$USERNAME -R /home/$USERNAME

RUN chmod 644 /etc/containers/containers.conf; sed -i -e 's|^#mount_program|mount_program|g' -e '/additionalimage.*/a "/var/lib/shared",' -e 's|^mountopt[[:space:]]*=.*$|mountopt = "nodev,fsync=0"|g' /etc/containers/storage.conf
RUN mkdir -p /var/lib/shared/overlay-images /var/lib/shared/overlay-layers /var/lib/shared/vfs-images /var/lib/shared/vfs-layers; touch /var/lib/shared/overlay-images/images.lock; touch /var/lib/shared/overlay-layers/layers.lock; touch /var/lib/shared/vfs-images/images.lock; touch /var/lib/shared/vfs-layers/layers.lock

ENV _CONTAINERS_USERNS_CONFIGURED=""

USER $USERNAME
WORKDIR /home/$USERNAME

GitLab pipeline error:

ERRO[0000] running `/usr/bin/newuidmap 28 0 1010 1 1 100000 65536`: newuidmap: write to uid_map failed: Operation not permitted 
Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1

When I enable the VOLUME instruction in above Containerfile:

VOLUME /home/$USERNAME/.local/share/containers
COPY podman-containers.conf /home/$USERNAME/.config/containers/containers.conf
...
...

I get error:

Error: creating runtime static files directory "/home/gitlab-runner/.local/share/containers/storage/libpod": mkdir /home/gitlab-runner/.local/share/containers/storage: permission denied

I also found this comment and tried the following in the /etc/gitlab-runner/config.toml on the gitlab-runner host:

    privileged = false
    cap_add = ["CAP_SETUID", "CAP_SETGID", "CAP_SYS_ADMIN", "CAP_DAC_OVERRIDE", "CAP_FOWNER"]
    host = "unix:///run/user/986/podman/podman.sock"
    security_opt = ["apparmor:unconfined"]

Again got the same permission denied error:

Error: creating runtime static files directory "/home/gitlab-runner/.local/share/containers/storage/libpod": mkdir /home/gitlab-runner/.local/share/containers/storage: permission denied

Note: All above errors show up when podman info command is run in the GitLab pipeline using the above non-root image.

Output of podman info on Ubuntu host:

$ podman info
host:
  arch: amd64
  buildahVersion: 1.33.7
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.10+ds1-1build2_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: unknown'
  cpuUtilization:
    idlePercent: 99.83
    systemPercent: 0.05
    userPercent: 0.12
  cpus: 16
  databaseBackend: sqlite
  distribution:
    codename: noble
    distribution: ubuntu
    version: "24.04"
  eventLogger: journald
  freeLocks: 2044
  hostname: devops-eu
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 980
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 986
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.8.0-55-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 20137873408
  memTotal: 33855602688
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.4.0-5_amd64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.4.0
    package: netavark_1.4.0-4_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.4.0
  ociRuntime:
    name: crun
    package: crun_1.14.1-1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.1
      commit: de537a7965bfbe9992e2cfae0baeb56a08128171
      rundir: /run/user/986/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_0.0~git20240220.1e6f92b-1_amd64
    version: |
      pasta unknown version
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/986/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.1-1build2_amd64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 19h 7m 54.00s (Approximately 0.79 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/gitlab-runner/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/gitlab-runner/.local/share/containers/storage
  graphRootAllocated: 534458142720
  graphRootUsed: 37885849600
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /tmp/containers-user-986/containers
  transientStore: false
  volumePath: /home/gitlab-runner/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.3
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.22.2
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.3

Whatever I try, I keep getting one of the above mentioned errors. I will really appreciate any help.