podman info hangs after changing uid mappings

[hwalinga]

After changing uid mapping we first got this:

ERRO [0000] running /usr/bin/newuidmap XXXX 0 XXXX 1 1 XXXXXX 65536: newuidmap : write to uid_map failed: Operation not permitted
Error: cannot set up namespace using "/usr/bin/newuidmap": should have setuid or have filecaps setuid: exit status 1

Then following https://access.redhat.com/solutions/7006710 podman info just hangs.

Output with log-level=debug

INFO[0000] podman filtering at log level debug
DEBU[0000] Called info.PersistentPreRunE(podman --log-level=debug info)
DEBU[0000] Using conmon: "/usr/bin/conmon"
INFO[0000] Using sqlite as database backend
DEBU[0000] systemd-logind: Unknown object '/'.
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /trinity/local/podman/P094451/containers/storage
DEBU[0000] Using run root /run/user/1241/containers
DEBU[0000] Using static dir /trinity/local/podman/P094451/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1241/libpod/tmp
DEBU[0000] Using volume path /home/P094451/my-podman-volumes
DEBU[0000] Using transient store: false
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: storage already configured with a mount-program
DEBU[0000] backingFs=tmpfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
DEBU[0000] Initializing event backend file
DEBU[0000] Configured OCI runtime crun-vm initialization failed: no valid executable found for OCI runtime crun-vm: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"

[hwalinga]

I tried debugging this further and the best I could produce was with:

strace -f -v -s 100 podman system reset

Which give me this interesting block:

[pid 1238045] ioctl(2, TCGETS, {c_iflags=0x4d00, c_oflags=0x5, c_cflags=0xbf, c_lflags=0x8a3b, c_line=0, c_cc="\x03\x1c\x08\x15\x04\x00\x01\x00\x11\x13\x1a\x00\x12\x0f\x17\x16\x00\x00\x00"}) = 0
[pid 1238045] write(2, "\33[31mERRO\33[0m[0000] running `/usr/bin/newuidmap 1238048 0 1241 1 1 100000 65536`: newuidmap: write t"..., 143ERRO[0000] running `/usr/bin/newuidmap 1238048 0 1241 1 1 100000 65536`: newuidmap: write to uid_map failed: Operation not permitted
) = 143
[pid 1238045] newfstatat(AT_FDCWD, "/usr/bin/newuidmap", {st_dev=makedev(0, 0x23), st_ino=88573, st_mode=S_IFREG|0755, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=80, st_size=39016, st_atime=1749655492 /* 2025-06-11T17:24:52.826398649+0200 */, st_atime_nsec=826398649, st_mtime=1726234254 /* 2024-09-13T15:30:54+0200 */, st_mtime_nsec=0, st_ctime=1749655462 /* 2025-06-11T17:24:22.079532076+0200 */, st_ctime_nsec=79532076}, 0) = 0
[pid 1238045] getxattr("/usr/bin/newuidmap", "security.capability", "\1\0\0\2\200\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 20) = 20
[pid 1238045] write(9, "1", 1)          = 1
[pid 1238048] <... read resumed>"1", 1) = 1
[pid 1238045] kill(1238048, SIGKILL)    = 0
[pid 1238045] wait4(1238048,  <unfinished ...>
[pid 1238048] +++ killed by SIGKILL +++
[pid 1238045] <... wait4 resumed>[{WIFSIGNALED(s) && WTERMSIG(s) == SIGKILL}], 0, NULL) = 1238048
[pid 1238045] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=1238048, si_uid=1241, si_status=SIGKILL, si_utime=0, si_stime=0} ---
[pid 1238045] rt_sigreturn({mask=[]})   = 1238048
[pid 1238045] close(9)                  = 0
[pid 1238045] close(7)                  = 0
[pid 1238045] write(2, "Error: cannot set up namespace using \"/usr/bin/newuidmap\": exit status 1\n", 73Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1
) = 73
[pid 1238045] exit_group(125 <unfinished ...>

[eriksjolund]

Error: cannot set up namespace using "/usr/bin/newuidmap": should have setuid or have filecaps setuid: exit status 1

These commands could provide interesting information:

ls -lZ /usr/bin/newuidmap 

getcap /usr/bin/newuidmap

cat /proc/sys/user/max_user_namespaces

I tried them out on my Fedora CoreOS system:

$ ls -lZ /usr/bin/newuidmap 
-rwxr-xr-x. 1 root root system_u:object_r:bin_t:s0 72088 Jan  1  1970 /usr/bin/newuidmap
$ getcap /usr/bin/newuidmap
/usr/bin/newuidmap cap_setuid=ep
$ cat /proc/sys/user/max_user_namespaces
22971
$