Custom networks stopped working, unable to get containers to communicate on Podman 5.3.1 with Pasta

[ElderBlade]

I'm on Fedora Server 40 and after a reboot, all my quadlets stopped working. I was previously on podman 5.2, and after reboot I'm on 5.3.1. Quadlets only work on pasta network or no network specified in the quadlet. Here's my simplest quadlet:

[Unit]
Description=Personal Knowledge Base System

[Container]
Image=docker.io/zefhemel/silverbullet
ContainerName=sbullet
Network=app_net
AutoUpdate=registry
PublishPort=3001:3000
Volume=podman_myspace:/space:Z

[Service]
Restart=unless-stopped

[Install]
WantedBy=multi-user.target default.target

Restarting my quadlets shows an udp listener error at port 53:

journalctl --user -xeu silverbullet.service

Mar 05 10:25:11 optimus-core podman[2612963]: 2025-03-05 10:25:11.757281065 -0800 PST m=+2.991212820 image pull 757bb6a2c84f023fab36fe33fd0bfddef176b269cef87f3e7e7e62de7feec526 docker.io/zefhemel/silverbu>
Mar 05 10:25:12 optimus-core podman[2612963]: 2025-03-05 10:25:12.445516504 -0800 PST m=+3.679448249 container create fcc1e292c1a3c9b49c41cf7394e953f8b1daf6d99992fadd22a9dee6acd5d00b (image=docker.io/zefh>
Mar 05 10:25:12 optimus-core silverbullet[2613789]: [WARN  netavark::dns::aardvark] Failed to delete aardvark-dns entries after failed start: IO error: aardvark-dns failed to start: Error from child proce>
Mar 05 10:25:12 optimus-core silverbullet[2613789]:     Error starting server failed to bind udp listener on 10.89.2.1:53: IO error: Cannot assign requested address (os error 99)
Mar 05 10:25:14 optimus-core podman[2612963]: 2025-03-05 10:25:14.218477945 -0800 PST m=+5.452409700 container remove fcc1e292c1a3c9b49c41cf7394e953f8b1daf6d99992fadd22a9dee6acd5d00b (image=docker.io/zefh>
Mar 05 10:25:14 optimus-core silverbullet[2612963]: Error: netavark: IO error: Error while applying dns entries: IO error: aardvark-dns failed to start: Error from child process
Mar 05 10:25:14 optimus-core silverbullet[2612963]: Error starting server failed to bind udp listener on 10.89.2.1:53: IO error: Cannot assign requested address (os error 99)
Mar 05 10:25:14 optimus-core systemd[1452]: silverbullet.service: Main process exited, code=exited, status=126/n/a

Creating a new network podman network create my_network does not resolve this error.

podman network inspect app_net

[
     {
          "name": "app_net",
          "id": "3ed0bdc15bf9c51b911ce761cb7c602828ddfdd77d17b25979f2b88e1e751e66",
          "driver": "bridge",
          "network_interface": "podman4",
          "created": "2025-03-03T20:47:27.597260687-08:00",
          "subnets": [
               {
                    "subnet": "10.89.0.0/24",
                    "gateway": "10.89.0.1"
               }
          ],
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": true,
          "ipam_options": {
               "driver": "host-local"
          },
          "containers": {}
     }
]

sudo netstat -tuln | grep :53

tcp        0      0 0.0.0.0:5355            0.0.0.0:*               LISTEN     
tcp6       0      0 :::5355                 :::*                    LISTEN     
udp        0      0 0.0.0.0:5355            0.0.0.0:*                          
udp6       0      0 :::5355                 :::*                               

I've tried

• podman network prune
• systemctl restart podman
• systemctl --user daemon-reload
• sudo systemctl enable podman.socket
• sudo systemctl start podman.socket
• podman network rm app_net
• podman network create app_net
• podman network create app_net --subnet 10.89.3.0/24 --gateway 10.89.3.1 --ip-range 10.89.3.0/24

So now I'm trying to use pasta since that is the only network any of my quadlets will even start with. The problem now is nginx proxy manager can't access any of the services.

I've tried adding the map-gw option which worked, and now doesn't, simply between two containers. I can't put them in a pod together because both containers use port 3000 internally.

nginx-proxy-manager.container

[Unit]
Description=Nginx Reverse Proxy

[Container]
Image=docker.io/jc21/nginx-proxy-manager:latest
ContainerName=npm
Network=pasta:--map-gw,-T,3001

PublishPort=80:80
PublishPort=81:81
PublishPort=443:443

Volume=podman_nginx:/data
Volume=podman_letsencrypt:/etc/letsencrypt

[Service]
Restart=always

[Install]
WantedBy=multi-user.target default.target

Accessing the encrypted url for silverbullet results in a bad gateway error.

podman info

host:
  arch: amd64
  buildahVersion: 1.38.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-2.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 86.02
    systemPercent: 12.12
    userPercent: 1.86
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: server
    version: "40"
  eventLogger: journald
  freeLocks: 2010
  hostname: optimus-core
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 70001
    - container_id: 70002
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 70001
    - container_id: 70002
      host_id: 524288
      size: 65536
  kernel: 6.10.12-200.fc40.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 21110853632
  memTotal: 66508005376
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.2-2.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.2
    package: netavark-1.12.2-1.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.2
  ociRuntime:
    name: crun
    package: crun-1.17-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.17
      commit: 000fa0d4eeed8938301f3bcf8206405315bc1017
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240906.g6b38f07-1.fc40.x86_64
    version: |
      pasta 0^20240906.g6b38f07-1.fc40.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 37h 4m 34.00s (Approximately 1.54 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 6
    paused: 0
    running: 4
    stopped: 2
  graphDriverName: btrfs
  graphOptions: {}
  graphRoot: /home/user/containers/storage
  graphRootAllocated: 1099511627776
  graphRootUsed: 49891356672
  graphStatus:
    Build Version: Btrfs v6.11
    Library Version: "104"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 29
  runRoot: /home/user/containers/run
  transientStore: false
  volumePath: /home/user/containers/storage/volumes
version:
  APIVersion: 5.3.1
  Built: 1732147200
  BuiltTime: Wed Nov 20 16:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.7
  Os: linux
  OsArch: linux/amd64
  Version: 5.3.1

I'm completely dead in the water here. Can't get anything working.

[Luap99]

runRoot: /home/user/containers/run

That seems totally wrong, did you configure a custom runroot anywhere?

The runroot should be on tmpfs or must be deleted on each boot before podman is run.

Generally the runroot default to /run/user/$UID/containers when you have properly logged in and have a valid systemd user session, i.e. not using su to switch users https://github.com/containers/podman/blob/main/troubleshooting.md#solution-28.

Note the podman db caches that path after first use so you need to wipe everything clean (podman system reset) and then run podman info again for a proper user session. Then it should have a valid path.
A short term fix might be to just remove all the network configs manually from the runroot, also see containers/netavark#1174 for the paths

#25487

[ElderBlade]

Yeah that was it.

I backed up my volumes with podman export my_volume --output my_volume.tar to a local directory in home.

Ran podman system reset and then modified my ~/.config/containers/storage.conf to this:

[storage]
driver = "btrfs"
runroot = "/run/user/1000"
graphroot = "/home/user/containers/storage"

Rebooted and now my quadlets work again. Restoring my volumes was as simple as using podman volume import my_volume my_volume.tar