Colima bridge network connectivity port issues

[GroveJay]

Hello,

I am not a networking expert, so please be patient with my ignorance.

TL;DR

Bridge networks don't seem to work as I expect them to?
My possibly misguided expectation from the docker documentation:

Containers connected to the same user-defined bridge network effectively expose all ports to each other.

However, ports seem to be randomly mapped on each connection and I have a fairly minimal test setup that appears to showcases this.

Context

I'm bringing up a Synapse (Matrix) server and hit issues adding mautrix-bridges, specifically the mautrix-meta bridge which i'm running as a container named mautrix-meta-messenger.

Synapse and its postgres db are working (after strife related to file permissions) but Synapse can't connect to the mautrix-meta bridge. When the bridge makes a request to Synapse (a request to test its connectivity with the bridge), it fails: synapse.http.client - 451 - INFO - POST-63 - Error sending request to POST http://mautrix-meta-messenger:29319/_matrix/app/v1/ping: ConnectionRefusedError Connection refused.

This category of error is mentioned in the mautrix FAQ here where they say to test the connectivity between Synapse and the bridge.

Initial Testing

Leaving the Synapse container running, I tested connectivity using nc in the mautrix-meta-messenger container and running a python one-liner from the Synapse container.

Setup Details

The existing setup details are the following:

$ colima list
PROFILE    STATUS     ARCH      CPUS    MEMORY    DISK     RUNTIME    ADDRESS
default    Stopped    x86_64    4       8GiB      60GiB
vz         Running    x86_64    4       8GiB      60GiB    docker
$ colima status -p vz
INFO[0001] colima [profile=vz] is running using macOS Virtualization.Framework
INFO[0001] arch: x86_64
INFO[0001] runtime: docker
INFO[0001] mountType: virtiofs
INFO[0001] socket: unix:///Users/johngrover/.colima/vz/docker.sock

$ docker ps
CONTAINER ID   IMAGE                                                COMMAND                  CREATED        STATUS                  PORTS                                                 NAMES
499659ee7863   matrixdotorg/synapse:latest                          "/start.py"              7 hours ago    Up 7 hours (healthy)    8009/tcp, 127.0.0.1:8008->8008/tcp, 8448/tcp          synapse-data-synapse-1
61173f4ac204   postgres:latest                                      "docker-entrypoint.s…"   7 hours ago    Up 7 hours              5432/tcp                                              synapse-data-db-1

$ docker network ls
NETWORK ID     NAME                           DRIVER    SCOPE
[...]
3bed792a6841   synapse-data_default           bridge    local
$ docker network inspect synapse-data_default
[
    {
        "Name": "synapse-data_default",
        "Id": "3bed792a6841e9a5aa6f5c81f34f3fa0aab175e4fec3c7a3b724dc68038ac3b9",
        "Created": "2025-03-20T10:57:16.423819649-07:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.21.0.0/16",
                    "Gateway": "172.21.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "499659ee7863c2e8069dc31f7391624434037f46a2841bafff7b0b0473252daf": {
                "Name": "synapse-data-synapse-1",
                "EndpointID": "73594c7959f861e12cf3124711ed63a99a656504ec5aeb846c9f1d4078a75e48",
                "MacAddress": "02:42:ac:15:00:03",
                "IPv4Address": "172.21.0.3/16",
                "IPv6Address": ""
            },
            "61173f4ac204fdfed3b32eb0412e5ee840c86bba25951513837be0aa6ee490df": {
                "Name": "synapse-data-db-1",
                "EndpointID": "3d50994247759e49b8dbce68288d8ce0cb0a485f8f9760ea14e2bfc34b33d99b",
                "MacAddress": "02:42:ac:15:00:02",
                "IPv4Address": "172.21.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "synapse-data",
            "com.docker.compose.version": "2.29.7"
        }
    }
]

Test Result

Running a basic test went as follows:

1. Run nc to listen on a port in the mautrix-meta-messenger container

% docker run --rm --name mautrix-meta-messenger --network synapse-data_default dock.mau.dev/mautrix/meta:latest nc -v -l -p 29319 "127.0.0.1"
listening on 0.0.0.0:29319 ...

2. Run a python one-liner to test connecting to the listener (nc wasn't available and this seems pretty minimal)

docker exec -u 0 -ti synapse-data-synapse-1 python -c 'import socket; s = socket.socket(); s.connect(("mautrix-meta-messenger", 29319)); s.close();'

Running the above three times produces this in the mautrix-meta-messenger output:

nc: connect from wrong ip/port 172.21.0.3:46748 ignored
nc: connect from wrong ip/port 172.21.0.3:46760 ignored
nc: connect from wrong ip/port 172.21.0.3:46770 ignored

So what gives? Why is the traffic coming in at these random ports?
I tried a bunch of other ports of lower ranges and it produced similar results.

Minimal Example

I created a more minimal example to observe the same thing:

• Create a test_bridge bridge network
• Run nc as a listener on a test_receiver container attached to test_bridge
• Repeatedly connect to that container from a test_sender container attached to test_bridge

Results

Running the test_receiver:

docker run --rm --name test_receiver --network test_bridge golang:1-alpine3.21 nc -v -l -p 29319 "127.0.0.1"
listening on 0.0.0.0:29319 ...

Running the test_sender three times:

$ echo "Hello" | docker run --rm -i --name test_sender --network test_bridge golang:1-alpine3.21 nc -v "test_receiver" 29319
test_receiver (172.22.0.2:29319) open
$ echo "Hello" | docker run --rm -i --name test_sender --network test_bridge golang:1-alpine3.21 nc -v "test_receiver" 29319
test_receiver (172.22.0.2:29319) open
$ echo "Hello" | docker run --rm -i --name test_sender --network test_bridge golang:1-alpine3.21 nc -v "test_receiver" 29319
test_receiver (172.22.0.2:29319) open

The resulting output in test_receiver:

nc: connect from wrong ip/port 172.22.0.3:33577 ignored
nc: connect from wrong ip/port 172.22.0.3:38187 ignored
nc: connect from wrong ip/port 172.22.0.3:41273 ignored

Summary

I can't say the test necessarily relates to the Synapse -> mautrix-meta connectivity failure. I know in general connectivity is working because we see output on the receiving side, but it seems like the port is randomly wrong.
I'm guessing the Synapse -> postgres connectivity and mautrix-meta -> postgres connectivity works because of the explicitly exposed port on the postgres container?
I tried explicitly exposing a port --expose 29319 and that didn't seem to help though.

[GroveJay]

I still don't understand anything, but changing the appservice.hostname property for the mautrix-meta bridge seems to have fixed this particular issue for me. This seems to mean that listening at 127.0.0.1 is distinctly different from 0.0.0.0 inside a docker container?
Re-running the tests using 0.0.0.0 still produced the port swapping so I still don't know what to think.