Proxy trying to reach unknown container

[Mohitsharma44]

When running the proxy with jupyterhub in a multi-node swarm configuration, as mentioned in swarmspawner example (https://github.com/jupyterhub/dockerspawner/tree/master/examples/swarm), proxy is not able to resolve to the correct ip of the hub container.
Attached is more information:

Logs from docker-compose up:

Creating jupyterhub_proxy_1 ... done
Creating jupyterhub_hub_1   ... done
Attaching to jupyterhub_proxy_1, jupyterhub_hub_1
proxy_1  | 19:18:56.653 - warn: [ConfigProxy] REST API is not authenticated.
hub_1    | [I 2018-12-06 19:18:58.298 JupyterHub app:1667] Using Authenticator: oauthenticator.generic.GenericOAuthenticator-0.8.0
hub_1    | [I 2018-12-06 19:18:58.298 JupyterHub app:1667] Using Spawner: dockerspawner.swarmspawner.SwarmSpawner-0.10.0
hub_1    | [D 2018-12-06 19:18:58.301 JupyterHub app:1048] Generating new cookie_secret
hub_1    | [I 2018-12-06 19:18:58.302 JupyterHub app:1053] Writing cookie_secret to /srv/jupyterhub/jupyterhub_cookie_secret
hub_1    | [D 2018-12-06 19:18:58.302 JupyterHub app:1069] Connecting to db: sqlite:///jupyterhub.sqlite
hub_1    | [D 2018-12-06 19:18:58.317 JupyterHub orm:656] Stamping empty database with alembic revision 896818069c98
hub_1    | [I 2018-12-06 19:18:58.322 alembic.runtime.migration migration:117] Context impl SQLiteImpl.
hub_1    | [I 2018-12-06 19:18:58.322 alembic.runtime.migration migration:122] Will assume non-transactional DDL.
hub_1    | [I 2018-12-06 19:18:58.330 alembic.runtime.migration migration:327] Running stamp_revision  -> 896818069c98
hub_1    | [D 2018-12-06 19:18:58.330 alembic.runtime.migration migration:516] new branch insert 896818069c98
hub_1    | [W 2018-12-06 19:18:58.361 JupyterHub app:1171] No admin users, admin interface will be unavailable.
hub_1    | [W 2018-12-06 19:18:58.361 JupyterHub app:1172] Add any administrative users to `c.Authenticator.admin_users` in config.
hub_1    | [I 2018-12-06 19:18:58.361 JupyterHub app:1199] Not using whitelist. Any authenticated user will be allowed.
hub_1    | [D 2018-12-06 19:18:58.391 JupyterHub app:1487] Loaded users:
hub_1    |     
hub_1    | [I 2018-12-06 19:18:58.400 JupyterHub app:1849] Hub API listening on http://0.0.0.0:8081/hub/
hub_1    | [I 2018-12-06 19:18:58.400 JupyterHub app:1851] Private Hub API connect url http://hub:8081/hub/
hub_1    | [I 2018-12-06 19:18:58.400 JupyterHub app:1864] Not starting proxy
hub_1    | [D 2018-12-06 19:18:58.400 JupyterHub proxy:296] Fetching routes to check
proxy_1  | 19:18:56.665 - info: [ConfigProxy] Proxying http://*:8000 to (no default)
proxy_1  | 19:18:56.665 - info: [ConfigProxy] Proxy API at http://0.0.0.0:8001/api/routes
hub_1    | [D 2018-12-06 19:18:58.401 JupyterHub proxy:678] Proxy: Fetching GET http://proxy:8001/api/routes
proxy_1  | 19:19:01.488 - info: [ConfigProxy] 200 GET /api/routes 
proxy_1  | 19:19:01.491 - info: [ConfigProxy] Adding route / -> http://hub:8081
hub_1    | [I 2018-12-06 19:19:01.488 JupyterHub proxy:301] Checking routes
hub_1    | [I 2018-12-06 19:19:01.488 JupyterHub proxy:370] Adding default route for Hub: / => http://hub:8081
hub_1    | [D 2018-12-06 19:19:01.488 JupyterHub proxy:678] Proxy: Fetching POST http://proxy:8001/api/routes/
proxy_1  | 19:19:01.493 - info: [ConfigProxy] 201 POST /api/routes/ 
proxy_1  | 19:19:14.216 - error: [ConfigProxy] 503 GET /hub/login Error: connect EHOSTUNREACH 10.0.1.8:8081
proxy_1  |     at Object.exports._errnoException (util.js:1020:11)
proxy_1  |     at exports._exceptionWithHostPort (util.js:1043:20)
proxy_1  |     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1090:14)
proxy_1  | 19:19:17.285 - error: [ConfigProxy] Failed to get custom error page Error: connect EHOSTUNREACH 10.0.1.8:80
proxy_1  |     at Object.exports._errnoException (util.js:1020:11)
proxy_1  |     at exports._exceptionWithHostPort (util.js:1043:20)
proxy_1  |     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1090:14)
proxy_1  | 19:19:20.356 - error: [ConfigProxy] 503 GET /favicon.ico Error: connect EHOSTUNREACH 10.0.1.8:8081
proxy_1  |     at Object.exports._errnoException (util.js:1020:11)
proxy_1  |     at exports._exceptionWithHostPort (util.js:1043:20)
proxy_1  |     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1090:14)
proxy_1  | 19:19:23.428 - error: [ConfigProxy] Failed to get custom error page Error: connect EHOSTUNREACH 10.0.1.8:80
proxy_1  |     at Object.exports._errnoException (util.js:1020:11)
proxy_1  |     at exports._exceptionWithHostPort (util.js:1043:20)
proxy_1  |     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1090:14)

> .... this goes on and on

So basically, its trying to reach a container at 10.0.1.8 ip address. However, there is no such container in the overlay network:

        "Containers": {
            "17c349b0d7a096c1983623f7cdb3b589cc4e054e97e2cb78927f9a77bb85c2e1": {
                "Name": "docker-gen",
                "EndpointID": "4048a39235e9918db99fb93e2bed80a22235c5db6e1cf0d267399f14803266f9",
                "MacAddress": "02:42:0a:00:01:05",
                "IPv4Address": "10.0.1.5/24",
                "IPv6Address": ""
            },
            "6200cdbae90fe78f14b894669fc712f6d085d374f1b044fb053799613b43d7db": {
                "Name": "jupyterhub_proxy_1",
                "EndpointID": "6cce3e159b8192bacf0d44e3736625458213456a0dfa96b862d0129aaea05cfe",
                "MacAddress": "02:42:0a:00:01:09",
                "IPv4Address": "10.0.1.9/24",
                "IPv6Address": ""
            },
            "64fb609962bf5aaa9f45bda64526e2c72c9e3970c7031a0a8dc608a3b05512dd": {
                "Name": "jupyterhub_hub_1",
                "EndpointID": "507cbf90b8f36f79c4fb45cb13bf51cf2c4a37de153d658606fc891c3e76a3be",
                "MacAddress": "02:42:0a:00:01:0a",
                "IPv4Address": "10.0.1.10/24",
                "IPv6Address": ""
            },
            "d433fbecb50a57dd2eec3504b456fe37a483610acd460bd3b03745a13747f2d5": {
                "Name": "nginx",
                "EndpointID": "ebe89a238bb9334cd21d30f5262ac26d85ea5479ed833091ef8bc47258c8df23",
                "MacAddress": "02:42:0a:00:01:06",
                "IPv4Address": "10.0.1.6/24",
                "IPv6Address": ""
            }
        },

Sometimes, proxy is able to obtain the correct IP of the hub but most of the times it simply complains that it cannot reach the host.
On the webpage, I get the Service Unavailable error.

(@minrk could it also be one of the reasons for the issue that I mentioned at dockerspawner #279 ?)

[Mohitsharma44]

After close inspection, I saw that the proxy server was always trying to reach the older ip of the hub (before docker-compose down).
Later, realized that I had the same overlay network attached to the nginx container for reverse proxying so docker-compose down was not cleaning up the overlay network.

The configurable-http-proxy is working perfectly fine. No issues whatsoever.