[emerg] 1#1: host not found in upstream

[truong-hua]

Currently Nginx will immediately exit if the server in upstream directive is not resolvable. I think it should be by passed so that we can start the nginx independently from other services. In Docker Swarm the nginx service may be started faster than others or any bug in the others service will cause service domain to be unresolvable.

[zdm]

To work in docker swarm as load balancer it also must remove ip addresses from the upstream, if dns request returns nothing.

[ZigzagAK]

DNS resolving begins on worker process start phase.
Resolving servers performed on init phase. Init phase is work when master process reads configuration.
To resolve your requirements we need to create new upstream module. Standard upstream modules like default round robin or least_conn perform standard resolve on init (parse config in master process context) phase and we can't change this behavior.
We can replace ip addresses with 0.0.0.0:1 for example in background resolving process, but on start (reload) we can't do it.

[ZigzagAK]

Resolve is performed here https://github.com/nginx/nginx/blob/master/src/http/ngx_http_upstream.c#L5982.

[ZigzagAK]

We can't use standard server directive if we want to skip not resolvable hosts. We need to create own directive (similar to server) to add servers into upstream, which will skip not resolvable hosts.

[truong-hua]

Thank @ZigzagAK and got it, so do you think that our healthcheck module will compatible with this https://github.com/nicholaschiasson/ngx_upstream_jdomain or not?

[truong-hua]

Dear, is it possible to override just the resolver to response 0.0.0.0:1 on that case? Seem like this module is doing that? https://github.com/GUI/nginx-upstream-dynamic-servers

[ZigzagAK]

@huaphuoctruong
No. ngx_dynamic_healthcheck requires zone directive. This module resolves servers independently in each worker. ngx_dynamic_healthcheck is work with peers in shared memory.

[ZigzagAK]

@zdm , yes we can, but we must implement our directive, as i wrote before.
This module has memory issue in https://github.com/GUI/nginx-upstream-dynamic-servers/blob/master/ngx_http_upstream_dynamic_servers.c#L528.
Memory pool can't be destroyed before it is still in use in other requests (addrs allocated from it).
This module also can't work with ngx_dynamic_healthcheck.

[zdm]

I understand, this is complex.
You can at least describe in readme this case, that it is not designed to use in environment, where ip addresses of upstreams can be not available at any moment, for example in cluster, where services can be started / stopped dynamically and get ip addresses via dhcp.

[truong-hua]

Thank @ZigzagAK. Btw, I have to do a workaround solution. I made a simple patch here. I see that these codes have been there for over 7 years already so this patch would work with many nginx version (I hope so and testing now).

From e3d469cde4f1150d8d8cd6096d65c8dbf2cdd0d0 Mon Sep 17 00:00:00 2001
From: "truong.hua" <[email protected]>
Date: Wed, 7 Apr 2021 14:55:57 +0700
Subject: [PATCH] Allow nginx to start even the upstream server hostname is not
 resolvable on startup

---
 src/http/ngx_http_upstream.c | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

diff --git a/src/http/ngx_http_upstream.c b/src/http/ngx_http_upstream.c
index e66ba328..f376a9fa 100644
--- a/src/http/ngx_http_upstream.c
+++ b/src/http/ngx_http_upstream.c
@@ -504,6 +504,8 @@ ngx_conf_bitmask_t  ngx_http_upstream_ignore_headers_masks[] = {
     { ngx_null_string, 0 }
 };
 
+static ngx_str_t ngx_http_upstream_server_null_route = ngx_string("127.255.255.255");
+
 
 ngx_int_t
 ngx_http_upstream_create(ngx_http_request_t *r)
@@ -6183,11 +6185,28 @@ ngx_http_upstream_server(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
 
     if (ngx_parse_url(cf->pool, &u) != NGX_OK) {
         if (u.err) {
-            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
+            ngx_conf_log_error(NGX_LOG_WARN, cf, 0,
                                "%s in upstream \"%V\"", u.err, &u.url);
         }
 
-        return NGX_CONF_ERROR;
+        // If the domain fails to resolve on start up, mark this server as down,
+        // and assign a static IP that should never route. This is to account for
+        // various things inside nginx that seem to expect a server to always have
+        // at least 1 IP.
+        us->down = 1;
+
+        u.url = ngx_http_upstream_server_null_route;
+        u.default_port = u.port;
+        u.no_resolve = 1;
+
+        if (ngx_parse_url(cf->pool, &u) != NGX_OK) {
+            if (u.err) {
+                ngx_conf_log_error(NGX_LOG_ERR, cf, 0,
+                                   "%s in upstream \"%V\"", u.err, &u.url);
+            }
+
+            return NGX_CONF_ERROR;
+        }
     }
 
     us->name = u.url;
-- 
2.18.0.windows.1

[ZigzagAK]

Yes, it is should work, but i don't like to patch nginx core. Separate directive is more attractive.

[ZigzagAK]

@zdm
I will implement your requirements when i will get free time.
I don't use this module in dynamic environment when dns lookups may fails because no addresses for hosts.
And I didn't think about this situation.

[zdm]

ok, thank you very much
i already solved this by created a service, that updates upstreams via http api, but dns will be preferred.

[ZigzagAK]

You may try https://github.com/ZigzagAK/ngx_http_upsync_upstream. This module has one limitation - chunked response is not supported. You must use http 1.0 or responses with content-length. Chunked response parse may be ported from ngx_dynamic_healthcheck but now it is not required for me.

[zdm]

i'll take a look, thanks