Air-Gapped DNS issues

[fleroviumactinium]

Hello!

Starting a discussion here as advised on Slack, where I asked this question last Friday.
(https://taloscommunity.slack.com/archives/CMARMBC4E/p1755862373071119)

I'm trying to get Talos Linux up and running in an air-gapped environment. I have an internal harbor registry running from which I can pull images (tested, works). However, when starting up the control plane, I get these errors where it tries to use 8.8.8.8 as DNS to resolve my internal registry:

192.168.57.51: user: warning: [2025-08-22T11:14:11.541086942Z]: level=info msg=trying next host error=failed to do request: Head "https://localregistry/v2/siderolabs/kubelet/manifests/v1.33.1?ns=ghcr.io": dial tcp: lookup localregistry on 8.8.8.8:53: read udp 192.168.57.51:47656->8.8.8.8:53: i/o timeout host=localregistry image=ghcr.io/siderolabs/kubelet:v1.33.1
192.168.57.51: user: warning: [2025-08-22T11:14:11.541995942Z]: level=info msg=trying next host error=failed to do request: Head "https://localregistry/v2/etcd-development/etcd/manifests/v3.5.21?ns=gcr.io": dial tcp: lookup localregistry on 8.8.8.8:53: read udp 192.168.57.51:47656->8.8.8.8:53: i/o timeout host=localregistry image=gcr.io/etcd-development/etcd:v3.5.21
192.168.57.51: user: warning: [2025-08-22T11:14:15.170415942Z]: level=info msg=fetch failed error=failed to do request: Head "https://gcr.io/v2/etcd-development/etcd/manifests/v3.5.21": dial tcp: lookup gcr.io on 127.0.0.53:53: server misbehaving host=gcr.io image=gcr.io/etcd-development/etcd:v3.5.21
192.168.57.51: user: warning: [2025-08-22T11:14:15.170991942Z]: level=info msg=fetch failed error=failed to do request: Head "https://ghcr.io/v2/siderolabs/kubelet/manifests/v1.33.1": dial tcp: lookup ghcr.io on 127.0.0.53:53: server misbehaving host=ghcr.io image=ghcr.io/siderolabs/kubelet:v1.33.1
192.168.57.51: user: warning: [2025-08-22T11:14:15.178871942Z]: [talos] retrying error: failed to pull image "gcr.io/etcd-development/etcd:v3.5.21": failed to resolve reference "gcr.io/etcd-development/etcd:v3.5.21": failed to do request: Head "https://localregistry/v2/etcd-development/etcd/manifests/v3.5.21?ns=gcr.io": dial tcp: lookup localregistry on 8.8.8.8:53: read udp 192.168.57.51:47656->8.8.8.8:53: i/o timeout
192.168.57.51: user: warning: [2025-08-22T11:14:15.181288942Z]: [talos] retrying error: failed to pull image "ghcr.io/siderolabs/kubelet:v1.33.1": failed to resolve reference "ghcr.io/siderolabs/kubelet:v1.33.1": failed to do request: Head "https://localregistry/v2/siderolabs/kubelet/manifests/v1.33.1?ns=ghcr.io": dial tcp: lookup localregistry on 8.8.8.8:53: read udp 192.168.57.51:47656->8.8.8.8:53: i/o timeout
192.168.57.51: user: warning: [2025-08-22T11:14:16.783073942Z]: [talos] etcd is waiting to join the cluster, if this node is the first node in the cluster, please run `talosctl bootstrap` against one of the following IPs:
192.168.57.51: user: warning: [2025-08-22T11:14:16.783437942Z]: [talos] [192.168.57.51]

This is the cp.patch.yaml I use:

- op: add
  path: /machine/network
  value:
    interfaces:
    - interface: eth0
      addresses:
      - 192.168.57.51/24 
      routes:
      - network: 0.0.0.0/0
        gateway: 192.168.57.254
    nameservers:
    - 192.168.52.51
    - 192.168.52.52

- op: replace
  path: /cluster/extraManifests
  value: []

- op: replace
  path: /cluster/discovery
  value:
    enabled: false

- op: add
  path: /machine/files
  value: []

- op: replace
  path: /machine/time
  value:
    disabled: false
    servers:
    - 10.182.227.253

- op: replace
  path: /machine/registries
  value:
    mirrors:
      ghcr.io:
        endpoints:
        - https://localregistry/
      gcr.io:
        endpoints:
        - https://localregistry/
      docker.io:
        endpoints:
        - https://localregistry/
      registry.k8s.io:
        endpoints:
        - https://localregistry/
      quay.io:
        endpoints:
        - https://localregistry/
    config:
      localregistry:
        auth:
          username: user
          password: password

- op: replace
  path: /cluster/network/cni
  value:
    name: none

How can I tell Talos to use our internal nameservers to resolve my registry? I tried using the IP of the registry, but got the, more or less, same error message.

Any hint or help is greatly appreciated!

support.zip

[smira]

You already configure your own DNS, and Talos might take a bit of time to pick it up in the resolving process, which is normal.

This seems to be a transient error which is retried.

[fleroviumactinium]

The errors for DNS and falling back to 8.8.8.8 only happen when starting up the control plane node. However, with this configuration, I cannot successfully bootstrap it.

Any tips on what else I can try or look into to startup a control plane in an air-gapped env?

[smira]

I don't see any DNS errors here.

You should check first Troubleshooting guide.