Broken DNS behind VPN on windows

[Nino-K]

As previously reported (#8088, #8055, and #8058), the DNS lookup behind the VPN was broken in version 1.17. The root cause was an upgrade to gvisor-tap-vsock v0.8.1. As part of the fix, we have downgraded it to v0.7.5, which should resolve the issue in version 1.17.1.

Additionally, we should investigate the fix in the upstream gvisor-tap-vsock repository and consider contributing a pull request to address the issue.

[Nino-K]

Submited the following issue in the upstream and they agreed to revert the PR that caused the issue in here: containers/gvisor-tap-vsock#467

[jankap]

Not sure if this is DNS related, but I can't pull any images when connected to the corporate VPN.

Commands done in WSL Ubuntu (not rancher-desktop WSL)

docker pull hello-world
Using default tag: latest
Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp 44.208.254.194:443: i/o timeout

curl on the same machine works

curl https://registry-1.docker.io/v2/
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

curl -v shows that the correct proxy, set via http(s)_proxy are used.

Since the domain is translated into the IP by rancher, this is not a DNS error, correct?

I'm on WSL2, v2.4.10.0, Win 11, rancher 1.17.1., using the new mirrored network mode

EDIT: Interesting, after restarting WSL and Rancher Desktop, after VPN connection, I get now a

docker pull hello-world
Using default tag: latest
Error response from daemon: Get "https://registry-1.docker.io/v2/": EOF

Maybe it's DNS related.

Logging into rancher-desktop distribution:

docker -d rancher-desktop

ping www.google.de
ping: bad address 'www.google.de'

ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=115 time=22.046 ms
64 bytes from 8.8.8.8: seq=1 ttl=115 time=29.356 ms
64 bytes from 8.8.8.8: seq=2 ttl=115 time=38.626 ms

cat /etc/resolv.conf
nameserver 192.168.127.1

ping 192.168.127.1
PING 192.168.127.1 (192.168.127.1): 56 data bytes
<timeout>

So what is 192.168.127.1?

[Qenupve]

So what is 192.168.127.1?

That looks to be the gateway IP hard coded in gvisor-tap-vsock

@jankap have you checked the Rancher Desktop WSL Proxy settings? Someone noted that it was causing them issues in 1.17.1 #8055 (comment)

[Nino-K]

@jankap are you able to try out the build from our main branch which can be found here (you would need to be logged into the github). Thanks.

[jankap]

I will. Please give me a few days, I'm sick at home, just wanted to comment, thanks :)

[JWood48]

I'm having a similar issue, don't know if it's related or something new, i'm behind a corporate vpn and when pulling images the docker host freezes and looses the connection to the net after some time eg.

start pulling an image, many layers are pulled but suddenly it hangs:

If the process is cancelled and rerun i get this error:

$ docker pull mcr.microsoft.com/dotnet/sdk:8.0 Error response from daemon: Get "https://mcr.microsoft.com/v2/": dial tcp: lookup mcr.microsoft.com on 192.168.127.1:53: dial udp 192.168.127.1:53: connect: network is unreachable

This is the same for any repository i pull from:

$ docker pull confluentinc/cp-kafka:7.0.0 Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 192.168.127.1:53: dial udp 192.168.127.1:53: connect: network is unreachable

I need to restart Rancher Desktop to get it working again, and after many tries i might get lucky and pull all layers before it freezes.

(I'm on windows 11 and Rancher Desktop 1.17.1)

[jankap]

@jankap are you able to try out the build from our main branch which can be found here (you would need to be logged into the github). Thanks.

@Nino-K

I tried Version: 1.17.1-369-gc3ccaba62. The error changed.

 docker pull curlimages/curl
Using default tag: latest
Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

When disconnecting VPN, everything works.

@jankap have you checked the Rancher Desktop WSL Proxy settings? Someone noted that it was causing them issues in 1.17.1 #8055 (comment)

Yes, the proxy integration is currently disabled.

Is there anything I can do to support searching the issue? Docker Desktop works BTW.

[Nino-K]

@jankap are you able to try out the build from our main branch which can be found here (you would need to be logged into the github). Thanks.

@Nino-K

I tried Version: 1.17.1-369-gc3ccaba62. The error changed.

 docker pull curlimages/curl
Using default tag: latest
Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

When disconnecting VPN, everything works.

@jankap have you checked the Rancher Desktop WSL Proxy settings? Someone noted that it was causing them issues in 1.17.1 #8055 (comment)

Yes, the proxy integration is currently disabled.

Is there anything I can do to support searching the issue? Docker Desktop works BTW.

@jankap, thanks for testing it out! One thing I suspect might be causing issues is the mirrored network mode in WSL, which we don't officially support. Are you able to disable it, or is it a must-have for your setup? Also, could you try pulling from the Windows CMD terminal on the host to see if you get a different result?

[jankap]

@jankap, thanks for testing it out! One thing I suspect might be causing issues is the mirrored network mode in WSL, which we don't officially support. Are you able to disable it, or is it a must-have for your setup? Also, could you try pulling from the Windows CMD terminal on the host to see if you get a different result?

@Nino-K the mirrored network mode is awesome because it gets rid of all kinds of issues with DNS, proxies, etc in WSL - no manual settings needed anymore, no messing around with resolv.conf etc. I think it's going to become the standard mode in the future.

But yes, I tried to disable the mode, used NAT again, and restarted WSL and Rancher. No change.

From inside WSL:

docker pull hello-world
Using default tag: latest
Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

From host powershell:

docker pull hello-world
Using default tag: latest
Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Is there any way to debug from inside rancher-desktop or rancher-desktop-data? How to find out if it's DNS or proxy related? I have not set any proxies, because I'm not sure how to do that, see #4289

[Nino-K]

@jankap, thanks for testing it out! One thing I suspect might be causing issues is the mirrored network mode in WSL, which we don't officially support. Are you able to disable it, or is it a must-have for your setup? Also, could you try pulling from the Windows CMD terminal on the host to see if you get a different result?

@Nino-K the mirrored network mode is awesome because it gets rid of all kinds of issues with DNS, proxies, etc in WSL - no manual settings needed anymore, no messing around with resolv.conf etc. I think it's going to become the standard mode in the future.

But yes, I tried to disable the mode, used NAT again, and restarted WSL and Rancher. No change.

From inside WSL:

docker pull hello-world
Using default tag: latest
Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

From host powershell:

docker pull hello-world
Using default tag: latest
Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Is there any way to debug from inside rancher-desktop or rancher-desktop-data? How to find out if it's DNS or proxy related? I have not set any proxies, because I'm not sure how to do that, see #4289

Trying to figure out if it's a DNS issue, you can do rdctl shell nslookup registry-1.docker.io to see if it resolves.