NordVPN prevents Networkd Netplan from renewing Cloud configs

[astromasoud]

Hi,

I had run and setup NordVPN on Linux without any issue before, but now I've been trying to use it on a Hetzner VPS running Ubuntu 20.04 or 22.04 and I'm facing an issue that I haven't been able to figure out. I tried contacting the NordVPN support, but their solutions were not very helpful, I think posting it here would be faster to find a workaround.

The environment:

• An Ubuntu 20.04/22.04 x64 VPS
• It uses systemd-networkd.service and netplan automated cloud configs
• Netplan automatically renews the Internet interface's configs (eth0) with their cloud every 36 hours.

NordVPN configs:

• Technology: NORDLYNX (because I need higher speeds)
• Firewall: disabled (no difference when enabled)
• Firewall Mark: 0xe1f1
• Routing: enabled
• Analytics: enabled
• Kill Switch: disabled
• Threat Protection Lite: disabled
• Notify: enabled
• Tray: enabled
• Auto-connect: enabled
• IPv6: disabled (no difference when enabled)
• Meshnet: disabled
• DNS: disabled
• LAN Discovery: enabled (no difference when disabled)
• Virtual Location: enabled
• Post-quantum VPN: disabled
• Allowlisted ports:
  -- 67 (UDP) for DHCP requests (no difference)
  -- SSH (TCP)
• Allowlisted subnets:
  -- 172.0.0.0/8 (The DC's internal subnets on which their DHCP and other servers exist)

So apparently EVERY 36 hours, Networkd uses Netplan's cloudconfigs to inquire whether there's been any change on the cloud side that needs to be reflected on this machine's eth0 Internet-facing interface (well, there's never any change, but anyway).

What happens is that WHEN NordVPN is Connected, somehow Networkd is unable to perform this task.
Which then results in the eth0 interface going down with the following error:
Nov 19 05:52:27 ubuntu-8gb-hel1-1 systemd-networkd[723]: eth0: Could not set route: IPv6 is disabled on nexthop device. Permission denied
Nov 19 05:52:27 ubuntu-8gb-hel1-1 systemd-networkd[723]: eth0: Failed
And then I have to access the VPS through direct Console, and perform:

• nordvpn disconnect
• sudo systemctl restart systemd-networkd.service
  -- Which works flawlessly, pulls the configs, and brings eth0 back up.
• nordvpn connect

Also to replicate the problem manually:

• nordvpn connect
• sudo systemctl restart systemd-networkd.service
  -- And your machine will lose Internet/network access entirely as eth0 will go down and you have to console into the device and do the steps above to fix it!

I've also tried testing it on new VPS instances in case this one was broken, same thing.

Any ideas would be appreciated.
If nothing works, probably the only workaround currently would be to setup a cronjob to disconnect NordVPN, restart Networkd, and reconnect NordVPN maybe every 35 hours before the OS does it at the 36th!

[devzbysiu]

Hi, sorry for a late response. I'll do some experiments and let you know

[devzbysiu]

Ok, I know more now (thx @mariusSincovici 🙇)

So the thing is that we are disabling IPv6 to prevent leaks (also, we don't have servers with IPv6 support currently, so it is not used by app). If anything is trying to interact with IPv6 stack without proper checks, it will probably fail. I think that's why you see this eth0 down error.

When you disconnect from VPN we re-enable the IPv6 so that's why it works.

So the weird thing on your side is that, for some reason, systemd tries to interact with IPv6 when it is disabled in the kernel.

What you can try is to check if you have IPv6-related config in netplan and maybe remove it if it's possible - so that it doesn't interact with IPv6 stack.

You can also check if you have IPv6 enabled specifically for some of your interfaces, see this for example https://www.flatcar.org/docs/latest/setup/customization/network-config-with-networkd/#turn-off-ipv6-on-specific-interfaces

---

There is potentially one thing that you can also check:

• we are disabling IPv6 with kernel setting net.ipv6.conf.all.disable_ipv6
• this setting should also influence the net.ipv6.conf.default.disable_ipv6 so after connecting to the vpn and doing sudo sysctl net.ipv6.conf.default.disable_ipv6 you should see:

net.ipv6.conf.default.disable_ipv6 = 1

This default.disable_ipv6 setting is disabling IPv6 for newly created interfaces also, so maybe, just maybe, this setting is not set on your side along with all.disable_ipv6 and if your net interfaces are recreated, for any reason, after connecting to VPN, they have IPv6 enabled.

That's a long shot IMO - according to kernel docs setting all.disable_ipv6 is the same as setting default.disable_ipv6. On my machine those two go hand in hand, but that's something you can check also

[astromasoud]

Sorry for the very late reply.

Initially I managed to create a workaround by having a cronjob run once a day that would stop NordVPN, restart systemd-networkd, and start NordVPN again. For the record and anyone interested, these are the steps:

• Create a bash file like /opt/renew.sh with the following content:

#!/bin/bash
nordvpn disconnect
systemctl restart systemd-networkd.service
nordvpn connect Netherlands

• Give it execution permission: sudo chmod +x /opt/renew.sh
• Add a line to your root crontab using sudo crontab -e like this, which would run at 14:05 every day:
  5 14 * * * /opt/renew.sh

---

After reading your message and that link I gave it more time and was finally able to figure it out.
Tho, that method for disabling IPv6 per interface doesn't work anymore as if-up-down is replaced with netplan, and it seems that netplan only cares about its own configs, not systemwide settings and such!

So to resolve the issue, you could edit the cloud config file which is usually placed here:
/etc/netplan/50-cloud-init.yaml

• Remove or comment out the sections that include pre-set IPv6 configs,
  and also add dhcp6: false in there. Like this example (DO NOT copy, do it manually):

network:
    version: 2
    ethernets:
        eth0:
                #addresses:
                #- 1234:567:89a:bcdf::1/64
            dhcp4: true
            dhcp6: false
            match:
                macaddress: 01:23:45:67:89:ab
                #nameservers:
                #addresses:
                #- 1234:567:89ab::cdf:1
                #- 1234:567:89ab::cdf:2
            routes:
            -   on-link: true
                to: ::/0
                via: fe80::1
            set-name: eth0

-- As you can see, I've commented out addresses and nameservers sections and disabled dhcp6 there.

• After making these changes you should also execute sudo netplan apply for the changes to take effect.

Now the server stays connected and eth0 does not try to get an IPv6 every time systemd-networkd renews!
You can test the results, while NordVPN is connected, by manually running sudo systemctl restart systemd-networkd, and if after that the server still has access to the Internet then it's working!

[astromasoud]

Update:

The netplan config changes above, although it seems to work when testing with sudo systemctl restart systemd-networkd, but it only lasted 24 hours and the server lost Internet connectivity again!
When I connected to the server via direct console, I saw the same outcome as before and the eth0 interface didn't have an IP address. Surprisingly doing sudo systemctl restart systemd-networkd fixed it, but there seem to be some other underlying communications with the cloud provider that somehow overrides this somewhere and messes things up! Tho, it used to happen every 36 hours, this happened 24 hours after the last restart of systemd-networkd yesterday...

So I tried that link's solution:

• Created a file named /etc/systemd/network/eth0 with the following content:

[Match]
Name=eth0

[Network]
LinkLocalAddressing=no
IPv6AcceptRA=no

• Ran sudo systemctl restart systemd-networkd to apply.

And it sorta worked and didn't! After multiple trials and errors, found that it only works when I uncomment the lines in /etc/netplan/50-cloud-init.yaml that I had previously commented out in the post above, but have to keep the dhcp6: false option there.

Still, tho, something funny happens: It works once in every 2 times I run sudo systemctl restart systemd-networkd!

I checked the logs for systemd-networkd, every time the error part (line 2-3) about IPv6 is the same, but every other time DHCPv4 is successful in getting the IP address (line 1) and the server goes online!

Jan 08 16:28:22 ubuntu systemd-networkd[81753]: eth0: DHCPv4 address 123.123.123.123/32 via 172.31.1.1
Jan 08 16:28:22 ubuntu systemd-networkd[81753]: eth0: Could not set route: IPv6 is disabled on nexthop device. Permission denied
Jan 08 16:28:22 ubuntu systemd-networkd[81753]: eth0: Failed

I guess what happens is that the changes work now, but next time networkd tries to renew network configs automatically, it fails, but then when you manually restart that service it works again, but if you do it twice it doesn't work lol!

So far the only solution that works all the time is still the daily cronjob that stops NordVPN, restarts systemd-networkd, and starts NordVPN again! It only takes a few seconds. :D

---

Update 2

Well, the server lost Internet access again after 24 hours.

So the only solution working right now is the cronjob! :)