Unable to connect to overlay network after hibernation or changing the docking status on macos

[christian-schlichtherle]

Describe the problem

After waking up my MacBook Pro from hibernation, Netbird fails to connect to the overlay network again:

$ sudo netbird status
Error: status failed: create wg interface: resource busy
$ sudo ifconfig utun100
utun100: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1280
	inet 100.90.208.249 --> 100.90.208.249 netmask 0xff000000
	inet6 fe80::f22f:4bff:fe13:efad%utun100 prefixlen 64 scopeid 0x20 
	inet6 fe80::%utun100 prefixlen 64 scopeid 0x20 
	nd6 options=201<PERFORMNUD,DAD>
$ sudo netbird up
Connected
$ sudo netbird status
Error: status failed: create wg interface: resource busy
$ sudo ifconfig utun100
utun100: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1280
	inet 100.90.208.249 --> 100.90.208.249 netmask 0xff000000
	inet6 fe80::f22f:4bff:fe13:efad%utun100 prefixlen 64 scopeid 0x20 
	inet6 fe80::%utun100 prefixlen 64 scopeid 0x20 
	nd6 options=201<PERFORMNUD,DAD>

I have Netbird UI 0.28.2 installed. My machine has multiple network interfaces: When undocked, it's only connected to my WiFi. When being docked, it's also connected via a Thunderbolt Ethernet adapter with 10Gbps, so when docking/undocking my notebook the OS is actually roaming. This may or may not be related to the problem.

I can do netbird down, netbird up, but it doesn't reconnect again. Wireguard stays unconnected:

$ sudo wg show
interface: utun100
  public key: (not shown)
  private key: (hidden)
  listening port: 51820

The only way to reconnect is to reboot the OS, which is very annoying. Is there another workaround at least?

Expected behavior

Netbird should automatically reconnect to the overlay network after waking up the machine from hibernation or undocking/docking it.

Are you using NetBird Cloud?

Yes.

NetBird version

0.28.2

NetBird status -d output:

$ sudo netbird status -d
Error: status failed: create wg interface: resource busy

[mlsmaycon]

@christian-schlichtherle can you upgrade to 0.28.3 and enable the network monitor with:

netbird down
netbird up -N

[hurricanehrndz]

This is related to #2130

[christian-schlichtherle]

After upgrading to 0.28.3, a first test seems to be successful: After wakeup from hibernation for some minutes, the Netbird client reconnects to the other peers one by one:

$ netbird status
OS: darwin/arm64
Daemon version: 0.28.3
CLI version: 0.28.3
Management: Connected
Signal: Connected
Relays: 2/2 Available
Nameservers: 0/0 Available
FQDN: (not shown)
NetBird IP: 100.90.208.249/16
Interface type: Userspace
Quantum resistance: false
Routes: -
Peers count: 10/10 Connected

I still have to test the docking/undocking scenario, so please don't close this ticket yet.

[hurricanehrndz]

Yeah it is because of multiple interfaces, I am going to get a PR ready, hopefully Netbird team can fix it and make it pretty for their standards

[hurricanehrndz]

@christian-schlichtherle feel free to test out the branch if you know how

[mlsmaycon]

@hurricanehrndz thanks for the PR, we will have a look and give you feedback ASAP.

@christian-schlichtherle If you want to test the PR change, you can download the files from here: https://github.com/netbirdio/netbird/actions/runs/9667635878/artifacts/1637361966 And replace the netbird bin in your system, probably /Applications/NetBird.app/Contents/MacOS/netbird with the one from the package. See the example steps below:

sudo netbird service stop
sudo cp extracted/bin/path/netbird /Applications/NetBird.app/Contents/MacOS/netbird
sudo chmod +x /Applications/NetBird.app/Contents/MacOS/netbird
sudo netbird service start

[christian-schlichtherle]

Following up my testing, after docking my notebook with 0.28.3 installed I run into the same problem again:

$ netbird status
Error: status failed: create wg interface: resource busy

Next, I will try the supplied patch.

PS: Same result when waking up from hibernation while being docked => The root cause is related to multiple NICs.

[christian-schlichtherle]

I've installed the new client:

$ netbird status
OS: darwin/arm64
Daemon version: 0.28.3-SNAPSHOT-2c869542
CLI version: 0.28.3-SNAPSHOT-2c869542
Management: Connected
Signal: Connected
Relays: 2/2 Available
Nameservers: 0/0 Available
FQDN: (not shown)
NetBird IP: 100.90.208.249/16
Interface type: Userspace
Quantum resistance: false
Routes: -
Peers count: 10/10 Connected

BTW: Following semantic versioning, the version tag should be 0.28.4-SNAPSHOT-2c869542 because 0.28.3-SNAPSHOT-2c869542 implies that it's a pre-release to 0.28.3, but it's not. Maybe, just maybe that's why the Netbird UI is now suggesting to "Download the latest version"?

[christian-schlichtherle]

I've completed the test series now: With the new snapshot version, I can dock/undock/hibernate my notebook in any fashion and it reconnects seamlessly - great!

I noticed something interesting however: When doing netbird status instantly after waking up from hibernation, it shows that there are still some connections available, e.g. 6/10. Then after the three seconds elapsed, it drops down to 0/10 and then ramps up to 10/10 again. This begs the question if the connection reset is really required after all?

As I understand this I can suppress it:

netbird down
netbird up --network-monitor=false

I will give that a try.

[hurricanehrndz]

Network monitor is beneficial when you go from dock to wifi, because the primary route would change. You can test the negative behaviour by sshing to a device on the wireguard interface.