Newt in Kubernetes #207

enorasec · 2025-12-16T18:30:16Z

enorasec
Dec 16, 2025

Hello All -

I'm having issues installing the latest version of Newt (1.7.0) with the latest Helm Chart (1.1.0). The pod is able to install and pick up the configuration file but the pods keep exiting and causing a CrashBackoffLoop.

Debug Logs provided. It's running an Oracle Kubernetes Engine

INFO: 2025/12/16 16:37:30 Newt version 1.7.0
DEBUG: 2025/12/16 16:37:30 Config already provided, skipping loading from file
INFO: 2025/12/16 16:37:30 Config file does not exist at /root/.config/newt-client/config.json, will create it
DEBUG: 2025/12/16 16:37:30 Endpoint: https://<<PANGOLIN_URL>>
DEBUG: 2025/12/16 16:37:30 Log Level: DEBUG
DEBUG: 2025/12/16 16:37:30 Docker Network Validation Enabled: false
DEBUG: 2025/12/16 16:37:30 Health Check Certificate Enforcement: false
DEBUG: 2025/12/16 16:37:30 Dns: 9.9.9.9
DEBUG: 2025/12/16 16:37:30 MTU: 1280
INFO: 2025/12/16 16:37:30 Setting up clients with netstack2...
INFO: 2025/12/16 16:37:30 Created shared UDP socket on port 61406 (refcount: 2)
DEBUG: 2025/12/16 16:37:30 Creating new health check monitor with certificate enforcement: false
DEBUG: 2025/12/16 16:37:30 Token response body: {"data":{"token":"<<TOKEN>>","serverVersion":"1.13.0"},"success":true,"error":false,"message":"Token created successfully","status":200}
INFO: 2025/12/16 16:37:30 Server version: 1.13.0
DEBUG: 2025/12/16 16:37:30 Received token: <<TOKEN>>
INFO: 2025/12/16 16:37:31 Saving config to: /root/.config/newt-client/config.json
DEBUG: 2025/12/16 16:37:31 Public key: 6bJ5yOtLHUCU4le5lJZMxpPHMpvzAo/3f0GwjgT7ODo=
INFO: 2025/12/16 16:37:31 Websocket connected
DEBUG: 2025/12/16 16:37:31 Requesting exit nodes from server
INFO: 2025/12/16 16:37:31 Requesting WireGuard configuration from remote server
DEBUG: 2025/12/16 16:37:31 Sending message: newt/wg/register, data: map[backwardsCompatible:true newtVersion:1.7.0 publicKey:6bJ5yOtLHUCU4le5lJZMxpPHMpvzAo/3f0GwjgT7ODo=]
DEBUG: 2025/12/16 16:37:31 Sending message: newt/ping/request, data: map[noCloud:false]
DEBUG: 2025/12/16 16:37:31 Sending message: newt/wg/get-config, data: map[port:61406 publicKey:<<PUBLIC_KEY>>]
DEBUG: 2025/12/16 16:37:31 Received ping message
DEBUG: 2025/12/16 16:37:31 Only one exit node available, using it directly: <<PANGOLIN_URL>>
DEBUG: 2025/12/16 16:37:31 Sending message: newt/wg/register, data: map[newtVersion:1.7.0 pingResults:[{ExitNodeID:1 LatencyMs:0 Weight:1 Error: Name:Exit Node A2HcWJ2B Endpoint:<<PANGOLIN_URL>> WasPreviouslyConnected:true}] publicKey:6bJ5yOtLHUCU4le5lJZMxpPHMpvzAo/3f0GwjgT7ODo=]
DEBUG: 2025/12/16 16:37:31 Received registration message
DEBUG: 2025/12/16 16:37:31 Received registration message data: map[endpoint:<<PANGOLIN_URL>>:51820 healthCheckTargets:[] publicKey:<<PUBLIC_KEY>> serverIP:100.89.128.1 targets:map[tcp:[] udp:[]] tunnelIP:100.89.128.8]
DEBUG: 2025/12/16 16:37:31 Received: {Type:newt/wg/connect Data:map[endpoint:<<PANGOLIN_URL>>:51820 healthCheckTargets:[] publicKey:<<PUBLIC_KEY>> serverIP:100.89.128.1 targets:map[tcp:[] udp:[]] tunnelIP:100.89.128.8]}
INFO: 2025/12/16 16:37:31 Connecting to endpoint: <<PANGOLIN_URL>>
DEBUG: gerbil-wireguard: 2025/12/16 16:37:31 Routine: encryption worker 1 - started
DEBUG: gerbil-wireguard: 2025/12/16 16:37:31 Routine: decryption worker 1 - started
DEBUG: gerbil-wireguard: 2025/12/16 16:37:31 Routine: handshake worker 1 - started
DEBUG: gerbil-wireguard: 2025/12/16 16:37:31 Routine: TUN reader - started
DEBUG: gerbil-wireguard: 2025/12/16 16:37:31 Routine: event worker - started
DEBUG: gerbil-wireguard: 2025/12/16 16:37:31 Interface up requested
DEBUG: gerbil-wireguard: 2025/12/16 16:37:31 UDP bind has been updated
DEBUG: gerbil-wireguard: 2025/12/16 16:37:31 Interface state was Down, requested Up, now Up
DEBUG: gerbil-wireguard: 2025/12/16 16:37:31 Routine: receive incoming v6 - started
DEBUG: gerbil-wireguard: 2025/12/16 16:37:31 Routine: receive incoming v4 - started
INFO: 2025/12/16 16:37:31 Starting UDP hole punch to 1 exit nodes with shared bind
INFO: 2025/12/16 16:37:31 Starting hole punch to <<PANGOLIN_IP>> with public key: <<PUBLIC_KEY>>
DEBUG: gerbil-wireguard: 2025/12/16 16:37:31 UAPI: Updating private key
DEBUG: gerbil-wireguard: 2025/12/16 16:37:31 peer(A2Hc…YhTs) - UAPI: Created
DEBUG: gerbil-wireguard: 2025/12/16 16:37:31 peer(A2Hc…YhTs) - UAPI: Adding allowedip
DEBUG: gerbil-wireguard: 2025/12/16 16:37:31 peer(A2Hc…YhTs) - UAPI: Updating endpoint
DEBUG: gerbil-wireguard: 2025/12/16 16:37:31 peer(A2Hc…YhTs) - UAPI: Updating persistent keepalive interval
DEBUG: gerbil-wireguard: 2025/12/16 16:37:31 peer(A2Hc…YhTs) - Starting
DEBUG: gerbil-wireguard: 2025/12/16 16:37:31 peer(A2Hc…YhTs) - Sending keepalive packet
DEBUG: gerbil-wireguard: 2025/12/16 16:37:31 peer(A2Hc…YhTs) - Sending handshake initiation
DEBUG: 2025/12/16 16:37:31 WireGuard device created. Lets ping the server now...
DEBUG: 2025/12/16 16:37:31 Testing initial connection with reliable ping...
INFO: 2025/12/16 16:37:31 Resolved exit node: <<PANGOLIN_IP>> -> <<PANGOLIN_IP>>:21820
DEBUG: 2025/12/16 16:37:31 Sent UDP hole punch to <<PANGOLIN_IP>>:21820: {"ephemeralPublicKey":"<<EPHEMERAL_PUB_KEY>>","nonce":"Z9EFSTk26TQxx+Qv","ciphertext":"<<CIPHER_TEXT>>"}
DEBUG: gerbil-wireguard: 2025/12/16 16:37:31 peer(A2Hc…YhTs) - Routine: sequential sender - started
DEBUG: gerbil-wireguard: 2025/12/16 16:37:31 peer(A2Hc…YhTs) - Routine: sequential receiver - started
DEBUG: 2025/12/16 16:37:32 Sent UDP hole punch to <<PANGOLIN_IP>>:21820: {"ephemeralPublicKey":"<<EPHEMERAL_PUB_KEY>>","nonce":"9g58gUBaT+Lmh3mT","ciphertext":"<<CIPHER_TEXT>>"}
DEBUG: 2025/12/16 16:37:32 Increased hole punch interval to 2s
DEBUG: 2025/12/16 16:37:33 Sending message: newt/wg/get-config, data: map[port:61406 publicKey:<<PUBLIC_KEY>>]
DEBUG: 2025/12/16 16:37:34 Sent UDP hole punch to <<PANGOLIN_IP>>:21820: {"ephemeralPublicKey":"<<EPHEMERAL_PUB_KEY>>","nonce":"mWEi/geDCgxWq4uN","ciphertext":"<<CIPHER_TEXT>>"}
DEBUG: 2025/12/16 16:37:34 Increased hole punch interval to 4s
DEBUG: 2025/12/16 16:37:35 Sending message: newt/wg/get-config, data: map[port:61406 publicKey:<<PUBLIC_KEY>>]
DEBUG: gerbil-wireguard: 2025/12/16 16:37:36 peer(A2Hc…YhTs) - Sending handshake initiation
DEBUG: 2025/12/16 16:37:36 Ping attempt 1/5 failed: failed to read ICMP packet: i/o timeout
DEBUG: 2025/12/16 16:37:37 Sending message: newt/wg/get-config, data: map[port:61406 publicKey:<<PUBLIC_KEY>>]
DEBUG: 2025/12/16 16:37:38 Sent UDP hole punch to <<PANGOLIN_IP>>:21820: {"ephemeralPublicKey":"<<EPHEMERAL_PUB_KEY>>","nonce":"VmNj+SAm0GvzOu+6","ciphertext":"<<CIPHER_TEXT>>"}
DEBUG: 2025/12/16 16:37:38 Increased hole punch interval to 8s
DEBUG: 2025/12/16 16:37:39 Sending message: newt/wg/get-config, data: map[port:61406 publicKey:<<PUBLIC_KEY>>]
DEBUG: 2025/12/16 16:37:41 Sending message: newt/wg/get-config, data: map[port:61406 publicKey:<<PUBLIC_KEY>>]
DEBUG: gerbil-wireguard: 2025/12/16 16:37:41 peer(A2Hc…YhTs) - Sending handshake initiation
DEBUG: 2025/12/16 16:37:41 Ping attempt 2/5 failed: failed to read ICMP packet: i/o timeout
DEBUG: 2025/12/16 16:37:43 Sending message: newt/wg/get-config, data: map[port:61406 publicKey:<<PUBLIC_KEY>>]
DEBUG: 2025/12/16 16:37:45 Sending message: newt/wg/get-config, data: map[port:61406 publicKey:<<PUBLIC_KEY>>]
DEBUG: gerbil-wireguard: 2025/12/16 16:37:46 peer(A2Hc…YhTs) - Sending handshake initiation
DEBUG: 2025/12/16 16:37:46 Sent UDP hole punch to <<PANGOLIN_IP>>:21820: {"ephemeralPublicKey":"<<EPHEMERAL_PUB_KEY>>","nonce":"a5Nr6xbkGI96pvNJ","ciphertext":"<<CIPHER_TEXT>>"}
DEBUG: 2025/12/16 16:37:46 Increased hole punch interval to 16s
DEBUG: 2025/12/16 16:37:47 Sending message: newt/wg/get-config, data: map[port:61406 publicKey:<<PUBLIC_KEY>>]
DEBUG: 2025/12/16 16:37:48 Ping attempt 3/5 failed: failed to read ICMP packet: i/o timeout
DEBUG: 2025/12/16 16:37:49 Sending message: newt/wg/get-config, data: map[port:61406 publicKey:<<PUBLIC_KEY>>]
INFO: 2025/12/16 16:37:51 SendMessageInterval timed out after 10 attempts for message type: newt/wg/get-config
DEBUG: gerbil-wireguard: 2025/12/16 16:37:51 peer(A2Hc…YhTs) - Sending handshake initiation
DEBUG: 2025/12/16 16:37:54 Ping attempt 4/5 failed: failed to read ICMP packet: i/o timeout
DEBUG: gerbil-wireguard: 2025/12/16 16:37:56 peer(A2Hc…YhTs) - Sending handshake initiation
DEBUG: gerbil-wireguard: 2025/12/16 16:38:01 peer(A2Hc…YhTs) - Sending handshake initiation
DEBUG: 2025/12/16 16:38:02 Ping attempt 5/5 failed: failed to read ICMP packet: i/o timeout
WARN: 2025/12/16 16:38:02 Initial reliable ping failed, but continuing: all 5 ping attempts failed, last error: failed to read ICMP packet: i/o timeout
ERROR: 2025/12/16 16:38:02 Failed to remove health file: remove /tmp/healthy: no such file or directory
DEBUG: 2025/12/16 16:38:02 Ping attempt 1
DEBUG: 2025/12/16 16:38:02 Sent UDP hole punch to <<PANGOLIN_IP>>:21820: {"ephemeralPublicKey":"<<EPHEMERAL_PUB_KEY>>","nonce":"wT+kBZcW4DzrErDa","ciphertext":"<<CIPHER_TEXT>>"}
DEBUG: 2025/12/16 16:38:02 Increased hole punch interval to 32s
DEBUG: gerbil-wireguard: 2025/12/16 16:38:06 peer(A2Hc…YhTs) - Sending handshake initiation
WARN: 2025/12/16 16:38:07 Ping attempt 1 failed: failed to read ICMP packet: i/o timeout
DEBUG: 2025/12/16 16:38:07 Starting ping check
DEBUG: 2025/12/16 16:38:07 Ping attempt 2
INFO: 2025/12/16 16:38:07 Started direct UDP relay on 100.89.128.8:61406 (bidirectional via SharedBind)
DEBUG: 2025/12/16 16:38:07 Adding 0 health check targets in bulk
DEBUG: 2025/12/16 16:38:07 Successfully added all 0 health check targets
DEBUG: 2025/12/16 16:38:07 Successfully added 0 health check targets
INFO: 2025/12/16 16:38:07 Direct UDP relay started (bidirectional through SharedBind)
DEBUG: gerbil-wireguard: 2025/12/16 16:38:11 peer(A2Hc…YhTs) - Sending handshake initiation
WARN: 2025/12/16 16:38:12 Ping attempt 2 failed: failed to read ICMP packet: i/o timeout
DEBUG: 2025/12/16 16:38:14 Ping attempt 3
INFO: 2025/12/16 16:38:14 Closing clients...
INFO: 2025/12/16 16:38:14 Stopping direct UDP relay
INFO: 2025/12/16 16:38:14 Hole punch manager stopped
DEBUG: 2025/12/16 16:38:14 Releasing shared bind (refcount before release: 2)
INFO: 2025/12/16 16:38:14 Released shared UDP bind
INFO: 2025/12/16 16:38:14 Stopping health check monitor with 0 targets
INFO: 2025/12/16 16:38:14 Health check monitor stopped
DEBUG: gerbil-wireguard: 2025/12/16 16:38:14 Device closing
DEBUG: gerbil-wireguard: 2025/12/16 16:38:14 Routine: TUN reader - stopped
DEBUG: 2025/12/16 16:38:14 Hole punch stopped by signal
INFO: 2025/12/16 16:38:14 UDP hole punch goroutine ended for all exit nodes
WARN: 2025/12/16 16:38:14 Ping attempt 3 failed: failed to read ICMP packet: ping read: endpoint is closed for receive
DEBUG: 2025/12/16 16:38:14 Ping attempt 1/2 failed: failed to read ICMP packet: ping read: endpoint is closed for receive
DEBUG: gerbil-wireguard: 2025/12/16 16:38:14 Routine: event worker - stopped
DEBUG: gerbil-wireguard: 2025/12/16 16:38:14 Routine: receive incoming v4 - stopped
DEBUG: gerbil-wireguard: 2025/12/16 16:38:14 Routine: receive incoming v6 - stopped
DEBUG: gerbil-wireguard: 2025/12/16 16:38:14 peer(A2Hc…YhTs) - Stopping
DEBUG: gerbil-wireguard: 2025/12/16 16:38:14 peer(A2Hc…YhTs) - Routine: sequential sender - stopped
DEBUG: gerbil-wireguard: 2025/12/16 16:38:14 peer(A2Hc…YhTs) - Routine: sequential receiver - stopped
DEBUG: gerbil-wireguard: 2025/12/16 16:38:14 Device closed
DEBUG: gerbil-wireguard: 2025/12/16 16:38:14 Routine: decryption worker 1 - stopped
DEBUG: gerbil-wireguard: 2025/12/16 16:38:14 Routine: handshake worker 1 - stopped
DEBUG: gerbil-wireguard: 2025/12/16 16:38:14 Routine: encryption worker 1 - stopped
DEBUG: 2025/12/16 16:38:14 Ping attempt 2/2 failed: failed to create ICMP socket: ping connect: no route to host
DEBUG: 2025/12/16 16:38:14 Periodic ping failed (1 consecutive failures): all 2 ping attempts failed, last error: failed to create ICMP socket: ping connect: no route to host
DEBUG: 2025/12/16 16:38:14 Ping attempt 1/2 failed: failed to create ICMP socket: ping connect: no route to host
DEBUG: 2025/12/16 16:38:14 Ping attempt 2/2 failed: failed to create ICMP socket: ping connect: no route to host
WARN: 2025/12/16 16:38:14 Periodic ping failed (2 consecutive failures): all 2 ping attempts failed, last error: failed to create ICMP socket: ping connect: no route to host
DEBUG: 2025/12/16 16:38:14 WebSocket connection closed during shutdown
INFO: 2025/12/16 16:38:14 Exiting...

Answered by oschwartz10612

Dec 17, 2025

Could you run a tcpdump on the vps and start the newt and see if you are getting inbound traffic on 51820 from it? sudo tcpdump -i any -n udp port 51820

View full answer

oschwartz10612 · 2025-12-17T15:06:51Z

oschwartz10612
Dec 17, 2025
Maintainer

Ping attempt 3/5 failed: failed to read ICMP packet: i/o timeout It looks like your newt is not able to ping the gerbil server. Could you verify you have UDP port 51820 open on your Pangolin and firewall rules are letting that port through?

0 replies

enorasec · 2025-12-17T15:46:45Z

enorasec
Dec 17, 2025
Author

Here is the output of iptables -L on the server. There is no other FW on my VPS. I also have another Newt client running with the same version in an LXC container on my proxmox host so I'm assuming connectivity into Pangolin and Gerbil are correctly setup.

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
f2b-sshd   tcp  --  anywhere             anywhere             multiport dports ssh
ACCEPT     udp  --  anywhere             anywhere             udp dpt:21820
ACCEPT     udp  --  anywhere             anywhere             udp dpt:51820
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DROP       all  --  anywhere             anywhere

Chain FORWARD (policy DROP)
target     prot opt source               destination
DOCKER-USER  all  --  anywhere             anywhere
DOCKER-FORWARD  all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (2 references)
target     prot opt source               destination
ACCEPT     udp  --  anywhere             172.18.0.3           udp dpt:51820
ACCEPT     udp  --  anywhere             172.18.0.3           udp dpt:21820
ACCEPT     tcp  --  anywhere             172.18.0.3           tcp dpt:https
ACCEPT     tcp  --  anywhere             172.18.0.3           tcp dpt:http
DROP       all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere

Chain DOCKER-BRIDGE (1 references)
target     prot opt source               destination
DOCKER     all  --  anywhere             anywhere
DOCKER     all  --  anywhere             anywhere

Chain DOCKER-CT (1 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED

Chain DOCKER-FORWARD (1 references)
target     prot opt source               destination
DOCKER-CT  all  --  anywhere             anywhere
DOCKER-INTERNAL  all  --  anywhere             anywhere
DOCKER-BRIDGE  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

Chain DOCKER-INTERNAL (1 references)
target     prot opt source               destination

Chain DOCKER-USER (1 references)
target     prot opt source               destination

Chain f2b-sshd (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Here is a sample of the gerbil logs

INFO: 2025/12/17 15:43:08 Peer Wd1x6OkjFsiVrySvwhca2ciZZtBcsyFGxb62o+lAwm4= added successfully
INFO: 2025/12/17 15:43:40 Cleared 0 sessions for WG IP: <<WG IP>
INFO: 2025/12/17 15:43:51 Clearing connections for removed peer with WG IP: 100.89.128.8
INFO: 2025/12/17 15:43:51 Cleared 0 connections for WG IP: 100.89.128.8
INFO: 2025/12/17 15:43:51 Peer Wd1x6OkjFsiVrySvwhca2ciZZtBcsyFGxb62o+lAwm4= removed successfully
INFO: 2025/12/17 15:43:51 Clearing connections for added peer with WG IP: 100.89.128.8
INFO: 2025/12/17 15:43:51 Cleared 0 connections for WG IP: 100.89.128.8
INFO: 2025/12/17 15:43:51 Peer FBk4s6YXgtfsI/wc1kxWfZJDgAFFgvEaJRGlaYkJ4D0= added successfully

0 replies

oschwartz10612 · 2025-12-17T15:58:58Z

oschwartz10612
Dec 17, 2025
Maintainer

Could you run a tcpdump on the vps and start the newt and see if you are getting inbound traffic on 51820 from it? sudo tcpdump -i any -n udp port 51820

0 replies

enorasec · 2025-12-17T16:16:36Z

enorasec
Dec 17, 2025
Author

User error on this one. Allowed UDP egress on the Oracle Cloud Side and it started connecting. Verified with your tcpdump command. Apologies for the discussion but the back and forth was appreciated after banging my head against this for a few days.

1 reply

oschwartz10612 Dec 17, 2025
Maintainer

NP glad it works! :}

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Newt in Kubernetes #207

Uh oh!

{{title}}

Uh oh!

Replies: 4 comments 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Newt in Kubernetes #207

Uh oh!

enorasec Dec 16, 2025

Replies: 4 comments · 1 reply

Uh oh!

oschwartz10612 Dec 17, 2025 Maintainer

Uh oh!

enorasec Dec 17, 2025 Author

Uh oh!

oschwartz10612 Dec 17, 2025 Maintainer

Uh oh!

enorasec Dec 17, 2025 Author

Uh oh!

oschwartz10612 Dec 17, 2025 Maintainer

enorasec
Dec 16, 2025

Replies: 4 comments 1 reply

oschwartz10612
Dec 17, 2025
Maintainer

enorasec
Dec 17, 2025
Author

oschwartz10612
Dec 17, 2025
Maintainer

enorasec
Dec 17, 2025
Author

oschwartz10612 Dec 17, 2025
Maintainer