Repository navigation
cni0 ignores the primary NIC MTU and DHCP replies carry no MTU option #24
Description
Activity
Evidence gathered on 2026-09-21 before deciding this one.
Why nothing has broken so far
- Every GCE node in this family sits in a VPC with MTU 1460 (instance metadata
network-interfaces/0/mtu= 1460, primary NICmtu 1460). - It did break once: cocoon #135 (2026-07-16) was exactly this symptom on GCP (large TLS responses time out while small requests work). The fix went into
cocoon-check, which writes the CNI conflist with the host egress MTU and installsiptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu. Every node runbook (cocoonset-node-setup, gke-full-stack-deployment, volcengine-node-setup) runssudo cocoon-check --fixbeforecocoon-net init, so the clamp is present on every cocoon-net node as a side effect of the doctor, not of cocoon-net. - Non-TCP traffic that exceeds 1460 with DF set gets an ICMP fragmentation-needed from the host, which is the router, so guest PMTUD converges for it.
- The sandboxd-based GCE node does not route guest traffic at all: guest egress is the host-side forward proxy over vsock (
sandbox/docs/egress.md), so TCP terminates on the host and the host MTU governs. The MTU never applies there.
What cocoon-net itself does today
cni0is created without an MTU, the generated conflist carries nomtu, and DHCP replies carry no option 26, so the guest sees 1500 and the node relies on the doctor's clamp.Rollout constraint for the fix
Firecracker clone and direct restore reject a target network whose MTU differs from the snapshot's (
hypervisor/firecracker/clone.govalidateCloneNetworkMTUs,direct.go). Lowering the bridge MTU on a node therefore invalidates the Firecracker fork and hibernate snapshots captured at 1500 until they are re-captured; Cloud Hypervisor clones hot-swap their NICs and are unaffected.The change itself is small (bridge MTU from the primary NIC,
mtuin the conflist, DHCP option 26). Whether to land it with a snapshot re-capture step, or to document the dependency oncocoon-checkand close, is the open decision.- Every GCE node in this family sits in a VPC with MTU 1460 (instance metadata
Fixed in ad7469c on the review branch (PR #28), scoped to cocoon-net: node setup reads the primary NIC MTU, creates or resets cni0 to it and writes it into the conflist so every veth follows; cocoon syncs the TAP to the veth and the hypervisor advertises the TAP MTU to the guest. Docs (306cd6f) note that a node whose MTU changes needs its Firecracker snapshots re-captured.
Problem
setupBridge(node/bridge_linux.go:19) createscni0at the default 1500 without reading the primary NIC's MTU, andbuildReply(dhcp/reply.go:11) sends no DHCP option 26. Nothing in cocoon-net mentions MTU.On a GCE VPC at the default 1460,
ens4is 1460 whilecni0, the VM veths and the TAPs are 1500; cocoon copies the bridge MTU onto the TAP and advertises it to the guest (network/bridge/bridge_linux.go,network/cni/lifecycle_linux.go, Firecracker NICmtu). Guests therefore emit full-size frames that the host must fragment on the way out, and any path that swallows ICMP frag-needed becomes a PMTUD black hole: small flows work, bulk transfers stall.Verified on Linux with
NET_ADMINagainst a dummyens4at 1460:cni0stays 1500 after a veth is enslaved, because the primary NIC is routed and never enslaved.Proposal
Set the bridge MTU from the resolved primary NIC in
setupBridge, and optionally add option 26 tobuildReplyfrom a newConfig.MTU. The bridge route also covers VM-to-VM traffic on the same node; the DHCP option alone does not.Acceptance
With the primary NIC at 1460,
cni0and every enslaved veth report 1460, and a guest booted through DHCP configures 1460.Found in the 2026-09-21 whole-repo review round (branch
review/whole-repo-2026-09-21). Filed as a behaviour question for the owner; no code change is included in the round PR.