Skip to content

Stop flooding other guests' unicast frames to isolated TAPs - #563

Draft
sjmiller609 wants to merge 1 commit into
mainfrom
hypeman/pin-guest-fdb
Draft

sjmiller609 wants to merge 1 commit into
mainfrom
hypeman/pin-guest-fdb

Conversation

@sjmiller609

@sjmiller609 sjmiller609 commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Port isolation stops guest-to-guest forwarding, but when the bridge has no FDB entry for a destination MAC it still floods the frame to every port. A guest that puts its NIC in promiscuous mode can therefore capture other guests' inbound unicast traffic on the same host.

For every TAP on an isolated network, createTAPDevice now:

  1. pins the guest's MAC to its TAP as a static FDB entry (bridge fdb replace <mac> dev <tap> master static), so delivery to the guest never depends on flooding;
  2. turns off unknown-unicast flooding on the TAP;
  3. turns off MAC learning on the TAP.

Notes:

  • The entry uses NUD_NOARP (static), not NUD_PERMANENT. NUD_PERMANENT makes the bridge treat the MAC as its own (local) rather than an address behind the port.
  • Learning must be off as well. With learning on, a guest sending frames with a neighbour's source MAC moves even a static entry to its own port and then receives the neighbour's frames. The unit test below demonstrates this.
  • Broadcast and multicast flooding are unchanged, so ARP and IPv6 ND still work.
  • Both CreateAllocation and RecreateAllocation pass the allocated MAC. Forked restores reconfigure the guest to the allocated MAC before the fork call returns. The kernel removes a port's FDB entries when the TAP is deleted, so no cleanup path is needed.
  • The change applies to TAPs created after deploy. Running VMs keep their current port settings until they are restored or recreated.
  • Non-isolated networks are unchanged.

Testing

Unit test

TestHardenIsolatedPortStopsUnicastLeak (root only, skips otherwise) builds a bridge in a throwaway network namespace with two hardened ports and one port left at kernel defaults. It transmits frames through the bridge device, as the host stack does for guest-bound traffic, and checks:

  • the hardened ports' FDB entries are static;
  • a frame for a pinned MAC reaches that guest and no other;
  • unknown unicast reaches the default port (the leak) but neither hardened port;
  • after one guest spoofs the other's source MAC, frames still go only to the real owner.

Run as root: passes 5/5. Mutation check: turning flooding back on, turning learning back on, or using NUD_PERMANENT each fails the test. go test ./lib/network/ and GOOS=darwin go vet ./lib/network/ pass.

Before/after on a dev hypeman host

Both revisions (before = merge base a096e53, after = this branch) ran on dev-yul-hypeman-0 (AMD EPYC 9334, XFS on md RAID) in a dedicated network namespace with their own data dirs, bridge, and subnet. Only one revision ran at a time, alternating by round. The host's staging Hypeman service was not touched. Image: onkernel/chromium-headless-private:869aaa2, Firecracker, file-backed snapshots. Forks follow the production shape: POST /snapshots/{id}/fork with target_state=Running, apply fork identity, CDP /json/version, confirm identity.

Leak repro. One guest sniffs in promiscuous mode while the host sends 20 unicast frames to a MAC that has no FDB entry:

frames to unknown MAC captured by the sniffing guest guest's own unicast received
before 20 / 20 35
after 0 / 20 37

The same result reproduced in a second run.

Latency. Every operation passed after one harness fix: the first run used a 10s per-attempt identity timeout, and identity calls stalled longer than that under burst load on both revisions.

p50 ms (avg) before after
snapshot fork API, single (n=40) 252 242
snapshot fork API, 25-way burst (n=100) 403 405
cold create → kernel-images ready (n=15) 24236 24265
standby → restore API (n=15) 33 33
restore → kernel-images ready (n=15) 52 51
fork → identity confirmed, single (n=40) 1038 1109
fork → identity confirmed, 25-way burst (n=100) 1170 1275

Hypeman-side time (fork, create, restore) is unchanged. Most of the end-to-end gap is guest-side identity apply, and it comes from each revision having its own seed snapshot. Crossing binaries and templates (40 single forks per cell):

identity apply p50 (avg) ms before template after template
before binary 708 (705) 771 (797)
after binary 734 (728) 831 (823)

The template accounts for ~65–100ms. What's left that's attributable to this change is ~25ms average, which is within the round-to-round spread of this harness (±20–40ms). Tail stalls of ~14s under 25-way bursts showed up on both revisions.

Not done: staging QA through CI.

Port isolation blocks guest-to-guest forwarding, but frames for a MAC the
bridge has no entry for are still flooded to every port, so a guest in
promiscuous mode can capture other guests' inbound traffic. Pin each guest
MAC to its TAP as a static FDB entry and turn off unknown-unicast flooding
and learning on isolated ports.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant