Skip to content

relay: add static-IP egress relay service (TLS 1.3 + yamux) - #21

Merged
shreyas-blacksmith merged 7 commits into
mainfrom
devin/1790372603-relay-service
Sep 29, 2026
Merged

shreyas-blacksmith merged 7 commits into
mainfrom
devin/1790372603-relay-service

Conversation

@shreyas-blacksmith

@shreyas-blacksmith shreyas-blacksmith commented Sep 25, 2026 •

Copy link
Copy Markdown

Summary

Server side of the macOS static-IP userspace egress dataplane (FastActions/fa agent/docs/macos-egress-proxy-design.md §5, PR 1 of the §8 split). Replaces route-to → gif/ESP → IPIP for Mac static IPs with:

VM -> pf rdr -> egress-proxy (mac, root) --TLS1.3/yamux, one session per VM--> vprox relay --bind(static_ip)--> dst

Port of the benchmarks/vzbench/relay prototype; the prototype's bench-only mTLS and KindMux byte are replaced by a versioned session hello with bearer auth (phase 1, parity with /connect-ipip). Client-cert mTLS is the phase-2 follow-up.

Protocol v1 (lib/relayproto) — new, not in the prototype

All big-endian. A byte-identical copy with the same hex fixtures ships in FastActions/fa (agent/egressproxy/relayproto); the fixtures are the cross-module compatibility contract since vprox can't import fa.

client hello   "BRLY" | u8 version=1 | u16 len | JSON {"auth","vm_id","static_ip"}   (len <= 4096)
server reply   u8 status | u16 len | JSON {"egress_ip"} | {"error"}
   0 ok · 1 unauthorized · 2 bad_static_ip · 3 session_limit · 4 protocol_error · 5 unavailable
--- yamux; per stream ---
stream header  u8 proto (6 TCP / 17 UDP) | 4B IPv4 dst | u16 port
stream reply   u8 status
   0 ok · 1 dial_failed · 2 policy_denied · 3 limit · 4 udp_limit · 5 bad_header · 6 unavailable
TCP            raw; yamux FIN = half-close
UDP            repeated u16 len | datagram  (0..65535; zero-length datagrams are forwarded)

lib/relay — §5 hardening as requirements

  • Source allow-list (--allow-source, required): checked on the raw accept, before the TLS handshake; vprox_relay_source_rejected_total.
  • TLS 1.3 only; embedded vprox cert by default (--tls-cert/--tls-key override). vprox relay --print-spki-pin prints the base64 SHA-256 SPKI the host proxy pins instead of InsecureSkipVerify.
  • Auth: subtle.ConstantTimeCompare against VPROX_PASSWORD. Hello must arrive within 10s of accept.
  • Static IP ownership: static_ip must be an address of the relay host (interface scan, 10s TTL) or the hello is refused bad_static_ip; the reply's egress_ip is what outbound sockets bind to (Dialer.LocalAddr / DialUDP(laddr)).
  • Destination policy (Policy.Check, runs before every dial — the FORWARD chain doesn't apply to a local process): 0/8, RFC1918, 100.64/10, 127/8, 169.254/16, 224/4, 240/4, IPv6, port 0, every address currently on the host's interfaces (so tunnel/vprox0/new static IPs are covered without restart), plus --deny-cidr (pass the WG block + internal CIDR) and --deny-port. Own-address discovery failing never disables the static list.
  • Limits with distinct refusal statuses: sessions 4096 total / 64 per source (session_limit), streams 4096 per session (limit), UDP 1024 per session (udp_limit), yamux window 256 KiB, accept backlog 256, dial timeout 5s, TCP idle 6h, UDP idle 60s. Refused streams don't tear down the session.
  • Drain on SIGTERM: listener closes → every session gets yamux GoAway (host proxy sees ErrRemoteGoAway and reconnects to the restarted relay) → in-flight streams get --drain-timeout (15s) to finish → cut. vprox_relay_draining gauge.
  • Metrics (--metrics-addr, default 127.0.0.1:9444): sessions open/total, hello refusals by status, streams open/total by proto, refusals by status, bytes by dir+static_ip, dial latency histogram by proto, Go/process collectors.
  • Payloads and the credential are never logged.

Wiring not in this repo

The systemd unit, UFW rule (relay port restricted to ipip_client_cidrs, same as /connect-ipip), and Alloy scrape target land in FastActions/fa infra/ansible/setup_vprox_server.yaml (PR E of the split), since that playbook is what clones this repo.

Tests

go test ./... (relay + relayproto): protocol fixtures and round-trips; policy table; hello refusals (bad password / foreign IP / garbage); TCP echo >1 MiB with half-close; dial-failed vs policy-denied on a live session; UDP echo incl. zero-length datagram; UDP idle close; stream and UDP limits; per-source session limit; source allow-list drop; drain = GoAway + in-flight stream completes + exit once drained. CI on this repo is build-only.

§6 acceptance (design doc)

Pending. Items 1–14 need the full stack (this relay on staging vprox .247, the fa egress-proxy, supervisor and vm/vz proxy mode on .16). Will be run once fa PR D exists and the numbers posted here. Prototype reference (same host pair): P1 11.9s (~9.1 Gbit/s) vs OLD 62.7s (~1.7 Gbit/s), P2 10.3s vs 10.6s, P4 8.3s vs 8.2s.

Link to Devin session: https://app.devin.ai/sessions/3732a5e335164324a71ec27bf19595aa
Open in Devin Desktop: https://app.devin.ai/desktop/session/3732a5e335164324a71ec27bf19595aa?variant=devin
Requested by: @shreyas-blacksmith


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Adds `vprox relay`, the server side of the macOS userspace egress
dataplane. A macOS host's egress proxy opens one TLS 1.3 session per VM,
authenticates with VPROX_PASSWORD in a session hello, and multiplexes the
VM's TCP and UDP flows over yamux; the relay dials each destination bound
to the granted static IP.

- lib/relayproto: protocol v1 (hello, stream header with proto, refusal
  status codes, UDP framing) with hex fixtures shared with FastActions/fa.
- lib/relay: source allow-list, bearer auth, static-IP ownership check,
  SSRF destination policy (RFC1918, loopback, link-local, CGNAT, multicast,
  240/4, own addresses, extra CIDRs/ports), per-session and per-source
  limits, idle timeouts, Prometheus metrics, SIGTERM drain via GoAway.
- cmd/relay: flags, /metrics endpoint, --print-spki-pin for the client pin.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration

Copy link
Copy Markdown

End-to-end numbers with this relay deployed on .247 (:9443) and the fa proxy dataplane on .16 are on FastActions/fa#5884: https://github.com/FastActions/fa/pull/5884#issuecomment-5843773110 — 13.4 GB single stream at ~9.1 Gbit/s vs ~1.4 Gbit/s over gif/ESP, 10k rapid port-reuse connections with 0 failures, NTP/DNS OK.

shreyas-blacksmith and others added 3 commits September 28, 2026 01:17
Dial timing is exported through vprox_relay_dial_seconds, so carrying it
in every stream reply duplicates data the host proxy never uses. Trim the
reply to the status byte and fold the shared post-dial bookkeeping of the
TCP and UDP paths into session.dialed.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Mirror of the fa egressproxy/relayproto change: the hello carries
HMAC-SHA256(password, "client" || TLS-exporter binding) and a successful
reply carries the matching "server" proof, so the host proxy can trust the
self-signed relay without an SPKI pin and the password never crosses the
wire. Drops --print-spki-pin.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A fixed SO_RCVBUF disables receive autotuning, so a single download
stream is capped at buffer/RTT: 4 MiB at ~100 ms RTT measured
~40 MB/s against ~205 MB/s with autotuning on the same host and path.
tcp_rmem/tcp_wmem already bound the autotuned buffers.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@shreyas-blacksmith
shreyas-blacksmith marked this pull request as ready for review September 28, 2026 14:40

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread lib/relay/server.go
Comment thread lib/relay/server.go
Comment thread cmd/relay.go Outdated
- a hello that arrives while draining is refused with HelloUnavailable
  (the client retries the next relay) instead of HelloSessionLimit
- the session's yamux mux is published under a mutex and a GoAway
  requested before the mux exists is delivered once it is, so a session
  admitted in the hello window still learns about the drain
- one Server serves one listener; --listen is no longer repeatable and
  the unused Server.ln field is gone

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 9f7a274. Configure here.

Comment thread lib/relay/server.go Outdated
…ut too

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@shreyas-blacksmith
shreyas-blacksmith merged commit b6f169a into main Sep 29, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants