Allow limited inbound ICMP to Nexus, add ICMP type/code filters to firewall rules #8194

FelixMcFelix · 2025-05-21T12:14:37Z

This PR adds more detail to firewall protocol filters. Rather than being specified purely as a string->enum mapping, each protocol filter is now a standard tagged enum like many other types in the Nexus API. Here, this allows us to express more control over ICMP traffic. For instance:

all ICMP ([{"type": "icmp"}]),
all ICMP echo ([{"type": "icmp", "value": { icmp_type: 0 }}, {"type": "icmp", "value": { icmp_type: 8 }}])
a subset of ICMP destination unreachable ({"type": "icmp", "value": { icmp_type: 0, code: "0-4" }}).

Building on this, Nexus now has a firewall entry to permit the receipt of e.g., Destination Unreachable messages. I've added a new system endpoint to control whether this is enabled or disabled.

As part of this, I've converted the CRDB representation from a fixed ENUM[] to a STRING(32)[]. There are a couple of reasons for this:

ICMPv6 will need a similar level of specificity,
Users will need to specify arbitrary (ranges) of L4 protocols, once OPTE is more permissive (Allow/support arbitrary L4 protocols in guest traffic opte#609).
Future protocols we give first-class support to might also require/want in-depth filters.

Should close #7998.

That seems to be be the primary reason for the entire test suite having a Very Bad Time.

Thanks, CRDB.

FelixMcFelix · 2025-05-23T11:08:44Z

I've setup these bits on dublin this morning. After performing the schema update, everything seems happily functional, and posting true/false to the new system endpoint correctly enables/disables the ICMP allowance as expected. Console is broken when we go to the firewall rules tab, but that is to be expected given the OpenAPI changes.

On the actual MTU dscovery front, it looks as though both this PR and main are doing the correct thing?

If from the web console I perform a 'Reload (Override Cache)' we can kick off large enough TCP transfers to run into LSO. When we trace this through:

Dublin

BRM23230018 # dtrace -n 'xde_mc_tx:entry{ this->m = (mblk_t *)arg1; this->mss = this->m->b_datap->db_struioun.cksum.pad } xde_mc_tx:entry/this->m->b_datap->db_struioun.cksum.flags && this-
>mss != 0xBADD && this->mss/{printf("packet with MSS %d", this->m->b_datap->db_struioun.cksum.pad);}'
dtrace: description 'xde_mc_tx:entry' matched 2 probes
CPU     ID                    FUNCTION:NAME
111   5301                  xde_mc_tx:entry packet with MSS 1368
 69   5301                  xde_mc_tx:entry packet with MSS 1368
 69   5301                  xde_mc_tx:entry packet with MSS 1368
 69   5301                  xde_mc_tx:entry packet with MSS 1368
 69   5301                  xde_mc_tx:entry packet with MSS 1368
 69   5301                  xde_mc_tx:entry packet with MSS 1368
 69   5301                  xde_mc_tx:entry packet with MSS 1368
 69   5301                  xde_mc_tx:entry packet with MSS 1368
 69   5301                  xde_mc_tx:entry packet with MSS 1368
 69   5301                  xde_mc_tx:entry packet with MSS 1368
 69   5301                  xde_mc_tx:entry packet with MSS 1368
 69   5301                  xde_mc_tx:entry packet with MSS 1368
 69   5301                  xde_mc_tx:entry packet with MSS 1368
 69   5301                  xde_mc_tx:entry packet with MSS 1368
 69   5301                  xde_mc_tx:entry packet with MSS 1368
  1   5301                  xde_mc_tx:entry packet with MSS 1368
 69   5301                  xde_mc_tx:entry packet with MSS 1368
 69   5301                  xde_mc_tx:entry packet with MSS 1368
^C

BRM23230018 # dtrace -n 't4_eth_tx:entry{ this->m = (mblk_t *)arg1; this->mss = this->m->b_datap->db_struioun.cksum.pad } t4_eth_tx:entry/this->m->b_datap->db_struioun.cksum.flags && this-
>mss != 0xBADD && this->mss/{printf("packet with MSS %d", this->m->b_datap->db_struioun.cksum.pad);}'
dtrace: description 't4_eth_tx:entry' matched 2 probes
CPU     ID                    FUNCTION:NAME
  9   6360                  t4_eth_tx:entry packet with MSS 8928
 34   6360                  t4_eth_tx:entry packet with MSS 8928
117   6360                  t4_eth_tx:entry packet with MSS 8928
120   6360                  t4_eth_tx:entry packet with MSS 8928
110   6360                  t4_eth_tx:entry packet with MSS 8928
120   6360                  t4_eth_tx:entry packet with MSS 8928
 86   6360                  t4_eth_tx:entry packet with MSS 8928
 86   6360                  t4_eth_tx:entry packet with MSS 8928
 77   6360                  t4_eth_tx:entry packet with MSS 8928
 78   6360                  t4_eth_tx:entry packet with MSS 8928
 87   6360                  t4_eth_tx:entry packet with MSS 1368
  8   6360                  t4_eth_tx:entry packet with MSS 8928
 69   6360                  t4_eth_tx:entry packet with MSS 1368
 69   6360                  t4_eth_tx:entry packet with MSS 1368
 69   6360                  t4_eth_tx:entry packet with MSS 1368
 78   6360                  t4_eth_tx:entry packet with MSS 8928
 69   6360                  t4_eth_tx:entry packet with MSS 1368
 69   6360                  t4_eth_tx:entry packet with MSS 1368
 69   6360                  t4_eth_tx:entry packet with MSS 1368
 69   6360                  t4_eth_tx:entry packet with MSS 1368
 43   6360                  t4_eth_tx:entry packet with MSS 1368
 69   6360                  t4_eth_tx:entry packet with MSS 1368
 69   6360                  t4_eth_tx:entry packet with MSS 1368
 69   6360                  t4_eth_tx:entry packet with MSS 1368
 69   6360                  t4_eth_tx:entry packet with MSS 1368
 69   6360                  t4_eth_tx:entry packet with MSS 1368
 43   6360                  t4_eth_tx:entry packet with MSS 1368
 43   6360                  t4_eth_tx:entry packet with MSS 1368
 69   6360                  t4_eth_tx:entry packet with MSS 1368
 69   6360                  t4_eth_tx:entry packet with MSS 1368
 69   6360                  t4_eth_tx:entry packet with MSS 1368
 78   6360                  t4_eth_tx:entry packet with MSS 8928
 69   6360                  t4_eth_tx:entry packet with MSS 1368
 69   6360                  t4_eth_tx:entry packet with MSS 1368
 43   6360                  t4_eth_tx:entry packet with MSS 1368
 43   6360                  t4_eth_tx:entry packet with MSS 1368
^C

The latter query is also catching underlay traffic, hence the 8928B entries. An MSS of 1368 matches what I'd maybe expect for a reduced MTU of 1402 bytes (resp. 1448 for a 1500B MTU). Do we have any hits on the new rules?

BRM23230018 # opteadm dump-layer -p opte0 firewall
Port opte0 - Layer firewall
======================================================================
Inbound Flows
----------------------------------------------------------------------
PROTO  SRC IP        SPORT  DST IP      DPORT  HITS  ACTION
TCP    172.20.17.94  50056  172.30.2.6  443    0     no-op
TCP    172.20.17.94  50062  172.30.2.6  443    0     no-op

Outbound Flows
----------------------------------------------------------------------
PROTO  SRC IP      SPORT  DST IP        DPORT  HITS  ACTION
TCP    172.30.2.6  443    172.20.17.94  50056  1     no-op
TCP    172.30.2.6  443    172.20.17.94  50062  1     no-op

Inbound Rules
----------------------------------------------------------------------
ID   PRI    HITS  PREDICATES                               ACTION
126  65534  2     inner.ip.proto=TCP                       "Stateful Allow"
                  inner.ulp.dst=80,=443

125  65534  0     inner.icmp.type=time exceeded            "Stateful Allow"
124  65534  0     inner.icmp.type=destination unreachable  "Stateful Allow"
                  inner.icmp.code=4

DEF  --     427   --                                       "deny"

Outbound Rules
----------------------------------------------------------------------
ID   PRI  HITS  PREDICATES  ACTION
DEF  --   2     --          "stateful allow"

Apparently not, although it's hard to tell given that the reconcile task is completely reinstating the rules periodically so we're losing those counters. Note that all the deny entries here are from my own ping https://recovery.sys.dublin.eng.oxide.computer command.

Dogfood

BRM44220011 # dtrace -n 'xde_mc_tx:entry{ this->m = (mblk_t *)arg1; this->mss = this->m->b_datap->db_struioun.cksum.pad } xde_mc_tx:entry/this->m->b_datap->db_struioun.cksum.flags && this-
>mss != 0xBADD && this->mss/{printf("packet with MSS %d", this->m->b_datap->db_struioun.cksum.pad);}'
dtrace: description 'xde_mc_tx:entry' matched 2 probes
CPU     ID                    FUNCTION:NAME
 41   4868                  xde_mc_tx:entry packet with MSS 1448
  9   4868                  xde_mc_tx:entry packet with MSS 1368
 68   4868                  xde_mc_tx:entry packet with MSS 1368
 68   4868                  xde_mc_tx:entry packet with MSS 1368
 68   4868                  xde_mc_tx:entry packet with MSS 1368
 68   4868                  xde_mc_tx:entry packet with MSS 1368
 68   4868                  xde_mc_tx:entry packet with MSS 1368
 68   4868                  xde_mc_tx:entry packet with MSS 1368
 68   4868                  xde_mc_tx:entry packet with MSS 1368
 68   4868                  xde_mc_tx:entry packet with MSS 1368
127   4868                  xde_mc_tx:entry packet with MSS 1368
 68   4868                  xde_mc_tx:entry packet with MSS 1368
 68   4868                  xde_mc_tx:entry packet with MSS 1368
 68   4868                  xde_mc_tx:entry packet with MSS 1368
 68   4868                  xde_mc_tx:entry packet with MSS 1368
 68   4868                  xde_mc_tx:entry packet with MSS 1368
 68   4868                  xde_mc_tx:entry packet with MSS 1368
 68   4868                  xde_mc_tx:entry packet with MSS 1368
 68   4868                  xde_mc_tx:entry packet with MSS 1368
 68   4868                  xde_mc_tx:entry packet with MSS 1368
 68   4868                  xde_mc_tx:entry packet with MSS 1368
 83   4868                  xde_mc_tx:entry packet with MSS 1368
 83   4868                  xde_mc_tx:entry packet with MSS 1368
 83   4868                  xde_mc_tx:entry packet with MSS 1368
 83   4868                  xde_mc_tx:entry packet with MSS 1368
^C

BRM44220011 # dtrace -n 't4_eth_tx:entry{ this->m = (mblk_t *)arg1; this->mss = this->m->b_datap->db_struioun.cksum.pad } t4_eth_tx:entry/this->m->b_datap->db_struioun.cksum.flags && this->mss != 0xBADD && this->mss/{printf("packet with MSS %d", this->m->b_datap->db_struioun.cksum.pad);}'
dtrace: description 't4_eth_tx:entry' matched 2 probes
CPU     ID                    FUNCTION:NAME
 18   5927                  t4_eth_tx:entry packet with MSS 8928
 52   5927                  t4_eth_tx:entry packet with MSS 8928
 52   5927                  t4_eth_tx:entry packet with MSS 1448
113   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 76   5927                  t4_eth_tx:entry packet with MSS 8928
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 44   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 44   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 44   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 44   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 44   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 44   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 44   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 44   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 44   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 44   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 44   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 44   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 44   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 44   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1448
 44   5927                  t4_eth_tx:entry packet with MSS 1448
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 18   5927                  t4_eth_tx:entry packet with MSS 1368
 18   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
120   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
^C
 76   5927                  t4_eth_tx:entry packet with MSS 8928
 95   5927                  t4_eth_tx:entry packet with MSS 1368
 78   5927                  t4_eth_tx:entry packet with MSS 1368
 43   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 43   5927                  t4_eth_tx:entry packet with MSS 1368
 43   5927                  t4_eth_tx:entry packet with MSS 1368
 60   5927                  t4_eth_tx:entry packet with MSS 1368
 43   5927                  t4_eth_tx:entry packet with MSS 1368
 60   5927                  t4_eth_tx:entry packet with MSS 1368
 60   5927                  t4_eth_tx:entry packet with MSS 1368
  1   5927                  t4_eth_tx:entry packet with MSS 1368
113   5927                  t4_eth_tx:entry packet with MSS 1368
  1   5927                  t4_eth_tx:entry packet with MSS 1368
 69   5927                  t4_eth_tx:entry packet with MSS 1368
 86   5927                  t4_eth_tx:entry packet with MSS 1368
 52   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 42   5927                  t4_eth_tx:entry packet with MSS 1368
 42   5927                  t4_eth_tx:entry packet with MSS 1368
 42   5927                  t4_eth_tx:entry packet with MSS 1368
103   5927                  t4_eth_tx:entry packet with MSS 1368
 19   5927                  t4_eth_tx:entry packet with MSS 1368
 19   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368
 68   5927                  t4_eth_tx:entry packet with MSS 1368

There's a mix in there, but I assume we have something on the lab-side hitting the API periodically, since I see those 1448 entries passively. But dogfood appears to be discovering and using the reduced MTU just fine.

Fixing oxidecomputer/opte#748 may have been sufficient to get this working, and that PR has been pulled into main already. As a fun bonus, due to this work I can now run traceroute from the Nexus zone, so we might want to expand the scope of the allow rule across external DNS too. The NTP zones can't really benefit since they're behind a SNAT.

FelixMcFelix added 3 commits May 21, 2025 13:09

Initial plumbing.

f315451

More OpteHdl fixups.

8a33170

Parity.

38bd2ff

FelixMcFelix force-pushed the felixmcfelix/icmp-for-nexus branch from 5594fc0 to 38bd2ff Compare May 21, 2025 12:46

FelixMcFelix added 13 commits May 21, 2025 13:59

Merge branch 'main' into felixmcfelix/icmp-for-nexus

c5aec33

CRDB didn't like *that* migration.

283fc56

TODO: migration test.

3d13775

Merge branch 'main' into felixmcfelix/icmp-for-nexus

da4c157

Iterating on launch failures.

29db3dd

serde_json::from_str 🫠

01f0f4c

That seems to be be the primary reason for the entire test suite having a Very Bad Time.

Merge branch 'main' into felixmcfelix/icmp-for-nexus

3f8d07e

Simple migration query.

14804cb

An idempotent column rename

983e6b7

Thanks, CRDB.

Some control over ICMP allowance.

3d1a58e

Help the deploy job along, a little.

62a8b3d

Last test fixup.

564424f

fmt.

0c19f5f

FelixMcFelix mentioned this pull request May 23, 2025

Nexus ICMP paranoia breaks Path MTU Discovery #7998

Open

Renaming, and some serde/integration tests.

32a4914

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Allow limited inbound ICMP to Nexus, add ICMP type/code filters to firewall rules #8194

Allow limited inbound ICMP to Nexus, add ICMP type/code filters to firewall rules #8194

Uh oh!

FelixMcFelix commented May 21, 2025 •

edited

Loading

Uh oh!

FelixMcFelix commented May 23, 2025 •

edited

Loading

Uh oh!

Uh oh!

Allow limited inbound ICMP to Nexus, add ICMP type/code filters to firewall rules #8194

Are you sure you want to change the base?

Allow limited inbound ICMP to Nexus, add ICMP type/code filters to firewall rules #8194

Uh oh!

Conversation

FelixMcFelix commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FelixMcFelix commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dublin

Dogfood

Uh oh!

Uh oh!

FelixMcFelix commented May 21, 2025 •

edited

Loading

FelixMcFelix commented May 23, 2025 •

edited

Loading