[gateway] ingest sensor measurements from SPs into oximeter (#6354)

This branch adds code to the Management Gateway Service for periodically
polling sensor measurements from SPs and emitting it to Oximeter. In
particular, this consists of:

- a task for managing the metrics endpoint, waiting until MGS knows its
  underlay network address to bind the endpoint and register it with the
  control plane,
- tasks for polling sensor measurements from each individual SP that MGS
  knows about,
- a task that waits until SP discovery has completed and the rack ID to
  be known, and then spawns a poller task for every discovered SP slot

The SP poller tasks send samples to the Oximeter producer endpoint using
a `tokio::sync::broadcast` channel, which I've chosen primarily because
it can be used as a bounded ring buffer that actually overwrites the
*oldest* value when the buffer is full. This mostway, we use a bounded
amount of memory for samples, but prioritize the most recent samples if
we have to throw anything away because Oximeter hasn't come along to
collect them recently.

The poller tasks cache the component inventory and identifying
information from the SP, so that we don't have to re-read all this data
from the SP on every poll. While MGS, running on a host, would probably
be fine with doing this, it seems better to avoid making the SP do
unnecessary work at a 1Hz poll frequency, especially when *both* switch
zones are polling them. Instead, every time we poll sensor data from an
SP, we first ask it for its current state, and only invalidate our
cached understanding of the SP when the state changes. This way, if a SP
starts reporting new metrics due to a firmware update, or gets replaced
with a different chassis with a new serial number, revision, etc, we
won't continue to report metrics for stale targets, but we don't have to
reload all of that once per second. To detect scenarios where the SP's
state and/or identity has changed in the midst of polling its sensors
(which may result in mislabeled metrics), we check whether the SP's
state at the end of the poll matches its state at the beginning, and if
it's not, we poll again immediately with its new identity. 

At present, the timestamps for these metric samples is generated by MGS
--- it's the time when MGS received the sensor data from the SP, as MGS
understands it. Because we don't currently collect data that was
recorded prior to the switch zone coming up, we don't need to worry
about figuring out timestamps for data recorded by the SP prior to the
existence of a wall clock. Figuring out the SP/MGS timebase
synchronization is probably a lot of additional work, although it would
be nice to do in the future. At present, [metrics emitted by sled-agent
prior to NTP sync will also be from 1987][1], so I think it's fine to do
something similar here, especially because the potential solutions to
that [also have their fair share of tradeoffs][2].

The new metrics use a schema in
`oximeter/oximeter/schema/hardware-component.toml`. The target of these
metrics is a `hardware_component` that includes:

- the rack ID and the identity of the MGS instance that collected the
  metric,
- information identifying the chassis[^1] and of the SP that recorded
them (its serial number, model number, revision, and whether it's a
switch, a sled, or a power shelf),
- the SP's Hubris archive version (since the reported sensor data may
  change in future firmware releases)
- the SP's ID for the hardware component (e.g. "dev-7"), the kind of
  device (e.g. "tmp117", "max5970"), and the humman-readable description
  (e.g. "Southeast temperature sensor", "U.2 Sharkfin A hot swap
  controller", etc.) reported by the SP

Each kind of sensor reading has an individual metric
(`hardware_component:temperature`, `hardware_component:current`,
`hardware_component:voltage`, and so on). These metrics are labeled with
the SP-reported name of the individual sensor measurement channel. For
instance, a MAX5970 hotswap controller on sharkfin will have a voltage
and current metric named "V12_U2A_A0" for the 12V rail, and a voltage
and current metric named "V3P3_U2A_A0" for the 3.3V rail. Finally, a
`hardware_component:sensor_errors` metric records sensor errors reported
by the SP, labeled with the sensor name, what kind of sensor it is, and
a string representation of the error.

[1]:
    #6354 (comment)
[2]:
    #6354 (comment)

[^1]: I'm using "chassis" as a generic term to refer to "switch, sled,
    or power shelf".

Author: hawkw

Cargo.lock

@@ -213,6 +213,7 @@ impl From<omicron_common::api::internal::nexus::ProducerKind>
     fn from(kind: omicron_common::api::internal::nexus::ProducerKind) -> Self {
         use omicron_common::api::internal::nexus::ProducerKind;
         match kind {
+            ProducerKind::ManagementGateway => Self::ManagementGateway,
             ProducerKind::SledAgent => Self::SledAgent,
             ProducerKind::Service => Self::Service,
             ProducerKind::Instance => Self::Instance,
@@ -390,6 +391,9 @@ impl From<types::ProducerKind>
     fn from(kind: types::ProducerKind) -> Self {
         use omicron_common::api::internal::nexus::ProducerKind;
         match kind {
+            types::ProducerKind::ManagementGateway => {
+                ProducerKind::ManagementGateway
+            }
             types::ProducerKind::SledAgent => ProducerKind::SledAgent,
             types::ProducerKind::Instance => ProducerKind::Instance,
             types::ProducerKind::Service => ProducerKind::Service,

clients/nexus-client/src/lib.rs

@@ -26,6 +26,7 @@ impl From<omicron_common::api::internal::nexus::ProducerKind>
     fn from(kind: omicron_common::api::internal::nexus::ProducerKind) -> Self {
         use omicron_common::api::internal::nexus;
         match kind {
+            nexus::ProducerKind::ManagementGateway => Self::ManagementGateway,
             nexus::ProducerKind::Service => Self::Service,
             nexus::ProducerKind::SledAgent => Self::SledAgent,
             nexus::ProducerKind::Instance => Self::Instance,

clients/oximeter-client/src/lib.rs

@@ -223,6 +223,8 @@ pub enum ProducerKind {
     Service,
     /// The producer is a Propolis VMM managing a guest instance.
     Instance,
+    /// The producer is a management gateway service.
+    ManagementGateway,
 }
 
 /// Information announced by a metric server, used so that clients can contact it and collect

common/src/api/internal/nexus.rs

@@ -14,6 +14,7 @@ futures.workspace = true
 gateway-messages.workspace = true
 gateway-test-utils.workspace = true
 libc.workspace = true
+omicron-gateway.workspace = true
 omicron-workspace-hack.workspace = true
 signal-hook-tokio.workspace = true
 tokio.workspace = true

dev-tools/mgs-dev/Cargo.toml

@@ -8,6 +8,7 @@ use clap::{Args, Parser, Subcommand};
 use futures::StreamExt;
 use libc::SIGINT;
 use signal_hook_tokio::Signals;
+use std::net::SocketAddr;
 
 #[tokio::main]
 async fn main() -> anyhow::Result<()> {
@@ -36,7 +37,12 @@ enum MgsDevCmd {
 }
 
 #[derive(Clone, Debug, Args)]
-struct MgsRunArgs {}
+struct MgsRunArgs {
+    /// Override the address of the Nexus instance to use when registering the
+    /// Oximeter producer.
+    #[clap(long)]
+    nexus_address: Option<SocketAddr>,
+}
 
 impl MgsRunArgs {
     async fn exec(&self) -> Result<(), anyhow::Error> {
@@ -46,9 +52,23 @@ impl MgsRunArgs {
         let mut signal_stream = signals.fuse();
 
         println!("mgs-dev: setting up MGS ... ");
-        let gwtestctx = gateway_test_utils::setup::test_setup(
+        let (mut mgs_config, sp_sim_config) =
+            gateway_test_utils::setup::load_test_config();
+        if let Some(addr) = self.nexus_address {
+            mgs_config.metrics =
+                Some(gateway_test_utils::setup::MetricsConfig {
+                    disabled: false,
+                    dev_nexus_address: Some(addr),
+                    dev_bind_loopback: true,
+                });
+        }
+
+        let gwtestctx = gateway_test_utils::setup::test_setup_with_config(
             "mgs-dev",
             gateway_messages::SpPort::One,
+            mgs_config,
+            &sp_sim_config,
+            None,
         )
         .await;
         println!("mgs-dev: MGS is running.");

dev-tools/mgs-dev/src/main.rs

@@ -141,9 +141,16 @@ SP DETAILS: type "Sled" slot 0
 
     COMPONENTS
 
-        NAME         DESCRIPTION             DEVICE          PRESENCE SERIAL 
-        sp3-host-cpu FAKE host cpu           sp3-host-cpu    Present  None   
-        dev-0        FAKE temperature sensor fake-tmp-sensor Failed   None   
+        NAME         DESCRIPTION                              DEVICE          PRESENCE SERIAL 
+        sp3-host-cpu FAKE host cpu                            sp3-host-cpu    Present  None   
+        dev-0        FAKE temperature sensor                  fake-tmp-sensor Failed   None   
+        dev-1        FAKE temperature sensor                  tmp117          Present  None   
+        dev-2        FAKE Southeast temperature sensor        tmp117          Present  None   
+        dev-6        FAKE U.2 Sharkfin A VPD                  at24csw080      Present  None   
+        dev-7        FAKE U.2 Sharkfin A hot swap controller  max5970         Present  None   
+        dev-8        FAKE U.2 A NVMe Basic Management Command nvme_bmc        Present  None   
+        dev-39       FAKE T6 temperature sensor               tmp451          Present  None   
+        dev-53       FAKE Fan controller                      max31790        Present  None   
 
     CABOOSES: none found
 
@@ -167,8 +174,16 @@ SP DETAILS: type "Sled" slot 1
 
     COMPONENTS
 
-        NAME         DESCRIPTION   DEVICE       PRESENCE SERIAL 
-        sp3-host-cpu FAKE host cpu sp3-host-cpu Present  None   
+        NAME         DESCRIPTION                              DEVICE       PRESENCE SERIAL 
+        sp3-host-cpu FAKE host cpu                            sp3-host-cpu Present  None   
+        dev-0        FAKE temperature sensor                  tmp117       Present  None   
+        dev-1        FAKE temperature sensor                  tmp117       Present  None   
+        dev-2        FAKE Southeast temperature sensor        tmp117       Present  None   
+        dev-6        FAKE U.2 Sharkfin A VPD                  at24csw080   Present  None   
+        dev-7        FAKE U.2 Sharkfin A hot swap controller  max5970      Present  None   
+        dev-8        FAKE U.2 A NVMe Basic Management Command nvme_bmc     Present  None   
+        dev-39       FAKE T6 temperature sensor               tmp451       Present  None   
+        dev-53       FAKE Fan controller                      max31790     Present  None   
 
     CABOOSES: none found
 

dev-tools/omdb/tests/successes.out

@@ -88,6 +88,15 @@ addr = "[::1]:0"
 ignition-target = 3
 location = { switch0 = ["sled", 1], switch1 = ["sled", 1] }
 
+#
+# Configuration for SP sensor metrics polling
+#
+[metrics]
+# Allow the Oximeter metrics endpoint to bind on the loopback IP. This is
+# useful in local testing and development, when the gateway service is not
+# given a "real" underlay network IP.
+dev_bind_loopback = true
+
 #
 # NOTE: for the test suite, if mode = "file", the file path MUST be the sentinel
 # string "UNUSED".  The actual path will be generated by the test suite for each

gateway-test-utils/configs/config.test.toml

@@ -20,13 +20,19 @@ device = "fake-tmp-sensor"
 description = "FAKE temperature sensor 1"
 capabilities = 0x2
 presence = "Present"
+sensors = [
+    {name = "Southwest", kind = "Temperature", last_data.value = 41.7890625, last_data.timestamp = 1234 },
+]
 
 [[simulated_sps.sidecar.components]]
 id = "dev-1"
 device = "fake-tmp-sensor"
 description = "FAKE temperature sensor 2"
 capabilities = 0x2
 presence = "Failed"
+sensors = [
+    { name = "South", kind = "Temperature", last_error.value = "DeviceError", last_error.timestamp = 1234 },
+]
 
 [[simulated_sps.sidecar]]
 multicast_addr = "::1"
@@ -56,6 +62,82 @@ device = "fake-tmp-sensor"
 description = "FAKE temperature sensor"
 capabilities = 0x2
 presence = "Failed"
+sensors = [
+    { name = "Southwest", kind = "Temperature", last_error.value = "DeviceError", last_error.timestamp = 1234 },
+]
+[[simulated_sps.gimlet.components]]
+id = "dev-1"
+device = "tmp117"
+description = "FAKE temperature sensor"
+capabilities = 0x2
+presence = "Present"
+sensors = [
+    { name = "South", kind = "Temperature", last_data.value = 42.5625, last_data.timestamp = 1234 },
+]
+
+[[simulated_sps.gimlet.components]]
+id = "dev-2"
+device = "tmp117"
+description = "FAKE Southeast temperature sensor"
+capabilities = 0x2
+presence = "Present"
+sensors = [
+    { name = "Southeast", kind = "Temperature", last_data.value = 41.570313, last_data.timestamp = 1234 },
+]
+
+[[simulated_sps.gimlet.components]]
+id = "dev-6"
+device = "at24csw080"
+description = "FAKE U.2 Sharkfin A VPD"
+capabilities = 0x0
+presence = "Present"
+
+[[simulated_sps.gimlet.components]]
+id = "dev-7"
+device = "max5970"
+description = "FAKE U.2 Sharkfin A hot swap controller"
+capabilities = 0x2
+presence = "Present"
+sensors = [
+    { name = "V12_U2A_A0", kind = "Current", last_data.value = 0.45898438, last_data.timestamp = 1234 },
+    { name = "V3P3_U2A_A0", kind = "Current", last_data.value = 0.024414063, last_data.timestamp = 1234 },
+    { name = "V12_U2A_A0", kind = "Voltage", last_data.value = 12.03125, last_data.timestamp = 1234 },
+    { name = "V3P3_U2A_A0", kind = "Voltage", last_data.value = 3.328125, last_data.timestamp = 1234 },
+]
+
+[[simulated_sps.gimlet.components]]
+id = "dev-8"
+device = "nvme_bmc"
+description = "FAKE U.2 A NVMe Basic Management Command"
+capabilities = 0x2
+presence = "Present"
+sensors = [
+    { name = "U2_N0", kind = "Temperature", last_data.value = 56.0, last_data.timestamp = 1234 },
+]
+[[simulated_sps.gimlet.components]]
+id = "dev-39"
+device = "tmp451"
+description = "FAKE T6 temperature sensor"
+capabilities = 0x2
+presence = "Present"
+sensors = [
+    { name = "t6", kind = "Temperature", last_data.value = 70.625, last_data.timestamp = 1234 },
+]
+[[simulated_sps.gimlet.components]]
+id = "dev-53"
+device = "max31790"
+description = "FAKE Fan controller"
+capabilities = 0x2
+presence = "Present"
+sensors = [
+    { name = "Southeast", kind = "Speed", last_data.value = 2607.0, last_data.timestamp = 1234 },
+    { name = "Northeast", kind = "Speed", last_data.value = 2476.0, last_data.timestamp = 1234 },
+    { name = "South", kind = "Speed", last_data.value = 2553.0, last_data.timestamp = 1234 },
+    { name = "North", kind = "Speed", last_data.value = 2265.0, last_data.timestamp = 1234 },
+    { name = "Southwest", kind = "Speed", last_data.value = 2649.0, last_data.timestamp = 1234 },
+    { name = "Northwest", kind = "Speed", last_data.value = 2275.0, last_data.timestamp = 1234 },
+]
+
 
 [[simulated_sps.gimlet]]
 multicast_addr = "::1"
@@ -72,6 +154,90 @@ capabilities = 0
 presence = "Present"
 serial_console = "[::1]:0"
 
+
+[[simulated_sps.gimlet.components]]
+id = "dev-0"
+device = "tmp117"
+description = "FAKE temperature sensor"
+capabilities = 0x2
+presence = "Present"
+sensors = [
+    { name = "Southwest", kind = "Temperature", last_data.value = 41.3629, last_data.timestamp = 1234 },
+]
+[[simulated_sps.gimlet.components]]
+id = "dev-1"
+device = "tmp117"
+description = "FAKE temperature sensor"
+capabilities = 0x2
+presence = "Present"
+sensors = [
+    { name = "South", kind = "Temperature", last_data.value = 42.5625, last_data.timestamp = 1234 },
+]
+
+[[simulated_sps.gimlet.components]]
+id = "dev-2"
+device = "tmp117"
+description = "FAKE Southeast temperature sensor"
+capabilities = 0x2
+presence = "Present"
+sensors = [
+    { name = "Southeast", kind = "Temperature", last_data.value = 41.570313, last_data.timestamp = 1234 },
+]
+
+[[simulated_sps.gimlet.components]]
+id = "dev-6"
+device = "at24csw080"
+description = "FAKE U.2 Sharkfin A VPD"
+capabilities = 0x0
+presence = "Present"
+
+[[simulated_sps.gimlet.components]]
+id = "dev-7"
+device = "max5970"
+description = "FAKE U.2 Sharkfin A hot swap controller"
+capabilities = 0x2
+presence = "Present"
+sensors = [
+    { name = "V12_U2A_A0", kind = "Current", last_data.value = 0.41893438, last_data.timestamp = 1234 },
+    { name = "V3P3_U2A_A0", kind = "Current", last_data.value = 0.025614603, last_data.timestamp = 1234 },
+    { name = "V12_U2A_A0", kind = "Voltage", last_data.value = 12.02914, last_data.timestamp = 1234 },
+    { name = "V3P3_U2A_A0", kind = "Voltage", last_data.value = 3.2618, last_data.timestamp = 1234 },
+]
+
+[[simulated_sps.gimlet.components]]
+id = "dev-8"
+device = "nvme_bmc"
+description = "FAKE U.2 A NVMe Basic Management Command"
+capabilities = 0x2
+presence = "Present"
+sensors = [
+    { name = "U2_N0", kind = "Temperature", last_data.value = 56.0, last_data.timestamp = 1234 },
+]
+[[simulated_sps.gimlet.components]]
+id = "dev-39"
+device = "tmp451"
+description = "FAKE T6 temperature sensor"
+capabilities = 0x2
+presence = "Present"
+sensors = [
+    { name = "t6", kind = "Temperature", last_data.value = 70.625, last_data.timestamp = 1234 },
+]
+[[simulated_sps.gimlet.components]]
+id = "dev-53"
+device = "max31790"
+description = "FAKE Fan controller"
+capabilities = 0x2
+presence = "Present"
+sensors = [
+    { name = "Southeast", kind = "Speed", last_data.value = 2510.0, last_data.timestamp = 1234 },
+    { name = "Northeast", kind = "Speed", last_data.value = 2390.0, last_data.timestamp = 1234 },
+    { name = "South", kind = "Speed", last_data.value = 2467.0, last_data.timestamp = 1234 },
+    { name = "North", kind = "Speed", last_data.value = 2195.0, last_data.timestamp = 1234 },
+    { name = "Southwest", kind = "Speed", last_data.value = 2680.0, last_data.timestamp = 1234 },
+    { name = "Northwest", kind = "Speed", last_data.value = 2212.0, last_data.timestamp = 1234 },
+]
+
+
 #
 # NOTE: for the test suite, the [log] section is ignored; sp-sim logs are rolled
 # into the gateway logfile.