reconfigurator does not try to remove expunged datasets

[davepacheco]

I expunged an external-dns zone from a freshly-deployed a4x2 system. Things seemed to work, but executing the blueprint that expunged the zone did not change the configured datasets at all (i.e., reconfigurator didn't even try to remove the dataset that was expunged).

This is closely related to #6177 but that ticket covers the sled-agent side, whereas this covers the client (reconfigurator) side. We need to be careful about the order in which we fix this issue, #6177, and #6999 for the reason mentioned here.

---

Concretely, I started from what's described in #7299, which is basically a freshly deployed system with a few extra blueprints (with no meaningful changes). Then I used reconfigurator-cli to create a new blueprint that expunged the external-dns zone:

root@oxz_switch:~# ./reconfigurator-cli
generated RNG seed: stonily-revered-klipspringer
〉load reconfigurator.out dbaa2665-95b0-4959-b85c-d113665f1c56
loaded data from "reconfigurator.out"
result:
  system:
    using collection dbaa2665-95b0-4959-b85c-d113665f1c56 as source of sled inventory data
    loaded sleds: 09301471-54be-465e-a674-45da58c0c59b, a21d85f9-ff13-4800-9e86-5eb7b9e62eb5, f0b13ea8-c304-4a03-ab7c-da1fb2d0c612
    loaded collections: a205c682-1efb-49d6-b829-08b56b815641, cc7db484-24da-40f8-873e-e6b60fb26353, 25f934e9-ae50-4dc5-ac2e-2e10f394b827, dbaa2665-95b0-4959-b85c-d113665f1c56
    loaded blueprints: 2e24acb1-21c8-469a-ace5-bec12f89a539, 831679c9-26f8-4e3b-9873-e2522cfdc087, f55291a6-4265-4178-a269-7df22eb0a2b7
    loaded service IP pool ranges: [V4(Ipv4Range { first: 198.51.100.20, last: 198.51.100.29 })]
    loaded internal DNS generations: (none)
    loaded external DNS generations: (none)
  config:
    configured external DNS zone name: oxide.test
    configured silo names: default-silo, recovery

〉blueprint-edit 2e24acb1-21c8-469a-ace5-bec12f89a539 expunge-zone a21d85f9-ff13-4800-9e86-5eb7b9e62eb5 5c88569a-d13a-483c-b276-90730292d35b
blueprint edac838d-b454-48b7-b4e8-0bd433fecc6c created from blueprint 2e24acb1-21c8-469a-ace5-bec12f89a539: expunged zone 5c88569a-d13a-483c-b276-90730292d35b from sled a21d85f9-ff13-4800-9e86-5eb7b9e62eb5

〉blueprint-save edac838d-b454-48b7-b4e8-0bd433fecc6c blueprint-expunged-external-dns.out
saved blueprint edac838d-b454-48b7-b4e8-0bd433fecc6c to "blueprint-expunged-external-dns.out"

The diff looks right:

〉blueprint-diff 2e24acb1-21c8-469a-ace5-bec12f89a539 edac838d-b454-48b7-b4e8-0bd433fecc6c
from: blueprint 2e24acb1-21c8-469a-ace5-bec12f89a539
to:   blueprint edac838d-b454-48b7-b4e8-0bd433fecc6c

 UNCHANGED SLEDS:

  sled 09301471-54be-465e-a674-45da58c0c59b (active):

    physical disks at generation 2:
    -------------------------------------------------------------
    vendor             model                serial
    -------------------------------------------------------------
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g3_0
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g3_1
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g3_2
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g3_3
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g3_4


    datasets at generation 2:
    -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    dataset name                                                                                                   dataset uuid                           quota     reservation   compression
    -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/cockroachdb                                                     355ddd2b-b336-45c6-903f-ecdc99b26289   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crucible                                                              36b554b7-d442-4c58-96fe-71f88d9145e6   none      none          off
    oxp_29f71bd8-8d07-44f9-a587-8dd6552d6b3d/crucible                                                              53b65adf-99c3-472b-849c-9385e61c7f80   none      none          off
    oxp_639dcdf6-d036-4e59-94d8-9a05706f2a6e/crucible                                                              ee41d385-8bc3-4932-971a-2f2b1b3461a1   none      none          off
    oxp_b5c23cec-84ac-4810-9077-3f960af72fcd/crucible                                                              ab04e945-3a16-4a33-8e8b-a87b7d51df6f   none      none          off
    oxp_b9dc2f06-bf04-463c-8ffe-79d8357d852b/crucible                                                              9f46899f-ee8c-46e1-b6a3-99f74974379f   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/clickhouse                                                      ff1f90df-76b9-46a1-b2aa-c960db481817   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/external_dns                                                    5c81c9ca-43b3-43a2-ae34-2b2d09544492   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/internal_dns                                                    c2df2f3e-9b96-48ec-bcf2-23ffccbc3c05   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/zone                                                            0e6b946e-f23d-402d-87d0-6531ef26c24a   none      none          off
    oxp_29f71bd8-8d07-44f9-a587-8dd6552d6b3d/crypt/zone                                                            48566631-3eba-4cd5-b4e9-48bb9657d8de   none      none          off
    oxp_639dcdf6-d036-4e59-94d8-9a05706f2a6e/crypt/zone                                                            644e63c6-ff9b-4ac0-a932-b1063b2a0a02   none      none          off
    oxp_b5c23cec-84ac-4810-9077-3f960af72fcd/crypt/zone                                                            c04b17c8-5626-409a-8eb1-f0b797fe9788   none      none          off
    oxp_b9dc2f06-bf04-463c-8ffe-79d8357d852b/crypt/zone                                                            f760e7e6-089d-4743-a339-1e0d882a40aa   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/zone/oxz_clickhouse_0208b07b-4ccd-4e8b-b334-1ec8ecc4bae7        d343db99-4aa1-4f6c-aa53-7be34e509d36   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/zone/oxz_cockroachdb_67989451-63b3-48c2-aef8-c6e511fb122f       9c747a15-0ce0-41bd-bc2f-d7ee792a3592   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/zone/oxz_crucible_79da9a6d-9db8-4fd9-b2dc-ba0d2fed0bdd          6ec06bb2-9fd0-45d0-a35b-d6cbf894537d   none      none          off
    oxp_639dcdf6-d036-4e59-94d8-9a05706f2a6e/crypt/zone/oxz_crucible_7c5ae2a6-95a7-4cd9-8631-68477b614d97          a4dd3c84-8616-4182-8214-c9677a57318e   none      none          off
    oxp_29f71bd8-8d07-44f9-a587-8dd6552d6b3d/crypt/zone/oxz_crucible_a0e1f460-0a7c-4788-b4a2-3ecc2ac909ad          a93a6377-6d78-4c34-b220-f2c4926e4795   none      none          off
    oxp_b5c23cec-84ac-4810-9077-3f960af72fcd/crypt/zone/oxz_crucible_b16765f7-de2f-44c2-b6ae-4fb60f5c4d53          660a8beb-683a-4c00-82e9-b8f19e9f86fe   none      none          off
    oxp_b9dc2f06-bf04-463c-8ffe-79d8357d852b/crypt/zone/oxz_crucible_f1497d76-dc9d-40ac-9670-3b17e1a64b84          37896114-bb11-402a-a1a2-f0fe800d794d   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/zone/oxz_crucible_pantry_31170c27-6736-418a-b496-b740a5668e4c   7fc29a07-ce43-41d9-9a1e-1bfd3be35a46   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/zone/oxz_external_dns_8ea2a399-24f4-44c1-a31c-9b83750de6c1      defada08-7725-4d45-acec-2fe308bae2e7   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/zone/oxz_internal_dns_0c752586-eb3b-4857-aa35-85a1206b6efb      28d664b6-8003-489b-b824-262025837f56   none      none          off
    oxp_29f71bd8-8d07-44f9-a587-8dd6552d6b3d/crypt/zone/oxz_nexus_bdf0578f-d601-44e9-9318-5b5083d3e57d             30179c95-0623-468d-8b25-b78499725884   none      none          off
    oxp_639dcdf6-d036-4e59-94d8-9a05706f2a6e/crypt/zone/oxz_ntp_d2d54e36-5a56-491e-8910-8a2cbe10cce9               c6649371-7506-4fa3-ad41-f4d7666bc1a3   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/debug                                                           50e80495-eb69-4716-8fa4-9f697b454498   100 GiB   none          gzip-9
    oxp_29f71bd8-8d07-44f9-a587-8dd6552d6b3d/crypt/debug                                                           f8a339ec-c736-44cd-b562-d845da329ece   100 GiB   none          gzip-9
    oxp_639dcdf6-d036-4e59-94d8-9a05706f2a6e/crypt/debug                                                           7f64c446-bc62-4885-8d74-7a40906c6f20   100 GiB   none          gzip-9
    oxp_b5c23cec-84ac-4810-9077-3f960af72fcd/crypt/debug                                                           59d97482-de4f-4f5a-9961-d2b00126a6a6   100 GiB   none          gzip-9
    oxp_b9dc2f06-bf04-463c-8ffe-79d8357d852b/crypt/debug                                                           914cc29f-b2cd-4128-9393-18e8d6715730   100 GiB   none          gzip-9


    omicron zones at generation 5:
    --------------------------------------------------------------------------------------------
    zone type         zone id                                disposition   underlay IP
    --------------------------------------------------------------------------------------------
    clickhouse        0208b07b-4ccd-4e8b-b334-1ec8ecc4bae7   in service    fd00:1122:3344:103::6
    cockroach_db      67989451-63b3-48c2-aef8-c6e511fb122f   in service    fd00:1122:3344:103::3
    crucible          79da9a6d-9db8-4fd9-b2dc-ba0d2fed0bdd   in service    fd00:1122:3344:103::8
    crucible          7c5ae2a6-95a7-4cd9-8631-68477b614d97   in service    fd00:1122:3344:103::a
    crucible          a0e1f460-0a7c-4788-b4a2-3ecc2ac909ad   in service    fd00:1122:3344:103::c
    crucible          b16765f7-de2f-44c2-b6ae-4fb60f5c4d53   in service    fd00:1122:3344:103::b
    crucible          f1497d76-dc9d-40ac-9670-3b17e1a64b84   in service    fd00:1122:3344:103::9
    crucible_pantry   31170c27-6736-418a-b496-b740a5668e4c   in service    fd00:1122:3344:103::7
    external_dns      8ea2a399-24f4-44c1-a31c-9b83750de6c1   in service    fd00:1122:3344:103::4
    internal_dns      0c752586-eb3b-4857-aa35-85a1206b6efb   in service    fd00:1122:3344:3::1
    internal_ntp      d2d54e36-5a56-491e-8910-8a2cbe10cce9   in service    fd00:1122:3344:103::d
    nexus             bdf0578f-d601-44e9-9318-5b5083d3e57d   in service    fd00:1122:3344:103::5


  sled f0b13ea8-c304-4a03-ab7c-da1fb2d0c612 (active):

    physical disks at generation 2:
    -------------------------------------------------------------
    vendor             model                serial
    -------------------------------------------------------------
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g1_0
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g1_1
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g1_2
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g1_3
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g1_4


    datasets at generation 2:
    -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    dataset name                                                                                                   dataset uuid                           quota     reservation   compression
    -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    oxp_89021d00-c808-49ec-ada3-427646a63b46/crypt/cockroachdb                                                     2cfac653-9dba-492a-b4e5-c8efe5a1e1b9   none      none          off
    oxp_9d453deb-a057-438f-b555-627c501d9257/crypt/cockroachdb                                                     12feda67-cf20-4f01-8094-69a956526990   none      none          off
    oxp_60155e33-c180-4490-9f28-0cb4003ede47/crucible                                                              d597754a-6fe3-4d18-b8be-8dc0a269628d   none      none          off
    oxp_89021d00-c808-49ec-ada3-427646a63b46/crucible                                                              beee4d0a-44b9-4298-9014-da523dd9eaa7   none      none          off
    oxp_9d453deb-a057-438f-b555-627c501d9257/crucible                                                              644161d1-8a5f-4609-ae2d-21c4ae54fdd4   none      none          off
    oxp_c1b87796-8265-4c6b-b41a-be7de82934f6/crucible                                                              5d8edd26-0cd7-434e-af7b-c50a319cd1ea   none      none          off
    oxp_f63bc936-be56-4092-a2cb-6a8508f8a23d/crucible                                                              cdf87aea-8f8f-4e49-b921-db848d6e25ef   none      none          off
    oxp_89021d00-c808-49ec-ada3-427646a63b46/crypt/internal_dns                                                    b1e1c105-638d-4401-a5af-534f219a11f4   none      none          off
    oxp_60155e33-c180-4490-9f28-0cb4003ede47/crypt/zone                                                            d621f634-68d8-4741-85bd-53e82095c094   none      none          off
    oxp_89021d00-c808-49ec-ada3-427646a63b46/crypt/zone                                                            7c00d8d6-86c6-4446-a90f-24c62ac741c4   none      none          off
    oxp_9d453deb-a057-438f-b555-627c501d9257/crypt/zone                                                            b5fcfc17-2954-484a-bbf8-8c51b97a7aa3   none      none          off
    oxp_c1b87796-8265-4c6b-b41a-be7de82934f6/crypt/zone                                                            aacb001f-2c30-427d-96cf-10f553de5446   none      none          off
    oxp_f63bc936-be56-4092-a2cb-6a8508f8a23d/crypt/zone                                                            662d12c6-b68b-4aac-b9be-9c999bf5b05e   none      none          off
    oxp_89021d00-c808-49ec-ada3-427646a63b46/crypt/zone/oxz_cockroachdb_2c00cc83-0211-4662-9f95-ecbf1c8fc26b       1478643a-989f-43e3-bc24-4ab2c9f40d15   none      none          off
    oxp_9d453deb-a057-438f-b555-627c501d9257/crypt/zone/oxz_cockroachdb_2eb1e0f2-72a3-48bc-a76e-cb5b8838938f       905d06de-7198-41a7-a696-35d1c66bc882   none      none          off
    oxp_60155e33-c180-4490-9f28-0cb4003ede47/crypt/zone/oxz_crucible_0a1d8035-c373-4de7-8d70-ebe0a4b16758          ac0ca38c-128a-42a5-b87a-670d7b807ed4   none      none          off
    oxp_f63bc936-be56-4092-a2cb-6a8508f8a23d/crypt/zone/oxz_crucible_476d5bde-b360-4c49-8093-5d56be92c63f          d642b0d0-1f61-4ad0-9840-98464d3b4ba2   none      none          off
    oxp_9d453deb-a057-438f-b555-627c501d9257/crypt/zone/oxz_crucible_74d4206f-6bdb-4c47-b01c-9e2a876d622e          19a81e6a-938a-4579-bf95-e1105fe1b0a5   none      none          off
    oxp_c1b87796-8265-4c6b-b41a-be7de82934f6/crypt/zone/oxz_crucible_a3cb63c4-0044-4705-a96f-87d83b09ea5d          7df8ef54-cb4a-4676-b163-7d0418043127   none      none          off
    oxp_89021d00-c808-49ec-ada3-427646a63b46/crypt/zone/oxz_crucible_c673bafa-fcd2-40c4-a7f9-20906391ba42          88a432f9-1f5f-4799-a303-ab5806f73f3f   none      none          off
    oxp_60155e33-c180-4490-9f28-0cb4003ede47/crypt/zone/oxz_crucible_pantry_47d0ca61-9861-4d61-948c-890c5f4ac5d5   3cbb1ed7-3290-428d-bebb-f54297c09909   none      none          off
    oxp_89021d00-c808-49ec-ada3-427646a63b46/crypt/zone/oxz_internal_dns_23d6123b-1b33-4ea9-8eb0-62607cc29679      74b11e9a-bf6a-458d-a2c7-740576494cc9   none      none          off
    oxp_89021d00-c808-49ec-ada3-427646a63b46/crypt/zone/oxz_nexus_7b9677c8-0daf-4ab1-bc1d-9d655b964b33             0d21ae90-6ee4-4422-9bc6-f336fd4b9ff5   none      none          off
    oxp_89021d00-c808-49ec-ada3-427646a63b46/crypt/zone/oxz_ntp_f4bcdbb7-c64b-4473-9346-e5c6c411e76d               7a9e077f-5512-4f8c-b780-36c326aa0ad5   none      none          off
    oxp_60155e33-c180-4490-9f28-0cb4003ede47/crypt/zone/oxz_oximeter_f4806e03-9ecc-49ef-8ee0-ff7aee4dab23          e48d0b5a-2a9e-4794-b45c-858bf8ebf833   none      none          off
    oxp_60155e33-c180-4490-9f28-0cb4003ede47/crypt/debug                                                           5f96f85f-15df-4b76-bc99-7cdd1e07ff16   100 GiB   none          gzip-9
    oxp_89021d00-c808-49ec-ada3-427646a63b46/crypt/debug                                                           5aa39aed-b571-4c7d-b8ae-040c64cfe25d   100 GiB   none          gzip-9
    oxp_9d453deb-a057-438f-b555-627c501d9257/crypt/debug                                                           45fcfb99-bc89-4c87-b5c6-ee0320d3420c   100 GiB   none          gzip-9
    oxp_c1b87796-8265-4c6b-b41a-be7de82934f6/crypt/debug                                                           d8ffad03-2ea6-432f-beb2-a5b2fc4bf885   100 GiB   none          gzip-9
    oxp_f63bc936-be56-4092-a2cb-6a8508f8a23d/crypt/debug                                                           9156026c-38ed-4ee5-b49f-b4682b7a2702   100 GiB   none          gzip-9


    omicron zones at generation 5:
    --------------------------------------------------------------------------------------------
    zone type         zone id                                disposition   underlay IP
    --------------------------------------------------------------------------------------------
    boundary_ntp      f4bcdbb7-c64b-4473-9346-e5c6c411e76d   in service    fd00:1122:3344:102::d
    cockroach_db      2c00cc83-0211-4662-9f95-ecbf1c8fc26b   in service    fd00:1122:3344:102::3
    cockroach_db      2eb1e0f2-72a3-48bc-a76e-cb5b8838938f   in service    fd00:1122:3344:102::4
    crucible          0a1d8035-c373-4de7-8d70-ebe0a4b16758   in service    fd00:1122:3344:102::c
    crucible          476d5bde-b360-4c49-8093-5d56be92c63f   in service    fd00:1122:3344:102::a
    crucible          74d4206f-6bdb-4c47-b01c-9e2a876d622e   in service    fd00:1122:3344:102::9
    crucible          a3cb63c4-0044-4705-a96f-87d83b09ea5d   in service    fd00:1122:3344:102::b
    crucible          c673bafa-fcd2-40c4-a7f9-20906391ba42   in service    fd00:1122:3344:102::8
    crucible_pantry   47d0ca61-9861-4d61-948c-890c5f4ac5d5   in service    fd00:1122:3344:102::7
    internal_dns      23d6123b-1b33-4ea9-8eb0-62607cc29679   in service    fd00:1122:3344:2::1
    nexus             7b9677c8-0daf-4ab1-bc1d-9d655b964b33   in service    fd00:1122:3344:102::5
    oximeter          f4806e03-9ecc-49ef-8ee0-ff7aee4dab23   in service    fd00:1122:3344:102::6


 MODIFIED SLEDS:

  sled a21d85f9-ff13-4800-9e86-5eb7b9e62eb5 (active):

    physical disks at generation 2:
    -------------------------------------------------------------
    vendor             model                serial
    -------------------------------------------------------------
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g0_0
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g0_1
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g0_2
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g0_3
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g0_4


    datasets generation 2 -> 3:
    -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    dataset name                                                                                                   dataset uuid                           quota     reservation   compression
    -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc/crypt/cockroachdb                                                     119fec8f-785e-4833-90f2-04311332041f   none      none          off
    oxp_fb97a84c-3ac3-494c-9c76-26d6357aebf6/crypt/cockroachdb                                                     1f8b0e1a-0545-44f1-9366-6aaf29a0fb83   none      none          off
    oxp_15b53b30-72cf-4edb-a7c4-325ee3f7c679/crucible                                                              aff65822-a39d-4b21-9b1e-d94ba1688057   none      none          off
    oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc/crucible                                                              76cc81e2-f740-4cb9-b495-10ae9cbb8cbd   none      none          off
    oxp_44152dd0-a9bc-40a0-a64f-d375b950e5bf/crucible                                                              305cdc80-78d1-4e19-8b68-ba68be9e7b1a   none      none          off
    oxp_abbce0e4-bda7-4853-b89a-37721aa638fb/crucible                                                              a3c3f7cc-b4e4-439b-9eb6-e97ad14b58fe   none      none          off
    oxp_fb97a84c-3ac3-494c-9c76-26d6357aebf6/crucible                                                              1c2913e3-adce-4a64-8c46-b47c8018b291   none      none          off
    oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc/crypt/internal_dns                                                    03492708-b9f6-4c40-a0fb-965734468a66   none      none          off
    oxp_15b53b30-72cf-4edb-a7c4-325ee3f7c679/crypt/zone                                                            cac4df64-07ac-4266-9c73-822fb620ff9f   none      none          off
    oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc/crypt/zone                                                            da1424d7-fe97-4f3e-84a1-4842ab4dee99   none      none          off
    oxp_44152dd0-a9bc-40a0-a64f-d375b950e5bf/crypt/zone                                                            97c712a4-cb43-4e83-960b-9382964bae74   none      none          off
    oxp_abbce0e4-bda7-4853-b89a-37721aa638fb/crypt/zone                                                            7fe4e11d-6cfb-4e35-973c-fe366a8cd9a1   none      none          off
    oxp_fb97a84c-3ac3-494c-9c76-26d6357aebf6/crypt/zone                                                            6f7a280d-2aca-4e64-9ecd-c9432c9a80f4   none      none          off
    oxp_fb97a84c-3ac3-494c-9c76-26d6357aebf6/crypt/zone/oxz_cockroachdb_792489e0-9d08-4d21-9caa-0440b12d6ade       2d4851d8-0d00-45a3-b351-9353339e3f17   none      none          off
    oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc/crypt/zone/oxz_cockroachdb_83e82c9d-e587-4a68-a18d-128712e2741f       c4351281-bd7d-41bd-b5a1-e9118dac2f16   none      none          off
    oxp_15b53b30-72cf-4edb-a7c4-325ee3f7c679/crypt/zone/oxz_crucible_049d9f96-6e06-43a0-a924-35146efd7b8c          801a8141-9e83-4cc0-9428-fb1db210657d   none      none          off
    oxp_fb97a84c-3ac3-494c-9c76-26d6357aebf6/crypt/zone/oxz_crucible_6d03d1db-c605-4a00-9717-3bf867969ad9          1728ac4b-67aa-451f-b674-ad9b60dd5fdb   none      none          off
    oxp_44152dd0-a9bc-40a0-a64f-d375b950e5bf/crypt/zone/oxz_crucible_8ab366b1-b6f7-4467-8468-36745617618c          05146ed5-3246-4591-adec-e8a7941b8e83   none      none          off
    oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc/crypt/zone/oxz_crucible_b26d1945-d24b-41f9-b80f-77af3638941a          3d967991-7f45-41f6-9cb5-827b90645547   none      none          off
    oxp_abbce0e4-bda7-4853-b89a-37721aa638fb/crypt/zone/oxz_crucible_b67c6455-b415-4404-b997-588d39ae4a6c          bd097130-13a7-4478-9a0e-e6eadeefabf4   none      none          off
    oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc/crypt/zone/oxz_crucible_pantry_8b81d199-146c-4ccf-aa23-f57140a6b582   d4ce5cf6-96e8-44f0-975f-b5601f099c09   none      none          off
    oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc/crypt/zone/oxz_internal_dns_c3e14e81-0d5a-4461-80f5-1144ccdf4da5      04128921-a9b4-40f1-bd7a-17e89c22fac9   none      none          off
    oxp_fb97a84c-3ac3-494c-9c76-26d6357aebf6/crypt/zone/oxz_nexus_b8da49f4-5b11-4773-be0c-8cbae8f31400             0ab36247-5490-4fe0-b3ab-cbd5d9170efc   none      none          off
    oxp_15b53b30-72cf-4edb-a7c4-325ee3f7c679/crypt/zone/oxz_ntp_2b3c2cf8-bf97-4a7c-9327-712f1d589c7b               43a80037-e23f-44be-84eb-bb30bd1f539e   none      none          off
    oxp_15b53b30-72cf-4edb-a7c4-325ee3f7c679/crypt/debug                                                           6f610524-4329-4634-adab-ffbd6f65a653   100 GiB   none          gzip-9
    oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc/crypt/debug                                                           eb609f8e-0bd4-4f07-8c8c-8ad72bc3b8e6   100 GiB   none          gzip-9
    oxp_44152dd0-a9bc-40a0-a64f-d375b950e5bf/crypt/debug                                                           dd9e2a12-9e1a-4c01-9090-2318a4ca5bf6   100 GiB   none          gzip-9
    oxp_abbce0e4-bda7-4853-b89a-37721aa638fb/crypt/debug                                                           87c3fca0-606e-47ff-81cb-0917fe240f85   100 GiB   none          gzip-9
    oxp_fb97a84c-3ac3-494c-9c76-26d6357aebf6/crypt/debug                                                           d60fcccf-fc65-404c-a2d9-0c780a7e1ea8   100 GiB   none          gzip-9
*   oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc/crypt/external_dns                                                    0712c964-bf3f-45e7-9e75-361652b53736   none      none          off
*   oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc/crypt/zone/oxz_external_dns_5c88569a-d13a-483c-b276-90730292d35b      5073be4d-3c52-4948-b7ce-2a708c724d02   none      none          off


    omicron zones generation 5 -> 6:
    ---------------------------------------------------------------------------------------------
    zone type         zone id                                disposition    underlay IP
    ---------------------------------------------------------------------------------------------
    boundary_ntp      2b3c2cf8-bf97-4a7c-9327-712f1d589c7b   in service     fd00:1122:3344:101::d
    cockroach_db      792489e0-9d08-4d21-9caa-0440b12d6ade   in service     fd00:1122:3344:101::4
    cockroach_db      83e82c9d-e587-4a68-a18d-128712e2741f   in service     fd00:1122:3344:101::3
    crucible          049d9f96-6e06-43a0-a924-35146efd7b8c   in service     fd00:1122:3344:101::a
    crucible          6d03d1db-c605-4a00-9717-3bf867969ad9   in service     fd00:1122:3344:101::9
    crucible          8ab366b1-b6f7-4467-8468-36745617618c   in service     fd00:1122:3344:101::c
    crucible          b26d1945-d24b-41f9-b80f-77af3638941a   in service     fd00:1122:3344:101::8
    crucible          b67c6455-b415-4404-b997-588d39ae4a6c   in service     fd00:1122:3344:101::b
    crucible_pantry   8b81d199-146c-4ccf-aa23-f57140a6b582   in service     fd00:1122:3344:101::7
    internal_dns      c3e14e81-0d5a-4461-80f5-1144ccdf4da5   in service     fd00:1122:3344:1::1
    nexus             b8da49f4-5b11-4773-be0c-8cbae8f31400   in service     fd00:1122:3344:101::6
*   external_dns      5c88569a-d13a-483c-b276-90730292d35b   - in service   fd00:1122:3344:101::5
     └─                                                      + expunged


 COCKROACHDB SETTINGS:
    state fingerprint:::::::::::::::::   d4d87aa2ad877a4cc2fddd0573952362739110de (unchanged)
    cluster.preserve_downgrade_option:   "22.1" (unchanged)

 METADATA:
    internal DNS version:   1 (unchanged)
    external DNS version:   2 (unchanged)


internal DNS:
* DNS zone: "control-plane.oxide.internal":
    name: 0208b07b-4ccd-4e8b-b334-1ec8ecc4bae7.host          (records: 1)
        AAAA fd00:1122:3344:103::6
    name: 049d9f96-6e06-43a0-a924-35146efd7b8c.host          (records: 1)
        AAAA fd00:1122:3344:101::a
    name: 0a1d8035-c373-4de7-8d70-ebe0a4b16758.host          (records: 1)
        AAAA fd00:1122:3344:102::c
    name: 0c752586-eb3b-4857-aa35-85a1206b6efb.host          (records: 1)
        AAAA fd00:1122:3344:3::1
    name: 23d6123b-1b33-4ea9-8eb0-62607cc29679.host          (records: 1)
        AAAA fd00:1122:3344:2::1
    name: 2b3c2cf8-bf97-4a7c-9327-712f1d589c7b.host          (records: 1)
        AAAA fd00:1122:3344:101::d
    name: 2c00cc83-0211-4662-9f95-ecbf1c8fc26b.host          (records: 1)
        AAAA fd00:1122:3344:102::3
    name: 2eb1e0f2-72a3-48bc-a76e-cb5b8838938f.host          (records: 1)
        AAAA fd00:1122:3344:102::4
    name: 31170c27-6736-418a-b496-b740a5668e4c.host          (records: 1)
        AAAA fd00:1122:3344:103::7
    name: 476d5bde-b360-4c49-8093-5d56be92c63f.host          (records: 1)
        AAAA fd00:1122:3344:102::a
    name: 47d0ca61-9861-4d61-948c-890c5f4ac5d5.host          (records: 1)
        AAAA fd00:1122:3344:102::7
-   name: 5c88569a-d13a-483c-b276-90730292d35b.host          (records: 1)
-       AAAA fd00:1122:3344:101::5
    name: 67989451-63b3-48c2-aef8-c6e511fb122f.host          (records: 1)
        AAAA fd00:1122:3344:103::3
    name: 6d03d1db-c605-4a00-9717-3bf867969ad9.host          (records: 1)
        AAAA fd00:1122:3344:101::9
    name: 74d4206f-6bdb-4c47-b01c-9e2a876d622e.host          (records: 1)
        AAAA fd00:1122:3344:102::9
    name: 792489e0-9d08-4d21-9caa-0440b12d6ade.host          (records: 1)
        AAAA fd00:1122:3344:101::4
    name: 79da9a6d-9db8-4fd9-b2dc-ba0d2fed0bdd.host          (records: 1)
        AAAA fd00:1122:3344:103::8
    name: 7b9677c8-0daf-4ab1-bc1d-9d655b964b33.host          (records: 1)
        AAAA fd00:1122:3344:102::5
    name: 7c5ae2a6-95a7-4cd9-8631-68477b614d97.host          (records: 1)
        AAAA fd00:1122:3344:103::a
    name: 83e82c9d-e587-4a68-a18d-128712e2741f.host          (records: 1)
        AAAA fd00:1122:3344:101::3
    name: 8ab366b1-b6f7-4467-8468-36745617618c.host          (records: 1)
        AAAA fd00:1122:3344:101::c
    name: 8b81d199-146c-4ccf-aa23-f57140a6b582.host          (records: 1)
        AAAA fd00:1122:3344:101::7
    name: 8ea2a399-24f4-44c1-a31c-9b83750de6c1.host          (records: 1)
        AAAA fd00:1122:3344:103::4
    name: _boundary-ntp._tcp                                 (records: 2)
        SRV  port   123 2b3c2cf8-bf97-4a7c-9327-712f1d589c7b.host.control-plane.oxide.internal
        SRV  port   123 f4bcdbb7-c64b-4473-9346-e5c6c411e76d.host.control-plane.oxide.internal
    name: _clickhouse-native._tcp                            (records: 1)
        SRV  port  9000 0208b07b-4ccd-4e8b-b334-1ec8ecc4bae7.host.control-plane.oxide.internal
    name: _clickhouse._tcp                                   (records: 1)
        SRV  port  8123 0208b07b-4ccd-4e8b-b334-1ec8ecc4bae7.host.control-plane.oxide.internal
    name: _cockroach._tcp                                    (records: 5)
        SRV  port 32221 2c00cc83-0211-4662-9f95-ecbf1c8fc26b.host.control-plane.oxide.internal
        SRV  port 32221 2eb1e0f2-72a3-48bc-a76e-cb5b8838938f.host.control-plane.oxide.internal
        SRV  port 32221 67989451-63b3-48c2-aef8-c6e511fb122f.host.control-plane.oxide.internal
        SRV  port 32221 792489e0-9d08-4d21-9caa-0440b12d6ade.host.control-plane.oxide.internal
        SRV  port 32221 83e82c9d-e587-4a68-a18d-128712e2741f.host.control-plane.oxide.internal
    name: _crucible-pantry._tcp                              (records: 3)
        SRV  port 17000 31170c27-6736-418a-b496-b740a5668e4c.host.control-plane.oxide.internal
        SRV  port 17000 47d0ca61-9861-4d61-948c-890c5f4ac5d5.host.control-plane.oxide.internal
        SRV  port 17000 8b81d199-146c-4ccf-aa23-f57140a6b582.host.control-plane.oxide.internal
    name: _crucible._tcp.049d9f96-6e06-43a0-a924-35146efd7b8c (records: 1)
        SRV  port 32345 049d9f96-6e06-43a0-a924-35146efd7b8c.host.control-plane.oxide.internal
    name: _crucible._tcp.0a1d8035-c373-4de7-8d70-ebe0a4b16758 (records: 1)
        SRV  port 32345 0a1d8035-c373-4de7-8d70-ebe0a4b16758.host.control-plane.oxide.internal
    name: _crucible._tcp.476d5bde-b360-4c49-8093-5d56be92c63f (records: 1)
        SRV  port 32345 476d5bde-b360-4c49-8093-5d56be92c63f.host.control-plane.oxide.internal
    name: _crucible._tcp.6d03d1db-c605-4a00-9717-3bf867969ad9 (records: 1)
        SRV  port 32345 6d03d1db-c605-4a00-9717-3bf867969ad9.host.control-plane.oxide.internal
    name: _crucible._tcp.74d4206f-6bdb-4c47-b01c-9e2a876d622e (records: 1)
        SRV  port 32345 74d4206f-6bdb-4c47-b01c-9e2a876d622e.host.control-plane.oxide.internal
    name: _crucible._tcp.79da9a6d-9db8-4fd9-b2dc-ba0d2fed0bdd (records: 1)
        SRV  port 32345 79da9a6d-9db8-4fd9-b2dc-ba0d2fed0bdd.host.control-plane.oxide.internal
    name: _crucible._tcp.7c5ae2a6-95a7-4cd9-8631-68477b614d97 (records: 1)
        SRV  port 32345 7c5ae2a6-95a7-4cd9-8631-68477b614d97.host.control-plane.oxide.internal
    name: _crucible._tcp.8ab366b1-b6f7-4467-8468-36745617618c (records: 1)
        SRV  port 32345 8ab366b1-b6f7-4467-8468-36745617618c.host.control-plane.oxide.internal
    name: _crucible._tcp.a0e1f460-0a7c-4788-b4a2-3ecc2ac909ad (records: 1)
        SRV  port 32345 a0e1f460-0a7c-4788-b4a2-3ecc2ac909ad.host.control-plane.oxide.internal
    name: _crucible._tcp.a3cb63c4-0044-4705-a96f-87d83b09ea5d (records: 1)
        SRV  port 32345 a3cb63c4-0044-4705-a96f-87d83b09ea5d.host.control-plane.oxide.internal
    name: _crucible._tcp.b16765f7-de2f-44c2-b6ae-4fb60f5c4d53 (records: 1)
        SRV  port 32345 b16765f7-de2f-44c2-b6ae-4fb60f5c4d53.host.control-plane.oxide.internal
    name: _crucible._tcp.b26d1945-d24b-41f9-b80f-77af3638941a (records: 1)
        SRV  port 32345 b26d1945-d24b-41f9-b80f-77af3638941a.host.control-plane.oxide.internal
    name: _crucible._tcp.b67c6455-b415-4404-b997-588d39ae4a6c (records: 1)
        SRV  port 32345 b67c6455-b415-4404-b997-588d39ae4a6c.host.control-plane.oxide.internal
    name: _crucible._tcp.c673bafa-fcd2-40c4-a7f9-20906391ba42 (records: 1)
        SRV  port 32345 c673bafa-fcd2-40c4-a7f9-20906391ba42.host.control-plane.oxide.internal
    name: _crucible._tcp.f1497d76-dc9d-40ac-9670-3b17e1a64b84 (records: 1)
        SRV  port 32345 f1497d76-dc9d-40ac-9670-3b17e1a64b84.host.control-plane.oxide.internal
    name: _dendrite._tcp                                     (records: 2)
        SRV  port 12224 dendrite-09301471-54be-465e-a674-45da58c0c59b.host.control-plane.oxide.internal
        SRV  port 12224 dendrite-a21d85f9-ff13-4800-9e86-5eb7b9e62eb5.host.control-plane.oxide.internal
*   name: _external-dns._tcp                                 (records: 2 -> 1)
-       SRV  port  5353 5c88569a-d13a-483c-b276-90730292d35b.host.control-plane.oxide.internal
-       SRV  port  5353 8ea2a399-24f4-44c1-a31c-9b83750de6c1.host.control-plane.oxide.internal
+       SRV  port  5353 8ea2a399-24f4-44c1-a31c-9b83750de6c1.host.control-plane.oxide.internal
    name: _internal-ntp._tcp                                 (records: 1)
        SRV  port   123 d2d54e36-5a56-491e-8910-8a2cbe10cce9.host.control-plane.oxide.internal
    name: _mgd._tcp                                          (records: 2)
        SRV  port  4676 dendrite-09301471-54be-465e-a674-45da58c0c59b.host.control-plane.oxide.internal
        SRV  port  4676 dendrite-a21d85f9-ff13-4800-9e86-5eb7b9e62eb5.host.control-plane.oxide.internal
    name: _mgs._tcp                                          (records: 2)
        SRV  port 12225 dendrite-09301471-54be-465e-a674-45da58c0c59b.host.control-plane.oxide.internal
        SRV  port 12225 dendrite-a21d85f9-ff13-4800-9e86-5eb7b9e62eb5.host.control-plane.oxide.internal
    name: _nameservice._tcp                                  (records: 3)
        SRV  port  5353 0c752586-eb3b-4857-aa35-85a1206b6efb.host.control-plane.oxide.internal
        SRV  port  5353 23d6123b-1b33-4ea9-8eb0-62607cc29679.host.control-plane.oxide.internal
        SRV  port  5353 c3e14e81-0d5a-4461-80f5-1144ccdf4da5.host.control-plane.oxide.internal
    name: _nexus._tcp                                        (records: 3)
        SRV  port 12221 7b9677c8-0daf-4ab1-bc1d-9d655b964b33.host.control-plane.oxide.internal
        SRV  port 12221 b8da49f4-5b11-4773-be0c-8cbae8f31400.host.control-plane.oxide.internal
        SRV  port 12221 bdf0578f-d601-44e9-9318-5b5083d3e57d.host.control-plane.oxide.internal
    name: _oximeter._tcp                                     (records: 1)
        SRV  port 12223 f4806e03-9ecc-49ef-8ee0-ff7aee4dab23.host.control-plane.oxide.internal
    name: a0e1f460-0a7c-4788-b4a2-3ecc2ac909ad.host          (records: 1)
        AAAA fd00:1122:3344:103::c
    name: a3cb63c4-0044-4705-a96f-87d83b09ea5d.host          (records: 1)
        AAAA fd00:1122:3344:102::b
    name: b16765f7-de2f-44c2-b6ae-4fb60f5c4d53.host          (records: 1)
        AAAA fd00:1122:3344:103::b
    name: b26d1945-d24b-41f9-b80f-77af3638941a.host          (records: 1)
        AAAA fd00:1122:3344:101::8
    name: b67c6455-b415-4404-b997-588d39ae4a6c.host          (records: 1)
        AAAA fd00:1122:3344:101::b
    name: b8da49f4-5b11-4773-be0c-8cbae8f31400.host          (records: 1)
        AAAA fd00:1122:3344:101::6
    name: bdf0578f-d601-44e9-9318-5b5083d3e57d.host          (records: 1)
        AAAA fd00:1122:3344:103::5
    name: boundary-ntp                                       (records: 2)
        AAAA fd00:1122:3344:101::d
        AAAA fd00:1122:3344:102::d
    name: c3e14e81-0d5a-4461-80f5-1144ccdf4da5.host          (records: 1)
        AAAA fd00:1122:3344:1::1
    name: c673bafa-fcd2-40c4-a7f9-20906391ba42.host          (records: 1)
        AAAA fd00:1122:3344:102::8
    name: d2d54e36-5a56-491e-8910-8a2cbe10cce9.host          (records: 1)
        AAAA fd00:1122:3344:103::d
    name: dendrite-09301471-54be-465e-a674-45da58c0c59b.host (records: 1)
        AAAA fd00:1122:3344:103::2
    name: dendrite-a21d85f9-ff13-4800-9e86-5eb7b9e62eb5.host (records: 1)
        AAAA fd00:1122:3344:101::2
    name: f1497d76-dc9d-40ac-9670-3b17e1a64b84.host          (records: 1)
        AAAA fd00:1122:3344:103::9
    name: f4806e03-9ecc-49ef-8ee0-ff7aee4dab23.host          (records: 1)
        AAAA fd00:1122:3344:102::6
    name: f4bcdbb7-c64b-4473-9346-e5c6c411e76d.host          (records: 1)
        AAAA fd00:1122:3344:102::d

external DNS:
  DNS zone: "oxide.test" (unchanged)
    name: recovery.sys                                       (records: 3)
        A    198.51.100.23
        A    198.51.100.24
        A    198.51.100.22

Ran into #7302 again -- those datasets that changed, they changed only in disposition from "in-service" to "expunged". I imported the blueprint:

root@oxz_switch:~# omdb -w nexus blueprints import blueprint-expunged-external-dns.out
note: Nexus URL not specified.  Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:101::6]:12221
uploaded new blueprint edac838d-b454-48b7-b4e8-0bd433fecc6c

then we can see in the database that that's the only column that changed:

root@[fd00:1122:3344:102::3]:32221/omicron> select * from bp_omicron_dataset where id = '5073be4d-3c52-4948-b7ce-2a708c724d02';
              blueprint_id             |               sled_id                |                  id                  | disposition |               pool_id                | kind |                       zone_name                       |          ip           | port  | quota | reservation | compression
---------------------------------------+--------------------------------------+--------------------------------------+-------------+--------------------------------------+------+-------------------------------------------------------+-----------------------+-------+-------+-------------+--------------
  2e24acb1-21c8-469a-ace5-bec12f89a539 | a21d85f9-ff13-4800-9e86-5eb7b9e62eb5 | 5073be4d-3c52-4948-b7ce-2a708c724d02 | in_service  | 16b6a5fa-aa93-4781-aede-4393c243a1fc | zone | oxz_external_dns_5c88569a-d13a-483c-b276-90730292d35b | NULL                  | NULL  |  NULL |        NULL | off
  831679c9-26f8-4e3b-9873-e2522cfdc087 | a21d85f9-ff13-4800-9e86-5eb7b9e62eb5 | 5073be4d-3c52-4948-b7ce-2a708c724d02 | in_service  | 16b6a5fa-aa93-4781-aede-4393c243a1fc | zone | oxz_external_dns_5c88569a-d13a-483c-b276-90730292d35b | fd00:1122:3344:101::8 | 32345 |  NULL |        NULL | off
  edac838d-b454-48b7-b4e8-0bd433fecc6c | a21d85f9-ff13-4800-9e86-5eb7b9e62eb5 | 5073be4d-3c52-4948-b7ce-2a708c724d02 | expunged    | 16b6a5fa-aa93-4781-aede-4393c243a1fc | zone | oxz_external_dns_5c88569a-d13a-483c-b276-90730292d35b | NULL                  | NULL  |  NULL |        NULL | off
  f55291a6-4265-4178-a269-7df22eb0a2b7 | a21d85f9-ff13-4800-9e86-5eb7b9e62eb5 | 5073be4d-3c52-4948-b7ce-2a708c724d02 | in_service  | 16b6a5fa-aa93-4781-aede-4393c243a1fc | zone | oxz_external_dns_5c88569a-d13a-483c-b276-90730292d35b | NULL                  | NULL  |  NULL |        NULL | off
(4 rows)

I confirmed that they appear expunged in the blueprint JSON file that was saved above:

...
        "0712c964-bf3f-45e7-9e75-361652b53736": {
          "disposition": "expunged",
          "id": "0712c964-bf3f-45e7-9e75-361652b53736",
          "pool": "oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc",
          "kind": "external_dns",
          "address": null,
          "quota": null,
          "reservation": null,
          "compression": {
            "type": "off"
          }
        },
...
        "5073be4d-3c52-4948-b7ce-2a708c724d02": {
          "disposition": "expunged",
          "id": "5073be4d-3c52-4948-b7ce-2a708c724d02",
          "pool": "oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc",
          "kind": "zone/oxz_external_dns_5c88569a-d13a-483c-b276-90730292d35b",
          "address": null,
          "quota": null,
          "reservation": null,
          "compression": {
            "type": "off"
          }
        },
...

Then I executed it:

root@oxz_switch:~# omdb -w nexus blueprints target set edac838d-b454-48b7-b4e8-0bd433fecc6c inherit
note: Nexus URL not specified.  Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:101::6]:12221
set target blueprint to edac838d-b454-48b7-b4e8-0bd433fecc6c
...
root@oxz_switch:~# omdb nexus background-tasks show blueprint_executor
note: Nexus URL not specified.  Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:101::6]:12221
task: "blueprint_executor"
  configured period: every 1m
  currently executing: no
  last completed activation: iter 20330, triggered by a dependent task completing
    started at 2025-01-03T03:17:32.415Z (14s ago) and ran for 14014ms
    target blueprint: edac838d-b454-48b7-b4e8-0bd433fecc6c
    execution:        enabled
    status:           completed (16 steps)
    error:            (none)

So that worked. The ledger on the affected sled was updated:

root@g0:~# ls -lrt /pool/int/*/config/omicron-datasets.json
-rw-r--r--   1 root     root        7746 Jan  3 03:17 /pool/int/28d71fca-496b-4b8c-ac69-6b14a14174e5/config/omicron-datasets.json
-rw-r--r--   1 root     root        7746 Jan  3 03:17 /pool/int/4ca557a1-5a52-49b3-bbf8-1a89b3134065/config/omicron-datasets.json

But the only change is that it went from generation 2 to 3 (as the diff output said it would):

$ diff a4x2datasets-g{2,3}-pretty.txt
2c2
<   "generation": 2,
---
>   "generation": 3,
$

and the expunged datasets are still there:

$ jq '.datasets["0712c964-bf3f-45e7-9e75-361652b53736"]' < a4x2datasets-g3.txt 
{
  "id": "0712c964-bf3f-45e7-9e75-361652b53736",
  "name": {
    "pool_name": "oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc",
    "kind": "external_dns"
  },
  "compression": {
    "type": "off"
  },
  "quota": null,
  "reservation": null
}

[davepacheco]

This sounds superficially similar to #6066 but it's distinct. Here we're talking about kicking off a process of removing datasets from the blueprint, which will cause them to be removed from the sled, which after #6066 will eventually cause them to be removed from the dataset table.

[davepacheco]

Datasets ledger (sled g0, generation 3): a4x2datasets-g3.txt