failure trying to expunge two external IP zones

[davepacheco]

Continuing from #7304, where I had a freshly installed a4x2 system and expunged an external DNS zone: I then generated a new blueprint (which added a new zone to get the count back up), executed it (which worked), and then tried to expunge it, but it failed with:

error: adding omicron zone external IP: associating Omicron zone 5c88569a-d13a-483c-b276-90730292d35b with Floating(OmicronZoneExternalFloatingIp { id: 81554757-e8aa-4df1-8ba3-d2ae1121e9fb (external_ip), ip: 198.51.100.21 }) failed due to duplicates: duplicate entry: OmicronZoneExternalIpEntry { zone_id: 5c88569a-d13a-483c-b276-90730292d35b (service), ip: Floating(OmicronZoneExternalFloatingIp { id: 81554757-e8aa-4df1-8ba3-d2ae1121e9fb (external_ip), ip: 198.51.100.21 }) } conflicts with existing: [OmicronZoneExternalIpEntry { zone_id: 13f9e4ae-4f94-4718-af29-4a5da01bac2f (service), ip: Floating(OmicronZoneExternalFloatingIp { id: 293d7d74-bf8e-4199-b1b3-5a64b283a63c (external_ip), ip: 198.51.100.21 }) }]

---

Here are the full steps:

root@oxz_switch:~# omdb -w nexus blueprints regenerate
note: Nexus URL not specified.  Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:101::6]:12221
generated new blueprint 4d07b7da-112f-4332-929e-4f2e483d32cc

root@oxz_switch:~# omdb -w nexus blueprints diff current 4d07b7da-112f-4332-929e-4f2e483d32cc
note: Nexus URL not specified.  Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:101::6]:12221
from: blueprint edac838d-b454-48b7-b4e8-0bd433fecc6c
to:   blueprint 4d07b7da-112f-4332-929e-4f2e483d32cc

 UNCHANGED SLEDS:

  sled 09301471-54be-465e-a674-45da58c0c59b (active):

    physical disks at generation 2:
    -------------------------------------------------------------
    vendor             model                serial
    -------------------------------------------------------------
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g3_0
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g3_1
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g3_2
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g3_3
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g3_4


    datasets at generation 2:
    -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    dataset name                                                                                                   dataset uuid                           quota     reservation   compression
    -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/cockroachdb                                                     355ddd2b-b336-45c6-903f-ecdc99b26289   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crucible                                                              36b554b7-d442-4c58-96fe-71f88d9145e6   none      none          off
    oxp_29f71bd8-8d07-44f9-a587-8dd6552d6b3d/crucible                                                              53b65adf-99c3-472b-849c-9385e61c7f80   none      none          off
    oxp_639dcdf6-d036-4e59-94d8-9a05706f2a6e/crucible                                                              ee41d385-8bc3-4932-971a-2f2b1b3461a1   none      none          off
    oxp_b5c23cec-84ac-4810-9077-3f960af72fcd/crucible                                                              ab04e945-3a16-4a33-8e8b-a87b7d51df6f   none      none          off
    oxp_b9dc2f06-bf04-463c-8ffe-79d8357d852b/crucible                                                              9f46899f-ee8c-46e1-b6a3-99f74974379f   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/clickhouse                                                      ff1f90df-76b9-46a1-b2aa-c960db481817   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/external_dns                                                    5c81c9ca-43b3-43a2-ae34-2b2d09544492   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/internal_dns                                                    c2df2f3e-9b96-48ec-bcf2-23ffccbc3c05   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/zone                                                            0e6b946e-f23d-402d-87d0-6531ef26c24a   none      none          off
    oxp_29f71bd8-8d07-44f9-a587-8dd6552d6b3d/crypt/zone                                                            48566631-3eba-4cd5-b4e9-48bb9657d8de   none      none          off
    oxp_639dcdf6-d036-4e59-94d8-9a05706f2a6e/crypt/zone                                                            644e63c6-ff9b-4ac0-a932-b1063b2a0a02   none      none          off
    oxp_b5c23cec-84ac-4810-9077-3f960af72fcd/crypt/zone                                                            c04b17c8-5626-409a-8eb1-f0b797fe9788   none      none          off
    oxp_b9dc2f06-bf04-463c-8ffe-79d8357d852b/crypt/zone                                                            f760e7e6-089d-4743-a339-1e0d882a40aa   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/zone/oxz_clickhouse_0208b07b-4ccd-4e8b-b334-1ec8ecc4bae7        d343db99-4aa1-4f6c-aa53-7be34e509d36   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/zone/oxz_cockroachdb_67989451-63b3-48c2-aef8-c6e511fb122f       9c747a15-0ce0-41bd-bc2f-d7ee792a3592   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/zone/oxz_crucible_79da9a6d-9db8-4fd9-b2dc-ba0d2fed0bdd          6ec06bb2-9fd0-45d0-a35b-d6cbf894537d   none      none          off
    oxp_639dcdf6-d036-4e59-94d8-9a05706f2a6e/crypt/zone/oxz_crucible_7c5ae2a6-95a7-4cd9-8631-68477b614d97          a4dd3c84-8616-4182-8214-c9677a57318e   none      none          off
    oxp_29f71bd8-8d07-44f9-a587-8dd6552d6b3d/crypt/zone/oxz_crucible_a0e1f460-0a7c-4788-b4a2-3ecc2ac909ad          a93a6377-6d78-4c34-b220-f2c4926e4795   none      none          off
    oxp_b5c23cec-84ac-4810-9077-3f960af72fcd/crypt/zone/oxz_crucible_b16765f7-de2f-44c2-b6ae-4fb60f5c4d53          660a8beb-683a-4c00-82e9-b8f19e9f86fe   none      none          off
    oxp_b9dc2f06-bf04-463c-8ffe-79d8357d852b/crypt/zone/oxz_crucible_f1497d76-dc9d-40ac-9670-3b17e1a64b84          37896114-bb11-402a-a1a2-f0fe800d794d   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/zone/oxz_crucible_pantry_31170c27-6736-418a-b496-b740a5668e4c   7fc29a07-ce43-41d9-9a1e-1bfd3be35a46   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/zone/oxz_external_dns_8ea2a399-24f4-44c1-a31c-9b83750de6c1      defada08-7725-4d45-acec-2fe308bae2e7   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/zone/oxz_internal_dns_0c752586-eb3b-4857-aa35-85a1206b6efb      28d664b6-8003-489b-b824-262025837f56   none      none          off
    oxp_29f71bd8-8d07-44f9-a587-8dd6552d6b3d/crypt/zone/oxz_nexus_bdf0578f-d601-44e9-9318-5b5083d3e57d             30179c95-0623-468d-8b25-b78499725884   none      none          off
    oxp_639dcdf6-d036-4e59-94d8-9a05706f2a6e/crypt/zone/oxz_ntp_d2d54e36-5a56-491e-8910-8a2cbe10cce9               c6649371-7506-4fa3-ad41-f4d7666bc1a3   none      none          off
    oxp_13a7b934-412b-4ea5-8391-d52998c2c6fe/crypt/debug                                                           50e80495-eb69-4716-8fa4-9f697b454498   100 GiB   none          gzip-9
    oxp_29f71bd8-8d07-44f9-a587-8dd6552d6b3d/crypt/debug                                                           f8a339ec-c736-44cd-b562-d845da329ece   100 GiB   none          gzip-9
    oxp_639dcdf6-d036-4e59-94d8-9a05706f2a6e/crypt/debug                                                           7f64c446-bc62-4885-8d74-7a40906c6f20   100 GiB   none          gzip-9
    oxp_b5c23cec-84ac-4810-9077-3f960af72fcd/crypt/debug                                                           59d97482-de4f-4f5a-9961-d2b00126a6a6   100 GiB   none          gzip-9
    oxp_b9dc2f06-bf04-463c-8ffe-79d8357d852b/crypt/debug                                                           914cc29f-b2cd-4128-9393-18e8d6715730   100 GiB   none          gzip-9


    omicron zones at generation 5:
    --------------------------------------------------------------------------------------------
    zone type         zone id                                disposition   underlay IP
    --------------------------------------------------------------------------------------------
    clickhouse        0208b07b-4ccd-4e8b-b334-1ec8ecc4bae7   in service    fd00:1122:3344:103::6
    cockroach_db      67989451-63b3-48c2-aef8-c6e511fb122f   in service    fd00:1122:3344:103::3
    crucible          79da9a6d-9db8-4fd9-b2dc-ba0d2fed0bdd   in service    fd00:1122:3344:103::8
    crucible          7c5ae2a6-95a7-4cd9-8631-68477b614d97   in service    fd00:1122:3344:103::a
    crucible          a0e1f460-0a7c-4788-b4a2-3ecc2ac909ad   in service    fd00:1122:3344:103::c
    crucible          b16765f7-de2f-44c2-b6ae-4fb60f5c4d53   in service    fd00:1122:3344:103::b
    crucible          f1497d76-dc9d-40ac-9670-3b17e1a64b84   in service    fd00:1122:3344:103::9
    crucible_pantry   31170c27-6736-418a-b496-b740a5668e4c   in service    fd00:1122:3344:103::7
    external_dns      8ea2a399-24f4-44c1-a31c-9b83750de6c1   in service    fd00:1122:3344:103::4
    internal_dns      0c752586-eb3b-4857-aa35-85a1206b6efb   in service    fd00:1122:3344:3::1
    internal_ntp      d2d54e36-5a56-491e-8910-8a2cbe10cce9   in service    fd00:1122:3344:103::d
    nexus             bdf0578f-d601-44e9-9318-5b5083d3e57d   in service    fd00:1122:3344:103::5


  sled f0b13ea8-c304-4a03-ab7c-da1fb2d0c612 (active):

    physical disks at generation 2:
    -------------------------------------------------------------
    vendor             model                serial
    -------------------------------------------------------------
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g1_0
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g1_1
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g1_2
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g1_3
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g1_4


    datasets at generation 2:
    -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    dataset name                                                                                                   dataset uuid                           quota     reservation   compression
    -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    oxp_89021d00-c808-49ec-ada3-427646a63b46/crypt/cockroachdb                                                     2cfac653-9dba-492a-b4e5-c8efe5a1e1b9   none      none          off
    oxp_9d453deb-a057-438f-b555-627c501d9257/crypt/cockroachdb                                                     12feda67-cf20-4f01-8094-69a956526990   none      none          off
    oxp_60155e33-c180-4490-9f28-0cb4003ede47/crucible                                                              d597754a-6fe3-4d18-b8be-8dc0a269628d   none      none          off
    oxp_89021d00-c808-49ec-ada3-427646a63b46/crucible                                                              beee4d0a-44b9-4298-9014-da523dd9eaa7   none      none          off
    oxp_9d453deb-a057-438f-b555-627c501d9257/crucible                                                              644161d1-8a5f-4609-ae2d-21c4ae54fdd4   none      none          off
    oxp_c1b87796-8265-4c6b-b41a-be7de82934f6/crucible                                                              5d8edd26-0cd7-434e-af7b-c50a319cd1ea   none      none          off
    oxp_f63bc936-be56-4092-a2cb-6a8508f8a23d/crucible                                                              cdf87aea-8f8f-4e49-b921-db848d6e25ef   none      none          off
    oxp_89021d00-c808-49ec-ada3-427646a63b46/crypt/internal_dns                                                    b1e1c105-638d-4401-a5af-534f219a11f4   none      none          off
    oxp_60155e33-c180-4490-9f28-0cb4003ede47/crypt/zone                                                            d621f634-68d8-4741-85bd-53e82095c094   none      none          off
    oxp_89021d00-c808-49ec-ada3-427646a63b46/crypt/zone                                                            7c00d8d6-86c6-4446-a90f-24c62ac741c4   none      none          off
    oxp_9d453deb-a057-438f-b555-627c501d9257/crypt/zone                                                            b5fcfc17-2954-484a-bbf8-8c51b97a7aa3   none      none          off
    oxp_c1b87796-8265-4c6b-b41a-be7de82934f6/crypt/zone                                                            aacb001f-2c30-427d-96cf-10f553de5446   none      none          off
    oxp_f63bc936-be56-4092-a2cb-6a8508f8a23d/crypt/zone                                                            662d12c6-b68b-4aac-b9be-9c999bf5b05e   none      none          off
    oxp_89021d00-c808-49ec-ada3-427646a63b46/crypt/zone/oxz_cockroachdb_2c00cc83-0211-4662-9f95-ecbf1c8fc26b       1478643a-989f-43e3-bc24-4ab2c9f40d15   none      none          off
    oxp_9d453deb-a057-438f-b555-627c501d9257/crypt/zone/oxz_cockroachdb_2eb1e0f2-72a3-48bc-a76e-cb5b8838938f       905d06de-7198-41a7-a696-35d1c66bc882   none      none          off
    oxp_60155e33-c180-4490-9f28-0cb4003ede47/crypt/zone/oxz_crucible_0a1d8035-c373-4de7-8d70-ebe0a4b16758          ac0ca38c-128a-42a5-b87a-670d7b807ed4   none      none          off
    oxp_f63bc936-be56-4092-a2cb-6a8508f8a23d/crypt/zone/oxz_crucible_476d5bde-b360-4c49-8093-5d56be92c63f          d642b0d0-1f61-4ad0-9840-98464d3b4ba2   none      none          off
    oxp_9d453deb-a057-438f-b555-627c501d9257/crypt/zone/oxz_crucible_74d4206f-6bdb-4c47-b01c-9e2a876d622e          19a81e6a-938a-4579-bf95-e1105fe1b0a5   none      none          off
    oxp_c1b87796-8265-4c6b-b41a-be7de82934f6/crypt/zone/oxz_crucible_a3cb63c4-0044-4705-a96f-87d83b09ea5d          7df8ef54-cb4a-4676-b163-7d0418043127   none      none          off
    oxp_89021d00-c808-49ec-ada3-427646a63b46/crypt/zone/oxz_crucible_c673bafa-fcd2-40c4-a7f9-20906391ba42          88a432f9-1f5f-4799-a303-ab5806f73f3f   none      none          off
    oxp_60155e33-c180-4490-9f28-0cb4003ede47/crypt/zone/oxz_crucible_pantry_47d0ca61-9861-4d61-948c-890c5f4ac5d5   3cbb1ed7-3290-428d-bebb-f54297c09909   none      none          off
    oxp_89021d00-c808-49ec-ada3-427646a63b46/crypt/zone/oxz_internal_dns_23d6123b-1b33-4ea9-8eb0-62607cc29679      74b11e9a-bf6a-458d-a2c7-740576494cc9   none      none          off
    oxp_89021d00-c808-49ec-ada3-427646a63b46/crypt/zone/oxz_nexus_7b9677c8-0daf-4ab1-bc1d-9d655b964b33             0d21ae90-6ee4-4422-9bc6-f336fd4b9ff5   none      none          off
    oxp_89021d00-c808-49ec-ada3-427646a63b46/crypt/zone/oxz_ntp_f4bcdbb7-c64b-4473-9346-e5c6c411e76d               7a9e077f-5512-4f8c-b780-36c326aa0ad5   none      none          off
    oxp_60155e33-c180-4490-9f28-0cb4003ede47/crypt/zone/oxz_oximeter_f4806e03-9ecc-49ef-8ee0-ff7aee4dab23          e48d0b5a-2a9e-4794-b45c-858bf8ebf833   none      none          off
    oxp_60155e33-c180-4490-9f28-0cb4003ede47/crypt/debug                                                           5f96f85f-15df-4b76-bc99-7cdd1e07ff16   100 GiB   none          gzip-9
    oxp_89021d00-c808-49ec-ada3-427646a63b46/crypt/debug                                                           5aa39aed-b571-4c7d-b8ae-040c64cfe25d   100 GiB   none          gzip-9
    oxp_9d453deb-a057-438f-b555-627c501d9257/crypt/debug                                                           45fcfb99-bc89-4c87-b5c6-ee0320d3420c   100 GiB   none          gzip-9
    oxp_c1b87796-8265-4c6b-b41a-be7de82934f6/crypt/debug                                                           d8ffad03-2ea6-432f-beb2-a5b2fc4bf885   100 GiB   none          gzip-9
    oxp_f63bc936-be56-4092-a2cb-6a8508f8a23d/crypt/debug                                                           9156026c-38ed-4ee5-b49f-b4682b7a2702   100 GiB   none          gzip-9


    omicron zones at generation 5:
    --------------------------------------------------------------------------------------------
    zone type         zone id                                disposition   underlay IP
    --------------------------------------------------------------------------------------------
    boundary_ntp      f4bcdbb7-c64b-4473-9346-e5c6c411e76d   in service    fd00:1122:3344:102::d
    cockroach_db      2c00cc83-0211-4662-9f95-ecbf1c8fc26b   in service    fd00:1122:3344:102::3
    cockroach_db      2eb1e0f2-72a3-48bc-a76e-cb5b8838938f   in service    fd00:1122:3344:102::4
    crucible          0a1d8035-c373-4de7-8d70-ebe0a4b16758   in service    fd00:1122:3344:102::c
    crucible          476d5bde-b360-4c49-8093-5d56be92c63f   in service    fd00:1122:3344:102::a
    crucible          74d4206f-6bdb-4c47-b01c-9e2a876d622e   in service    fd00:1122:3344:102::9
    crucible          a3cb63c4-0044-4705-a96f-87d83b09ea5d   in service    fd00:1122:3344:102::b
    crucible          c673bafa-fcd2-40c4-a7f9-20906391ba42   in service    fd00:1122:3344:102::8
    crucible_pantry   47d0ca61-9861-4d61-948c-890c5f4ac5d5   in service    fd00:1122:3344:102::7
    internal_dns      23d6123b-1b33-4ea9-8eb0-62607cc29679   in service    fd00:1122:3344:2::1
    nexus             7b9677c8-0daf-4ab1-bc1d-9d655b964b33   in service    fd00:1122:3344:102::5
    oximeter          f4806e03-9ecc-49ef-8ee0-ff7aee4dab23   in service    fd00:1122:3344:102::6


 MODIFIED SLEDS:

  sled a21d85f9-ff13-4800-9e86-5eb7b9e62eb5 (active):

    physical disks at generation 2:
    -------------------------------------------------------------
    vendor             model                serial
    -------------------------------------------------------------
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g0_0
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g0_1
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g0_2
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g0_3
    synthetic-vendor   synthetic-model-U2   synthetic-serial-g0_4


    datasets generation 3 -> 4:
    -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    dataset name                                                                                                   dataset uuid                           quota     reservation   compression
    -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc/crypt/cockroachdb                                                     119fec8f-785e-4833-90f2-04311332041f   none      none          off
    oxp_fb97a84c-3ac3-494c-9c76-26d6357aebf6/crypt/cockroachdb                                                     1f8b0e1a-0545-44f1-9366-6aaf29a0fb83   none      none          off
    oxp_15b53b30-72cf-4edb-a7c4-325ee3f7c679/crucible                                                              aff65822-a39d-4b21-9b1e-d94ba1688057   none      none          off
    oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc/crucible                                                              76cc81e2-f740-4cb9-b495-10ae9cbb8cbd   none      none          off
    oxp_44152dd0-a9bc-40a0-a64f-d375b950e5bf/crucible                                                              305cdc80-78d1-4e19-8b68-ba68be9e7b1a   none      none          off
    oxp_abbce0e4-bda7-4853-b89a-37721aa638fb/crucible                                                              a3c3f7cc-b4e4-439b-9eb6-e97ad14b58fe   none      none          off
    oxp_fb97a84c-3ac3-494c-9c76-26d6357aebf6/crucible                                                              1c2913e3-adce-4a64-8c46-b47c8018b291   none      none          off
    oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc/crypt/external_dns                                                    0712c964-bf3f-45e7-9e75-361652b53736   none      none          off
    oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc/crypt/internal_dns                                                    03492708-b9f6-4c40-a0fb-965734468a66   none      none          off
    oxp_15b53b30-72cf-4edb-a7c4-325ee3f7c679/crypt/zone                                                            cac4df64-07ac-4266-9c73-822fb620ff9f   none      none          off
    oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc/crypt/zone                                                            da1424d7-fe97-4f3e-84a1-4842ab4dee99   none      none          off
    oxp_44152dd0-a9bc-40a0-a64f-d375b950e5bf/crypt/zone                                                            97c712a4-cb43-4e83-960b-9382964bae74   none      none          off
    oxp_abbce0e4-bda7-4853-b89a-37721aa638fb/crypt/zone                                                            7fe4e11d-6cfb-4e35-973c-fe366a8cd9a1   none      none          off
    oxp_fb97a84c-3ac3-494c-9c76-26d6357aebf6/crypt/zone                                                            6f7a280d-2aca-4e64-9ecd-c9432c9a80f4   none      none          off
    oxp_fb97a84c-3ac3-494c-9c76-26d6357aebf6/crypt/zone/oxz_cockroachdb_792489e0-9d08-4d21-9caa-0440b12d6ade       2d4851d8-0d00-45a3-b351-9353339e3f17   none      none          off
    oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc/crypt/zone/oxz_cockroachdb_83e82c9d-e587-4a68-a18d-128712e2741f       c4351281-bd7d-41bd-b5a1-e9118dac2f16   none      none          off
    oxp_15b53b30-72cf-4edb-a7c4-325ee3f7c679/crypt/zone/oxz_crucible_049d9f96-6e06-43a0-a924-35146efd7b8c          801a8141-9e83-4cc0-9428-fb1db210657d   none      none          off
    oxp_fb97a84c-3ac3-494c-9c76-26d6357aebf6/crypt/zone/oxz_crucible_6d03d1db-c605-4a00-9717-3bf867969ad9          1728ac4b-67aa-451f-b674-ad9b60dd5fdb   none      none          off
    oxp_44152dd0-a9bc-40a0-a64f-d375b950e5bf/crypt/zone/oxz_crucible_8ab366b1-b6f7-4467-8468-36745617618c          05146ed5-3246-4591-adec-e8a7941b8e83   none      none          off
    oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc/crypt/zone/oxz_crucible_b26d1945-d24b-41f9-b80f-77af3638941a          3d967991-7f45-41f6-9cb5-827b90645547   none      none          off
    oxp_abbce0e4-bda7-4853-b89a-37721aa638fb/crypt/zone/oxz_crucible_b67c6455-b415-4404-b997-588d39ae4a6c          bd097130-13a7-4478-9a0e-e6eadeefabf4   none      none          off
    oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc/crypt/zone/oxz_crucible_pantry_8b81d199-146c-4ccf-aa23-f57140a6b582   d4ce5cf6-96e8-44f0-975f-b5601f099c09   none      none          off
    oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc/crypt/zone/oxz_external_dns_5c88569a-d13a-483c-b276-90730292d35b      5073be4d-3c52-4948-b7ce-2a708c724d02   none      none          off
    oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc/crypt/zone/oxz_internal_dns_c3e14e81-0d5a-4461-80f5-1144ccdf4da5      04128921-a9b4-40f1-bd7a-17e89c22fac9   none      none          off
    oxp_fb97a84c-3ac3-494c-9c76-26d6357aebf6/crypt/zone/oxz_nexus_b8da49f4-5b11-4773-be0c-8cbae8f31400             0ab36247-5490-4fe0-b3ab-cbd5d9170efc   none      none          off
    oxp_15b53b30-72cf-4edb-a7c4-325ee3f7c679/crypt/zone/oxz_ntp_2b3c2cf8-bf97-4a7c-9327-712f1d589c7b               43a80037-e23f-44be-84eb-bb30bd1f539e   none      none          off
    oxp_15b53b30-72cf-4edb-a7c4-325ee3f7c679/crypt/debug                                                           6f610524-4329-4634-adab-ffbd6f65a653   100 GiB   none          gzip-9
    oxp_16b6a5fa-aa93-4781-aede-4393c243a1fc/crypt/debug                                                           eb609f8e-0bd4-4f07-8c8c-8ad72bc3b8e6   100 GiB   none          gzip-9
    oxp_44152dd0-a9bc-40a0-a64f-d375b950e5bf/crypt/debug                                                           dd9e2a12-9e1a-4c01-9090-2318a4ca5bf6   100 GiB   none          gzip-9
    oxp_abbce0e4-bda7-4853-b89a-37721aa638fb/crypt/debug                                                           87c3fca0-606e-47ff-81cb-0917fe240f85   100 GiB   none          gzip-9
    oxp_fb97a84c-3ac3-494c-9c76-26d6357aebf6/crypt/debug                                                           d60fcccf-fc65-404c-a2d9-0c780a7e1ea8   100 GiB   none          gzip-9
+   oxp_15b53b30-72cf-4edb-a7c4-325ee3f7c679/crypt/external_dns                                                    e517594e-729d-40dc-9b05-75f0866cc720   none      none          off
+   oxp_15b53b30-72cf-4edb-a7c4-325ee3f7c679/crypt/zone/oxz_external_dns_13f9e4ae-4f94-4718-af29-4a5da01bac2f      9b229bcf-1e70-4c34-a34f-20c022968645   none      none          off


    omicron zones generation 6 -> 7:
    ---------------------------------------------------------------------------------------------
    zone type         zone id                                disposition   underlay IP
    ---------------------------------------------------------------------------------------------
    boundary_ntp      2b3c2cf8-bf97-4a7c-9327-712f1d589c7b   in service    fd00:1122:3344:101::d
    cockroach_db      792489e0-9d08-4d21-9caa-0440b12d6ade   in service    fd00:1122:3344:101::4
    cockroach_db      83e82c9d-e587-4a68-a18d-128712e2741f   in service    fd00:1122:3344:101::3
    crucible          049d9f96-6e06-43a0-a924-35146efd7b8c   in service    fd00:1122:3344:101::a
    crucible          6d03d1db-c605-4a00-9717-3bf867969ad9   in service    fd00:1122:3344:101::9
    crucible          8ab366b1-b6f7-4467-8468-36745617618c   in service    fd00:1122:3344:101::c
    crucible          b26d1945-d24b-41f9-b80f-77af3638941a   in service    fd00:1122:3344:101::8
    crucible          b67c6455-b415-4404-b997-588d39ae4a6c   in service    fd00:1122:3344:101::b
    crucible_pantry   8b81d199-146c-4ccf-aa23-f57140a6b582   in service    fd00:1122:3344:101::7
    external_dns      5c88569a-d13a-483c-b276-90730292d35b   expunged      fd00:1122:3344:101::5
    internal_dns      c3e14e81-0d5a-4461-80f5-1144ccdf4da5   in service    fd00:1122:3344:1::1
    nexus             b8da49f4-5b11-4773-be0c-8cbae8f31400   in service    fd00:1122:3344:101::6
+   external_dns      13f9e4ae-4f94-4718-af29-4a5da01bac2f   in service    fd00:1122:3344:101::21


 COCKROACHDB SETTINGS:
    state fingerprint:::::::::::::::::   d4d87aa2ad877a4cc2fddd0573952362739110de (unchanged)
    cluster.preserve_downgrade_option:   "22.1" (unchanged)

 METADATA:
*   internal DNS version:   1 -> 2
    external DNS version:   2 (unchanged)


root@oxz_switch:~# omdb -w nexus blueprints target set 4d07b7da-112f-4332-929e-4f2e483d32cc inherit
note: Nexus URL not specified.  Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:101::6]:12221
set target blueprint to 4d07b7da-112f-4332-929e-4f2e483d32cc


root@oxz_switch:~# omdb nexus background-tasks show blueprint_executor
note: Nexus URL not specified.  Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:101::6]:12221
task: "blueprint_executor"
  configured period: every 1m
  currently executing: iter 20369, triggered by a periodic timer firing
    started at 2025-01-03T03:55:48.268Z, running for 4327ms
  last completed activation: iter 20368, triggered by a dependent task completing
    started at 2025-01-03T03:55:25.601Z (26s ago) and ran for 19482ms
    target blueprint: 4d07b7da-112f-4332-929e-4f2e483d32cc
    execution:        enabled
    status:           completed (16 steps)
    error:            (none)

root@oxz_switch:~# omdb db reconfigurator-save reconfigurator-2.out
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:102::3]:32221,[fd00:1122:3344:102::4]:32221,[fd00:1122:3344:103::3]:32221,[fd00:1122:3344:101::4]:32221,[fd00:1122:3344:101::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (117.0.0)
assembling reconfigurator state ... done
wrote reconfigurator-2.out

root@oxz_switch:~# ./reconfigurator-cli
generated RNG seed: belatedly-encouraged-sidewinder

〉file-contents reconfigurator-2.out
sled: 09301471-54be-465e-a674-45da58c0c59b (subnet: fd00:1122:3344:103::/64, zpools: 5)
sled: a21d85f9-ff13-4800-9e86-5eb7b9e62eb5 (subnet: fd00:1122:3344:101::/64, zpools: 5)
sled: f0b13ea8-c304-4a03-ab7c-da1fb2d0c612 (subnet: fd00:1122:3344:102::/64, zpools: 5)
collection: 7253276d-595a-437a-b611-3b98c9ed107a (errors: 0, completed at: 2025-01-03T03:56:02.317Z)
collection: 673127c7-f791-4498-a345-bc307bc914bb (errors: 0, completed at: 2025-01-03T03:56:07.738Z)
collection: bf2dc06d-2ec3-4938-a3a8-01f66cc5710f (errors: 0, completed at: 2025-01-03T03:56:30.796Z)
collection: de0ce350-7ab7-4b57-8104-d085b82d00ac (errors: 0, completed at: 2025-01-03T03:56:36.233Z)
blueprint:  2e24acb1-21c8-469a-ace5-bec12f89a539 (created at: 2024-12-21 19:15:33.575638 UTC)
blueprint:  4d07b7da-112f-4332-929e-4f2e483d32cc (created at: 2025-01-03 03:54:27.496120 UTC)
blueprint:  831679c9-26f8-4e3b-9873-e2522cfdc087 (created at: 2024-12-20 00:26:36.176391 UTC)
blueprint:  edac838d-b454-48b7-b4e8-0bd433fecc6c (created at: 2025-01-03 03:09:24.436330 UTC)
blueprint:  f55291a6-4265-4178-a269-7df22eb0a2b7 (created at: 2025-01-02 18:58:07.719269 UTC)
internal DNS generations: [Generation(1), Generation(2), Generation(3)]
external DNS generations: [Generation(2)]
silo names: [Name("default-silo"), Name("recovery")]
external DNS zone names: oxide.test

〉load reconfigurator-2.out de0ce350-7ab7-4b57-8104-d085b82d00ac
loaded data from "reconfigurator-2.out"
result:
  system:
    using collection de0ce350-7ab7-4b57-8104-d085b82d00ac as source of sled inventory data
    loaded sleds: 09301471-54be-465e-a674-45da58c0c59b, a21d85f9-ff13-4800-9e86-5eb7b9e62eb5, f0b13ea8-c304-4a03-ab7c-da1fb2d0c612
    loaded collections: 7253276d-595a-437a-b611-3b98c9ed107a, 673127c7-f791-4498-a345-bc307bc914bb, bf2dc06d-2ec3-4938-a3a8-01f66cc5710f, de0ce350-7ab7-4b57-8104-d085b82d00ac
    loaded blueprints: 2e24acb1-21c8-469a-ace5-bec12f89a539, 4d07b7da-112f-4332-929e-4f2e483d32cc, 831679c9-26f8-4e3b-9873-e2522cfdc087, edac838d-b454-48b7-b4e8-0bd433fecc6c, f55291a6-4265-4178-a269-7df22eb0a2b7
    loaded service IP pool ranges: [V4(Ipv4Range { first: 198.51.100.20, last: 198.51.100.29 })]
    loaded internal DNS generations: (none)
    loaded external DNS generations: (none)
  config:
    configured external DNS zone name: oxide.test
    configured silo names: default-silo, recovery

〉blueprint-edit 4d07b7da-112f-4332-929e-4f2e483d32cc expunge-zone a21d85f9-ff13-4800-9e86-5eb7b9e62eb5 13f9e4ae-4f94-4718-af29-4a5da01bac2f
error: adding omicron zone external IP: associating Omicron zone 5c88569a-d13a-483c-b276-90730292d35b with Floating(OmicronZoneExternalFloatingIp { id: 81554757-e8aa-4df1-8ba3-d2ae1121e9fb (external_ip), ip: 198.51.100.21 }) failed due to duplicates: duplicate entry: OmicronZoneExternalIpEntry { zone_id: 5c88569a-d13a-483c-b276-90730292d35b (service), ip: Floating(OmicronZoneExternalFloatingIp { id: 81554757-e8aa-4df1-8ba3-d2ae1121e9fb (external_ip), ip: 198.51.100.21 }) } conflicts with existing: [OmicronZoneExternalIpEntry { zone_id: 13f9e4ae-4f94-4718-af29-4a5da01bac2f (service), ip: Floating(OmicronZoneExternalFloatingIp { id: 293d7d74-bf8e-4199-b1b3-5a64b283a63c (external_ip), ip: 198.51.100.21 }) }]