Configuration change leaves device in the inconsistent state (although it's not reflected anywhere)

[cmohorea]

Is there a limit for 10 device updates at a time?

Environment: 12 devices defined, 12 sdwan_attach_feature_device_template resources.
Steps:
1: change to a feature template common for all of them (in my case, sdwan_cisco_system_feature_template used in all of them)
2: terraform plan/apply reports changes to all 12 devices
3: after apply:
╷
│ Warning: Client Warning
│
│ with sdwan_attach_feature_device_template.LAB1_SITE7_LARGE_SITE_MAIN_R1,
│ on cnrail-lab1-branch-pod7.tf line 444, in resource "sdwan_attach_feature_device_template" "LAB1_SITE7_LARGE_SITE_MAIN_R1":
│ 444: resource "sdwan_attach_feature_device_template" "LAB1_SITE7_LARGE_SITE_MAIN_R1" {
│
│ No changes detected to trigger an attachment.
│
│ (and one more similar warning elsewhere)
╵

4: Now, intended config and real config on a device are in mismatch. Re-apply template (via "Change device values") in the GUI and it would show config differences (although it would not show that template is out-of-sync!). I'm not sure how to catch that in the real network other than re-pushing all templates to be 100% sure (terraform only reports warning for 1 device). From terraform perspective, in the meantime, all is good, "plan" shows that no changes needed

I noted in terraform log that resource changes for routers in question are planned:

  # sdwan_attach_feature_device_template.LAB1_SITE7_LARGE_SITE_MAIN_R1 will be updated in-place
  ~ resource "sdwan_attach_feature_device_template" "LAB1_SITE7_LARGE_SITE_MAIN_R1" {
        id      = "f2daf042-09aa-43b6-b053-752f1a440306"
      ~ version = 27 -> (known after apply)
        # (1 unchanged attribute hidden)
    }

  # sdwan_feature_device_template.LAB1_SITE7_LARGE_SITE_MAIN_R1 will be updated in-place
  ~ resource "sdwan_feature_device_template" "LAB1_SITE7_LARGE_SITE_MAIN_R1" {
      ~ general_templates  = [
          - {
              - id            = "ef2298e4-7628-4e7c-b95c-6eb875a59fbe" -> null
              - sub_templates = [
                  - {
                      - id   = "94dcc980-bc10-4078-af75-04654a9372f6" -> null
                      - type = "cisco_logging" -> null
                    },
                  - {
                      - id      = "f30a8713-128b-4107-893d-7147b0f62a21" -> null
                      - type    = "cisco_ntp" -> null
                      - version = 0 -> null
                    },
                ] -> null
              - type          = "cisco_system" -> null
              - version       = 2 -> null
            },
          + {
              + id            = "ef2298e4-7628-4e7c-b95c-6eb875a59fbe"
              + sub_templates = [
                  + {
                      + id   = "94dcc980-bc10-4078-af75-04654a9372f6"
                      + type = "cisco_logging"
                    },
                  + {
                      + id      = "f30a8713-128b-4107-893d-7147b0f62a21"
                      + type    = "cisco_ntp"
                      + version = 0
                    },
                ]
              + type          = "cisco_system"
              + version       = (known after apply)
            },
            # (10 unchanged elements hidden)
        ]
        id                 = "f2daf042-09aa-43b6-b053-752f1a440306"
        name               = "LAB1_SITE7_LARGE_SITE_MAIN_R1"
      ~ version            = 27 -> (known after apply)
        # (5 unchanged attributes hidden)
    }

but during update, only device template gets modified:

sdwan_feature_device_template.LAB1_SITE7_LARGE_SITE_MAIN_R1: Modifications complete after 0s [id=f2daf042-09aa-43b6-b053-752f1a440306]

and that's it + final message

and it's not a missed dependency. If I update another setting that is more local (e.g. only impacts all branch routers), then it works just fine:

sdwan_cisco_bfd_feature_template.SITE_BFD: Modifications complete after 2s [id=a03ea8a9-eeb7-4448-860b-34c59ef7c541]
...
sdwan_feature_device_template.LAB1_SITE7_LARGE_SITE_MAIN_R1: Modifying... [id=f2daf042-09aa-43b6-b053-752f1a440306]
sdwan_feature_device_template.LAB1_SITE7_LARGE_SITE_MAIN_R1: Modifications complete after 0s [id=f2daf042-09aa-43b6-b053-752f1a440306]
...
sdwan_attach_feature_device_template.LAB1_SITE7_LARGE_SITE_MAIN_R1: Modifying... [id=f2daf042-09aa-43b6-b053-752f1a440306]
...
sdwan_attach_feature_device_template.LAB1_SITE7_LARGE_SITE_MAIN_R1: Still modifying... [id=f2daf042-09aa-43b6-b053-752f1a440306, 10s elapsed]
...
sdwan_attach_feature_device_template.LAB1_SITE7_LARGE_SITE_MAIN_R1: Still modifying... [id=f2daf042-09aa-43b6-b053-752f1a440306, 20s elapsed]
...
sdwan_attach_feature_device_template.LAB1_SITE7_LARGE_SITE_MAIN_R1: Modifications complete after 27s [id=f2daf042-09aa-43b6-b053-752f1a440306]

resource definition, for the reference:

resource "sdwan_feature_device_template" "LAB1_SITE7_LARGE_SITE_MAIN_R1" {
  # id = "c2ef199f-9949-417c-a484-b3bc799bc601"
  name = "LAB1_SITE7_LARGE_SITE_MAIN_R1"
  description = "Lab Pod4 - Large Sites, Main Building - Primary Router"
  device_role = "sdwan-edge"
  device_type = "vedge-C8300-1N1S-4T2X"
  general_templates =   [
    {
      id = sdwan_cisco_system_feature_template.SYSTEM.id,
      version = sdwan_cisco_system_feature_template.SYSTEM.version,
      sub_templates = [
        {
          id = sdwan_cisco_logging_feature_template.Factory_Default_Cisco_Logging_Template.id,
          version = sdwan_cisco_logging_feature_template.Factory_Default_Cisco_Logging_Template.version,
          type = "cisco_logging",
        },
        {
          id = sdwan_cisco_ntp_feature_template.NTP.id,
          version = sdwan_cisco_ntp_feature_template.NTP.version,
          type = "cisco_ntp",
        },
      ],
      type = "cisco_system",
    },
....
  ]
}

resource "sdwan_attach_feature_device_template" "LAB1_SITE7_LARGE_SITE_MAIN_R1" {
  id = sdwan_feature_device_template.LAB1_SITE7_LARGE_SITE_MAIN_R1.id
  version = sdwan_feature_device_template.LAB1_SITE7_LARGE_SITE_MAIN_R1.version
  devices = [
    {
      id = "C8300-1N1S-4T2X-FLM2739115M"
      variables = {
...
      },
    },
  ]
}