Problem with resource "nxos_port_channel_interface_member" with list of interfaces

[hani010]

Hello,

we wrote an modul that create interfaces and an portchannel but got an error when trying to attached multiple interfaces to the portchannel. This first interfaces is fine, but we get an error to add the other interfaces. When apply terrafrom again, everthing is fine.

Module main.tf

terraform {
  required_providers {
    nxos = {
      source = "CiscoDevNet/nxos"
      #version = "0.5.1"
    }
  }
}
resource "nxos_physical_interface" "r_nxos_physical_interface" {
  for_each = toset(var.physical_intfs)
  interface_id             =  each.key
  admin_state              = var.admin_state
  description              =  var.description
  mode                      = var.switchport_mode
  trunk_vlans               = "1010-1500"
  device                    = "dc3-spktaggr-pw01"
}

resource "nxos_port_channel_interface" "r_nxos_port_channel_interface" {
  interface_id          = var.port-channel_id
  port_channel_mode     = var.port_channel_mode
  admin_state           = var.admin_state
  description           = local.updated_description
  link_logging          = "enable"
  mode                  = var.switchport_mode
  trunk_vlans           = var.trunk_vlans_po
  }
  
resource "nxos_port_channel_interface_member" "r_nxos_port_channel_interface_member" {
 for_each = toset(var.physical_intfs)
 interface_dn = "sys/intf/phys-[${each.key}]"
 interface_id = nxos_port_channel_interface.r_nxos_port_channel_interface.interface_id
 depends_on =[nxos_physical_interface.r_nxos_physical_interface]
 device = var.device 

Calling the module

module "nxos_port_channel_dc3-spktaggr-pw01-po520" {
  source                    = "./modules/port_channel"
  physical_intfs            = ["eth1/29", "eth1/30"]
  admin_state               = "up"
  description               = "meine_neue_description"
  switchport_mode           = "trunk"
  trunk_vlans_po            = "1010-1500"
  port-channel_id           = "po520"
  port_channel_mode         = "active"
  vpc-port-channel_id       = null
  device                    = "dc3-spktaggr-pw01"
}

Error Message

module.nxos_port_channel_dc3-spktaggr-pw01-po520.nxos_port_channel_interface_member.r_nxos_port_channel_interface_member-sec-intf["eth1/29"]: Creation complete after 0s [id=sys/intf/aggr-[po520]/rsmbrIfs-[sys/intf/phys-[eth1/29]]]
╷
│ Error: Client Error
│ 
│   with module.nxos_port_channel_dc3-spktaggr-pw01-po520.nxos_port_channel_interface_member.r_nxos_port_channel_interface_member-sec-intf["eth1/30"],
│   on modules/port_channel/nxos_port_channel.tf line 68, in resource "nxos_port_channel_interface_member" "r_nxos_port_channel_interface_member-sec-intf":
│   68: resource "nxos_port_channel_interface_member" "r_nxos_port_channel_interface_member-sec-intf" {
│ 
│ Failed to post object, got error: JSON error: {"imdata":[{"error": {"attributes": {"code": "1","text": "Command failed: Port not compatible \n  Buffer boost : Port-channel has buffer-boost capability unset and interface has the
│ capability set \n** You can use force option to override the port's parameters \n** (e.g. \"channel-group X force\") \n** Use \"show port-channel compatibility-parameters\" to get more information on
│ failure\n\nfaulty_dn=sys\/intf\/aggr-[po520]\/rsmbrIfs-[sys\/intf\/phys-[eth1\/30]]"}}}]}`

[danischm]

Hi @hani010 ! Can you try adding an explicit dependency to the nxos_port_channel_interface resource and check if that makes a difference:

resource "nxos_port_channel_interface" "r_nxos_port_channel_interface" {
  ...
  depends_on = [nxos_port_channel_interface_member.r_nxos_port_channel_interface_member]
}

[hani010]

Hi,
yes i try this and other depends, but nothing works correct. My workaround is, i start with one member and then i add all other members, or i execute terraform apply two times...

[huntx]

Hi @hani010,

I had the same problem before.

You could try putting the "nxos_physical_interface" resource part of the dependecy.

resource "nxos_physical_interface" "po_member_phy_interface" {
  device                     = each.value.device
  description                = each.value.description
  layer                      = "Layer2"
  interface_id               = each.value.interface_dn
  admin_state                = "up"
  mode                       = "trunk"
  trunk_vlans                = "1-4094"
}

resource "nxos_port_channel_interface" "po_inteface" {
  device                      = each.value.device
  interface_id                = each.value.interface_id
  description                 = each.value.description
  port_channel_mode           = "active"
  suspend_individual          = "enable"
  auto_negotiation            = "on"
  speed                       = "auto"
  admin_state                 = "up"
  layer                       = "Layer2"
  mode                        = "trunk"
  trunk_vlans                 = "1-4094"
}

resource "nxos_port_channel_interface_member" "po_inteface_member" {
   device                = each.value.device
   interface_id          = each.value.interface_id
   interface_dn          = "sys/intf/phys-[${each.value.interface_dn}]"

  depends_on = [
    nxos_physical_interface.po_member_phy_interface, nxos_port_channel_interface.po_inteface
  ]
}

[huntx]

@danischm Is it possible to add a new schema item to perform 'Channel group force' for "nxos_port_channel_interface_member" resource ?

It seems possible from the API documentation.
https://pubhub.devnetcloud.com/media/dme-docs-10-2-2/docs/Interfaces/pc:RsMbrIfs/

[danischm]

4524b8c

[danischm]

Added the force option in v0.5.2 release. Can this be considered resolved now?

[huntx]

I tested the force is working.

But, it can't be used to update existing port-channel config (eg: vlans, switch mode) as the nxos_physical_interface resource cannot be dynamically updated for subsequent "terraform apply"

I guess this is the nature of terraform, which it is great as to deploy the infrastructure. But for operational tasks, such as port configuration changes, it's necessary to fall back to using Ansible.

[LimpensE]

We are experiencing a similar issue, when keeping the vlan list empty and specifying vlans on the port channel interface, a change in the vlans results in an error:

│ Error: Client Error
│
│   with nxos_physical_interface.leafs-default-interfaces["LEAF01-G3.30"],
│   on leafs-interfaces.tf line 1, in resource "nxos_physical_interface" "leafs-default-interfaces":
│    1: resource "nxos_physical_interface" "leafs-default-interfaces" {
│
│ Failed to update object, got error: JSON error: {"imdata":[{"error": {"attributes": {"code": "1","text": "ERROR: : port already in a port-channel, no config
│ allowed\nfaulty_dn=sys\/intf\/phys-[eth1\/30], Pkg=l1, biStatus=endCommonCb"}}}]}

Using the force parameter does not change the behavior and dependencies should be in correct order.

As mentioned above, it looks like the provider (0.5.4) is having issues when changing the vlan list on a port-channel.

[jgomezve]

Hi

What about using the lifecycle meta-argument on the nxos_physical_interface resource?

resource "nxos_physical_interface" "r_nxos_physical_interface" {
  for_each     = toset(var.physical_intfs)
  interface_id = each.key
  admin_state  = var.admin_state
  description  = var.description
  mode         = var.switchport_mode
#   trunk_vlans  = var.trunk_vlans_po
  lifecycle {
    ignore_changes = [
      trunk_vlans,
      mode,
    ]
  }
}

[jgomezve]

Would you mind sharing the NXOS version you are using? I have experiences some issues while destroying the resource nxos_physical_interface on NXOS version 9.3(8). The POST operation to restore the resource times out. I do not see any issues on version 9.3(10)

[LimpensE]

Hi Jorge,

We are running with version NXOS: version 10.3(5) [Maintenance Release]

Your hint to use the lifecycle argument looks very promising. Changing the number of configured vlans on the port-channel interface is now properly reflected on the physical interfaces. Will have to do some more testing, but it looks you resolved our issue.

Regards, Eric.