dependsOn indexing is broken if it skips over an existing resource with a different for loop or indexing expression

[slavizh]

Bicep version
Bicep CLI version 0.31.34 (ec82b47)

Describe the bug
The change in #13674 broke existing syntax for language version 2 in critical way. The example is using Graph but it applies to all resource types with existing. Note that I am also using user assigned identities which is also Azure resource.

To Reproduce
This template:

extension 'br:mcr.microsoft.com/bicep/extensions/microsoftgraph/v1.0:0.1.8-preview'  as microsoftGraphV1
extension 'br:mcr.microsoft.com/bicep/extensions/microsoftgraph/beta:0.1.8-preview'  as microsoftGraphBeta

param entraGroup object = {
  name: 'ExampleGroup2'
  type: 'Security'
  members: [
    {
      name: '{application name}'
      type: 'Application'
    }
  ]
  owners: [
    {
      name: '{user identity name}'
      resourceGroup: '{resource group name}'
      type: 'UserAssignedManagedIdentity'
    }
  ]
}

var defaultMember = {
  subscriptionId: subscription().subscriptionId
  resourceGroup: ''
  name: ''
  appId: ''
}

resource memberManagedIdentities 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-07-31-preview' existing = [
  for (member, i) in entraGroup.members: if (member.type =~ 'UserAssignedManagedIdentity') {
    //https://github.com/Azure/bicep/issues/13937
    name: empty(union(defaultMember, member).name) ? 'dummy${i}' : member.name
    scope: resourceGroup(union(defaultMember, member).subscriptionId, union(defaultMember, member).resourceGroup)
  }
]

resource memberApplications 'Microsoft.Graph/[email protected]' existing = [
  for (member, i) in entraGroup.members: if (member.type =~ 'Application') {
    //https://github.com/Azure/bicep/issues/13937
    uniqueName: empty(union(defaultMember, member).name) ? 'dummy${i}' : member.name
  }
]

resource memberServicePrincipals 'Microsoft.Graph/[email protected]' existing = [
  for (member, i) in entraGroup.members: if (member.type =~ 'Application') {
    appId: memberApplications[i].appId
  }
]

resource memberServicePrincipalsStandalone 'Microsoft.Graph/[email protected]' existing = [
  for (member, i) in entraGroup.members: if (member.type =~ 'ServicePrincipal') {
    //https://github.com/Azure/bicep/issues/13937
    appId: empty(union(defaultMember, member).appId) ? 'dummy${i}' : member.appId
  }
]

resource memberGroups 'Microsoft.Graph/[email protected]' existing = [
  for (member, i) in entraGroup.members: if (member.type =~ 'Group') {
    //https://github.com/Azure/bicep/issues/13937
    uniqueName: empty(union(defaultMember, member).name) ? 'dummy${i}' : member.name
  }
]

resource ownerManagedIdentities 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-07-31-preview' existing = [
  for (owner, i) in entraGroup.owners: if (owner.type =~ 'UserAssignedManagedIdentity') {
    //https://github.com/Azure/bicep/issues/13937
    name: empty(union(defaultMember, owner).name) ? 'dummy${i}' : owner.name
    scope: resourceGroup(union(defaultMember, owner).subscriptionId, union(defaultMember, owner).resourceGroup)
  }
]

resource ownerApplications 'Microsoft.Graph/[email protected]' existing = [
  for (owner, i) in entraGroup.owners: if (owner.type =~ 'Application') {
    //https://github.com/Azure/bicep/issues/13937
    uniqueName: empty(union(defaultMember, owner).name) ? 'dummy${i}' : owner.name
  }
]

resource ownerServicePrincipals 'Microsoft.Graph/[email protected]' existing = [
  for (owner, i) in entraGroup.owners: if (owner.type =~ 'Application') {
    appId: ownerApplications[i].appId
  }
]

resource ownerServicePrincipalsStandalone 'Microsoft.Graph/[email protected]' existing = [
  for (owner, i) in entraGroup.owners: if (owner.type =~ 'ServicePrincipal') {
    //https://github.com/Azure/bicep/issues/13937
    appId: empty(union(defaultMember, owner).appId) ? 'dummy${i}' : owner.appId
  }
]

resource entraGroupRes 'Microsoft.Graph/[email protected]' = {
  uniqueName: entraGroup.name
  displayName: entraGroup.name
  mailEnabled: false
  mailNickname: entraGroup.name
  securityEnabled: true
  members: [
    for (member, i) in entraGroup.members: member.type =~ 'UserAssignedManagedIdentity'
      ? memberManagedIdentities[i].properties.principalId
      : member.type =~ 'Application'
          ? memberServicePrincipals[i].id
          : member.type =~ 'ServicePrincipal'
              ? memberServicePrincipalsStandalone[i].id
              : member.type =~ 'Group' ? memberGroups[i].id : member.type =~ 'PrincipalId' ? member.principalId : ''
  ]
  owners: [
    for (owner, i) in entraGroup.owners: owner.type =~ 'UserAssignedManagedIdentity'
      ? ownerManagedIdentities[i].properties.principalId
      : owner.type =~ 'Application'
          ? ownerServicePrincipals[i].id
          : owner.type =~ 'ServicePrincipal' ? ownerServicePrincipalsStandalone[i].id : owner.type =~ 'PrincipalId' ? owner.principalId : ''
  ]
}

Works fine with 0.30.23 (ec3612e) but it fails with 0.31.34 (ec82b47).

Error:

  "properties": {
        "statusCode": "InternalServerError",
        "serviceRequestId": null,
        "statusMessage": "{\"error\":{\"code\":\"MultipleErrorsOccurred\",\"message\":\"Multiple error occurred: . Please see details.\",\"details\":[{\"code\":\"InternalServerError\",\"message\":\"Encountered internal server error. Diagnostic information: timestamp '20241107T093958Z', subscription id '{redacted}', tracking id '8290acda-6a98-4c58-834f-a6056ede609e', request correlation id '8290acda-6a98-4c58-834f-a6056ede609e'.\"},{\"code\":\"InternalServerError\",\"message\":\"Encountered internal server error. Diagnostic information: timestamp '20241107T093958Z', subscription id '{redacted}', tracking id '8290acda-6a98-4c58-834f-a6056ede609e', request correlation id '8290acda-6a98-4c58-834f-a6056ede609e'.\"},{\"code\":\"InternalServerError\",\"message\":\"Encountered internal server error. Diagnostic information: timestamp '20241107T093958Z', subscription id '{redacted}', tracking id '8290acda-6a98-4c58-834f-a6056ede609e', request correlation id '8290acda-6a98-4c58-834f-a6056ede609e'.\"},{\"code\":\"InternalServerError\",\"message\":\"Encountered internal server error. Diagnostic information: timestamp '20241107T093958Z', subscription id '{redacted}', tracking id '8290acda-6a98-4c58-834f-a6056ede609e', request correlation id '8290acda-6a98-4c58-834f-a6056ede609e'.\"}]}}",

Additional context
Add any other context about the problem here.

[slavizh]

@jeskew @alex-frankel no idea how many people write the code like me and use language version 2 but I think this is pretty critical thus opted to mention you.

[slavizh]

Actually managed to scope the issue to the error generated when only Graph resources are referenced as existing. If I comment all the code around existing for Microsoft.Graph/applications, Microsoft.Graph/servicePrincipals and Microsoft.Graph/groups and their references the template deploys successfully. This is good as it seems affects only the Graph resources which is still in preview.

[jeskew]

@slavizh It looks like this isn't related to extensibility but happens when a non-copy resource depends on an existing copy resource that in turn depends on a non-existing copy resource. In that case, Bicep may sometimes try to skip over the existing resource in the dependency graph. The following template reproduces the issue without extensibility resources:

resource vnets 'Microsoft.Network/virtualNetworks@2024-03-01' = [for i in range(0, 10): {
  name: 'vnet${i}'
}]

resource subnets 'Microsoft.Network/virtualNetworks/subnets@2024-03-01' existing = [for i in range(0, 10): {
  parent: vnets[i]
  name: 'subnet'
}]

resource vault 'Microsoft.KeyVault/vaults@2023-07-01' = {
  name: 'vault'
  location: resourceGroup().location
  properties: {
    sku: {
      name: 'standard'
      family: 'A'
    }
    tenantId: subscription().tenantId
    networkAcls: {
      virtualNetworkRules: [for i in range(0, 10): {
        id: subnets[i].id
      }]
    }
  }
}

When compiled without symbolic name support (and therefore no existing resources in the compiled template), the generated template is invalid:

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "metadata": {
    "_generator": {
      "name": "bicep",
      "version": "0.30.23.60470",
      "templateHash": "1546815409293910064"
    }
  },
  "resources": [
    {
      "copy": {
        "name": "vnets",
        "count": "[length(range(0, 10))]"
      },
      "type": "Microsoft.Network/virtualNetworks",
      "apiVersion": "2024-03-01",
      "name": "[format('vnet{0}', range(0, 10)[copyIndex()])]"
    },
    {
      "type": "Microsoft.KeyVault/vaults",
      "apiVersion": "2023-07-01",
      "name": "vault",
      "location": "[resourceGroup().location]",
      "properties": {
        "sku": {
          "name": "standard",
          "family": "A"
        },
        "tenantId": "[subscription().tenantId]",
        "networkAcls": {
          "copy": [
            {
              "name": "virtualNetworkRules",
              "count": "[length(range(0, 10))]",
              "input": {
                "id": "[resourceId('Microsoft.Network/virtualNetworks/subnets', format('vnet{0}', range(0, 10)[range(0, 10)[range(0, 10)[copyIndex('virtualNetworkRules')]]]), 'subnet')]"
              }
            }
          ]
        }
      },
      "dependsOn": [
        "[resourceId('Microsoft.Network/virtualNetworks', format('vnet{0}', range(0, 10)[range(0, 10)[copyIndex()]]))]"
      ]
    }
  ]
}

(Note the copyIndex() expression within the vault resource's dependsOn property, even though vault is not a copy resource.)

This did not surface for symbolic name templates in v0.30 and below because vault would take a dependency on the existing resource:

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "languageVersion": "2.0",
  "contentVersion": "1.0.0.0",
  "metadata": {
    "_generator": {
      "name": "bicep",
      "version": "0.30.23.60470",
      "templateHash": "13202434162406894311"
    }
  },
  "resources": {
    "vnets": {
      "copy": {
        "name": "vnets",
        "count": "[length(range(0, 10))]"
      },
      "type": "Microsoft.Network/virtualNetworks",
      "apiVersion": "2024-03-01",
      "name": "[format('vnet{0}', range(0, 10)[copyIndex()])]"
    },
    "subnets": {
      "copy": {
        "name": "subnets",
        "count": "[length(range(0, 10))]"
      },
      "existing": true,
      "type": "Microsoft.Network/virtualNetworks/subnets",
      "apiVersion": "2024-03-01",
      "name": "[format('{0}/{1}', format('vnet{0}', range(0, 10)[range(0, 10)[copyIndex()]]), 'subnet')]",
      "dependsOn": [
        "[format('vnets[{0}]', range(0, 10)[copyIndex()])]"
      ]
    },
    "vault": {
      "type": "Microsoft.KeyVault/vaults",
      "apiVersion": "2023-07-01",
      "name": "vault",
      "location": "[resourceGroup().location]",
      "properties": {
        "sku": {
          "name": "standard",
          "family": "A"
        },
        "tenantId": "[subscription().tenantId]",
        "networkAcls": {
          "copy": [
            {
              "name": "virtualNetworkRules",
              "count": "[length(range(0, 10))]",
              "input": {
                "id": "[resourceId('Microsoft.Network/virtualNetworks/subnets', format('vnet{0}', range(0, 10)[range(0, 10)[range(0, 10)[copyIndex('virtualNetworkRules')]]]), 'subnet')]"
              }
            }
          ]
        }
      },
      "dependsOn": [
        "subnets"
      ]
    }
  }
}

The dependency graph builder recognizes that vault is not a copy resource and just takes a dependency on all of the subnet resources.

In v0.31, the graph builder tries to be a bit smarter and skip over the existing resource since vault is only using the subnets' ID, but this makes it run headfirst into the error that affects non-symbolic name templates:

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "languageVersion": "2.0",
  "contentVersion": "1.0.0.0",
  "metadata": {
    "_generator": {
      "name": "bicep",
      "version": "0.31.34.60546",
      "templateHash": "1523914663347343621"
    }
  },
  "resources": {
    "vnets": {
      "copy": {
        "name": "vnets",
        "count": "[length(range(0, 10))]"
      },
      "type": "Microsoft.Network/virtualNetworks",
      "apiVersion": "2024-03-01",
      "name": "[format('vnet{0}', range(0, 10)[copyIndex()])]"
    },
    "subnets": {
      "copy": {
        "name": "subnets",
        "count": "[length(range(0, 10))]"
      },
      "existing": true,
      "type": "Microsoft.Network/virtualNetworks/subnets",
      "apiVersion": "2024-03-01",
      "name": "[format('{0}/{1}', format('vnet{0}', range(0, 10)[range(0, 10)[copyIndex()]]), 'subnet')]",
      "dependsOn": [
        "[format('vnets[{0}]', range(0, 10)[copyIndex()])]"
      ]
    },
    "vault": {
      "type": "Microsoft.KeyVault/vaults",
      "apiVersion": "2023-07-01",
      "name": "vault",
      "location": "[resourceGroup().location]",
      "properties": {
        "sku": {
          "name": "standard",
          "family": "A"
        },
        "tenantId": "[subscription().tenantId]",
        "networkAcls": {
          "copy": [
            {
              "name": "virtualNetworkRules",
              "count": "[length(range(0, 10))]",
              "input": {
                "id": "[resourceId('Microsoft.Network/virtualNetworks/subnets', format('vnet{0}', range(0, 10)[range(0, 10)[range(0, 10)[copyIndex('virtualNetworkRules')]]]), 'subnet')]"
              }
            }
          ]
        }
      },
      "dependsOn": [
        "[format('vnets[{0}]', range(0, 10)[copyIndex()])]"
      ]
    }
  }
}

The graph builder needs to be aware of when it is skipping over a looped existing resource.

[anthony-c-martin]

@slavizh we're working on a fix for this now - I think we'll need to do a new Bicep release for it.

[slavizh]

@anthony-c-martin great! I am happy that the root cause was found so fast.

[jeskew]

@slavizh The change included in v0.31 was reverted in the latest version. Leaving this open while we work on fixing the underlying issue for 0.32

[scrocquesel-ml150]

Hi, could be the same root cause. Azure Devops agent have recently been updated to 0.31.34 and we now have build issues. Note that when using version v0.31.92 locally, we have no issue.

Error BCP036: The property "scope" expected a value of type "resourceGroup" but the provided value is of type "{ }". [https://aka.ms/bicep/core-diagnostics#BCP036]

and the corresponding code is

resource serviceBusResource 'Microsoft.ServiceBus/namespaces@2024-01-01' existing = {
  name: '${env}-svcbus'
  scope: env == 'dev' ? resourceGroup('x-${env}') : resourceGroup('x-${env}-rg')
}

I changed to scope: resourceGroup(env == 'dev' ? 'x-${env}' : 'x-${env}-rg') but I have other build errors.

[jeskew]

@scrocquesel-ml150 There's some discussion on that in #15517, and it should be fixed in 0.31.92