DevOps Automation using Bicep

[tonydean37]

Hey @DCMattyG,

Our PoC is coming along pretty nicely and our Automation team have produced a working script to create a Landing zone with a VNet CIDR picked by IPAM, which is great. :)

Our next challenge is to enable users to input the subnet masks that they would like in the same script and the IPAM solution hand these out to the pipeline. As currently the IPAM solution API for subnets only provides the next available CIDR instead of reserving it, then I was wondering if you knew of a method to automate these?

Our current process (from a very high level) looks something like this:

This is the process that I think would need to happen if we tried assigning multiple Subnets:

It would also only work if the IPAM Solution had updated its' database with the new VNet info, which could potentially cause issues.

Any thoughts would be really appreciated :)

Cheers!

[tonydean37]

Hey @DCMattyG

I have another question regarding the reservations too please. :) We use multiple address prefixes per VNet (so for example two reservations: /24, /25 from IPAM, but a single VNet). When the VNet is created, the script can only assign a single tag with the name: ipam-res-id.

In IPAM I get an error message saying: "A vNET with the assigned CIDR has already been associated with the target IP Block" when I assign one of the Reservation IDs to the VNet and both reservations stay.

I have also tried adding both reservation IDs into the tag using an array ["xxxx", "yyyy"] and using a comma xxxx, yyyy. The IPAM Solution doesn't appear to recognise this and leaves the reservations in place with the status of pending

I wondered if there is a way we can still use the tagging solution, but pass an array of Reservation IDs back to the IPAM API to delete both reservations?

Hope this makes sense. :)

[DCMattyG]

Hi there @tonydean37, there's quite a bit to digest from the above, but I promise I'll get to all of it. I'm going to break it down and ask some clarifying questions first:

1. Can you help me better understand what the actual user input looks like for the Pipeline? Are they providing the number of subnets and the size, or are they more explicitly providing a list of subnet sizes? I just want to understand better what the intended input would be.
2. Can you help me better understand the business case where you're assigning multiple non-contiguous Address Spaces during the creation of the Virtual Network? You're correct that we don't specifically support that today, but I could easily add that in....just want to understand a bit better the "why" (so I know when someone asks me haha).

Thank you for all of the excellent feedback and questions. We'll make sure you get taken good care of 😄

[tonydean37]

That's great thanks :) - I've been trying to draw it out a bit to get the flow sorted in my head!

1. (Apologies if this is too much detail and hopefully this makes sense) So currently our process for creating new Landing Zones is as follow:
   i. User emails networks advising size of VNet required and engineer checks spreadsheet (I know!) and sends back an appropriate prefix (e.g. 10.0.0.0/24)
   ii. User creates a new repo in Azure DevOps and enters the CIDR into network.vnetCidr parameter.
   iii. As part of our script a "core" resource group is created, which has a /28 network in it (e.g. 10.0.0.0/28) for things like Private Endpoint for KeyVault. So the user has to carve up the vnet CIDR taking that into account (e.g. 10.0.0.16/28, 10.0.0.128/25), which they would do by adding the CIDRs into the network.subnets parameter (this would include a name, addressprefix and NSG).

So currently it is very much down to the user to complete those activities, but they may require network support to assist in carving up the network.

We thought that we may try and use some logic that allows the user to input how many hosts they will need in the VNet and Subnets, which would then auto assign the required prefixes for each (but this is undecided at the moment). For the time being we will be asking the user to input the vnet CIDR they want (e.g. [/24]) and the subnet CIDRs they want (e.g. [/28, /25]).

2. I think you make a really good point asking why a we would require two address prefixes that are non-contiguous because I can't really think of a good use case, but we do have a few VNets in our subs setup like this (I can check with colleagues why if you would like) :). Where I think I can see a use case is if an existing VNet ran out of usable IPs and we needed to add an additional/second CIDR to the VNet. From what I saw earlier with the error message, I don't think the reservation would clear when the additional VNet is added (but I could well be wrong here)?

As always really appreciate you taking the time to investigate our queries. :)

Have a good weekend!

[DCMattyG]

Hey @tonydean37, apologies for my delayed response. I've been OOO this week.

Tagging @hbendana on this item as well, as I believe he'll have some great feedback on the above workflow. I'll add my thoughts shortly as well. Just letting you know we didn't forget about you!

[tonydean37]

Hi @DCMattyG, No worries at all - I was away at an internal conference last week, so didn't get as much time on this one. We have, however, come up with a bit of a solution surrounding the multiple VNet issue:

I've included an updated Visio of the workflow, but basically, after the VNet has been created in Azure, we send a GET request from the pipeline to the IPAM server for all the VNets in the block. If the new VNet Resource ID doesn't appear, then send a PUT request to IPAM to add the VNet to the correct block. We use a loop to ensure that the VNet is in the correct block and then send a DELETE request to IPAM to remove the reservation ID once it is:

We've tested a couple of times and it all appears to work correctly, but I'm not too sure what you think of the logic?

In terms of the Subnet assigning, do you think this will require a similar reservation system to the one used for the VNets?

Cheers!

[hbendana]

Hello @tonydean37 thank you for your patience while I circle back with you on this. I love how you are integrating IPAM into your automated workflow, we love to see it :) A few thoughts/comments come to mind regarding your workflow. I'll do my best to articulate over this medium but know that we are happy to meet more formally (virtually and/or in person if you are on the west coast).

• remember that when you create a reservation within a space/block and are returned the CIDR and reservation ID, once that information is leveraged to create the VNET resource in Azure, the reservation is deleted and the VNET will automatically be allocated to the corresponding space/block. On its next scan, IPAM will discover the VNET (along with its reservation ID tag) make the connection between it and the pending reservation, and then cleanup said reservation.
• regarding non-contiguous address space reservations, as @DCMattyG mentioned, we don't support it yet BUT you could handle each individual CIDR for the VNET as its own reservation request (two separate calls to the IPAM api). You then create the VNET with the two distinct CIDRS and apply each individual reservation ID tag to the VNET. Not idea, we know, but an operationally sound way to go about it for now.
• With regards to subnets....we have a new feature coming in our next release that will provide the ability to look up the next available subnet within a VNET (no reservation functionality though). That should solve your subnet workflow step. We hope to have this released very, very soon!

Let us know if this helps or if we can dive deeper with you, we are always here to help :)

[tonydean37]

Morning @hbendana, sorry for the slow reply - I've been on annual leave for the past week, so just catching up now. As always really appreciate you taking the time to respond and help us get the most out of our deployment.

We came across a few issues when trying to add multiple address prefixes to the VNET and trying to clean up the reservations in IPAM using the tags:

• The first being that we could not assign both reservation IDs due to the VNET not allowing multiple tags with the same name.
• The second being that even with one of the reservation IDs being tagged on the VNET, we had an error appear in IPAM and the reservation was not cleaned up.

But we have overcome these issues with the workflow above, by sending a PUT request to IPAM, which assigns the VNET to the correct block, then a DELETE request that removes the reservations.

In terms of the subnets, that sounds really promising. I would love to hear more about this as the current API command for providing the next available is great, but when working with multiple subnets in a VNET doesn't really give us the flexibility we require. I don't suppose we could set up a meeting to discuss that and our automation to see where we could improve things? My colleague Ash and I are based in the UK, but I'm happy to work around you in terms of a meeting time. :)

[DCMattyG]

Hey @tonydean37, I've already responded to @jonprattLBG in #116, but I'll re-post the detail here for you as well...

I took a look at this closer and I think I found a very simple way to make this work.

The implementation would be adding N number of reservations to a Virtual Network TAG, separate by commas, like this:

If that would work, I have this functioning in our DEV containers, and you're welcome to give it a whirl and provide some feedback. Here are the configuration changes you'd need to test the development version:

Under the "Development Center" for the Azure App Service, add the "dev" suffix to the ACR address (e.g. azureipamdev) for all 3 containers. Click save, allow the App Service to pull the new containers and restart (usually ~5 minutes), then give it a try!

Excited for your feedback on this item, and thank you for your patience!

[DCMattyG]

Looking good @tonydean37! Adding @hbendana back in for his feedback on the pipeline ☝️

On the topic of the logo, you are (unsurprisingly) not the first person to ask for this. I do have this on my "to-do" list, but this is likely 60-ish days out as I have some other higher priority features that I'm working on. It will definitely happen, just a matter of a bit more time.

Thanks for the continuous feedback and patience!

[DCMattyG]

BTW, there are ways to change it if you're compiling your own Azure IPAM containers and replace the logo files in the code before building them, but there's no "end user" way of accomplishing this YET, which is what I'm working on for an upcoming release.

[tonydean37]

Morning @DCMattyG - thanks for confirming the logo details. There's no rush for this change, so waiting ~60days is no problem at all.

We have tried out the new feature in Dev, but have come across a bit of an issue, which I was wondering if you could help with? We have vended a new vnet with 2 address space: 10.101.2.0/23 and 10.101.0.0/24 and tagged the vnet with the reservation IDs for both of those:

When we check IPAM, the vnet is in the correct block, but the second address space that is created still appears in the reservation area with the error: A vNET with the assigned CIDR has already been associated with the target IP Block.

The vnet also only shows one prefix under the Virtual Network Association screen:

If I remove the reservation ID then the prefix does get added though:

We have also tried adding the tags with a comma and then space in between each, but the same results were seen.

[tonydean37]

Hi @DCMattyG - I was wondering if you'd been able to investigate this one at all?

[DCMattyG]

Hi @tonydean37, apologies for the delay. I finally took some PTO after a very long time, lol.

I haven't been able to reproduce this unfortunately, would you have some time perhaps that we could do a 1:1 to look further into this together? You can drop me an email or Teams message at [email protected].

I'll keep attempting a few permutations on my side, but so far I'm not sure why you're seeing different results than I am here.