Terraform Controller Support

[csantanapr]

Currently the terraform blueprints are deploy using argo-workflows, we want to move away from this approach and instead use some type TF Controller, that will allow user to edit their terraform in git, and the controller be able to re-apply, and maybe destroy the terraform plan

Here are some terraform k8s controller projects to look into:

• https://github.com/kubevela/terraform-controller/blob/master/getting-started.md
• https://github.com/upbound/provider-terraform
• https://github.com/flux-iac/tofu-controller
• terraform cloud https://developer.hashicorp.com/terraform/cloud-docs/integrations/kubernetes
• https://terranetes.appvia.io/terranetes-controller/

We need to pick one to do implement as a references architecture, the other would be homework to the user or can be contributed as examples.

cc @nimakaviani @elamaran11 @nabuskey

[nabuskey]

Some reasons I did not go with other open source solutions in the initial example:

1. I did not like double reconciliation process by Flux controller when deployed with ArgoCD.
2. I did not want to use Crossplane TF provider because I did not want to confine ourselves to Crossplane.
3. Resource requirements. I didn't like the idle resource usage required by controllers.
4. Ease of visualization in Argo Workflows.
5. DoEKS has its own install and cleanup scripts.

I think the way it was implemented was a mistake. We should go with an existing solution instead.

From my testing, the flux TF controller or tofu controller works well. I just did not like the way reconciliations are handled with ArgoCD. Flamingo was an attempt at resolving this but I don't think we should use it.

I think my second point above is still relevant . If we want to integrate with TF, we should try to find a solution that does that specifically.

We also need to figure out how to handle scripts that wrap TF.

[csantanapr]

@nabuskey Could you expand on “I just did not like the way reconciliations are handled with ArgoCD” ?

What exactly is the blocker is there an issue with more info in ArgoCD?
Or is it a problem in TF Controller to make it work with ArgoCD without requiring FluxCD?

[nimakaviani]

Based on everything discussed above, we are going to go down the route of using Flux + the Tofu controller Should keep the licenses in check for broader modification and use.

[nabuskey]

@nabuskey Could you expand on “I just did not like the way reconciliations are handled with ArgoCD” ?
What exactly is the blocker is there an issue with more info in ArgoCD?
Or is it a problem in TF Controller to make it work with ArgoCD without requiring FluxCD?

You must have the repository controller installed for the TF controller to work. At least it used to be, maybe it changed.

So it's very similar to the problem that Flamingo tried to solve. That is, defining repository spec in two different places.

[elamaran11]

@nabuskey Could you expand on “I just did not like the way reconciliations are handled with ArgoCD” ?
What exactly is the blocker is there an issue with more info in ArgoCD?
Or is it a problem in TF Controller to make it work with ArgoCD without requiring FluxCD?

You must have the repository controller installed for the TF controller to work. At least it used to be, maybe it changed.

So it's very similar to the problem that Flamingo tried to solve. That is, defining repository spec in two different places.

I just tried this out, you definately need a Flux repository controller to install tofu-controller via argo. Please see the error im facing in Argo console:

The Kubernetes API could not find source.toolkit.fluxcd.io/OCIRepository for requested resource argocd/aws-package. Make sure the "OCIRepository" CRD is installed on the destination cluster.

[nimakaviani]

that's fine. @elamaran11. we make those two controllers the prereqs for terraform integrations

[elamaran11]

@csantanapr @nimakaviani @nabuskey What about this - https://github.com/redhat-developer/argocd-terraform-controller ?

[nimakaviani]

doesn't appear well maintained. commits are from 2 years ago.

[elamaran11]

doesn't appear well maintained. commits are from 2 years ago.

Make sense, im POCing the approach of using two controllers, will keep you all posted.

[elamaran11]

I got tofu-controller working along with flux-controller dependency. I was able to create aws resources with default aws-package but only problem is the package comes corrupted with this issue. We have to do manual override for now to make this work.