Add extra_args_array and windows_extra_args

[HarrisonWAffel]

Issue: #348

Problem

Rancher and RKE allow users to specify arguments for RKE services multiple times through the use of ExtraArgsArray and WindowsExtraArgsArray, however terraform does not support these fields. Additionally, terraform does not support WindowsExtraArgs.

A previous implementation of this PR encountered difficulties suppressing diffs being detected by Terraform, however it seems that this issue has been resolved with the recent SDK update and by updating the version of terraform used during development.

Solution

Add the missing fields to the schema's for each RKE service. Some services do not have arguments that can be passed multiple times, however their schema has been updated for the sake of consistency.

For each service the extra_args_array and windows_extra_args_array fields expect a JSON encoded string value with the following format:

extra_args_array = jsonencode({
      command_one = ["value1", "value2"],
      command_two = ["value1", "value2"]
})

Testing

Use the terraform listed at the bottom of the ticket to deploy a single node all roles RKE cluster. Follow the same test steps as outlined in rancher/rancher#25500, with the only difference being the modification of the terraform as opposed to the yaml within Rancher UI.

Engineering Testing

Manual Testing

Automated Testing

• Test types added/modified:
  • Unit tests have been expanded to cover the new fields

QA Testing Considerations

The terraform included in this PR will work by default, however a section has been commented out. This section will test the schema changes for the scheduler service, however it will result in terraform apply failing to create the cluster. This failure is expected, as properly configuring the environment such that the arguments are valid is outside the scope of this issue.

Each k8 service will log all of the arguments it was started with, so to confirm that particular changes to the terraform have been properly passed to the service via RKE you can simply look at the first few log messages of each relevant service.

Regressions Considerations

This PR adds in net-new schema fields, and does not modify existing logic. There should be no regressions.

Terraform

Click to Expand Terraform

terraform {
  required_providers {
    rke = {
      source  = "rancher.com/rkeprovider/rke"
      version = "1.3.2"
    }
  }
}

provider "rke" {
  log_file = "/Users/haffel/terraform/extra-args/log"
  debug    = "true"
}

resource "rke_cluster" "rke_cluster" {
  kubernetes_version = "v1.26.4-rancher2-1"
  enable_cri_dockerd = true
  nodes {
    address = "<IP_ADDRESS>"
    user    = "<USER>"
    role    = ["etcd", "controlplane", "worker"]
    ssh_key = file("<YOUR_RSA_FILE>")
  }

  services {

    kube_api {
      extra_args_array = jsonencode(
            {
            service-account-key-file = ["/etc/kubernetes/ssl/kube-apiserver-key.pem"],
            service-account-issuer = ["aws", "gcp", "azure"]
            }
        )

      pod_security_policy = false
    }

    etcd {
      extra_args_array = jsonencode(
            {
            listen-peer-urls = ["http://localhost:2381", "http://localhost:2382"],
            }
        )
    }

    # kube-proxy doesn't apply
    # kubelet doesn't apply

    # uncomment the below with the understanding that `tf apply` will not complete successfully. The error message will indicate that the expected arguments were passed, but are not valid. 
    # scheduler {
    #  extra_args_array = jsonencode(
     #       {
     #      tls-sni-cert-key = ["*.foo.com,foo.com", "*.foo2.com,foo.com", "*.foo3.com,foo.com"],
     #      }
     #  )
  }
}

[a-blender]

Looks like the CI is failing due to a flattener test failure

[a-blender]

Based on discussion #348 (comment), this PR just needs to be retested on the latest repo version to make sure the reordering bug is fixed.

[HarrisonWAffel]

I've reworked this PR to use jsonencode when working with the extra_args_array fields. This results in the schema representation I wanted, but loses some of the clarity when working with the fields. This will need to be documented so the proper JSON structure expected by this field is well known by users. This is however a better solution considering the limitations of the current provider and its unit tests (cannot properly leverage the TypeSet field type due to use of DeepEqual).

I've double checked this change and did not see any instances of terraform reporting a change every time tf plan is executed.

I've updated the documentation and the full/example.tf file to record what the expected format of that JSON value is.