spotinst_ocean_aws does not detect diff for instance_metadata_options

[trc-ikeskin]

Terraform Version

Terraform v1.6.5
on darwin_arm64
+ provider registry.terraform.io/alekc/kubectl v2.0.4
+ provider registry.terraform.io/hashicorp/aws v5.31.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.3.3
+ provider registry.terraform.io/hashicorp/helm v2.12.1
+ provider registry.terraform.io/hashicorp/kubernetes v2.24.0
+ provider registry.terraform.io/hashicorp/random v3.6.0
+ provider registry.terraform.io/hashicorp/time v0.10.0
+ provider registry.terraform.io/hashicorp/tls v4.0.5
+ provider registry.terraform.io/spotinst/spotinst v1.156.0

Your version of Terraform is out of date! The latest version
is 1.6.6. You can update by downloading from https://www.terraform.io/downloads.html

Affected Resource(s)

• spotinst_ocean_aws
• spotinst_ocean_aws_launch_spec (possibly)

Terraform Configuration Files

resource "spotinst_ocean_aws" "this" {
  name          = module.eks.cluster_name
  controller_id = module.eks.cluster_name
  region        = data.aws_region.current.name

  # --- Autoscaling

  max_size         = terraform.workspace == "prod" ? 10 : 3
  desired_capacity = terraform.workspace == "prod" ? 3 : 1
  min_size         = terraform.workspace == "prod" ? 3 : 1

  autoscaler {
    autoscale_is_enabled                 = true
    autoscale_is_auto_config             = true
    enable_automatic_and_manual_headroom = true
    auto_headroom_percentage             = 5

    resource_limits {
      max_vcpu       = 1024
      max_memory_gib = 1500
    }
  }

  # --- Instance Types

  filters {
    architectures     = ["x86_64"]
    categories        = ["General_purpose", "Compute_optimized", "Memory_optimized"]
    disk_types        = ["EBS", "SSD"]
    exclude_families  = ["f*", "g*", "p*", "x*", "z*"]
    exclude_metal     = true
    max_memory_gib    = 64
    min_memory_gib    = 4
    root_device_types = ["ebs"]
  }

  # --- Networking

  subnet_ids = module.vpc.private_subnets

  # --- Launch Configuration

  image_id        = data.aws_ami.bottlerocket.id
  security_groups = [module.eks.node_security_group_id]
  user_data = templatefile("${path.module}/files/templates/user_data.toml", {
    CLUSTER_NAME           = module.eks.cluster_name
    CLUSTER_ENDPOINT       = module.eks.cluster_endpoint
    CLUSTER_AUTH_BASE64    = module.eks.cluster_certificate_authority_data
    ENABLE_ADMIN_CONTAINER = true
  })
  iam_instance_profile = aws_iam_instance_profile.eks_nodes.name
  monitoring           = true # CloudWatch detailed monitoring
  ebs_optimized        = true

  block_device_mappings {
    device_name = "/dev/xvda"
    ebs {
      volume_type           = "gp3"
      encrypted             = "true"
      kms_key_id            = module.kms["ebs"].key_arn
      delete_on_termination = "true"
      throughput            = 125
      iops                  = 3000

      dynamic_volume_size {
        base_size              = 2
        resource               = "CPU"
        size_per_resource_unit = 2
      }
    }
  }

  block_device_mappings {
    device_name = "/dev/xvdb"
    ebs {
      volume_type           = "gp3"
      encrypted             = "true"
      kms_key_id            = module.kms["ebs"].key_arn
      delete_on_termination = "true"
      throughput            = 125
      iops                  = 3000

      dynamic_volume_size {
        base_size              = 20
        resource               = "CPU"
        size_per_resource_unit = 10
      }
    }
  }

  # --- IMDS

  instance_metadata_options {
    http_tokens                 = "required"
    http_put_response_hop_limit = 2 # Container -hop-> EC2 instance -hop-> IMDSv2
  }

  # --- Load Balancing/Target Group Registration

  draining_timeout = 120
  grace_period     = 300

  # --- Purchasing Options

  fallback_to_ondemand       = true
  spot_percentage            = 100
  utilize_commitments        = true # use saving plans if existing
  utilize_reserved_instances = true # use reserved instances if existing

  # --- Tagging

  resource_tag_specification {
    should_tag_volumes = true
  }

  tags {
    key   = "Name"
    value = "xxxxxxx"
  }

  tags {
    key   = "managed_by"
    value = "xxxxxxx"
  }

  tags {
    key   = "repository"
    value = "xxxxxxx"
  }

  tags {
    key   = "kubernetes.io/cluster/${module.eks.cluster_name}"
    value = "owned"
  }

  tags {
    key   = "eks:cluster-name"
    value = module.eks.cluster_name
  }
}

Debug Output

https://gist.github.com/trc-ikeskin/a5338bb1a02484e605245bb09af40f2b

Expected Behavior

The provider should detect that there is drift between the cloud cluster resource in Spot Ocean and the Terraform state:
The state contains a list instance_metadata_options while the response object returned from https://api.spotinst.io/ocean/aws/k8s/cluster/o-55xxxx?accountId=act-3cxxxx does not contain a corresponding object response.items[0].compute.launchSpecification.instanceMetadataOptions. This should trigger the provider to update the cloud cluster resource in Spot Ocean.

Actual Behavior

The provider does not detect a drift and therefore does not trigger an update on the cloud cluster resource, which causes the instance_metadata_options to be missing from Spot Ocean.

Steps to Reproduce

1. Configure an instance_metadata_options block for the spotinst_ocean_aws resource.
2. terraform apply

Community Note

• Please vote on this issue by adding a 👍 reaction to help the community and maintainers prioritize this request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

[anuragsharma-123]

@trc-ikeskin - The team is on a year-end weeklong vacation. We will handle this issue in the first week of Jan 2024.

[trc-ikeskin]

@anuragsharma-123 no worries, thank you for the heads up! I wish you and your team a relaxing vacation.

[chandra1-n]

@trc-ikeskin Could you give more details on this, I have tried reproducing this as per the steps mentioned but couldn't reproduce it. I could see the diff showing for both cluster and launchSpec and also I suggest you to use the latest version of terraform provider. (1.157.0)
Steps followed for both cluster and launchSpec:
Scenario1- Created a ocean aws cluster without instance_metadata_options block configured.
Updated the cluster to configure the instance_metadata_options block, during plan or apply changes are reflected as intended to be and after apply changes are updated in cluster configuration.

Scenario2- Created a ocean aws cluster with instance_metadata_options block configured
Updated the values of instance_metadata_options - values updated as intended

[trc-ikeskin]

@chandra1-n Thanks for testing. I reproduced this on 3 different clusters. For each of them no diff was shown.

Is it possible that this error is caused when clusters are imported to the Terraform state after they were created manually? This is the only obvious difference I can observe compared to the provided scenarios.

[trc-ikeskin]

@chandra1-n @anuragsharma-123 any news on this? Could it be an issue that is caused by importing existing clusters into the state (see my last comment)?

[chandra1-n]

@trc-ikeskin that could be possible, but once you create a ocean cluster using spot TF provider it would create a new state file which will display diff for instance_metadata_option if there is change in it. Would like to know what exactly the use case here and if possible please share the files by masking the actual values, so that we can look into this further.

[trc-ikeskin]

@chandra1-n Not sure what you mean by "use case"? The Spot Cloud Cluster was created manually and it was later decided, that it should be managed by Terraform - hence the resource was imported to a Terraform state. Development was then continued via Terraform, where it became obvious that changes were not detected properly.

Providing all the files, will take sometime because I have to obfuscate a lot of customer related data from the code. I will get back to you as soon as it is ready.

[sharadkesarwani]

Hello @trc-ikeskin

We tried the scenario, creating spot-ocean-aws manually (without configuring instance_metadata_options) and then imported terraform state and then added " instance_metadata_options" to cluster config which is showing the difference as expected.