provider/stack: add new stack resource for Stacks API integration

[rjeczalik]

This PR adds a "scylladbcloud_stack" resource which is used for accounting managed resources with the Stacks API.

It uses new client, as the payloads for Stacks API are HMAC signed.

[ksinica]

I've done some quick manual testing for the following scenarios:

• creating a new cluster using work in this pull request: no issues
• updating a cluster created with v1.6.1: no issues, no state differences

LGTM!

I wasn't able to delete the cluster after I approved the PR.

[dkropachev]

@rjeczalik , could you please point me to the docs/code of stack API, I having problems with locating it, is it on siren? Could you also please provide test/example for scylladbcloud_stack?

[rjeczalik]

I wasn't able to delete the cluster after I approved the PR.

I will look into that.

@rjeczalik , could you please point me to the docs/code of stack API, I having problems with locating it, is it on siren?

siren-byok

Could you also please provide test/example for

Relevant part of the example:

{
        "provider": {
                "scylladbcloud": {
                        "endpoint": "...",
                        "token": "${ format(\"%s:%s\", var.StackID, var.StackSecret) }"
                }
        },
        "resource": {
                "scylladbcloud_stack": {
                        "scylladb_cloud": {
                                "attributes": {
                                        "google_kms_crypto_key_id": "${ google_kms_crypto_key.scylladb_cloud.id }",
                                        "google_kms_key_ring_id": "${ google_kms_key_ring.scylladb_cloud.id }",
                                        "stack_id": "${ var.StackID }"
                                }
                        }
                }
        }
}

It is on purpose not documented, as it is meant to be used only for stack creation flows with siren-byok.

[dkropachev]

I wasn't able to delete the cluster after I approved the PR.

I will look into that.

@rjeczalik , could you please point me to the docs/code of stack API, I having problems with locating it, is it on siren?

siren-byok

Could you also please provide test/example for

Relevant part of the example:

{
        "provider": {
                "scylladbcloud": {
                        "endpoint": "...",
                        "token": "${ format(\"%s:%s\", var.StackID, var.StackSecret) }"
                }
        },
        "resource": {
                "scylladbcloud_stack": {
                        "scylladb_cloud": {
                                "attributes": {
                                        "google_kms_crypto_key_id": "${ google_kms_crypto_key.scylladb_cloud.id }",
                                        "google_kms_key_ring_id": "${ google_kms_key_ring.scylladb_cloud.id }",
                                        "stack_id": "${ var.StackID }"
                                }
                        }
                }
        }
}

It is on purpose not documented, as it is meant to be used only for stack creation flows with siren-byok.

It is not clear, it is public tf provider using public API, what exactly are we trying to hide by not providing example?

[rjeczalik]

It is not clear, it is public tf provider using public API, what exactly are we trying to hide by not providing example?

I don't understand the question. I replied very clear, that this resource is not intended for direct usage, because from the perspective of Terraform Provider user it does not bring any features. It is used from automation flow

Your assumption that it uses public api is also incorrect. If there are any more further questions, please contact me off-list.

[dkropachev]

It is not clear, it is public tf provider using public API, what exactly are we trying to hide by not providing example?

I don't understand the question. I replied very clear, that this resource is not intended for direct usage, because from the perspective of Terraform Provider user it does not bring any features. It is used from automation flow

Your assumption that it uses public api is also incorrect. If there are any more further questions, please contact me off-list.

What is automation flow mean in this context ? It is testing ?
Sorry for not being clear, I was trying to say that my perception is existance of PUBLIC API only for testing purposes is wrong.
It is just a note and does not block this PR review, I am working on it.

[dkropachev]

It failed on following:

1. Create cluster with v1.6.1 provider:

resource "scylladbcloud_cluster" "aws" {
  name       = "AWS"
  cloud      = "AWS"
  region     = "us-east-1"
  node_count = 3
  node_type  = "t3.micro"
  cidr_block = "172.31.0.0/24"
  enable_dns = true
}

2. Install this plugin, and run terraform apply and see it is stuck:

2024-07-01T12:02:54.413-0400 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/scylladb/scylladbcloud\"] (close)" is waiting for "scylladbcloud_cluster.aws (expand)"
2024-07-01T12:02:54.463-0400 [TRACE] dag/walk: vertex "root" is waiting for "scylladbcloud_cluster.aws"
2024-07-01T12:02:59.414-0400 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/scylladb/scylladbcloud\"] (close)" is waiting for "scylladbcloud_cluster.aws (expand)"
2024-07-01T12:02:59.414-0400 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/scylladb/scylladbcloud\"] (close)"
2024-07-01T12:02:59.464-0400 [TRACE] dag/walk: vertex "root" is waiting for "scylladbcloud_cluster.aws"
2024-07-01T12:03:04.414-0400 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/scylladb/scylladbcloud\"] (close)" is waiting for "scylladbcloud_cluster.aws (expand)"
2024-07-01T12:03:04.414-0400 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/scylladb/scylladbcloud\"] (close)"
2024-07-01T12:03:04.465-0400 [TRACE] dag/walk: vertex "root" is waiting for "scylladbcloud_cluster.aws"

3. Rollback plugin, and see if terraform apply works:

terraform apply      
╷
│ Warning: Provider development overrides are in effect
│ 
│ The following provider development overrides are set in the CLI configuration:
│  - scylladb/scylladbcloud in /extra/scylladb/terraform-provider-scylladbcloud
│ 
│ The behavior may therefore not match any released version of the provider and applying changes may cause the state to become incompatible with published releases.
╵
random_string.postfix: Refreshing state... [id=7Xp1sK6w]
scylladbcloud_cluster.aws: Refreshing state... [id=6]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

[rjeczalik]

I wasn't able to delete the cluster after I approved the PR.

@ksinica Please take a look, I took different approach - I left the metadata loading on by default with an env var that can turn it off. Should be less error-prone.

[ksinica]

LGTM, I did a few quick manual tests and was able to create and destroy the clusters.