[p2] IAM role collision, suggested use of a fallback strategy #224

jordiroura-infotrust · 2024-11-01T22:55:15Z

When following the quick install, upon running Terraform it fails claiming the plugin crashed.

Looking at GCP logs, a fallback strategy is suggested indicating API throttling. I've tried at different times and still get the same error.

Plan: 10 to add, 0 to change, 0 to destroy.
module.data_store.module.dataform-workflow-prod[0].google_service_account.scheduler: Creating...
module.data_store.google_bigquery_dataset_iam_member.dataform-ga4-export-reader: Creating...
module.data_store.module.dataform-workflow-prod[0].google_workflows_workflow.dataform-incremental-workflow: Creating...
module.data_store.google_bigquery_dataset_iam_member.dataform-ads-export-reader[0]: Creating...
module.data_store.module.dataform-workflow-prod[0].google_project_iam_member.worflow-dataform-dataform-editor: Creating...
module.data_store.google_project_iam_member.dataform-serviceaccount["roles/bigquery.jobUser"]: Creating...
module.data_store.google_project_iam_member.dataform-bigquery-data-owner["roles/bigquery.dataOwner"]: Creating...
module.data_store.google_project_iam_member.dataform-serviceaccount["roles/secretmanager.secretAccessor"]: Creating...
╷
│ Error: Plugin did not respond
│ 
│   with module.data_store.google_project_iam_member.dataform-serviceaccount["roles/bigquery.jobUser"],
│   on modules/data-store/iam-binding.tf line 60, in resource "google_project_iam_member" "dataform-serviceaccount":
│   60: resource "google_project_iam_member" "dataform-serviceaccount" {
│ 
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may
│ contain more details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with module.data_store.google_project_iam_member.dataform-serviceaccount["roles/secretmanager.secretAccessor"],
│   on modules/data-store/iam-binding.tf line 60, in resource "google_project_iam_member" "dataform-serviceaccount":
│   60: resource "google_project_iam_member" "dataform-serviceaccount" {
│ 
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may
│ contain more details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with module.data_store.google_project_iam_member.dataform-bigquery-data-owner["roles/bigquery.dataOwner"],
│   on modules/data-store/iam-binding.tf line 77, in resource "google_project_iam_member" "dataform-bigquery-data-owner":
│   77: resource "google_project_iam_member" "dataform-bigquery-data-owner" {
│ 
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may
│ contain more details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with module.data_store.google_bigquery_dataset_iam_member.dataform-ga4-export-reader,
│   on modules/data-store/iam-binding.tf line 92, in resource "google_bigquery_dataset_iam_member" "dataform-ga4-export-reader":
│   92: resource "google_bigquery_dataset_iam_member" "dataform-ga4-export-reader" {
│ 
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may
│ contain more details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with module.data_store.google_bigquery_dataset_iam_member.dataform-ads-export-reader[0],
│   on modules/data-store/iam-binding.tf line 105, in resource "google_bigquery_dataset_iam_member" "dataform-ads-export-reader":
│  105: resource "google_bigquery_dataset_iam_member" "dataform-ads-export-reader" {
│ 
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may
│ contain more details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with module.data_store.module.dataform-workflow-prod[0].google_workflows_workflow.dataform-incremental-workflow,
│   on modules/dataform-workflow/dataform-workflow.tf line 23, in resource "google_workflows_workflow" "dataform-incremental-workflow":
│   23: resource "google_workflows_workflow" "dataform-incremental-workflow" {
│ 
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may
│ contain more details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with module.data_store.module.dataform-workflow-prod[0].google_service_account.scheduler,
│   on modules/dataform-workflow/service-account.tf line 15, in resource "google_service_account" "scheduler":
│   15: resource "google_service_account" "scheduler" {
│ 
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may
│ contain more details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with module.data_store.module.dataform-workflow-prod[0].google_project_iam_member.worflow-dataform-dataform-editor,
│   on modules/dataform-workflow/service-account.tf line 108, in resource "google_project_iam_member" "worflow-dataform-dataform-editor":
│  108: resource "google_project_iam_member" "worflow-dataform-dataform-editor" {
│ 
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may
│ contain more details.
╵

Stack trace from the terraform-provider-google_v5.44.1_x5 plugin:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1fd5e77]

goroutine 105 [running]:
github.com/hashicorp/terraform-provider-google/google/tpgiamresource.ResourceIamMember.resourceIamMemberCreate.func1.1(0x0)
	github.com/hashicorp/terraform-provider-google/google/tpgiamresource/resource_iam_member.go:217 +0x17
github.com/hashicorp/terraform-provider-google/google/tpgiamresource.iamPolicyReadModifyWrite({0x48e3d50, 0xc001169740}, 0xc001412eb0)
	github.com/hashicorp/terraform-provider-google/google/tpgiamresource/iam.go:104 +0x406
github.com/hashicorp/terraform-provider-google/google/tpgiamresource.ResourceIamMember.resourceIamMemberCreate.func1(0x0?, {0x40eeae0?, 0xc0005f9500})
	github.com/hashicorp/terraform-provider-google/google/tpgiamresource/resource_iam_member.go:225 +0x23e
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).create(0x48e1818?, {0x48e1818?, 0xc00140e1b0?}, 0xd?, {0x40eeae0?, 0xc0005f9500?})
	github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:766 +0x163
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc000f53c00, {0x48e1818, 0xc00140e1b0}, 0xc001402750, 0xc001374680, {0x40eeae0, 0xc0005f9500})
	github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:909 +0xa89
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0xc000f68360, {0x48e1818?, 0xc0013a64e0?}, 0xc00138c230)
	github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/grpc_provider.go:1078 +0xdbc
github.com/hashicorp/terraform-plugin-mux/tf5muxserver.(*muxServer).ApplyResourceChange(0x48e1850?, {0x48e1818?, 0xc0013a61e0?}, 0xc00138c230)
	github.com/hashicorp/[email protected]/tf5muxserver/mux_server_ApplyResourceChange.go:36 +0x193
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ApplyResourceChange(0xc0005a4640, {0x48e1818?, 0xc0012fd9b0?}, 0xc0012e9810)
	github.com/hashicorp/[email protected]/tfprotov5/tf5server/server.go:865 +0x3d0
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler({0x408f180?, 0xc0005a4640}, {0x48e1818, 0xc0012fd9b0}, 0xc0017d6a80, 0x0)
	github.com/hashicorp/[email protected]/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:518 +0x169
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000f6a400, {0x48e1818, 0xc0012fd920}, {0x48ecd28, 0xc00049fc80}, 0xc001334ea0, 0xc001038690, 0x6654cf8, 0x0)
	google.golang.org/[email protected]/server.go:1379 +0xe23
google.golang.org/grpc.(*Server).handleStream(0xc000f6a400, {0x48ecd28, 0xc00049fc80}, 0xc001334ea0)
	google.golang.org/[email protected]/server.go:1790 +0x1016
google.golang.org/grpc.(*Server).serveStreams.func2.1()
	google.golang.org/[email protected]/server.go:1029 +0x8b
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 7
	google.golang.org/[email protected]/server.go:1040 +0x135

Error: The terraform-provider-google_v5.44.1_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
[<ipython-input-9-d277c5c61f36>](https://localhost:8080/#) in <cell line: 1>()
----> 1 get_ipython().run_cell_magic('bash', '', 'export PATH="$PATH:~/.tfenv/bin"\nexport PATH="/root/.local/bin:$PATH"\nexport PATH="$PATH:$(which gcloud)"\nexport GOOGLE_APPLICATION_CREDENTIALS=/content/.config/application_default_credentials.json\nTERRAFORM_RUN_DIR=$(pwd)/infrastructure/terraform\nterraform -chdir="${TERRAFORM_RUN_DIR}" apply -auto-approve\n')

4 frames
<decorator-gen-103> in shebang(self, line, cell)

[/usr/local/lib/python3.10/dist-packages/IPython/core/magics/script.py](https://localhost:8080/#) in shebang(self, line, cell)
    243             sys.stderr.flush()
    244         if args.raise_error and p.returncode!=0:
--> 245             raise CalledProcessError(p.returncode, cell, output=out, stderr=err)
    246 
    247     def _run_script(self, p, cell, to_close):

CalledProcessError: Command 'b'export PATH="$PATH:~/.tfenv/bin"\nexport PATH="/root/.local/bin:$PATH"\nexport PATH="$PATH:$(which gcloud)"\nexport GOOGLE_APPLICATION_CREDENTIALS=/content/.config/application_default_credentials.json\nTERRAFORM_RUN_DIR=$(pwd)/infrastructure/terraform\nterraform -chdir="${TERRAFORM_RUN_DIR}" apply -auto-approve\n'' returned non-zero exit status 1.

The text was updated successfully, but these errors were encountered:

chmstimoteo · 2024-11-07T20:02:54Z

This seems to be instability on the network causing issues, we're using fixed versions of plugins previously tested and validated: https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart/blob/main/infrastructure/terraform/.terraform.lock.hcl#L46

Give it a try later.

chmstimoteo · 2024-11-07T20:18:08Z

It can also be a quota issue. Which links back to another Issue: #163

chmstimoteo · 2024-11-15T20:30:53Z

This happens on the Colab quick install notebook. Looking for a fix to this issue.

chmstimoteo · 2024-11-15T20:31:27Z

@jordiroura-infotrust did you find a fix for this issue?

chmstimoteo · 2024-11-26T17:19:56Z

Workaround:

Run terraform .. destroy

terraform -chdir="${TERRAFORM_RUN_DIR}" destroy -target=module.data_store -auto-approve
terraform -chdir="${TERRAFORM_RUN_DIR}" destroy -target=module.feature_store -auto-approve
terraform -chdir="${TERRAFORM_RUN_DIR}" destroy -target=module.activation -auto-approve

and deploy module by module:

terraform -chdir="${TERRAFORM_RUN_DIR}" apply -target=module.data_store -auto-approve
terraform -chdir="${TERRAFORM_RUN_DIR}" apply -target=module.feature_store -auto-approve
terraform -chdir="${TERRAFORM_RUN_DIR}" apply -target=module.pipelines -auto-approve
terraform -chdir="${TERRAFORM_RUN_DIR}" apply -target=module.activation -auto-approve
terraform -chdir="${TERRAFORM_RUN_DIR}" apply -target=module.monitoring -auto-approve

chmstimoteo self-assigned this Nov 4, 2024

chmstimoteo added invalid This doesn't seem right duplicate This issue or pull request already exists labels Nov 7, 2024

chmstimoteo added bug Something isn't working and removed invalid This doesn't seem right labels Nov 15, 2024

chmstimoteo changed the title ~~IAM role collision, suggested use of a fallback strategy~~ [p0] IAM role collision, suggested use of a fallback strategy Nov 20, 2024

chmstimoteo added enhancement New feature or request and removed bug Something isn't working duplicate This issue or pull request already exists labels Dec 2, 2024

chmstimoteo changed the title ~~[p0] IAM role collision, suggested use of a fallback strategy~~ [p2] IAM role collision, suggested use of a fallback strategy Dec 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[p2] IAM role collision, suggested use of a fallback strategy #224

[p2] IAM role collision, suggested use of a fallback strategy #224

jordiroura-infotrust commented Nov 1, 2024

chmstimoteo commented Nov 7, 2024

chmstimoteo commented Nov 7, 2024

chmstimoteo commented Nov 15, 2024

chmstimoteo commented Nov 15, 2024

chmstimoteo commented Nov 26, 2024

[p2] IAM role collision, suggested use of a fallback strategy #224

[p2] IAM role collision, suggested use of a fallback strategy #224

Comments

jordiroura-infotrust commented Nov 1, 2024

chmstimoteo commented Nov 7, 2024

chmstimoteo commented Nov 7, 2024

chmstimoteo commented Nov 15, 2024

chmstimoteo commented Nov 15, 2024

chmstimoteo commented Nov 26, 2024