Resources not cleaned up from test, incurring ongoing charges in Aiven account

[AaronFriel]

• Suspend tests
• Document account (likely [email protected]) used: answer - in engineering team 1password vault.
• Clean up outstanding resources in account
• Follow up with @isaac-pulumi on approach to re-enable tests or future steps.

[AaronFriel]

Removing P1 and keeping this as a tracking issue.

[guineveresaenger]

I recommend a cleaner-upper, which has been used as a solution in other providers: https://github.com/pulumi/civo-account-cleanup

[t0yv0]

Just merged something that enables tests back, should they still be skipped?

[AaronFriel]

We should keep this open for a while, I'll track our Aiven portal and cron workflow to see if we're leaving resources alive. There's an upstream issue enabling us to tweak timeouts via a fork + patch or ldflags:

• Error creating resources: unable to wait for service state change: context deadline exceeded aiven/terraform-provider-aiven#1054

[t0yv0]

Also pulumi/ci-mgmt#379 perhaps relevant.

I don't mind keeping this open, just FYI more tests are running now in Aiven that were skipped before today. Just in case this causes further problems.

[AaronFriel]

In the case of this issue, the cleanup failed due to provider timeouts. The physical resource creation succeeded async after the provider timed out, so the logical resource create call failed. The state file thus had no resources to clean up when programtest ran pulumi destroy.