Cloudformation stack creation fails due to insufficient Elastic IPs

[MikeTheCanuck]

When creating a new stack based on our CloudFormation stack templates, I have repeatedly run into situations where the stack creation fails and rolls back, claiming some combination of failures to do with the VPC and its resources, e.g.:

[MikeTheCanuck]

I've learned that this problem has a viable (if frustrating) explanation:

• the resources that repeatedly CREATE_FAILED are Internet Gateway and VPC

When digging through the Stacks in CloudFormation, I finally noticed the Deleted filter (whose stacks are hidden from the default filter) and was able to find a single, repeated type of error from this morning's repeated failures:

[MikeTheCanuck]

Well, it appears that we're allocating "too many" Elastic IPs in our account/region:
hashicorp/terraform#6018 (comment)

And this will be a problem, since we've already got 4 of the 5 allowable addresses allocated:

This is where the limit is documented:
https://docs.aws.amazon.com/vpc/latest/userguide/amazon-vpc-limits.html#vpc-limits-eips

[MikeTheCanuck]

So for us to continue developing a test-vs-staging-vs-production infrastructure (plus all the one-off deploys from other teams), we're likely to need to request an increase in the Elastic IPs limit on our account - otherwise, only one test stack can be deployed at a time, making it very hard for multiple engineers to work simultaneously.

Requesting an increase to these limits has one somewhat-mystifying question we need to resolve:
https://console.aws.amazon.com/support/home#/case/create?issueType=service-limit-increase&limitType=vpc

Are we currently rate-limited on "EC2-Classic Elastic IP Addresses" or "New VPC Elastic IP Addresses"?

[MikeTheCanuck]

Here's how to request an increase:
https://stackoverflow.com/questions/24275080/aws-eip-limit-increase

[DingoEatingFuzz]

I just mentioned this in #239, but I'll say it here also.

I don't think we need separate VPCs in order to have isolated test/staging/production environments.

Budget aside, as long as we have separate ECS clusters and ALBs, then we have isolation.

Including budgets, it's important to be aware that a single NAT gateway costs ~$30/mo.

[danieldn]

Will merge this with #239