Consider using a private cluster with Shared VPC

[macaptain]

With the current design, the Vault cluster isn't private and the nodes have external IP addresses:

% k get nodes -o wide | awk '{ print $1, $7}'                                                                                   (vault/default)
NAME EXTERNAL-IP
gke-vault-vault-node-pool-199ab1a5-cclv 35.247.182.159
gke-vault-vault-node-pool-199ab1a5-jc2d 35.240.209.196
gke-vault-vault-node-pool-199ab1a5-ks2x 34.87.75.11
gke-vault-vault-node-pool-9477141a-nrjn 35.240.209.142
gke-vault-vault-node-pool-9477141a-qbkq 35.198.233.255
gke-vault-vault-node-pool-9477141a-sl7m 34.87.125.4
gke-vault-vault-node-pool-95883582-3j54 34.87.43.9
gke-vault-vault-node-pool-95883582-nfmc 35.198.253.189
gke-vault-vault-node-pool-95883582-xlzz 35.240.142.150

This doesn't have to be so!

We can create a private cluster by adding this to the GKE cluster resource in Terraform:

 private_cluster_config {
    enable_private_nodes = true
    ...
  }

Then use a Shared VPC. We can run Vault in a separate project, the app project would then be the Host project, the Vault project is a Service project. Then there's no need for peering. For talking out to Datadog/Infura, we can define this in Cloud NAT.

Advantages:

• Reduce attack surface by isolating the nodes from the internet
• Simplifies project structure in development (only need one Vault project per project, rather than one Vault project per application cluster)