feat: Add encrypted .env creds

[kgpayne]

Closes #463

• Add encrypted .env creds #463

[kgpayne]

@aaronsteers I have tested this with a publickey from the sandbox account and, as expected, our Google Analytics credentials are too long to fit into one ciphertext. I have created this issue to track chunking long plaintext into arrays of ciphertexts in secrets.yml, but we need not necessarily block on it.

[aaronsteers]

Confirmed:

• We don't have a critical dependency on google analytics data as of now, so we can ignore the GA creds issue as of now.

Related:

• This adds a KMS key with permissions: https://github.com/meltano/dragon-infra/pull/91
• We can preinstall the kms-ext utility in the dockerfile here: https://github.com/meltano/dragon-infra/blob/main/deployments/06_project_orchestrate/docker/Dockerfile
• We can decrypt the contents into a .env file on the running container here: https://github.com/meltano/dragon-infra/blob/main/deployments/06_project_orchestrate/docker/entry.sh

Still TBD/TODO:

• How to deliver the public key (.pem file or similar) to the project owner. @magreenbaum - any thoughts on this?
• Even if using a short-term approach, can we get Ken a public key for the squared project today or tomorrow?

cc @magreenbaum, @WillDaSilva

[magreenbaum]

How to deliver the public key (.pem file or similar) to the project owner. @magreenbaum - any thoughts on this?

Maybe a meltano command that downloads it. I would just need to add kms:GetPublicKey permissions for the container. https://docs.aws.amazon.com/kms/latest/APIReference/API_GetPublicKey.html

[magreenbaum]

Even if using a short-term approach, can we get Ken a public key for the squared project today or tomorrow?

We can manually generate a KMS key for now or we can deploy the infra off https://github.com/meltano/dragon-infra/pull/91 and see how it goes in the new staging account. Right now nothing is deployed in staging.

[magreenbaum]

I've got this PR deployed into our new staging account. If it looks good to everyone, I'll merge it and share the public key with @kgpayne. cc @aaronsteers

[pnadolny13]

@kgpayne this looks great!

As a side not since I'm curious - are we expecting to have one of these secrets.yml files for each meltano environment? It would be really helpful in CI to just pull secrets from KMS vs storing them in GitHubs secret store. Having a secrets key attached to each meltano env seems super useful, each slightly different than each other.

[pnadolny13]

@kgpayne are these env variables required in the other meltano environments (cicd/staging/prod) as well?

[pnadolny13]

@aaronsteers I have tested this with a publickey from the sandbox account and, as expected, our Google Analytics credentials are too long to fit into one ciphertext. I have created meltano/kms-ext#1 to track chunking long plaintext into arrays of ciphertexts in secrets.yml, but we need not necessarily block on it.

@kgpayne I'm not sure what the size limit is exactly but fwiw we could split the TAP_GOOGLE_ANALYTICS_CLIENT_SECRETS secret into its parts using the dot syntax so they live in the meltano.yml file since most of them arent sensitive, like:

      client_secrets:
        type: service_account
        token_uri: https://oauth2.googleapis.com/token

and only leave the truly secret sections like private_key and private_key_id? The private key in that GA credentials json is almost 1800 characters in itself so thats mostly the problem.

[aaronsteers]

https://github.com/meltano/squared/actions/runs/3482409891/jobs/5824784907

2022-11-16T19:28:26.430364Z [info     ] 19:28:26  Database Error in test source_unique_tap_spreadsheets_anywhere_azure_ips_id (snapshots/spreadsheets_anywhere/sources.yml) cmd_type=command name=dbt-snowflake stdio=stderr
2022-11-16T19:28:26.431071Z [info     ] 19:28:26    002003 (42S02): SQL compilation error: cmd_type=command name=dbt-snowflake stdio=stderr
2022-11-16T19:28:26.431896Z [info     ] 19:28:26    Object 'CICD_RAW.BFBC13D0FB1F7549736BC8FFF3A544484D893C035_TAP_SPREADSHEETS_ANYWHERE.AZURE_IPS' does not exist or not authorized. cmd_type=command name=dbt-snowflake stdio=stderr

[pnadolny13]

@aaronsteers related to the failing test I created #480 that should fix it, once thats pulled in this PR should pass. Optionally since this is high priority we can force merge since this PR doesnt affect anything thats being tested anyways.

#354 is the actual issue that periodically causes the test to fail since we dont have a way to dynamically generate the url yet.