Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Terraform for production environment #3641

Merged
merged 26 commits into from
Feb 20, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
bbca0ee
Remove old IaC configuration [#3365]
ohrite Feb 4, 2025
3e2196b
Add cal-itp-data-infra-staging terraform to iac directory
ohrite Feb 10, 2025
6858b00
Add cal-itp-data-infra terraform to iac directory
ohrite Feb 10, 2025
424afed
Update cal-itp-data-infra terraform to add us-west2
ohrite Feb 13, 2025
fb744ba
Add terraform validation
ohrite Feb 13, 2025
d9c91d9
Format staging terraform files
ohrite Feb 13, 2025
ffc6fa7
Fix target resolution
ohrite Feb 13, 2025
fcf52c0
Disable github commenting for terraform plan
ohrite Feb 13, 2025
366d9b7
Combine all Terraform linting
ohrite Feb 13, 2025
a1451f8
Set terraform google provider version to 4.59.0
ohrite Feb 13, 2025
8070fdf
Disable fast failures during validation
ohrite Feb 13, 2025
6c1fccc
Reimport cal-itp-data-infra-staging using Terraformer
ohrite Feb 14, 2025
72f0fc5
Ignore root-level terraform files
ohrite Feb 14, 2025
dc90431
Remove empty resource definitions from staging and production
ohrite Feb 18, 2025
348b85b
Update GCS IAM membership in Terraform scripts
ohrite Feb 18, 2025
ebcd9bd
Clean terraform run against staging and production
erikamov Feb 18, 2025
ac815a7
Use Workload Identity Provider in Github Action
ohrite Feb 19, 2025
020dc0b
Split production and staging validation
ohrite Feb 19, 2025
ccbcf55
Update workflow and service account
ohrite Feb 19, 2025
c19e0f9
Remove jupyterhub disks and instance group resources
ohrite Feb 19, 2025
a027ec4
Add makefile to run Terraform locally
ohrite Feb 19, 2025
9db8947
Add Terraform apply Github action
ohrite Feb 19, 2025
d60bdda
Move staging state to gcs
ohrite Feb 19, 2025
0a26441
Move production state to gcs
ohrite Feb 19, 2025
bccf94d
Do not include local tfstate
ohrite Feb 20, 2025
d55c8a9
Add labels to terraform jobs
ohrite Feb 20, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 106 additions & 0 deletions .github/workflows/terraform-apply.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
name: Terraform Apply

on:
push:
branches:
- 'main'
paths:
- 'iac/*'

permissions:
contents: read
pull-requests: write

jobs:
targets:
name: Find targets

runs-on: ubuntu-latest

outputs:
staging: ${{ steps.staging.outputs.paths }}
production: ${{ steps.production.outputs.paths }}

steps:
- name: Checkout
uses: actions/checkout@v4

- name: Staging Terraform targets
id: staging
run: echo "paths=$(find iac -name 'provider.tf' | grep cal-itp-data-infra-staging/ | xargs dirname | jq --raw-input --slurp --compact-output 'split("\n")[:-1]')" >> ${GITHUB_OUTPUT}

- name: Production Terraform targets
id: production
run: echo "paths=$(find iac -name 'provider.tf' | grep cal-itp-data-infra/ | xargs dirname | jq --raw-input --slurp --compact-output 'split("\n")[:-1]')" >> ${GITHUB_OUTPUT}

staging:
name: Staging

needs: targets

runs-on: ubuntu-latest

permissions:
contents: 'read'
id-token: 'write'

strategy:
fail-fast: false
matrix:
path: ${{ fromJson(needs.targets.outputs.staging) }}

steps:
- name: Checkout
uses: actions/checkout@v4

- uses: 'google-github-actions/auth@v2'
with:
create_credentials_file: 'true'
project_id: cal-itp-data-infra-staging
workload_identity_provider: 'projects/473674835135/locations/global/workloadIdentityPools/github-actions-pool/providers/github-actions-provider'
service_account: 'github-actions-terraform@cal-itp-data-infra-staging.iam.gserviceaccount.com'

- uses: google-github-actions/setup-gcloud@v2

- name: Terraform Apply
uses: dflook/terraform-apply@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
path: ${{ matrix.path }}

production:
name: Production

needs: targets

runs-on: ubuntu-latest

permissions:
contents: 'read'
id-token: 'write'

strategy:
fail-fast: false
matrix:
path: ${{ fromJson(needs.targets.outputs.production) }}

steps:
- name: Checkout
uses: actions/checkout@v4

- uses: 'google-github-actions/auth@v2'
with:
create_credentials_file: 'true'
project_id: cal-itp-data-infra
workload_identity_provider: 'projects/1005246706141/locations/global/workloadIdentityPools/github-actions-pool/providers/github-actions-provider'
service_account: 'github-actions-terraform@cal-itp-data-infra.iam.gserviceaccount.com'

- uses: google-github-actions/setup-gcloud@v2

- name: Terraform Apply
uses: dflook/terraform-apply@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
path: ${{ matrix.path }}
48 changes: 0 additions & 48 deletions .github/workflows/terraform-deploy.yml

This file was deleted.

122 changes: 122 additions & 0 deletions .github/workflows/terraform-plan.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
name: Terraform Plan

on:
pull_request:
paths:
- 'iac/*'

jobs:
targets:
name: Find targets

runs-on: ubuntu-latest

outputs:
staging: ${{ steps.staging.outputs.paths }}
production: ${{ steps.production.outputs.paths }}

steps:
- name: Checkout
uses: actions/checkout@v4

- name: Staging Terraform targets
id: staging
run: echo "paths=$(find iac -name 'provider.tf' | grep cal-itp-data-infra-staging/ | xargs dirname | jq --raw-input --slurp --compact-output 'split("\n")[:-1]')" >> ${GITHUB_OUTPUT}

- name: Production Terraform targets
id: production
run: echo "paths=$(find iac -name 'provider.tf' | grep cal-itp-data-infra/ | xargs dirname | jq --raw-input --slurp --compact-output 'split("\n")[:-1]')" >> ${GITHUB_OUTPUT}

staging:
name: Staging

needs: targets

runs-on: ubuntu-latest

permissions:
contents: 'read'
id-token: 'write'

strategy:
fail-fast: false
matrix:
path: ${{ fromJson(needs.targets.outputs.staging) }}

steps:
- name: Checkout
uses: actions/checkout@v4

- uses: 'google-github-actions/auth@v2'
with:
create_credentials_file: 'true'
project_id: cal-itp-data-infra-staging
workload_identity_provider: 'projects/473674835135/locations/global/workloadIdentityPools/github-actions-pool/providers/github-actions-provider'
service_account: 'github-actions-terraform@cal-itp-data-infra-staging.iam.gserviceaccount.com'

- uses: google-github-actions/setup-gcloud@v2

- name: Terraform Formatting
uses: dflook/terraform-fmt-check@v1
with:
path: ${{ matrix.path }}

- name: Terraform Validation
uses: dflook/terraform-validate@v1
with:
path: ${{ matrix.path }}

- name: Terraform Plan
uses: dflook/terraform-plan@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
add_github_comment: changes-only
path: ${{ matrix.path }}

production:
name: Production

needs: targets

runs-on: ubuntu-latest

permissions:
contents: 'read'
id-token: 'write'

strategy:
fail-fast: false
matrix:
path: ${{ fromJson(needs.targets.outputs.production) }}

steps:
- name: Checkout
uses: actions/checkout@v4

- uses: 'google-github-actions/auth@v2'
with:
create_credentials_file: 'true'
project_id: cal-itp-data-infra
workload_identity_provider: 'projects/1005246706141/locations/global/workloadIdentityPools/github-actions-pool/providers/github-actions-provider'
service_account: 'github-actions-terraform@cal-itp-data-infra.iam.gserviceaccount.com'

- uses: google-github-actions/setup-gcloud@v2

- name: Terraform Format Check
uses: dflook/terraform-fmt-check@v1
with:
path: ${{ matrix.path }}

- name: Terraform Validate
uses: dflook/terraform-validate@v1
with:
path: ${{ matrix.path }}

- name: Terraform Plan
uses: dflook/terraform-plan@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
add_github_comment: changes-only
path: ${{ matrix.path }}
70 changes: 0 additions & 70 deletions .github/workflows/terraform-report.yml

This file was deleted.

1 change: 0 additions & 1 deletion iac/.engine
Submodule .engine deleted from 3175c1
6 changes: 6 additions & 0 deletions iac/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
.terraform/
.terraform.tfstate.*.backup
terraform.tfstate.backup
terraform.tfstate
/.terraform.lock.hcl
/provider.tf
9 changes: 9 additions & 0 deletions iac/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
TARGETS := init plan apply fmt migrate-state

PATHS := $(wildcard */.)

$(TARGETS): $(PATHS)
$(PATHS):
$(MAKE) -C $@ $(MAKECMDGOALS)

.PHONY: $(TARGETS) $(PATHS)
Loading