Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: add retry step to terragrunt apply #444

Closed
wants to merge 5 commits into from
Closed

fix: add retry step to terragrunt apply #444

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
37ece17
fix: add retry step to terragrunt apply
samuelarogbonlo May 10, 2024
890f8ec
fix terrafrom ci
samuelarogbonlo May 10, 2024
440f2cc
Merge branch 'main' into samuel/add-retry
samuelarogbonlo May 14, 2024
9620ebb
add create before destroy flag
samuelarogbonlo May 14, 2024
d50b36e
fix fmt ci
samuelarogbonlo May 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/deploy-daily-snapshot.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ jobs:
r2_access_key: ${{ secrets.R2_ACCESS_KEY }}
r2_secret_key: ${{ secrets.R2_SECRET_KEY }}
slack_token: ${{ secrets.SLACK_TOKEN }}
max_retries: '5'
retry_delay_seconds: '60s'
working_directory: tf-managed/live/environments/prod/applications/${{ matrix.replica }}
service_name: ${{ matrix.replica }}
new_relic_account_id: ${{ secrets.NEW_RELIC_ACCOUNT_ID }}
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/deploy-forest-butterflynet.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@ jobs:
aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
working_directory: tf-managed/live/environments/prod/applications/forest-butterflynet
service_name: forest-butterflynet
max_retries: '5'
retry_delay_seconds: '60s'
new_relic_account_id: ${{ secrets.NEW_RELIC_ACCOUNT_ID }}
new_relic_api_key: ${{ secrets.NEW_RELIC_API_KEY }}
ssh_private_key: ${{ secrets.SSH_PRIVATE_KEY }}
2 changes: 2 additions & 0 deletions .github/workflows/deploy-forest-calibnet.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@ jobs:
aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
working_directory: tf-managed/live/environments/prod/applications/forest-calibnet
service_name: forest-calibnet
max_retries: '5'
retry_delay_seconds: '60s'
new_relic_account_id: ${{ secrets.NEW_RELIC_ACCOUNT_ID }}
new_relic_api_key: ${{ secrets.NEW_RELIC_API_KEY }}
ssh_private_key: ${{ secrets.SSH_PRIVATE_KEY }}
2 changes: 2 additions & 0 deletions .github/workflows/deploy-forest-mainnet.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@ jobs:
aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
working_directory: tf-managed/live/environments/prod/applications/forest-mainnet
service_name: forest-mainnet
max_retries: '5'
retry_delay_seconds: '60s'
new_relic_account_id: ${{ secrets.NEW_RELIC_ACCOUNT_ID }}
new_relic_api_key: ${{ secrets.NEW_RELIC_API_KEY }}
ssh_private_key: ${{ secrets.SSH_PRIVATE_KEY }}
2 changes: 2 additions & 0 deletions .github/workflows/deploy-snapshot-monitoring.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ jobs:
slack_token: ${{ secrets.SLACK_TOKEN }}
working_directory: tf-managed/live/environments/prod/applications/snapshot-monitoring
service_name: Snapshot Service
max_retries: '5'
retry_delay_seconds: '60s'
new_relic_account_id: ${{ secrets.NEW_RELIC_ACCOUNT_ID }}
new_relic_api_key: ${{ secrets.NEW_RELIC_API_KEY }}
ssh_private_key: ${{ secrets.SSH_PRIVATE_KEY }}
2 changes: 2 additions & 0 deletions .github/workflows/deploy-sync-check.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ jobs:
slack_token: ${{ secrets.SLACK_TOKEN }}
working_directory: tf-managed/live/environments/prod/applications/sync-check
service_name: Sync Check Service
max_retries: '5'
retry_delay_seconds: '60s'
new_relic_account_id: ${{ secrets.NEW_RELIC_ACCOUNT_ID }}
new_relic_api_key: ${{ secrets.NEW_RELIC_API_KEY }}
ssh_private_key: ${{ secrets.SSH_PRIVATE_KEY }}
24 changes: 23 additions & 1 deletion composite-action/terragrunt/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,12 @@ inputs:
description: 'CloudFlare R2 access key id'
r2_secret_key:
description: 'CloudFlare R2 private access key'
max_retries:
description: 'The maximum number of retry attempts for the Terragrunt apply command if it fails initially.'
required: true
retry_delay_seconds:
description: 'The number of seconds to wait between each retry attempt.'
required: true

runs:
using: "composite"
Expand Down Expand Up @@ -155,7 +161,22 @@ runs:
else
echo "Changes detected. Redeploying everything..."
terragrunt destroy -auto-approve --terragrunt-non-interactive
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this is the issue that should be addressed. We should use lifecycle.create_before_destroy and not force-destroy the resource - it should get destroyed after the new one is created successfully, leaving it untouched in case of errors. What do you think? We can do it in a separate issue/PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah makes sense. i will implement in this PR

terragrunt apply -auto-approve --terragrunt-non-interactive
max_retries=${{ inputs.max_retries }}
delay_seconds=${{ inputs.retry_delay_seconds }}
n=0
while [ $n -lt $max_retries ]; do
echo "Attempt $(($n + 1)) of $max_retries"
terragrunt apply -auto-approve --terragrunt-non-interactive && break
n=$((n+1))
if [ $n -lt $max_retries ]; then
echo "Retry $n/$max_retries of terragrunt apply failed. Retrying in $delay_seconds seconds..."
sleep $delay_seconds
fi
done
if [ $n -eq $max_retries ]; then
echo "Failed to apply changes after $max_retries attempts."
exit 1
fi
fi
working-directory: ${{ inputs.working_directory }}
env:
Expand All @@ -170,6 +191,7 @@ runs:
TF_VAR_new_relic_api_key: ${{ inputs.new_relic_api_key }}
TF_VAR_new_relic_account_id: ${{ inputs.new_relic_account_id }}


- name: Terragrunt Force Apply
if: github.ref == 'refs/heads/main' && github.event_name == 'workflow_dispatch'
working-directory: ${{ inputs.working_directory }}
Expand Down
11 changes: 7 additions & 4 deletions tf-managed/modules/daily-snapshot/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,10 +75,9 @@ resource "digitalocean_droplet" "forest" {
graceful_shutdown = false

connection {
host = self.ipv4_address
user = "root"
type = "ssh"
timeout = "30m"
LesnyRumcajs marked this conversation as resolved.
Show resolved Hide resolved
host = self.ipv4_address
user = "root"
type = "ssh"
}

# Push the sources.tar file to the newly booted droplet
Expand All @@ -90,6 +89,10 @@ resource "digitalocean_droplet" "forest" {
provisioner "remote-exec" {
inline = local.init_commands
}

lifecycle {
create_before_destroy = true
}
}


Expand Down
10 changes: 5 additions & 5 deletions tf-managed/modules/forest-droplet/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,9 @@ resource "digitalocean_droplet" "forest" {
graceful_shutdown = false

connection {
host = self.ipv4_address
user = "root"
type = "ssh"
timeout = "30m"
host = self.ipv4_address
user = "root"
type = "ssh"
}

provisioner "file" {
Expand All @@ -56,7 +55,8 @@ resource "digitalocean_droplet" "forest" {
}

lifecycle {
replace_triggered_by = [local_sensitive_file.bootstrap_script]
replace_triggered_by = [local_sensitive_file.bootstrap_script]
create_before_destroy = true
}
}

Expand Down
10 changes: 6 additions & 4 deletions tf-managed/modules/sync-check/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,10 +74,9 @@ resource "digitalocean_droplet" "forest" {
graceful_shutdown = false

connection {
host = self.ipv4_address
user = "root"
type = "ssh"
timeout = "30m"
host = self.ipv4_address
user = "root"
type = "ssh"
}

# Push the sources.tar file to the newly booted droplet
Expand All @@ -89,6 +88,9 @@ resource "digitalocean_droplet" "forest" {
provisioner "remote-exec" {
inline = local.init_commands
}
lifecycle {
create_before_destroy = true
}
}

data "digitalocean_project" "forest_project" {
Expand Down
Loading