fix: retry failed ssl renewal with jittered backoff 🐛

[meysam81]

fixes #338

For your reference, you can see the templated file below:

#!/bin/bash
#
# This file is managed by Ansible. Only change it through the rust-lang/simpleinfra repository.
#
set -euo pipefail
IFS=$'\n\t'

# During the initial renew nginx will fail to run as the certificate is
# missing, so lego will have to serve port 80.
if [[ $# -eq 1 ]] && [[ $1 = "initial-renew" ]]; then
    renew_kind="--http.port=:80"
    action="run"
else
    renew_kind="--http.webroot=/var/run/acme-challenges"
    action="renew"
fi

# Use the local Pebble instance to get a dummy certificate
server="https://localhost:14000/dir"
# Use Pebble's CA to validate connections to it
export LEGO_CA_CERTIFICATES=/etc/pebble/certs/pebble.minica.pem
# Remove the account as Pebble doesn't persist it
rm -rf "/etc/ssl/letsencrypt/accounts/localhost_14000/[email protected]"

retries="10"
wait_time="5"

lego_cmd="lego --email '[email protected]' \
        --server '${server}' \
        --accept-tos \
        --path /etc/ssl/letsencrypt \
        --http \
        ${renew_kind} \
        -d 'dummy' \
        ${action}"

function run_with_retries {
    command=$1
    local i=0
    while true; do
        ${command}
        exit_code=$?

        if [ ${exit_code} -eq 0 ]; then
            break
        fi

        if [ ${i} -ge ${retries} ]; then
            exit ${exit_code}
        fi

        jitter=$(($RANDOM % 10))
        wait_time_with_jitter=$((${wait_time} + ${jitter}))

        echo "Command failed with exit code ${exit_code}. Retrying in ${wait_time_with_jitter} seconds..."
        sleep ${wait_time_with_jitter}

        i=$(($i + 1))
    done
}

set +e
run_with_retries $lego_cmd
set -e

sudo /etc/ssl/letsencrypt/after-renew

Running the executable on the machine gave me the output you can see in the screenshot below.

[meysam81]

@jdno Please have a look and let me know if there's anything missing 🙏

[Mark-Simulacrum]

Can we structure this implementation to use retries at the systemd level (https://github.com/rust-lang/simpleinfra/blob/master/ansible/roles/letsencrypt/templates/renew-ssl-certs.service)? I feel like that might be less ad-hoc in terms of implementation and a little cleaner than the current solution.

[shepmaster]

Superseded by #393