Tasks: Graceful stopping and restarting

[nphilipp]

Story

As an operator of Duffy,
I want that running tasks are finished when stopping or restarting the task workers,
so that the work won’t be restarted unnecessarily and all provisioned nodes are known to Duffy.

Acceptance Criteria

• The task worker systemd unit file is adapted to wait reasonably long so current tasks can finish.

Background

Stopping the task workers while e.g. nodes are provisioned will finish the running Ansible task, so these nodes exist and use up resources, but aren’t known to Duffy and won’t ever get used.

Celery documentation describes how to structure a systemd service accordingly:

https://docs.celeryq.dev/en/latest/userguide/daemonizing.html#usage-systemd

This uses the celery multi command instead of worker, i.e. might need to change how Duffy invokes Celery.