Skip to content

Ensure graceful shutdown for Edge Tasks in Airflow 3 #60086

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jscheffl wants to merge 1 commit into from
Closed

Ensure graceful shutdown for Edge Tasks in Airflow 3 #60086

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions providers/edge3/src/airflow/providers/edge3/cli/worker.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,8 @@ def signal_handler(sig: signal.Signals, frame):
def shutdown_handler(self, sig, frame):
logger.info("SIGTERM received. Terminating all jobs and quit")
for job in EdgeWorker.jobs:
os.killpg(job.process.pid, signal.SIGTERM)
if job.process.pid:
os.kill(job.process.pid, signal.SIGTERM)
EdgeWorker.drain = True

def _get_sysinfo(self) -> dict:
Expand Down Expand Up @@ -193,7 +194,7 @@ def _run_job_via_supervisor(workload, execution_api_server_url) -> int:
from airflow.sdk.execution_time.supervisor import supervise

# Ignore ctrl-c in this process -- we don't want to kill _this_ one. we let tasks run to completion
signal.signal(signal.SIGINT, signal.SIG_IGN)
os.setpgrp()

logger.info("Worker starting up pid=%d", os.getpid())
setproctitle(f"airflow edge worker: {workload.ti.key}")
Expand Down