Skip to content

Fix SSHRemoteJobOperator still orphaning the remote job on cancellation#69490

Open
kaxil wants to merge 1 commit into
apache:mainfrom
astronomer:fix-ssh-remote-job-pgid-orphan
Open

Fix SSHRemoteJobOperator still orphaning the remote job on cancellation#69490
kaxil wants to merge 1 commit into
apache:mainfrom
astronomer:fix-ssh-remote-job-pgid-orphan

Conversation

@kaxil

@kaxil kaxil commented Jul 6, 2026

Copy link
Copy Markdown
Member

When an SSHRemoteJobOperator task is cancelled, on_kill signals the process group recorded in the job's pid file so the whole job tree is torn down. The detached job launches under setsid to lead its own process group, and the wrapper recorded $! as that group's leader.

$! is the group leader only when setsid(1) runs in place. If job control is on in the launching shell, setsid(1) forks (setsid(2) cannot create a new session from an existing process-group leader), so $! names the short-lived setsid parent instead of the job. Cancellation then signals a dead group, the real command keeps running as an orphan, and because the wrapper subshell is what writes the exit_code file, that file never appears and the task stays deferred until the trigger times out.

Fix

Record the job's own pid instead of the launcher's $!. Right after setsid(2), POSIX guarantees pid == pgid == sid for the caller, and that identity survives the following exec, so $$ written from inside the job script is always the true process-group id, whether or not setsid forked. The launcher no longer records $! at all, and build_posix_kill_command is unchanged.

The pid file is read only at cancellation time, in on_kill, which runs long after submission, so the job's own write is reliably in place before any reader and the launcher does not need to wait for it.

Relationship to #68644

#68644 added the setsid + process-group-kill behavior but assumed $! was always the group leader. That holds only when the launching shell is not already a process-group leader. Submission runs with get_pty=False, so the default path is unaffected. The orphaning surfaces on remotes where job control ends up on: a controlling terminal, a ForceCommand wrapper, or set -m in the remote user's shell init.

Regression test (follow-up to #69384)

The end-to-end kill test was marked @pytest.mark.flaky(reruns=5) in #69384 because it depended on incidental process-group timing. It now forces job control through a real controlling terminal (pty.fork() + bash -mc), exercising the exact fork path that orphaned the job, and asserts the recorded pid equals the job's true process-group id before checking teardown. The flaky marker is removed; test markers are unique per test and per worker so the suite stays isolated under pytest -n auto.

Gotcha

On hosts without setsid (some macOS/BSD images) the job is not a process-group leader, so $$ is just its own pid and cancellation degrades to the previous single-process kill, unchanged from before.

The detached job runs under `setsid` so it leads its own process group, and
`on_kill` signals that group. The wrapper recorded `$!` as the group leader,
but `$!` is only the leader when `setsid(1)` runs in place. When job control is
on in the launching shell, `setsid(1)` forks (`setsid(2)` cannot create a new
session from an existing group leader), so `$!` names the short-lived setsid
parent, not the job. Cancellation then signals a dead group, the real command
keeps running as an orphan, and the `exit_code` file is never written, so the
task stays deferred until the trigger times out.

Record the job's own pid instead: right after `setsid(2)`, POSIX guarantees
`pid == pgid == sid` for the caller and that identity survives the following
`exec`, so `$$` written from inside the job script is always the true PGID,
whether or not setsid forked. The launcher no longer records `$!`, and since
the pid file is read only at cancellation time (long after submission) the
job's own write is always in place first.

Make the end-to-end kill test deterministic by forcing job control through a
real controlling terminal, exercising the exact fork path that orphaned the
job, and remove the `flaky(reruns=5)` marker. Test markers are unique per test
and per worker so the suite stays isolated under `pytest -n auto`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant