Repository navigation
Commit e979b7b
ci: don't let a dead Depot runner cancel a merge-queue run (#1376)
Between 22:15Z and 22:52Z on 2026-10-09, 58 of 65 failed Depot jobs
(merge_group, pull_request and push alike, on depot-ubuntu-24.04-4 and
-16) died because their runner went away: "Step canceled by GitHub",
"The runner has received a shutdown signal", or "lost communication".
The step-cancel deaths came 34-91 s (median ~63 s) after the job
started, whatever it ran (Checkout, clippy, cargo, unittest, bundler),
and the lost-communication ones are GitHub's 10-minute heartbeat timeout
on runners that died at about the same point. None of the victim runs
had been cancelled through the API first.
merge-queue-fail-fast.py treated each death as a failed job and
cancelled the whole run, so every dead runner evicted the queue head
and threw away its other jobs. It now reads the failed job's check-run
annotations and skips jobs whose runner died, leaving the run to finish
so ci-ok decides. A job re-run is not possible here: the jobs rerun API
refuses until the whole run has completed, by which time ci-ok has
reported.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>1 parent 92e001a commit e979b7b
2 files changed
Lines changed: 112 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
16 | 23 | | |
17 | 24 | | |
18 | 25 | | |
| |||
31 | 38 | | |
32 | 39 | | |
33 | 40 | | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
34 | 47 | | |
35 | 48 | | |
36 | 49 | | |
37 | 50 | | |
38 | | - | |
39 | | - | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
40 | 57 | | |
41 | 58 | | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
42 | 67 | | |
43 | 68 | | |
44 | 69 | | |
| |||
74 | 99 | | |
75 | 100 | | |
76 | 101 | | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
77 | 107 | | |
78 | 108 | | |
79 | 109 | | |
| |||
89 | 119 | | |
90 | 120 | | |
91 | 121 | | |
| 122 | + | |
92 | 123 | | |
93 | 124 | | |
94 | 125 | | |
| |||
100 | 131 | | |
101 | 132 | | |
102 | 133 | | |
103 | | - | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
104 | 143 | | |
105 | 144 | | |
106 | 145 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
| 4 | + | |
4 | 5 | | |
5 | 6 | | |
| 7 | + | |
6 | 8 | | |
| 9 | + | |
7 | 10 | | |
8 | 11 | | |
9 | 12 | | |
| |||
37 | 40 | | |
38 | 41 | | |
39 | 42 | | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
40 | 110 | | |
41 | 111 | | |
42 | 112 | | |
| |||
0 commit comments