Skip to content

[ci] Add Datadog monitor definitions for unattended workflows - #98557

Draft
eps1lon wants to merge 1 commit into
canaryfrom
sebbie/ci-workflow-health-monitors
Draft

[ci] Add Datadog monitor definitions for unattended workflows#98557
eps1lon wants to merge 1 commit into
canaryfrom
sebbie/ci-workflow-health-monitors

Conversation

@eps1lon

@eps1lon eps1lon commented Sep 11, 2026

Copy link
Copy Markdown
Member

Adds two import-ready Datadog monitor definitions under scripts/datadog/workflow-health/ covering the workflows that run without intentional user input: the scheduled (cron) workflows and the ones driven purely by workflow_run, issues, or issue_comment events. An audit of these workflows found that several had been failing unnoticed for weeks (four of them with startup_failure since 2026-08-21), which is the gap these monitors close.

failure-monitor.json is a multi-alert ci-pipelines alert that notifies on the very first failed run of each workflow, with no consecutive-failure tolerance, and its message links the specific failed execution via the ci.pipeline.url attribute. skip-rate-monitor.json warns when a workflow's share of skipped runs over the trailing day exceeds 95 percent and alerts at 100 percent, where every completed run was skipped, which usually means a condition broke and the workflow silently stopped doing work. Both monitors keep messages minimal, notify on recovery and on no data, and renotify once a week while they stay in a triggered state. For the skip-rate monitor, whose queries match every run, the no-data path also acts as the detector for the GitHub integration itself stopping to report.

Nothing applies these definitions automatically. After merging, import each JSON via https://app.datadoghq.com/monitors/create/import and replace @slack-TODO with the real channel handle. The README lists the covered workflows (kept as a plain list so it can be re-derived from the trigger criteria) and the assumptions to verify in the CI Pipelines explorer before relying on the monitors: pipeline names match the workflow display names, skipped runs carry a distinguishable status, startup_failure runs surface as error events, and the ci.pipeline.url variable renders a run URL in the alert message.

@github-actions

github-actions Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Tests Passed

Commit: 077c11b

@eps1lon
eps1lon force-pushed the sebbie/ci-workflow-health-monitors branch from 56b5ff1 to 80f034b Compare September 11, 2026 12:09
@eps1lon eps1lon changed the title Add Datadog monitor definitions for unattended workflows [ci] Add Datadog monitor definitions for unattended workflows Sep 11, 2026
@eps1lon
eps1lon force-pushed the sebbie/ci-workflow-health-monitors branch 5 times, most recently from 8b51580 to b35d43c Compare September 11, 2026 14:57
Comment thread scripts/datadog/workflow-health/skip-rate-monitor.json Outdated
@eps1lon
eps1lon force-pushed the sebbie/ci-workflow-health-monitors branch 6 times, most recently from 28bdb20 to 0a56500 Compare September 11, 2026 18:04
This change adds two import-ready Datadog monitor definitions under `scripts/datadog/workflow-health/` covering the 16 workflows that run without intentional user input, meaning the scheduled workflows and the ones driven purely by `workflow_run`, `issues`, or `issue_comment` events. The failure monitor is a multi-alert `ci-pipelines alert` that notifies on the very first failed run of each workflow and links the specific failed execution through the `ci.pipeline.url` attribute. The skip-rate monitor warns when a workflow's share of skipped runs over the trailing day exceeds 50 percent. Both monitors notify on recovery, and the accompanying README documents the mapping from workflow files to Datadog pipeline names, the import steps, and the assumptions that need to be verified in the CI Pipelines explorer before the monitors can be relied on.
@eps1lon
eps1lon force-pushed the sebbie/ci-workflow-health-monitors branch from 0a56500 to 077c11b Compare September 11, 2026 18:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant