Skip to content

Commit 23b7f81

Browse files
LouisParkinLouisParkin
authored
Merge pull request #366 from StackVista/STAC-25533-cerberus-notify
STAC-25533: report release-branch CI failures to Slack via Cerberus
2 parents 09b68c1 + 2198208 commit 23b7f81

16 files changed

Lines changed: 460 additions & 475 deletions

File tree

‎.github/scripts/select-checks.sh‎

Lines changed: 66 additions & 70 deletions
Original file line numberDiff line numberDiff line change
@@ -15,45 +15,28 @@
1515
# EVERY suite (GitLab: the `base_changes` anchor).
1616
# * Otherwise only the suites whose own directory changed run.
1717
# * GitLab's `splunk_base_build_rule` -- a change to splunk_base also runs the
18-
# other three splunk suites, which import its test helpers -- is not ported
19-
# here because no splunk suite runs yet. It lands with them in phase 2
20-
# (STAC-25531).
18+
# other three splunk suites, which import its test helpers.
2119
# * push / workflow_dispatch run everything (GitLab: `master_branch`,
2220
# `release_branch`).
2321
#
24-
# Writes three arrays to $GITHUB_OUTPUT for `fromJson()` in a matrix:
25-
# checks -- suites that need no credentials
26-
# private_checks -- suites that install from the private GitLab PyPI
27-
# index, and are cleared to run on this event
28-
# deferred_private_checks -- private-index suites withheld from this event
29-
# (always empty outside pull requests)
22+
# Writes two arrays to $GITHUB_OUTPUT for `fromJson()` in a matrix:
23+
# checks -- suites that run in the shared BCI container
24+
# docker_checks -- suites that need a live Docker daemon and so run directly
25+
# on the runner (STAC-25531)
3026
#
31-
# The split is a security boundary, not a convenience. The credential-free suites
32-
# run with no secrets in scope at all. The private-index suites need a registry
33-
# password, so they are kept in a separate job -- and, on pull requests, are not
34-
# run at all (STAC-25540, second review pass).
35-
#
36-
# That last part is the whole point, so it is worth stating plainly: a
37-
# `pull_request` run executes the pull request's own copy of the workflow and of
38-
# every script it calls. Hardening the job cannot keep a determined pull request
39-
# away from a secret the run is holding -- it can always edit the thing that holds
40-
# it. The only run that cannot leak the credential is a run that never receives
41-
# it, so these suites are deferred to push, tag and workflow_dispatch events,
42-
# whose contents are reviewed before they reach the release branch.
27+
# Every suite here is credential-free, and that is worth keeping. Until
28+
# STAC-25544 `vsphere` resolved only against a private package registry, which
29+
# meant withholding the credential from pull requests and therefore not running
30+
# the suite on them at all -- a real coverage gap, because a `pull_request` run
31+
# executes the pull request's own copy of the workflow and of every script it
32+
# calls, so a run holding a secret cannot be hardened against the pull request
33+
# that edits it. Modernising the VMware pin onto public PyPI removed the secret
34+
# and with it the gap. If a suite ever appears to need a registry credential
35+
# again, removing that need is the fix; splitting the matrix is not.
4336

4437
set -euo pipefail
4538

46-
# Suites currently running on GitHub Actions. Phase 1 is the 15 suites that need
47-
# no Docker daemon.
48-
#
49-
# Deliberately NOT here yet (phase 2, STAC-25531 -- needs a docker client in the
50-
# job image):
51-
# splunk_base, splunk_health, splunk_metric, splunk_topology
52-
# -- each drives a real Splunk container via docker-compose.
53-
# stackstate_checks_dev
54-
# -- its tests exercise the toolkit's own Docker helpers.
55-
# ubuntu-latest already provides a working Docker daemon, so this is a matter of
56-
# giving the job a docker client rather than provisioning a runner.
39+
# Suites currently running on GitHub Actions.
5740
#
5841
# Deliberately dropped, not pending:
5942
# postgres -- .gitlab-ci.yml carried a `test_postgres` job for a check that does
@@ -69,27 +52,45 @@ CHECKS=(
6952
kubelet
7053
openmetrics
7154
servicenow
55+
splunk_base
56+
splunk_health
57+
splunk_metric
58+
splunk_topology
7259
stackstate_checks_base
60+
stackstate_checks_dev
7361
static_health
7462
static_topology
7563
vsphere
7664
zabbix
7765
)
7866

79-
# Suites whose requirements resolve only against the private GitLab PyPI index.
80-
# `vsphere` pins vsphere-automation-sdk, which VMware never published to public
81-
# PyPI (the name is squatted there by an unrelated 0.0.1 placeholder), so it is
82-
# mirrored into the StackVista package registry and needs authentication.
67+
# Suites that need a real Docker daemon: the four splunk suites drive a Splunk
68+
# container through docker-compose, and stackstate_checks_dev tests the toolkit's
69+
# own Docker helpers (STAC-25531).
8370
#
84-
# Everything not listed here is credential-free and must stay that way: adding a
85-
# suite to this list stops it running on pull requests altogether, and removing
86-
# the need for the private index is always the better fix. For vsphere that fix
87-
# looks reachable -- VMware now publishes the SDK to public PyPI under renamed
88-
# packages (vmware-vapi-runtime, vmware-vapi-common-client, pyvmomi) and ships
89-
# the NSX/VMC wheels from its own public index -- so this list should shrink to
90-
# nothing once the pin is modernised.
91-
PRIVATE_INDEX_CHECKS=(
92-
vsphere
71+
# These run as their own matrix directly on the runner, not inside the BCI
72+
# container the other suites use. That is not a preference -- the tests resolve
73+
# their target host through `get_docker_hostname()`, which reads DOCKER_HOST and
74+
# falls back to `localhost`. Compose publishes its ports on the Docker host, so
75+
# `localhost` is correct only when the test process shares a network namespace
76+
# with the daemon. Inside a job container it would resolve to the container
77+
# itself and every connection would be refused. GitLab avoided this by pointing
78+
# DOCKER_HOST at a `docker:dind` service, whose hostname then resolved for both.
79+
DOCKER_CHECKS=(
80+
splunk_base
81+
splunk_health
82+
splunk_metric
83+
splunk_topology
84+
stackstate_checks_dev
85+
)
86+
87+
# splunk_health, splunk_metric and splunk_topology all build on splunk_base, so a
88+
# change there has to run all four. Ported from the `splunk_base_build_rule`
89+
# anchor in .gitlab-ci.yml, which added the same fan-out to every splunk job.
90+
SPLUNK_DEPENDENTS=(
91+
splunk_health
92+
splunk_metric
93+
splunk_topology
9394
)
9495

9596
# A change anywhere here invalidates every suite: the base classes and the test
@@ -112,53 +113,37 @@ to_json() {
112113
fi
113114
}
114115

115-
is_private_index() {
116+
is_docker() {
116117
local candidate=$1 check
117-
for check in "${PRIVATE_INDEX_CHECKS[@]}"; do
118+
for check in "${DOCKER_CHECKS[@]}"; do
118119
[ "${candidate}" = "${check}" ] && return 0
119120
done
120121
return 1
121122
}
122123

123124
emit() {
124125
local -a selected=("$@")
125-
local -a public=() private=() deferred=()
126+
local -a public=() docker=()
126127
local check
127128
for check in ${selected[@]+"${selected[@]}"}; do
128-
if is_private_index "${check}"; then
129-
private+=("${check}")
129+
if is_docker "${check}"; then
130+
docker+=("${check}")
130131
else
131132
public+=("${check}")
132133
fi
133134
done
134135

135-
# Pull requests do not run the private-index suites at all (STAC-25540, second
136-
# review pass). See the security-boundary note at the top of this file: a
137-
# `pull_request` run executes the pull request's own copy of the workflow and
138-
# scripts, so the credential can only be protected by withholding it. These
139-
# suites run on the release branch instead, where the code has been reviewed.
140-
if [ "${EVENT_NAME}" = "pull_request" ] && [ "${#private[@]}" -gt 0 ]; then
141-
deferred=("${private[@]}")
142-
private=()
143-
fi
144-
145-
local public_json private_json deferred_json
136+
local public_json docker_json
146137
public_json=$(to_json ${public[@]+"${public[@]}"})
147-
private_json=$(to_json ${private[@]+"${private[@]}"})
148-
deferred_json=$(to_json ${deferred[@]+"${deferred[@]}"})
138+
docker_json=$(to_json ${docker[@]+"${docker[@]}"})
149139

150140
{
151141
echo "checks=${public_json}"
152-
echo "private_checks=${private_json}"
153-
echo "deferred_private_checks=${deferred_json}"
142+
echo "docker_checks=${docker_json}"
154143
} >>"${GITHUB_OUTPUT}"
155144

156145
echo "Selected credential-free suites: ${public_json}"
157-
echo "Selected private-index suites: ${private_json}"
158-
if [ "${deferred_json}" != "[]" ]; then
159-
echo "Deferred private-index suites: ${deferred_json}"
160-
echo "::notice title=Private-index suites do not run on pull requests::${deferred_json} resolve only against the private package registry. Pull requests are deliberately given no credential to reach it, so these suites run on ${BASE_REF:-the release branch} after merge."
161-
fi
146+
echo "Selected docker-daemon suites: ${docker_json}"
162147
}
163148

164149
# Anything that is not a pull request is a full run. On the release branch the
@@ -207,4 +192,15 @@ for file in "${CHANGED[@]}"; do
207192
done
208193
done
209194

195+
# splunk_base is a library for the other three splunk suites, so pull them in
196+
# whenever it changes. Ported from `splunk_base_build_rule` in .gitlab-ci.yml.
197+
# `emit` sorts and de-duplicates, so adding them unconditionally is safe.
198+
for check in ${SELECTED[@]+"${SELECTED[@]}"}; do
199+
if [ "${check}" = "splunk_base" ]; then
200+
echo "'splunk_base' changed: also running ${SPLUNK_DEPENDENTS[*]}."
201+
SELECTED+=("${SPLUNK_DEPENDENTS[@]}")
202+
break
203+
fi
204+
done
205+
210206
emit "${SELECTED[@]+"${SELECTED[@]}"}"

‎.github/workflows/cerberus-notify.yml‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
name: Cerberus notify
2+
3+
# New capability, not a port (STAC-25142 / STAC-25533). Unlike stackstate-agent
4+
# and stackstate-process-agent, this repo's retired .gitlab-ci.yml had no notify
5+
# job and no .cerberus directory, so a failed release-branch pipeline has always
6+
# been silent here. STAC-25510 is what that costs: process-agent's image
7+
# publishing broke on 2026-07-23 and went unnoticed for 12 days.
8+
#
9+
# Structure and calling convention follow
10+
# stackstate-process-agent/.github/workflows/cerberus-notify.yml, which in turn
11+
# follows `cerberus-block-on-master-fail` in StackVista/stackstate. Cerberus is
12+
# the internal notify/block Lambda (source: https://github.com/StackVista/cerberus).
13+
# `platform: github` makes it build GitHub pipeline/commit URLs rather than
14+
# GitLab ones.
15+
#
16+
# `action: notify`, never `action: block`. Policy for migrated repos is notify by
17+
# default. Blocking locks the branch (`lock_branch`), additionally requires the
18+
# Cerberus GitHub App to be installed here, and mutates branch protection that
19+
# pulumi-infra owns (STAC-25522) out from under it -- a subsequent pulumi apply
20+
# would silently unlock the branch again.
21+
#
22+
# Prerequisites: CERBERUS_LAMBDA_URL and CERBERUS_API_TOKEN must both reach this
23+
# repo as REPO-level secrets. The org-level copies are visibility=private, which
24+
# excludes this PUBLIC repo. pulumi-infra provisions the pair together
25+
# (github/repoVariables/resources.yaml, StackVista/pulumi-infra#277). If either
26+
# is missing, this workflow warns and exits 0 rather than adding a second red job
27+
# to an already-failed run -- the annotation is the signal.
28+
#
29+
# The bearer token is not optional going forward. StackVista/cerberus#4
30+
# (STAC-24889) adds `Authorization: Bearer <token>` verification to every
31+
# non-Slack request; before it, the endpoint was entirely unauthenticated.
32+
# Sending the header is forward-compatible -- the currently deployed Lambda
33+
# ignores unknown headers -- so this works either side of that deploy. Without
34+
# it, the first failure after cerberus#4 ships would get a 401 and no Slack
35+
# message.
36+
#
37+
# The Slack channel is deliberately not sent. Cerberus resolves it as
38+
# `util.GetOrDefault(req.Context, "channel", s.Channel)`, and GetOrDefault treats
39+
# an empty or whitespace value as absent, so omitting `channel` falls back to the
40+
# Lambda's own SLACK_CHANNEL.
41+
42+
on:
43+
workflow_call:
44+
inputs:
45+
suite:
46+
description: Suite label shown in the Slack message, e.g. checks.
47+
required: true
48+
type: string
49+
secrets:
50+
# `required: false`. A caller passing `${{ secrets.X }}` for a secret the
51+
# repo does not hold yields an empty string, which GitHub rejects as "not
52+
# provided" against a required secret and fails the call before the run
53+
# step's guard can warn -- the failure mode this workflow exists to avoid.
54+
CERBERUS_LAMBDA_URL:
55+
required: false
56+
CERBERUS_API_TOKEN:
57+
required: false
58+
59+
# Nothing here reads the repository; the payload is built entirely from the
60+
# github context.
61+
permissions: {}
62+
63+
jobs:
64+
notify:
65+
name: Notify Slack via Cerberus
66+
runs-on: ubuntu-24.04
67+
timeout-minutes: 5
68+
steps:
69+
- name: Post the failure to Cerberus
70+
env:
71+
CERBERUS_LAMBDA_URL: ${{ secrets.CERBERUS_LAMBDA_URL }}
72+
CERBERUS_API_TOKEN: ${{ secrets.CERBERUS_API_TOKEN }}
73+
REPOSITORY: ${{ github.repository }}
74+
BRANCH: ${{ github.ref_name }}
75+
PIPELINE: ${{ github.run_id }}
76+
COMMIT_SHA: ${{ github.sha }}
77+
# Empty on tag pushes, which carry no head_commit. COMMIT_TITLE below
78+
# falls back to the sha so the Slack message is never blank.
79+
COMMIT_MESSAGE: ${{ github.event.head_commit.message }}
80+
SUITE: ${{ inputs.suite }}
81+
run: |
82+
set -euo pipefail
83+
84+
if [ -z "${CERBERUS_LAMBDA_URL}" ] || [ -z "${CERBERUS_API_TOKEN}" ]; then
85+
echo "::warning title=Cerberus not configured::CERBERUS_LAMBDA_URL and/or CERBERUS_API_TOKEN is not visible to this repo, so the ${SUITE} failure was not reported to Slack. Both are provisioned as repo-level secrets by pulumi-infra (STAC-25533)."
86+
exit 0
87+
fi
88+
89+
COMMIT_TITLE=$(printf '%s' "${COMMIT_MESSAGE}" | head -n1)
90+
if [ -z "${COMMIT_TITLE}" ]; then
91+
COMMIT_TITLE="${COMMIT_SHA}"
92+
fi
93+
94+
# Not --verbose: it echoes request headers, and the Authorization
95+
# header carries the shared token. GitHub would mask it, but not
96+
# emitting it is better than relying on masking.
97+
curl --fail --silent --show-error \
98+
-X POST "${CERBERUS_LAMBDA_URL}" \
99+
-H "Content-Type: application/json" \
100+
-H "Authorization: Bearer ${CERBERUS_API_TOKEN}" \
101+
-d "$(jq -n \
102+
--arg repo "${REPOSITORY}" \
103+
--arg branch "${BRANCH}" \
104+
--arg pipeline "${PIPELINE}" \
105+
--arg sha "${COMMIT_SHA}" \
106+
--arg title "${COMMIT_TITLE}" \
107+
--arg suite "${SUITE}" \
108+
'{
109+
action: "notify",
110+
context: {
111+
platform: "github",
112+
"project.id": $repo,
113+
"project.slug": $repo,
114+
"project.name": "StackState Agent Integrations",
115+
branch: $branch,
116+
pipeline: $pipeline,
117+
"commit.sha": $sha,
118+
"commit.title": $title,
119+
suite: $suite
120+
}
121+
}'
122+
)"

0 commit comments

Comments
 (0)