checklist: Add golden tests for substitutions

This test evaluates that the version substitutions in the release
templates will work correctly. The downside is that any change to
templates will require modifying both the template and the golden file.
The good thing is that this can help to detect problems where we write
the template in a way that doesn't properly apply the version string
substitutions that the tool is intended to apply.

Use 'make generate-golden' to regenerate the golden test content from
the templates.

Signed-off-by: Joe Stringer <[email protected]>

Author: joestringer

Makefile

@@ -9,6 +9,16 @@ all: tests release
 docker-image:
 	docker build $(DOCKER_BUILD_FLAGS) -t cilium/release-tool:${IMAGE_TAG} .
 
+.PHONY: generate-golden
+generate-golden:
+	$(MAKE) $(patsubst %.input,%.golden,$(shell find ./testdata/checklist/ -name "*.input"))
+
+%.golden: %.input
+	$(GO) run ./cmd/ checklist open --dry-run \
+		--target-version "v1.10.0-pre.0" \
+		--template $< \
+		> $@ \
+
 .PHONY: tests
 tests:
 	$(GO) test -mod=vendor ./...

cmd/checklist/template_test.go

@@ -4,6 +4,8 @@
 package checklist
 
 import (
+	"os"
+	"path/filepath"
 	"testing"
 
 	"github.com/stretchr/testify/assert"
@@ -71,3 +73,34 @@ func Test_assembleVersionSubstitutions(t *testing.T) {
 		assert.Equal(t, sub, tt.want, tt.name+" generated unexpected substitutions")
 	}
 }
+
+func Test_prepareChecklist(t *testing.T) {
+	cfg := ChecklistConfig{
+		TargetVer: "v1.10.0-pre.0",
+	}
+	testdataPath := filepath.Join("..", "..", "testdata", "checklist")
+
+	paths, err := filepath.Glob(filepath.Join(testdataPath, "*.input"))
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	for _, path := range paths {
+		_, filename := filepath.Split(path)
+		testname := filename[:len(filename)-len(filepath.Ext(path))]
+
+		t.Run(testname, func(t *testing.T) {
+			source, err := os.ReadFile(path)
+			assert.Nil(t, err, "failed to read input template: ", err)
+
+			output, err := prepareChecklist(source, cfg)
+			assert.Nil(t, err, "failed to render checklist: ", err)
+
+			golden := filepath.Join(testdataPath, testname+".golden")
+			want, err := os.ReadFile(golden)
+			assert.Nil(t, err, "error reading golden output: ", err)
+
+			assert.Equal(t, output, string(want), "template processing did not match golden output. Check for bugs or run 'make generate-golden' to update the golden tests.")
+		})
+	}
+}

testdata/checklist/release_template_minor.md.golden

@@ -0,0 +1,127 @@
+---
+name: Release a new minor version of Cilium from a stable branch
+about: Create a checklist for an upcoming release
+title: 'v1.10.0 release'
+labels: kind/release
+assignees: ''
+
+---
+
+## Setup preparation
+
+- [ ] Depending on your OS, make sure Docker is running
+- [ ] Export a [`GITHUB_TOKEN`](https://github.com/settings/tokens/new?description=Cilium%20Release%20Script&scopes=write:org,public_repo) that has access to the repository
+- [ ] Make sure a setup (GPG, SSH, S/MIME) is in place for [signing tags] with Git
+- [ ] Make sure the `GOPATH` environment variable is set and pointing to the relevant path
+- [ ] Make sure the [Cilium helm charts][Cilium charts] and [release][Cilium release-notes tool] repositories are installed locally:
+  - [ ] Run `git clone https://github.com/cilium/charts.git "$GOPATH/src/github.com/cilium/charts"`
+  - [ ] Run `git clone https://github.com/cilium/release.git "$GOPATH/src/github.com/cilium/release"`
+    - [ ] If you already have the repo checked out, make sure the `release` binary is up to date:
+
+          git checkout master && git pull && make
+
+## Pre-release
+
+- [ ] Announce in Cilium slack channel #launchpad: `Starting v1.10.0 release process :ship:`
+- [ ] Create a thread for that message and ping current top-hat to not merge any
+      PRs until the release process is complete.
+- [ ] Change directory to the local copy of Cilium repository.
+- [ ] Check that there are no [release blockers] for the targeted release version
+- [ ] Ensure that outstanding [backport PRs] are merged
+- [ ] Check with @cilium/security team if there are any security fixes to include
+      in the release.
+- [ ] Execute `release --current-version 1.10.0 --next-dev-version 1.10.1` to
+      automatically move any unresolved issues/PRs from old release project
+      into the new project. The `release` binary is located in the
+      [current repository][Cilium release-notes tool].
+- [ ] Push a PR including the changes necessary for the new release:
+  - [ ] Pull latest changes from the branch being released
+  - [ ] The next step will generate a `CHANGELOG.md` that will not be correct.
+        That is expected, and it is fixed with a follow-up step. Don't worry.
+  - [ ] Run `../release/internal/start-release.sh 1.10.0 <GH-PROJECT> 1.9`
+        Note that this script produces some files at the root of the Cilium
+        repository, and that these files are required at a later step for
+        tagging the release.
+  - [ ] `rm CHANGELOG.md`
+  - [ ] Regenerate the log since the previous release with `prep-changelog.sh <last-patch-release> v1.10.0`
+  - [ ] Check and edit the `CHANGELOG.md` to ensure all PRs have proper release notes
+  - [ ] Edit the `v1.10.0-changes.txt` files locally to replace the text with "See CHANGELOG.md for more details"
+  - [ ] Add the 'stable' tag as part of the GitHub workflow and remove the
+        'stable' tag from the last stable branch.
+  - [ ] Commit all changes with title `Prepare for release v1.10.0`
+  - [ ] Submit PR (`../release/internal/submit-release.sh`)
+  - [ ] Submit a PR that removes the 'stable' tag from the last stable branch.
+- [ ] Merge PR
+- [ ] Create and push *both* tags to GitHub (`v1.10.0`, `1.10.0`)
+  - [ ] Pull latest branch locally and run `../release/internal/tag-release.sh`.
+- [ ] Ask a maintainer to approve the build in the following link (keep the URL
+      of the GitHub run to be used later):
+      [Cilium Image Release builds](https://github.com/cilium/cilium/actions?query=workflow:%22Image+Release+Build%22)
+  - [ ] Check if all docker images are available before announcing the release:
+        `make -C install/kubernetes/ check-docker-images`
+- [ ] Get the image digests from the build process and make a commit and PR with
+      these digests.
+  - [ ] Run `../release/internal/post-release.sh URL` to fetch the image
+        digests and submit a PR to update these, use the `URL` of the GitHub
+        run here
+  - [ ] Get someone to review the PR. Do not trigger the full CI suite, but
+        wait for the automatic checks to complete. Merge the PR.
+- [ ] Update helm charts
+  - [ ] Create helm charts artifacts in [Cilium charts] repository using
+        [cilium helm release tool] for the `v1.10.0` release.
+  - [ ] Check the output of the [chart workflow] and see if the test was
+        successful.
+- [ ] Check [read the docs] configuration:
+    - [ ] Set the [default version] and mark the EOL version as active, and
+          hidden and configure the new minor version as active and **not**
+          hidden in [active versions].
+    - [ ] Update algolia configuration search in [docsearch-scraper-webhook].
+- [ ] Check draft release from [releases] page
+  - [ ] Update the text at the top with 2-3 highlights of the release
+  - [ ] Publish the release
+- [ ] Announce the release in #general on Slack (Use [@]channel for v1.10.0)
+- [ ] Update Grafana dashboards
+
+## Post-release
+
+- [ ] Update the upgrade guide and [roadmap](https://github.com/cilium/cilium/blob/main/Documentation/community/roadmap.rst) for any features that changed status.
+- [ ] For new minor version update [security policy]
+- [ ] Prepare post-release changes to main branch using `../release/internal/bump-readme.sh`
+  - [ ] Make sure to update the `.github/maintainers-little-helper.yaml` so that
+        upcoming PRs are tracked correctly for the next release.
+  - [ ] Bump the main testsuite to upgrade from v1.10 branch to main
+  - [ ] `echo 1.10.0 > stable.txt`.
+  - [ ] `echo '{"results":[{"slug":"v1.10"}]}' > Documentation/_static/stable-version.json`.
+  - [ ] Commit / amend the commit to add all of the changes above and push the PR.
+  - [ ] Merge the post-release PR.
+- [ ] Notify #development on Slack that deprecated features may now be removed.
+- [ ] This is the list of links for known external installers that depend on
+      the release process. Ideally, work toward updating external tools and
+      guides to point to the new Cilium version. If you find where to submit
+      the update, please add the relevant links to this template.
+  - [kops]
+  - [kubespray]
+  - [network policy]
+  - [cluster administration networking]
+  - [cluster administration addons]
+
+
+[signing tags]: https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-tags
+[release blockers]: https://github.com/cilium/cilium/labels/release-blocker%2F1.10
+[backport PRs]: https://github.com/cilium/cilium/pulls?q=is%3Aopen+is%3Apr+label%3Abackport%2F1.10
+[Cilium release-notes tool]: https://github.com/cilium/release
+[Cilium charts]: https://github.com/cilium/charts
+[releases]: https://github.com/cilium/cilium/releases
+[kops]: https://github.com/kubernetes/kops/
+[kubespray]: https://github.com/kubernetes-sigs/kubespray/
+[cilium helm release tool]: https://github.com/cilium/charts/blob/master/RELEASE.md
+[cilium-runtime images]: https://quay.io/repository/cilium/cilium-runtime
+[read the docs]: https://readthedocs.org/projects/cilium/
+[active versions]: https://readthedocs.org/projects/cilium/versions/
+[default version]: https://readthedocs.org/dashboard/cilium/advanced/
+[docsearch-scraper-webhook]: https://github.com/cilium/docsearch-scraper-webhook
+[security policy]: https://github.com/cilium/cilium/security/policy
+[network policy]: https://kubernetes.io/docs/tasks/administer-cluster/network-policy-provider/cilium-network-policy/
+[cluster administration networking]: https://kubernetes.io/docs/concepts/cluster-administration/networking/
+[cluster administration addons]: https://kubernetes.io/docs/concepts/cluster-administration/addons/
+[chart workflow]: https://github.com/cilium/charts/actions/workflows/validate-cilium-chart.yaml

testdata/checklist/release_template_minor.md.input

@@ -0,0 +1 @@
+../../.github/ISSUE_TEMPLATE/release_template_minor.md

testdata/checklist/release_template_patch.md.golden

@@ -0,0 +1,150 @@
+---
+name: NEW Release a new patch version of Cilium from a stable branch
+about: Create a checklist for an upcoming release
+title: 'v1.10.0 release'
+labels: kind/release
+assignees: ''
+
+---
+
+## Setup preparation
+
+- [ ] Depending on your OS, make sure Docker is running
+- [ ] Export a [`GITHUB_TOKEN`](https://github.com/settings/tokens/new?description=Cilium%20Release%20Script&scopes=write:org,public_repo,project) that has access to the repository. Only classic tokens are
+      [supported at the moment][GitHub PAT tracker], the needed scopes are `write:org`, `public_repo` and `project`.
+- [ ] Make sure a setup (GPG, SSH, S/MIME) is in place for [signing tags]
+  with Git and install [gh](https://cli.github.com).
+- [ ] Make sure the [Cilium helm charts][Cilium charts] and [release][Cilium release-notes tool] repositories are installed locally:
+  - [ ] Run `git clone https://github.com/cilium/charts.git "$GOPATH/src/github.com/cilium/charts"`
+  - [ ] Run `git clone https://github.com/cilium/release.git "$GOPATH/src/github.com/cilium/release"`
+    - [ ] If you already have the repo checked out, make sure the `release` binary is up to date:
+
+          git checkout main && git pull && make
+- [ ] Read the documentation of `release start --help` tool to understand what
+  each automated step does.
+
+## Pre-check (run ~1 week before release date)
+
+- [ ] When you create a GitHub issue using this issue template, GitHub Slack app posts a
+      message in #launchpad Slack channel. Create a thread for that message and ping the
+      current backporter to merge the outstanding [backport PRs] and stop merging any new
+      backport PRs until the GitHub issue is closed (to avoid generating incomplete
+      release notes).
+- [ ] Run `./release start --steps 1-pre-check --target-version v1.10.0`
+  - [ ] Check that there are no [release blockers] for the targeted release
+        version.
+  - [ ] Ensure that outstanding [backport PRs] are merged (these may be
+        skipped on case by case basis in coordination with the backporter).
+  - [ ] Check with @cilium/security team in case there are any CVEs found in the
+        docker image.
+  - [ ] Check with @cilium/security team if there are any security fixes to
+        include in the release.
+
+## Preparation PR (run ~1 day before release date. It can be re-run multiple times.)
+
+- [ ] Go to [release workflow] and Run the workflow from "Branch: main", for
+  step "2-prepare-release" and version for v1.10.0
+  - [ ] Check if the workflow was successful and check the PR opened by the
+        Release bot.
+- [ ] Merge PR
+
+## Tagging
+
+- [ ] Ask a maintainer if there are any known issues that should hold up the release
+- [ ] FYI, do not wait too much time between a tag is created and the helm charts are published.
+      Once the tags are published the documentation will be pointing to them. Until we release
+      the helm chart, users will face issues while trying out our documentation.
+- [ ] Run `./release start --steps 3-tag --target-version v1.10.0`
+- [ ] Ask a maintainer to approve the build in the following link (keep the URL
+      of the GitHub run to be used later):
+      [Cilium Image Release builds](https://github.com/cilium/cilium/actions?query=workflow:%22Image+Release+Build%22)
+
+## Post Tagging (run after docker images are published)
+
+- [ ] Go to [release workflow] and Run the workflow from "Branch: main", for
+  step "4-post-release" and version for v1.10.0
+    - [ ] Check if the workflow was successful and check the PR opened by the
+      Release bot.
+- [ ] Merge PR
+- [ ] This step opened a [GitHub project](https://github.com/orgs/cilium/projects?query=is%3Aopen++v+)
+      to track the PRs in the release. Close the corresponding project.
+
+## Publish helm (run after docker images are published)
+
+- [ ] Update helm charts `./release start --steps 5-publish-helm --target-version v1.10.0`
+- [ ] Open [Charts Workflow] and check if the workflow run is successful.
+
+## Publish docs (only for pre/rc releases)
+
+- [ ] Check [read the docs] configuration:
+  - [ ] Set a new build as active and hidden in [active versions].
+  - [ ] Deactivate previous RCs.
+  - [ ] Update algolia configuration search in [docsearch-scraper-webhook].
+    - Update the versions in `docsearch.config.json`, commit them and push a
+      trigger the workflow [here](https://github.com/cilium/docsearch-scraper-webhook/actions/workflows/update-algolia-index.yaml)
+
+## Post-release
+
+- [ ] Check draft release from [releases] page
+  - [ ] Update the text at the top with 2-3 highlights of the release
+  - [ ] Check with @cilium/security if the release addresses any open security
+        advisory. If it does, include the list of security advisories at the
+        top of the release notes.
+  - [ ] Check whether the new release should be set as the _latest_ release
+        (via the checkbox at the bottom of the page). It should be the new
+        _latest_ if the version number is strictly superior to the current
+        _latest_ release displayed on GitHub (e.g. 1.11.13 does not become the
+        new latest release over 1.12.5, but version 1.12.6 will).
+  - [ ] Publish the release
+- [ ] Announce the release in #general on Slack (do not use [@]channel).
+      See below for templates.
+- [ ] Prepare post-release changes to main branch using `../release/internal/bump-readme.sh`.
+
+---
+
+## Slack example text templates
+
+### Patch releases
+
+```
+:confetti_ball: :cilium-radiant: Release Announcement :cilium-radiant::confetti_ball:
+
+Cilium v1.10.0, vA.B.C, and vD.E.F have been released. Thanks all for your contributions! Please see the release notes below for details :cilium-gopher:
+
+v1.10.0: https://github.com/cilium/cilium/releases/tag/v1.10.0
+vA.B.C: https://github.com/cilium/cilium/releases/tag/vA.B.C
+vD.E.F: https://github.com/cilium/cilium/releases/tag/vD.E.F
+```
+
+### First pre-release
+
+```
+:cilium-new: *Cilium v1.10.0-pre.0 has been released:*
+https://github.com/cilium/cilium/releases/tag/v1.10.0-pre.0
+
+This is the first monthly snapshot for the v1.10 development cycle. There are [v1.10.0-pre.0 OSS docs](https://docs.cilium.io/en/v1.10.0-pre.0) available if you want to pull this version & try it out.
+```
+
+### Subsequent pre-/rc- releases
+
+```
+*Announcement* :tada: :tada:
+
+:cilium-new: *Cilium v1.10.0-pre.0 has been released:*
+https://github.com/cilium/cilium/releases/tag/v1.10.0-pre.0
+
+Thank you for the testing and contributing to the previous pre-releases. There are [v1.10.0-pre.0 OSS docs](https://docs.cilium.io/en/v1.10.0-pre.0) available if you want to pull this version & try it out.
+```
+
+[release workflow]: https://github.com/cilium/cilium/actions/workflows/release.yaml
+[GitHub PAT tracker]: https://github.com/orgs/community/discussions/36441
+[signing tags]: https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-tags
+[release blockers]: https://github.com/cilium/cilium/labels/release-blocker%2F1.10
+[backport PRs]: https://github.com/cilium/cilium/pulls?q=is%3Aopen+is%3Apr+draft%3Afalse+label%3Abackport%2F1.10
+[Cilium release-notes tool]: https://github.com/cilium/release
+[Cilium charts]: https://github.com/cilium/charts
+[Charts Workflow]: https://github.com/cilium/charts/actions/workflows/validate-cilium-chart.yaml
+[releases]: https://github.com/cilium/cilium/releases
+[cilium helm release tool]: https://github.com/cilium/charts/blob/master/RELEASE.md
+[cilium-runtime images]: https://quay.io/repository/cilium/cilium-runtime
+[chart workflow]: https://github.com/cilium/charts/actions/workflows/validate-cilium-chart.yaml

testdata/checklist/release_template_patch.md.input

@@ -0,0 +1 @@
+../../.github/ISSUE_TEMPLATE/release_template_patch.md