Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fcos-release-notes: add job for generating release notes #261

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

zonggen
Copy link
Member

@zonggen zonggen commented Aug 11, 2020

Adds a job for generating release-notes.yaml from release-notes.d/ in testing-devel and next-devel and puts the output release-notes.yaml file to testing and next branch correspondingly.

Also adds the functionality in Jenkinsfile such that after building the image, replaces the premature next-release placeholder key with the real build id. Then it uploads the final release-notes.yaml to the S3 bucket as well as pushing to fedora-coreos-config repo.

Related: coreos/fedora-coreos-tracker#194
Signed-off-by: Allen Bai [email protected]

Adds a job for generating `release-notes.yaml` from `release-notes.d/`
in `testing-devel` and `next-devel` and puts the output `release-notes.yaml`
file to `testing` and `next` branch correspondingly.

Also adds the functionality in `Jenkinsfile` such that after building the image,
replaces the premature `next-release` placeholder key with the real build id.
Then it uploads the final `release-notes.yaml` to the S3 bucket as well as
pushing to `fedora-coreos-config` repo.

Related: coreos/fedora-coreos-tracker#194
Signed-off-by: Allen Bai <[email protected]>
./ s3://fcos-builds
""")
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something in my mind keeps telling me this is not correct / desired. Since Jenkinsfile primarily used only used cosa, and not sure if I could use botCreds and aws-fcos-builds-bot here. The logic comes from bump-lockfile.Jenkinsfile and sync-stream-metadata.Jenkinsfile.

Could you sanity check please? @jlebon

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still trying to understand how everything fits together (so the next statements might not fit with the intended design), but ISTM like we shouldn't do anything related to release notes as part of the main build pipeline. That should instead be in the release job (or a job triggered by the release job), and only on production streams.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Discussed at #261 (comment).

@zonggen
Copy link
Member Author

zonggen commented Aug 11, 2020

The PR makes it possible to:

  • Run the generate-release-notes.Jenkinsfile to build a pre-mature release-notes.yaml with the field next-release as the placeholder before we know the real build id ; also clean up the release-notes.d/ for next release cycle
  • Run the Jenkinsfile to replace the next-release field with actual build id and push the finalized release-notes.yaml to S3 bucket for website consumption

shwrap("mkdir ${branch}")

dir(branch) {
shwrap("cosa init --branch ${branch} https://github.com/${config_repo}")
Copy link
Member

@dustymabe dustymabe Aug 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to use cosa here? Should we just use git clone --depth=1 instead?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All I needed was a FCOS config repo so no hard need for a cosa init.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, let's just use git clone --depth=1 to keep things simple then.

shwrap("""
cd ${branch}/src/config
git checkout ${branch}
rm -rf release-notes.d/*.yaml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we probably need to use git rm here and git commit -m below.

Comment on lines +48 to +49
// should gracefully handle race conditions here
sh("git push https://\${GHUSER}:\${GHTOKEN}@github.com/${config_repo} ${branch}")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doing a git pull --rebase before doing the git push might help with race conditions.

@dustymabe
Copy link
Member

In the most recent design proposal we have (snippet):


Proposal:

Have a tool that runs independently of the pipeline like bump-lockfile and sync-stream-metadata jenkins jobs. This tool will trigger based on a new release of FCOS or can be triggered manually. The responsibility of this tool includes:


In this PR the "tool" is jobs/generate-release-notes.Jenkinsfile, which is great! However, shouldn't it be able to run independently without the need for the Jenkinsfile code in this PR that runs release-note-generator.py update?

If jobs/generate-release-notes.Jenkinsfile basically polls the release metadata to see what the latest release is it should be able to run completely independent of the build pipeline.

@dustymabe
Copy link
Member

For jobs/generate-release-notes.Jenkinsfile are we sticking by the design proposal (snippet):


Git repository manipulation

  • look at the release-notes.yaml file for the related branch (i.e. testing)
    • if an entry for that release exists, it does nothing to the git repo
    • if an entry for that release does not exist
      • create a new entry in release-notes.yaml for that release version
        • merge the release-notes.d/* files into release-notes.yaml
        • delete release-notes.d/* files from that branch
        • delete corresponding release-notes.d/* files from testing-devel branch

It seems like we can do these three bullet points:

- merge the `release-notes.d/*` files into `release-notes.yaml`
- delete `release-notes.d/*` files from that branch
- delete corresponding `release-notes.d/*` files from `testing-devel` branch

in a single stage and not worry about breaking it up further. From those bullets it seems like there should be 2 git push operations.

  • one git push to testing with updated release-notes.yaml and deleted release-notes.d/* files
  • one git push to testing-devel with deleted release-notes.d/* files
    • We have to be careful with this one. We need to make sure we only delete release-notes.d/ files that were in the testing branch

We really are just primarily operating on the testing branch, the only reason we touch testing-devel is so we don't have to manually clean up those release-notes.d/ files.

@zonggen
Copy link
Member Author

zonggen commented Aug 17, 2020

jobs/generate-release-notes.Jenkinsfile basically polls the release metadata to see what the latest release

@dustymabe Yes, the only portion that I depend on Jenkinsfile was the retrieving the build id part, so this should be fixing the current issue 👍

We really are just primarily operating on the testing branch, the only reason we touch testing-devel is so we don't have to manually clean up those release-notes.d/ files.

I thought we mainly operate on release-notes.d in testing-devel and generate the corresponding release-notes.yaml in testing so there would be no release-notes.d in testing branch. So my question is, what is the relationship between release-notes.d in testing-devel and release-notes.d in testing since according from the comment

We need to make sure we only delete release-notes.d/ files that were in the testing branch

it seems like release-notes.d in testing-devel is the superset of that in testing branch.

@dustymabe
Copy link
Member

jobs/generate-release-notes.Jenkinsfile basically polls the release metadata to see what the latest release

@dustymabe Yes, the only portion that I depend on Jenkinsfile was the retrieving the build id part, so this should be fixing the current issue +1

👍

We really are just primarily operating on the testing branch, the only reason we touch testing-devel is so we don't have to manually clean up those release-notes.d/ files.

I thought we mainly operate on release-notes.d in testing-devel and generate the corresponding release-notes.yaml in testing so there would be no release-notes.d in testing branch. So my question is, what is the relationship between release-notes.d in testing-devel and release-notes.d in testing since according from the comment

The release-notes.d/ files from testing-devel will get copied into testing when the release person runs the promote-config.sh script. So we'll be primarily operating on the testing branch.

Though, now that I think about it, we might need to leave the release-notes.d/ in the testing branch even after we've processed them, so that they can be used later for the stable branch. We'd just want to make sure any processed files get removed from testing-devel. We'd just need the promote-config.sh to clean up all the release-notes.d/ files before promoting content to the stable, testing, next branches. If this doesn't make sense let's grab some time tomorrow and talk it through.

We need to make sure we only delete release-notes.d/ files that were in the testing branch

it seems like release-notes.d in testing-devel is the superset of that in testing branch.

Right. Basically testing-devel continuously gets new PRs, so a new PR could have come in since we promoted content to testing (for the next release). We'd want to make sure that we only remove release-notes.d/ files that we actually made it into a release and was processed by our tools.

@zonggen
Copy link
Member Author

zonggen commented Aug 17, 2020

we might need to leave the release-notes.d/ in the testing branch even after we've processed them, so that they can be used later for the stable branch

That makes a lot of sense. I was concerning about testing -> stable transition and promote-config.sh seems like the piece of information I was missing.

new PR could have come in since we promoted content to testing (for the next release). We'd want to make sure that we only remove release-notes.d/ files that we actually made it into a release and was processed by our tools.

I thought twice about the process, and it makes sense now. Thanks for explaining!

@zonggen
Copy link
Member Author

zonggen commented Aug 19, 2020

From coreos/fedora-coreos-config#550 (comment) I've reorganized the logic to reflect the workflow without a next-release field:

# Before promotion, engineers add yaml snippets under release-notes.d in testing-devel
testing-devel:
  release-notes.d/1.yaml
  release-notes.d/2.yaml
testing:
  release-notes.d/

stable:
  release-notes.d/

---

# During promotion from testing-devel -> testing stream, 1.yaml and 2.yaml would be moved to testing branch and removed from testing-devel
# New release-notes.yaml is also generated / updated
  release-notes.d/
testing:
  release-notes.d/1.yaml
  release-notes.d/2.yaml
  release-notes.yaml
stable:
  release-notes.d/
 
---

# During promotion from testing -> stable stream, 1.yaml and 2.yaml would be moved to stable branch and removed from testing
# New release-notes.yaml is also generated / updated
# We no longer need to aggregate yaml snippets in stable, since we get future release notes from testing branch and directly generate them with latest release id. Remove yaml files under release-notes.d/, though in reality we probably should just not move the yaml files to stable in the first place.
testing-devel
  release-notes.d/
testing:
  release-notes.d/
  release-notes.yaml
stable:
  release-notes.d/1.yaml (removed)
  release-notes.d/2.yaml (removed)
  release-notes.yaml

Note: all of the release-notes.yaml will be updated with latest release id.

@dustymabe
Copy link
Member

From coreos/fedora-coreos-config#550 (comment) I've reorganized the logic to reflect the workflow without a next-release field:

# Before promotion, engineers add yaml snippets under release-notes.d in testing-devel
testing-devel:
  release-notes.d/1.yaml
  release-notes.d/2.yaml
testing:
  release-notes.d/

stable:
  release-notes.d/

Looks good.


# During promotion from testing-devel -> testing stream, 1.yaml and 2.yaml would be moved to testing branch and removed from testing-devel
# New release-notes.yaml is also generated / updated
testing-devel:
  release-notes.d/
testing:
  release-notes.d/1.yaml
  release-notes.d/2.yaml
  release-notes.yaml
stable:
  release-notes.d/
 

Looks good

# During promotion from testing -> stable stream, 1.yaml and 2.yaml would be moved to stable branch and removed from testing
# New release-notes.yaml is also generated / updated
# We no longer need to aggregate yaml snippets in stable, since we get future release notes from testing branch and directly generate them with latest release id. Remove yaml files under release-notes.d/, though in reality we probably should just not move the yaml files to stable in the first place.
testing-devel
  release-notes.d/
testing:
  release-notes.d/
  release-notes.yaml
stable:
  release-notes.d/1.yaml (removed)
  release-notes.d/2.yaml (removed)
  release-notes.yaml

I'm a bit torn on the "we no longer need to aggregate yaml snippets in stable" bit. I think it might more simple to just generate them the same way as we do for testing, but I do see value in just taking the release notes directly from the corresponding release in testing because it's possible we've gone in and fixed up formatting or added new information.

Though the more I think about it, we often do multiple testing releases before we do a stable. I think the snippets approach might be the way to go.

Note: all of the release-notes.yaml will be updated with latest release id.

I think there are a few things that are important to call out to see if we are on the same page.

  1. For testing-devel and next-devel release-notes.d/{1,2}.yaml files should get removed by the release-notes-generator job
  2. For stable, testing, and next the release-notes.d/{1,2}.yaml files should only get removed by the promote-config.sh process

Does that match with what you are thinking?

@jlebon jlebon removed their request for review September 11, 2023 18:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants