How-to release a new version of SF

This document how to create a release of the software factory project (SF).

Terminology

SemVer

SF projects are using SemVer, or Breaking.Feature.Fix, for example sf-config:

version	features
2.5.0	add diskimage builder support
2.5.1	add minor fix that can be applied without running upgrade process
2.6.0	add zuul-launcher support in paralell of jenkins, upgrade from 2.5.x is trivial
3.0.0	drop jenkins support, upgrade from 2.y is non trivial

A feature release, or feature version only include the first two numbers.

SF release is only the two first component, for commands in this document please replace ‘X.X’ by the feature release number for example ‘2.7’.

Koji targets

Koji target defines a build environment: an architecture and base repository SF is using a koji target per feature, e.g. 2.6 or 3.0

Koji tags

Koji tags defines a collection of package build. By default a target needs two tags:

targetname-build: includes the baserepos and extrarepos packages.
targetname: only includes the SF packages. Packages are published by the CI in this tag.

Then we use two extra tag for stable branch:

targetname-candidate: include the set of package we want to release. New CI build doesn’t end up there.
targetname-release: a copy of candidate once it’s fully tested. Packages in this tag will be signed.

Prepare the release environment

All commands in this documents should be run from koji.softwarefactory-project.io. Please connect as kojiadmin via SSH.

You should connect on the koji node with your SSH agent up and your key added in the agent to be able to send reviews.

git clone https://softwarefactory-project.io/r/software-factory/sfinfo

or

pushd sfinfo && git checkout master && git fetch -a && git reset --hard origin/master

Run a ~~git review -s~~ in sfinfo or set your username in sfinfo/.git/config.

Set the ~kojiadmin/.gitconfig file with your identity.

git config --global user.name "SF initial configurator"
git config --global user.email "admin@softwarefactory-project.io"

Create the release branch from master

Prepare the distro info

GOAL: define the koji target and the list of packages we are going to include in the new release RESULT: a YAML file

In the sfinfo repository, create a new description for the release. It should be a copy of the sf-master.yaml file as we create the new release from the master state:

cp sf-master.yaml sf-X.X.yaml

Change the

koji-target: the name of the new target
koji-url: the url to be used in CI script
branch: the git branch that will use the target

Send the review and approve the change to have it merged. You might need to do small fixes in that YAML file during the release process so you can submit later but do not forget to submit it before the step “Update the sf-release package”.

git add sf-X.X.yaml
git commit -m "Add stable info file for release sf-X.X"
git review

Finally, approve the change on Gerrit

Create the koji target for the new release

GOAL: create the koji resources RESULT: the koji target and 4 tags

As the koji admin user, run the script (included in sfinfo):

./zuul_koji_set_target.py --distro-info sf-X.X.yaml

Expected results: newRepo (sf-X.X-el7-build) completed successfully

In order to have the correct Release dist (el7), a package needs to be added to the build tag:

koji add-pkg sf-X.X-el7-build buildsys-macros-el7 --owner sfci
koji tag-build sf-X.X-el7-build buildsys-macros-el7-1.0-0.el7.centos

This last command may fails with “FAILED: TagError: build buildsys-macros-el7-1.0-0.el7.centos already tagged (sf-X.X-el7-build)” which is fine.

Also be sure to verify that the ‘dist’ macro in /etc/rpm/macros.dist on the koji host is as follow ~~’%dist .el7’~~.

Populate the build target with the current master

GOAL: validate before creating all the branches RESULT: the sf-X.X-el7 target populated with the master branches

Make sure no koji build are pending (e.g. gate is free and no tag are expected), This step generates a lot of tag notifications which are currently not configured and the task queue will take times to be cleared.

As the koji admin user, run the script (included in sfinfo):

./zuul_koji_populate_target.py --distro-info sf-X.X.yaml --branch master

Expected results: SUCCESS: sf-X.X-el7 is populated

When master HEAD points at ‘merge commit’, the tool may not be able to discover the n-v-r and it will fail with: “Failed to populate target, it’s missing sf-config-3.0.0.78.g7d9273b-3.el7” In that case, manually import the last build using:

koji tag-build sf-X.X-el7 sf-config-LAST_NVR.el7

If the package is added for the first time to the target the following command must be used (it used to be the case for sf-ui package):

koji add-pkg sf-X.X-el7 sf-ui --owner=sfci

Then remove the sf-release-999 packages:

koji untag-build sf-X.X-el7 sf-release-9999.8.g4a0900f-8.el7

Finally, create a Mash repository:

./zuul_koji_mash.py --distro-info sf-X.X.yaml --build

You can use this baseurl to do deployment test: https://softwarefactory-project.io/kojifiles/repos/sf-X.X-el7/

Repeat this step until the upgrade and the features are working as expected on your test environment.

Create the project branches

GOAL: branch all repositories to prevent new/unstable change to be included in the release RESULT: two config project changes to submit and approve on Gerrit.

Run the script (also included in sfinfo):

pushd ~/koji-git/config && git pull && popd
./zuul_set_distro_branch.py --project-resources ~/koji-git/config/resources/software-factory.yaml --distro-info sf-X.X.yaml
pushd ~/koji-git/config && git add -A . && git commit -m "sf: add main branches for X.X" && git review && popd
./zuul_set_distro_branch.py --project-resources ~/koji-git/config/resources/rpms.yaml --distro-info sf-X.X.yaml
pushd ~/koji-git/config && git add -A . && git commit -m "sf: add rpms distgit branches for X.X" && git review && popd

Verify, approve and then submit the changes on Gerrit. Finally verify on some repositories that the X.X branches have been well created and a ‘ManageSF commit’ have been added on those branches.

Populate the target

GOAL: import built packages from the master koji target that match the content of the branch RESULT: a target populated

Make sure no koji build are pending (e.g. gate is free and no tag are expected), This step generates a lot of tag notifications which are currently not configured and the task queue will take times to be cleared.

Each repository part of the X.X release will be cloned or refreshed and checkouted on the branch X.X. Then exaxt NVR of each package will be determined from each RPM spec file. Finally each NVR will be expected to be found inside the master koji target (because already built there) and be added to the tag sf-X.X-el7.

This command will take some time to execute (~ 1 hour). You can see the koji sf-X.X-el7 tag being populated by checking this page: http://koji.softwarefactory-project.io/koji/buildtargetinfo?name=sf-X.X-el7 on the ‘Destination Tag’ link.

As the koji admin user, run the script (included in sfinfo):

./zuul_koji_populate_target.py --distro-info sf-X.X.yaml --update

It may occurs the command fails when a repository does not contain a valid .spec file (eg. an empty repository). In that case just comment or remove the related repository in the sf-X.X.yaml file and run again the command.

Expected results: INFO [zuulkoji.ZuulKojiPopulateTarget] SUCCESS: sf-X.X-el7 is populated

If you modified the sf-X.X.yaml do not forget to send the change and approve it on Gerrit.

Prepare the release

Update the sf-release package

GOAL: Update the release repository package for the release X.X RESULT: a usable sf-release package

git clone https://softwarefactory-project.io/r/software-factory/sf-release
git checkout X.X

In sf-release.spec change:

Version: X.X
echo X.X > %{buildroot}%{_sysconfdir}/sf-release
Add a new entry in the changelog section

In sf-release.repo change:

[sfrelease-X.X]
name=SF X.X packages
baseurl=https://softwarefactory-project.io/kojifiles/repos/sf-X.X-el7-release/
gpgcheck=1

git add -A
git commit -m "Release sf-X.X"
git review

Make the change validate the CI then approve to submit the change on Gerrit.

Tag all the internal projects

GOAL: get real version number instead of -dev git describe RESULT: all internal project are tagged and a corresponding package has been built and published in the sf-X.X-el7 koji tag.

Internal “projects” (with the ‘internal’ flag in sf-X.X.yaml file) must be tagged if needed (when the project was modified since the last tag). This will result to package (NVR) with a clean version number. Please note that only projects that have changed since the last X.X-1 release that must be tagged.

A job will be executed in the tag pipeline in order to build the SRPM then submit it to Koji in the sf-master-el7 target. Indeed as explained below the tag must be done on the HEAD^1 that is supposed to be a commit in common with the master branch as we just branched the repository in a previous step.

Tag projects with relevant version number, for example managesf:

git review -s # to set the gerrit remote
git tag -l # to read the last tag number
git checkout origin/X.X
git tag -a -m "Z.Z.Z" Z.Z.Z HEAD^   # Z.Z.Z must be at least 'last tag' + 1
git push gerrit Z.Z.Z

Note: do not tag the .gitreview change, use HEAD^ instead so that the tag applies to master branch too. If master and stable branch content are identical (minus the .gitreview update), then master tip can be tag instead.

Wait for zuul tag pipeline to finish.

Import newly tagged build to the stable X.X target

GOAL: first tag on the branch shall be shared with master and the branch, thus it has been published on master target RESULT: import newly tagged build from master target to stable target

Internal “project” builds resulted in packages landed in the sf-master-el7 koji tag then we must run again zuul_koji_populate_target.py in order to add the new builds to the sf-X.X-el7 koji tag.

As the koji admin user, run the script:

./zuul_koji_populate_target.py --update --internal --distro-info sf-X.X.yaml

Expected results: INFO [zuulkoji.ZuulKojiPopulateTarget] SUCCESS: sf-X.X-el7 is populated

Populate the candidate target

GOAL: import all packages from the stable tag (sf-X.X-el7) to the candidate tag (sf-X.X-el7-candidate) RESULT: a release candidate tag populated

As the koji admin user run the command below. Please note the command will take ~ 1 hour to execute. You can follow the tag populate on that page: http://koji.softwarefactory-project.io/koji/tags by clicking on sf-X.X-el7-candidate.

./zuul_koji_populate_target.py --distro-info sf-X.X.yaml --candidate

Then we create a “flat” RPM repository from the koji tag thank to the mash tool.

./zuul_koji_mash.py --distro-info sf-X.X.yaml

A working RPM repository is now available under: http://koji.softwarefactory-project.io/kojifiles/repos/sf-2.7-el7-candidate/ Note the Mash directory that contains the release candidate packages. The repodata directory links to the packages from the Mash directory.

Try an installation of the candidate release

Start a fresh CentOS 7 VM. Then run the following commands:

sudo yum update -y
sudo yum install -y http://koji.softwarefactory-project.io/kojifiles/repos/sf-X.X-el7-candidate/Mash/sf-release-X.X.0-1.el7.noarch.rpm
sudo sed -i 's/-release/-candidate/' /etc/yum.repos.d/sf-release.repo
sudo sed -i 's/gpgcheck=1/gpgcheck=0'/ /etc/yum.repos.d/sf-release.repo
sudo yum install sf-config
sudo sfconfig

Send an annonce on softwarefactory-dev@redhat.com

Subject: Software Factory X.X RC available

Hello folks,

The release candidate of Software Factory X.X is available on our repository. This is a beta version of the next Software Factory so DO NOT use it in production or update a production deployment with the release candidate.

Fell free to test it; Any feedback is welcome. If you find an issue then do not hesitate to report it on the issue tracker: https://tree.taiga.io/project/morucci-software-factory/issues?q=&tags=software%20factory or contact us on our IRC channel on Freenode, #softwarefactory.

Here is the process to deploy the RC on a fresh Centos 7 system: $ sudo -i

Software Factory X.X will be released once the RC is validated.

Best Regards, The Software Factory team.

Create the release tag

GOAL: freeze the candidate tag RESULT: a release tag

As the koji admin user, run:

koji clone-tag --verbose --pkgs --builds --latest-only sf-X.X-el7-candidate sf-X.X-el7-release

Sign the release packages

As the kojiadmin user:

# Install key if needed
gpg --list-keys
gpg --import $signing_key_path
cat ~/.rpmmacros
echo "%_gpg_name release@softwarefactory-project.io" > ~/.rpmmacros

You’ll need the key pass phrase.

./zuul_koji_sign_release.py --distro-info sf-X.X.yaml

Then execute the commands listed by the command.

Create the release repository

./zuul_koji_mash.py --distro-info sf-X.X.yaml --release

Update a release container

GOAL: update a release container RESULT: a new container NVR publish on quay.io/software-factory/

When a change is needed for a release container, the related container image definition must be updated in containers/images-sf/X.X.

The container project’s Makefile must be updated to handle the build of the Dhall defintion for the new container image in order to create the Zuul job definition.

Ensure the new container image is published on quay.io.

Then a sf-config update will be needed in X.X branch and Koji target to make use of the new container image.

Update a release

GOAL: update the release with new changes RESULT: an updated repository

If/when bugs are fixed or unbreaking features in master are backported to the release X.X branch you’ll need to update the release candidate then update the release koji tag and Mash repo.

Tag internal package

If changes occured on some internal (internal flag in sf-X.X.yaml) sources like managesf then you need to git tag the source (so a commit from the X.X git branch of the source). As usual Zuul will run and populate the koji tag sf-X.X-el7.

cd src/$project
git checkout X.X
git tag -a -m X.X.X X.X.X
git push --follow-tags

When tagging a version on a stable branch, it may be required to create a tag for the master branch to ensure upgrade job works as expected. For example:

When doing `sf-config-3.6.1`, then master `git describe` has `sf-config-3.6.0~dev…` so the first commit in master not in 3.6 needs to be tag `3.7.0` so that `git describe` shows `sf-config-3.7.0~dev…` and ensure upgrade job works.

To compare the package list between the master tag and the X.X tag:

./zuul_koji_compare_tag.py --distro-info sf-master.yaml sf-master-el7 sf-X.X-el7

Tag on master

For some package it is easier to simply tag the build from the master repository:

# First submit a review on the 3.4 branch to pass the CI
koji tag-build sf-X.X-el7 nvr
# Then you can push the review

Update and sign the release

To compare the package list between the release candidate tag and the sf-X.X-el7 tag:

./zuul_koji_compare_tag.py --distro-info sf-X.X.yaml sf-X.X-el7-candidate sf-X.X-el7

To update the release candidate, run:

./zuul_koji_populate_target.py --distro-info sf-X.X.yaml --candidate
./zuul_koji_mash.py --distro-info sf-X.X.yaml

You might need to do some test on the candidate before releasing the release update. When you are OK then clone the candidate tag to the release tag.

koji clone-tag --verbose --pkgs --builds --latest-only sf-X.X-el7-candidate sf-X.X-el7-release

Execute the ‘Sign the release’ process

./zuul_koji_sign_release.py --distro-info sf-X.X.yaml

Then execute the commands listed by the command.

./zuul_koji_mash.py --distro-info sf-X.X.yaml --release

Finalise the release

Publish the release RPM

From the softwarefactory-project.io instance:

sudo curl -o /var/www/repos/sf-release-X.X.rpm https://softwarefactory-project.io/kojifiles/repos/sf-X.X-el7-release/Mash/sf-release-X.X.X-X.el7.noarch.rpm

Add the release to the website

In the www.softwarefactory-project.io repository:

Add a new line to the release table in website/content/pages/about.rst
Create a new page in website/content/pages/releases/X.X.rst

Software Factory X.X
####################

:date: 2018-MM-DD hh:mm
:modified: 2018-MM-DD hh:mm
:authors: SF
:status: hidden
:url: releases/X.X/
:save_as: releases/X.X/index.html

Prelude
-------

What is the big new feature/change?


Doc
---

Here_ is the documentation of the X.X release.

.. _Here: https://docs.softwarefactory-project.io/sf-config-X.X/index.html


Release Notes (YYYY-MM-DD)
--------------------------

<insert package changelog, e.g.>
sf-config-X.X.X
~~~~~~~~~~~~~~~

managesf-X.X.X
~~~~~~~~~~~~~~

<insert package list>


Digest
------

The packages are signed with this key:
E46E04A2344803E5A808BDD7E8C203A71C3BAE4B - release@softwarefactory-project.io

<insert sf-release.rpm signature>

Generate changelog

For each internal package:

git log --oneline gerrit/Y.Y..gerrit/X.X

Add changelog for other packages to the release page

Generate new packages list

Using pkgtreediff ( https://github.com/juhp/pkgtreediff ):

export hub=http://koji.softwarefactory-project.io/kojihub
pkgtreediff --rst koji://sf-Y.Y-el7-candidate@$hub koji://sf-X.X-el7-candidate@$hub

Generate sf-release package digest

Since the chain of trust start with the sf-release.rpm file, let’s sign it:

sha256sum sf-release-X.X.X-1.el7.noarch.rpm > digest
gpg -u release@softwarefactory-project.io --clearsign digest

If the passphrase pop up doesn’t show, ensure no gpg-agent are running.

Publish the docs:

On the managesf instance:

cd /var/www/pages/docs.softwarefactory-project.io/
mkdir sf-config-X.X
cd sf-config-X.X
curl -OL https://softwarefactory-project.io/kojifiles/repos/sf-X.X-el7-release/Mash/sf-docs-X.X.0-3.el7.noarch.rpm
mkdir -p ./usr/share/doc/software-factory/
rpm2cpio sf-docs-X.X.0-3.el7.noarch.rpm | cpio -i
mv usr/share/doc/software-factory/* .
rm -Rf usr/ sf-docs-3.5.0-3.el7.noarch.rpm

Send announce

Subject: SF-X.X has been released!

Here is the template to fill and send the Release anounce on softwarefactory-dev@redhat.com.

Hello everyone,

We are pleased to announce the release of the X.X version of Software Factory: https://docs.softwarefactory-project.io/releases/X.X

Best regards, The Software Factory Team

Update SF jobs

Branch the sf-ci project

git clone https://softwarefactory-project.io/r/config
pushd config
# Create a branch for sf-ci
popd

Update the master jobs

git clone https://softwarefactory-project.io/sf-ci
pushd sf-ci
edit zuul.d/jobs.yaml # change sf_version variable in upgrade job (not for sf-ci-base)
git commit -m "sf-ci-upgrade: update stable version"
git review

Update the jobs

git checkout X.X
edit zuul.d/projects.yaml # remove periodic pipeline
git commit -m "sf-ci: remove periodic job from stable branch"
git review

Remove upgrade job from n-1

git checkout X.X-1
edit zuul.d/templates.yaml # remove upgrade jobs
git commit -m "sf-ci-upgrade: remove upgrade jobs from unsupported version"
git review
#+ENG_SRC

*** Remove un-supported branches
Old branch can also be removed from projects. In the config projects, do:

sed -i -e "s/2.6:.*/2.6: '0'/" resources/software-factory.yaml resources/rpms.yaml
git commit -m "sf: remove old branches

sf-resources: allow-delete"
git review

Freeze containers definition

Make a copy of SF master containers definition. This is done to easily handle the bump of a release containers during maintainance period.

cd ~/git/softwarefactory-project.io/containers
cp -R images-sf/master images-sf/X.X
git add images-sf/X.X && git commit -m"Copy master defintion into X.X directory"
git review
#+ENG_SRC

Files

create-release.org

Latest commit

History

create-release.org

File metadata and controls

How-to release a new version of SF

Terminology

SemVer

Koji targets

Koji tags

Prepare the release environment

Create the release branch from master

Prepare the distro info

Create the koji target for the new release

Populate the build target with the current master

Create the project branches

Populate the target

Prepare the release

Update the sf-release package

Tag all the internal projects

Import newly tagged build to the stable X.X target

Populate the candidate target

Try an installation of the candidate release

Send an annonce on [email protected]

Create the release tag

Sign the release packages

Create the release repository

Update a release container

Update a release

Tag internal package

Tag on master

Update and sign the release

Finalise the release

Publish the release RPM

Add the release to the website

Generate changelog

Generate new packages list

Generate sf-release package digest

Publish the docs:

Send announce

Update SF jobs

Freeze containers definition