downstream: enhance release process

[jasonbrooks]

@cgwalters We've talked a bit about tagging downstream releases so that we can associate a particular state of the metadata in the repo with a given release. I was looking at how you do alpha releases, but that seems tied up in ci -- do you have thoughts about how we should do this for downstream releases?

Also, @kbsingh and I were talking in the meeting last week about setting up a process where, upon some release tagging event that happens in the repo, we'd have the nightly signed tree that KB is producing cut over to the official release location, allowing us to bring more automation to downstream releases.

Does that sound like something that'd work with the sort of release tagging I mentioned above?

[cgwalters]

Can you elaborate a bit on what you mean by tagging? And what's a "release"? Are we trying to e.g. snapshot media/images together with the ostree commit?

[jasonbrooks]

This is the conversation I was thinking of: #206 (comment)

[cgwalters]

Oh, we should definitely add git tags. Why not?

[jasonbrooks]

So by release, I'm thinking 7.20160224, 7.20160404, etc. So we'd add a git tag for downstream-7.20170117. And then my thought is that KB's script could pick up on that.

I was just noticing that you do releases differently for alpha, so I wanted to ask about that.

[cgwalters]

Oh, I see. Using git tags instead of the releases.yml model. It's an interesting idea. It seems pretty sane to me.

[jasonbrooks]

What do you think? https://github.com/CentOS/sig-atomic-buildscripts/releases It creates the tag along w/ a space to put some release info. I just linked to the ML message.

[cgwalters]

LGTM other than that I'd always GPG sign git tags (preferably with https://github.com/cgwalters/git-evtag )

[jasonbrooks]

OK, I re-did it w/ git-evtag.

@kbsingh Can we script the process of looking for new downstream releases, and rsyncing into place the associated version of the tree from your nightly composer? So, for the next release, I'd update the metadata as needed, we'd test the updated tree, and when things are ready, I'd cut a new release, and the signed tree would fall into place?

[jasonbrooks]

@kbsingh Any comment on this?

I'd like to get a new release out asap.

[kbsingh]

Since there is no real state maintained in the CI infra, it would be tricky working out when a new tag shows up, however its worth a try. If you go ahead and create a tag, lets try it.

[jberkus]

@kbsingh it it looks OK, can you go ahead and do it? Or grant access to Jbrooks so that he can? It would be really nice if CAH releases were as reliable as Fedora ones are now.

[jasonbrooks]

OK, I created a new tag: https://github.com/CentOS/sig-atomic-buildscripts/tree/downstream-7.20170209

[jasonbrooks]

Fedora has added a new "updates" ref, which automatically updates as new packages become stable out of bodhi, so if a user wants to be running the latest stable pkgs, rather than wait for the next two week release, they can do that.

I'd like to have a similar thing for centos atomic. I have some ideas on how that could work:

First, we could tweak the existing centos-atomic-host.json to target the ref centos-atomic-host/7/x86_64/updates/standard, run the build and sign script nightly, and automatically push updated trees into place where any cah user that wanted to rebase to the updates ref could access them.

Second, we could start including the id of a tested commit from the updates ref in the body of the release tags we're now creating for downstream releases.

Then we put in place a new script that would look at the latest tag in the downstream git repo (git tag | sort -nr | head -1 or similar) to check the downstream version listed there, and then would look at the current "stable" (aka not updates) cah tree and check its version (w/ something like: ostree pull --depth=0 --repo=/srv/repo --commit-metadata-only centos-atomic-host centos-atomic-host/7/x86_64/standard && ostree log --repo=/srv/repo centos-atomic-host/7/x86_64/standard | grep Version).

If the version in the tag and the version in the tree didn't match, that'd mean that there's a new release that's tested and tagged and ready to be made available in the centos-atomic-host/7/x86_64/standard ref.

The script would then mirror the repo, and add the updates ref commit referenced in the release tag to the "stable" ref, including the version number from the tag:

ostree commit -b centos-atomic-host/7/x86_64/standard --tree=ref=49e973a2fef90f724acdd9d7bff80761c24dd1146d45431c8c1b8ad911ce7adc --repo=/srv/repo --add-metadata-string=Version=7.20170301

Then we'd rsync the updated tree back into place.

cc @kbsingh @cgwalters

[cgwalters]

But in CentOS we don't have an updates stream apart from the Core rebuilds right? Or are you thinking of changing that?

Are we trying to solve "test the core rebuild ref before promoting it"? Or are we trying to solve e.g. async security errata, like what happened with kernel a while back?

[jasonbrooks]

We released a new version of the host this week, but the updated packages for that release have been in centos extras since Jan 23. I want to narrow that sort of gap. If there was an async security errata situation, this could help with that as well, but in a case like that, I think we'd consider updating the regular ref as well. A setup like this would both speed our regular releases and allow users to access updated content as soon as it's available, if they chose.

[jberkus]

Relevant to this: CAH is now an entire release cycle behind. So clearly "let's just keep doing what we're doing" is not a realistic option.

[jasonbrooks]

proposed release changes

• Align the centos atomic host release schedule with regular centos: new media, testing and announcements every month.
• Assuming that the primary audience for cah is centos users, this gives users a general idea of what’s in a given monthly release, based on what’s in the equivalent non-atomic release.
• rhelah ships roughly every six weeks, which means that there may be times when a centos atomic host could arrive almost a month after the rhelah release on which it’s based.
• To solve this issue we add a second ref, that’s updated nightly (when there are updated packages). A user that wanted it could rebase to this ref and always be up to date w/ what’s in the repos.
• Fedora does its two-week releases by copying a particular commit from its continuously updated ref to the two-week release ref, and then creating media based on that. We could do the same thing, but on a monthly schedule. https://pagure.io/releng/blob/master/f/scripts/push-two-week-atomic.py#_561

move to buildlogs

• We serve our cah repo from http://mirror.centos.org/centos/7/atomic/x86_64/repo, which isn’t accessible via https. We want to switch to serving the repo from https://buildlogs.centos.org/centos/7/atomic/x86_64/repo, over https
• We need to modify the centos-release-atomic package (https://git.centos.org/summary/rpms!centos-release-atomic) to include the new repo location, and we need to leave the old repo location in place for a while so that users pointed at the old location will receive the update pointing them to the new location.
• Do we plan to serve the media from buildlogs, too? It seems to make sense to continue serving that from the same location as the rest of the media.

[cgwalters]

With baseline CentOS - the monthly new media rolls up the RHEL z-stream security errata mostly right? Or is there anything else going on there?

The monthly centos cycle being potentially out-of-line with RHELAH/Extras seems...weird. Why not try to align?

I'm a big fan of moving to https.

[jasonbrooks]

Updating every six weeks means that cah users must wait longer than regular centos users for access to security errata. That's one big downside of alignment, but it'd be less of an issue if we added an updates ref.

We've lacked regularity in our cah releases. I want that. CentOS revs on a monthly schedule now, with naming attached to that. I'd like to be on that same train. And on that model, if core CentOS opted to do an extra release for a security issue, cah could follow suit.

Whether we ship monthly or ~6 weekly, I'd still like to have the second updates ref.

[cgwalters]

I guess my vote is following base centos - rolling up security errata feels like a very important property.

The out-of-sync nature of Extras/RHELAH with this will be an issue but if we do the updates ref too, seems OK.