Release: new helper script to improve the release process

[portersrc]

During the CoCo release process, there are a couple of tedious steps in the operator repo that can be automated. The script in this PR attempts to do that.

We just used this to help with the v0.10.0 release. The checklist is here. This script helps with steps 4 and 5.

This script has been used for v0.9.0 and now v0.10.0, so it's been run a couple of rounds and been found useful enough to submit for review.

[wainersm]

Hi @portersrc !

That's a great idea to have a script to help on the release. Also it's always nice to see (and learn) new different ways to doing things in bash!

[portersrc]

Hi @wainersm thanks for the helpful feedback. I've tried to address each one.

[ldoktor]

are you deliberately avoiding the use of yq? The use of grep can be fragile in case things re-order or somehow modify... Or just a bit of in-line python, it should be installed everywhere ;-) python -c "import yaml; print(yaml.load(open('$kata_versions_dest'), Loader=yaml.SafeLoader)['externals']['nydus-snapshotter']['version'])"

[wainersm]

In the past we discussed the problem of using yq to write yaml that will result on formatting changes, if I'm not mistaken even with python's lib. However, use yq or python just to read yaml should be acceptable.

[portersrc]

Yes, I left a note in the comments at the top of the script partly for myself about this:

# Note: This script intentionally uses sed. yq (or even python libraries like   
# "yaml" or "ruamel.yaml") do not preserve yaml formatting.                     

I wasted a bit of my life trying to find a good solution for this, and I never found one. It's true that we could read the values out with yq, but in this case we end up writing it back (so, to preserve formatting, we'd still depend on sed, even if we eliminate the grep). In general, I'd prefer this entire script to be python. In fact, I wrote it in python, partly to solve this problem, but then I couldn't find a solution to this issue.

We also considered reading and then writing back to file all of the yaml files with one of these python libraries -- the idea being that once they're formatted by that library, future writes with that library would maintain the same formatting. But this idea has multiple problems, too (introduces a large, ugly PR across at least this repo; when the library version changes, its own formatting can change; and I think I noticed inconsistencies when testing this, too). So, unless I've missed something, grep and sed seems the least bad to me at the moment.

[ldoktor]

I's fun watching developers argue about yaml formatting while they use go which enforces gofmt on compile... Anyway it's not on me to judge, I'd prefer using yq at least to get the values but it's a mild suggestion.

As for having this script in python, IMO it'd be more readable and writing could be performed by re string manipulation and writing the full content afterwards. The re.sub should be safer. Anyway I'm a pythonist so I'm probably biased so take this also as a mild suggestion.

As for now the script is fine, just a bit fragile.

[wainersm]

This debate of yq/python vs sed/grep shouldn't this script any longer, IMHO. We could have it used on v0.11.0 release. Let's resume (or not) that topic in a future opportunity; for now I will approve it as is.

[mythi]

Let's resume (or not) that topic in a future opportunity

It's most certainly going to be that "or not" case. That's why it's better to do it right from the very beginning (and use kustomize edit set image)

[ldoktor]

Could you please document in code how this works rather than a simple comment that knowing the latest commit helps? IIUC you're looking for the oldest merge commit that contains the latest change to the pre-install-payload, am I right?

[portersrc]

Yes, I added a more verbose comment above the function. I'd be grateful for a cleaner way to do this, too, if you see a way.

[ldoktor]

This is definitely a step to the right direction although there are many places with unquoted variables. Would be nice to treat them (you can use shellcheck to find them all ;-) )

[portersrc]

Thanks @ldoktor for the review! I've addressed the minor things and left a couple comments open for you.

[wainersm]

Thanks @portersrc ! I'm sure it will be very useful!

For the v0.110 release I looked at this script to review the changes on #465 were sufficient. It would had saved time if we had used this script!

[wainersm]

@fitzthum @stevenhorsman @mythi could you please review it?

[fitzthum]

LGTM

[ldoktor]

Thanks, functional-wise it seems fine, could you please address the quotation issues shellcheck complains about? (do you have it handy or do you want me to paste the comments in-line?)

[portersrc]

could you please address the quotation issues shellcheck complains about?

Sorry for this -- yes, I addressed everything from shellcheck v0.8.0.

[portersrc]

I rewrote this in python per @ldoktor's advice but never polished it and pushed it. I will double-check this by end of day to see if it's worth the confusion. Adding DO NOT MERGE to title.

[portersrc]

... okay nevermind, I see the discussion above regarding yq/sed/python/etc. I'm fine either way. Any other comments welcome but I'll change title back. Thanks for all reviews everyone.