WIP Upgrade to rebar3 3.18, with improvements in packaging and release assembly

[hmmr]

This is a patchset inspired by the work I have been doing for some months on Riak CS/Stanchion 3.0. It includes:

• Upgrade to rebar3 3.17 + a patch fixing a call to git revlist in repos with same-name branch and tag, and eventually, to 3.18 (where that patch has been merged). The bug this patch addresses is triggered when (for example) a build is attempted on dockerhub.com.
• Brush up packaging for rpm- and deb-based distros:
  • all now include riak.service unit files and require systemd;
  • the full list of distros where packaging was verified now includes Centos 7 and 8, Debian 11 and Amazon Linux 2;
  • make package now works on OSX 14, where it is essentially the result of make rel, tarred.
• Get rid of the - option to su in the launcher script, which removes the su barrier for env vars (something requested in another PR).
• Avoid copying leveldb git history when making rel- targets.
• Refactoring of the release assembly:
  • make make rel relocatable again (specifically, to enable OSX package);
  • make riak-admin command equivalent to riak admin command (also applies to riak-repl and riak-debug);
  • point cuttlefish and rebar3_cuttlefish to TI-Tokyo repos, with support for fqdn type and bin_script template synchronized with that in relx-4.5.0.

There are no changes in Erlang code.

Testing was done as follows:

• on each of the OSes covered, Riak package is built natively, with make package, and installed;
• Riak CS and Stanchion packages are built and installed, just as above;
• riak pid, riak ping, riak-admin status work as expected;
• riak-admin test passes;
• riak-admin member-status and other cluster-related commands work;
• riak-debug succeeds;
• riak-cs-admin test succeeds;
• Riak can be built with this Dockerfile and run as part of Riak CS suite in Riak CS Docker Bundle.

[martincox]

Why does each command need to be present in this case - is this not already handled in the extended start script as extensions?

[hmmr]

It's an effort to make hyphenated and non-hyphenated forms equivalent.

[martincox]

I don't think it works as expected, and I'm not even sure that resurrecting the hyphenated commands is warranted - what purpose does it serve? It's been a couple of years since we moved to start script extensions. You now have to re-insert a bunch of boilerplate environment stuff at the top of the script files to make them work, whereas that was all inherited with from the root start script with extensions.

My opinion is that it's a backwards step, unless there's a compelling reason that I'm not aware of / considering.

The tidy up across the rest of the release config is very decent though, and much needed. Some of the deps currently reference TI forks, which would need to target basho, but guess that would be fixed up before everything gets merged in.

[hmmr]

To deal with the easier part first: re-pinning deps to TI-Tokyo repos has a single purpose: pull in cuttlefish and rebar3_cuttlefish, not so much for the fqdn data type in cuttlefish but rather for several commits in the plugin that make it compatible with rebar3 3.18 (by ioolkos, see this one and thereabout), synchronize its internal install_update_escript with the "upstream" version of it in relx, and last but not least, this little gem.

I note that the fqdn-type patch has already been merged in basho/cuttlefish, but there is no tag. If you do the tagging, I will then prepare the necessary PRs to bump cuttlefish version where necessary (it is an explicit dependency in bitcask, clique, eleveldb and riak_sysmon), and collect a patchset for another PR in rebar3_cuttlefish.

Regarding the phasing out of hyphenated forms, I have no strong opinions and will gladly follow the party line (although I would invite @Bob-The-Marauder to this discussion, who has been in the business of dealing with customers for some time).

For the record, my understanding of the underlying machinery is as follows:

• The riak script generated by relx and/or cuttlefish provides basic commands like start, stop, console, pid, ping, rpc. It is how riak is started/stopped in riak_test setting. It is not under our control.
• The other riak, the one created from the template in rel/files, has a distinct and specific purpose:
  • it takes care of the pid file (relx-generated one will not do this for us);
  • it does su as appropriate, and is therefore installed in PATH when riak is installed from a package;
  • it launches riak-admin etc when called as riak admin.

I saw this last item precisely as a way to enable non-hyphenated forms (if we remove riak-admin and its hyphenated friends from PATH, the move to hide hyphenated forms will be complete, from the user standpoint). Apparently, by "move to start script extensions" you mean to do it differently -- perhaps by injecting code in the script produced by relx? If so, I would be happy to rework my contribution accordingly, but I didn't see how relx can be massaged to achieve that.

[hmmr]

@martincox Re:

If it is absolutely necessary to bridge the gap of hyphenated forms being deprecated, why not take a less invasive route and create proxy scripts for the hyphenated forms that live in /usr/sbin/ with something in the form of:

With 1ce0262, when installed from a package on a debian11 system, we now have:

root@debian:~# riak admin
Usage: riak-admin { cluster | join | leave | backup | restore | test | 
                    reip | erl-reload | wait-for-service | 
...
root@debian:~# riak-admin
The hyphenated form of this command is deprecated. Please fix your scripts to call `riak admin` instead.

Usage: riak-admin { cluster | join | leave | backup | restore | test | 
                    reip | erl-reload | wait-for-service | 
                    ringready | transfers | force-remove | down |

With regards to the PID writing - does that not work? It should be handled as part of the start script. I mean, I've just checked over a few clusters (3.*) and all have PIDs in the expected directory.

It's another possible regression. I have the extended_start_script_hooks stanza in relx/deb section in rebar.config, unchanged since 3.0.9, but if I delete this line, the pid file is not written. riak pid still works (pid is obtained by calling os:getpid/0, so no wonder). I will investigate.

[martinsumner]

@hmmr wanted to try test this today, but had issues fetching the referred to tag in riak_repl. Is the TI-Tokyo riak_repl repo public?

[hmmr]

@martinsumner It is indeed private. The change I pushed in TI-Tokyo repo is just s|git://|https://| on top of tag riak_kv-3.0.9. I guess you can do this change manually in the official repo, tag it riak_kv-3.0.10, and I will then re-point riak_repl to the new tag accordingly in this PR.

I have traced the pid file regression, which was in rebar3_cuttlefish plugin (it was introduced by vernemq people in their more recent changes, and adopted by too eager me). I have reverted to version 0.2.0, tweaking it to work with rebar3.18 (and noticed that @martincox has already done the same). I have just re-pointed rebar3_cuttlefish to basho/develop branch, and it is great again.

There are still issues with f75df62, which I am trying to fix, but you should be able to build it locally with make rel.

[hmmr]

Superseded by #1108.