fix: avoid stopping nginx-agent service on package upgrade

[defanator]

Proposed changes

This change introduces a number of improvements around packaging nginx-agent to ensure that the following behavior is maintained:

1. The nginx-agent service is called with stop argument on package removal.
2. The nginx-agent service is called with restart argument on package upgrade if a service is running at the moment of package upgrade (previously, a service has just been stopped without any notice).
3. On fresh installations, or on upgrades when nginx-agent service from older package is not running, it will not be started automatically.

While here, added OpenRC init scripts to make service nginx-agent command fully functional on Alpine Linux.

Tested on:

• Debian 11 (.deb)
• RHEL 8 (.rpm)
• Alpine 3.18 (.apk)

Unfortunately, FreeBSD is failing to follow the 2nd rule (restart on upgrade if a service was running) due to limitation of the pkg package manager (see freebsd/pkg#2128 for more details).

Fixes #303.

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING document
• I have run make install-tools and have attached any dependency changes to this pull request
• If applicable, I have added tests that prove my fix is effective or that my feature works
• If applicable, I have checked that any relevant tests pass after adding my changes
• If applicable, I have updated any relevant documentation (README.md)
• If applicable, I have tested my cross-platform changes on Ubuntu 22, Redhat 8, SUSE 15 and FreeBSD 13

[netlify]

✅ Deploy Preview for agent-public-docs canceled.

Name	Link
🔨 Latest commit	9d70b92
🔍 Latest deploy log	https://app.netlify.com/sites/agent-public-docs/deploys/65549d7ae600b00007989eee

[defanator]

Tagging @thresheek @fblr @oxpa for the sake of visibility.

[dhurley]

I think the addition of openrc for alpine looks good, but I'm not sure about changing the current behaviour of the agent during upgrade. We require the agent to be stopped by the user before upgrade and then started again after the upgrade by the user.

[defanator]

We require the agent to be stopped by the user before upgrade and then started again after the upgrade by the user.

@dhurley this is quite counterintuitive and stumbling behavior; what was the rationale behind that decision?

To elaborate on a context: we came across the situation when nginx-agent service silently stopped during the usual package upgrade process with the appropriate package manager (yum/dnf, apt, apk) on a large fleet of instances. There were no any errors from corresponding package scripts during the upgrade, and there were no any signs of broken agent's configuration etc - so from the administrator's point of view, upgrade went fine - except the fact that we ended up with stopped monitoring for all affected instances.

I'm wondering where that requirement of manual stopping and starting came from. Agent's process can be restarted gracefully, so why prevent it to do so?

[dhurley]

We require the agent to be stopped by the user before upgrade and then started again after the upgrade by the user.

@dhurley this is quite counterintuitive and stumbling behavior; what was the rationale behind that decision?

To elaborate on a context: we came across the situation when nginx-agent service silently stopped during the usual package upgrade process with the appropriate package manager (yum/dnf, apt, apk) on a large fleet of instances. There were no any errors from corresponding package scripts during the upgrade, and there were no any signs of broken agent's configuration etc - so from the administrator's point of view, upgrade went fine - except the fact that we ended up with stopped monitoring for all affected instances.

I'm wondering where that requirement of manual stopping and starting came from. Agent's process can be restarted gracefully, so why prevent it to do so?

Hi @defanator, actually it looks like our docs might be a bit out of date. Yes you actually shouldn't need to stop the agent before upgrading. When we upgrade the agent, the old version of the agent should continue to run and its not until the user restarts the agent that the new version of the agent is used. Sorry about the confusion.

[dhurley]

Thanks for fixing the issue where the agent was being stopped during upgrades. I just have questions related to upgrading on alpine.

[defanator]

When we upgrade the agent, the old version of the agent should continue to run and its not until the user restarts the agent that the new version of the agent is used.

@dhurley such behavior will introduce an inconsistent setup when your running binary will no longer be available on disk, which is potentially just another delayed point of failure (aka time bomb): imagine a situation when users won't restart service after upgrade till system reboot or another component(s) upgrade that will involve restarting services affected by particular changed shared library - which is a common scenario in modern Ubuntu/Debian - in this situation, in case of any unexpected issues with agent, it will be impossible to just restart the service without doing rollback to previous package version.

Restarting service on upgrade (and failing immediately if a restart was unsuccessful) solves this inconsistency + allows a user to be alarmed on possible issues as soon as possible.

We spent a while building similar patterns for other nginx products (just noticing - given the fact agent's trying to follow established principles / support matrix of nginx/nginx-plus); I'm happy to elaborate more on this if required.

[defanator]

@dhurley @oliveromahony any updates on this one?

[defanator]

TWIMC, rebased this against latest main and tested on these platforms:

• Debian 11 "bullseye",
• Alpine 3.18,
• RHEL 9.3.

Testing included:

1. Building local .deb/.apk/.rpm with VERSION=v2.30.3 (latest tagged) and VERSION=v2.30.4 (bumped for tests).
2. Installing v2.30.3 package on target OS, without any previous version, verifying that the nginx-agent service wasn't started after initial installation.
3. Upgrading v2.30.3 to v2.30.4 via standard package manager tool on target OS, verifying that the service was restarted after upgrade.
4. Removing the package via standard package manager tool, verifying that the service was stopped before package removal/purge.