Prevent release to be triggered depending on a setting saved "as code"

[lemeurherve]

Release has been inadvertally triggered. The setting preventing it was set manually via the UI, but unfortunately nothing allows registering it as code at the moment so it was automatically reverted when changes were applied to the infra configuration as code.

(To be completed)

I propose we add a kind of semaphore in the form of an environment variable which could be setup as code in the infra, and to add a test on this env var in the release pipeline.

WDYT?

EDIT: related issue

[timja]

I think last time this happened @olblak or maybe someone else got some jobdsl that can be added to disable the job just using regular Jenkins built-in functionality

[daniel-beck]

The previous workaround was to disable puppet for a day or two. I think @dduportal did that last time?

[timja]

The Kubernetes management job, yes that's been done before as well.

I checked through history and couldn't see the jobdsl but I do think that was looked at

[dduportal]

• Last time(s), you remember well: we disabled Kubernetes Management
• Disabling either Kubernetes management (or puppet when it's ci.jenkins.io), as a reminder, disable the ability of the infra team to deliver changes and fixes: we have to keep this until a viable solution is found but it's an additional risk (manual == unsustainable)

The root cause here is the fact that the only way to configure Jenkins with Infra as code is through JobDSL that does NOT expose the attribute "enable/disable" job as far as we can tell.
Since we use a configuration as code system that is applied frequently (every 30 min in worst case) to ensure that changes are auditable (compared to manual clicks on the UIs), then the JobDSL process is re-enabling the job, which is annying (we all agree on that).

The proposal from Herve is a good one: if we can implement, in the release pipeline, a check on a variable/file/lock that is independant from the infra as code and from Jenkins (file seems a good way), then we would have a very nice kill switch.

Additionally (and really nice) proposal from @wadeckfollonier is to add a notifications (IRC, mails, slack, whatever) when the release build start: since it takes 1-2 hours before doing the maven release, it would add an additional layer of protection as we could stop the build manually when everything else fails.

[olblak]

The proposal from Herve is a good one: if we can implement, in the release pipeline, a check on a variable/file/lock that is independant from the infra as code and from Jenkins (file seems a good way), then we would have a very nice kill switch.

I like this idea very much, what about having a specific json page on status.jenkins.io?

[dduportal]

The proposal from Herve is a good one: if we can implement, in the release pipeline, a check on a variable/file/lock that is independant from the infra as code and from Jenkins (file seems a good way), then we would have a very nice kill switch.

I like this idea very much, what about having a specific json page on status.jenkins.io?

Not sure to see the relationship with status.jenkins.io ? Could you elaborate ?

[olblak]

what I have in mind is purely theoretical but the idea would be to publish a specific page on status.jenkins.io
with a message saying that the weekly release is delayed until the security release happen.

From this script we could check if the page is up on status.jenkins.io and based on that information we abort or continue the release process.

We could use the environment variable "RELEASE_PROFILE" == weekly to know if we want to fail early `
https://github.com/jenkins-infra/release/blob/075401a0117d9a5313fdc554d9c3a786d58be329/Jenkinsfile.d/core/weekly#L32

Hugo allows to publish json page as well