ci.jenkins.io: upgrade datadog plugin from 7.x to 8.x

[dduportal]

Service(s)

ci.jenkins.io

Summary

Same as #4236

As the past attempts to upgrade the datadog plugin (ref. #4080) did corrupt the Jenkins Home, we have to be careful.

We need to:

• Check changelog and user feedbacks (as it's been a few weeks since last release)
  • https://plugins.jenkins.io/datadog/releases/#version_8.0.0
• Announce operation
• Take a backup of the data disk (snapshot)
• Perform upgrade and check carefully the results
• Rollback snapshot if any error happens due to datadog

Reproduction steps

No response

[dduportal]

Planned for Thursday 14 Nov. (ref. jenkins-infra/status#556)

[dduportal]

Update:

• Announced on status.jio: https://status.jenkins.io/issues/2024-11-14-ci-maintenance/
• Announced on ci.jenkins.io with a top-level message
• Latest available version is now 8.1.x: changelog summary incoming

[dduportal]

Update:

• Backup taken (snapshot of the data disk of ci.jenkins.io named backup-ci.jio-data-20241114-08h20Z)
• Changelog from 7.1.3 to 8.1.0 (note Datadog plugin developer do not publish changelog informations in plugins.jenkins.io but instead they use their own CHANGELOG.md file in the repository, with timestamped-title so not deterministic either 🥲) :
  • 7.2.0 => https://github.com/jenkinsci/datadog-plugin/blob/master/CHANGELOG.md#720--2024-09-24
    • Mostly bug fixes. A refactor implying intermediate states cleanups though: we have to watch it carefully (see Refactor Datadog clients jenkinsci/datadog-plugin#408)
  • 7.2.1 => https://github.com/jenkinsci/datadog-plugin/blob/master/CHANGELOG.md#721--2024-09-24
    • Bugfix only
  • 8.0.0 => https://github.com/jenkinsci/datadog-plugin/blob/master/CHANGELOG.md#800-beta--2024-10-22 (beta and RC are the same)
    • Useful bugfix (dealying pipeline start) which will avoid a lot of warning log messages for us
    • Introduces support for "flare" (e.g. generating support archive to diagnose plugin when failing) 🥳
    • Refactor of logs submission: should make ci.jio a bit faster (or less overloaded) at the cost of a bit more of CPU (compression): Refactor logs submission jenkinsci/datadog-plugin#451. Should not change resource usages but worth watching metrics in the upcoming days
  • 8.1.0 => https://github.com/jenkinsci/datadog-plugin/blob/master/CHANGELOG.md#810--2024-11-06
    • New feature for warning collections and a bugfix
• Checking issues on the plugin tracker:
  • Absent from the changelog: Error Loading Configuration: 'retryLogs' is deprecated jenkinsci/datadog-plugin#467 (comment) we need to update the JCasc setup of ci.jio as part of the 8.0.0 upgrade
  • No other issues recently opened/commented with obvious problems or evidence it could be risky

=> LGTM, let's roll!

[dduportal]

Update:

• Plugin upgrade performed:
  • Stopped Puppet agent
  • Removed manually the retryLogs CasC directive (ref. Error Loading Configuration: 'retryLogs' is deprecated jenkinsci/datadog-plugin#467 (comment))
  • Performed plugin upgrade
  • Restarted controller
• No ERROR logs message found until now
  • Watching for the next hours
  • Triggered Infra Acceptance test job and Jenkins Infra Test plugin main job for sanity check

[dduportal]

Update:

• Closing operation on status.jio (Close CI maintenance (datadog 7.x -> 8.x plugin upgrade) status#557)

• All builds were successful, no ERROR message in ci.jio logs and no corruption

• Metrics observation: it is a bit soon to draw conclusion but we seem to have a minor CPU usage increase:

  

  • The "%user" means Jenkins (and other user processes such as Apache) are using a bit more of CPU resources
  • However it seems we have a bit of "%iowait" + "%system" which hints to contention on the block devices level (system disk? data disk?)
  • Note that, since we are moving ci.jenkins.io to AWS with unified disk an better instance capacity (along with arm64 CPU), this should not be a concern at all (modern arm64 CPUs should manage compression better, and wider PCI lanes means less I/O, along with unified block storage device for less interruptions)