[elastic_agent] add system.cpu.cores to CPU utilization calculation in dashboards

[leehinman]

Proposed commit message

Add system.cpu.cores field and normalize CPU usage metrics

• Adds system.cpu.cores as a new gauge field to the elastic_agent_metrics data stream, capturing the number of CPUs on the system.
• Updates the Agent Metrics dashboard CPU usage formula to normalize by core count when available: if system.cpu.cores is present, CPU time is divided by the core count before computing the rate, falling back to the raw system.process.cpu.total.pct value. This gives a more accurate per-core CPU utilization view on multi-core hosts.

Formula for calculating CPU went from

pick_max(
  pick_max(
    counter_rate(
      defaults(max(system.process.cpu.total.time.ms),0)
      )/counter_rate(max(@timestamp)),
    0
  ),
  defaults(max(system.process.cpu.total.pct),0)
)

to

pick_max(
  pick_max(
    counter_rate(
      defaults(
        divide(
          max(system.process.cpu.total.time.ms),
          ifelse(
            count(kql='system.cpu.cores : *') > 0,
            max(system.cpu.cores),
            1
          )
        )
        ,0)
      )/counter_rate(max(@timestamp)),
    0
  ),
  defaults(max(system.process.cpu.total.pct),0)
)

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

• Install elastic-agent on host with single CPU and multiple CPU.
• Install version 2.9.4 of elastic_agent package
• Use version of elastic-agent with and without add system.cpu.cores field to monitoring elastic-agent#14885
• Verify Agent Metrics dashboard still works and CPU utilization is normalized per CPU
• Verify that agents that don't have the system.cpu.cores field continue to display as they do today

Related issues

• Relates elastic-agent self monitoring metrics do not include normalized cpu information elastic-agent#14862

Screenshots

Agent one CPU

Agent 4 CPUs

Agent before elastic/elastic-agent#14885

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 2bdb98c

[infra-vault-gh-plugin-prod]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[elastic-vault-github-plugin-prod]

Package elastic_agent - 2.9.4 containing this change is available at https://epr.elastic.co/package/elastic_agent/2.9.4/