Over the past 24 hours: NuttX CI Checks have become much slower, and it's beyond our control. According to our NuttX Build Monitor at https://lupyuen.github.io/nuttx-github-jobs/build-monitor
The Top Bar shows that our CI Checks are taking up to 9 hours to complete. Normally they complete in 3.5 hours: https://lupyuen.github.io/nuttx-github-jobs/build-monitor
Why are our CI Checks so slow? Has something changed in our CI Workflow?
Let's click into PR 18903. The Elapsed Duration is indeed 8 hour plus (instead of 3.5 hours): https://github.com/apache/nuttx/actions/runs/26104003327
That's the Elapsed Time, but it's not the actual Execution Time for building NuttX. To show the Actual GitHub Runner Minutes consumed: Click "Run Details > Usage". This CI Build consumed 30 hours of GitHub Runners (1 day 6 hours): https://github.com/apache/nuttx/actions/runs/26104003327/usage
And 30 hours of GitHub Runners is quite normal for a Complete CI Build.
Why is Elapsed Time so high, when GitHub Runner Time is normal?
It means that NuttX CI is waiting for an available GitHub Runner. NuttX CI wasted time on the waiting, not the execution. Remember that All Apache Projects are sharing the Same Pool of GitHub Runners? It's highly likely that Other Apache Projects are consuming Too Many GitHub Runners, and depriving us of GitHub Runners.
Recently we seem to be running out of GitHub Runners. Why?
Possibly because Other Apache Projects are overwhelmed by AI-Generated PRs? Watch out for illogical AI-Generated PRs like this, we shouldn't waste any GitHub Runners on them (should we block them?)
Are we really sure it's not something we changed in NuttX CI?
Check the NuttX Mirror Repo. Complete CI Builds are still completing in 3 hours, which means that NuttX CI is still working OK. NuttX Mirror Repo uses its own GitHub Runners, not from the Apache Pool of GitHub Runners: https://github.com/NuttX/nuttx/actions
Why is our GitHub Full-Time Runner Metric so high?
Our own GitHub Full-Time Runner Metric shows that we consumed 30 Full-Time GitHub Runners over the past day, way beyond our quota of 25 Full-Time GitHub Runners: https://lupyuen.github.io/nuttx-metrics/github-fulltime-runners.png
But this number is incorrect. The official report from ASF shows that we consumed only 16 Full-Time GitHub Runners: https://infra-reports.apache.org/#ghactions&hours=24&limit=15&group=name
What's wrong with our GitHub Full-Time Runner Metric?
That's because we don't have access to the Special GitHub APIs needed to extract the Actual Execution Time from each GitHub Job. We only have the Elapsed Time, which is incorrectly inflated, that's why our GitHub Full-Time Runner Metric incorrectly estimates the Execution Time based on the Elapsed Time.
In future we might use the Downloaded GitHub Job Status to compute more accurately the Actual Execution Time and GitHub Runner Minutes. (Work in progress)
Be careful when using PR Review with GitHub Copilot, they will consume our precious GitHub Runners (but ASF doesn't have a report that tells us how many GitHub Runners are used by Copilot sigh)
FYI @simbit18 :-)
Over the past 24 hours: NuttX CI Checks have become much slower, and it's beyond our control. According to our NuttX Build Monitor at https://lupyuen.github.io/nuttx-github-jobs/build-monitor
The Top Bar shows that our CI Checks are taking up to 9 hours to complete. Normally they complete in 3.5 hours: https://lupyuen.github.io/nuttx-github-jobs/build-monitor
Why are our CI Checks so slow? Has something changed in our CI Workflow?
Let's click into PR 18903. The Elapsed Duration is indeed 8 hour plus (instead of 3.5 hours): https://github.com/apache/nuttx/actions/runs/26104003327
That's the Elapsed Time, but it's not the actual Execution Time for building NuttX. To show the Actual GitHub Runner Minutes consumed: Click "Run Details > Usage". This CI Build consumed 30 hours of GitHub Runners (1 day 6 hours): https://github.com/apache/nuttx/actions/runs/26104003327/usage
And 30 hours of GitHub Runners is quite normal for a Complete CI Build.
Why is Elapsed Time so high, when GitHub Runner Time is normal?
It means that NuttX CI is waiting for an available GitHub Runner. NuttX CI wasted time on the waiting, not the execution. Remember that All Apache Projects are sharing the Same Pool of GitHub Runners? It's highly likely that Other Apache Projects are consuming Too Many GitHub Runners, and depriving us of GitHub Runners.
Recently we seem to be running out of GitHub Runners. Why?
Possibly because Other Apache Projects are overwhelmed by AI-Generated PRs? Watch out for illogical AI-Generated PRs like this, we shouldn't waste any GitHub Runners on them (should we block them?)
Are we really sure it's not something we changed in NuttX CI?
Check the NuttX Mirror Repo. Complete CI Builds are still completing in 3 hours, which means that NuttX CI is still working OK. NuttX Mirror Repo uses its own GitHub Runners, not from the Apache Pool of GitHub Runners: https://github.com/NuttX/nuttx/actions
Why is our GitHub Full-Time Runner Metric so high?
Our own GitHub Full-Time Runner Metric shows that we consumed 30 Full-Time GitHub Runners over the past day, way beyond our quota of 25 Full-Time GitHub Runners: https://lupyuen.github.io/nuttx-metrics/github-fulltime-runners.png
But this number is incorrect. The official report from ASF shows that we consumed only 16 Full-Time GitHub Runners: https://infra-reports.apache.org/#ghactions&hours=24&limit=15&group=name
What's wrong with our GitHub Full-Time Runner Metric?
That's because we don't have access to the Special GitHub APIs needed to extract the Actual Execution Time from each GitHub Job. We only have the Elapsed Time, which is incorrectly inflated, that's why our GitHub Full-Time Runner Metric incorrectly estimates the Execution Time based on the Elapsed Time.
In future we might use the Downloaded GitHub Job Status to compute more accurately the Actual Execution Time and GitHub Runner Minutes. (Work in progress)
Be careful when using PR Review with GitHub Copilot, they will consume our precious GitHub Runners (but ASF doesn't have a report that tells us how many GitHub Runners are used by Copilot sigh)
FYI @simbit18 :-)