Cumulative metrics for job/pipeline durations

[slsyy]

Right now we have only gitlab_ci_pipeline_job_duration_seconds or similar metrics, which cannot be used to answer questions like this:

• how 95th percentile of particular job duration changes over time
• what is the average duration of all pipelines over time

, because most of metrics returns data only for the latest job/pipeline.

Do you think that adding metrics such as histogram of durations will be consistent with the idea of the project and bring some added value?

[wnobres-sr]

I'm looking for the same possibility actually so we can properly track the impact of improvements or identify decrease of performance of the jobs overtime. Is there a way to achieve that using the existing implementation or would require development effort?

[slsyy]

Is there a way to achieve that using the existing implementation or would require development effort?

I don't think so. Current metrics keep only status for the last run. I will try to implement this kind of metrics to see, if it makes sense. Please inform me, if you are working on something similar or you have an alternative solution

[cgill27]

Agree this would be good data to either collect or some way to pull it together in Grafana.
Would love to be able to set thresholds and alerts in Grafana to notify when there's an obvious problem across all pipelines/jobs etc.