Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Date filters not working in Grafana/Prometheus with gitlab-ci-pipeline-exporter #863

Open
mguassone-enginium opened this issue Jun 19, 2024 · 2 comments

Comments

@mguassone-enginium
Copy link

I'm encountering an issue with date filters in Grafana when using the gitlab-ci-pipeline-exporter. It appears that the filters are not applying correctly to the exported data, and all time series are being displayed as if they occurred today (I think because I started today the gitlab-cipipeline-exporter container), even when filtering for specific dates in the past.

This behavior persists even when attempting to filter directly through Prometheus, suggesting that the issue might lie within the exporter itself or its interaction with Prometheus.

The inability to effectively filter data by date significantly hinders my ability to analyze historical pipeline trends and identify patterns over time.

Observed behavior

  • Date filters in Grafana do not apply correctly to the exported data.
  • All time series are displayed as if they occurred today (I think because I started today the gitlab-cipipeline-exporter container) , regardless of the selected date range.
  • Filtering directly through Prometheus yields the same erroneous results.

N.B.
Note that the date of the pipeline execution (or other pipeline details) is shown correctly, it is the filter on the timeseries that seems to refer to the date with which the exporter exported the data.

Expected behavior

  • Grafana should accurately filter data based on the specified date range.
  • Time series should be displayed for the selected dates, reflecting historical pipeline trends.

Impact

  • Inability to effectively analyze historical pipeline data.
  • Difficulty in identifying patterns and trends over time.
  • Reduced visibility into pipeline performance and health.

Possible solutions

Additional questions

Has anyone else encountered similar issues with date filters and the gitlab-ci-pipeline-exporter?
Are there any known workarounds or community-supported solutions for addressing this filtering problem?
Thank you for your attention to this matter. I appreciate your assistance in resolving this issue and ensuring the accurate handling of date filters in Grafana with the gitlab-ci-pipeline-exporter.

Please let me know if you have any other questions or require further clarification.

Additional Context

docker-compose.yml

---
version: '3.8'
services:
  gitlab-ci-pipelines-exporter:
    image: quay.io/mvisonneau/gitlab-ci-pipelines-exporter:v0.5.8
    ports:
      - 8080:8080
    environment:
      GCPE_GITLAB_TOKEN: ${GCPE_GITLAB_TOKEN}
      GCPE_CONFIG: /etc/gitlab-ci-pipelines-exporter.yml
      GCPE_INTERNAL_MONITORING_LISTENER_ADDRESS: tcp://127.0.0.1:8082
    volumes:
      - type: bind
        source: ./gitlab-ci-pipelines-exporter.yml
        target: /etc/gitlab-ci-pipelines-exporter.yml

  prometheus:
    image: docker.io/prom/prometheus:v2.44.0
    ports:
      - 9090:9090
    links:
      - gitlab-ci-pipelines-exporter
    user: root
    volumes:
      - ./prometheus/config.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    command:
      - "--storage.tsdb.retention.size=50GB"
      - "--storage.tsdb.retention.time=2y"
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--web.console.libraries=/usr/share/prometheus/console_libraries"
      - "--web.console.templates=/usr/share/prometheus/consoles"


  grafana:
    image: docker.io/grafana/grafana:9.5.2
    ports:
      - 3000:3000
    environment:
      GF_AUTH_ANONYMOUS_ENABLED: 'true'
      GF_INSTALL_PLUGINS: grafana-polystat-panel,yesoreyeram-boomtable-panel
    links:
      - prometheus
    volumes:
      - ./grafana/dashboards.yml:/etc/grafana/provisioning/dashboards/default.yml
      - ./grafana/datasources.yml:/etc/grafana/provisioning/datasources/default.yml
      - ./grafana/dashboards:/var/lib/grafana/dashboards

networks:
  default:

volumes:
  prometheus-data:

gitlab-ci-pipelines-exporter.yml

---
log:
  level: debug

gitlab:
  url: ****
  token: ****

pull:
  projects_from_wildcards:
    on_init: true

  environments_from_projects:
    on_init: true

  refs_from_projects:
    on_init: true

  metrics:
    on_init: true
# Pull jobs related metrics on all projects
project_defaults:
  pull:
    pipeline:
      jobs:
        enabled: true
        most_recent: 0
        max_age_seconds: 0
    environments:
      enabled: true
      regexp: ".*"
      exclude_stopped: true
    merge_requests:
      enabled: true
      most_recent: 0
      max_age_seconds: 0
#

wildcards:
  - {}

prometheus/config.yml

global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'gitlab-ci-pipelines-exporter'
    scrape_interval: 10s
    scrape_timeout: 5s
    static_configs:
      - targets: ['gitlab-ci-pipelines-exporter:8080']

grafana/dashboards/dashboard_pipelines.json

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "description": "This dashboard leverages the Prometheus exporter I wrote to fetch information about GitLab CI pipelines statuses. More information here: https://github.com/mvisonneau/gitlab-ci-pipelines-exporter",
  "editable": true,
  "gnetId": 10620,
  "graphTooltip": 0,
  "iteration": 1604494964194,
  "links": [],
  "panels": [
    {
      "datasource": null,
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 2,
        "x": 0,
        "y": 0
      },
      "id": 116,
      "options": {
        "content": "<p style=\"text-align:center;\"><img src=\"https://about.gitlab.com/images/press/logo/png/gitlab-logo-500.png\" width=100px/></p>",
        "mode": "html"
      },
      "pluginVersion": "7.3.1",
      "timeFrom": null,
      "timeShift": null,
      "title": "",
      "transparent": true,
      "type": "text"
    },
    {
      "cacheTimeout": null,
      "datasource": null,
      "fieldConfig": {
        "defaults": {
          "custom": {},
          "decimals": 0,
          "mappings": [
            {
              "id": 0,
              "op": "=",
              "text": "N/A",
              "type": 1,
              "value": "null"
            }
          ],
          "nullValueMode": "connected",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "semi-dark-blue",
                "value": null
              },
              {
                "color": "#d44a3a"
              }
            ]
          },
          "unit": "none"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 2,
        "x": 2,
        "y": 0
      },
      "id": 107,
      "interval": null,
      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "background",
        "fieldOptions": {
          "calcs": [
            "lastNotNull"
          ]
        },
        "graphMode": "area",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "last"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "pluginVersion": "7.3.1",
      "targets": [
        {
          "expr": "count(gitlab_ci_pipeline_run_count{project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\"})",
          "format": "time_series",
          "instant": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "",
          "refId": "A"
        }
      ],
      "timeFrom": null,
      "timeShift": null,
      "title": "PIPELINES #",
      "type": "stat"
    },
    {
      "cacheTimeout": null,
      "datasource": null,
      "fieldConfig": {
        "defaults": {
          "custom": {},
          "decimals": 0,
          "mappings": [
            {
              "id": 0,
              "op": "=",
              "text": "N/A",
              "type": 1,
              "value": "null"
            }
          ],
          "nullValueMode": "connected",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "#299c46",
                "value": null
              },
              {
                "color": "#C4162A",
                "value": 1
              },
              {
                "color": "#d44a3a"
              }
            ]
          },
          "unit": "none"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 2,
        "x": 4,
        "y": 0
      },
      "id": 117,
      "interval": null,
      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "background",
        "fieldOptions": {
          "calcs": [
            "lastNotNull"
          ]
        },
        "graphMode": "area",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "last"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "pluginVersion": "7.3.1",
      "targets": [
        {
          "expr": "count(gitlab_ci_pipeline_status{status=\"failed\", project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\"} > 0) or vector(0)",
          "format": "time_series",
          "instant": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "",
          "refId": "A"
        }
      ],
      "timeFrom": null,
      "timeShift": null,
      "title": "FAILED PIPELINES #",
      "type": "stat"
    },
    {
      "cacheTimeout": null,
      "datasource": null,
      "fieldConfig": {
        "defaults": {
          "custom": {},
          "decimals": 0,
          "mappings": [
            {
              "id": 0,
              "op": "=",
              "text": "N/A",
              "type": 1,
              "value": "null"
            }
          ],
          "nullValueMode": "connected",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "semi-dark-blue",
                "value": null
              },
              {
                "color": "#d44a3a"
              }
            ]
          },
          "unit": "none"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 3,
        "x": 6,
        "y": 0
      },
      "id": 118,
      "interval": null,
      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "background",
        "fieldOptions": {
          "calcs": [
            "lastNotNull"
          ]
        },
        "graphMode": "area",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "last"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "pluginVersion": "7.3.1",
      "targets": [
        {
          "expr": "sum(increase(gitlab_ci_pipeline_run_count{project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\"}[1h]))",
          "format": "time_series",
          "instant": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "",
          "refId": "A"
        }
      ],
      "timeFrom": null,
      "timeShift": null,
      "title": "RUNS # (in the last hour)",
      "type": "stat"
    },
    {
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a",
        "#4040a0"
      ],
      "datasource": null,
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "gridPos": {
        "h": 6,
        "w": 15,
        "x": 9,
        "y": 0
      },
      "id": 110,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "polystat": {
        "animationSpeed": 2500,
        "columnAutoSize": true,
        "columns": "",
        "defaultClickThrough": "",
        "defaultClickThroughNewTab": false,
        "defaultClickThroughSanitize": false,
        "displayLimit": "",
        "fontAutoColor": true,
        "fontAutoScale": true,
        "fontSize": 4,
        "fontType": "Roboto",
        "globalDecimals": 2,
        "globalDisplayMode": "all",
        "globalDisplayTextTriggeredEmpty": "OK",
        "globalOperatorName": "current",
        "globalThresholds": [
          {
            "$$hashKey": "object:618",
            "color": "#299c46",
            "state": 0,
            "value": 1
          },
          {
            "$$hashKey": "object:625",
            "color": "#3274D9",
            "state": 3,
            "value": 2
          },
          {
            "$$hashKey": "object:628",
            "color": "#d44a3a",
            "state": 2,
            "value": 3
          },
          {
            "$$hashKey": "object:631",
            "color": "#959595",
            "state": 3,
            "value": 4
          }
        ],
        "globalUnitFormat": "short",
        "gradientEnabled": true,
        "hexagonSortByDirection": 1,
        "hexagonSortByField": "name",
        "maxMetrics": 0,
        "polygonBorderColor": "#10111c",
        "polygonBorderSize": 1,
        "polygonGlobalFillColor": "#0a50a1",
        "radius": "",
        "radiusAutoSize": true,
        "rowAutoSize": false,
        "rows": 4,
        "shape": "hexagon_pointed_top",
        "tooltipDisplayMode": "all",
        "tooltipDisplayTextTriggeredEmpty": "OK",
        "tooltipFontSize": 12,
        "tooltipFontType": "Roboto",
        "tooltipPrimarySortDirection": 2,
        "tooltipPrimarySortField": "thresholdLevel",
        "tooltipSecondarySortDirection": 2,
        "tooltipSecondarySortField": "value",
        "tooltipTimestampEnabled": false,
        "valueEnabled": false
      },
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "savedComposites": [],
      "savedOverrides": [
        {
          "$$hashKey": "object:591",
          "clickThrough": "",
          "colors": [
            "#299c46",
            "#e5ac0e",
            "#bf1b00",
            "#ffffff"
          ],
          "decimals": "",
          "enabled": true,
          "label": "OVERRIDE 1",
          "metricName": "/.*/",
          "operatorName": "current",
          "prefix": "",
          "sanitizeURLEnabled": true,
          "scaledDecimals": null,
          "suffix": "",
          "unitFormat": "short"
        }
      ],
      "targets": [
        {
          "expr": "(gitlab_ci_pipeline_status{status=\"success\", project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\"} * 1 > 0) or (gitlab_ci_pipeline_status{status=\"running\", project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\"} * 2 > 0) or (gitlab_ci_pipeline_status{status=~\"failed|canceled\", project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\"} * 3 > 0) or (gitlab_ci_pipeline_status{status!~\"success|running|failed|canceled\", project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\"} * 4 > 0)",
          "format": "time_series",
          "hide": false,
          "instant": true,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "                                          {{project}} - {{ref}}",
          "refId": "A"
        }
      ],
      "timeFrom": null,
      "timeShift": null,
      "title": "",
      "transparent": true,
      "type": "grafana-polystat-panel",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ]
    },
    {
      "cacheTimeout": null,
      "datasource": null,
      "fieldConfig": {
        "defaults": {
          "custom": {},
          "decimals": 0,
          "mappings": [
            {
              "id": 0,
              "op": "=",
              "text": "N/A",
              "type": 1,
              "value": "null"
            }
          ],
          "nullValueMode": "connected",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "#299c46",
                "value": null
              },
              {
                "color": "rgba(237, 129, 40, 0.89)",
                "value": 5184000
              },
              {
                "color": "#C4162A",
                "value": 15552000
              }
            ]
          },
          "unit": "dtdurations"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 5,
        "x": 0,
        "y": 3
      },
      "id": 108,
      "interval": null,
      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "background",
        "fieldOptions": {
          "calcs": [
            "lastNotNull"
          ]
        },
        "graphMode": "area",
        "justifyMode": "center",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "/^Value$/",
          "values": false
        },
        "textMode": "auto"
      },
      "pluginVersion": "7.3.1",
      "targets": [
        {
          "expr": "avg(time() - gitlab_ci_pipeline_timestamp{project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\"})",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "",
          "refId": "A"
        }
      ],
      "timeFrom": null,
      "timeShift": null,
      "title": "Average Pipeline Run Frequency",
      "type": "stat"
    },
    {
      "cacheTimeout": null,
      "datasource": null,
      "fieldConfig": {
        "defaults": {
          "custom": {},
          "decimals": 0,
          "mappings": [
            {
              "id": 0,
              "op": "=",
              "text": "N/A",
              "type": 1,
              "value": "null"
            }
          ],
          "nullValueMode": "connected",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "#299c46",
                "value": null
              },
              {
                "color": "rgba(237, 129, 40, 0.89)",
                "value": 600
              },
              {
                "color": "#d44a3a",
                "value": 900
              }
            ]
          },
          "unit": "dtdurations"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 3,
        "w": 4,
        "x": 5,
        "y": 3
      },
      "id": 106,
      "interval": null,
      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "background",
        "fieldOptions": {
          "calcs": [
            "lastNotNull"
          ]
        },
        "graphMode": "area",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "pluginVersion": "7.3.1",
      "targets": [
        {
          "expr": "avg(gitlab_ci_pipeline_duration_seconds{project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\"})",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "",
          "refId": "A"
        }
      ],
      "timeFrom": null,
      "timeShift": null,
      "title": "Average Pipeline Duration",
      "type": "stat"
    },
    {
      "aliasColors": {},
      "bars": true,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "description": "",
      "fieldConfig": {
        "defaults": {
          "custom": {},
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 4,
        "w": 12,
        "x": 0,
        "y": 6
      },
      "hiddenSeries": false,
      "id": 114,
      "legend": {
        "alignAsTable": true,
        "avg": false,
        "current": false,
        "hideZero": true,
        "max": false,
        "min": false,
        "rightSide": true,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": false,
      "linewidth": 1,
      "nullPointMode": "null as zero",
      "options": {
        "alertThreshold": false
      },
      "percentage": false,
      "pluginVersion": "7.3.1",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": true,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum(increase(gitlab_ci_pipeline_run_count{project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\"}[1m])) by (project, ref) / sum(increase(gitlab_ci_pipeline_run_count{project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\"}[1m])) by (project, ref)",
          "format": "time_series",
          "instant": false,
          "interval": "",
          "legendFormat": "{{ project }} - {{ ref }}",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "PIPELINE RUNS",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "datasource": null,
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "custom": {
            "align": "center",
            "displayMode": "auto",
            "filterable": false
          },
          "links": [],
          "mappings": [
            {
              "from": "",
              "id": 1,
              "text": "",
              "to": "",
              "type": 1
            }
          ],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          }
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "ID"
            },
            "properties": [
              {
                "id": "links",
                "value": [
                  {
                    "targetBlank": true,
                    "title": "View pipeline #${__value.numeric}",
                    "url": "https://${GITLAB_HOST}/${__data.fields.project}/-/pipelines/${__value.numeric}"
                  }
                ]
              },
              {
                "id": "custom.displayMode",
                "value": "color-background"
              },
              {
                "id": "color",
                "value": {
                  "fixedColor": "dark-blue",
                  "mode": "fixed"
                }
              },
              {
                "id": "custom.width",
                "value": 85
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "Status"
            },
            "properties": [
              {
                "id": "custom.displayMode",
                "value": "color-background"
              },
              {
                "id": "mappings",
                "value": [
                  {
                    "from": "",
                    "id": 1,
                    "text": "SUCCESS",
                    "to": "",
                    "type": 1,
                    "value": "1"
                  }
                ]
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "Job"
            },
            "properties": [
              {
                "id": "custom.align",
                "value": "left"
              },
              {
                "id": "custom.width",
                "value": 230
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "Project"
            },
            "properties": [
              {
                "id": "custom.align",
                "value": "left"
              },
              {
                "id": "custom.width",
                "value": 218
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "Ref Kind"
            },
            "properties": [
              {
                "id": "custom.width",
                "value": 98
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "Ref Name"
            },
            "properties": [
              {
                "id": "custom.width",
                "value": 122
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "Duration"
            },
            "properties": [
              {
                "id": "unit",
                "value": "dtdurations"
              },
              {
                "id": "thresholds",
                "value": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "green",
                      "value": null
                    },
                    {
                      "color": "light-orange",
                      "value": 900
                    },
                    {
                      "color": "semi-dark-red",
                      "value": 1200
                    }
                  ]
                }
              },
              {
                "id": "custom.displayMode",
                "value": "color-background"
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "Date"
            },
            "properties": [
              {
                "id": "unit",
                "value": "dtdurations"
              },
              {
                "id": "custom.displayMode",
                "value": "color-background"
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 26,
        "w": 12,
        "x": 12,
        "y": 6
      },
      "id": 122,
      "links": [],
      "options": {
        "showHeader": true,
        "sortBy": [
          {
            "desc": true,
            "displayName": "Date"
          }
        ]
      },
      "pluginVersion": "7.3.1",
      "targets": [
        {
          "expr": "-max(time() - gitlab_ci_pipeline_timestamp{project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\"}) by (project, ref, kind) unless max(gitlab_ci_pipeline_status{status!~\"success\", project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\"}) by (project, ref, kind) > 0",
          "format": "table",
          "hide": false,
          "instant": true,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "",
          "refId": "B"
        },
        {
          "expr": "max(gitlab_ci_pipeline_duration_seconds{project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\"}) by (project, ref, kind) unless (max(gitlab_ci_pipeline_status{status!~\"success\", project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\"}) by (project, ref, kind) > 0)",
          "format": "table",
          "hide": false,
          "instant": true,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "",
          "refId": "C"
        },
        {
          "expr": "max(gitlab_ci_pipeline_id{project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\"}) by (project, ref, kind) unless (max(gitlab_ci_pipeline_status{status!~\"success\", project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\"}) by (project, ref, kind) > 0)",
          "format": "table",
          "instant": true,
          "interval": "",
          "legendFormat": "",
          "refId": "D"
        }
      ],
      "timeFrom": null,
      "timeShift": null,
      "title": "SUCCESSFULLY COMPLETED",
      "transformations": [
        {
          "id": "merge",
          "options": {}
        },
        {
          "id": "organize",
          "options": {
            "excludeByName": {
              "Time": true,
              "Value #A": false,
              "job_name": false
            },
            "indexByName": {
              "Time": 0,
              "Value #A": 9,
              "Value #B": 6,
              "Value #C": 7,
              "Value #D": 1,
              "job_name": 3,
              "kind": 4,
              "project": 2,
              "ref": 5,
              "status": 8
            },
            "renameByName": {
              "Value #A": "Status",
              "Value #B": "Date",
              "Value #C": "Duration",
              "Value #D": "ID",
              "job_name": "Job",
              "kind": "Ref Kind",
              "project": "Project",
              "ref": "Ref Name",
              "status": "Status"
            }
          }
        },
        {
          "id": "calculateField",
          "options": {
            "alias": "Status",
            "mode": "reduceRow",
            "reduce": {
              "include": [
                "ID"
              ],
              "reducer": "count"
            }
          }
        }
      ],
      "type": "table"
    },
    {
      "datasource": null,
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "custom": {
            "align": "center",
            "displayMode": "auto",
            "filterable": false
          },
          "links": [],
          "mappings": [
            {
              "from": "",
              "id": 1,
              "text": "",
              "to": "",
              "type": 1
            }
          ],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          }
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "ID"
            },
            "properties": [
              {
                "id": "links",
                "value": [
                  {
                    "targetBlank": true,
                    "title": "View pipeline #${__value.numeric}",
                    "url": "https://${GITLAB_HOST}/${__data.fields.project}/pipelines/${__value.numeric}"
                  }
                ]
              },
              {
                "id": "custom.displayMode",
                "value": "color-background"
              },
              {
                "id": "color",
                "value": {
                  "fixedColor": "dark-blue",
                  "mode": "fixed"
                }
              },
              {
                "id": "custom.width",
                "value": 85
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "Status"
            },
            "properties": [
              {
                "id": "custom.displayMode",
                "value": "color-background"
              },
              {
                "id": "color",
                "value": {
                  "mode": "thresholds"
                }
              },
              {
                "id": "mappings",
                "value": [
                  {
                    "from": "",
                    "id": 1,
                    "text": "RUNNING",
                    "to": "",
                    "type": 1,
                    "value": "2"
                  },
                  {
                    "from": "",
                    "id": 2,
                    "text": "FAILED",
                    "to": "",
                    "type": 1,
                    "value": "3"
                  },
                  {
                    "from": "",
                    "id": 3,
                    "text": "CANCELED",
                    "to": "",
                    "type": 1,
                    "value": "4"
                  },
                  {
                    "from": "",
                    "id": 4,
                    "text": "CREATED",
                    "to": "",
                    "type": 1,
                    "value": "5"
                  },
                  {
                    "from": "",
                    "id": 5,
                    "text": "WAITING FOR RESOURCE",
                    "to": "",
                    "type": 1,
                    "value": "6"
                  },
                  {
                    "from": "",
                    "id": 6,
                    "text": "PREPARING",
                    "to": "",
                    "type": 1,
                    "value": "7"
                  },
                  {
                    "from": "",
                    "id": 7,
                    "text": "PENDING",
                    "to": "",
                    "type": 1,
                    "value": "8"
                  },
                  {
                    "from": "",
                    "id": 8,
                    "text": "SKIPPED",
                    "to": "",
                    "type": 1,
                    "value": "9"
                  },
                  {
                    "from": "",
                    "id": 9,
                    "text": "MANUAL",
                    "to": "",
                    "type": 1,
                    "value": "10"
                  },
                  {
                    "from": "",
                    "id": 10,
                    "text": "SCHEDULED",
                    "to": "",
                    "type": 1,
                    "value": "11"
                  }
                ]
              },
              {
                "id": "thresholds",
                "value": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "green",
                      "value": null
                    },
                    {
                      "color": "light-blue",
                      "value": 1
                    },
                    {
                      "color": "semi-dark-red",
                      "value": 3
                    },
                    {
                      "color": "light-orange",
                      "value": 5
                    },
                    {
                      "color": "rgb(135, 135, 134)",
                      "value": 9
                    }
                  ]
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "Job"
            },
            "properties": [
              {
                "id": "custom.align",
                "value": "left"
              },
              {
                "id": "custom.width",
                "value": 230
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "Project"
            },
            "properties": [
              {
                "id": "custom.align",
                "value": "left"
              },
              {
                "id": "custom.width",
                "value": 218
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "Ref Kind"
            },
            "properties": [
              {
                "id": "custom.width",
                "value": 98
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "Ref Name"
            },
            "properties": [
              {
                "id": "custom.width",
                "value": 122
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "Duration"
            },
            "properties": [
              {
                "id": "unit",
                "value": "dtdurations"
              },
              {
                "id": "thresholds",
                "value": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "green",
                      "value": null
                    },
                    {
                      "color": "light-orange",
                      "value": 900
                    },
                    {
                      "color": "semi-dark-red",
                      "value": 1200
                    }
                  ]
                }
              },
              {
                "id": "custom.displayMode",
                "value": "color-background"
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "Date"
            },
            "properties": [
              {
                "id": "unit",
                "value": "dtdurations"
              },
              {
                "id": "custom.displayMode",
                "value": "color-background"
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 22,
        "w": 12,
        "x": 0,
        "y": 10
      },
      "id": 120,
      "links": [],
      "options": {
        "showHeader": true,
        "sortBy": [
          {
            "desc": true,
            "displayName": "Date"
          }
        ]
      },
      "pluginVersion": "7.3.1",
      "targets": [
        {
          "expr": "-max(time() - gitlab_ci_pipeline_timestamp{project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\"}) by (project, ref, kind) unless max(gitlab_ci_pipeline_status{status=~\"success\", project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\"}) by (project, ref, kind) > 0",
          "format": "table",
          "hide": false,
          "instant": true,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "",
          "refId": "B"
        },
        {
          "expr": "max(gitlab_ci_pipeline_duration_seconds{project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\"}) by (project, ref, kind) unless (max(gitlab_ci_pipeline_status{status=~\"success\", project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\"}) by (project, ref, kind) > 0)",
          "format": "table",
          "hide": false,
          "instant": true,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "",
          "refId": "C"
        },
        {
          "expr": "(max(gitlab_ci_pipeline_status{project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\", status=~\"running\"}) by (project, ref, kind) * 2) > 0 or\n(max(gitlab_ci_pipeline_status{project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\", status=~\"failed\"}) by (project, ref, kind) * 3) > 0 or\n(max(gitlab_ci_pipeline_status{project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\", status=~\"canceled\"}) by (project, ref, kind) * 4) > 0 or\n(max(gitlab_ci_pipeline_status{project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\", status=~\"created\"}) by (project, ref, kind) * 5) > 0 or\n(max(gitlab_ci_pipeline_status{project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\", status=~\"waiting_for_resource\"}) by (project, ref, kind) * 6) > 0 or\n(max(gitlab_ci_pipeline_status{project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\", status=~\"preparing\"}) by (project, ref, kind) * 7) > 0 or\n(max(gitlab_ci_pipeline_status{project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\", status=~\"pending\"}) by (project, ref, kind) * 8) > 0 or\n(max(gitlab_ci_pipeline_status{project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\", status=~\"skipped\"}) by (project, ref, kind) * 9) > 0 or\n(max(gitlab_ci_pipeline_status{project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\", status=~\"manual\"}) by (project, ref, kind) * 10) > 0 or\n(max(gitlab_ci_pipeline_status{project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\", status=~\"scheduled\"}) by (project, ref, kind) * 11) > 0",
          "format": "table",
          "hide": false,
          "instant": true,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "",
          "refId": "A"
        },
        {
          "expr": "max(gitlab_ci_pipeline_id{project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\"}) by (project, ref, kind, job_name) unless (max(gitlab_ci_pipeline_status{status=~\"success\", project=~\"($OWNER).*\",project=~\"$PROJECT\", ref=~\"$REF\"}) by (project, ref, kind, job_name) > 0)",
          "format": "table",
          "instant": true,
          "interval": "",
          "legendFormat": "",
          "refId": "D"
        }
      ],
      "timeFrom": null,
      "timeShift": null,
      "title": "RUNNING, FAILED OR NOT COMPLETED",
      "transformations": [
        {
          "id": "merge",
          "options": {}
        },
        {
          "id": "organize",
          "options": {
            "excludeByName": {
              "Time": true,
              "Value #A": false,
              "job_name": false
            },
            "indexByName": {
              "Time": 0,
              "Value #A": 9,
              "Value #B": 6,
              "Value #C": 7,
              "Value #D": 1,
              "job_name": 3,
              "kind": 4,
              "project": 2,
              "ref": 5,
              "status": 8
            },
            "renameByName": {
              "Value #A": "Status",
              "Value #B": "Date",
              "Value #C": "Duration",
              "Value #D": "ID",
              "job_name": "Job",
              "kind": "Ref Kind",
              "project": "Project",
              "ref": "Ref Name",
              "status": "Status"
            }
          }
        }
      ],
      "type": "table"
    }
  ],
  "refresh": "10s",
  "schemaVersion": 26,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": [
      {
        "current": {
          "selected": false,
          "text": "corporate-gitlab.gigroup.com",
          "value": "corporate-gitlab.gigroup.com"
        },
        "error": null,
        "hide": 2,
        "label": null,
        "name": "GITLAB_HOST",
        "options": [
          {
            "selected": true,
            "text": "corporate-gitlab.gigroup.com",
            "value": "corporate-gitlab.gigroup.com"
          }
        ],
        "query": "corporate-gitlab.gigroup.com",
        "skipUrlSync": false,
        "type": "constant"
      },
      {
        "allValue": ".*",
        "current": {
          "selected": true,
          "text": [
            "All"
          ],
          "value": [
            "$__all"
          ]
        },
        "datasource": "prometheus",
        "definition": "label_values(gitlab_ci_pipeline_id, project)",
        "error": null,
        "hide": 0,
        "includeAll": true,
        "label": "owner",
        "multi": true,
        "name": "OWNER",
        "options": [],
        "query": "label_values(gitlab_ci_pipeline_id, project)",
        "refresh": 1,
        "regex": "/(.*)\\/.*$/",
        "skipUrlSync": false,
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": "",
        "current": {
          "selected": true,
          "text": [
            "All"
          ],
          "value": [
            "$__all"
          ]
        },
        "datasource": "prometheus",
        "definition": "label_values(gitlab_ci_pipeline_id{project=~\"($OWNER).*\"}, project)",
        "error": null,
        "hide": 0,
        "includeAll": true,
        "label": "project",
        "multi": true,
        "name": "PROJECT",
        "options": [],
        "query": "label_values(gitlab_ci_pipeline_id{project=~\"($OWNER).*\"}, project)",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 1,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": ".*",
        "current": {
          "selected": true,
          "text": [
            "All"
          ],
          "value": [
            "$__all"
          ]
        },
        "datasource": "prometheus",
        "definition": "label_values(gitlab_ci_pipeline_id{project=~\"($OWNER).*\",project=~\"$PROJECT\"}, ref)",
        "error": null,
        "hide": 0,
        "includeAll": true,
        "label": "ref",
        "multi": true,
        "name": "REF",
        "options": [],
        "query": "label_values(gitlab_ci_pipeline_id{project=~\"($OWNER).*\",project=~\"$PROJECT\"}, ref)",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 1,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      }
    ]
  },
  "time": {
    "from": "now-1h",
    "to": "now"
  },
  "timepicker": {
    "refresh_intervals": [
      "10s",
      "30s",
      "1m",
      "5m",
      "15m",
      "30m",
      "1h",
      "2h",
      "1d"
    ],
    "time_options": [
      "5m",
      "15m",
      "1h",
      "6h",
      "12h",
      "24h",
      "2d",
      "7d",
      "30d"
    ]
  },
  "timezone": "",
  "title": "GitLab CI pipelines",
  "uid": "gitlab_ci_pipelines",
  "version": 2
}

grafana/datasources.yml

datasources:
- name: 'prometheus'
  type: 'prometheus'
  access: 'proxy'
  org_id: 1
  url: 'http://prometheus:9090'
  is_default: true
  version: 1
  editable: true
@tomina-s
Copy link

tomina-s commented Jul 2, 2024

Hello, @mguassone-enginium
I faced the same problem in Grafana widjet "SUCCESSFULLY COMPLETED", column "Data".
I used this grafana-dashbord from author of exporter: https://grafana.com/grafana/dashboards/10620-gitlab-ci-pipelines/

This fix help me with column "Data" in dashbord:
change "max(time()..." to "min(time()..."

- "expr": "-max(time() - gitlab_ci_pipeline_timestamp{project=~\"$PROJECT\", ref=~\"$REF\"}) by (project, ref, kind) unless max(gitlab_ci_pipeline_status{status!~\"success\", project=~\"$PROJECT\", ref=~\"$REF\"}) by (project, ref, kind) > 0",
+ "expr": "-min(time() - gitlab_ci_pipeline_timestamp{project=~\"$PROJECT\", ref=~\"$REF\"}) by (project, ref, kind) unless max(gitlab_ci_pipeline_status{status!~\"success\", project=~\"$PROJECT\", ref=~\"$REF\"}) by (project, ref, kind) > 0",

I changed it in Grafana GUI (B is the "Date" column).
image

I think it's an original mistake in dashbord.

@tomina-s
Copy link

tomina-s commented Jul 2, 2024

1 - About your wish:

ability to analyze historical pipeline trends and identify patterns over time.

I think gitlab-ci-pipelines-exporter unfortunately don't suposed to do this.
bcs, the main gitlab-ci-pipelines-exporter metrics, that used in dashboard_pipelines.json, says "the most recent pipeline" (see metrics.md?plain=1#L25):

  • gitlab_ci_pipeline_duration_seconds Duration in seconds of the most recent pipeline project, topics, ref, kind, source, variables available by default
  • gitlab_ci_pipeline_id ID of the most recent pipeline project, topics, ref, kind, source, variables available by default
  • gitlab_ci_pipeline_status Status of the most recent pipeline project, topics, ref, kind, source, variables, status available by default
  • gitlab_ci_pipeline_timestamp Timestamp of the last update of the most recent pipeline project, topics, ref, kind, source, variables available by default

2 - And problem with date range:

All time series are displayed as if they occurred today (I think because I started today the gitlab-cipipeline-exporter container) , regardless of the selected date range.

Grafana should accurately filter data based on the specified date range.

Maybe you mean "time range"?
image

I also dissatisfied about this behavior, but I think gitlab-ci-pipelines-exporter just not for this, just only for recent pipelines.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants