Many pipelines not displayed

[Alex7851]

We have thousends pipelines in our gitlab.
Me exclude jobs, environments data and increase refresh intervals but problem about not displaying some many pipelines is actual.

Its not a problem with some projects because i see some pipes from project and same pipes from this projects don't displayed and its still have existing branches, not removed (after merge or smth).

I suppose it's about big data from Gitlab. I see about 300-400 pipes in PIPELINES panel but we have several thousends pipes

My current config is:

gitlab:
  url: 'gitlab instance url'
  token: 'mytoken'

wildcards:
  - 
    pull:
          refs:
            branches:
              regexp: ^.*$
    
pull:
  pipelines:
    enabled: true
    on_init: true
    refs:
      regexp: ^.*$
  projects_from_wildcards:
    on_init: true
    scheduled: true
    interval_seconds: 3600

  environments_from_projects:
    on_init: false
    scheduled: false

  refs_from_projects:
    on_init: true
    scheduled: true
    interval_seconds: 900

  metrics:
    on_init: true
    scheduled: true
    interval_seconds: 180

project_defaults:
  pull:
    pipeline:
      jobs:
        enabled: false
metrics:

  pipeline:
    enabled: true
  job:
    enabled: false
  lint:
    enabled: false
  project:
    enabled: false
  group:
    enabled: false

How can i improove this config to see all pipes in gitlab without omitted pipelines?

[steelecliftonberry]

Hey Alex, how did you go?

Did you notice any pattern yet about which refs do export pipeline metrics and which don't?

When you turned on trace level debugging, did you see anything out of ordinary in the logs? Anything about queueing, warnings, errors, rate limiting, etc. Perhaps you can post a representative snapshot of some logs that cover 1 or 2 entire runs?

Maybe you can also cross-check your config against this link documenting it? I don't see any metrics top-level key yet you use one in your config - if so maybe something else is also off with your config too? https://github.com/mvisonneau/gitlab-ci-pipelines-exporter/blob/main/docs/configuration_syntax.md

I would also consider the max_age_seconds setting - probably you don't care about pipelines from a month ago right?

You might also want to take the query grafana runs in that panel and check the result directly with a Prometheus query - do you still only see 300-400 results? If not, maybe Grafana simply does not display them all