Pipeline running for days

[danielemoraschi]

I have a project with 18 github repositories in scope and 15 jira boards.
Whenever the pipeline hits the largest repository it gets stuck.

Task13165
refdiff:github:GithubRepo:2:275766
All Subtasks completed
< 1min

Task13166
github_graphql:[repo]
Subtasks running8/41
a day

Task13167
gitextractor:[repo]
All Subtasks completed
< 1min

Task13168
Jira:190
All Subtasks completed

No errors. Just endless:
devlake-1 | time="2025-09-15 08:06:11" level=info msg=" [pipeline service] [pipeline #32] [task #13166] [Collect Job Runs] finished records: 312(not exactly)" devlake-1 | time="2025-09-15 08:06:14" level=info msg=" [pipeline service] [pipeline #32] [task #13166] [Collect Job Runs] finished records: 313(not exactly)" devlake-1 | time="2025-09-15 08:06:21" level=info msg=" [pipeline service] [pipeline #32] [task #13166] [Collect Job Runs] finished records: 315(not exactly)" devlake-1 | time="2025-09-15 08:06:26" level=info msg=" [pipeline service] [pipeline #32] [task #13166] [Collect Job Runs] finished records: 317(not exactly)" devlake-1 | time="2025-09-15 08:06:30" level=info msg=" [pipeline service] [pipeline #32] [task #13166] [Collect Job Runs] finished records: 318(not exactly)" devlake-1 | time="2025-09-15 08:06:36" level=info msg=" [pipeline service] [pipeline #32] [task #13166] [Collect Job Runs] finished records: 319(not exactly)" devlake-1 | time="2025-09-15 08:06:39" level=info msg=" [pipeline service] [pipeline #32] [task #13166] [Collect Job Runs] finished records: 320(not exactly)" devlake-1 | time="2025-09-15 08:06:45" level=info msg=" [pipeline service] [pipeline #32] [task #13166] [Collect Job Runs] finished records: 322(not exactly)" devlake-1 | time="2025-09-15 08:06:48" level=info msg=" [pipeline service] [pipeline #32] [task #13166] [Collect Job Runs] finished records: 324(not exactly)" devlake-1 | time="2025-09-15 08:06:53" level=info msg=" [pipeline service] [pipeline #32] [task #13166] [Collect Job Runs] finished records: 326(not exactly)"

Any idea what it means?

[dosubot[bot]]

Hi @danielemoraschi! I'm Dosu and I’m helping the incubator-devlake team.

The log messages showing slow, steady increments in "finished records" for the "Collect Job Runs" step—without errors—usually mean DevLake is still collecting data, but it's progressing very slowly. This often happens with large repositories if the backend runs out of memory or resources. There are two common causes for this:

1. Resource constraints: When DevLake runs with limited memory (e.g., 4GB RAM), the backend can freeze or become unresponsive during large data collection tasks, especially with GitHub or Jira. Increasing the container memory to 6GB has resolved similar issues for others, allowing the pipeline to complete successfully on large projects while smaller ones finish quickly regardless of memory limits [source].

2. GitHub GraphQL plugin bugs: In some versions (notably v1.0.2-beta7), there was a bug where the "Collect Job Runs" task would collect data as check runs, but the extraction step expected check suite data, causing the process to hang or produce incomplete results. Upgrading to v1.0.2-beta8 or later and recollecting data fixes this issue [source].

Try increasing your DevLake container's memory allocation and make sure you're running the latest version. If you recently upgraded, you may need to recollect data for the affected repositories.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[danielemoraschi]

Hi @dosu thanks for your reply.

We have recently upgraded to v1.0.2 which is now causing issue with Jira actually {"errorMessages":["The requested API has been removed. Please migrate to the /rest/api/3/search/jql API

However, the problem I have raised is with a local version on devlake:v1.0.3-beta5.
I have no set resources constrains locally. I'll look into memory constrains, and if as suggested it is still collecting data I'll leave it run while longer.

[dosubot[bot]]

For the Jira error, Atlassian removed the V2 API endpoint, so DevLake needs to use /rest/api/3/search/jql instead. This was fixed in PR #8559—upgrading to a version that includes this change (like v1.0.3-beta5) should resolve the issue for Jira integration [source].

On the pipeline hanging with your largest GitHub repo: even without explicit resource constraints, large data collections can silently stall if memory is insufficient. Users have resolved similar issues by increasing container memory to at least 6GB [source]. If the pipeline still progresses slowly, check for plugin-specific bugs: in recent versions, the GitHub GraphQL plugin sometimes mismatches data formats between "Collect Job Runs" and "Extract Jobs," causing incomplete extraction or stalls [source]. If you upgraded recently, a full data recollection may help.

For large repos, you can also try enabling shallow cloning, setting SkipCommitStat and SkipCommitFiles to true, and ensuring NoShallowClone is false to reduce memory usage [source]. Batch processing and double cloning methods are available if you need further optimization.

If the pipeline continues to hang or crawl, sharing more logs or your pipeline config could help pinpoint the bottleneck.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[danielemoraschi]

Is it via NO_SHALLOW_CLONE in .env? I cannot find any documentation regarding this.

[dosubot[bot]]

NO_SHALLOW_CLONE is not a recognized environment variable in DevLake, which is why you can't find documentation for it. Shallow cloning is controlled via the NoShallowClone option in the plugin or pipeline configuration (not via .env). If you want to enable shallow cloning (to reduce memory usage for large repos), make sure NoShallowClone is set to false in your plugin/task config. The gitextractor plugin only reads environment variables like UseGoGit, SKIP_COMMIT_STAT, and SKIP_COMMIT_FILES for git behavior—not NO_SHALLOW_CLONE or anything similar (source).

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[danielemoraschi]

Pipeline been running now for 3 days.
Still unclear how to enable NoShallowClone given I'm using the docker setup. But anyway, I'll leave it run for a few more days :)

Sample logs: devlake.log

CONTAINER ID   NAME                  CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS 
c7cc338bea1b   devlake-config-ui-1   0.00%     20.16MiB / 11.67GiB   0.17%     171MB / 170MB     9.24MB / 8.19kB   15 
91daa6e27923   devlake-devlake-1     3.22%     195.3MiB / 11.67GiB   1.63%     9.66MB / 51.3MB   131kB / 7.27MB    16 
0e400bc74fda   devlake-grafana-1     0.12%     131.8MiB / 11.67GiB   1.10%     714kB / 359kB     6.65MB / 1.57MB   19 
5830135a0bf4   devlake-mysql-1       3.81%     865.7MiB / 11.67GiB   7.25%     5.1GB / 1.64GB    15.2GB / 82.6GB   45 

Thanks for your help!

[klesh]

Thanks for the log file. The content of devlake.log doesn't appear to be from the DevLake backend. Could you please confirm it was generated by the devlake-devlake-1 container?

[klesh]

NoShallowClone is irrelevant to your problem.
It seems like you are running into #8563
It should be fixed in the latest beta https://github.com/apache/incubator-devlake/releases/tag/v1.0.3-beta6

[danielemoraschi]

I shared logs here #8576 (reply in thread)
It was the only stage running.

[klesh]

Thanks for the log file. The content of devlake.log doesn't appear to be from the DevLake backend. Could you please confirm it was generated by the devlake-devlake-1 container?

[danielemoraschi]

That is from devlake. Tailing logs via docker compose logs -f. I've only split into new lines the content of the [Collect Job Runs] fetchAsync >>> done outputs.
Here's the same log file without new lines added devlake.log

[klesh]

Based on the following log message, it appears the failure was caused by the response body size exceeding the allowable limit on GitHub's servers, which resulted in the response being killed.

Can you file a Bug Report in the issues section? Thanks.

[danielemoraschi]

Hi @klesh

Thanks. I have now raised #8614