Bugwarrior not starting with Gitlab

[Andonome]

task version: 2.6.2

bugwarrior-pull version: unsure. I installed once through pip3 on Void (python 3.11.2), and tried again with the AUR version (and python 3.10.10). There is no output from bugwarrior-pull --version, and no bugwarrior executable in the $PATH.

bugwarriorrc file:

[general]
targets = gitlab
taskrc = ~/.taskrc # this is a temporary task directory, which has only one task called 'test'

[gitlab]
service = gitlab
gitlab.login = andonome
gitlab.token = [ same token used for glab, which is working ]
gitlab.host = gitlab.com

Running bugwarrior-pull means it hangs for a long time, then outputs:

INFO:bugwarrior.db:Service-defined UDAs exist: you can optionally use the `bugwarrior-uda` command to export a list of UDAs you can add to your taskrc file.
INFO:bugwarrior.services:Starting to aggregate remote issues.
INFO:bugwarrior.services:Spawning 1 workers.
INFO:bugwarrior.services:Working on [gitlab]
ERROR:bugwarrior.services:Worker for [gitlab] failed: Non-200 status code 500; 'https://gitlab.com/api/v4/projects?simple=True&page=89&per_page=100'; '{"message":"500 Internal Server Error"}'
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/bugwarrior/services/__init__.py", line 504, in _aggregate_issues
    for issue in service.issues():
  File "/usr/lib/python3.10/site-packages/bugwarrior/services/gitlab.py", line 484, in issues
    all_repos = self._fetch_paged(tmpl + '?' + urlencode(querystring))
  File "/usr/lib/python3.10/site-packages/bugwarrior/services/gitlab.py", line 377, in _fetch_paged
    items = self._fetch(tmpl, params=params)
  File "/usr/lib/python3.10/site-packages/bugwarrior/services/gitlab.py", line 366, in _fetch
    return self.json_response(response)
  File "/usr/lib/python3.10/site-packages/bugwarrior/services/__init__.py", line 481, in json_response
    raise IOError(
OSError: Non-200 status code 500; 'https://gitlab.com/api/v4/projects?simple=True&page=89&per_page=100'; '{"message":"500 Internal Server Error"}'
INFO:bugwarrior.services:Done with [gitlab] in 260.110228s
INFO:bugwarrior.services:Terminating workers
ERROR:bugwarrior.command:Aborted (critical error in target 'gitlab')
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/bugwarrior/command.py", line 73, in pull
    synchronize(issue_generator, config, main_section, dry_run)
  File "/usr/lib/python3.10/site-packages/bugwarrior/db.py", line 357, in synchronize
    for issue in issue_generator:
  File "/usr/lib/python3.10/site-packages/bugwarrior/services/__init__.py", line 572, in aggregate_issues
    raise RuntimeError(
RuntimeError: critical error in target 'gitlab'

Is this a problem with the Gitlab section, or my setup?

[ryneeverett]

Thanks for the report. Looks like a bug to me.

I've noticed some similar but not identical errors on the latest development branch for the past month or so. However they aren't fatal so I'd ignored them. Since you're seeing this on an older version of bugwarrior -- presumably the latest 1.8.0 release -- I'm guessing that there was a change in the gitlab API (at least as implemented by gitlab.com) that we have not yet accommodated. I'll try to take a look at this soon.

In the meantime, as a workaround, you might try the development version of bugwarrior. We have instructions for installing from source but they are dated (#987). The development branch is develop rather than master.

There is no output from bugwarrior-pull --version

Yes, the --version flag has been added since the latest release.

[ryneeverett]

Looking at this further, i think your issue is likely due to not having any filters on your service definition, causing bugwarrior to try to pull all issues from all the thousands of public repositories on gitlab.com, which can result in a variety of errors. Since the last release we've added validation to make sure gitlab.com users use at least one filter -- such as gitlab.owned = True or gitlab.membership = True -- to prevent this.

[ryneeverett]

I pinned down the error I was seing in #988.

[Andonome]

I've confirmed this works with both commands (though gitlab.membership = True crashed once, probably due to many tasks and no other filters).

The documentation on Gitlab doesn't mention this requirement. Should I open a new issue for that?

[ryneeverett]

In the unreleased development version there is a validation check which requires gitlab.com users to apply some kind of filter: https://github.com/ralphbean/bugwarrior/blob/a37750e45555bd7fe6a7ac749f271d7159d034cd/bugwarrior/services/gitlab.py#L76. I don't think we should document this now that bugwarrior catches it and gives immediate feedback.

[Andonome]

Okay. Thanks for your help.

[ryneeverett]

It would be nice if we could more broadly catch rate-limit errors and give a nice error message explaining the problem. Historically though, I've found gitlab to demonstrate inconsistent error messages due to high volume requests -- several different error codes which could also indicate different problems. Without consistency it's unfortunately difficult to improve this aspect of the service.

[Andonome]

If the example file recommended gitlab.only_if_assigned then the problem would present less often, and if that first pull worked, it would be clear that the subsequent pulls without it were due to rate limiting.

[ryneeverett]

I like that line of thought -- it is a bit weird that our example configuration is actually invalid.

However, I think including gitlab.owned = True in the example -- with a comment explaining that it's not strictly required, would be a better option because it is closer to the way most other services, such as github, operate by default.

In fact, the best improvement might be to change the default to gitlab.owned = True. I'd want to make that change incrementally, starting with requiring all users to explicitly define gitlab.owned in the next breaking release.