Configurable granularity for PR rate limiting

[bmish]

What would you like Renovate to be able to do?

The current prHourlyLimit option does not provide as granular control as we would like for rate limiting the number of PRs that are created/modified. It also doesn't provide a global or per-run rate limit (also requested in #4279), instead only providing a per-repository rate limit currently.

We have hundreds of repositories and thousands of out-of-date dependencies. In order to onboard these repositories and work through all the dependency updates in an efficient, timely manner, without overwhelming our CI/build/package systems, we need a more fine-grained rate limiting option.

Note: We use Bitbucket Server and a cron job to run renovate every 15 minutes.

Describe the solution you'd like

We would like to be able to define multiple rate limits. Each rate limit would include the PR limit, timeframe or per-run, and the scope of global or per-repository.

Example rate limits:

1. A global rate limit of 10 PRs per 15 minute time period, to avoid overwhelming our CI/builds/npm package registry.
2. A global, per-run rate limit of 10 PRs, to avoid overwhelming our CI/builds/npm package registry.
3. A per-repository rate limit of 20 PRs per two hours. This could help avoid opening up too many PRs that are likely to end up with conflicts after each dependency merge.

Here's how that example could look with a new, more sophisticated replacement option for prHourlyLimit:

prRateLimiting: [
  { limit: 10, timeframe: "15m", scope: "global" },
  { limit: 10, timeframe: "run", scope: "global" },
  { limit: 20, timeframe: "2h", scope: "repository" }
]

It would also help if the options and documentation for rate limiting clarified:

1. Whether the rate limiting is per-repository, per-run, or global.
2. That rate limiting applies to both PR creations and modifications.

References

• See previous discussion about rate limiting in Rate limit PR creation rate #1363.

[rarkins]

I assume that PR creation or updating/rebasing cause the same load for you? Ie what you need is to rate limit commits per time period instead of PRs.

[rarkins]

Also any solution here needs to handle the situation where two or more Renovate runs overlap. Ie storage of state must handle multiple readers/writers even if that’s an undesirable situation.

[rarkins]

Now is probably a good time to also consider the different types of "noise" in a repo.

When we make or update commits to branches, each commit usually causes a build on CI. i.e. commits cause CI noise, not really PRs. A PR could have many commits over its lifetime and each causes CI to run.

If you use the default Renovate setting of prCreation=immediate then you can approximate branch creation to PR creation, so limiting creation of either has the same effect. We currently use "PR" in our rate limiting but actually we are doing branch creation limiting, not PR.

But if you use settings like prCreation=not-pending or prCreation=approval then it's possible to get multiple commits to a branch before the PR is created. Once a branch/PR is created, it may be rebased (e.g. due to conflicts) or updated (e.g. due to newer versions available).

When it comes down it, people normally care about one or both of these types of "noise":

• CI noise (every commit potentially causes a build, maybe creating a backlog)
• PR creation noise (PRs may disturb developers due to notifications, and then keep dominating the PR list of a repo)

@bmish it sounds like the first type of noise (commits/CI) is your main concern. In this case though it may be useful to finally take a commits-focused approach to rate limiting rather than branch/PR-based. The commitFilesToBranch function in our git layer could track and check counters and then throw a rate-limited error if any limit is hit.

For scopes, we should probably support "repository", "org" and "global". For the hosted app, we naturally couldn't let people set global limits, but "org" would make sense for them. For self-hosted, you may want it to be global if you have multiple orgs/projects and want a shared limit.

[bmish]

@rarkins all of that sounds right. In addition to making this based on commits (to address the CI load) and allowing the scope to be configured, do you like the ideas of allowing multiple rate limits and configurable timeframes?

[rarkins]

@bmish yes, I think multiple rate limits and configurable timeframes are both good

limit: always a number
timeframe: can be in hours, minutes, or "run", with an upper bound of 24 hours
scope: global, repository

I've removed "org" from here as it was for the app only and I realised it's pretty complicated and should be done later.

Per run is the easiest to implement because it can simply be an in-memory counter that's initialized at startup (e.g. part of initPlatform) and then incremented until it reaches its limit. commitFilesToBranch can then throw a new error to indicate global or repository run rage exceeded.

Time-based limits (minutes or hours) need to persist between runs. We can either:

• Read the existing counter at the start of a repo and then write an updated one at the end, or
• Read/write every single commit

[viceice]

Global timeframe based limits would require a storage, repo based we could use the commit /pr creation times

[lephuongbg]

We would also like renovate to be able to rate limit per manager. Our use case is like this:

• We have poetry and npm managers enabled.
• With poetry, as soon as one renovated branch is merged, any other renovated branch will surely become conflicted due to hash conflict in lockfile
  ⇒ Set pr concurrent limit for poetry to just 1.
• With npm, the lockfile won't be conflicted between different renovated branches
  ⇒ Set concurrent limit for npm to be more than 1.