Timeout running job when updating NuGet packages after updater logic was re-written in C# #9375
Comments
github-actions
bot
added
L: dotnet:nuget
abdulapopoola
removed
L: go:modules
|
Today's log Nothing between @brettfo, hi! Do you have any ideas or know someone who can help? |
|
@batkaevruslan Two questions for you:
There's obviously something very slow happening and I'd like to experiment a bit to see where we could potentially speed things up. There are some instances, however, where there's not much we can do, e.g., if dependabot is running against a huge repo and trying to process all updates. The C# rewrite calls into the real NuGet and dotnet code which does much more checking, etc., and is certainly slower than the original code which was a sort of best-guess. |
|
@brettfo ,
Can you add some more logs to see what is going between 11:12:02 and 11:52:15 ? |
|
@brettfo , And there was a timeout. There is also some unknown error, but I don't see it in logs... P.S. the update log is so huge that GitHub fails to return it (50x errors) and if GitHub returns a response, Chrome fails to render it :) |
|
@batkaevruslan Thank you for this, I'll start experimenting. |
|
@brettfo , hi! I've seen your pr #9566 and run Dependabot update on my public repo: at least, GH does not fail rendering the update log :) Timeouts are still here. Log show 5k+ requests to |
|
The telemetry calls shouldn't be the determining factor in the timeouts, but I'll keep experimenting with your repo. |
|
For a quick follow-up, I removed the telemetry calls and started running an update on the sample repo provided and right now it's still going at 4 hours. There will certainly be more work to get the time down, but at some point we'll hit a wall because much of the work now is done either directly with the |
|
@brettfo, thanks for update!
|
|
@batkaevruslan I'm not aware of any setting to change the max run length, but given how long I've left this running locally with no end in sight, I don't think that's a viable option. As for modifying the dependabot.yml, you should be able to split it up a bit by dependency name. I haven't tried this yet, but you might be able to do something like this: version: 2
registries:
public-nuget:
type: nuget-feed
url: https://api.nuget.org/v3/index.json
updates:
- package-ecosystem: nuget
directory: "/"
registries:
- public-nuget
schedule:
interval: daily
time: "07:00"
timezone: "America/Los_Angeles"
open-pull-requests-limit: 15
groups:
# each group here should run as a separate update job
MicrosoftExtensions: # this group name is purely for you
patterns:
- "Microsoft.Extensions.*" # here we group a few dependencies at a time
FluentAssertions:
patterns:
- "FluentAssertions" # you can list multiple patterns
- "FluentAssertions.*"
# add any other groups you see fit |
|
Just for history: using grouping does not help to avoid timeouts. An attempt to use several
|
I'm also seeing this on a private repo and I'm curious about this comment. Did the old solution not have to call make similar API calls to determine out of date dependencies? When I run I know dependabot collects release notes and the commit diff. Is it this extra information that causes the wall to be hit? |
|
@brettfo, hi! Any news on the issue? I was wondering if our product solution is the largest in the world and the only one suffering from timeouts after this release...
|
|
@batkaevruslan This is something we're actively working on, but the short version is that we don't want to generate PRs that will fail the end user's CI and with the example you shared, updating the requested package resulted in one or more |
Looks like you are about my another issue #6710
Is it possible to build a graph of all dependencies once and create PRs only for leaf level dependencies? Even if PR grouping feature of Dependabot won't work, it is better for us to have one PR per leaf dependency every day than no PRs at all. |
|
Same issue here. After 60 minutes, Azure is killing our Dependabot run due to timeout and it is not possible with our tier to increase the timeout to more then 60 minutes. Also, I am not sure what Dependabot is doing: we see 20.000 lines of GET/POST log entries in our pipeline log. |
|
Yep, same issue here, it times out after 60 minutes. Last run had 70k log lines and then got cancelled. |
|
@roldengarm: I managed to figure out the issue and a solution: Microsoft is killing the pipeline after 60 minutes, no matter which timeout we specify, due to the fact that we used free Microsoft-hosted agents. Now we use the paid Microsoft-hosted agents and have a timeout of 360 minutes. To archive this, you have to order the paid agents and set the timeout value of the pipeline to 0. But I still don't understand why we need > 100k requests and > 2 hours to update a handful of dependencies. I observed that a lot of requests result in 401/403 errors, followed by 200 status code responses. For me (as somebody who has absolutely no background knowledge of Dependabot) is seems as if Dependabot first tries to authenticate without credentials, then receives an authentication error as a response and only then includes credentials. |
|
I'm using Github, not ADO, so not sure how to increase the timeout. Thanks @Patrick-3000 |
|
There ideally shouldn't be any 401/403 codes in the log, are you passing in a token via As for lots of requests, the NuGet protocol dictates that every package needs to be downloaded to properly determine the dependencies so there is expected to be a lot of network traffic during a NuGet update. |
|
@brettfo: Yes, I understand. However, the current algorithm leads to issues which prevents us from using Dependabot on our main repo: due to the runtime of > 2.5 hours we get pipeline timeouts and our agents run out of disc space due to log files of > 10 GB. My (to be honest, very naive) approach would be to just update every Nuget dependency. If there is no newer version, fine. If there is, we update it. In this case, we would save a lot of round trips (and therefore runtime and disc space for logs). And I am sure, that ~100 Nuget packages won't take > 2.5 hours to update. |
|
By the way, Dependabot fails to update the repro repository, that I mentioned earlier, with the error about disk space: |
|
@batkaevruslan: for this problem, I opened issue tinglesoftware/dependabot-azure-devops#1473 |
|
@Patrick-3000 Even looking for newer packages means we have to download and extract it to see what it's dependencies are. It's not uncommon for a package update to not be possible because some transitive dependency won't work. There's not really a quick way around it, otherwise we'd be generating massive amounts of PRs that don't build. |
@brettfo, on the other hand, for myself and many others, dependabot simply no longer works and isn't generating PRs because it times out most or even all of the time. |
I may be speaking for myself, but I'd be fine with seeing lots of update PRs that don't build. It highlights that newer versions are available and it's up to the human developer if they want to make the necessary changes to fix compatibility issues. I'd be very happy if there was an option to force the Dependabot NuGet module into "dumb mode", where it simply makes PRs for any immediate dependencies that have a new version available (grouping and filtering by Dependabot config settings). Alternatively, have you guys considered any kind of NuGet version cache? On a cache miss download the package and extract, examine the dependencies, store in a database somewhere. |
|
@brettfo: thank you for your response. In my mind, I was thinking about something similar to the "Manage NuGet Packages" tab in Visual Studio. There, I can see and update all NuGet updates. In Visual Studio this works very quickly, that is why I can't comprehend why it takes so long for Dependabot to figure out if new updates are available. |
|
@Patrick-3000 When running locally, your machine has a package cache where downloaded packages are saved. For example, you may have downloaded Newtonsoft.Json/10.0.1 several years ago and that now exists on your machine and never needs to be downloaded again, but when dependabot starts, it doesn't have anything so has to download all packages. Further, Visual Studio has already loaded all packages and transitive dependencies, but dependabot is working from a fresh call to We've discussed pre-loading common packages in the dependabot Docker image, but that would make the updater image absolutely huge and from a security standpoint, isn't good. Meaning we don't want to pre-load the package Newtonsoft.Json/13.0.3 if your code uses a private NuGet feed and Newtonsoft 13.0.3 hasn't been approved; your IT department has only loaded version 12 onto your feed. Dependabot needs to honor that. |
|
@brettfo: thank your very much for your explanation. |
|
Is there any news here? Any workarounds? Back in November Dependabot stopped working for me as well. I noticed back then, but I had a lot of other, more important things to work on. I also assumed it would be fixed in a couple of days/weeks, but this is seemingly still an issue. Turning off grouping seems to make it work again - at least it's generating a couple of PRs - but generates way too many overlapping PRs which makes it a show-stopper. I want a single PR that updates everything; Makes testing a lot easier. Nov 18 was the last time my weekly dependabot update worked. |
|
As a workaround and probably a permanent solution, we switched to https://github.com/renovatebot/renovate instead of using dependabot because of this issue. |
Is there an existing issue for this?
Package ecosystem
NuGet
Package manager version
No response
Language version
No response
Manifest location and content before the Dependabot update
No response
dependabot.yml content
No response
Updated dependency
No response
What you expected to see, versus what you actually saw
Hi! Since the release of https://github.blog/changelog/2023-11-28-improvements-to-nuget-support-for-dependabot/
Dependabot stopped working in our non-tiny mono repository. Before that, it had been working for more than half a year.
What is going on between
05:49:57and06:28:32?Native package manager behavior
No response
Images of the diff or a link to the PR, issue, or logs
No response
Smallest manifest that reproduces the issue
No response
The text was updated successfully, but these errors were encountered: