nuget authentication issues from version 1.25 #921
Comments
|
hmm not sure why it is using such large fonts in the end. sorry it's unintentional... |
|
Just for reference, this is related to #919 |
|
It seems to only happen with dotnet/nuget projects. Our frontend/npm repos run fine. I temporarily set dependabot task to run 1.24 image for now to have everything fully functional again. Also, regarding that comment:
I recognized that I only counted log lines, so real number of requests would be half of it I guess... Not sure if it matters a lot. |
|
Not sure if related but we get tihs error |
mburumaxwell
changed the title
|
As @JensSchadron has stated, this is indeed an issue resulting from the merge of #919 but it was something on hold for a while. See comment. The issue is only affecting NuGet when private registries/feeds are in use. All other scenarios and ecosystems seem to continue working as usual. The causeThe Microsoft team decided to overhaul the way NuGet works in dependabot so that it supports the new GitHub Advanced Security for Azure DevOps in PR #8179. How come it does not affects the GitHub-hosted version, you asked? Well, there are two things I can think of:
SolutionFirst some context:
Hopefully, this gets fixed in the core repo. The solution is probably similar to that for PNPM and it's open for anyone to give a try. Secretly, I am hoping that this break would result in Azure DevOps getting native support and this extension being killed. It's better for everyone (or so I think). WorkaroundBefore a solution is found, you can do something. If you are using the server, set the If you are using the extension:
I hope this gives the much needed clarity. |
|
thanks @mburumaxwell ; using 1.24 for now worked. |
|
Tracking dependabot/dependabot-core#8597 which will possibly be fixed by dependabot/dependabot-core#8679 |
|
I am tracking fixes in the core repo, such as dependabot/dependabot-core#8748, then adding them to #927 for testing purposes. If anyone is interested in helping me test you can check out #927 (comment) and subscribe to the PR for updates. |
|
Not sure, but according to this MS article, the problem could be that for v3 nuget protocol at Azure DevOps Artifact Feed it is not sufficient to assemble a nuget.config containing the PAT (as done here by dependabot-core), but Azure Artifacts Credential Provider needs to be used... |
This is an interesting find. Thank you. Seeing that Microsoft (Azure DevOps) guys are the ones that changed the implementation, I hope they can either rollback or fix it. |
|
Or we add the nuget plugin to the image and provide it the PAT via env variable as described here. |
@cyberblast can you contribute a PR for that? |
|
Not sure if I can manage to do that.
If anybody can support with that, it would be highly appreciated. |
|
I found some piece of dockerfile code which our developers are usually using to create images for our own apps. ARG PAT
ENV NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED true
ENV VSS_NUGET_EXTERNAL_FEED_ENDPOINTS '{"endpointCredentials":[{"endpoint":"https://pkgs.dev.azure.com/<org>/_packaging/<feed>/nuget/v3/index.json","username":"docker","password":"'${PAT}'"}]}'
RUN curl --proto "=https" --tlsv1.2 -sSf -L https://raw.githubusercontent.com/Microsoft/artifacts-credprovider/master/helpers/installcredprovider.sh | sh
COPY ./nuget.config ./I guess the missing piece would be to assemble the correct endpointCredentials json format from dependabot config and set as VSS_NUGET_EXTERNAL_FEED_ENDPOINTS before resolving dependencies .. or something like that |
Just linking these here as I think they're related: |
|
May be fixed by dependabot/dependabot-core#8927 |
|
This seems to have fixed the issue for me. |
|
Was the fix released? Ho to get it working with the latest dependabot version? |
|
Not really sure anymore what is happening: the day before yesterday it worked. Yesterday our dependabot pipeline failed with Today it worked again. And today I see this error in a different dependabot pipeline of ours, which worked yesterday. Now I am in the situation that I can`t use dependabot for nuget packages due to this issue, and neither for npm packages due to #729 😟 |
|
I just updated the image version from 1.24 to 1.26.4 because suddenly we were no longer getting work items linked to the PRs created. However 1.26.4 seems to give an error accessing the private feed we have like so:
If anyone has any idea how we can get the private feed to work that would be great. Seems this issue is still actively preventing people from going beyond 1.24. |
|
Downgrading from 1.26.671 -> 1.24 "fixed" our pipeline (Telerik nuget feed gave authentication errors), hoping on a permanent fix :) |
mburumaxwell
changed the title
What updated that broke linking work items? It's still works on my side with 1.24 as imageTag |
I am not sure if it still is broken. Haven't used this in a while. The challenge with this is how much the NuGet space was changing. It is still not back to what it was before Microsoft overhauled it. On GitHub there are still issues like the whole update failing because of multiple target frameworks in Directory.Build.props or timing out for big repository (I have one with 50 projects in it). Overall, the changes in typings are quite a number so I kept off till it stabilizes to focus on where my time is better spent than chasing my own rail.
Deleting the use of Dependabot in your repository is one way to go about it. There are better options like contributing a fix, pressurize Microsoft to bring official support, migrate to GitHub, or keep using 1.24. |
|
1.24 works fine for me too, including work item linking. @Blackunknown it might be worth double checking that the user account Dependabot is running as has read access to the project and can read work items as originally I had your same issue but it started working after giving the user account project read access. Despite the regression in dependabot-core, this project is still huge time saver for me and I really appreciate all your work @mburumaxwell. |
|
Are there any considerations when pinning the 1.24 version? E.g. is the image going to be retained for the foreseeable future? |
|
We just set this up for the first time yesterday (Azure DevOps repos, private Azure DevOps nuget feed) and naturally ran into various of these problems. Pinning back to However we then wanted to implement "grouping" of dependabot updates, and the settings didn't seem to be working no matter how we configured them in the dependabot.yml file. We then discovered that grouped updates for nuget was also broken and only "fixed" recently in upstream dependabot-core here - AFTER the above extension version 1.2.4 (roughly around 1.27 of this extension I believe)... however due to the private feed nuget auth issue it seems there is no workable solution if you want grouping of updates? So I guess I would say one consideration when pinning to 1.24 is that you wont get "grouping" It's a bit hard to follow all the open issues etc so Im not sure where to look to understand if this is a known issue and there is a PR with proposed fixes we could help test on our setups. Do we just keep trying |
|
Today my dependabot pipelines are failing even on |
|
@DaleMckeown worked 1h ago for me with a private nuget repository. |
We run this periodically and the last time was 5 AM and all our repositories worked :) |
|
Cheers both, I'll check over my config. I might have broken something else. |
|
@SeMuell @Rutix Would you be able to tell me how you're setting the PAT token? We currently have it set in a template file (which used to work) but now the auth docs specify that it doesn't work from template variables. Are you creating the PAT as a variable group in DevOps -> Pipelines -> Libraries, as an environment variable (not sure how that is done), or adding them to each pipeline directly? |
|
Closing this, everything should be working fine now, especially with version 1.29.0. Maybe some private feeds may not be working if they need dependabot/dependabot-core#8927 to be resolved. Let us use that issue to track that particular behaviour/problem. |
|
For me updating to 1.29 worked with a private Azure DevOps artifact feed. |
|
Same, 1.29 works much better now.
If you are having the issue where the NuGet/MSBuild/dotnet tools are failing with "error NU1301: Unable to load the service index for source" then the below workaround resolved it for me. EDIT: This workaround is not required if using v1.30.1 or higher. See: #1233. This workaround hijacks the fix for #834 to install the NuGet credential provider, then injects the creds via environment variables. This ensures that dotnet CLI tools are able to auth with the NuGet feed. It's really hacky, but it works. - task: dependabot@1
inputs:
dockerImageTag: '1.29'
...snip...
extraEnvironmentVariables: 'WORKAROUND_CMD=sh -c "$(curl -fsSL https://aka.ms/install-artifacts-credprovider.sh)";NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED=true;VSS_NUGET_EXTERNAL_FEED_ENDPOINTS={"endpointCredentials":[{"endpoint":"https://pkgs.dev.azure.com/...your_feed_slug.../nuget/v3/index.json","username":"unused","password":"$(System.AccessToken)"}]}'
@mburumaxwell I'm not sure if this is a dependabot-core issue or not, but from my limited understanding auth should be passed from dependabot down to the dotnet CLI tools right? it doesn't appear to do this correctly for me and it seems like dependabot/dependabot-core#8927 would remove the need to do this workaround right? |
|
@rhyskoedijk this is a perfect workaround. Hadn't envisioned the PR by @BobSilent being of use this way. Guess it will stick around indefinitely. What you have done in the workaround is exactly what dependabot/dependabot-core#8927 intends to do. From what I understand, this only affects NuGet feeds hosted by Azure DevOps. The hosted version of dependabot handles authentication quite differently and this might be why they haven't seen the issue and needed to fix it. IIRC, they do not pass credentials to the job but instead inject them on outgoing requests that match the feed which is why it works on GitHub but not on other self-hosted places. This is must be what is being explained in dependabot/dependabot-core#8927 (comment) One way would have been to override settings when building docker images but that can't be generalized for everyone given the feed URL needs to be known before hand. (Forgetting the extras scoped just to the NuGet ecosystem). I may be a bit wrong with my approach/understanding but before moving to doing updates with NuGet/dotnet CLI, credentials worked just fine. They still do for other ecosystems unless in cases where one is fighting with the Azure DevOps permissions mess. My take is keep running the workaround and simultaneously apply pressure on the PR to be merged. |
|
Marvelous. Unfortunately, now I have errors in all npm based repos as described in #1046 (uninitialized constant RequirementsUpdateStrategy). But seems thats a different issue again... Anyway, thanks for solving it that far, first time it runs without pinning to 1.24 😄 A bit sad that it's so slow now. |
|
Pinning it back to 1.24 as sometimes it keeps hanging on a task until the job times out (after 300 minutes), keeping subsequent repos unprocessed. |
…d 1.24 no longer exists
|
@cyberblast Any luck on this? I'm having the same issue in my npm registry. It just runs until timeout. I see a bunch of 404 when it tries to locate a specific version of a package. |
|
@eli-gc there are several known authentication issues when using Task V1; Assuming you are using V1 and it is possible to do so, I'd recommend trying to migrate to Task V2; If you are already using Task V2, please log an issue with a copy of your dependabot and pipeline configuration and logs so we can investigate. |
|
@rhyskoedijk Am running into this issue with the V2 task. I'll make a new issue. Thanks. |
and others
Describe the bug
Hi,
we are having sudden issues with Dependabot in Azure DevOps since 1.25. We did not change anything on the setup but it is simply failing.
Here's some excerpt from the Pipe log:
I also recognized, that it is suddenly fetching > 1000 nuget dependency files. Where as before it were only like ~90.
Eventually the pipe has been cancelled by DevOps due to long run execution.
I also doublechecked permission of build service on the private Artifact stream (due to the 401) and it is unchanged and should work as before. But as seen in the log excerpt, it is also failing while checking for a public package and we are not maintaining upstream in our private artifact feed but fetching public packages directly from nuget instead.
To Reproduce
We are running Dependabot nightly like this:
With a git repo in Azure DevOps and this configuration:
Expected behavior
We did not change anything. It should simply run as before.
Screenshots
-
Extension (please complete the following information):
Server (please complete the following information):
-
Additional context
The text was updated successfully, but these errors were encountered: