Update LibGit2 to version 1.1.1 #1905

jairbubbles · 2021-08-22T08:21:14Z

Use package produced by Update libgit2 to v1.1.1 libgit2sharp.nativebinaries#114
Rename some native methods an change return type (see Change API instances of fromnoun to from_noun (with an underscore) libgit2#5117 and Return int from non-free functions libgit2#5365)
- Thanks @A-Ovchinnikov-mx for poviding the fix! (cherry-picked from mendix/libgit2sharp@f958478)
Update some tests to match the fact that "origin/HEAD" is now added when cloning (see clone: update origin's HEAD libgit2#5651)

codecov-commenter · 2021-08-22T08:24:45Z

Codecov Report

Merging #1905 (9941d39) into master (f5098f4) will decrease coverage by 0.00%.
The diff coverage is 85.71%.

@@            Coverage Diff             @@
##           master    #1905      +/-   ##
==========================================
- Coverage   84.81%   84.80%   -0.01%     
==========================================
  Files         231      231              
  Lines        9125     9120       -5     
==========================================
- Hits         7739     7734       -5     
  Misses       1386     1386

Impacted Files	Coverage Δ
LibGit2Sharp/Core/NativeMethods.cs	`68.11% <ø> (ø)`
LibGit2Sharp/Core/Proxy.cs	`94.12% <81.81%> (-0.03%)`	⬇️
LibGit2Sharp/ObjectDatabase.cs	`86.28% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f5098f4...9941d39. Read the comment docs.

skysb · 2021-08-24T10:07:48Z

Hi, is there an update on this draft or any ETA on when it will be merged to master? It contains an important update to allow git version 1 which would be amazing to have for sparse checkouts, partial clones etc.

jairbubbles · 2021-08-24T10:23:03Z

Hi @skysb! The PR is pretty new so give some time to the maintainers 😅.

It's still draft as it's using as pre-release version of LibGit2Sharp.NativeBinaries.

@bording Do you think it's possible to publish a pre-release version of this package? Also I feel like it would make sense to publish the current as a stable 0.27.0 and 0.28.0-preview would be for the next version with a recent libgit2.

LibGit2Sharp.Tests/CloneFixture.cs

LibGit2Sharp/Core/NativeMethods.cs

bording · 2021-08-24T13:25:26Z

Do you think it's possible to publish a pre-release version of this package?

Yes, I should be able to publish a new preview once this gets merged.

Also I feel like it would make sense to publish the current as a stable 0.27.0 and 0.28.0-preview would be for the next version with a recent libgit2.

I'm a bit reluctant to do that as this point because the 0.27.0 preview packages include the change over to a managed HTTPS implementation that is definitely a bit experimental, and does seem to be causing some problems. I'd prefer to wait for libgit2/libgit2#5974 to be merged, pull that into the native binaries package, and then revert the managed HTTPs changes.

jairbubbles · 2021-08-24T14:50:18Z

Yes, I should be able to publish a new preview once this gets merged.

But I don't think we can merge with the prerelease package referenced?

bording · 2021-08-24T14:53:01Z

But I don't think we can merge with the prerelease package referenced?

Why not? The prerelease native package is on nuget.org, so a preview LibGit2Sharp could be released that references it.

jairbubbles · 2021-08-24T14:57:14Z

Why not? The prerelease native package is on nuget.org, so a preview LibGit2Sharp could be released that references it.

Let me undraft then! 😊

bording · 2021-08-24T15:36:12Z

0.27.0-preview-0119 is up on nuget.org now.

alfonsarmi · 2021-08-24T16:45:20Z

Thanks for this!

In Windows is working like a charm with partial cloned repos but in Linux I'm having the next error message:

LibGit2Sharp.LibGit2SharpException: 'unsupported extension name extensions.partialclone'

Any clue?

ethomson · 2021-08-24T16:51:02Z

Hi, is there an update on this draft or any ETA on when it will be merged to master? It contains an important update to allow git version 1 which would be amazing to have for sparse checkouts, partial clones etc.

libgit2 doesn't support partial clones or sparse checkouts. I think that the update that you're referring to allows us to understand the repository format v1, which adds in new optional extensions (like partial clones or sparse checkouts). The update in libgit2 only allows us to know that we don't support the repository's extensions.

skysb · 2021-08-25T06:40:30Z

Hi, is there an update on this draft or any ETA on when it will be merged to master? It contains an important update to allow git version 1 which would be amazing to have for sparse checkouts, partial clones etc.

libgit2 doesn't support partial clones or sparse checkouts. I think that the update that you're referring to allows us to understand the repository format v1, which adds in new optional extensions (like partial clones or sparse checkouts). The update in libgit2 only allows us to know that we don't support the repository's extensions.

Hi @ethomson , You are right, I didn't explain myself correctly in the post. The idea was to be able to identify v0 from v1 only, as when we tried instantiating a new repository with a partially cloned repository, we got an error saying it wasn't supported.

skysb · 2021-08-25T06:44:00Z

Thanks for this!

In Windows is working like a charm with partial cloned repos but in Linux I'm having the next error message:

LibGit2Sharp.LibGit2SharpException: 'unsupported extension name extensions.partialclone'

Any clue?

Hi @alfonsarmi ,

I am getting the same issue. When trying to instantiate a partially cloned repository, I get the same error.

@jairbubbles would this be something related to the native binaries? as in windows it works well but in linux (docker images) it fails for me with the error

LibGit2Sharp.LibGit2SharpException: 'unsupported extension name extensions.partialclone'

jairbubbles · 2021-08-25T07:49:08Z

@jairbubbles would this be something related to the native binaries? as in windows it works well but in linux (docker images) it fails for me with the error

I don't explain why there's a different behavior between Linux and Windows. I quickly looked at the code and there's no #IFDEF around it. (https://github.com/libgit2/libgit2/blob/main/src/repository.c#L1669).

If I understand correctly @ethomson's answer, we should get the error 'unsupported extension name extensions.partialclone' on both platforms.

For my use case, I was also able to open a partially cloned repository and access the info I needed (on Windows). Please note that I don't use libgit2sharp to run actions on the repository, it's only used to inspect it. I guess that it mostly works correctly even for partially cloned repositories.

@alfonsarmi @skysb Can you share the git commands you use to clone your repository?

skysb · 2021-08-25T07:57:20Z

@jairbubbles

The gitclone command I am currently using is

git clone --filter=blob:limit=1m <repo_url> <folder>

Using libgit2sharp, just doing a

Repository repo = new Repository(folder);

raises the exception stated above, but only in linux, and not in windows.
I am not trying to run any actions on the repository, I am just trying to inspect the repository.

I hope this helps as I am pretty new to this!

Thanks!

jairbubbles · 2021-08-25T08:28:52Z

@skysb Did you test it thoroughly? I guess it will fail as soon as we need to get file content on an object which is not yet retrieved. For instance when using Lookup.

alfonsarmi · 2021-08-25T09:04:26Z

@skysb Did you test it thoroughly? I guess it will fail as soon as we need to get file content on an object which is not yet retrieved. For instance when using Lookup.

Exactly the same problem in Linux, cannot "just read to inspect the directory tree" the repository folder.

I'm also using a --filter=blob:limit command in git to get a partial clone

skysb · 2021-08-25T09:12:53Z

@skysb Did you test it thoroughly? I guess it will fail as soon as we need to get file content on an object which is not yet retrieved. For instance when using Lookup.

@jairbubbles

Well, I can't test it until it lets me instantiate the repository right?
I get the exact same error as @alfonsarmi .

jairbubbles · 2021-08-25T09:21:24Z

Sorry @skysb I guess you only work on Linux on your side.

jairbubbles · 2021-08-25T09:44:42Z

And my question was more: "Did you test it thoroughly on Windows?"

alfonsarmi · 2021-08-25T13:41:50Z

And my question was more: "Did you test it thoroughly on Windows?"

Hi guys,

Just in case it helps... @jairbubbles @skysb @ethomson @bording

I tested this thoroughly in Windows with a partially cloned repository.

So I could instanciate the Repository folder (new Repository(folder)), and also I could inspect the repository, lookup commits and navigate through the file system (tree) of a partially cloned repo (--filter=blob:limit=1m).

The only difference is that those filtered files (more than 1 megabyte) have no blob content, but everything else works.

It is not the case on a Linux container, as I cannot even instanciate the Repository (new Repository(folder) raises the "unsupported extension name extensions.partialclone" error).

I think it something with the NativeLibraries for Linux... but I'm not sure.

jairbubbles · 2021-08-25T16:58:04Z

@alfonsarmi @skysb What version of git you're using on Linux to do the clone? It might explain the different behaviors.

When looking at https://git-scm.com/docs/repository-version#_partialclone it seems that "extensions.partialclone" should be set in the .git config of the repo.

skysb · 2021-08-27T07:40:34Z

Hi @jairbubbles

The version of git we use in docker is always the latest version available. In fact, if I understand the documentation correctly, what it means to say is that when cloning the repository, in the config file created, the variables should already be set in that way if its a partial clone. It's not a manual config I need to modify to be able to clone the repository in a certain way.

The following is an example of the config file in windows when I carried out the partial clone using the git filter commands:

And this is an example of the config file downloaded in the linux machine:

both configs seem to be set correctly and the repositories seem to be downloaded correctly, the only issue is the instantiation of the repository in windows and linux, where in linux it fails, and in windows it doesn't which is why I suspected it to be related to native libraries maybe? as they seem to have a build for each OS?

Hope this helps in some way!
Thanks

jairbubbles · 2021-08-27T08:07:50Z

Thx @skysb,

That's very weird because in both cases "extensions.partialclone" is set so you should get the error on both platforms when opening the repo with libgit2sharp.

On my side I didn't get this in the .git config for Windows & WSL which confuses me even more:

[core]
	repositoryformatversion = 1
	filemode = false
	bare = false
	logallrefupdates = true
	ignorecase = true
[remote "origin"]
	url = https://github.com/libgit2/libgit2.git
	fetch = +refs/heads/*:refs/remotes/origin/*
	promisor = true
	partialclonefilter = blob:none
[branch "main"]
	remote = origin
	merge = refs/heads/main
	rebase = true

EDIT: when I manually set "extensions.partialclone" I can't open the repository on Windows, I get the error which doesn't surprise me as the code in libgit2 is pretty straight forward.

jairbubbles · 2021-08-27T08:47:43Z

@ethomson How would you feel about logging only a warning when "extensions.partialclone" is set indicating that some features won't work properly rather than not opening the repo at all?

jairbubbles · 2021-08-27T09:08:06Z

Or maybe add a new error code?

ethomson · 2021-08-27T12:22:23Z

Hi @jairbubbles - the purpose of the extensions is truly to communicate non-optional functionality and prevent unsupported git tools from opening the repository because they may corrupt the repository. A warning would not be appropriate in this case.

We should provide a mechanism to libgit2 users to indicate that they understand the extension - for example, the extension indicating that objects cannot be removed is a concept that users would need to "opt in" to. This is something I was thinking about after the upcoming 1.2 release.

ethomson · 2021-08-27T13:02:33Z

(I'm happy to add an unsupported extension error code.)

jairbubbles · 2021-08-27T13:12:31Z

(I'm happy to add an unsupported extension error code.)

Cool but if it exists before the repo is initialized it won't fix what we try to achieve, ie inspecting partially cloned repository through libgit2. Some kind of read-only open would make sense?

skysb · 2021-09-02T09:14:24Z

(I'm happy to add an unsupported extension error code.)

Cool but if it exists before the repo is initialized it won't fix what we try to achieve, ie inspecting partially cloned repository through libgit2. Some kind of read-only open would make sense?

This would actually help us a lot. Being able to at least have a read-only open to inspect it and it would be a significant improvement over just raising an error, as now there many people that are opting for partially cloned repos where the repo size is huge.

ethomson · 2021-09-02T12:13:59Z

Sorry if my comment about writing corrupt repositories suggested that writing was the only concern. Reading is a similar problem. There's no read only view of a repo with sha256, for example. So it's not like we can ignore extensions if you are opening a repo read only. We would still need this mechanism for you to tell us what extensions you understand.

jairbubbles · 2021-09-02T18:02:31Z

Yes @ethomson but in our case we're speaking only about the "partialclone" extension. The behavior regarding other extensions is another concern.

ethomson · 2021-09-02T18:46:40Z

I'm sorry, this is just not a viable path forward. Adding read-only support for repositories is not a trivial undertaking, and while it might solve your needs, it's not a particularly elegant solution nor a general purpose one. Adding a mechanism where users tell us the extensions that they support has the advantage of being both easier and a necessary for the library anyway.

jairbubbles · 2021-09-02T19:52:11Z

Adding a mechanism where users tell us the extensions that they support has the advantage of being both easier and a necessary for the library anyway.

I really don't have any preference, I was just brainstorming around our use case. I don't exacly see what you have in mind and I don't see how as a client of the library I can "handle" an extension. In our case it would be just a way to tell "I know what I do, the repo is partially cloned that's fine with me, I'm just going to inspect it".

ethomson · 2021-09-02T20:20:39Z

In our case it would be just a way to tell "I know what I do, the repo is partially cloned that's fine with me, I'm just going to inspect it".

Ah! Sorry, my bad. Yes, that's precisely what I have in mind. libgit2 needs a system where the user can say "hey, I know how to handle this extension" and then when we see a repo that has that extension, we continue. Otherwise, we'll block as we do today. At app startup, you'll just say "hey I know what to do with these extensions" and then we'll open any repositories that have that extension and you can do what you wish.

Eventually, we'll have some set of extensions that we support ourselves: new sha format, for example, won't need to be opt-in (though it could be opt out). But you'll be able to opt in to extensions that we don't know anything about (like this one).

I'll see if I can tackle this over the weekend.

jairbubbles · 2021-09-03T08:59:17Z

@ethomson Well that would all make us very happy here. I can look at the C# part if you'd like.

jairbubbles · 2021-09-11T18:24:19Z

There are some good things coming here: #1908 😏

skysb · 2021-09-13T09:27:03Z

@jairbubbles That sounds awesome and promising! Once approved, would there be a nuget package created for it too?
Thank you so much for the quick responses to this!

jairbubbles · 2021-09-13T12:51:36Z

@skysb Yes as soon as it's merged in libgit2 I'll make all the PR so that it's available here.

Please note that I also started a PR to make the code more friendly when using blobless repos: #1909

skysb · 2021-09-13T13:22:25Z

@jairbubbles youre a ⭐ !! Would #1908 mean we can download partially cloned repos (like blobless files) and initialize the repo or would we need to wait for #1909 for the feature?

Cheers

jairbubbles · 2021-09-13T15:12:23Z

#1908 will be enough.

#1909 is a quality of life PR on top of it. Many methods throw when the blobs are not available, you can already try / catch but I feel like it's not very pretty as it generates many different exceptions depending on the called method (sometimes NullReferenceException).

It's just a draft, I'm also considering throwing a new BlobNotAvailable exception.

alfonsarmi · 2021-09-15T12:12:29Z

This is fantastic guys! Many thanks!

@jairbubbles looks like the merge's checks have failed. Any plan to get this fixed any time soon?

Thanks!

jairbubbles · 2021-09-15T13:16:38Z

@alfonsarmi Yes I need to consume the new native binaries but they are not yet on nuget.org. I'll make a PR as soon as version 1.3 is tagged, should be this week-end.

skysb · 2021-09-23T08:32:38Z

Hello! Any updates on the PR? should this discussion move to the other PR instead?? @jairbubbles

jairbubbles · 2021-09-23T18:08:27Z

Hello @skysb libgit2 was not tagged, I will update to a commit containing the fix. As long as we're sill in pre-release I think it's ok.

jairbubbles · 2021-09-23T18:26:10Z

See libgit2/libgit2sharp.nativebinaries#128

ethomson · 2021-09-23T18:30:03Z

Hello @skysb libgit2 was not tagged, I will update to a commit containing the fix. As long as we're sill in pre-release I think it's ok.

👍 LibGit2Sharp has a long history of using unreleased commits. 😁

I wanted to do a little more testing before I land v1.3.0 over in libgit2.

skysb · 2021-11-18T11:16:27Z

@ethomson @jairbubbles Any updates on this?

Update LibGit2 to version 1.1.1

85ad389

jairbubbles force-pushed the update-to-libgit-1-1-1 branch from a8a546e to 85ad389 Compare August 22, 2021 08:23

Revert commented test

9941d39

jairbubbles mentioned this pull request Aug 22, 2021

Binaries for newer versions of libgit2? libgit2/libgit2sharp.nativebinaries#110

Closed

bording reviewed Aug 24, 2021

View reviewed changes

LibGit2Sharp.Tests/CloneFixture.cs Show resolved Hide resolved

LibGit2Sharp/Core/NativeMethods.cs Show resolved Hide resolved

jairbubbles marked this pull request as ready for review August 24, 2021 14:57

bording approved these changes Aug 24, 2021

View reviewed changes

bording merged commit 6329bea into libgit2:master Aug 24, 2021

jairbubbles deleted the update-to-libgit-1-1-1 branch August 24, 2021 16:30

Update LibGit2 to version 1.1.1 #1905

Update LibGit2 to version 1.1.1 #1905

Conversation

jairbubbles commented Aug 22, 2021 • edited Loading

codecov-commenter commented Aug 22, 2021 • edited Loading

Codecov Report

skysb commented Aug 24, 2021

jairbubbles commented Aug 24, 2021 • edited Loading

bording commented Aug 24, 2021

jairbubbles commented Aug 24, 2021

bording commented Aug 24, 2021

jairbubbles commented Aug 24, 2021

bording commented Aug 24, 2021

alfonsarmi commented Aug 24, 2021

ethomson commented Aug 24, 2021

skysb commented Aug 25, 2021

skysb commented Aug 25, 2021

jairbubbles commented Aug 25, 2021

skysb commented Aug 25, 2021

jairbubbles commented Aug 25, 2021

alfonsarmi commented Aug 25, 2021 • edited Loading

skysb commented Aug 25, 2021

jairbubbles commented Aug 25, 2021

jairbubbles commented Aug 25, 2021

alfonsarmi commented Aug 25, 2021 • edited Loading

jairbubbles commented Aug 25, 2021

skysb commented Aug 27, 2021

jairbubbles commented Aug 27, 2021 • edited Loading

jairbubbles commented Aug 27, 2021

jairbubbles commented Aug 27, 2021

ethomson commented Aug 27, 2021

ethomson commented Aug 27, 2021

jairbubbles commented Aug 27, 2021

skysb commented Sep 2, 2021

ethomson commented Sep 2, 2021

jairbubbles commented Sep 2, 2021

ethomson commented Sep 2, 2021

jairbubbles commented Sep 2, 2021

ethomson commented Sep 2, 2021

jairbubbles commented Sep 3, 2021

jairbubbles commented Sep 11, 2021 • edited Loading

skysb commented Sep 13, 2021

jairbubbles commented Sep 13, 2021

skysb commented Sep 13, 2021

jairbubbles commented Sep 13, 2021

alfonsarmi commented Sep 15, 2021

jairbubbles commented Sep 15, 2021

skysb commented Sep 23, 2021 • edited Loading

jairbubbles commented Sep 23, 2021

jairbubbles commented Sep 23, 2021

ethomson commented Sep 23, 2021

skysb commented Nov 18, 2021

jairbubbles commented Aug 22, 2021 •

edited

Loading

codecov-commenter commented Aug 22, 2021 •

edited

Loading

jairbubbles commented Aug 24, 2021 •

edited

Loading

alfonsarmi commented Aug 25, 2021 •

edited

Loading

alfonsarmi commented Aug 25, 2021 •

edited

Loading

jairbubbles commented Aug 27, 2021 •

edited

Loading

jairbubbles commented Sep 11, 2021 •

edited

Loading

skysb commented Sep 23, 2021 •

edited

Loading