Repository.Clone doesn't support SSH

[JustinBeckwith]

When performing a Repository.Clone against an SSH uri, it fails with the following message:

An error was raised by libgit2. Category = Net (Error).
This transport isn't implemented. Sorry

I'm guessing this just isn't implemented in libgit2 yet, though it looks to be on its way:
libgit2/libgit2#1103

[markhildreth]

Now that libgit2 has support for SSH, I was looking at what was needed to get this working. Changing the C# code looks straightforward, although I'm not sure the preferred way to change the API. The simplest would be to just throw a bunch more properties onto the Credentials object. I would prefer something like multiple Credentials objects, each with their own implementation of the aquire_credentials callback method, like...

public interface ICredentials
{
    int Authenticate(out IntPtr cred, IntPtr url, IntPtr username_from_url, uint types, IntPtr payload);
}

public sealed class UserPassCredentials : ICredentials
{
    /// <summary>
    /// Username for username/password authentication (as in HTTP basic auth).
    /// </summary>
    public string Username { get; set; }

    /// <summary>
    /// Password for username/password authentication (as in HTTP basic auth).
    /// </summary>
    public string Password { get; set; }

    public int Authenticate(out IntPtr cred, IntPtr url, IntPtr username_from_url, uint types, IntPtr payload)
    {
        return NativeMethods.git_cred_userpass_plaintext_new(out cred, this.Username, this.Password);
    }
}

Then we use this authenticate method in RemoteCallbacks.

Either way, I could do this. The issue really is that in order for SSH to work, libgit2 will need to be built with libssh2, (which itself has even more dependencies). I was able to get some of these libraries building on Windows, but I'm sort of out of my league on this one; what is the right way to have this project build libgit2 with SSH support? Does the powershell script need updating to include all of the dependencies? Or is there a better way?

[dahlbyk]

👍 for multiple credential types, though that Authenticate() method is probably not one we would want to expose publicly. Perhaps a public abstract class Credentials base that has protected internal abstract int Authenticate(...)?

[nulltoken]

@dahlbyk 👍

@markhildreth I've recently discussed a bit about this topic in GitTools/GitVersion#38 (comment).

what is the right way to have this project build libgit2 with SSH support? Does the powershell script need updating to include all of the dependencies? Or is there a better way?

1. Maybe the first step would be to automate the building of libssh2 (and its dependencies) and produce x86 and x64 Windows binaries.
2. Then we could update the .ps1 and .sh build scripts to make them build libgit2 with ssh support

I was able to get some of these libraries building on Windows,

These are awesome news! Do you think you could try and produce a script to automate this process?

Warning: I'm not sure libgit2/ssh has ever been tested on Windows. There may be dragons out there 😉

/cc @andreasohlund @SimonCropp @ethomson @carlosmn

[ben]

Regarding the API, I just helped out with the design for Rugged's implementation of this. Here are the use cases we came up with for Rugged; Libgit2Sharp might be different, but maybe not.

1. You're writing a script to automate some git things, and you know the url and credentials you'll be needing.
2. You're writing a client where the remote URLs will be coming from someone else, and you'll be collecting and caching several sets of credentials.

Case 1 wants a simple way to provide a single Credentials instance. Case 2 wants some way to choose the credentials based on the URL. The lambda syntax in C# is a bit lighter than in Ruby, though, so a Func<Uri, Credentials> might be all you need.

[markhildreth]

@dahlbyk: 👍 to the abstract method.

@nulltoken: I think you're right with what you said in the other ticket regarding the Crypt API: nice idea, but does limit the platform support. Also, it isn't available yet. I'll see what I can do on a script for automating the libssh2 build including dependencies. We'll see how things go.

@ben: I think I'm missing the point of the callback. What is lost by creating the Credentials object before calling Repository.Clone()? Is it so that we let libgit2 parse pieces of the URL for us (like the username)? Or perhaps it is to allow us to open a dialog box to the user with a "password" field for them to fill in?

[ben]

Yeah, that's basically it. Here's what libgit2 does:

1. Try to make the connection.
2. If the response is 401, use a callback to get appropriate credentials.
3. GOTO 1.

Say I'm writing a GUI git client, and the user wants to clone a private repo from GitHub. Libgit2 will try the connection and get a 401, which returns control to my code. I can check my credential cache, and try the creds I've got stored there, and if that doesn't work, I can fall back to asking the user for credentials, and telling them it failed until they get it right or hit cancel.

Passing a Credentials object to Repository.Clone() can work, but you have to understand that there are several kinds of credential, and you may not know ahead of time which one is required. Libgit2 supports four different kinds of authentication right now, and there may be more in the future. As of right now, you can usually tell which kind you'll need by looking at the URL, but someday it might come down to what the server asks for, so you'd have to wait until it asks.

Either way, this:

Repository.Clone( /* … */, credentials: (url, types) => new Credentials("user", "pass"));

Isn't much harder to write than this:

Repository.Clone( /* … */, credentials: new Credentials("user", "pass"));

…and gives the caller a lot more flexibility.

[markhildreth]

Okay, I went WAY too far down the rabbit hole on this one. Here is my result of trying to get all these dependencies compiled (note: I'm trying to use Visual Studio for the compile, I didn't even look at MinGW)...

To compile libgit2 with SSH, we need libssh2. libssh2 requires openssl or gcrypt.

• openssl uses a custom perl build system to call the compiler/linker directly (no VS solutions), but still can be easily scripted:

perl Configure VC-WIN32 no-asm --prefix="C:\OpenSSL-Win32"
ms\do_ms.bat
nmake -f ms\ntdll.mak
nmake -f ms\ntdll.mak install

• libssh2 has an old VS6 project (.dsp), which when I converted to VS I was able to compile fine. However, it would be difficult to get this to compile from a script directly from sources without including our own newer project file or script the entire build.
• I didn't really try to use libgcrypt, but it seems the only way it provides to build it on Windows would be to use MinGW.

The main problem that I ran into was that libgit2 would not link against the libssh2 library unless I changed libssh2's calling convention from __cdecl to __stdcall. However, if I did that then I couldn't link against OpenSSL. Attempting to compile OpenSSL with __stdcall lead to...

C:\libssh\openssl-1.0.1e\tmp32\e_os.h(314) : error C2373: '__iob_func' : redefin
ition; different type modifiers
        C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\INCLUDE\stdio.h(1
29) : see declaration of '__iob_func'

Like I said before, this is not an area of expertise for me, so I might be missing something that someone who works with C more than once a decade would understand.

Another thing I noticed was that in addition to compiling SSH support, src/transport.c in libgit2 will need to be patched, as right now even if you have the GIT_SSH compiler flag enabled it will still not allow you to use SSH on Windows (see here).

So, I see three main hurdles here:

1. Even with all this correct, libgit2 would need to be patched to allow for SSH from Windows. I doubt this is a big deal, but something that would have to be done.
2. I cannot seem to compile OpenSSL with the __stdcall calling convention. This could be overcome by some other solution that doesn't require me changing the calling convention on OpenSSL and libssh2 to __stdcall, perhaps I'm just being C-illiterate here.
3. There isn't an easy way to build libssh2 with a modern Visual Studio install without first converting the .dsp project. There are a few potential solutions to this, but none that I know of that would be as easy as a script that says "point me to the libssh2 source directory and I'll do all the work".

Unless I can get pointed in the right direction, I'm probably going to stop work on this. The Windows Cryptography Patch for libssh2 makes all of this moot at the cost of not working on XP/Windows Server 2003 and lower, but it would be better than nothing and probably easier to build.

[Therzok]

The patch for the NG Crypto patch for libssh2 is in.

http://git.libssh2.org/?p=libssh2.git;a=commit;h=4440e05d488e115bb4c398532fd24707f5dc1915;js=1

[nulltoken]

As discussed in #739, maybe could we also resolve this through a NuGet dependency chain. Make libssh2 a NuGet package, make libgit2 depend on it and eventually make LibGit2Sharp depend on libgit2.

This would solve the Windows side.
Regarding the Linux/OS X side, well.. we'd have to build everything by hand...

[carlosmn]

This may be confusing for Windows folk, but on the unix side, we have these things called package managers. They're a bit like NuGet but for the software on our machines :p

On Linux/OSX and apt-get/yum/brew install libssh2(-dev) is all you need. The script might want to complain/advise if it's missing, but other than that, it doesn't need to bother with this.

[nulltoken]

You *nix guys are mean! You stole us the whole NuGet idea years before we came up with the concept. You can't be trusted 👅

On Linux/OSX and apt-get/yum/brew install libssh2(-dev) is all you need.

Is there a way to select a specific commit sha?

[carlosmn]

No, but you don't need to; they don't break the API like we do. Whatever you have in the 1.x timeframe, we can deal with; or at least >= 1.4, since that's been their current release the whole time we've used libssh2, but that's been out for 18 months or so now.

Or did I misunderstand the question?

[nulltoken]

Or did I misunderstand the question?

http://git.libssh2.org/?p=libssh2.git;a=summary;js=1

1.4.3 has been released 18 months ago.
The Windows crypto seems to be in their development (unstable?) branch.

How can we target this unreleased branch while still knowing which "commit" we're working against (for troubleshooting purpose mainly)?

[carlosmn]

On unix, we're already fine with ssh support and don't need to do anything. On Windows, we can target our own packages until there's something more official.

[nulltoken]

On unix, we're already fine with ssh support and don't need to do anything

Pffff. You really need to start using a decent OS 🚎

[nulltoken]

A work in progress in libgit2 (see libgit2/libgit2#2428) should help make LibGit2Sharp support Ssh more easily.

[ghuntley]

👍

[yannisgu]

The WIP mentioned by @nulltoken is now implemented: libgit2/libgit2#2600

Should be a bit eeasier now to implement it..

[nulltoken]

Related #771 and #852

[dannyzhan]

Guys, is ssh still not supported in libgit2 and libgit2sharp, right?
I really need ssh support, it's more secure than http.

[nulltoken]

@dannyzhan See my answer to your very similar question in libgit2/libgit2sharp.nativebinaries#7

[CumpsD]

Which issue exactly is the main one now to follow progress of SSH implementation? #1072?

Trying to figure out the current state to fetch repo's using an SSH key (used as deployment key in github)

[carlosmn]

libgit2 and libgit2 do support SSH. The libgit2 we ship does not, as we do not want to ship 3rd-party crypto software in our packages.

[mback2k]

@carlosmn If libgit2 already requires libssh2 to use ssh, then you do not need to ship an additional 3rd-party crypto software in your package, since libssh2 now has native Windows support using WinCNG.

[ethomson]

I think libssh2 is the third-party crypto that he was referring to.

[carlosmn]

Yes. While the cryptographic primitives are implemented by OpenSSL or
Windows, libssh2 is still crypto code and packaging that means having to
keep up with its releases and security updates.

And on UNIX we want to use the system libraries, but that would make the
nuget package not be self-contained anymore.

dload/GetProcAddress is an alternative to linking directly, but nobody has
done the work to make that work.
On Apr 20, 2016 9:59 PM, "Edward Thomson" [email protected] wrote:

I think libssh2 is the third-party crypto that he was referring to.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#255 (comment)

[rubberduck203]

I'm a little confused about the state of this issue. Is the ssh protocol supported for windows?

[ethomson]

@ckuhn203 Not out of the box, you have to build it and link it yourself.

[rubberduck203]

Got it. Thanks @ethomson.

[smuda]

@ethomson is there some documentation how to do that?