zef cannot not install distributions that list SSH source-urls (e.g. "[email protected]:…")

[codesections]

When a distribution lists an SSH address as its source-url, I cannot use zef to install that distribution. This may be true for all users, or it may only effect users who need to unlock their SSH key before it can be used (the issue might be that zef does not prompt to unlock the SSH key).

For example, consider this command/output:

 $ zef install Data::MessagePack --debug
===> Searching for: Data::MessagePack
===> Found: Data::MessagePack:ver<0.1.2> [via Zef::Repository::Ecosystems<p6c>]
===> Fetching: Data::MessagePack
[Data::MessagePack] Fetching [email protected]:pierre-vigier/Perl6-Data-MessagePack.git with plugin: Zef::Service::Shell::git+{<anon|1>}
===> Fetching [FAIL]: Data::MessagePack:ver<0.1.2> from [email protected]:pierre-vigier/Perl6-Data-MessagePack.git
Aborting due to fetch failure: Data::MessagePack:ver<0.1.2> (use --force-fetch to override)

I can think of a few different solutions:

• do nothing, and state that all source-urls should be https URLs (in which case I should close this issue and open a documentation issue, since that isn't currently documented).
• Add a better error message that suggests the user to specify an https URL when fetching from an SSH URL.
• Automatically fall back to fetching from an https URL when fetching from an SSH URL fails.
• Fix whatever issue stops zef from fetching from SSH URLs (maybe just prompting the user to unlock their SSH keys when needed).

I also mentioned this issue in pierre-vigier/Perl6-Data-MessagePack#15.

[ugexe]

It’s not up to zef to enforce ecosystem policy like what source-url is allowed. For all I know users have configured their own fetcher that actually uses that uri format. There is no blessed format, and any format can be made to work then rough configuration and code. And because there are many possible fetchers how could it possibly report “set your ssh key” when that plug-in cannot know a later plug-in might actually understand that uri? Admittedly it’s possible but it’s a lot of work to fix a problem that stems from elsewhere.

It’s not up to zef to tell the user about errors the author made (wrong source-url) — that is for authoring tools or ecosystem prechecks. Zef will never tell you e.g. “hey your META6.json is wrong“ because it isn’t an authoring tool.

[Xliff]

@ugexe: Would you be adverse to a PR that will handle conversion of these URLs to something that zef can handle?

[ugexe]

Why not just fix the problem at the source? Automatically rewriting git:// is one thing (and works AFAIK), rewriting git@ is another.

[Xliff]

Fair enough. I do believe an issue has already been raised on Data::MessagePack. One question: Can zef determine if a URL has a valid fetcher? If not, would THAT be something that could be used to improve the type of error returned?

[ugexe]

$ zef install fart://fart.com
Enabled fetching backends [git path curl wget] don't understand fart://fart.com
You may need to configure one of the following backends, or install its underlying software - [pswebrequest]

[ugexe]

fwiw for zef to reject git@ requires modifying the fetch-matcher method of the git plugin. However -- a user being told the git fetching backend is available but is failing to install [email protected]:xxx/zzz.git would be confusing. I think the git@ url should work (assuming the git env is setup properly) though, and that something here just needs to be tweaked to not assume a scheme.

[codesections]

I don't feel qualified to have much of an opinion on this – I'm very new to the Raku ecosystem and actually ran into this issue while still trying to get my dev environment fully set up for Raku work.

That said, I think it comes down to something you said earlier:

Why not just fix the problem at the source?

But what does that mean in this context? If the problem is that some distribution authors aren't correctly following "ecosystem policy" and are using an incorrect source-url format, then the "fixing the problem at the source" means fixing the errors in those authors' META6.json files.

However – at least as far as I can tell – there isn't any ecosystem policy against using SSH protocol addresses as a distribution's source-url. At least, there doesn't seem to be a source-url specified in the (old) spec or in the current Roast test. Similarly, the relevant docs don't seem to mention a prescribed format. I could definitely be missing something – as I mentioned, I'm very new here – but it looks to me as though there isn't any rule that source-urls should use HTTP instead of SSH.

If that's the case, then the problem isn't that distribution authors are using an incorrect format for their URLs. Instead, (arguably) the problem is that zef can't currently fetch from those valid URLs. Under that argument, "fixing the problem at the source" would mean adding support to zef.

After saying all that, however, I'm coming around to your point of view: that there really is an ecosystem policy against using SSH URLs. I say this because, based on a scan of the 2,864 distributions zif knows about, only 8 use a URL starting with git@. If there really is an ecosystem policy, then I guess the correct answer is to document that policy and to issue PRs to those 8 projects.

But, that's just my ¢2. I'm happy to defer to others on this. If you say that there is an ecosystem policy against SSH URLs, I'd be happy to open a documentation issue to better document that policy. Do you know where that should be documented?

[ugexe]

At least, there doesn't seem to be a source-url specified in the (old) spec or in the current Roast test

Correct, source-url has never been spec. And it shouldn't be -- rakudo does not interact in any way with urls. There are some suggested fields in s22 like source but so is telephone number.

If that's the case, then the problem isn't that distribution authors are using an incorrect format for their URLs. Instead, (arguably) the problem is that zef can't currently fetch from those valid URLs.

Should zef also be able to install things from magnet links, ftp, etc in core? The thing is zef cannot ever hope to understand every URI format, including stuff like [email protected]:ugexe/zef.git which isn't even a valid uri reference. Someone is welcome to write a plugin to do this, or modify this to maybe work with that uri-like, but the fact it does not work currently, to me, is not seen as a bug.

If you say that there is an ecosystem policy against SSH URLs, I'd be happy to open a documentation issue to better document that policy.

There aren't really any ecosystem policies... but of course that doesn't mean there couldn't/shouldn't be.

[ugexe]

Surprising to me the ssh-git uri format was fairly simple to make work, including the tag style ala [email protected]:ugexe/[email protected]. It should be merged sometime in the next couple weeks.

However, I still think its wise to encourage users to just use https urls.