Gemstash server integration

[micheelengronne]

What would you like Renovate to be able to do?

Renovate should be able to fetch gems in a private Gemstash server. It should be able to handle credentials for this server.

AFAIK, the url /api/v1/gems seems implemented in Gemstash but only for pushing gems https://github.com/rubygems/gemstash/blob/9d0e45d80e70f7e2d3987e5e1cdef68cd1119a91/lib/gemstash/web.rb#L48

It seems that urls used to fetch gems are /api/v1/dependencies. That's what fetched when I use bundle lock -V.

For instance, here fetching ffi and nokogiri gems:

HTTP GET https://username:password@INTERNALDOMAIN/private/api/v1/dependencies
HTTP 200 OK https://username:password@INTERNALDOMAIN/private/api/v1/dependencies
Fetching gem metadata from https://INTERNALDOMAIN/private/
Query List: ["ffi", "nokogiri"]
Query Gemcutter Dependency Endpoint API: ffi,nokogiri
HTTP GET https://username:password@INTERNALDOMAIN/private/api/v1/dependencies?gems=ffi%2Cnokogiri
HTTP 200 OK https://username:password@INTERNALDOMAIN/private/api/v1/dependencies?gems=ffi%2Cnokogiri

[rarkins]

Can we locate any documentation for this API? We need to (a) support it and (b) have a way to know how/when to switch to it

[micheelengronne]

Here is the doc from the user POV for setting auth https://github.com/rubygems/gemstash/blob/master/docs/gemstash-private-gems.7.md

The files handling auth and fetch are:

https://github.com/rubygems/gemstash/blob/master/lib/gemstash/api_key_authorization.rb

https://github.com/rubygems/gemstash/blob/master/lib/gemstash/authorization.rb

https://github.com/rubygems/gemstash/blob/master/lib/gemstash/gem_fetcher.rb

But I don't know if they documented an API. I think we should empirically test with a Gemstash docker container.

[micheelengronne]

I opened an issue on their side rubygems/gemstash#272

[rarkins]

If it works fine with Bundler then I'd hope there's an answer out there already, we just haven't found it yet. Thanks for your help

[micheelengronne]

I investigated further. Gemstash gives the api/v1/dependencies.json endpoint.

It is used that way (example with the ffi gem):

api/v1/dependencies.json?gems=ffi

and gives this result:

[{"name":"ffi","number":"1.9.11","platform":"x86-mingw32","dependencies":[]},{"name":"ffi","number":"0.4.0","platform":"java","dependencies":[]},{"name":"ffi","number":"1.0.11","platform":"java","dependencies":[]},{"name":"ffi","number":"1.2.0.pre6","platform":"ruby","dependencies":[]},{"name":"ffi","number":"1.2.0.pre5","platform":"ruby","dependencies":[]},{"name":"ffi","number":"1.7.0.dev","platform":"ruby","dependencies":[]},{"name":"ffi","number":"1.9.14","platform":"x64-mingw32","dependencies":[]},{"name":"ffi","number":"1.9.8","platform":"java","dependencies":[]},{"name":"ffi","number":"1.4.0","platform":"x86-mingw32","dependencies":[]},{"name":"ffi","number":"1.9.12","platform":"x64-mingw32","dependencies":[]},{"name":"ffi","number":"1.8.1","platform":"java","dependencies":[]},{"name":"ffi","number":"1.2.0.dev","platform":"ruby","dependencies":[]}...

etc...

It looks similar to the result of api/v1/gems (that works with https://rubygems.org) but gives less informations (no created_at, rubygems_version, or ruby_version).

But, we only need number and platform and it gives that.

So there is smthg to do here :

renovate/lib/datasource/rubygems/get.ts

Line 56 in b45502c

const releases = versions.map(

And here:

renovate/lib/datasource/rubygems/get.ts

Line 10 in b45502c

const INFO_PATH = '/api/v1/gems';

BTW, this endpoint also exist in rubygems. That would make the manager simpler and more reliable by removing:

https://github.com/renovatebot/renovate/blob/master/lib/datasource/rubygems/releases.ts#L10

and the file:

https://github.com/renovatebot/renovate/blob/b45502cd28dee7b583dd2f00da76a3807304d92f/lib/datasource/rubygems/get-rubygems-org.ts

[rarkins]

Thanks for the analysis. FYI rubygems.org will block any IP that uses the regular API more than one query per second, so it is definitely not possible to replace that functionality.

I think we want to default to the API that gives us the. Eat metadata and then fall back to the less informative one.

[rarkins]

Also note that gemfury uses the same bulk approach as rubygems

[micheelengronne]

Ok, as bundler uses the api/v1/dependencies the implementation of it will be the most compatible as any gem repository will at least implement this endpoint. Otherwise, they would not be compatible with bundler (smthg impossible in the ruby world).

[rarkins]

Bundler itself does not use the Rubygems API for most queries due to the load. It uses the append-only flat versions file that Renovate now uses too. Hopefully other servers support api v1 though

[micheelengronne]

I think it is better to improve gemstash to make it more compatible with the official rubygems API. So, I commited there instead. rubygems/gemstash#277

[jume-dev]

Did anybody fix this already?
Because we found that within our setup, it works with private gems from a gemstash server.

[micheelengronne]

Sorry, I was busy on other matters and this sunk down in my backlog. I don't when I would be able to fix this properly.

[jume-dev]

Yeah, no problem.
Indeed, maybe I worded it wrong, but we found it works. Something fixed it somehow. Maybe there isn't even something to do anymore if it works for others too.

[jboyens]

So prior to #16312 we were at least able to utilize renovate with our private Gemstash server. After that PR, all our renovate runs that have a reference to the private Gemstash server abort and fail to successfully complete a run. We've locked our runner back to the version prior to that PR, but that's far from a long-term plan.

I'm attempting to dig through the renovate code to see how/where I can patch out or correct the behavior to return at least partial functionality while using a private Gemstash. Any pointers would be most welcome...

[jume-dev]

We recently had the same problem and made a pull request on the gemstash repository:
rubygems/gemstash#326
If you update your gemstash server you can use the newest renovate version again

[jboyens]

Ahh wonderful. I ended up disabling the private gemserver via hostRules and setting packageRules to ignore anything stored there... It worked, but it was real ugly.

I'll give the newer version of Gemstash a whirl. Appreciate your comment and your PR.