User not found

[yangavin]

A regular request to get user's contributions stopped working, getting a response of:

{
"error": "User \"grubersjoe\" not found."
}

[grubersjoe]

Hey. Thanks for reporting this! GitHub seems to have changed how the contribution graph is rendered. Now client-side JavaScript appears to be required. You can test this by disabling JavaScript in your browser (the chart never shows up, but you will only see a spinner forever). Consequently, the contribution data can no longer be scraped by simply parsing the HTML source.

So this will make this a lot harder, unfortunately, and I have to think about this. There probably is no way around something like Puppeteer which sounds quite heavy.

[grubersjoe]

I think I've found a way around Puppeteer so that DOM parsing can still be used. There's an endpoint that prerenders some of the calendar, but it should be all we need:

https://github.com/users/grubersjoe/contributions

[adriangalilea]

Hey @grubersjoe, why not just use graphql?

source

EDIT: managed to implement a custom version of your component github-activity-calendar (dunno how storyblooks work so I just copypasted all src 😆 ) and then hooked it up to the graphql endpoint, then flattened the data, works flawless.

[grubersjoe]

Oh wow, there's an API now? I'm pretty sure there wasn't when I started the project.

In this case of course this is way to go. I'll look into it later. Thanks for the hint!

[adriangalilea]

Oh wow, there's an API now? I'm pretty sure there wasn't when I started the project.

I could not find the documentation anywhere, just that post.

You can take a look at my crude implementation here

Glad that I helped ;)

[yangavin]

Hey @grubersjoe,

It's really unfortunate that GitHub changed their rendering strategy. I was actually using GitHub's GraphQL API and I had two major issues with it that ultimately lead me to using your API:

1. Rate limits
   GitHub's API has a rate limit that I found to be too low for fetching large amounts of data, such as contributions:
   https://docs.github.com/en/graphql/overview/rate-limits-and-node-limits-for-the-graphql-api
   I could be querying it wrong, but I ran into the limit for personal use, I can imagine this may be an issue for a public-facing API.

2. Only showing previous year
   Currently there does not seem to be a way to fetch all total contributions, you can only fetch total contributions for the past year (similar to what's shown on your GitHub profile on initial load. There isn't even an option to fetch total contributions within a date range. More about this here:
   Repositories contributed to is only the last year anuraghazra/github-readme-stats#2269

I resorted to using your API because your web-scraping strategy bypasses these two issues. These are just some things to consider as your working on a fix if you're using GraphQL. While something like Puppeteer may be heavy, it could perhaps bring in a tremendous amount of value that is not currently being supported by GraphQL.

Hope this helps!

[d3or]

Hi. PR here that should fix this: #8

[grubersjoe]

Deployed latest version, everything should work again. Thanks everyone ❤️!

[grubersjoe]

@yangavin I think you made two very good points 👍. The rate limit would probably be reached pretty fast. Scraping is fragile but it solves these two issues.