Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rate limiting by github fails silently #480

Open
kasbah opened this issue Nov 3, 2022 · 2 comments
Open

Rate limiting by github fails silently #480

kasbah opened this issue Nov 3, 2022 · 2 comments

Comments

@kasbah
Copy link
Member

kasbah commented Nov 3, 2022

Only noticed this in gitea logs when doing migration e2e tests:

Run task failed: GET https://api.github.com/repos/kitspace-test-repos/DIY_particle_detector: 403 API rate limit of 60 still exceeded until 2022-11-03 13:23:52 +0000 GMT, not making remote request. [rate reset in 8m39s]

Not sure why Gitea is using the API in the first place (to fetch the description?) but it's making the whole migration fail. We should avoid the failure but also report such migration failures in a better way.

@kasbah kasbah added this to the alpha-2 milestone Nov 3, 2022
@kasbah
Copy link
Member Author

kasbah commented Nov 6, 2022

@AbdulrhmnGhanem
Copy link
Member

Moving this out of alpha-2; we are getting all repos' descriptions in a single request via the graphql API for now

/**
* Use the github graphql API to get the description of all the repos.
* We can't use the REST API because it doesn't allow us to get the description of multiple repos in one request.
* And if we requested the description of each repo separately, we would hit the rate limit.
* @param reposUrls
*/
async function getAllGithubReposDescriptions(
reposUrls: string[],
): Promise<Map<string, string>> {
const githubRepos = reposUrls.filter(
url => getGitServiceFromUrl(url) === 'github',
)
const reposOwnerAndName = githubRepos.map(
repoName => repoName.split('/', 5).slice(-2) as [string, string],
)
const query = `
query {
${reposOwnerAndName.map(
([owner, name], index) => `
r${index}: repository(owner: "${owner}", name: "${name}") {
fullName: nameWithOwner
description
}`,
)}
}
`
const { data: reposInfo }: GitHubRepoInfoGQLResponse = await fetch(
'https://api.github.com/graphql',
{
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `bearer ${flags.githubToken}`,
},
body: JSON.stringify({ query }),
},
).then(res => res.json())
const repoDescriptionsMap = new Map<string, string>()
for (const entry of Object.values(reposInfo)) {
repoDescriptionsMap.set(entry.fullName, entry.description)
}
return repoDescriptionsMap
}

@AbdulrhmnGhanem AbdulrhmnGhanem removed this from the alpha-2 milestone Jul 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants