Sync with Github error (Request failed with status code: 504)

[thawn]

This is a re-opening of #957 and #1107. Please see the discussions there.

Issue is (additionally) related to the number of organizations and repositories a user is a member of.

The only workaround currently is to disconnect repositories and reconnect them, which for users with many repositories is very sub-optimal.

Related to Zenodo Support email [Ticket#332699]

I suggest reopening #957 until the issue is actually fixed (and not just by the suboptimal workaround).

[cboettig]

I consistently get this error too.

[richelbilderbeek]

Same here, on 2025-02-18: I clicked the sync button multiple times.

Directly after clicking on the sync button for the ~10th time, the server crashed ... :-/ . Unsure if it was due to that...

When the server came back up, I tried again, no success yet.

[joshmoore]

This is a re-opening of #957 and #1107

Thanks for taking on the torch, @thawn! 🔥 I will update my support ticket with this version as well.

[alin256]

I managed to go around the issue by log-out and then log-in sequence.

But it would be great to see it fixed.

[richelbilderbeek]

I managed to go around the issue by log-out and then log-in sequence.

I tried so too. Does not work for me ... :-/

[alin256]

I tried so too. Does not work for me ...

I tried creating a new public repository to test the functionality. I cannot reproduce my previous solution but at some point (within few minutes) it showed up in the Zenodo list despite 504 errors. If you have a list but a new repository is missing it can also might be to e.g. organizational policies etc. Happened to me...

https://github.com/alin256/zenodo-test

[richelbilderbeek]

I get a 400 error after trying too often to sync (and getting the 504 error many times).

I assume the problem is that I am a member of multiple organisations, some having hundreds of repositories...? I can imagine that it may take too long to sync...

[joshmoore]

Here a summary of those involved in this protracted discussion to date:

User	Public Repos	Public Orgs	First Comment
	Issue 957
AliciaMstt	32	1	Apr 13, 2024
yarikoptic	1.1k	25*	Jun 16, 2024 (issue 895)
joshmoore	377	32*	Jul 16, 2024
wsnoble	2	0	Jul 26, 2024
zimolzak	135	0	Jul 30, 2024
Blake Naccarato	62	1	Sep 22, 2024
Peter Dudfield	11	0	Oct 25, 2024
thawn	34	0	Nov 18, 2024
e-kotov	74	5	Dec 18, 2024
alin256	144	0	Jan 29
	Issue 1107
krokicki	47	5*	Jan 7
bethac07	32	5	Jan 15
caufieldjh	57	3*	Jan 23
lldelisle	75	1	Jan 27
Vladimir Alexiev	113	8*	Jan 29
	Issue 1118 (here)
cboettig	179	10*	Feb 16
Richèl Bilderbeek	1.4K	5*	Feb 18
Philip Chase	106	1	Apr 15

*: at least one org with more than 100 repositories

cc top-contributors: @slint @alejandromumo @zzacharo @yashlamba

[joshmoore]

I'm now wondering if there could be some correlation with e.g. a server-side deployment or cache invalidation... 🤔

[thawn]

@joshmoore thanks for taking the systematic approach! I think you may be on to something here.
Added info: I am a member of two private organizations, one of which has 50 repos, the other 21.

[joshmoore]

An update from Zenodo Support email [Ticket#332699]:

Thank you for your investigative work in the attached issue, that is really helpful for us to see the scale of the issue at hand.

We're currently looking into how to circumvent this sort of limitation and reduce both the number of repositories that need to be synchronised and displayed on the GitHub settings page. Some possibilities would be to group repositories by their GitHub organisation, depending on if they're forks/sources, or how recently they were active. If you have any opinions on this matter we would be grateful to hear them

[Hoeze]

I'm also affected... :/
Let me know if I can help debugging this issue somehow

[joshmoore]

Carlin from Zenodo reached out saying that the sync method is the one that is being investigated. I extracted this gist and found that it takes a few seconds to choose some 1250 of 4400 repositories for further processing (Those starting with "ok"). If you want to run that, it might be a more useful metric than what I did above.