Instance re-distribution after incomplete submissions

[jl3676]

Hi, I recently used potato to collect annotation data on Prolific and I set the labels_per_instance variable to a fixed number. I ran into the issue that incomplete submissions (some workers didn't finish all the instances assigned to them) remained in the instance distribution counts, so after data collection stopped (new workers were told that they came too late and there were no tasks remaining, so they must return their submissions), I didn't get as much data as I wanted per instance sometimes. I wonder if there is a known solution to this problem? Additionally, for the participants who "came too late," being told that could be frustrating because it could feel like a waste of their time (according to some of the messages they sent me).

[Jiaxin-Pei]

Hi @jl3676, have you tried to connect to your prolific api? Please follow the guide here to set up the automatic task management with prolific. We also have an example project for this!

[jl3676]

@Jiaxin-Pei Thanks for the suggestion! I haven't tried to connect to the prolific api, but I will next time. What if my task wasn't run on prolific, but some other distribution platform? Is there a more general way of handling task redistribution?