Parallel Processing in Crop_plot_data function

[Nageshbansal]

@MarconiS, We have to implement parallel processing for the clip_plot function over the aop data rows.

[Nageshbansal]

@MarconiS, i will look into this meanwhile.

[ConnectBhawna]

Hi @Nageshbansal can i work on this issue?

[Nageshbansal]

hey @ConnectBhawna , sure. Let me know if you need any help

[Nageshbansal]

@MarconiS , Parallel processing involves two main steps:

1. Downloading data from multiple websites and years concurrently (instead of doing it one after the other, which would be slower).
2. Taking the downloaded data and carrying out additional processing in functions like retrieve_coords_itc concurrently as well.
   Is that correct?

[MarconiS]

yep, I think getting the data to be fetched in batches of 1 site - 1 year should fix the issue of being timeouted. If we have problems with the number of times per second we can query the API we can jitter the calls, I guess? like allow only 2 (or 4 if API key in provided) instances per second?

[Nageshbansal]

downloading the data in batches sounds good to me. Just to clarify , for e.g 1 batch of ABBY-2019 and another batch of ABBY-2020.

[MarconiS]

yeah, if that works fine to you, that sounds good to me! I believe we were discussing that it may send too many requests; if that's so, I am happy also with the ABBY-[2019:2020]. Whatever works best for you