12 job splitters

[GriffinBabe]

Closes #11 and #12

• Added job splitter using h3 library. Adding the h3 library and using it directly is more lightweight in terms of package and drastically more computationally efficient than having a pre-generated dataset containing the h3 grid. Especially when it comes to low resolutions.
• Added job manager, extending MultiBackendJobManager. The job manager also implements post-job actions that are computed on other threads to keep the status update smooth.

TODO before merging:

• Polish and add a demo notebook to the branch

[GriffinBabe]

@soxofaan I'm confused, there is a lint check that fails on black but the black check is disabled in the workflow file? I also tried fixing the files with black but doesn't help

[GriffinBabe]

@soxofaan Do you think I should be using multiprocessing instead of multithreading? In case GFMAP is running on a VM with multiple CPUs that would really accelerate possibly heavy post-job actions

[soxofaan]

@soxofaan I'm confused, there is a lint check that fails on black but the black check is disabled in the workflow file?

That's indeed weird. What might be the case is Gihub runs the workflow from main (instead of your branch) because you are making a pull request targeting main.

[soxofaan]

Do you think I should be using multiprocessing instead of multithreading? In case GFMAP is running on a VM with multiple CPUs that would really accelerate possibly heavy post-job actions

If I remember correctly, threading and multiprocessing have a common API, so I guess it's most future proof to stay within this common API, so that you can switch when one approach doesn't cut it

[GriffinBabe]

@kvantricht also corresponds to #43

[GriffinBabe]

Problem remaining with the "to_scl_dilation_mask", except that everything should be ready for merge

[kvantricht]

Problem remaining with the "to_scl_dilation_mask", except that everything should be ready for merge

is that something that needs to be fixed before merging or do we create another issue for that? What exactly was the problem?

[kvantricht]

Huge work done, nice! I added some comments/questions. Don't entirely get the STAC yet and how much it's tuned now to WorldCereal. Also band names still need to be updated there. In the end is this just the example belonging to Sentinel-2 extraction for WorldCereal?

What I didn't check but what's important as you know is proper testing. I trust you've been writing the proper tests along the way.

[kvantricht]

I will approve for now. After Monday discussion we can merge this and add remaining issues if needed.

[GriffinBabe]

Created issue for to_scl_dilation_mask and for the STAC AssetDefinition