Per executor output v02

[EmileSonneveld]

Change for spatial and spatialTemporal cubes. A sync request that loads a single file remains unchanged.
Need to assure sample by feature is tested correctly
Double check if per-date tiffs are written from executor. See if when they have speculative execution, the problem is there too

[bossie]

Basic algorithm looks fine, just some remarks to tidy up the code.

[bossie]

Avoid string manipulation and call relativePath.getParent() (see above).

[EmileSonneveld]

It does something different thatn getParent.
I added a comment:
// Remove the executorAttemptDirectory part from the path

[bossie]

I'm not sure what happens if collect returns before competing executors have all finished; I suppose some files don't get cleaned up?

Java has a way to clean up temp files upon JVM exit that we could look into if needed (that would be a separate issue).

[bossie]

More importantly, I also think that this only works if the batch job bucket is mounted because AFAICT you're treating every path as a file on disk.

[EmileSonneveld]

collect should launch the executors and wait till they are finished.
I'll see if I can test without a mounted s3 bucket.
This trick is implemented to avoid an issue we only had with fusemounts, but I think it makes sense to keep it even with direct S3 access, as it avoids a racing condition.

[bossie]

collect should launch the executors and wait till they are finished.

Yes, but speculative execution launches additional redundant executors and I should hope that collect does not wait for the slow ones.

[EmileSonneveld]

Speculative execution seems to kill remaining attempts when one succeeded: killed: another attempt succeeded
cleanUpExecutorAttemptDirectory handles their partial written files

[EmileSonneveld]

• This is tested with writing to standard files in the Scala code. (WriteRDDToGeotiffTest and others)
• And tested with s3 output here: https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/tests/deploy/test_batch_job.py#L1309
• Once gone trough CDSE integration tests, it is best to re-run the tests with fusemount disabled on CDSE dev.