batch_job.py write_metadata: avoid ad-hoc file selection for upload

[soxofaan]

openeo-geopyspark-driver/openeogeotrellis/deploy/batch_job.py

Lines 511 to 519 in b63280c

	logger.info("Writing results to object storage")
	for file_path in filter(lambda x: x.is_file(), job_dir.rglob("*")):
	# TODO: Get list of files to upload from metadata file.
	# AFAIK, nothing else is created locally and needs to be uploaded to s3 afterwards.
	if UDF_PYTHON_DEPENDENCIES_FOLDER_NAME in str(file_path):
	logger.warning(f"Omitting {file_path} as the executors will not be able to access it")
	else:
	full_path = str(file_path.absolute())
	s3_instance.upload_file(full_path, bucket, full_path.strip("/"))

Here we're building an ugly ad-hoc deny-list for "files" that should not be uploaded to S3

As mentioned in the TODO, we should use an explicit asset list to upload instead of blindly assuming everything from the job dir should be uploaded (minus some hand-picked exceptions)

[soxofaan]

cc @EmileSonneveld

[soxofaan]

As an illustration that this does not scale:

if UDF_PYTHON_DEPENDENCIES_FOLDER_NAME in str(file_path):

doesn't even work as UDF_PYTHON_DEPENDENCIES_FOLDER_NAME is not in play anymore on CDSE since #845

[EmileSonneveld]

In export_workspace list of files that exists locally an on s3 is determined by the list of stac metadata files.
For example colection.json + item.tiff.json. Probably the same can be used here

[EmileSonneveld]

This issue is a direct result of changes introduced in #877

[EmileSonneveld]

Logged what files do get uploaded. All logs on cdse dev where one of the following 2:

Writing results to object storage. paths=[PosixPath('/batch_jobs/j-XXX/job_specification.json'), PosixPath('/batch_jobs/j-XXX/job_metadata.json'), PosixPath('/batch_jobs/j-XXX/openEO_2017-03-07Z.tif'), PosixPath('/batch_jobs/j-XXX/openEO_2017-03-07Z.tif.aux.xml'), PosixPath('/batch_jobs/j-XXX/openEO_2017-03-07Z.tif.json'), PosixPath('/batch_jobs/j-XXX/collection.json')]

Writing results to object storage. paths=[PosixPath('/batch_jobs/j-XXX/job_specification.json'), PosixPath('/batch_jobs/j-XXX/job_metadata.json')]

[EmileSonneveld]

Tested on CDSE dev without fusemount. Running the test suite seems to be slightly faster now (40min->30min, but only checked on a few CI runs)