Downloads fail with shuffle problem

[marcos-lg]

Some downloads fail with errors like these:

Caused by: java.lang.RuntimeException: java.nio.file.NoSuchFileException: /data/spark-x/executor-x/blockmgr-f5aa0006-89ef-4294-b499-2586060192f8/2a/shuffle_0_358_0.index

e.g.: 0003512-250218110819086: https://airflow.gbif.org/log?execution_date=2025-02-20T15%3A08%3A16.382463%2B00%3A00&task_id=download_query_monitor&dag_id=gbif_occurrence_download_dag&map_index=-1

[muttcg]

This is a typical indicator of a lack of executor memory. There could be multiple reasons for this:

1. Memory is ok, but there was no space available for the executor pod, causing it to remain in pending mode and putting pressure on the rest of the executors.
2. Memory is ok, but the executor failed to bind the PVC and remained in pending mode, putting pressure on the rest of the executors.
3. The memory settings are incorrect for the particular download