[Bug] Intermittent doc_id test-src failures in ci/cd.

[daw3rd]

Search before asking

• I searched the issues and found no similar issues.

Component

Transforms/universal/doc_id

What happened + What you expected to happen

PR merging is failing a build of the universal transforms in doc_id test-src. Failing one of the tests
See log below, but problem is the that the "processing_time" key seems to be missing. Can't see how that would happen yet.

Reproduction script

Don't know how to reliably reproduce.

Anything else

The following from a failed build (full log here https://github.com/IBM/data-prep-kit/actions/runs/10721471796/job/29730072310#step:4:738)

...
Successfully installed argparse-1.4.0 dpk_doc_id_transform_ray-0.2.1.dev0
source venv/bin/activate;       \
export PYTHONPATH=../src;       \
cd test; pytest -s .
============================= test session starts ==============================
platform linux -- Python 3.10.12, pytest-8.3.2, pluggy-1.5.0
rootdir: /home/runner/work/data-prep-kit/data-prep-kit/transforms/universal/doc_id/ray
configfile: pyproject.toml
plugins: cov-5.0.0, anyio-4.4.0
collected 1 item

13:37:18 INFO - Running locally
13:37:18 INFO - Doc id parameters are : {'doc_column': 'contents', 'hash_column': 'hash_column', 'int_column': 'int_id_column', 'start_id': 5}
13:37:18 INFO - data factory data_ is using local data access: input_folder - /home/runner/work/data-prep-kit/data-prep-kit/transforms/universal/doc_id/ray/test-data/input output_folder - /tmp/doc_idb9v9ci15
13:37:18 INFO - data factory data_ max_files -1, n_sample -1
13:37:18 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']
13:37:18 INFO - pipeline id pipeline_id
13:37:18 INFO - code location None
13:37:18 INFO - number of workers 1 worker options {'num_cpus': 0.8, 'max_restarts': -1}
13:37:18 INFO - actor creation delay 0
13:37:18 INFO - job details {'job category': 'preprocessing', 'job name': 'doc_id', 'job type': 'ray', 'job id': 'job_id'}
2024-09-05 13:37:21,603	INFO worker.py:1744 -- Started a local Ray instance. View the dashboard at 127.0.0.1:8265 
(orchestrate pid=2372) 13:37:22 INFO - orchestrator started at 2024-09-05 13:37:22
(orchestrate pid=2372) 13:37:22 INFO - Number of files is 1, source profile {'max_file_size': 0.034458160400390625, 'min_file_size': 0.034458160400390625, 'total_file_size': 0.034458160400390625}
(orchestrate pid=2372) 13:37:22 INFO - Cluster resources: {'cpus': 4, 'gpus': 0, 'memory': 8.689183045178652, 'object_store': 4.344591521658003}
(orchestrate pid=2372) 13:37:22 INFO - Number of workers - 1 with {'num_cpus': 0.8, 'max_restarts': -1} each
(orchestrate pid=2372) 13:37:22 INFO - Completed 0 files (0.0%)  in 0.0 min. Waiting for completion
(orchestrate pid=2372) 13:37:23 INFO - Completed processing 1 files in 0.012 min
(orchestrate pid=2372) 13:37:23 INFO - done flushing in 0.001 sec
(orchestrate pid=2372) Traceback (most recent call last):
(orchestrate pid=2372)   File "/home/runner/work/data-prep-kit/data-prep-kit/data-processing-lib/ray/src/data_processing_ray/runtime/ray/transform_orchestrator.py", line 138, in orchestrate
**(orchestrate pid=2372)     stats["processing_time"] = round(stats["processing_time"], 3)
(orchestrate pid=2372) KeyError: 'processing_time'**
(orchestrate pid=2372) 13:37:23 ERROR - Exception during execution 'processing_time': None
13:37:33 INFO - Completed execution in 0.251 min, execution result 1
test_doc_id_ray.py Using temporary output path /tmp/doc_idb9v9ci15

OS

Ubuntu

Python

3.10.x

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[daw3rd]

I have run this locally on 3.10 and 3.11 10 times straight w/o errors (on mac m1).
Same on redhat 3.11.

make PYTHON=python3.10 venv
source venv/bin/activate
cd test
export PYTHONPATH=../src
for i in 1 2 3 4 5 6 7 8 9 10; do pytest -s test_doc_id_ray.py ; done