[FEA] Distributed processing of Event Logs

[parthosa]

Currently, we run the Tool (python+jar) on a single machine which is limited by the memory and compute of the host machine. However, Tools should have the capability to process large scale event logs.

Although, we do support running the Tools as a Spark Listener but is not useful for apps that are already processed.

Some of the ideas are:

1. Distributed Processing:
   • If the JAR can be submitted as an Spark App.
2. Batch Processing on a Single Machine:
   • If the Tool can do batching and write the JAR output to multiple directories.
   • Then the Python Tool could process multiple rapids_4_spark_qualification_output directories.
   • Batching can be done based on size of event logs or a config

Update: 11/18/2024:

After our internal POCs, we decided to implement distributed processing of event logs by using PySpark to submit multiple map tasks that take a event log file and run the Tools JAR on it. Finally the results will be collected and merged at the parent Python CLI process. A public design document could be shared if required.

Todo Tasks

Preview Give feedback

1. Support recursive look up of event logs
2. Improve Error Handling and Fault Tolerance for Distributed Mode Failures
3. Add Testing and CI jobs

cc: @viadea @kuhushukla

[amahussein]

Currently, we run the Tool (python+jar) on a single machine which is limited by the memory and compute of the host machine. However, Tools should have the capability to process large scale event logs.

I am not sure I understand the problem. Is it about processing Apps in runtime or about tools resources requirements?

Processing eventlogs require large resources. As instance, Spark History Server is known to require large memory and resources to process eventlogs.
We have issues opened for performance optimizations which mainly target possibility of OOME while processing large eventlogs.

[amahussein]

Previously, the python CLI had option to submit the Tools jar as a Spark job. This was mainly a way to work with large eventlogs since the CLI will be able to spin distributed Spark jobs.
Based on feature requests, the python CLI was converted to be a single Dev machine despite knowing that large scale processing would be a problem.

[tgravescs]

Note that scaling can also be done via making a single machine run more efficient by storing the data in a database vs in memory. For instance like RocksDB. This issue should likely be split up into multiple for the various improvements being made

[tgravescs]

linking #1377 to this for handling lots and lots of event logs.
Also linking #1378 to this for processing huge event logs

[amahussein]

Note that scaling can also be done via making a single machine run more efficient by storing the data in a database vs in memory. For instance like RocksDB. This issue should likely be split up into multiple for the various improvements being made

@tgravescs , yes I agree. We had a previous issue #815 to track that
I am sort of confused about how each of those issues are connected together.
For example, what is the outcome from this issue (1249) Vs. what's in 1378?
IMHO, we should close 1249. Then we can file something specific to Distributed-Tools execution.

[tgravescs]

Please note there are 2 other issues to improve processing of event logs on a single machine:

• [FEA] Scale: Support huge single event log size without using a lot of memory #1378
• [FEA] Scale: Support huge number of eventlogs in single qualification process run #1377