Skip processing apps with invalid platform and spark runtime configurations

[parthosa]

Fixes #1420.

This PR updated the tools behavior to skip processing apps that have a runtime not supported by the platform.

Changes

Enhancements to Platform Support:

• core/src/main/scala/com/nvidia/spark/rapids/tool/Platform.scala: Added a default runtime and supported runtimes to the Platform class, and introduced a method to check if a given runtime is supported. Updated DatabricksPlatform to include PHOTON as a supported runtime. [1] [2]

Runtime Validation:

• core/src/main/scala/org/apache/spark/sql/rapids/tool/AppBase.scala: Added a validateSparkRuntime method to ensure the parsed Spark runtime is supported by the platform, throwing an UnsupportedSparkRuntimeException if not. Updated the constructor to include the platform. [1] [2]

Integration with Profiling and Qualification Tools:

• core/src/main/scala/com/nvidia/spark/rapids/tool/profiling/Profiler.scala: Integrated platform support into the Profiler class, ensuring that the platform is correctly instantiated and passed to ApplicationInfo. [1] [2]
• core/src/main/scala/org/apache/spark/sql/rapids/tool/qualification/QualificationAppInfo.scala: Updated to include platform support in the QualificationAppInfo class.

Test

Unit Tests

• core/src/test/scala/com/nvidia/spark/rapids/tool/ToolTestUtils.scala: Modified test utilities to create platform instances based on the provided platform name.
• core/src/test/scala/com/nvidia/spark/rapids/tool/planparser/BasePlanParserSuite.scala: Updated test cases to include platform names when creating applications from event logs.
• core/src/test/scala/com/nvidia/spark/rapids/tool/profiling/AnalysisSuite.scala: Enhanced test cases to validate platform-specific behavior, including handling of Databricks Photon runtime logs. [1] [2]

Behave Tests

• [user_tools/tests/spark_rapids_tools_e2e/features/event_log_processing.feature]: Updated Python behave tests to include case for onprem and databricks-aws platforms for photon event logs.

[amahussein]

I opened a discussion on the issue. I am not convinced that this is the best way the tools should behave.

[parthosa]

@amahussein Updated the behavior to skip processing apps that have a runtime not supported by the platform.

[amahussein]

Thanks @parthosa
I was thinking about postponing the creation of the platform as much as possible. The platform would be created after there is enough information to decide which platform it is. If not, then we use the argument.
This is a risky change and it requires some considerable amount of testing; especially that we need to revisit the python side.

• At least the platform argument in python would have to be "non-optional". This will reduce the possibility of user hitting the problem because of a wrong CLI guess
• Revisit the workflow from python, to Scala, to AutoTuner, then back to Python. Previously, we had to initialize the platform prior to processing the eventlogs because we were using the CSP SDK. With the StorageLib in place, we can finally avoid that flow on python side.
• I am also concerned about the impact of that on the user + qualX

We can discuss further online and get feedback from @leewyang about how this would impact QualX

[amahussein]

That's not what I exactly thought we are heading to. Creating that platform before processing the eventlog implies that we are not using the information/cluster-detection logic from the eventlog.

[parthosa]

Thanks @amahussein for the review.

I agree that platform should be decided after processing of the event log. However, I think the concerns related to incorrect platform detection is outside the scope of this issue. This issue aims to solve the problem that if the platform (user provided or detected by our CLI) is incompatible with the Spark Runtime, then we should skip processing it.

• QualificationAppInfo already stores the platform that is provided by the user or detected by our CLI.
• This PR adds the platform field in AppBase that will be populated by QualificationAppInfo (by qualification tool) and ApplicationInfo (by profiling tool).
• Now, this platform will be used by AppBase to validate the compatibility with the Spark Runtime.

[parthosa]

Based on offline discussion, this requires that the correct platform is always provided.

Converting this to draft untill #1462 is merged.

[parthosa]

This PR is ready to be reviewed as #1462 is merged.

[amahussein]

This PR is ready to be reviewed as #1462 is merged.

Sorry, I overlooked something in #1463
if platform is required, we probably want to change the argument in the tools_cli to be non-optional. right?

[parthosa]

@amahussein

If platform is required, we probably want to change the argument in the tools_cli to be non-optional. right?

The platform argument is validated later in the argprocessor. This ensures a consistent error message from Pydantic when the platform argument is not specified (similar to all other args validation). Therefore, I did not make the platform argument mandatory in tools_cli.

Error Message when platform is not specified:

2024-12-18 09:13:02,002 ERROR spark_rapids_tools.argparser: Validation err: 1 validation error for QualifyUserArgModel
  Cannot run tool cmd without platform argument. Re-run the command providing the platform argument.
  Error: [type=invalid_argument, input_value=ArgsKwargs((), {'eventlog.../tools_config_00.yaml'}), input_type=ArgsKwargs]

[amahussein]

LGTME