v24.10.0

Packages

• Maven Release: https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/24.10.0/
• PyPI Package: https://pypi.org/project/spark-rapids-user-tools/24.10.0/

Changes

User Tools

• [FEA] Allow users to specify custom Dependency jars (#1395)
• Reduce default memory allocation to the java process (#1407)
• Update error handling in python for parsing cluster information (#1394)
• user-tools should add xms argument to java cmd (#1391)
• Use environment variables to set thresholds in static yaml configurations (#1389)
• Use StorageLib to download dependencies (#1383)
• Remove total core second heuristic and filter apps only in top candidate view (#1376)
• Generate log files for Python Profiling cli (#1366)
• Update models for updated datasets and latest code (#1365)
• Isolate dataset for qualx plugin invocations (#1361)
• [FEA] Add total core seconds into top candidate view (#1342)
• Fix python tool picking up wrong JAR version in Fat wheel mode (#1357)
• [FOLLOWUP-1326] Set Spark version to 3.4.2 by default for onprem environment (#1358)
• Disable too-many-positional-arguments in pylintrc (#1353)
• Reduce console output tree level, exclude JAR tool output files and remove incorrect logging (#1340)

Core

• Add support for Photon-specific SQL Metrics (#1390)
• Add support for processing Photon event logs in Scala (#1338)
• Add Reflection to support custom Spark Implementation at Runtime (#1362)
• Improve AQE support by capturing SQLPlan versions (#1354)
• Add PartitionFilters and DataFilters to the dataSourceInfo table (#1346)
• Add support to ArrayJoin in Qualification tool (#1345)

Miscellaneous

• Cluster information should handle dynamic allocation and nodes being removed and added (#1369)
• Rename tag core to core_tools (#1350)