Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] BasePythonRunner has a new parameter metrics in Spark 4.0 #11895

Open
hyperbolic2346 opened this issue Dec 19, 2024 · 1 comment
Open
Labels
? - Needs Triage Need team to review and classify bug Something isn't working

Comments

@hyperbolic2346
Copy link
Collaborator

hyperbolic2346 commented Dec 19, 2024

Describe the bug
CI found an issue with the GPUBasePythonRunner where it is not passing the metrics parameter to the parent class BasePythonRunner.

Steps/Code to reproduce bug
Build the image for spark 4.0

[2024-12-19T16:52:14.524Z] [INFO] --- scala-maven-plugin:4.9.1:compile (scala-compile-first) @ rapids-4-spark-sql_2.13 ---

[2024-12-19T16:52:14.779Z] [INFO] Compiler bridge file: /home/jenkins/agent/workspace/jenkins-rapids_scala213_nightly-dev-github-345/scala2.13/target/spark400/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.13-1.10.0-bin_2.13.14__61.0-1.10.0_20240505T232140.jar

[2024-12-19T16:52:41.249Z] [INFO] compiling 479 Scala sources and 58 Java sources to /home/jenkins/agent/workspace/jenkins-rapids_scala213_nightly-dev-github-345/scala2.13/sql-plugin/target/spark400/classes ...

[2024-12-19T16:52:41.250Z] [ERROR] [Error] /home/jenkins/agent/workspace/jenkins-rapids_scala213_nightly-dev-github-345/sql-plugin/src/main/spark341db/scala/org/apache/spark/sql/rapids/execution/python/shims/GpuBasePythonRunner.scala:36: not enough arguments for constructor BasePythonRunner: (funcs: Seq[org.apache.spark.api.python.ChainedPythonFunctions], evalType: Int, argOffsets: Array[Array[Int]], jobArtifactUUID: Option[String], metrics: Map[String,org.apache.spark.util.AccumulatorV2[Long,Long]]): org.apache.spark.api.python.BasePythonRunner[IN,org.apache.spark.sql.vectorized.ColumnarBatch].

[2024-12-19T16:52:41.250Z] Unspecified value parameter metrics.
@hyperbolic2346 hyperbolic2346 added ? - Needs Triage Need team to review and classify bug Something isn't working labels Dec 19, 2024
@gerashegalov
Copy link
Collaborator

It will come up in the audit for apache/spark@2b1369b

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
? - Needs Triage Need team to review and classify bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants