GPU Integration Challenges with Spark Rapids on Standalone Spark Clusters

[reysand]

When attempting to start spark with RAPIDS, I consistently face issues where the executors lose connection, indicated by the message "command exited with code 1," followed by reconnection attempts, which are repeatedly made.

24/06/07 13:13:21 ERROR TaskSchedulerImpl: Lost executor 0 on {IP}: Command exited with code 1
24/06/07 13:13:23 ERROR TaskSchedulerImpl: Lost executor 1 on {IP}: Command exited with code 1

Additionally, when using spark-submit, I encounter a warning regarding GPU overrides:

24/06/07 13:15:29 WARN GpuOverrides:
! <RowDataSourceScanExec> cannot run on GPU because GPU does not currently support the operator class org.apache.spark.sql.execution.RowDataSourceScanExec
  @Expression <AttributeReference> min#0 could run on GPU
  @Expression <AttributeReference> max#1 could run on GPU

This warning is followed by further losses of executors.

nvidia-smi:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.42.02              Driver Version: 555.42.02      CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3050        On  |   00000000:01:00.0 Off |                  N/A |
|  0%   26C    P8             N/A /  115W |      49MiB /   8192MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

versions:

Spark: 3.5.1
Scala: 2.12.18
Java: 11
RAPIDS: 24.04.1

[tgravescs]

Hello,

24/06/07 13:13:21 ERROR TaskSchedulerImpl: Lost executor 0 on {IP}: Command exited with code 1

Can you check the executor logs? Many times those will have details as to why the executors are lost. Depending on the resource manager you are using, many times you can see them in the Spark UI under the "Executors" tab. If you are running on something like Kubernetes you may have to try to use kubectl logs on the pod or if you are aggregating them somewhere look there.

[reysand]

Hi, I've checked the logs:

24/06/10 09:10:56 ERROR RapidsExecutorPlugin: Exception in the executor plugin, shutting down!
ai.rapids.cudf.CudaFatalException: Fatal CUDA error encountered at: /home/jenkins/agent/workspace/jenkins-spark-rapids-jni-release-35-cuda11/thirdparty/cudf/java/src/main/native/src/CudaJni.cpp:206: 999 cudaErrorUnknown unknown error
	at ai.rapids.cudf.Cuda.getComputeCapabilityMajor(Native Method)
	at com.nvidia.spark.rapids.RapidsPluginUtils$.validateGpuArchitecture(Plugin.scala:368)
	at com.nvidia.spark.rapids.RapidsExecutorPlugin.init(Plugin.scala:499)
	at org.apache.spark.internal.plugin.ExecutorPluginContainer.$anonfun$executorPlugins$1(PluginContainer.scala:125)
	at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
	at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
	at org.apache.spark.internal.plugin.ExecutorPluginContainer.<init>(PluginContainer.scala:113)
	at org.apache.spark.internal.plugin.PluginContainer$.apply(PluginContainer.scala:211)
	at org.apache.spark.internal.plugin.PluginContainer$.apply(PluginContainer.scala:199)
	at org.apache.spark.executor.Executor.$anonfun$plugins$1(Executor.scala:337)
	at org.apache.spark.util.Utils$.withContextClassLoader(Utils.scala:178)
	at org.apache.spark.executor.Executor.<init>(Executor.scala:337)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$receive$1.applyOrElse(CoarseGrainedExecutorBackend.scala:181)
	at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:115)
	at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213)
	at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
	at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75)
	at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

[reysand]

My issue has been resolved. I reinstalled both CUDA and the NVIDIA driver. Additionally, users on the nodes need to join the video group.