Generating speedup factors for Dataproc GKE L4 GPU instances

[parthosa]

Contributes to #253. This PR adds support for Dataproc GKE with L4 instances in core tools.

Changes:

• Operator scores are generated using the validate_qualification_estimates.py script.

Metrics

• Actual GPU Speed Up : 3.74
• Estimated Speed Up. : 3.09
• Error : -21.04%

Spark Config

GPU:

spark.kubernetes.container.image=knobby/spark-rapids:latest 
spark.executor.resource.gpu.vendor=nvidia.com 
spark.executor.extraClassPath=/opt/spark/jars/rapids-4-spark_2.12-23.10.0-20230828.131939-24-cuda11.jar 
spark.driver.extraClassPath=/opt/spark/jars/rapids-4-spark_2.12-23.10.0-20230828.131939-24-cuda11.jar 
spark.executor.resource.gpu.amount=1 
spark.executor.resource.gpu.discoveryScript=/opt/spark/getGpusResources.sh 
spark.plugins=com.nvidia.spark.SQLPlugin 
spark.dynamicAllocation.enabled=false 
spark.task.resource.gpu.amount=0.0625 
spark.driver.maxResultSize=2GB 
spark.driver.memory=50G 
spark.executor.cores=16 
spark.executor.instances=8 
spark.rapids.sql.batchSizeBytes=1GB 
spark.locality.wait=0 
spark.rapids.sql.concurrentGpuTasks=2 
spark.executor.memory=16G 
spark.sql.files.maxPartitionBytes=2gb 
spark.rapids.memory.host.spillStorageSize=32G 
spark.rapids.memory.pinnedPool.size=8g 
spark.executor.memoryOverhead=16G 
spark.scheduler.minRegisteredResourcesRatio=1.0 
spark.rapids.shuffle.multiThreaded.writer.threads=16 
spark.rapids.shuffle.multiThreaded.reader.threads=16 
spark.rapids.shuffle.mode=MULTITHREADED

• Workload Information:
  • Controller: 1 X n1-standard-16
  • Driver: 1 X n1-standard-16
  • Executors: 8 X n1-standard-32, (with 8 X L4 24GB)

CPU:

spark.kubernetes.container.image=knobby/spark-rapids:latest
spark.dynamicAllocation.enabled=false
spark.driver.maxResultSize=2GB
spark.driver.memory=50G
spark.executor.cores=16
spark.executor.instances=8
spark.locality.wait=0
spark.executor.memory=16G
spark.sql.files.maxPartitionBytes=2gb
spark.executor.memoryOverhead=16G
spark.scheduler.minRegisteredResourcesRatio=1.0
spark.sql.adaptive.enabled=true
spark.sql.broadcastTimeout=1200

• Workload Information:
  • Controller: 1 X n1-standard-16
  • Driver: 1 X n1-standard-16
  • Executors: 8 X n1-standard-32