py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. : java.lang.NoClassDefFoundError: scala/Product$class

[kommuembroker]

Hi All,

I have downloaded jar file as:

wget https://repos.spark-packages.org/qubole/sparklens/0.3.0-s_2.11/sparklens-0.3.0-s_2.11.jar

and I am trying to run sparklens as below, in Spark 3.0.0 in aws EMR

spark-submit --jars ./sparklens-0.3.0-s_2.11.jar --conf spark.network.timeout=100000 --conf spark.extraListeners=com.qubole.sparklens.QuboleJobListener --conf spark.executor.heartbeatInterval=10000 --conf spark.sql.shuffle.partitions=125 --conf spark.default.parallelism=88 --py-files config.zip,jobs.zip,DDL.zip main.py

However, I am getting an error as

py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.NoClassDefFoundError: scala/Product$class

Thanks,
Ad