No module named Tensorframes on EMR installation

[nikparmar]

Hey I performed the steps to install tensorframes but at the end when Spark shell launches and I run import tensorframes as tf I get no module tensorframes error

[taklwu]

I had the same issue when just use pyspark with automatic package installation.

pyspark --packages databricks:tensorframes:0.2.8-s_2.11

So, at the end I decided to build the jars from source and use the jar as extra library, also, you might need to install pandas before running pyspark

PYTHONPATH=tensorframes-assembly-0.2.8.jar \
pyspark --jars tensorframes-assembly-0.2.8.jar

[thunterdb]

@nikparmar can you tell me more about your current spark setup?

[nikparmar]

@thunterdb So it's an Amazon EMR setup where I was installing this.

@taklwu I will try and update you

[thunterdb]

@nikparmar I am not very familiar with EMR's setup. Did the suggestion above work for you?

[ghost]

@nikparmar Hello Nik, can you tell me the steps to install tensorframes, i.e. if you have started using it. Thanks,
Kartik

[tanjialiang]

Hi @thunterdb ,

I have similar issues. I have Spark running in yarn-cluster mode (nodes running CentOS7), version 2.0.1. I added databricks:tensorframes:0.2.9-s_2.11 to spark.jars.packages and it seemed to load the jar according to Spark UI. But I still get no module named tensorframes error when I try to import tensorframes. I have python version 2.7.5 installed across all nodes. I cannot go the other method that sets PYTHONPATH to every node because as I am running Spark in cluster mode, and in this case I will have to maintain it in every single node along with the assemly jars. Is this a bug? If not where do you see the issues are?

Thanks,
Jialiang