Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EventHistoryToSparklensJson doesn't work with local or HDFS events file/directory #52

Open
Harshit22 opened this issue May 3, 2020 · 1 comment

Comments

@Harshit22
Copy link

EventHistoryToSparklensJson class treats input events file argument as local file or directory. However, EventHistoryReporter class, used internally, reads it as HDFS file.

This makes both local and HDFS events file unusable with EventHistoryToSparklensJson.
Doc mentions that input file should be local path.

To circumvent this issue, I had to keep events file in both local and HDFS filesystems at identical paths.

Jar used: https://mvnrepository.com/artifact/qubole/sparklens/0.3.1-s_2.11
Java 8/Scala 2.11/Spark 2.4.3/AWS EMR

@iamrohit
Copy link
Contributor

iamrohit commented May 4, 2020

@Harshit22 it works for local files and directories, assuming the "local" doesn't have HDFS setup. Are you running it from one of the cluster machines with HDFS configured?
The primary reason to support HDFS was to ensure that while running sparklens with spark application, one can save the sparklens json file to known s3 or HDFS location, which is useful if one doesn't have ssh access to machine running the driver.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants