Skip to content
This repository has been archived by the owner on Oct 8, 2019. It is now read-only.

Implement Spark Native UDF interface #346

Open
myui opened this issue Sep 14, 2016 · 1 comment
Open

Implement Spark Native UDF interface #346

myui opened this issue Sep 14, 2016 · 1 comment

Comments

@myui
Copy link
Owner

myui commented Sep 14, 2016

Related to #345, Hive UDF invocation is slow in Spark.
We can do better at least for UDF, currently not for UDAF/UDTF, by implementing Spark's Java UDF{1,...,22} as well as implementing Hive's UDF.

class AngularDistanceUDF extends GenericUDF implements org.apache.spark.sql.api.java.UDF2
https://github.com/myui/hivemall/blob/master/core/src/main/java/hivemall/knn/distance/AngularDistanceUDF.java

Also, we can prepare some helper methods for Spark API in
https://github.com/myui/hivemall/blob/master/core/src/main/java/hivemall/UDFWithOptions.java

@maropu How do you think?

@maropu
Copy link
Contributor

maropu commented Sep 14, 2016

yea, I think it's a good idea. I'll try later.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

2 participants