[FEATURE REQUEST]: Extensibility of UDAF

[samueleresca]

Hi,
I was trying to replicating the following implementation
awslabs/deequ/StatefulDataType using .NET Spark.

The StatefulDataType class extends the UserDefinedAggregateFunction and it internally uses the AggregateExpression to parse a sequence of Columns in the apply method.

Are these APIs already supported in .NET spark? Do you have any plan to support them? Is there any alternative way I can implement the StatefulDataType class using the already existing .NET Spark API?

Thanks

[imback82]

@samueleresca Unfortunately, UserDefinedAggregateFunction is not supported in .NET for Apache Spark. If you already have a Scala implementation, you can use RegisterJavaUDAF to register it and use it in SQL commands.

Just FYI, UserDefinedAggregateFunction will be deprecated in Spark 3.0 in favor of Aggregator.

[AnakinSkyCN]

waiting for the UserDefinedAggregateFunction, so that u wouldn't have to sort the entire DataFrame just to find a Row that holds the max value of a column.