how do I call AvroSchemaUtils.toAvroSchema from python?

[kothapet]

Hello,
How do I call the following methods described in the "Generate schema from data" from python. Can you show some example and update the python documentation?

import za.co.absa.abris.avro.parsing.utils.AvroSchemaUtils

// generate schema for all columns in a dataframe
AvroSchemaUtils.toAvroSchema(dataFrame)

// generate schema for one column in a dataframe
AvroSchemaUtils.toAvroSchema(dataFrame, "input")

// generate schema for multiple columns in a dataframe
AvroSchemaUtils.toAvroSchema(dataFrame, Seq("input", "numbers"))

I tried the following but that gives an error like below. Guessing the pyspark dataframe needs to convert to scala dataframe?

keyAvroSchema = jvm_gateway.za.co.absa.abris.avro.parsing.utils.AvroSchemaUtils.toAvroSchema(dataframe, "key")

Traceback (most recent call last):
  File "D:\GitProjects\PySparkProject\PySparkProject\pySpark\StructStreaming\StructuredStreamingAbris.py", line 160, in <module>
    keySchemaODF = jvm_gateway.za.co.absa.abris.avro.parsing.utils.AvroSchemaUtils.toAvroSchema(df2, "key")
  File "D:\SPARK\spark-3.2.1-bin-without-hadoop\python\lib\py4j-0.10.9.3-src.zip\py4j\java_gateway.py", line 1313, in __call__
  File "D:\SPARK\spark-3.2.1-bin-without-hadoop\python\lib\py4j-0.10.9.3-src.zip\py4j\java_gateway.py", line 1283, in _build_args
  File "D:\SPARK\spark-3.2.1-bin-without-hadoop\python\lib\py4j-0.10.9.3-src.zip\py4j\java_gateway.py", line 1283, in <listcomp>
  File "D:\SPARK\spark-3.2.1-bin-without-hadoop\python\lib\py4j-0.10.9.3-src.zip\py4j\protocol.py", line 298, in get_command_part
  File "D:\SPARK\spark-3.2.1-bin-without-hadoop\python\pyspark\sql\dataframe.py", line 1659, in __getattr__
    raise AttributeError(
AttributeError: 'DataFrame' object has no attribute '_get_object_id'

[cerveada]

Hi, sorry. I don't know myself.

I tried the following but that gives an error like below. Guessing the pyspark dataframe needs to convert to scala dataframe?

You may be right. Seems like using the tem view could work as described in the following page:
https://stackoverflow.com/questions/37263300/moving-spark-dataframe-from-python-to-scala-whithn-zeppelin

If you figure it out, let us know nad we can add the proper solution to the examples.

[kothapet]

I figured it out. This will work.
Looks like to convert from python dataframe to java dataframe you have to do df._jdf

LINK

Create a Python wrapper method
Similarly create other Python wrapper methods for the toAvroSchema variations

def toAvroSchema(dataFrame, columnName, recordName="defaultName", nameSpace ="defaultNameSpace"):

    jvm_gateway = SparkContext._active_spark_context._gateway.jvm

    return jvm_gateway.za.co.absa.abris.avro.parsing.utils.AvroSchemaUtils.toAvroSchema(dataFrame._jdf, columnName, recordName, nameSpace ).toString()


keyAvroSchema = toAvroSchema(df, "key")