Customized Listener

[lantse]

We are using mappartition function in Spark to transform data in batch. The data read()/write() for each record are in a user defined adapter class. Is there a way to let the spline spark agent to listen to the write() function so that this specific transformation would not be missing in the data lineage record?

In another word, how do we supplement or add to the data lineage record for the area that the agent cannot collect by default? Thanks.

[wajda]

It sounds like you are looking for this - https://github.com/AbsaOSS/spline-spark-agent?tab=readme-ov-file#filters
Also see the example project - https://github.com/AbsaOSS/spline-getting-started/tree/main/spark-agent-extension-example

[lantse]

Hi @wajda , thanks for replying. The example project only shows the creation of a filter. Is there any example of adding missing data by the filter?
Thanks again!

[wajda]

Sure, you can add any additional metadata into the resulting execution plan or event JSON by overriding one of AbstractPostProcessingFilter's methods and adding the stuff you like as a map, like this:

  import za.co.absa.spline.harvester.ExtraMetadataImplicits._
  
  override def processExecutionPlan(plan: ExecutionPlan, ctx: HarvestingContext): ExecutionPlan = {
    plan.withAddedExtra(Map("my_param" -> "my_value"))
  }

You can add extra metadata to an event, plan, or any operation according to your choice.

[lantse]

Thank you. Will try it out.