Custom Plugin development

[rohank-dev]

Hello All,

Thanks for all your inputs earlier and a quick response.

I am having a query related to custom plugin development. I am planning to pass some additional information in the form of params in Map<"table_name", Map<"additional_info",tabledesc>>.

As per the documentation I understand I can write a custom plugin with the implementation I need, in my case there are 2 Plugin's where I am planning to make this changes -

1. SQLPlugin --> I am planning to create a copy of this with my implmentation, renaming it to MyCustomSqlPlugin and adding it to higher precedence than the default one, so to prioritize it.
2. SaveIntoDataSourceCommandPlugin --> here I am not sure a bit since RelationProviderProcessing and WriteNodeProcessing both uses this Plugin and I want to make sure that opts is adding the the additional information needed wherever this Plugin is getting used.

Please suggest.

Thanks

[cerveada]

SaveIntoDataSourceCommandPlugin extends WriteNodeProcessing. So it's one of the plugins doing WriteNodeProcessing.

SaveIntoDataSourceCommandPlugin tries to extract the relation and when successful it will send it to a group of plugins implementing RelationProviderProcessing. One of the plugins will then finish the work and return WriteNodeInfo

---

You may also consider using or implementing a Post Processing Filter. They are more high level, and are able to modify any lineage that Agent captured via Plugins. Even in filter, you still have access to all Spark lineage data via HarvestingContext.

see: https://github.com/AbsaOSS/spline-spark-agent#post-processing-filters

[rohank-dev]

Thanks for your response, but let me more specific for the use case of SaveIntoDataSourceCommandPlugin, I am planning to add Table and identifier information for the tables which are of Option tables. Need to extract source table from the options tables.

Also for SQLPlugin as mentioned earlier I will be renaming it to MySqlplugin and set the precedence higher so that my plugin will be picked instead of the sqlplugin.

please let me know if for both plug-ins development the approach is good since post processing filter won’t help much

[cerveada]

Also for SQLPlugin as mentioned earlier I will be renaming it to MySqlplugin and set the precedence higher so that my plugin will be picked instead of the sqlplugin.

Yes, I think that was the intention behind @Priority(Precedence.User) to give users Plugins the option to override the default behavior.

for the use case of SaveIntoDataSourceCommandPlugin, I am planning to add Table and identifier information for the tables which are of Option tables. Need to extract source table from the options tables.

Not sure what is your question here? Are you talking about the case where RelationProviderExtractor is not matched?

[rohank-dev]

Not sure what is your question here? Are you talking about the case where RelationProviderExtractor is not matched?

Currently if we look at the code in SaveIntoDataSourceCommandPlugin under RelationProviderExtractor we are having below message when table is option table

val uri = opts.get("path")
  .map(pathQualifier.qualify) 
  .getOrElse(sys.error(s"Cannot extract source URI from the options: ${opts.keySet mkString ","}"))

I am planning to handle it via code, where we can check if opts is having path as key and that can be used to populate URI else dont populate it. Also populate Identifier by fetching details from opts table details.

[cerveada]

we can check if opts is having path as key and that can be used to populate URI

That is already done currently, isn't it?

Also populate Identifier by fetching details from opts table details.

This part is not done currently, yes

---

If you want to improve the extraction of the url or add some additional info it could be made into PR that we could merge back into the Agent.

One thing I am not sure about is gathering the lineage when the URL is not populated. Generally that lineage is not usable by spline because knowing you store data somewhere without knowing where is kind of useless.

[rohank-dev]

One thing I am not sure about is gathering the lineage when the URL is not populated. Generally that lineage is not usable by spline because knowing you store data somewhere without knowing where is kind of useless.

Yes, you are correct that's the reason the way I am thinking is adding at least the database and table name as an Identifier information if URI is nor populated so we know the source of data in database name and table name form.

And in case if we neither get URI nor the identifier details then, we can log error message as the current way the code is handling mentioning we cannot extract the relevant information. I hope it clears the doubt.