Support for Databricks Notebook Task Lineage as Assets

[Prajwal214]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
-- Currently, OpenMetadata ingests Databricks workflows (pipelines) and displays the sequence of tasks within the "List" and "DAG" sections. However, the individual tasks are not treated as separate assets, and their lineage is not explicitly captured.

On Databricks, each task has its own lineage, showing both input and output tables. Users expect to see the same level of granularity in OpenMetadata, with tasks represented as assets and their lineage captured accordingly.

Describe the solution you'd like
A clear and concise description of what you want to happen.

• Each task within a Databricks pipeline should be ingested as an individual asset.
• The lineage of each task should include its source and destination tables, similar to what is visible in Databricks.
• The DAG representation should accurately reflect the relationships between tasks and their respective lineage.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.