feat(lookml): Use Looker API to get fields of a View

[askumar27]

Use Looker APIs to get fields for a Views

Current Approach

• We use lkml parser to get View attributes, and this has limitation of when Views are very complex

Proposed Approach

• Use Looker APIs to get fields of a View, this API need to be called via the Explore name.
• As many Views exist within an Explore - number of API calls <= number of Views with local LRU cache
• The code tries to groups as many Views to the same Explore as possible, so as to minimize the number of API calls. As a single View can exist in many Explore
• Add reliability towards lineage extraction in the next step where SQL query is prepared based on the fields

Other changes

• More debug logging
• Automatic fallback to view_context based fields extraction with better error handling
• Best effort fields extraction - continue on failure (atleast one field is required) instead of falling back to different upstream strategy

[codecov]

Codecov Report

❌ Patch coverage is 72.34043% with 26 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...c/datahub/ingestion/source/looker/view_upstream.py	61.76%	26 Missing ⚠️

📢 Thoughts on this report? Let us know!

[sgomezvillamor]

• We use lkml parser to get View attributes, and this has limitation of when Views are very complex
• Use Looker APIs to get fields of a View, this API need to be called via the Explore name.

Are we adding a fallback strategy? or do we replace with new strategy?
If the second, that could lead to some breaking change. If so, how can we address or mitigate it?

[sgomezvillamor]

Looker/LookML ingestion is becoming a very complex code.
That's noted in the amount of debug logging required.

Given the amount of detail we require for troubleshooting, I wonder if there is python tooling that we can use to trace execution and no require explicit logger debug lines. 🤔

Not a blocker, just thinking loudly.

[askumar27]

• We use lkml parser to get View attributes, and this has limitation of when Views are very complex
• Use Looker APIs to get fields of a View, this API need to be called via the Explore name.

Are we adding a fallback strategy? or do we replace with new strategy? If the second, that could lead to some breaking change. If so, how can we address or mitigate it?

There is a fallback strategy in place if there are any issues with Looker APIs to get the fields, it falls back to view_context based solution (current). There should not be any backward incompatibility issues here with either of the strategy. Worst case scenarios is no CLL or partial CLL (same as today)

• Automatic fallback to view_context based fields extraction with better error handling
• Best effort fields extraction - continue on failure (atleast one field is required) instead of falling back to different upstream strategy

[askumar27]

Looker/LookML ingestion is becoming a very complex code. That's noted in the amount of debug logging required.

Given the amount of detail we require for troubleshooting, I wonder if there is python tooling that we can use to trace execution and no require explicit logger debug lines. 🤔

Not a blocker, just thinking loudly.

This is a great suggestion for tracing, we should certainly explore options.
Regarding the debug logs, Sorry I forgot to mention in the PR - this is also a dev build to be used for a customer debug. I will then trim out the debug logs before merging.