Enable the creation of external TableReferences #139
Conversation
|
Enabling external views for DuckDB depends on duckdb/duckdb#10322 |
We could implement a workaround since this problem is quite common among some SQLAlchemy backends. We have such a workaround in other places of the codebase. |
| dict( | ||
| name=tbl.name, | ||
| schema=store.get_schema(tbl.stage.current_name).get(), | ||
| schema=store.get_schema(tbl.stage.current_name, tbl).get(), |
There was a problem hiding this comment.
It is good to add external tables here. However, it might be worth noting which ones are external ones. We use this list to request a shared reader lock for DB2 on source tables. This list is not used to produce the actual query. Thus it might be ok to exclude external tables for some actions. An external table might also be a federated production table in DB2 which we should not lock or don't have the permission to lock.
=> This is important to keep in mind during roll-out for DB2
| schema=store.get_schema(tbl.stage.current_name, tbl).get(), | |
| schema=store.get_schema(tbl.stage.current_name, tbl).get(), | |
| external_schema=tbl.external_schema, |
If we transport external_schema separately, we might not need to add tbl as argument to store.get_schema
There was a problem hiding this comment.
I added a new argument db2_shared_lock_allowed (still to be tested). Does that make sense?
There was a problem hiding this comment.
A DB2 test has been added in a5769bc
Done. |
# Conflicts: # docs/source/changelog.md
|
@windiana42 Let me know if it looks good now 😃 |
Also adjusted documentation to be more generic. Redefined default to False.
| src_tables = [ | ||
| f"{preparer.format_schema(tbl['schema'])}.{preparer.quote(tbl['name'])}" | ||
| for tbl in create.source_tables | ||
| if tbl["shared_lock_allowed"] |
There was a problem hiding this comment.
Just to remind myself: this is needed to prevent DB2 dialect from locking external table reference input tables if not allowed.
|
@nicolasmueller good catch! did you find something else? |
@windiana42 No, I am fine with merging then 👍 |
# Conflicts: # docs/source/changelog.md
3 participants
Enable the creation of external TableReferences that reference tables in external schemas. This is useful to tell pipedag about tables that already exist in the database and should not be copied. It avoid manually reflecting them, which can sometimes be cumbersome as one might want to call a
Taskwith a pipedagTableand sometimes with an external table.Checklist
docs/source/changelog.mdentry