You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The SCD library doesn't seems working on handling new rows (insertion).
For example:
Existing source table has rows with id : 1,2,3
After initial Dataform run
The table_scd_updates has rows with id : 1,2,3
When there is a new row in the source table : 1,2,3,4
The current logic doesn't handle the new row (id 4).
File code : dataform-scd/index.js
Code line :
"// Create an incremental table with just pure updates, for a full history of the table."
...
"where ${timestamp} > (select max(${timestamp}) from ${ctx.self()})"
The above logic will only insert new row when the id already exists from the initial SCD creation.
Is it intentional or bug?
How if adding additional clause e.g
"where ${timestamp} > (select max(${timestamp}) from ${ctx.self()})"
OR ${uniqueKey} NOT IN (select distinct ${uniqueKey} from ${ctx.self()}))
The text was updated successfully, but these errors were encountered:
The SCD library doesn't seems working on handling new rows (insertion).
For example:
Existing source table has rows with id : 1,2,3
After initial Dataform run
The table_scd_updates has rows with id : 1,2,3
When there is a new row in the source table : 1,2,3,4
The current logic doesn't handle the new row (id 4).
File code : dataform-scd/index.js
Code line :
"// Create an incremental table with just pure updates, for a full history of the table."
...
"
where ${timestamp} > (select max(${timestamp}) from ${ctx.self()})
"The above logic will only insert new row when the id already exists from the initial SCD creation.
Is it intentional or bug?
How if adding additional clause e.g
"
where ${timestamp} > (select max(${timestamp}) from ${ctx.self()})
"OR ${uniqueKey} NOT IN (select distinct ${uniqueKey} from ${ctx.self()}))
The text was updated successfully, but these errors were encountered: