Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handling new insertion on existing SCD table #1

Open
adiwijaya opened this issue Jan 19, 2022 · 1 comment
Open

Handling new insertion on existing SCD table #1

adiwijaya opened this issue Jan 19, 2022 · 1 comment

Comments

@adiwijaya
Copy link

The SCD library doesn't seems working on handling new rows (insertion).

For example:
Existing source table has rows with id : 1,2,3

After initial Dataform run
The table_scd_updates has rows with id : 1,2,3

When there is a new row in the source table : 1,2,3,4

The current logic doesn't handle the new row (id 4).

File code : dataform-scd/index.js
Code line :
"// Create an incremental table with just pure updates, for a full history of the table."
...
"where ${timestamp} > (select max(${timestamp}) from ${ctx.self()})"

The above logic will only insert new row when the id already exists from the initial SCD creation.

Is it intentional or bug?

How if adding additional clause e.g
"where ${timestamp} > (select max(${timestamp}) from ${ctx.self()})"
OR ${uniqueKey} NOT IN (select distinct ${uniqueKey} from ${ctx.self()}))

@kkulczak
Copy link

I cannot reproduce the bug.

In my experiments new row (id 4) was inserted correctly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants