Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Override database config is not applied to Auto generated assertions #1295

Open
newuniverse opened this issue Dec 14, 2021 · 5 comments · Fixed by #1454
Open

Override database config is not applied to Auto generated assertions #1295

newuniverse opened this issue Dec 14, 2021 · 5 comments · Fixed by #1454
Assignees
Labels

Comments

@newuniverse
Copy link

newuniverse commented Dec 14, 2021

About this https://docs.dataform.co/guides/assertions#auto-generated-assertions .

I am using dataform run --vars option to embed a new database(GCP project_id) into each sqlx files to
override the default_database in my CI/CD.

Not from code base but from behavior base, I found that override is not applied to auto generated assertions.

Is it an expected behavior or just a bug?

For example:

dataform.json

{
    "warehouse": "bigquery",
    "defaultSchema": "dataform",
    "assertionSchema": "dataform_assertions",
    "defaultDatabase": "my-dev-project"
}

some_query.sqlx

config {
  type: "table",
  database: dataform.projectConfig.vars.embed_my_prd_database,
  assertions: {
    uniqueKey: ["user_id"],
    nonNull: ["user_id", "customer_id"],
    rowConditions: [
      'signup_date is null or signup_date > "2019-01-01"',
      'email like "%@%.%"'
    ]
  }
}
select ...

If you run dataform run --actions=some_query --vars=embed_my_prd_database=my-prd-project

expect: success

but I got followings because I was using SA credential of my-prd-project not of my-dev-project

Assertion failed:  dataform_assertions.some_query_assertions_uniqueKey_0
  > 
  >       create or replace view `my-dev-project.dataform_assertions.some_query_assertions_uniqueKey_0` as
@Ekrekr
Copy link
Contributor

Ekrekr commented Feb 10, 2023

A more elegant solution that we have planned for this is CLI flags for dataform.json parameters.

@newuniverse
Copy link
Author

I'm looking forward to it!

@Ekrekr
Copy link
Contributor

Ekrekr commented Dec 4, 2023

Config option was not the fix, #1602 demonstrates however confirms that this was not an issue with Core's dataform.json interpretation.

Will create a new PR to add these to the tests, and fix the underlying cause.

@Ekrekr
Copy link
Contributor

Ekrekr commented Dec 4, 2023

I'm a bit stumped by this, but I've narrowed down the issue and it is clear it only happens in the CLI.

Adding more tests for Dataform Core (#1603) demonstrates that:

  • Variables are correctly loaded from the project config override.
  • Variables are correctly loaded as the database for tables.
  • Variables are correctly loaded as the database for auto assertions.

Adding more tests for Dataform CLI (#1604) demonstrates that:

  • Variables are correctly loaded from the project config override.
  • Variables are correctly loaded as the database for tables.
  • However for auto assertions, the variable is not applied.

@btelFD
Copy link

btelFD commented Apr 17, 2024

Also running into this. We set the database in the config block of each .sqlx and assertions in the config block don't work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants