-
Notifications
You must be signed in to change notification settings - Fork 189
Description
Describe the bug
Running dbt build or run with dbt-databricks 1.11.6 fails if two or more foreign_key constraints for the same model relation are defined in the YAML of an incremental model. Running an incremental run or the first run raises:
> Database Error in model incremental_model (models/incremental_model.sql)
> [DELTA_CONSTRAINT_ALREADY_EXISTS] Constraint 'incremental_model_relation_model_fk' already
exists. Please delete the old constraint first.
Old constraint:
incremental_model_relation_model_fk FOREIGN KEY (`column_1`) REFERENCES `catalog`.`schema`.`relation_model` (`column_pk`)
After adding names to the foreign constraints, the error is not raised anymore, but this condition was not necessary in previous versions of dbt-databricks (previously using 1.10.9).
This only happens for incremental models.
Steps To Reproduce
models.yml
models:
- name: incremental_model
config:
contract:
enforced: true
meta:
owner: '@owner'
columns:
- name: column_1
data_type: string
description: Foreign Key.
data_tests:
- relationships:
arguments:
to: ref('relation_model')
field: column_pk
constraints:
- type: foreign_key
to: ref('relation_model')
to_columns: [column_pk]
- name: column_2
data_type: string
description: Foreign Key.
data_tests:
- relationships:
arguments:
to: ref('relation_model')
field: column_pk
constraints:
- type: foreign_key
to: ref('relation_model')
to_columns: [column_pk]
- name: relation_model
config:
contract:
enforced: true
meta:
owner: '@owner'
columns:
- name: column_pk
data_type: string
data_tests:
- not_null
- unique
constraints:
- type: primary_key
- type: not_null
incremental_model.sql:
{{
config(
materialized="incremental",
on_schema_change="fail",
unique_key="accounting_category_id",
)
}}
with
import_cte as (
select
-- Primary key
incremental_pk,
-- Foreign keys
column_1,
column_2,
-- Timestamp fields
updated_at
from {{ ref('stg_model') }}
{% if is_incremental() %}
where updated_at >= (select max(updated_at) from {{ this }})
{% endif %}
)
select * from import_cte
The target/run incremental_model.sql file that causes the error:
ALTER TABLE `catalog`.`schema`.`incremental_model` ADD FOREIGN KEY (column_2) REFERENCES `catalog`.`schema`.`relation_model` (column_pk);
Expected behavior
Both foreign keys are correctly added to the incremental model even when no name is specified, as it happens for non incremental models.
Screenshots and log output
Terminal output:
> Database Error in model incremental_model (models/incremental_model.sql)
> [DELTA_CONSTRAINT_ALREADY_EXISTS] Constraint 'incremental_model_relation_model_fk' already
exists. Please delete the old constraint first.
Old constraint:
incremental_model_relation_model_fk FOREIGN KEY (`column_1`) REFERENCES `catalog`.`schema`.`relation_model` (`column_pk`)
System information
The output of dbt --version:
Core:
- installed: 1.11.6
- latest: 1.11.7 - Update available!
Your version of dbt-core is out of date!
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation
Plugins:
- databricks: 1.11.5 - Up to date!
- spark: 1.10.1 - Up to date!
The operating system you're using:
MacOS Tahoe 26.3
The output of python --version:
Python 3.10.11
Additional context
This problem first happens after upgrading from
dbt-databricks 1.10.9 . Running tests in local before upgrading in production.