Skip to content

1.11.5 fails with DELTA_CONSTRAINT_ALREADY_EXISTS in incremental models with two or more foreign keys to same relation #1344

@marta-fernandez-mews

Description

@marta-fernandez-mews

Describe the bug

Running dbt build or run with dbt-databricks 1.11.6 fails if two or more foreign_key constraints for the same model relation are defined in the YAML of an incremental model. Running an incremental run or the first run raises:

> Database Error in model incremental_model (models/incremental_model.sql)
>   [DELTA_CONSTRAINT_ALREADY_EXISTS] Constraint 'incremental_model_relation_model_fk' already
 exists. Please delete the old constraint first.
  Old constraint:
  incremental_model_relation_model_fk FOREIGN KEY (`column_1`) REFERENCES `catalog`.`schema`.`relation_model` (`column_pk`)

After adding names to the foreign constraints, the error is not raised anymore, but this condition was not necessary in previous versions of dbt-databricks (previously using 1.10.9).
This only happens for incremental models.

Steps To Reproduce

models.yml

models:
    - name: incremental_model
        config:
          contract:
            enforced: true
          meta:
            owner: '@owner'
        columns:
          - name: column_1
                  data_type: string
                  description: Foreign Key.
                  data_tests:
                    - relationships:
                        arguments:
                          to: ref('relation_model')
                          field: column_pk
                  constraints:
                    - type: foreign_key
                      to: ref('relation_model')
                      to_columns: [column_pk]
          
          - name: column_2
                  data_type: string
                  description: Foreign Key.
                  data_tests:
                    - relationships:
                        arguments:
                          to: ref('relation_model')
                          field: column_pk
                  constraints:
                    - type: foreign_key
                      to: ref('relation_model')
                      to_columns: [column_pk]
    - name: relation_model
      config:
        contract:
          enforced: true
        meta:
          owner: '@owner'
      columns:
        - name: column_pk
          data_type: string
          data_tests:
            - not_null
            - unique
          constraints:
            - type: primary_key
            - type: not_null

incremental_model.sql:

{{
    config(
        materialized="incremental",
        on_schema_change="fail",
        unique_key="accounting_category_id",
    )
}}
with
import_cte as (
    select
        -- Primary key
        incremental_pk,

        -- Foreign keys
        column_1,
        column_2,

        -- Timestamp fields
        updated_at

        from {{ ref('stg_model') }}
    {% if is_incremental() %}
    where updated_at >= (select max(updated_at) from {{ this }})
    {% endif %}
)

select * from import_cte

The target/run incremental_model.sql file that causes the error:

ALTER TABLE `catalog`.`schema`.`incremental_model` ADD FOREIGN KEY (column_2) REFERENCES `catalog`.`schema`.`relation_model` (column_pk);

Expected behavior

Both foreign keys are correctly added to the incremental model even when no name is specified, as it happens for non incremental models.

Screenshots and log output

Terminal output:

> Database Error in model incremental_model (models/incremental_model.sql)
>   [DELTA_CONSTRAINT_ALREADY_EXISTS] Constraint 'incremental_model_relation_model_fk' already
 exists. Please delete the old constraint first.
  Old constraint:
  incremental_model_relation_model_fk FOREIGN KEY (`column_1`) REFERENCES `catalog`.`schema`.`relation_model` (`column_pk`)

System information

The output of dbt --version:

Core:
  - installed: 1.11.6
  - latest:    1.11.7 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - databricks: 1.11.5 - Up to date!
  - spark:      1.10.1 - Up to date!

The operating system you're using:
MacOS Tahoe 26.3

The output of python --version:
Python 3.10.11

Additional context

This problem first happens after upgrading from
dbt-databricks 1.10.9 . Running tests in local before upgrading in production.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions