Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

insert_overwrite (insert+replace) partitions incremental strategy #201

Merged
merged 10 commits into from
Aug 1, 2024
Merged

insert_overwrite (insert+replace) partitions incremental strategy #201

merged 10 commits into from
Aug 1, 2024
+136 −16

Commits on Jul 15, 2024

  1. dbt/include/clickhouse/macros/materializations/incremental.sql: imple… 

    …ment insert+replace strategy

TODO: convert partition_expression to ClickHouse literals
    @bryzgaloff
    bryzgaloff committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    ae4ca48 View commit details
    Browse the repository at this point in the history

  2. dbt/include/clickhouse/macros/materializations/incremental.sql: check… 

    … nonempty partition_by
    @bryzgaloff
    bryzgaloff committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    b6cbca9 View commit details
    Browse the repository at this point in the history

  3. dbt/include/clickhouse/macros/materializations/incremental.sql: selec… 

    …ting affected partitions from system.parts

This allows to avoid translation of agate.Row to ClickHouse literals.
    @bryzgaloff
    bryzgaloff committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    5c44c60 View commit details
    Browse the repository at this point in the history

  4. dbt/include/clickhouse/macros/materializations/incremental.sql: syste… 

    …m.parts.partition_id -> partition

partition_id is a String field with internal partition ID, it cannot be used in REPLACE PARTITION clause. "partition" field is a string representation of partition expression and can be used in a REPLACE PARTITION query as-is.
    @bryzgaloff
    bryzgaloff committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    06cafb1 View commit details
    Browse the repository at this point in the history

  5. dbt/include/clickhouse/macros/materializations/incremental.sql: ident… 

    …ify partition by ID

system.parts.partition field does not work for strings. ClickHouse allows to manipulate partitions referencing their IDs.
    @bryzgaloff
    bryzgaloff committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    add2ff8 View commit details
    Browse the repository at this point in the history

  6. README.md: inserts_only documented as overwriting incremental_strategy

    @bryzgaloff
    bryzgaloff committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    b966767 View commit details
    Browse the repository at this point in the history

  7. README.md: insert+replace strategy described

    @bryzgaloff
    bryzgaloff committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    6869ed2 View commit details
    Browse the repository at this point in the history

  8. tests/integration/adapter/incremental/test_incremental.py: covered in… 

    …sert+replace with tests
    @bryzgaloff
    bryzgaloff committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    e4bd2cc View commit details
    Browse the repository at this point in the history

  9. Renamed insert+replace strategy to insert_overwrite 

    According to a PR review comments
    @bryzgaloff
    bryzgaloff committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    616ea7a View commit details
    Browse the repository at this point in the history

  10. tests/integration/adapter/incremental/test_base_incremental.py: linte… 

    …r fix applied
    @bryzgaloff
    bryzgaloff committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    6383bef View commit details
    Browse the repository at this point in the history