diff --git a/CHANGELOG.md b/CHANGELOG.md index ebf6bfea..44cb880d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ - Fix `get_query_results_as_dict` integration test with consistent ordering ([#322](https://github.com/fishtown-analytics/dbt-utils/pull/322)) - All macros are now properly dispatched, making it possible for non-core adapters to implement a shim package for dbt-utils ([#312](https://github.com/fishtown-analytics/dbt-utils/pull/312)) Thanks [@chaerinlee1](https://github.com/chaerinlee1) and [@swanderz](https://github.com/swanderz) - Small, non-breaking changes to accomdate TSQL (can't group by column number references, no real TRUE/FALSE values, aggregation CTEs need named columns) ([#310](https://github.com/fishtown-analytics/dbt-utils/pull/310)) Thanks [@swanderz](https://github.com/swanderz) +- Make `get_relations_by_pattern` and `get_relations_by_prefix` more powerful by returning `relation.type` # dbt-utils v0.6.3 diff --git a/README.md b/README.md index 4ce32043..f5d96b34 100644 --- a/README.md +++ b/README.md @@ -453,56 +453,81 @@ Usage: ... ``` -#### get_relations_by_prefix + + +#### get_relations_by_pattern ([source](macros/sql/get_relations_by_pattern.sql)) + Returns a list of [Relations](https://docs.getdbt.com/docs/writing-code-in-dbt/class-reference/#relation) -that match a given prefix, with an optional exclusion pattern. It's particularly -handy paired with `union_relations`. +that match a given schema- or table-name pattern. + +This macro is particularly handy when paired with `union_relations`. + **Usage:** ``` --- Returns a list of relations that match schema.prefix% -{% set relations = dbt_utils.get_relations_by_prefix('my_schema', 'my_prefix') %} +-- Returns a list of relations that match schema_pattern%.table +{% set relations = dbt_utils.get_relations_by_pattern('schema_pattern%', 'table_pattern') %} + +-- Returns a list of relations that match schema_pattern.table_pattern% +{% set relations = dbt_utils.get_relations_by_pattern('schema_pattern', 'table_pattern%') %} -- Returns a list of relations as above, excluding any that end in `deprecated` -{% set relations = dbt_utils.get_relations_by_prefix('my_schema', 'my_prefix', '%deprecated') %} +{% set relations = dbt_utils.get_relations_by_pattern('schema_pattern', 'table_pattern%', '%deprecated') %} -- Example using the union_relations macro -{% set event_relations = dbt_utils.get_relations_by_prefix('events', 'event_') %} +{% set event_relations = dbt_utils.get_relations_by_pattern('venue%', 'clicks') %} {{ dbt_utils.union_relations(relations = event_relations) }} ``` **Args:** -* `schema` (required): The schema to inspect for relations. -* `prefix` (required): The prefix of the table/view (case insensitive) -* `exclude` (optional): Exclude any relations that match this pattern. +* `schema_pattern` (required): The schema pattern to inspect for relations. +* `table_pattern` (required): The name of the table/view (case insensitive). +* `exclude` (optional): Exclude any relations that match this table pattern. * `database` (optional, default = `target.database`): The database to inspect for relations. -#### get_relations_by_pattern -> This was built from the get_relations_by_prefix macro. +**Examples:** +Generate drop statements for all Relations that match a naming pattern: +```sql +{% set relations_to_drop = dbt_utils.get_relations_by_pattern( + schema_pattern='public', + table_pattern='dbt\_%' +) %} +{% set sql_to_execute = [] %} + +{{ log('Statements to run:', info=True) }} + +{% for relation in relations_to_drop %} + {% set drop_command -%} + -- drop {{ relation.type }} {{ relation }} cascade; + {%- endset %} + {% do log(drop_command, info=True) %} + {% do sql_to_execute.append(drop_command) %} +{% endfor %} +``` + +#### get_relations_by_prefix ([source](macros/sql/get_relations_by_prefix.sql)) +> This macro will soon be deprecated in favor of the more flexible `get_relations_by_pattern` macro (above) Returns a list of [Relations](https://docs.getdbt.com/docs/writing-code-in-dbt/class-reference/#relation) -that match a given schema or table pattern and table/view name with an optional exclusion pattern. Like its cousin -get_relations_by_prefix, it's particularly handy paired with `union_relations`. +that match a given prefix, with an optional exclusion pattern. It's particularly +handy paired with `union_relations`. **Usage:** ``` --- Returns a list of relations that match schema%.table -{% set relations = dbt_utils.get_relations_by_pattern('schema_pattern%', 'table_pattern') %} - --- Returns a list of relations that match schema.table% -{% set relations = dbt_utils.get_relations_by_pattern('schema_pattern', 'table_pattern%') %} +-- Returns a list of relations that match schema.prefix% +{% set relations = dbt_utils.get_relations_by_prefix('my_schema', 'my_prefix') %} -- Returns a list of relations as above, excluding any that end in `deprecated` -{% set relations = dbt_utils.get_relations_by_pattern('schema_pattern', 'table_pattern%', '%deprecated') %} +{% set relations = dbt_utils.get_relations_by_prefix('my_schema', 'my_prefix', '%deprecated') %} -- Example using the union_relations macro -{% set event_relations = dbt_utils.get_relations_by_pattern('venue%', 'clicks') %} +{% set event_relations = dbt_utils.get_relations_by_prefix('events', 'event_') %} {{ dbt_utils.union_relations(relations = event_relations) }} ``` **Args:** -* `schema_pattern` (required): The schema pattern to inspect for relations. -* `table_pattern` (required): The name of the table/view (case insensitive). -* `exclude` (optional): Exclude any relations that match this table pattern. +* `schema` (required): The schema to inspect for relations. +* `prefix` (required): The prefix of the table/view (case insensitive) +* `exclude` (optional): Exclude any relations that match this pattern. * `database` (optional, default = `target.database`): The database to inspect for relations. diff --git a/macros/sql/get_relations_by_pattern.sql b/macros/sql/get_relations_by_pattern.sql index 59a5ea08..eb365ac9 100644 --- a/macros/sql/get_relations_by_pattern.sql +++ b/macros/sql/get_relations_by_pattern.sql @@ -15,7 +15,12 @@ {%- if table_list and table_list['table'] -%} {%- set tbl_relations = [] -%} {%- for row in table_list['table'] -%} - {%- set tbl_relation = api.Relation.create(database, row.table_schema, row.table_name) -%} + {%- set tbl_relation = api.Relation.create( + database=database, + schema=row.table_schema, + identifier=row.table_name, + type=row.table_type + ) -%} {%- do tbl_relations.append(tbl_relation) -%} {%- endfor -%} diff --git a/macros/sql/get_relations_by_prefix.sql b/macros/sql/get_relations_by_prefix.sql index 1fde612d..c4e7146f 100644 --- a/macros/sql/get_relations_by_prefix.sql +++ b/macros/sql/get_relations_by_prefix.sql @@ -15,7 +15,12 @@ {%- if table_list and table_list['table'] -%} {%- set tbl_relations = [] -%} {%- for row in table_list['table'] -%} - {%- set tbl_relation = api.Relation.create(database, row.table_schema, row.table_name) -%} + {%- set tbl_relation = api.Relation.create( + database=database, + schema=row.table_schema, + identifier=row.table_name, + type=row.table_type + ) -%} {%- do tbl_relations.append(tbl_relation) -%} {%- endfor -%} diff --git a/macros/sql/get_tables_by_pattern_sql.sql b/macros/sql/get_tables_by_pattern_sql.sql index 3000d001..3970aa38 100644 --- a/macros/sql/get_tables_by_pattern_sql.sql +++ b/macros/sql/get_tables_by_pattern_sql.sql @@ -6,8 +6,13 @@ {% macro default__get_tables_by_pattern_sql(schema_pattern, table_pattern, exclude='', database=target.database) %} select distinct - table_schema as "table_schema", table_name as "table_name" - from {{database}}.information_schema.tables + table_schema as "table_schema", + table_name as "table_name", + case table_type + when 'BASE TABLE' then 'table' + else lower(table_type) + end as "table_type" + from {{ database }}.information_schema.tables where table_schema ilike '{{ schema_pattern }}' and table_name ilike '{{ table_pattern }}' and table_name not ilike '{{ exclude }}' @@ -26,7 +31,12 @@ {% set sql %} {% for schema in schemata %} select distinct - table_schema, table_name + table_schema, + table_name, + case table_type + when 'BASE TABLE' then 'table' + else lower(table_type) + end as table_type from {{ adapter.quote(database) }}.{{ schema }}.INFORMATION_SCHEMA.TABLES where lower(table_name) like lower ('{{ table_pattern }}')