Merge pull request #323 from fishtown-analytics/add-relation-type

Add relation type to get_relations_by_pattern and get_relations_by_prefix

CHANGELOG.md

@@ -9,6 +9,7 @@
 - Fix `get_query_results_as_dict` integration test with consistent ordering ([#322](https://github.com/fishtown-analytics/dbt-utils/pull/322))
 - All macros are now properly dispatched, making it possible for non-core adapters to implement a shim package for dbt-utils ([#312](https://github.com/fishtown-analytics/dbt-utils/pull/312)) Thanks [@chaerinlee1](https://github.com/chaerinlee1) and [@swanderz](https://github.com/swanderz)
 - Small, non-breaking changes to accomdate TSQL (can't group by column number references, no real TRUE/FALSE values, aggregation CTEs need named columns) ([#310](https://github.com/fishtown-analytics/dbt-utils/pull/310)) Thanks [@swanderz](https://github.com/swanderz)
+- Make `get_relations_by_pattern` and `get_relations_by_prefix` more powerful by returning `relation.type`
 
 # dbt-utils v0.6.3
 

README.md

@@ -453,56 +453,81 @@ Usage:
 
 ...
 ```
-#### get_relations_by_prefix
+
+
+#### get_relations_by_pattern ([source](macros/sql/get_relations_by_pattern.sql))
+
 Returns a list of [Relations](https://docs.getdbt.com/docs/writing-code-in-dbt/class-reference/#relation)
-that match a given prefix, with an optional exclusion pattern. It's particularly
-handy paired with `union_relations`.
+that match a given schema- or table-name pattern.
+
+This macro is particularly handy when paired with `union_relations`.
+
 **Usage:**
 ```
--- Returns a list of relations that match schema.prefix%
-{% set relations = dbt_utils.get_relations_by_prefix('my_schema', 'my_prefix') %}
+-- Returns a list of relations that match schema_pattern%.table
+{% set relations = dbt_utils.get_relations_by_pattern('schema_pattern%', 'table_pattern') %}
+
+-- Returns a list of relations that match schema_pattern.table_pattern%
+{% set relations = dbt_utils.get_relations_by_pattern('schema_pattern', 'table_pattern%') %}
 
 -- Returns a list of relations as above, excluding any that end in `deprecated`
-{% set relations = dbt_utils.get_relations_by_prefix('my_schema', 'my_prefix', '%deprecated') %}
+{% set relations = dbt_utils.get_relations_by_pattern('schema_pattern', 'table_pattern%', '%deprecated') %}
 
 -- Example using the union_relations macro
-{% set event_relations = dbt_utils.get_relations_by_prefix('events', 'event_') %}
+{% set event_relations = dbt_utils.get_relations_by_pattern('venue%', 'clicks') %}
 {{ dbt_utils.union_relations(relations = event_relations) }}
 ```
 
 **Args:**
-* `schema` (required): The schema to inspect for relations.
-* `prefix` (required): The prefix of the table/view (case insensitive)
-* `exclude` (optional): Exclude any relations that match this pattern.
+* `schema_pattern` (required): The schema pattern to inspect for relations.
+* `table_pattern` (required): The name of the table/view (case insensitive).
+* `exclude` (optional): Exclude any relations that match this table pattern.
 * `database` (optional, default = `target.database`): The database to inspect
 for relations.
 
-#### get_relations_by_pattern
-> This was built from the get_relations_by_prefix macro.
+**Examples:**
+Generate drop statements for all Relations that match a naming pattern:
+```sql
+{% set relations_to_drop = dbt_utils.get_relations_by_pattern(
+    schema_pattern='public',
+    table_pattern='dbt\_%'
+) %}
 
+{% set sql_to_execute = [] %}
+
+{{ log('Statements to run:', info=True) }}
+
+{% for relation in relations_to_drop %}
+    {% set drop_command -%}
+    -- drop {{ relation.type }} {{ relation }} cascade;
+    {%- endset %}
+    {% do log(drop_command, info=True) %}
+    {% do sql_to_execute.append(drop_command) %}
+{% endfor %}
+```
+
+#### get_relations_by_prefix ([source](macros/sql/get_relations_by_prefix.sql))
+> This macro will soon be deprecated in favor of the more flexible `get_relations_by_pattern` macro (above)
 Returns a list of [Relations](https://docs.getdbt.com/docs/writing-code-in-dbt/class-reference/#relation)
-that match a given schema or table pattern and table/view name with an optional exclusion pattern. Like its cousin
-get_relations_by_prefix, it's particularly handy paired with `union_relations`.
+that match a given prefix, with an optional exclusion pattern. It's particularly
+handy paired with `union_relations`.
 **Usage:**
 ```
--- Returns a list of relations that match schema%.table
-{% set relations = dbt_utils.get_relations_by_pattern('schema_pattern%', 'table_pattern') %}
-
--- Returns a list of relations that match schema.table%
-{% set relations = dbt_utils.get_relations_by_pattern('schema_pattern', 'table_pattern%') %}
+-- Returns a list of relations that match schema.prefix%
+{% set relations = dbt_utils.get_relations_by_prefix('my_schema', 'my_prefix') %}
 
 -- Returns a list of relations as above, excluding any that end in `deprecated`
-{% set relations = dbt_utils.get_relations_by_pattern('schema_pattern', 'table_pattern%', '%deprecated') %}
+{% set relations = dbt_utils.get_relations_by_prefix('my_schema', 'my_prefix', '%deprecated') %}
 
 -- Example using the union_relations macro
-{% set event_relations = dbt_utils.get_relations_by_pattern('venue%', 'clicks') %}
+{% set event_relations = dbt_utils.get_relations_by_prefix('events', 'event_') %}
 {{ dbt_utils.union_relations(relations = event_relations) }}
 ```
 
 **Args:**
-* `schema_pattern` (required): The schema pattern to inspect for relations.
-* `table_pattern` (required): The name of the table/view (case insensitive).
-* `exclude` (optional): Exclude any relations that match this table pattern.
+* `schema` (required): The schema to inspect for relations.
+* `prefix` (required): The prefix of the table/view (case insensitive)
+* `exclude` (optional): Exclude any relations that match this pattern.
 * `database` (optional, default = `target.database`): The database to inspect
 for relations.
 

macros/sql/get_relations_by_pattern.sql

@@ -15,7 +15,12 @@
     {%- if table_list and table_list['table'] -%}
         {%- set tbl_relations = [] -%}
         {%- for row in table_list['table'] -%}
-            {%- set tbl_relation = api.Relation.create(database, row.table_schema, row.table_name) -%}
+            {%- set tbl_relation = api.Relation.create(
+                database=database,
+                schema=row.table_schema,
+                identifier=row.table_name,
+                type=row.table_type
+            ) -%}
             {%- do tbl_relations.append(tbl_relation) -%}
         {%- endfor -%}
 

macros/sql/get_relations_by_prefix.sql

@@ -15,7 +15,12 @@
     {%- if table_list and table_list['table'] -%}
         {%- set tbl_relations = [] -%}
         {%- for row in table_list['table'] -%}
-            {%- set tbl_relation = api.Relation.create(database, row.table_schema, row.table_name) -%}
+            {%- set tbl_relation = api.Relation.create(
+                database=database,
+                schema=row.table_schema,
+                identifier=row.table_name,
+                type=row.table_type
+            ) -%}
             {%- do tbl_relations.append(tbl_relation) -%}
         {%- endfor -%}
 

macros/sql/get_tables_by_pattern_sql.sql

@@ -6,8 +6,13 @@
 {% macro default__get_tables_by_pattern_sql(schema_pattern, table_pattern, exclude='', database=target.database) %}
 
         select distinct
-            table_schema as "table_schema", table_name as "table_name"
-        from {{database}}.information_schema.tables
+            table_schema as "table_schema",
+            table_name as "table_name",
+            case table_type
+                when 'BASE TABLE' then 'table'
+                else lower(table_type)
+            end as "table_type"
+        from {{ database }}.information_schema.tables
         where table_schema ilike '{{ schema_pattern }}'
         and table_name ilike '{{ table_pattern }}'
         and table_name not ilike '{{ exclude }}'
@@ -26,7 +31,12 @@
     {% set sql %}
         {% for schema in schemata %}
             select distinct
-                table_schema, table_name
+                table_schema,
+                table_name,
+                case table_type
+                    when 'BASE TABLE' then 'table'
+                    else lower(table_type)
+                end as table_type
 
             from {{ adapter.quote(database) }}.{{ schema }}.INFORMATION_SCHEMA.TABLES
             where lower(table_name) like lower ('{{ table_pattern }}')