Consolidate source + remove tests (#211)

* Consolidate dbt source into transform

* Test/empty actor (#206)

* Update packages.yml

* Update ticket_chat_event_data.csv

* pr updates

* tmp model fix

* tmp fixes and readme change

* Generate dbt docs via GitHub Actions

* Update dbt_project.yml

* Generate dbt docs via GitHub Actions

* Update CHANGELOG.md

Co-authored-by: fivetran-catfritz <[email protected]>

---------

Co-authored-by: fivetran-catfritz <[email protected]>
Co-authored-by: Joe Markiewicz <[email protected]>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Author: 4 people

CHANGELOG.md

@@ -1,3 +1,23 @@
+# dbt_zendesk v1.0.0
+
+[PR #211](https://github.com/fivetran/dbt_zendesk/pull/211) includes the following updates:
+
+## Breaking Changes
+
+### Source Package Consolidation
+- Removed the dependency on the `fivetran/zendesk_source` package.
+  - All functionality from the source package has been merged into this transformation package for improved maintainability and clarity.
+  - If you reference `fivetran/zendesk_source` in your `packages.yml`, you must remove this dependency to avoid conflicts.
+  - Any source overrides referencing the `fivetran/zendesk_source` package will also need to be removed or updated to reference this package.
+  - Update any zendesk_source-scoped variables to be scoped to only under this package. See the [README](https://github.com/fivetran/dbt_zendesk/blob/main/README.md) for how to configure the build schema of staging models.
+- As part of the consolidation, vars are no longer used to reference staging models, and only sources are represented by vars. Staging models are now referenced directly with `ref()` in downstream models.
+
+### dbt Fusion Compatibility Updates
+- Updated package to maintain compatibility with dbt-core versions both before and after v1.10.6, which introduced a breaking change to multi-argument test syntax (e.g., `unique_combination_of_columns`).
+- Temporarily removed unsupported tests to avoid errors and ensure smoother upgrades across different dbt-core versions. These tests will be reintroduced once a safe migration path is available.
+  - Removed all `dbt_utils.unique_combination_of_columns` tests.
+  - Moved `loaded_at_field: _fivetran_synced` under the `config:` block in `src_zendesk.yml`.
+
 # dbt_zendesk v0.25.0
 
 ## Schema/Data Changes

README.md

@@ -1,4 +1,4 @@
-# Zendesk Support Modeling dbt Package ([Docs](https://fivetran.github.io/dbt_zendesk/))
+# Zendesk Support dbt Package ([Docs](https://fivetran.github.io/dbt_zendesk/))
 
 <p align="left">
     <a alt="License"
@@ -16,7 +16,7 @@
 </p>
 
 ## What does this dbt package do?
-- Produces modeled tables that leverage Zendesk Support data from [Fivetran's connector](https://fivetran.com/docs/applications/zendesk) in the format described by [this ERD](https://fivetran.com/docs/applications/zendesk#schemainformation) and build off the output of our [zendesk source package](https://github.com/fivetran/dbt_zendesk_source).
+- Produces modeled tables that leverage Zendesk Support data from [Fivetran's connector](https://fivetran.com/docs/applications/zendesk) in the format described by [this ERD](https://fivetran.com/docs/applications/zendesk#schemainformation).
 - Enables you to better understand the performance of your Support team. It calculates metrics focused on response times, resolution times, and work times for you to analyze. It performs the following actions:
   - Creates an enriched ticket model with relevant resolution, response time, and other metrics
   - Produces a historical ticket field history model to see velocity of your tickets over time
@@ -68,9 +68,9 @@ Include the following zendesk package version in your `packages.yml` file:
 ```yml
 packages:
   - package: fivetran/zendesk
-    version: [">=0.25.0", "<0.26.0"]
+    version: [">=1.0.0", "<1.1.0"]
 ```
-> **Note**: Do not include the Zendesk Support source package. The Zendesk Support transform package already has a dependency on the source in its own `packages.yml` file.
+> All required sources and staging models are now bundled into this transformation package. Do not include `fivetran/zendesk_source` in your `packages.yml` since this package has been deprecated.
 
 ### Step 3: Define database and schema variables
 #### Option A: Single connection
@@ -93,7 +93,7 @@ To use this functionality, you will need to set the `zendesk_sources` variable i
 # dbt_project.yml
 
 vars:
-  zendesk_sources:
+  zendesk:
     - database: connection_1_destination_name # Required
       schema: connection_1_schema_name # Rquired
       name: connection_1_source_name # Required only if following the step in the following subsection
@@ -109,7 +109,7 @@ vars:
 By default, this package defines one single-connection source, called `zendesk`, which will be disabled if you are unioning multiple connections. This means that your DAG will not include your Zendesk sources, though the package will run successfully.
 
 To properly incorporate all of your Zendesk connections into your project's DAG:
-1. Define each of your sources in a `.yml` file in your project. Utilize the following template for the `source`-level configurations, and, **most importantly**, copy and paste the table and column-level definitions from the package's `src_zendesk.yml` [file](https://github.com/fivetran/dbt_zendesk_source/blob/main/models/src_zendesk.yml#L15-L351).
+1. Define each of your sources in a `.yml` file in your project. Utilize the following template for the `source`-level configurations, and, **most importantly**, copy and paste the table and column-level definitions from the package's `src_zendesk.yml` [file](https://github.com/fivetran/dbt_zendesk/blob/main/models/staging/src_zendesk.yml#L15-L351).
 
 ```yml
 # a .yml file in your root project
@@ -124,16 +124,16 @@ sources:
       warn_after: {count: 72, period: hour}
       error_after: {count: 168, period: hour}
 
-    tables: # copy and paste from zendesk_source/models/src_zendesk.yml - see https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/ for how to use anchors to only do so once
+    tables: # copy and paste from zendesk/models/staging/src_zendesk.yml - see https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/ for how to use anchors to only do so once
 ```
 
-> **Note**: If there are source tables you do not have (see [Step 4](https://github.com/fivetran/dbt_zendesk_source?tab=readme-ov-file#step-4-disable-models-for-non-existent-sources)), you may still include them, as long as you have set the right variables to `False`. Otherwise, you may remove them from your source definition.
+> **Note**: If there are source tables you do not have (see [Step 4](https://github.com/fivetran/dbt_zendesk?tab=readme-ov-file#step-4-disable-models-for-non-existent-sources)), you may still include them, as long as you have set the right variables to `False`. Otherwise, you may remove them from your source definition.
 
-2. Set the `has_defined_sources` variable (scoped to the `zendesk_source` package) to `True`, like such:
+2. Set the `has_defined_sources` variable (scoped to the `zendesk` package) to `True`, like such:
 ```yml
 # dbt_project.yml
 vars:
-  zendesk_source:
+  zendesk:
     has_defined_sources: true
 ```
 
@@ -221,7 +221,7 @@ Example usage:
 ```yml
 # dbt_project.yml
 vars:
-  zendesk_source:
+  zendesk:
     internal_user_criteria: "lower(email) like '%@fivetran.com' or external_id = '12345' or name in ('Garrett', 'Alfredo')" # can reference any non-custom field in USER
 ```
 
@@ -265,22 +265,18 @@ By default this package will build the Zendesk Support staging models within a s
 
 ```yml
 models:
-  zendesk:
-    +schema: my_new_schema_name # leave blank for just the target_schema
-    intermediate:
-      +schema: my_new_schema_name # leave blank for just the target_schema
-    sla_policy:
-      +schema: my_new_schema_name # leave blank for just the target_schema
-    ticket_history:
-      +schema: my_new_schema_name # leave blank for just the target_schema
-  zendesk_source:
-    +schema: my_new_schema_name # leave blank for just the target_schema
+    zendesk:
+      +schema: my_new_schema_name # Leave +schema: blank to use the default target_schema.
+      intermediate:
+        +schema: my_new_schema_name # Leave +schema: blank to use the default target_schema.
+      staging:
+        +schema: my_new_schema_name # Leave +schema: blank to use the default target_schema.
 ```
 
 #### Change the source table references
 If an individual source table has a different name than the package expects, add the table name as it appears in your destination to the respective variable:
 
-> IMPORTANT: See this project's [`dbt_project.yml`](https://github.com/fivetran/dbt_zendesk_source/blob/main/dbt_project.yml) variable declarations to see the expected names.
+> IMPORTANT: See this project's [`dbt_project.yml`](https://github.com/fivetran/dbt_zendesk/blob/main/dbt_project.yml) variable declarations to see the expected names.
 
 ```yml
 vars:
@@ -302,9 +298,6 @@ This dbt package is dependent on the following dbt packages. These dependencies
 
 ```yml
 packages:
-    - package: fivetran/zendesk_source
-      version: [">=0.18.0", "<0.19.0"]
-
     - package: fivetran/fivetran_utils
       version: [">=0.4.0", "<0.5.0"]
 

dbt_project.yml

@@ -1,9 +1,10 @@
 name: 'zendesk'
-version: '0.25.0'
+version: '1.0.0'
 
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 on-run-start: '{{ fivetran_utils.empty_variable_warning("ticket_field_history_columns", "zendesk_ticket_field_history") }}'
+
 models:
   zendesk:
     +schema: zendesk
@@ -35,30 +36,40 @@ models:
       +materialized: table
     utils:
       +materialized: ephemeral
+    staging:
+      +schema: zendesk_source
+      +materialized: table
+      tmp:
+        +materialized: view
+
 vars:
   zendesk:
     ticket_field_history_columns: ['assignee_id', 'status', 'priority']
     ticket_field_history_updater_columns: []
-    audit_log: "{{ ref('stg_zendesk__audit_log') }}"
-    brand: "{{ ref('stg_zendesk__brand') }}"
-    daylight_time: "{{ ref('stg_zendesk__daylight_time') }}"
-    domain_name: "{{ ref('stg_zendesk__domain_name') }}"
-    field_history: "{{ ref('stg_zendesk__ticket_field_history') }}"
-    group: "{{ ref('stg_zendesk__group') }}"
-    organization_tag: "{{ ref('stg_zendesk__organization_tag') }}"
-    organization: "{{ ref('stg_zendesk__organization') }}"
-    schedule_holiday: "{{ ref('stg_zendesk__schedule_holiday') }}"
-    schedule: "{{ ref('stg_zendesk__schedule') }}"
-    ticket_comment: "{{ ref('stg_zendesk__ticket_comment') }}"
-    ticket_form_history: "{{ ref('stg_zendesk__ticket_form_history') }}"
-    ticket_schedule: "{{ ref('stg_zendesk__ticket_schedule') }}"
-    ticket_tag: "{{ ref('stg_zendesk__ticket_tag') }}"
-    ticket: "{{ ref('stg_zendesk__ticket') }}"
-    time_zone: "{{ ref('stg_zendesk__time_zone') }}"
-    user_tag: "{{ ref('stg_zendesk__user_tag') }}"
-    user: "{{ ref('stg_zendesk__user') }}"
-    ticket_chat: "{{ ref('stg_zendesk__ticket_chat') }}"
-    ticket_chat_event: "{{ ref('stg_zendesk__ticket_chat_event') }}"
+    zendesk__ticket_passthrough_columns: []
+    zendesk__user_passthrough_columns: []
+    zendesk__organization_passthrough_columns: []
+    zendesk_sources: []
+    audit_log: "{{ source('zendesk', 'audit_log') }}"
+    brand: "{{ source('zendesk', 'brand') }}"
+    daylight_time: "{{ source('zendesk', 'daylight_time') }}"
+    domain_name: "{{ source('zendesk', 'domain_name') }}"
+    field_history: "{{ source('zendesk', 'ticket_field_history') }}"
+    group: "{{ source('zendesk', 'group') }}"
+    organization_tag: "{{ source('zendesk', 'organization_tag') }}"
+    organization: "{{ source('zendesk', 'organization') }}"
+    schedule_holiday: "{{ source('zendesk', 'schedule_holiday') }}"
+    schedule: "{{ source('zendesk', 'schedule') }}"
+    ticket_comment: "{{ source('zendesk', 'ticket_comment') }}"
+    ticket_form_history: "{{ source('zendesk', 'ticket_form_history') }}"
+    ticket_schedule: "{{ source('zendesk', 'ticket_schedule') }}"
+    ticket_tag: "{{ source('zendesk', 'ticket_tag') }}"
+    ticket: "{{ source('zendesk', 'ticket') }}"
+    time_zone: "{{ source('zendesk', 'time_zone') }}"
+    user_tag: "{{ source('zendesk', 'user_tag') }}"
+    user: "{{ source('zendesk', 'user') }}"
+    ticket_chat: "{{ source('zendesk', 'ticket_chat') }}"
+    ticket_chat_event: "{{ source('zendesk', 'ticket_chat_event') }}"
 
     using_schedules: true
     using_domain_names: true

docs/catalog.json

@@ -1,13 +1,13 @@
 config-version: 2
 
 name: 'zendesk_integration_tests'
-version: '0.25.0'
+version: '1.0.0'
 
 profile: 'integration_tests'
 
 vars:
   zendesk_schema: zendesk_integration_tests_63
-  zendesk_source:
+  zendesk:
     zendesk_organization_identifier:           "organization_data"
     zendesk_schedule_identifier:               "schedule_data"
     zendesk_schedule_holiday_identifier:       "schedule_holiday_data"

docs/manifest.json

@@ -7,5 +7,7 @@ chat_id,created_at,_Company_synced,actor_id,chat_index,external_message_id,filen
 4323.953441.T6OrgbvMIqPplK,2023-12-08 05:07:28.408000,2024-03-19 16:51:30.989000,403959218893,0,,,,No problem. Just share a few details and I'll connect you with someone from our team.,,,,"{""author"":{""avatar_url"":""url"",""display_name"":""Company Bot"",""type"":""BUSINESS""},""content"":{""text"":{""actions"":[],""text"":""Thanks. Let me connect you with a customer support agent.""}},""id"":""658d843188000ddefeeba3f2"",""metadata"":{""fields"":{}},""received"":""2023-12-28T14:20:33.333Z"",""source"":{""type"":""zd:answerBot""}}",,,,,,,ChatMessage,
 4323.953441.TzkBGrapJoA,2023-12-28 14:22:12.840000,2024-03-19 16:51:21.702000,403969141094,0,,,,No problem. Just share a few details and I'll connect you with someone from our team.,,,,"{""author"":{""avatar_url"":""url"",""display_name"":""Company Bot"",""type"":""BUSINESS""},""content"":{""form"":{""block_chat_input"":false,""fields"":[{""label"":""Name"",""name"":""dataCapture.systemField.requester.name"",""text"":{""max_size"":256,""min_size"":1,""text"":""Name""}},{""email"":{""email"":""email""},""label"":""Email"",""name"":""dataCapture.systemField.requester.email""}]}},""id"":""658d841e69c2dcac0bc5a"",""metadata"":{""fields"":{""formId"":{""string_value"":""DataCaptureForm""},""nextNodeId"":{""string_value"":""01HEYB2GDS9HSRDAV_01HJRCQ1Y5EDYFHKZ6""},""nodeId"":{""string_value"":""01HJRCQ1Y4MJEYB2GDS9HSRDAV_01HJRCQ1NZJ77B4""}}},""received"":""2023-12-28T14:20:14.563Z"",""source"":{""type"":""zd:answerBot""}}",,,,,,,ChatMessage,
 4323.953441.TzkArUd3CjKI,2023-12-28 14:20:35.032000,2024-03-19 16:51:23.267000,403959218893,0,,,,Hi there. Got a question? I'm here to help.,,,,"{""author"":{""avatar_url"":""url"",""display_name"":""Company Bot"",""type"":""BUSINESS""},""content"":{""text"":{""actions"":[],""text"":""No problem. Just share a few details and I'll connect you with someone from our team.""}},""id"":""658d8481d00e97935de863ab"",""metadata"":{""fields"":{}},""received"":""2023-12-28T14:21:53.345Z"",""source"":{""type"":""zd:answerBot""}}",,,,,,,ChatMessage,
-4323.953441.TzAcedetjkcKyq,2023-12-22 10:37:20.489000,2024-03-19 16:51:25.454000,403969141094,0,,,,,,,,"{""author"":{""avatar_url"":""url"",""display_name"":""Company Bot"",""type"":""BUSINESS""},""content"":{""text"":{""actions"":[],""text"":""No problem. Just share a few details and I'll connect you with someone from our team.""}},""id"":""65bce6a92a2fe21c"",""metadata"":{""fields"":{}},""received"":""2024-02-02T12:57:13.130Z"",""source"":{""type"":""zd:answerBot""}}",,,,,,,ChatMessage,
-4323.953441.TyDt23helloZt4E,2023-12-12 11:20:10.181000,2024-03-19 16:51:27.764000,403959218893,0,,,,TALK TO A HUMAN,,,,"{""author"":{""avatar_url"":""url"",""display_name"":""Company Bot"",""type"":""BUSINESS""},""content"":{""text"":{""actions"":[{""reply"":{""payload"":""goto_node=01HJRCSAC466_01HJRCSAC4FV78SSW""},""text"":""Talk to a human""}],""text"":""Hi there. Got a question? I'm here to help.""}},""id"":""65bce6a06765305e2a2fd629"",""metadata"":{""fields"":{}},""received"":""2024-02-02T12:57:04.171Z"",""source"":{""type"":""zd:answerBot""}}",,,,,,,ChatMessage,
+4323.953441.TzAcedetjkcKyq,2023-12-22 10:37:20.489000,2024-03-19 16:51:25.454000,__system,0,,,,,,,,"{""author"":{""avatar_url"":""url"",""display_name"":""Company Bot"",""type"":""BUSINESS""},""content"":{""text"":{""actions"":[],""text"":""No problem. Just share a few details and I'll connect you with someone from our team.""}},""id"":""65bce6a92a2fe21c"",""metadata"":{""fields"":{}},""received"":""2024-02-02T12:57:13.130Z"",""source"":{""type"":""zd:answerBot""}}",,,,,,,ChatMessage,
+4323.953441.TyDt23helloZt4E,2023-12-12 11:20:10.181000,2024-03-19 16:51:27.764000,__trigger,0,,,,TALK TO A HUMAN,,,,"{""author"":{""avatar_url"":""url"",""display_name"":""Company Bot"",""type"":""BUSINESS""},""content"":{""text"":{""actions"":[{""reply"":{""payload"":""goto_node=01HJRCSAC466_01HJRCSAC4FV78SSW""},""text"":""Talk to a human""}],""text"":""Hi there. Got a question? I'm here to help.""}},""id"":""65bce6a06765305e2a2fd629"",""metadata"":{""fields"":{}},""received"":""2024-02-02T12:57:04.171Z"",""source"":{""type"":""zd:answerBot""}}",,,,,,,ChatMessage,
+4323.953441.TzBlankActor,2023-12-31 11:00:00.000000,2024-03-19 16:51:25.454000,"",0,,,,Blank id example,,,,"{""author"":{""avatar_url"":""url"",""display_name"":""Company Bot"",""type"":""BUSINESS""},""content"":{""text"":{""actions"":[],""text"":""Example message""}},""id"":""dummy_blank_actor"",""metadata"":{""fields"":{}},""received"":""2024-02-02T12:57:13.130Z"",""source"":{""type"":""zd:answerBot""}}",,,,,,,ChatMessage,
+4323.953441.TzBlankActorTwo,2023-12-31 11:00:00.000000,2024-03-19 16:51:25.454000,agent:,0,,,,Blank id example two,,,,"{""author"":{""avatar_url"":""url"",""display_name"":""Company Bot"",""type"":""BUSINESS""},""content"":{""text"":{""actions"":[],""text"":""Example message""}},""id"":""dummy_blank_actor"",""metadata"":{""fields"":{}},""received"":""2024-02-02T12:57:13.130Z"",""source"":{""type"":""zd:answerBot""}}",,,,,,,ChatMessage,

integration_tests/dbt_project.yml

@@ -0,0 +1,17 @@
+{% macro get_audit_log_columns() %}
+
+{% set columns = [
+    {"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
+    {"name": "id", "datatype": dbt.type_int()},
+    {"name": "action", "datatype": dbt.type_string()},
+    {"name": "actor_id", "datatype": dbt.type_int()},
+    {"name": "change_description", "datatype": dbt.type_string()},
+    {"name": "created_at", "datatype": dbt.type_timestamp()},
+    {"name": "source_id", "datatype": dbt.type_int()},
+    {"name": "source_label", "datatype": dbt.type_string()},
+    {"name": "source_type", "datatype": dbt.type_string()}
+] %}
+
+{{ return(columns) }}
+
+{% endmacro %}

integration_tests/seeds/ticket_chat_event_data.csv

@@ -0,0 +1,29 @@
+{% macro get_brand_columns() %}
+
+{% set columns = [
+    {"name": "_fivetran_deleted", "datatype": "boolean"},
+    {"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
+    {"name": "active", "datatype": "boolean"},
+    {"name": "brand_url", "datatype": dbt.type_string()},
+    {"name": "has_help_center", "datatype": "boolean"},
+    {"name": "help_center_state", "datatype": dbt.type_string()},
+    {"name": "id", "datatype": dbt.type_int()},
+    {"name": "logo_content_type", "datatype": dbt.type_string()},
+    {"name": "logo_content_url", "datatype": dbt.type_string()},
+    {"name": "logo_deleted", "datatype": "boolean"},
+    {"name": "logo_file_name", "datatype": dbt.type_string()},
+    {"name": "logo_height", "datatype": dbt.type_int()},
+    {"name": "logo_id", "datatype": dbt.type_int()},
+    {"name": "logo_inline", "datatype": "boolean"},
+    {"name": "logo_mapped_content_url", "datatype": dbt.type_string()},
+    {"name": "logo_size", "datatype": dbt.type_int()},
+    {"name": "logo_url", "datatype": dbt.type_string()},
+    {"name": "logo_width", "datatype": dbt.type_int()},
+    {"name": "name", "datatype": dbt.type_string()},
+    {"name": "subdomain", "datatype": dbt.type_string()},
+    {"name": "url", "datatype": dbt.type_string()}
+] %}
+
+{{ return(columns) }}
+
+{% endmacro %}

macros/staging/get_audit_log_columns.sql

@@ -0,0 +1,14 @@
+{% macro get_daylight_time_columns() %}
+
+{% set columns = [
+    {"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
+    {"name": "daylight_end_utc", "datatype": "datetime"},
+    {"name": "daylight_offset", "datatype": dbt.type_int()},
+    {"name": "daylight_start_utc", "datatype": "datetime"},
+    {"name": "time_zone", "datatype": dbt.type_string()},
+    {"name": "year", "datatype": dbt.type_int()}
+] %}
+
+{{ return(columns) }}
+
+{% endmacro %}