feature/freshness-updates (#41)

* feature/freshness-updates

* changelog pr link update

* update package version in README

---------

Co-authored-by: Jamie Rodriguez <[email protected]>

Author: fivetran-joemarkiewicz

.buildkite/pipeline.yml

@@ -3,7 +3,7 @@ steps:
     key: "run-dbt-postgres"
     plugins:
       - docker#v3.13.0:
-          image: "python:3.8"
+          image: "python:3.10.13"
           shell: [ "/bin/bash", "-e", "-c" ]
           environment:
             - "BASH_ENV=/tmp/.bashrc"
@@ -18,7 +18,7 @@ steps:
     key: "run_dbt_snowflake"
     plugins:
       - docker#v3.13.0:
-          image: "python:3.8"
+          image: "python:3.10.13"
           shell: [ "/bin/bash", "-e", "-c" ]
           environment:
             - "BASH_ENV=/tmp/.bashrc"
@@ -35,7 +35,7 @@ steps:
     key: "run_dbt_bigquery"
     plugins:
       - docker#v3.13.0:
-          image: "python:3.8"
+          image: "python:3.10.13"
           shell: [ "/bin/bash", "-e", "-c" ]
           environment:
             - "BASH_ENV=/tmp/.bashrc"
@@ -47,7 +47,7 @@ steps:
     key: "run_dbt_redshift"
     plugins:
       - docker#v3.13.0:
-          image: "python:3.8"
+          image: "python:3.10.13"
           shell: [ "/bin/bash", "-e", "-c" ]
           environment:
             - "BASH_ENV=/tmp/.bashrc"
@@ -62,7 +62,7 @@ steps:
     key: "run_dbt_databricks"
     plugins:
       - docker#v3.13.0:
-          image: "python:3.8"
+          image: "python:3.10.13"
           shell: [ "/bin/bash", "-e", "-c" ]
           environment:
             - "BASH_ENV=/tmp/.bashrc"

.buildkite/scripts/run_models.sh

@@ -17,6 +17,7 @@ echo `pwd`
 cd integration_tests
 dbt deps
 dbt seed --target "$db" --full-refresh
+dbt source freshness --target "$db" || echo "...Only verifying freshness runs…"
 dbt run --target "$db" --full-refresh
 dbt test --target "$db"
 dbt run --vars '{asana__using_tags: false, asana__using_task_tags: false, asana_tag_identifier: missing, asana_task_tag_identifier: missing}' --target "$db" --full-refresh

CHANGELOG.md

@@ -1,3 +1,30 @@
+# dbt_asana_source v0.9.0
+
+[PR #41](https://github.com/fivetran/dbt_asana_source/pull/41) includes the following updates:
+
+## Breaking Change for dbt Core < 1.9.6
+
+> *Note: This is not relevant to Fivetran Quickstart users.*
+
+Migrated `freshness` from a top-level source property to a source `config` in alignment with [recent updates](https://github.com/dbt-labs/dbt-core/issues/11506) from dbt Core. This will resolve the following deprecation warning that users running dbt >= 1.9.6 may have received:
+
+```
+[WARNING]: Deprecated functionality
+Found `freshness` as a top-level property of `asana` in file
+`models/src_asana.yml`. The `freshness` top-level property should be moved
+into the `config` of `asana`.
+```
+
+**IMPORTANT:** Users running dbt Core < 1.9.6 will not be able to utilize freshness tests in this release or any subsequent releases, as older versions of dbt will not recognize freshness as a source `config` and therefore not run the tests.
+
+If you are using dbt Core < 1.9.6 and want to continue running Asana freshness tests, please elect **one** of the following options:
+  1. (Recommended) Upgrade to dbt Core >= 1.9.6
+  2. Do not upgrade your installed version of the `asana_source` package. Pin your dependency on v0.8.2 in your `packages.yml` file.
+  3. Utilize a dbt [override](https://docs.getdbt.com/reference/resource-properties/overrides) to overwrite the package's `asana` source and apply freshness via the previous release top-level property route. This will require you to copy and paste the entirety of the previous release `src_asana.yml` file and add an `overrides: asana_source` property.
+
+## Under the Hood
+- Updates to ensure integration tests use latest version of dbt.
+
 # dbt_asana_source v0.8.2
 This release includes the following updates:
 

README.md

@@ -17,6 +17,7 @@
   - Names columns for consistency across all packages and for easier analysis
   - Removes any rows that are soft-deleted
   - Adds freshness tests to source data
+    - dbt Core >= 1.9.6 is required to run freshness tests out of the box.
   - Adds column-level testing where applicable. For example, all primary keys are tested for uniqueness and non-null values.
 - Generates a comprehensive data dictionary of your Asana data through the [dbt docs site](https://fivetran.github.io/dbt_asana_source/).
 - These tables are designed to work simultaneously with our [Asana transformation package](https://github.com/fivetran/dbt_asana).
@@ -34,7 +35,7 @@ Include the following asana_source package version in your `packages.yml` file.
 ```yaml
 packages:
   - package: fivetran/asana_source
-    version: [">=0.8.0", "<0.9.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.9.0", "<0.10.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 
 ### Step 3: Define database and schema variables

dbt_project.yml

@@ -1,6 +1,6 @@
 config-version: 2
 name: 'asana_source'
-version: 0.8.2
+version: 0.9.0
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:
   asana_source:

integration_tests/ci/sample.profiles.yml

@@ -2,9 +2,6 @@
 # HEY! This file is used in the dbt package integrations tests with Buildkite.
 # You should __NEVER__ check credentials into version control. Thanks for reading :)
 
-config:
-    send_anonymous_usage_stats: False
-    use_colors: True
 
 integration_tests:
   target: redshift

integration_tests/dbt_project.yml

@@ -1,6 +1,6 @@
 config-version: 2
 name: 'asana_source_integration_tests'
-version: 0.8.2
+version: 0.9.0
 profile: 'integration_tests'
 vars:
   asana_schema: asana_source_integrations_tests_3
@@ -81,4 +81,7 @@ seeds:
 
 dispatch:
   - macro_namespace: dbt_utils
-    search_order: ['spark_utils', 'dbt_utils']
+    search_order: ['spark_utils', 'dbt_utils']
+
+flags:
+  send_anonymous_usage_stats: False

integration_tests/requirements.txt

@@ -5,4 +5,4 @@ dbt-postgres>=1.3.0,<2.0.0
 dbt-spark>=1.3.0,<2.0.0
 dbt-spark[PyHive]>=1.3.0,<2.0.0
 dbt-databricks>=1.3.0,<2.0.0
-certifi==2025.1.31
+certifi==2025.1.31

models/src_asana.yml

@@ -7,17 +7,19 @@ sources:
 
     loader: fivetran
     loaded_at_field: _fivetran_synced
-      
-    freshness:
-      warn_after: {count: 72, period: hour}
-      error_after: {count: 96, period: hour}
+
+    config:
+      freshness:
+        warn_after: {count: 72, period: hour}
+        error_after: {count: 96, period: hour}
 
 
     tables:
       - name: user
         identifier: "{{ var('asana_user_identifier', 'user')}}"
         description: Table of all accounts in the organization
-        freshness: null
+        config:
+          freshness: null
         columns:
           - name: id
             description: System generated unique ID for a user
@@ -68,7 +70,8 @@ sources:
       - name: project
         identifier: "{{ var('asana_project_identifier', 'project')}}"
         description: Table containing all projects
-        freshness: null
+        config:
+          freshness: null
         columns:
           - name: id
             description: System generated unique ID of the project.
@@ -121,7 +124,8 @@ sources:
       - name: team
         identifier: "{{ var('asana_team_identifier', 'team')}}"
         description: Table of the teams within the organization
-        freshness: null
+        config:
+          freshness: null
         columns:
           - name: id
             description: System generated unique ID for each team
@@ -134,9 +138,9 @@ sources:
       - name: tag
         identifier: "{{ var('asana_tag_identifier', 'tag')}}"
         description: Table of the custom tags made in the organization
-        config: 
+        config:
           enabled: "{{ var('asana__using_tags', True) }}"  # Disable if asana__using_tags is False
-        freshness: null
+          freshness: null
         columns:
           - name: id
             description: System generated ID for each tag
@@ -151,7 +155,8 @@ sources:
       - name: section
         identifier: "{{ var('asana_section_identifier', 'section')}}"
         description: Table of all sections within projects. If tasks in a project are not in an explicit user-made section, a default section called "(no section)" is created here
-        freshness: null
+        config:
+          freshness: null
         columns:
           - name: id
             description: System generated unique ID for each section
@@ -165,7 +170,8 @@ sources:
       - name: project_task
         identifier: "{{ var('asana_project_task_identifier', 'project_task')}}"
         description: Table of tasks and the project(s) they each belong to. If a project doesn't have a task, it won't be in here (and vice versa)
-        freshness: null
+        config:
+          freshness: null
         columns:
           - name: project_id
             description: Foreign key referencing a PROJECT the task belongs to
@@ -176,7 +182,8 @@ sources:
       - name: task_follower
         identifier: "{{ var('asana_task_follower_identifier', 'task_follower')}}"
         description: Table of tasks with the users that are following them. Tasks without users do not appear here.
-        freshness: null
+        config:
+          freshness: null
         columns:
           - name: task_id
             description: Foreign key referencing the TASK being followed
@@ -187,9 +194,9 @@ sources:
       - name: task_tag
         identifier: "{{ var('asana_task_tag_identifier', 'task_tag')}}"
         description: Table of tasks with the tags attached to them. Tasks without tags (and vice versa) do not appear here.
-        config: 
+        config:
           enabled: "{{ var('asana__using_task_tags', True) }}"  # Disable if asana__using_task_tags is False
-        freshness: null
+          freshness: null
         columns:
           - name: task_id
             description: Foreign key referencing the TASK that is tagged
@@ -200,7 +207,8 @@ sources:
       - name: task_section
         identifier: "{{ var('asana_task_section_identifier', 'task_section')}}"
         description: Table of tasks and the section that they live under. Section-less tasks within projects appear here, but project-less tasks do not
-        freshness: null
+        config:
+          freshness: null
         columns:
           - name: section_id
             description: Foreign key referencing the SECTION that the task lives under.