bug/too-many-partitions (#131)

* bug/too-many-partitions

* updates

* updates and regen docs

* incremental updates

* simplify incremental

* update materialization

* regen docs

* update yml

* review updates

* Apply suggestions from code review

Co-authored-by: Avinash Kunnath <[email protected]>

* update readme

* update changelog

---------

Co-authored-by: Avinash Kunnath <[email protected]>

Author: fivetran-catfritz

CHANGELOG.md

@@ -1,3 +1,28 @@
+# dbt_jira v0.18.0
+[PR #131](https://github.com/fivetran/dbt_jira/pull/131) contains the following updates:
+## Breaking Changes
+> Since the following changes are breaking, a `--full-refresh` after upgrading will be required.
+
+- Changed the partitioning from days to weeks in the following models for BigQuery and Databricks All Purpose Cluster destinations:
+  - `int_jira__pivot_daily_field_history`
+    - Added field `valid_starting_at_week` for use with the new weekly partition logic.
+  - `jira__daily_issue_field_history`
+    - Added field `date_week` for use with the new weekly partition logic.
+- This adjustment reduces the total number of partitions, helping avoid partition limit issues in certain warehouses.
+- For Databricks All Purpose Cluster destinations, updated the `file_format` to `delta` for improved performance.
+- Updated the default materialization of `int_jira__issue_calendar_spine` from incremental to ephemeral to improve performance and maintainability.
+
+## Documentation Update
+- Updated [README](https://github.com/fivetran/dbt_jira/blob/main/README.md#lookback-window) with the new default of 1 week for the `lookback_window` variable.
+
+## Under the Hood
+- Replaced the deprecated `dbt.current_timestamp_backcompat()` function with `dbt.current_timestamp()` to ensure all timestamps are captured in UTC for the following models:
+  - `int_jira__issue_calendar_spine`
+  - `int_jira__issue_join`
+  - `jira__issue_enhanced`
+- Updated model `int_jira__issue_calendar_spine` to prevent errors during compilation.
+- Added consistency tests for the `jira__daily_issue_field_history` and `jira__issue_enhanced` models.
+
 # dbt_jira v0.17.0
 [PR #127](https://github.com/fivetran/dbt_jira/pull/127) contains the following updates:
 

README.md

@@ -66,7 +66,7 @@ Include the following jira package version in your `packages.yml` file:
 ```yaml
 packages:
   - package: fivetran/jira
-    version: [">=0.17.0", "<0.18.0"]
+    version: [">=0.18.0", "<0.19.0"]
 
 ```
 ### Step 3: Define database and schema variables
@@ -142,14 +142,14 @@ vars:
 ```
 
 #### Lookback Window
-Records from the source can sometimes arrive late. Since several of the models in this package are incremental, by default we look back 3 days to ensure late arrivals are captured while avoiding the need for frequent full refreshes. While the frequency can be reduced, we still recommend running `dbt --full-refresh` periodically to maintain data quality of the models.
+Records from the source may occasionally arrive late. To handle this, we implement a one-week lookback in our incremental models to capture late arrivals without requiring frequent full refreshes. The lookback is structured in weekly increments, as the incremental logic is based on weekly periods. While the frequency of full refreshes can be reduced, we still recommend running `dbt --full-refresh` periodically to maintain data quality of the models. 
 
 To change the default lookback window, add the following variable to your `dbt_project.yml` file:
 
 ```yml
 vars:
   jira:
-    lookback_window: number_of_days # default is 3
+    lookback_window: number_of_weeks # default is 1
 ```
 
 ### (Optional) Step 6: Orchestrate your models with Fivetran Transformations for dbt Core™

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'jira'
-version: '0.17.0'
+version: '0.18.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 vars:

docs/catalog.json

@@ -1,5 +1,5 @@
 name: 'jira_integration_tests'
-version: '0.17.0'
+version: '0.18.0'
 config-version: 2
 profile: 'integration_tests'
 
@@ -28,11 +28,13 @@ vars:
     jira_user_identifier: "user"
     jira_version_identifier: "version"
 
+  # Comment out the below when generating docs
   issue_field_history_columns: ['summary', 'story points', 'components']
 
 models:
   jira:
     +schema: "{{ 'jira_integrations_tests_sqlw' if target.name == 'databricks-sql' else 'jira' }}"
+    # +schema: "jira_{{ var('directed_schema','dev') }}"
 
 seeds:
   jira_integration_tests:

docs/index.html

@@ -0,0 +1,58 @@
+
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select
+        date_day,
+        issue_id,
+        status,
+        status_id,
+        sprint,
+        issue_day_id
+    from {{ target.schema }}_jira_prod.jira__daily_issue_field_history
+),
+
+dev as (
+    select
+        date_day,
+        issue_id,
+        status,
+        status_id,
+        sprint,
+        issue_day_id
+    from {{ target.schema }}_jira_dev.jira__daily_issue_field_history
+),
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final

docs/manifest.json

@@ -0,0 +1,46 @@
+
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select *
+    from {{ target.schema }}_jira_prod.jira__issue_enhanced
+),
+
+dev as (
+    select *
+    from {{ target.schema }}_jira_dev.jira__issue_enhanced
+),
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final