feature/update-materialization-for-sap-models (#34)

* feature/update-materialization-for-sap-models

* Remove combo tests and move `loaded_at_field` (#32)

* Remove YAML tests and bump patch to v0.3.0

* Update README.md

* Update dbt_project.yml

* Apply suggestions from code review

---------

Co-authored-by: fivetran-catfritz <[email protected]>
Co-authored-by: Savage Fivetran <[email protected]>

* move_loaded_at_to_config

* Generate dbt docs via GitHub Actions

* Apply suggestions from code review

Co-authored-by: Joe Markiewicz <[email protected]>

* Update README.md

* Update CHANGELOG.md

* Generate dbt docs via GitHub Actions

---------

Co-authored-by: Fivetran Maintainer Bot <[email protected]>
Co-authored-by: fivetran-catfritz <[email protected]>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Joe Markiewicz <[email protected]>

Author: 5 people

CHANGELOG.md

@@ -1,3 +1,20 @@
+# dbt_sap v0.3.0
+
+[PR #34](https://github.com/fivetran/dbt_sap/pull/34) includes the following updates:
+
+## Schema and Data Changes **(--full-refresh required after upgrading)**
+**97 total changes • 94 of which are breaking in the form of materialization updates**
+| **Data Model** | **Change type** | **Old** | **New** | **Notes** |
+| -------------- | --------------- | ------------ | ------------ | --------- |
+|[stg_* ](https://github.com/fivetran/dbt_sap/tree/main/models/staging)(all staging models: 72)| Materialization | Table | View | These staging models are now views. Upon full refresh, the package should automatically drop the tables and recreate them as views. However, we recommend reviewing your schema and manually dropping any remaining staging models materialized as tables as they will no longer be updated in favor of the materialized views.|
+| [int_*](https://github.com/fivetran/dbt_sap/tree/main/models/sales_and_procurement/star_schema/intermediate) (all sales and procurement intermediate models: 22) | Materialization | Table | Ephemeral |  These intermediate models are now views. We therefore recommend manually dropping all sap data model sales and procurement intermediate tables as they will no longer be updated in favor of the materialized views.|
+
+## dbt Fusion Compatibility Updates
+- Updated package to maintain compatibility with dbt-core versions both before and after v1.10.6, which introduced a breaking change to multi-argument test syntax (e.g., `unique_combination_of_columns`).
+- Temporarily removed unsupported tests to avoid errors and ensure smoother upgrades across different dbt-core versions. These tests will be reintroduced once a safe migration path is available.
+  - Removed all `dbt_utils.unique_combination_of_columns` tests.
+  - Moved `loaded_at_field: _fivetran_synced` under the `config:` block in `src_sap.yml`
+
 # dbt_sap v0.2.1
 [PR #28](https://github.com/fivetran/dbt_sap/pull/28) includes the following updates:
 

README.md

@@ -21,7 +21,7 @@
     - Brings in general ledger models like General Ledger: Balances, Leading Ledger (`sap__0fi_gl_10`) and Line Items Leading Ledger (`sap__0fi_gl_14`).
     - Brings in master text models like Company Code (`sap__0comp_code_text`), Company (`sap__0company_text`), and Vendor (`sap__0vendor_text`).
     - Provides sales and procurement models including facts and dimensions for purchase and sales orders.
-- Produces modeled tables that leverage SAP data from [Fivetran's SAP connectors, like LDP SAP Netweaver](https://fivetran.com/docs/local-data-processing/requirements/source-and-target-requirements/sap-netweaver-requirements), [HVA SAP](https://fivetran.com/docs/databases/sap-erp/high-volume-agent) or [SAP ERP on HANA](https://fivetran.com/docs/databases/sap-erp/sap-erp-hana) and build off the output of our [SAP source package](https://github.com/fivetran/dbt_sap_source).
+- Produces modeled tables that leverage SAP data from [Fivetran's SAP connectors, like LDP SAP Netweaver](https://fivetran.com/docs/local-data-processing/requirements/source-and-target-requirements/sap-netweaver-requirements), [HVA SAP](https://fivetran.com/docs/databases/sap-erp/high-volume-agent) or [SAP ERP on HANA](https://fivetran.com/docs/databases/sap-erp/sap-erp-hana).
 - Generates a comprehensive data dictionary of your source and modeled sap data through the [dbt docs site](https://fivetran.github.io/dbt_sap/).
 
 <!--section=“sap_transformation_model"-->
@@ -42,7 +42,6 @@ The following table provides a detailed list of all tables materialized within t
 | [sap__0material_attr](https://fivetran.github.io/dbt_sap/#!/model/model.sap.sap__0material_attr)    |  This model is used to display material attribute information, originating from the `mara` source.                                                           |
 | [sap__0vendor_attr](https://fivetran.github.io/dbt_sap/#!/model/model.sap.sap__0vendor_attr)    |  This model is used to display vendor attributes, originating from the `lfa1` source.                           |
 | [sap__0vendor_text](https://fivetran.github.io/dbt_sap/#!/model/model.sap.sap__0vendor_text) |  This model is used to display vendor text, originating from the `lfa1` source.     |
-
 | [sap__dim_customer](https://fivetran.github.io/dbt_sap/#!/model/model.sap.sap__dim_customer) | Represents customer dimension data from the `kna1` source to support dimensional reporting. SAP field names are mapped to English readable column names. |
 | [sap__dim_material](https://fivetran.github.io/dbt_sap/#!/model/model.sap.sap__dim_material) | Provides enriched material and material type data by combining attributes from the `mara`, `makt`, `t134`, and `t134t` sources to support dimensional reporting. SAP field names are mapped to English readable column names. |
 | [sap__dim_plant](https://fivetran.github.io/dbt_sap/#!/model/model.sap.sap__dim_plant) | Delivers plant-level dimension data from the `t001w` source to support dimensional reporting. SAP field names are mapped to English readable column names. |
@@ -54,7 +53,7 @@ The following table provides a detailed list of all tables materialized within t
 | [sap__fact_sales_order](https://fivetran.github.io/dbt_sap/#!/model/model.sap.sap__fact_sales_order) | Contains fact-level sales order data, integrating records from `vbak`, `vbap`, `vbuk`, and `vbup` sources to provide visibility into sales transaction performance. SAP field names are mapped to English readable column names. |
 
 ### Materialized Models
-Each Quickstart transformation job run materializes 46 models if all components of this data model are enabled. This count includes all staging, intermediate, and final models materialized as `view`, `table`, or `incremental`.
+Each Quickstart transformation job run materializes 95 models if all components of this data model are enabled. This count includes all staging, intermediate, and final models materialized as `view`, `table`, or `incremental`.
 <!--section-end-->
 
 ## How do I use the dbt package?
@@ -129,12 +128,14 @@ vars:
 ```
 
 #### Change the build schema
-By default, this package builds the SAP staging models within a schema titled (`<target_schema>` + `stg_sap`) and the SAP final models within a schema titled (<target_schema> + `_sap`) in your target database. If this is not where you would like your modeled sap data to be written to, add the following configuration to your root `dbt_project.yml` file:
+By default, this package builds the SAP staging models within a schema titled (`<target_schema>` + `sap_source`) and the SAP final models within a schema titled (<target_schema> + `sap`) in your target database. If this is not where you would like your modeled sap data to be written to, add the following configuration to your root `dbt_project.yml` file:
 
 ```yml
 models:
     sap:
       +schema: my_new_schema_name # leave blank for just the target_schema
+      staging:
+        +schema: my_new_schema_name # leave blank for just the target_schema
 ```
 
 #### Change the source table references
@@ -143,7 +144,7 @@ If an individual source table has a different name than the package expects, add
 
 ```yml
 vars:
-    sap_<default_source_table_name>_identifier: your_table_name 
+    sap_<default_source_table_name>_identifier: your_table_name
 ```
 
 </details>

dbt_project.yml

@@ -1,12 +1,21 @@
 config-version: 2
 name: 'sap'
-version: '0.2.1'
+version: '0.3.0'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 
 models:
-  sap:
-    +materialized: table
-    +schema: sap 
+ sap:
+   +materialized: table
+   +schema: sap
+   staging:
+     +materialized: view
+   extractor_reports:
+     intermediate:
+       +materialized: table
+   sales_and_procurement:
+     star_schema:
+       intermediate:
+         +materialized: ephemeral
 
 vars:
   sap:

docs/catalog.json

@@ -1,5 +1,5 @@
 name: 'sap_integration_tests'
-version: '0.2.1'
+version: '0.3.0'
 profile: 'integration_tests'
 config-version: 2
 

docs/manifest.json

@@ -1,2 +1,2 @@
 packages:
-  - local: ../
+  - local: ../

integration_tests/dbt_project.yml

@@ -3,11 +3,6 @@ version: 2
 models:
   - name: sap__0comp_code_attr
     description: This model is used for loading company code attributes, extracting from the t001 data source.
-    tests:
-      - dbt_utils.unique_combination_of_columns:
-          combination_of_columns:
-            - mandt
-            - bukrs
     columns:
       - name: mandt
         description: Client
@@ -30,11 +25,6 @@ models:
 
   - name: sap__0comp_code_text
     description: This model is used for loading company code text information, extracting from the t001 data source. 
-    tests:
-      - dbt_utils.unique_combination_of_columns:
-          combination_of_columns:
-            - mandt
-            - bukrs
     columns: 
       - name: mandt
         description: Client
@@ -47,11 +37,6 @@ models:
 
   - name: sap__0company_text
     description: This model is used for loading customer text data, extracting from the t880 data source. 
-    tests:
-      - dbt_utils.unique_combination_of_columns:
-          combination_of_columns:
-            - mandt
-            - rcomp
     columns: 
       - name: mandt 
         description: Client
@@ -62,11 +47,6 @@ models:
 
   - name: sap__0customer_attr
     description: This model is used for loading customer master data, originating from the kna1 source.      
-    tests:
-      - dbt_utils.unique_combination_of_columns:
-          combination_of_columns:
-            - mandt
-            - kunnr    
     columns:
       - name: mandt 
         description: Client
@@ -131,12 +111,6 @@ models:
 
   - name: sap__0employee_attr
     description: This model contains information that concerns the employee's work relationship, extracting master data from the personnel administration tables.
-    tests:
-      - dbt_utils.unique_combination_of_columns:
-          combination_of_columns:
-            - pernr
-            - begda
-            - endda
     columns:
       - name: begda
         description: Vaild from date
@@ -611,12 +585,6 @@ models:
 
   - name: sap__0gl_account_attr
     description: This model is used for loading G/L Account Number master data, originating from the ska1 source.  
-    tests:
-      - dbt_utils.unique_combination_of_columns:
-          combination_of_columns:
-            - mandt
-            - ktopl
-            - saknr
     columns:
       - name: mandt
         description: Client
@@ -655,11 +623,6 @@ models:
 
   - name: sap__0material_attr
     description: This model is used to display material attribute information, originating from the mara source. 
-    tests:
-      - dbt_utils.unique_combination_of_columns:
-          combination_of_columns:
-            - mandt
-            - matnr
     columns:
       - name: mandt
         description: Client
@@ -916,11 +879,6 @@ models:
 
   - name: sap__0vendor_attr
     description: This model is used to display vendor attributes, originating from the lfa1 source.  
-    tests:
-      - dbt_utils.unique_combination_of_columns:
-          combination_of_columns:
-            - mandt
-            - lifnr
     columns:
       - name: mandt 
         description: Client
@@ -981,11 +939,6 @@ models:
 
   - name: sap__0vendor_text
     description: This model is used to display vendor text, originating from the lfa1 source.  
-    tests: 
-      - dbt_utils.unique_combination_of_columns:
-          combination_of_columns:
-            - mandt
-            - lifnr
     columns:
       - name: mandt
         description: Client

integration_tests/packages.yml

@@ -6,7 +6,8 @@ sources:
     database: "{% if target.type != 'spark' %}{{ var('sap_database', target.database) }}{% endif %}"
 
     loader: fivetran
-    loaded_at_field: _fivetran_synced
+    config:
+      loaded_at_field: _fivetran_synced
 
     tables:
       - name: bkpf