Skip to content

Commit 4b27b61

Browse files
fivetran-catfritzfivetran-joemarkiewiczfivetran-avinashfivetran-data-model-botgithub-actions[bot]
authored
Feature/source consolidation and star schema (#25)
* feature/source-consolidation-and-star-schema * Feature/source consolidation and star schema * minor working updates * Feature/sales and procurement (#12) * changelog * style(sql): normalize procurement models (#11) * manual updates * add staging * add missing int and stg models * get cols macro work * Add explicit column lists for remaining lookup macros (#14) * seems to be working * Populate seed files with realistic dummy values (#15) * Add missing seed files and update integration tests config (#13) * seeds seem to be working * yml * fix seeds * redshift working * Normalize seed timestamps (#16) * snowflake working * fixed redshift again * fix postgres add snowflake seeds * bigquery working * add bq seeds * seed issue * formatting * Prefix table toggle variables with sap_ (#17) * add remove slashes macro * update staging with remove slashes macro * Add variable-based enablement to star schema and intermediate models (#18) * Restore missing descriptions (#19) * update src yml descriptions * Fill column descriptions (#20) * add missing stg descriptions * add t024e * fix seed * fix cast * Sort staging model documentation (#21) * Update staging YAML with all model columns (#22) * fill missing stg descriptions from src (#23) * add docs for new models * Document sales_and_procurement models (#24) * update final model descriptions * add combo tests * add mandt vars * readme * updates for variable configs * fixes * add consistency * consistency tests * update seeds * fix seeds again * regen docs * run_models * fix intermediate nulls * fix cast * Apply suggestions from code review Co-authored-by: Joe Markiewicz <[email protected]> * Q2 FY26 Automatic Package Updates (#26) * Q2 FY26: Apply automated update. * Q2 FY26: Update auto-release workflow only. * Update CHANGELOG.md --------- Co-authored-by: fivetran-catfritz <[email protected]> * Feature/sales and procurement (#27) * changelog * style(sql): normalize procurement models (#11) * manual updates * add staging * add missing int and stg models * get cols macro work * Add explicit column lists for remaining lookup macros (#14) * seems to be working * Populate seed files with realistic dummy values (#15) * Add missing seed files and update integration tests config (#13) * seeds seem to be working * yml * fix seeds * redshift working * Normalize seed timestamps (#16) * snowflake working * fixed redshift again * fix postgres add snowflake seeds * bigquery working * add bq seeds * seed issue * formatting * Prefix table toggle variables with sap_ (#17) * add remove slashes macro * update staging with remove slashes macro * Add variable-based enablement to star schema and intermediate models (#18) * Restore missing descriptions (#19) * update src yml descriptions * Fill column descriptions (#20) * add missing stg descriptions * add t024e * fix seed * fix cast * Sort staging model documentation (#21) * Update staging YAML with all model columns (#22) * fill missing stg descriptions from src (#23) * add docs for new models * Document sales_and_procurement models (#24) * update final model descriptions * add combo tests * add mandt vars * readme * updates for variable configs * fixes * add consistency * consistency tests * update seeds * fix seeds again * regen docs * run_models * fix intermediate nulls * fix cast * update filters and vars * add comma * changelog readme * fixes * update public models * readme * Generate dbt docs via GitHub Actions * rename ymls * Generate dbt docs via GitHub Actions * changelog * Apply suggestions from code review Co-authored-by: Avinash Kunnath <[email protected]> * changelog * Update models/sales_and_procurement/star_schema/intermediate/int_sap__company.sql Co-authored-by: Avinash Kunnath <[email protected]> * t001 fix and regen docs * changelog * vendor * readme * update var * Generate dbt docs via GitHub Actions * Apply suggestions from code review Co-authored-by: Avinash Kunnath <[email protected]> * add test * changelog * Generate dbt docs via GitHub Actions * add links * Generate dbt docs via GitHub Actions * Apply suggestions from code review Co-authored-by: Avinash Kunnath <[email protected]> * Update CHANGELOG.md Co-authored-by: Avinash Kunnath <[email protected]> * Update CHANGELOG.md --------- Co-authored-by: Joe Markiewicz <[email protected]> Co-authored-by: Avinash Kunnath <[email protected]> Co-authored-by: Fivetran Maintainer Bot <[email protected]> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Avinash Kunnath <[email protected]>
1 parent de09c62 commit 4b27b61

File tree

226 files changed

+27196
-836
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

226 files changed

+27196
-836
lines changed

‎.buildkite/pipeline.yml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ steps:
33
key: "run-dbt-postgres"
44
plugins:
55
- docker#v3.13.0:
6-
image: "python:3.8"
6+
image: "python:3.10.13"
77
shell: [ "/bin/bash", "-e", "-c" ]
88
environment:
99
- "BASH_ENV=/tmp/.bashrc"
@@ -18,7 +18,7 @@ steps:
1818
key: "run_dbt_snowflake"
1919
plugins:
2020
- docker#v3.13.0:
21-
image: "python:3.8"
21+
image: "python:3.10.13"
2222
shell: [ "/bin/bash", "-e", "-c" ]
2323
environment:
2424
- "BASH_ENV=/tmp/.bashrc"
@@ -35,7 +35,7 @@ steps:
3535
key: "run_dbt_bigquery"
3636
plugins:
3737
- docker#v3.13.0:
38-
image: "python:3.8"
38+
image: "python:3.10.13"
3939
shell: [ "/bin/bash", "-e", "-c" ]
4040
environment:
4141
- "BASH_ENV=/tmp/.bashrc"
@@ -47,7 +47,7 @@ steps:
4747
key: "run_dbt_redshift"
4848
plugins:
4949
- docker#v3.13.0:
50-
image: "python:3.8"
50+
image: "python:3.10.13"
5151
shell: [ "/bin/bash", "-e", "-c" ]
5252
environment:
5353
- "BASH_ENV=/tmp/.bashrc"
@@ -62,7 +62,7 @@ steps:
6262
key: "run_dbt_databricks"
6363
plugins:
6464
- docker#v3.13.0:
65-
image: "python:3.8"
65+
image: "python:3.10.13"
6666
shell: [ "/bin/bash", "-e", "-c" ]
6767
environment:
6868
- "BASH_ENV=/tmp/.bashrc"

‎.buildkite/scripts/run_models.sh‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,5 +20,17 @@ dbt seed --target "$db" --full-refresh
2020
dbt run --target "$db" --full-refresh
2121
dbt test --target "$db"
2222

23+
dbt run --vars '{sap_using_ekko: False, sap_using_ekpo: False, sap_using_eket: False}' --target "$db" --full-refresh
24+
dbt test --vars '{sap_using_ekko: False, sap_using_ekpo: False, sap_using_eket: False}' --target "$db"
25+
26+
dbt run --vars '{sap_using_vbak: False, sap_using_vbap: False, sap_using_vbuk: False, sap_using_vbup: False}' --target "$db" --full-refresh
27+
dbt test --vars '{sap_using_vbak: False, sap_using_vbap: False, sap_using_vbuk: False, sap_using_vbup: False}' --target "$db"
28+
29+
dbt run --vars '{sap_using_mara: False, sap_using_makt: False, sap_using_t134: False, sap_using_t134t: False}' --target "$db" --full-refresh
30+
dbt test --vars '{sap_using_mara: False, sap_using_makt: False, sap_using_t134: False, sap_using_t134t: False}' --target "$db"
31+
32+
dbt run --vars '{sap_using_dd07t: False, sap_using_t161t: False, sap_using_t001w: False}' --target "$db" --full-refresh
33+
dbt test --vars '{sap_using_dd07t: False, sap_using_t161t: False, sap_using_t001w: False}' --target "$db"
34+
2335
dbt run-operation fivetran_utils.drop_schemas_automation --target "$db"
2436

‎.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md‎

Lines changed: 45 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,45 @@
1-
## PR Overview
2-
**This PR will address the following Issue/Feature:**
3-
4-
**This PR will result in the following new package version:**
5-
<!--- Please add details around your decision for breaking vs non-breaking version upgrade. If this is a breaking change, were backwards-compatible options explored? -->
6-
7-
**Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:**
8-
<!--- Copy/paste the CHANGELOG for this version below. -->
9-
10-
## PR Checklist
11-
### Basic Validation
12-
Please acknowledge that you have successfully performed the following commands locally:
13-
- [ ] dbt run –full-refresh && dbt test
14-
- [ ] dbt run (if incremental models are present) && dbt test
15-
16-
Before marking this PR as "ready for review" the following have been applied:
17-
- [ ] The appropriate issue has been linked, tagged, and properly assigned
18-
- [ ] All necessary documentation and version upgrades have been applied
19-
- [ ] docs were regenerated (unless this PR does not include any code or yml updates)
20-
- [ ] BuildKite integration tests are passing
21-
- [ ] Detailed validation steps have been provided below
22-
23-
### Detailed Validation
24-
Please share any and all of your validation steps:
25-
<!--- Provide the steps you took to validate your changes below. -->
26-
27-
### If you had to summarize this PR in an emoji, which would it be?
28-
<!--- For a complete list of markdown compatible emojis check our this git repo (https://gist.github.com/rxaviers/7360908) -->
29-
:dancer:
1+
<!--
2+
Pre-Submission Reminders
3+
Before marking this PR as "ready for review":
4+
5+
- `dbt run --full-refresh && dbt test`
6+
- `dbt run` && `dbt test` (if incremental models are present)
7+
- The related issue is linked, tagged, and appropriately assigned
8+
- Documentation and version updates are included, if applicable
9+
- `docs` have been regenerated (unless there are no code or YAML changes)
10+
- BuildKite integration tests are passing
11+
-->
12+
13+
## PR Overview
14+
**Package version introduced in this PR:**
15+
16+
**This PR addresses the following Issue/Feature(s):**
17+
<!-- Add Issue # or internal ticket reference -->
18+
19+
**Summary of changes:**
20+
<!-- 1-2 sentences describing PR changes. -->
21+
22+
-
23+
24+
**This PR addresses the following Issue/Feature(s):**
25+
<!-- Add Issue # or internal ticket reference -->
26+
-
27+
28+
**Summary of changes:**
29+
<!-- 1-2 sentences describing PR changes. -->
30+
-
31+
32+
### Submission Checklist
33+
- [ ] Alignment meeting with the reviewer (if needed)
34+
- [ ] Timeline and validation requirements discussed
35+
- [ ] Provide validation details:
36+
- [ ] **Validation Steps:** Check for unintentional effects (e.g., add/run consistency & integrity tests)
37+
- [ ] **Testing Instructions:** Confirm the change addresses the issue(s)
38+
- [ ] **Focus Areas:** Complex logic or queries that need extra attention
39+
- [ ] Merge any relevant open PRs into this PR
40+
41+
### Changelog
42+
<!-- Recommend drafting changelog notes, then refining via ChatGPT using:
43+
"Draft a changelog entry based on the following notes." -->
44+
- [ ] Draft changelog for PR
45+
- [ ] Final changelog for release review

‎.github/workflows/auto-release.yml‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,12 @@ on:
33
pull_request:
44
types:
55
- closed
6-
branches:
7-
- main
6+
- labeled
87

98
jobs:
10-
call-workflow-passing-data:
11-
if: github.event.pull_request.merged
9+
release:
10+
if: |
11+
(github.event.pull_request.merged == true && github.event.pull_request.base.ref == 'main') ||
12+
github.event.label.name == 'pre-release'
1213
uses: fivetran/dbt_package_automations/.github/workflows/auto-release.yml@main
13-
secrets: inherit
14+
secrets: inherit

‎.github/workflows/generate-docs.yml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
name: 'generate dbt docs'
2+
on:
3+
pull_request:
4+
types:
5+
- labeled
6+
7+
jobs:
8+
generate-docs:
9+
if: github.event.label.name == 'docs:ready'
10+
uses: fivetran/dbt_package_automations/.github/workflows/generate-docs.yml@main
11+
secrets: inherit
12+
with:
13+
schema_var_name: sap_schema

‎.gitignore‎

Lines changed: 70 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,73 @@
1-
2-
target/
1+
# dbt
2+
**/package-lock.yml
3+
package-lock.yml
4+
.dbt/
35
dbt_modules/
6+
dbt_packages/
47
logs/
8+
profiles.yml
9+
target/
10+
*.log
11+
12+
# IDE files
13+
.idea/
14+
.vscode/
15+
*~
16+
*.swp
17+
*.swo
18+
19+
# Jupyter Notebook
20+
.ipynb_checkpoints
21+
22+
# OS generated files
23+
**/.DS_Store
524
.DS_Store
6-
dbt_packages/
25+
.Spotlight-V100
26+
.Trashes
27+
._*
28+
Thumbs.db
29+
ehthumbs.db
30+
31+
# Python
32+
*.egg
33+
*.egg-info/
34+
*.py[cod]
35+
*.so
36+
*$py.class
37+
.Python
38+
__pycache__/
39+
build/
40+
develop-eggs/
41+
dist/
42+
downloads/
43+
eggs/
44+
.env
45+
.installed.cfg
46+
lib/
47+
lib64/
48+
MANIFEST
49+
parts/
50+
sdist/
51+
var/
52+
wheels/
53+
54+
# Secrets and credentials
55+
.env.*
56+
.secrets
57+
credentials.json
58+
service-account.json
59+
60+
# Temporary files
61+
.cache/
62+
*.temp
63+
*.tmp
64+
65+
# Virtual environments
66+
.conda/
67+
.env
68+
.venv
69+
ENV/
70+
env/
71+
env.bak/
72+
venv/
73+
venv.bak/

‎.quickstart/quickstart.yml‎

Lines changed: 84 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,80 @@ schema_key: sap_schema
33

44
dbt_versions: ">=1.3.0 <2.0.0"
55

6+
table_variables:
7+
sap_using_bkpf:
8+
- bkpf
9+
sap_using_bseg:
10+
- bseg
11+
sap_using_dd07l:
12+
- dd07l
13+
sap_using_dd07t:
14+
- dd07t
15+
sap_using_ekbe:
16+
- ekbe
17+
sap_using_eket:
18+
- eket
19+
sap_using_ekko:
20+
- ekko
21+
sap_using_ekpo:
22+
- ekpo
23+
sap_using_faglflexa:
24+
- faglflexa
25+
sap_using_faglflext:
26+
- faglflext
27+
sap_using_kna1:
28+
- kna1
29+
sap_using_lfa1:
30+
- lfa1
31+
sap_using_makt:
32+
- makt
33+
sap_using_mara:
34+
- mara
35+
sap_using_pa0000:
36+
- pa0000
37+
sap_using_pa0001:
38+
- pa0001
39+
sap_using_pa0007:
40+
- pa0007
41+
sap_using_pa0008:
42+
- pa0008
43+
sap_using_pa0031:
44+
- pa0031
45+
sap_using_ska1:
46+
- ska1
47+
sap_using_t001:
48+
- t001
49+
sap_using_t001w:
50+
- t001w
51+
sap_using_t024:
52+
- t024
53+
sap_using_t024e:
54+
- t024e
55+
sap_using_t134:
56+
- t134
57+
sap_using_t134t:
58+
- t134t
59+
sap_using_t161:
60+
- t161
61+
sap_using_t161t:
62+
- t161t
63+
sap_using_t503:
64+
- t503
65+
sap_using_t880:
66+
- t880
67+
sap_using_tvag:
68+
- tvag
69+
sap_using_tvagt:
70+
- tvagt
71+
sap_using_vbak:
72+
- vbak
73+
sap_using_vbap:
74+
- vbap
75+
sap_using_vbuk:
76+
- vbuk
77+
sap_using_vbup:
78+
- vbup
79+
680
destination_configurations:
781
databricks:
882
dispatch:
@@ -20,5 +94,14 @@ public_models: [
2094
"sap__0customer_attr",
2195
"sap__0comp_code_text",
2296
"sap__0comp_code_attr",
23-
"sap__0vendor_text"
97+
"sap__0vendor_text",
98+
"sap__dim_customer",
99+
"sap__dim_material",
100+
"sap__dim_plant",
101+
"sap__dim_purchasing_order",
102+
"sap__dim_purchasing_organization",
103+
"sap__dim_rejection_reason",
104+
"sap__dim_vendor",
105+
"sap__fact_purchasing_order",
106+
"sap__fact_sales_order"
24107
]

0 commit comments

Comments
 (0)