Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

expose all ntd annual reporting staging tables as mart #3550

Merged
merged 6 commits into from
Nov 22, 2024
Merged

expose all ntd annual reporting staging tables as mart #3550

merged 6 commits into from
Nov 22, 2024

Conversation

charlie-costanzo
Copy link
Member

@charlie-costanzo charlie-costanzo commented Nov 14, 2024 •
edited
Loading

Description

As requested in #3531, we are making all of the base staging NTD annual reporting tables available in mart schemas so as to access them in Metabase.

Resolves #3531

Type of change

  • New feature

How has this been tested?

tested locally with dbt
Screenshot 2024-11-22 at 5 18 42 PM

Post-merge follow-ups

  • Actions required (specified below)
    observe post-merge for expected behavior, change any outstanding/new models to use these new schemas and communicate out new schema.

@charlie-costanzo charlie-costanzo self-assigned this Nov 14, 2024
@charlie-costanzo charlie-costanzo changed the base branch from main to rename-ntd-api-tables November 14, 2024 20:59
@charlie-costanzo charlie-costanzo force-pushed the rename-ntd-api-tables branch 3 times, most recently from 282b6a2 to 0bc53d0 Compare November 21, 2024 21:08
Base automatically changed from rename-ntd-api-tables to main November 21, 2024 21:13
@charlie-costanzo charlie-costanzo marked this pull request as ready for review November 22, 2024 22:22
@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 22, 2024 •
edited
Loading

Warehouse report 📦

Checks/potential follow-ups

Checks indicate the following action items may be necessary.

  • For new models, do they all have a surrogate primary key that is tested to be not-null and unique?

New models 🌱

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__breakdowns

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__breakdowns_by_agency

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__capital_expenses_by_capital_use

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__capital_expenses_by_mode

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__capital_expenses_for_existing_service

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__capital_expenses_for_expansion_of_service

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__employees_by_agency

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__employees_by_mode

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__employees_by_mode_and_employee_type

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__fuel_and_energy

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__fuel_and_energy_by_agency

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__funding_sources_by_expense_type

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__funding_sources_directly_generated

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__funding_sources_federal

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__funding_sources_local

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__funding_sources_state

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__funding_sources_taxes_levied_by_agency

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__maintenance_facilities

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__maintenance_facilities_by_agency

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__metrics

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__operating_expenses_by_function

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__operating_expenses_by_function_and_agency

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__operating_expenses_by_type

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__operating_expenses_by_type_and_agency

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__service_by_agency

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__service_by_mode

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__service_by_mode_and_time_period

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__stations_and_facilities_by_agency_and_facility_type

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__stations_by_mode_and_age

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__track_and_roadway_by_agency

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__track_and_roadway_by_mode

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__track_and_roadway_guideway_age_distribution

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__vehicles_age_distribution

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__vehicles_type_count_by_agency

DAG

Legend (in order of precedence)

Resource type Indicator Resolution
Large table-materialized model Orange Make the model incremental
Large model without partitioning or clustering Orange Add partitioning and/or clustering
View with more than one child Yellow Materialize as a table or incremental
Incremental Light green
Table Green
View White

erikamov
erikamov reviewed Nov 22, 2024
View reviewed changes
warehouse/models/mart/ntd_fct_annual/_mart_ntd_fct_annual.yml Outdated
@@ -0,0 +1,102 @@
version: 2

x-common-fields:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These common fields are not in use here. I think would be better to add when adding documentation.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah good catch, removed!

erikamov
erikamov approved these changes Nov 22, 2024
View reviewed changes
Copy link
Contributor

@erikamov erikamov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Excited to see analysts using those tables!!

@charlie-costanzo charlie-costanzo merged commit 078bd26 into main Nov 22, 2024
4 checks passed
@charlie-costanzo charlie-costanzo deleted the expose-ntd-annual-tables-as-mart branch November 22, 2024 22:45
@charlie-costanzo charlie-costanzo added the data-pipeline-ingestion-and-modeling Ingesting, parsing and modeling data. Evan Siroky is product owner. label Nov 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@erikamov erikamov erikamov approved these changes

@evansiroky evansiroky Awaiting requested review from evansiroky evansiroky is a code owner

@vevetron vevetron Awaiting requested review from vevetron vevetron is a code owner

@tiffanychu90 tiffanychu90 Awaiting requested review from tiffanychu90 tiffanychu90 is a code owner

Assignees

@charlie-costanzo charlie-costanzo

Labels
data-pipeline-ingestion-and-modeling Ingesting, parsing and modeling data. Evan Siroky is product owner.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

make remaining ntd staging tables available in Metabase in mart schema
2 participants
@charlie-costanzo @erikamov