NTD time series data: create warehouse tables

[charlie-costanzo]

Description

Following up on #3655, this PR creates new warehouse tables and associated yml files for the new NTD time series endpoints that we are ingesting:

• staging tables with cleaning and casting
• intermediate tables that pivot on year
• mart tables that reflect the intermediate transformations

Type of change

• New feature

How has this been tested?

Post-merge follow-ups

• No action required

[github-actions]

Warehouse report 📦

Checks/potential follow-ups

Checks indicate the following action items may be necessary.

• For new models, do they all have a surrogate primary key that is tested to be not-null and unique?

New models 🌱

calitp_warehouse.mart.ntd_funding_and_expenses.fct_capital_expenditures_time_series_facilities

calitp_warehouse.mart.ntd_funding_and_expenses.fct_capital_expenditures_time_series_other

calitp_warehouse.mart.ntd_funding_and_expenses.fct_capital_expenditures_time_series_rolling_stock

calitp_warehouse.mart.ntd_funding_and_expenses.fct_capital_expenditures_time_series_total

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_capital_federal

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_capital_local

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_capital_other

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_capital_state

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_capital_total

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_decommissioned_operatingfares

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_decommissioned_operatingother

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_operating_federal

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_operating_local

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_operating_other

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_operating_state

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_operating_total

calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_summary_total

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_drm

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_fares

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_opexp_ga

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_opexp_nvm

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_opexp_total

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_opexp_vm

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_opexp_vo

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_pmt

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_upt

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_voms

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_vrh

calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_vrm

calitp_warehouse.mart.gtfs.fct_vehicle_locations_grouped

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__capital_expenditures_time_series_facilities

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__capital_expenditures_time_series_other

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__capital_expenditures_time_series_rolling_stock

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__capital_expenditures_time_series_total

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__operating_and_capital_funding_time_series_capital_federal

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__operating_and_capital_funding_time_series_capital_local

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__operating_and_capital_funding_time_series_capital_other

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__operating_and_capital_funding_time_series_capital_state

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__operating_and_capital_funding_time_series_capital_total

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__operating_and_capital_funding_time_series_operating_federal

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__operating_and_capital_funding_time_series_operating_local

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__operating_and_capital_funding_time_series_operating_other

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__operating_and_capital_funding_time_series_operating_state

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__operating_and_capital_funding_time_series_operating_total

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_drm

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_fares

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_opexp_ga

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_opexp_nvm

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_opexp_total

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_opexp_vm

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_opexp_vo

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_pmt

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_upt

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_voms

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_vrh

calitp_warehouse.intermediate.ntd_funding_and_expenses.int_ntd__service_data_and_operating_expenses_time_series_by_mode_vrm

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__capital_expenditures_time_series__facilities

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__capital_expenditures_time_series__other

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__capital_expenditures_time_series__rolling_stock

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__capital_expenditures_time_series__total

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__capital_federal

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__capital_local

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__capital_other

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__capital_state

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__capital_total

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__decommissioned_operatingfares

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__decommissioned_operatingother

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__operating_federal

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__operating_local

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__operating_other

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__operating_state

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__operating_total

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__summary_total

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__drm

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__fares

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__opexp_ga

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__opexp_nvm

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__opexp_total

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__opexp_vm

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__opexp_vo

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__pmt

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__upt

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__voms

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__vrh

calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__vrm

DAG

Legend (in order of precedence)

Resource type	Indicator	Resolution
Large table-materialized model	Orange	Make the model incremental
Large model without partitioning or clustering	Orange	Add partitioning and/or clustering
View with more than one child	Yellow	Materialize as a table or incremental
Incremental	Light green
Table	Green
View	White

[erikamov]

Looks great!

[vevetron]

There is mentioned in a resolved conversation - I'm okay with merging this in as is, but I don't like how we have all these columns listed out even at the cal-itp-data-infra.external_ntd__funding_and_expenses.historical__capital_expenditures_time_series__facilities stage - but realistically if we start using these tables more often, we can work on making it sustainable year after year at a later date. For now we can hard code it in.