BASIS data updates

[theocharides]

Code changes to update BAUS input data to 2020

Structural input data updates

• Removed the model's preprocessing.py, since inputs are now stable post-BASIS data processing
  • Ensured any data edits previously maintained in the model are moved into BASIS as needed and remove code
  • Ensured job assignment to buildings is moved into BASIS and remove code
  • Ensured any validation of the input datasets is moved into BASIS and remove code
  • Removed the orca tables for "preprocessed" versions since we will only have one version of each going forward
• Removed the model's base year set of sub-models, since the base year is not a simulation year
  • Retained the base year model summaries sub-models, for easy comparison against future year summaries
• Refreshed the model's data store
  • Maintained the new core datasets as csvs (parcels, buildings, residential_units, jobs, households) and also as an HDF5 file, packaged in this file format for runtime use only
  • Removed configs/paths.yaml which only held the path to find the h5 for use in the code, and store the path in datasources.py as is done with the other datasets
  • Removed code that used Amazon s3 storage since we'll store data in BASIS now
• Removed mapping.py
  • Moved building mapping into developer_settings.yaml
  • Other mappings like naics_to_empsix are no longer needed since we're using empsix throughout the inputs and code
• Added residential units table with assigned households to inputs
  • Removed the runtime code that was creating residential units and assigning households to them
  • Add this table to the model's datastore
• Added deed restricted units to inputs
  • Removed the runtime code adding this to buildings
  • Remove the zonal final used in the model to assign deed restricted units, since this now happens in BASIS at the disaggregate level

Dataset specific related updates

• Removed code no longer needed with new parcels dataset
  • geom_id_to_parcel_id and parcel_id_to_geom_id functions and use of them
• Brought in the single table for zoning_existing parcel level information, removing zoning_lookup and zoning_parcels
• Updated the parcels join key on other tables that are not being updated with this base year data update, such as zoning_strategy
• Updated parcels_geography table to growth_geographies table, removing BAUS edits and columns that are held in the inputs, such as juris_tra (concatenated juris + tra column)
• Updated development projects table
  • remove the reprocess_dev_projects function since that shifts duplicate projects to "add" to the parcel, do this on the inputs side (this also create a fourth dev projects function making it harder to track what happens to these projects)
  • Moved the editing of this table into the BASIS side as well
  • Pulled out all strategy projects, which will still be implemeted in the same but but will be store in inputs/plan_strategies
• Updated nodev
  • Removed manual_nodev and parcels_rejections (which it reads in), since these should be fed in through BASIS if carried forward
  • Transition nodev code from a collection of sources (zoning, static parcels, sea level rise parcels), to columns that can be pulled from the BASIS nodev table:
    • Removed "nodev from zoning" which @yuqiww found is a duplication of the nodev parcel column
    • The nodev column now only reads from the BASIS nodev table, then adds "static parcels" (including Institutions) to the list.
  • Relatedly, transitioned the parcel_filter for parcel development consideration from a collection of sources (nodev, pipeline projects, rules on which parcel/building types can be developed), to columns that can be pulled from the BASIS nodev table.
    • Removed filters on building age, small-lot single family parcels, ad building type
    • The parcel filter now only filters out nodev parcels and development pipeline projects
  • Moved the preprocessing.py logic that makes parking building types off limits in to nodev as well
• Updated calculate_vmt_fees and calculate_jobs_housing_fees which use parcels and other geography columns

---

• To see the specific tasks where the Asana app for GitHub is being used, see below:
  • https://app.asana.com/0/0/1204291927873726