Merge pull request #260 from SANDAG/ABM3_develop

Updating branch to most recent changes

Author: kelvinnguyenn

docs/design/demand/external.md

@@ -1,7 +1,42 @@
 # External Models
+The external aggregate travel models predict characteristics of US-SD and SD-US/MX travel behavior for all non-commercial, non-visitor vehicle trips and selected transit trips. Note that non-commercial MX-SD trips are forecast in the crossborder model, and non-commercial SD-US and SD-MX trips are forecast in the resident model.
 
-Details aggregate external models.
+![](../../images/design/external_external_county_cordons.png)
 
-## External Internal Model
+## External Model Estimation of Trip Counts by Type
+The total count of trips by production and attraction location was estimated in a series of steps:
 
-## External External Model
+1. The number of crossborder trips made by Mexican residents to attractions in San Diego was previously determined.
+2. The trips in the resident travel survey were expanded to estimate the total number of trips made by San Diego residents to attractions in Mexico.
+3. To derive an estimate of the number of US-MX trips, the number of MX-SD (1) and SD-MX (2) trips was subtracted from the total number of border-crossings.
+    * The distribution of US-MX trips among external stations on the US-side of San Diego County will be assumed to be proportional to the total volume at each external station, regardless of the point of entry at the Mexican border.
+4. The number of US-MX trips was then subtracted from the total number of trips in the SCAG cordon survey to arrive at an estimate of the combined total of US-US, US-SD, and SD-US trips with routes through San Diego County.
+5. Finally, the actual amounts of US-US, US-SD, and SD-US trips at each external station were estimated from the remaining trips (4) according to their proportions in the successfully geocoded responses in the SCAG cordon survey.
+
+## External Model Design Overview
+The behavioral characteristics of the different types of external trip were derived from the various data sources available as follows:
+
+* US-US trips: a fixed external station OD trip matrix was estimated from the SCAG cordon survey.
+    * */src/main/emme/toolbox/model/external_external.py*
+* US-MX trips: a fixed external station OD trip matrix was estimated from the SCAG cordon survey, Customs and Border Protection vehicle counts, and Mexican resident border-crossing survey as described in the previous section.
+    * */src/main/emme/toolbox/model/external_external.py*
+* US-SD trips: rates of vehicle trips per household for each external county were developed from the SCAG cordon survey, and the trips were distributed to locations in San Diego County according to a destination choice model estimated from the interregional survey.
+    * Intermediate stops and transit trips are not modeled in this segment due to the small contribution of these events to the total demand in the segment.
+    * */src/main/emme/toolbox/model/external_internal.py*
+
+### External-Internal Destination Choice Model
+The external-internal destination choice model distributes the EI trips to destinations within San Diego County. The EI destination choice model explanatory variables are:
+
+* Distance
+* The size of each sampled MGRA
+
+Diurnal and vehicle occupancy factors are then applied to the total daily trip tables to distribute the trips among shared ride modes and different times of day.
+
+### External-Internal Toll Choice Model
+The trips are then split among toll and non-toll paths according to a simplified toll choice model. The toll choice model included the following explanatory variables:
+
+* In-vehicle-time
+* Toll cost
+
+## Sources
+information primarily taken from this SANDAG document: [link to pdf](https://www.sandag.org/-/media/SANDAG/Documents/PDF/data-and-research/applied-research-and-performance-monitoring/surveys/appendix-t-SANDAG-travel-demand-model-documentation.pdf)

docs/design/report/report.md

@@ -1,3 +1,22 @@
 # Reporting Framework
 
-Details of reporting components.
+**Reporting Process Overview:**
+
+1. **ABM3 model output files are stored to data lake:**
+    - Model outputs are written to the data lake immediately following ABM3 model run completion.
+     - Output CSV files are converted to Parquet format before writing to the data lake.
+    - Each model run is assigned a unique scenario ID.
+
+2. **Data lake files are loaded to Delta tables:**
+    - Each output file in the data lake is loaded into its corresponding Delta table. For example, the trips output file is loaded into the trips Delta table, the persons output file is loaded into the persons Delta table, etc.
+    - Delta tables store the results from all model runs, organized by scenario ID.
+
+3. **Delta Tables are processed in Databricks:**
+    - Delta tables are read, transformed, and aggregated as needed to support analysis and reporting requirements.
+    - Once transformations are complete, the resulting data is written back to the data lake as new Delta tables or used to update existing tables.
+    - These new Delta tables are also organized by scenario ID, making it easier to manage and query specific versions of processed data.
+
+4. **Delta tables are ingested by Power BI:**
+    - Power BI reads the data from the Delta tables.
+    - Power BI report templates with various metrics of interest are automatically refreshed with new model run outputs.
+    - Metrics can easily be compared across different scenario IDs.

docs/design/supply/network-import-tned.md

@@ -1,3 +1,60 @@
 # Network Import from TNED
 
-Details of importing and processing the ETL network.
+This section describes the procedure by which the ABM3 model system imports (into Emme) network (highway and transit) files along with a general description of the different network files.
+
+## Network Files
+
+The ABM3 model system has been configured to be compatible with SANDAG's Transportation Network Editing Database (TNED) system, which is utilized to edit, maintain and generate transportation networks. The TNED network files, generated via an ETL (i.e., Extract, Tranform, Load) procedure, serve as inputs to the ABM3 model system's import network procedure and are produced in text file, shapefile, geodatabase table and geodatabase feature class geodatabase formats. There are, additionally, some non-TNED input network files which are manually maintained.
+
+The following are the required network files used during the Emme import network procedure:
+
+| **File**                       | **Source**          | **Description**                                                                              |
+|--------------------------------|---------------------|----------------------------------------------------------------------------------------------|
+| EMMEOutputs.gdb/TNED_HwyNet    | TNED                | Roadway network links                                                                        |
+| EMMEOutputs.gdb/TNED_HwyNodes  | TNED                | Roadway network nodes                                                                        |
+| EMMEOutputs.gdb/TNED_RailNet   | TNED                | Rail network links                                                                           |
+| EMMEOutputs.gdb/TNED_RailNodes | TNED                | Rail network nodes                                                                           |
+| EMMEOutputs.gdb/Turns          | TNED                | Turn prohibition records                                                                     |
+| special_fares.txt              | Manually Maintained | Special fares in terms of boarding and incremental in-vehicle costs                          |
+| timexfer_{time_of_day}.csv     | Manually Maintained | Timed transfer pairs of lines, by period. Where time_of_day refers to EA, AM, MD, PM, or EV. |
+| trrt.csv                       | TNED                | Attribute data (modes, headways) for the transit lines                                       |
+| trlink.csv                     | TNED                | Sequence of links (routing) for the transit lines                                            |
+| trstop.csv                     | TNED                | Stop data for the transit lines                                                              |
+| MODE5TOD.csv                   | Manually Maintained | Global (per-mode) transit cost and perception attributes                                     |
+| vehicle_class_toll_factors.csv | Manually Maintained | Factors to adjust the toll cost by facility name and class                                   |
+
+## Import Network Procedure
+
+This section describes the main steps carried out during the Emme import network procedure. The entire process is executed by the [import_network.py](https://github.com/SANDAG/ABM/blob/ABM3_develop/src/main/emme/toolbox/import/import_network.py) script.
+
+#### Create Modes
+
+This step creates the different combinations of traffic and transit modes that will get applied to the network links. A mode defines a group of vehicles or users which have access to the same parts of the network. Modes are used in both the traffic and transit assignments to define the available network for each class of demand. Each mode is uniquely identified by a single case-sensitive character. The modes which have access to a given link are listed on that link, and each link must allow at least one mode.
+
+#### Create Roadway Base Network
+
+This step creates the base roadway network by importing it from the EMMEOutputs.gdb/TNED_HwyNet and EMMEOutputs.gdb/TNED_HwyNodes. The nodes and links (referred to as the base network in Emme) for the traffic network are imported from the TNED_HwyNode and TNED_HwyNet geodatabase feature classes. The nodes are created first and the links connect between them. The I-node (from node, field AN) and J-node (to node, field BN) are used to associate the nodes and links and uniquely identify the link in the Emme network. Separate forward (AB) and reverse (BA) links are generated for links that have been coded as two-way. 
+
+#### Create Turns
+
+This step processes the EMMEOutputs.gdb/Turns input network file to generate turn restrictions by to- and from- link ID. If the indicated link IDs do not make a valid turn (links not adjacent) an error is reported.
+
+#### Calculate Traffic Attributes
+
+This step calculates derived traffic attributes. It utilizes the vehicle_class_toll_factors.csv to adjust toll costs by facility name and class.
+
+#### Check Zone Access
+
+This step verifies that every centroid has at least one available access and egress connector.
+
+#### Create Rail Base Network
+
+This step creates the base roadway network by importing it from the EMMEOutputs.gdb/TNED_RailNet and EMMEOutputs.gdb/TNED_RailNodes. The nodes and links (referred to as the base network in Emme) for the rail network are imported from the TNED_RailNode and TNED_RailNet geodatabase feature classes. The nodes are created first and the links connect between them. The I-node (from node, field AN) and J-node (to node, field BN) are used to associate the nodes and links and uniquely identify the link in the Emme network. Separate forward (AB) and reverse (BA) links are generated for links that have been coded as two-way. 
+
+#### Create Tranist Lines
+
+This step creates the transit lines by importing them from the trrt.csv, trlink.csv and trstop.csv input network files and matched to the transit base network. The mode-level attributes from MODE5TOD.csv, which vary by mode, are copied to transit line attributes and used in  transit assignment. It is in this step also where the timexfer_{time_of_day}.csv files are used to explicitly set route-to-route specific transfer transit times.
+
+#### Calculate Transit Attributes
+
+The transit line and stop / segment attributes (including fares) are imported to Emme attributes. The special_fares.txt lists network-level incremental fares by boarding (line and/or stop) and in-vehicle segment. They specify additive fares based on the network elements encountered on a transit journey and are used to represent the Coaster (or other) zonal fare system.

docs/design/supply/transit-skims-assign.md

@@ -1,3 +1,11 @@
 # Transit Skimming and Assignment
 
-Details of transit skimming and assignment.
+The transit assignment uses a headway-based approach, where the average headway between vehicle arrivals for each transit line is known, but not exact schedules. Passengers and vehicles arrive at stops randomly and passengers choose their travel itineraries considering the expected average waiting time.
+
+The Emme Extended transit assignment is based on the concept of optimal strategy but extended to support a number of behavioral variants. The optimal strategy is a set of rules which define sequence(s) of walking links, boarding and alighting stops which produces the minimum expected travel time (generalized cost) to a destination. At each boarding point the strategy may include multiple possible attractive transit lines with different itineraries. A transit strategy will often be a tree of options, not just a single path. A line is considered attractive if it reduces the total expected travel time by its inclusion. The demand is assigned to the attractive lines in proportion to their relative frequencies.
+
+The shortest "travel time" is a generalized cost formulation, including perception factors (or weights) on the different travel time components, along with fares, and other costs / perception biases such as transfer penalties which vary over the network and transit journey.
+
+The model has four access modes to transit (walk, park-and-ride (PNR), kiss-and-ride (KNR), and Transportation Network Company (TNC)) and three transit sets (local bus only, premium transit only, and local bus and premium transit sets), for 12 total demand classes by 5 TOD. These classes are assigned by slices, one at a time, to produce the total transit passenger flows on the network.
+
+While there are 12 slices of demand, there are only three classes of skims: Local bus only, premium only, and all modes. The access mode does not change the assignment parameters or skims.

docs/images/design/external_external_county_cordons.png

@@ -166,15 +166,7 @@ The table below contains brief descriptions of the input files required to execu
 | emp_non_ws_wfh | Non-wage and salary work from home employments | 
 | emp_non_ws_oth | Non-wage and salary other employments | 
 | emp_total | Total employment | 
-| pseudomsa | Pseudo MSA - | 
-|  | 1: Downtown | 
-|  | 2: Central | 
-|  | 3: North City | 
-|  | 4: South Suburban | 
-|  | 5: East Suburban | 
-|  | 6: North County West | 
-|  | 7: North County East | 
-|  | 8: East County | 
+| pseudomsa | Pseudo MSA<br>1: Downtown<br>2: Central<br>3: North City<br>4: South Suburban<br>5: East Suburban<br>6: North County West<br>7: North County East<br>8: East County | 
 | zip09 | 2009 Zip Code | 
 | enrollgradekto8 | Grade School K-8 enrollment | 
 | enrollgrade9to12 | Grade School 9-12 enrollment | 
@@ -184,12 +176,11 @@ The table below contains brief descriptions of the input files required to execu
 | parkactive | Acres of Active Park | 
 | openspaceparkpreserve | Acres of Open Park or Preserve | 
 | beachactive | Acres of Active Beach | 
-| district27 |  | 
+| district27 | Special layer reg employer shuttle service around Sorrento Valley | 
 | milestocoast | Distance (miles) to the nearest coast | 
 | acres | Total acres in the mgra (used in CTM) | 
 | land_acres | Acres of land in the mgra (used in CTM) | 
 | effective_acres | Effective acres in the mgra (used in CTM) | 
-| truckregiontype |  | 
 | exp_hourly | Expected hourly prking cost | 
 | exp_daily | Expected daily prking cost | 
 | exp_monthly | Expected monthly prking cost | 
@@ -335,14 +326,22 @@ The table below contains brief descriptions of the input files required to execu
 | pnum                 | Person Number                                                                                 |
 | age                  | Age of person                                                                                 |
 | sex                  | Gender of person<br>1 = Male<br>2 = Female                                                     |
-| military             | Military status of person:<br>0 = N/A Less than 17 Years Old<br>1 = Yes, Now on Active Duty    |
+| miltary             | Military status of person:<br>0 = N/A Less than 17 Years Old<br>1 = Yes, Now on Active Duty    |
 | pemploy              | Employment status of person:<br>1 = Employed Full-Time<br>2 = Employed Part-Time<br>3 = Unemployed or Not in Labor Force<br>4 = Less than 16 Years Old |
 | pstudent             | Student status of person:<br>1 = Pre K-12<br>2 = College Undergrad+Grad and Prof. School<br>3 = Not Attending School |
 | ptype                | Person type:<br>1 = Full-time Worker<br>2 = Part-time Worker<br>3 = College Student<br>4 = Non-working Adult<br>5 = Non-working Senior<br>6 = Driving Age Student<br>7 = Non-driving Student<br>8 = Pre-school |
 | educ                 | Educational attainment:<br>1 = No schooling completed<br>9 = High school graduate<br>13 = Bachelor's degree |
 | grade                | School grade of person:<br>0 = N/A (not attending school)<br>2 = K to grade 8<br>5 = Grade 9 to grade 12<br>6 = College undergraduate |
 | occen5               | Occupation:<br>0 = Not in universe (Under 16 years or LAST-WRK = 2)<br>1..997 = Legal census occupation code |
 | occsoc5              | Detailed occupation codes defined by the Bureau of Labor Statistics |
+| indcen              | Industry code.<br>0 = default<br>9970 = NAICS2 is MIL |
+| weeks              | Weeks worked during past 12 months<br>0 .N/A (less than 16 years old/did not work during the past 12 .months)<br>1 .50 to 52 weeks worked during past 12 months<br>2 .48 to 49 weeks worked during past 12 months<br>3 .40 to 47 weeks worked during past 12 months<br>4 .27 to 39 weeks worked during past 12 month<br>5 .14 to 26 weeks worked during past 12 months<br>6 .13 weeks or less worked during past 12 months|
+| hours              | Hours worked per week past 12 months<br>0 .N/A (less than 16 years old/did not work during the past .12 months)<br>1..98 .1 to 98 usual hours<br>99 .99 or more usual hours |
+| rac1p              | Race:<br>1 = White alone<br>2 = Black or African American alone<br>3 = American Indian alone<br>4 = Alaska Native alone<br>5 = American Indian and Alaska Native tribes specified; or .American Indian or Alaska Native, not specified and no other .races<br>6 = Asian alone<br>7 = Native Hawaiian and Other Pacific Islander alone<br>8 = Some Other Race alone<br>9 =Two or More Races |
+| hisp              | Hispanic origin:<br>1 = Not Hispanic<br>2 = Hispanic |
+| version              | Synthetic population run version. Presently set to 0. |
+| naics2_original_code              | 2 digit North American Industry Classification System (NAICS) |
+| soc2              | 2 digit Standard Occupational Classification |
 
 
 <a id="vehicle_class_toll"></a>