-
Notifications
You must be signed in to change notification settings - Fork 113
Agency Visualization Requirements
This is part of Task 4 of the Phase 6B work plan.
Stakeholders: Please fill out this survey of the existing tools and workflows in use at your agency. We are interested in how you manage your model runs, what tools you use when analyze the results of those runs, and what capabilities you particularly like or particularly feel are missing/need improvement.
After your agency has filled out the document, we will follow up with each of you for a brief interview or "show and tell" so we can better understand what is working and what is missing.
Please consider the use cases mentioned in the task order when you are answering:
- Organize/access model results
- Model debugging
- Calibration/validation
- Project analysis
- Scenario comparisons
- Results presentation
This is all going into one Wiki document so you can all see each other's responses: but PLEASE only edit the section for your own agency! Thanks! ✨
1. Run management. How do you currently manage model run inputs and outputs? Think about the different contexts such as model development, calibration/validation, and project-based model application work.
2. Visualization now: which tools do you currently use for visualizing model outputs? Do you have proprietary packages, agency-written scripts/tools, etc. Is any of it web-based?
3. Analysis needs: what do you feel is currently missing from your analyst visualization toolkit? Are there things you can't do at all that you need? Are there things you can do, but they are onerous or annoying or difficult?
4. Outward facing visualization: Stepping away from an internal analyst role and thinking about outward presentation, what would you need to help convey model outputs? What tools do you use to get from model outputs to Board, TRB presentations, and to the public?
5. At your agency, is a common viz toolkit something that would be added on top of existing tools that you already love, or more likely to be something that replaces existing workflows (if any)? We ask because the task clearly states that the consortium will coalesce around a common set of tools, and that will require changes for all of you!
1. Run management. How do you currently manage model run inputs and outputs? Think about the different contexts such as model development, calibration/validation, and project-based model application work.
2. Visualization now: which tools do you currently use for visualizing model outputs? Do you have proprietary packages, agency-written scripts/tools, etc. Is any of it web-based?
3. Analysis needs: what do you feel is currently missing from your analyst visualization toolkit? Are there things you can't do at all that you need? Are there things you can do, but they are onerous or annoying or difficult?
4. Outward facing visualization: Stepping away from an internal analyst role and thinking about outward presentation, what would you need to help convey model outputs? What tools do you use to get from model outputs to Board, TRB presentations, and to the public?
5. At your agency, is a common viz toolkit something that would be added on top of existing tools that you already love, or more likely to be something that replaces existing workflows (if any)? We ask because the task clearly states that the consortium will coalesce around a common set of tools, and that will require changes for all of you!
1. Run management. How do you currently manage model run inputs and outputs? Think about the different contexts such as model development, calibration/validation, and project-based model application work.
2. Visualization now: which tools do you currently use for visualizing model outputs? Do you have proprietary packages, agency-written scripts/tools, etc. Is any of it web-based?
3. Analysis needs: what do you feel is currently missing from your analyst visualization toolkit? Are there things you can't do at all that you need? Are there things you can do, but they are onerous or annoying or difficult?
4. Outward facing visualization: Stepping away from an internal analyst role and thinking about outward presentation, what would you need to help convey model outputs? What tools do you use to get from model outputs to Board, TRB presentations, and to the public?
5. At your agency, is a common viz toolkit something that would be added on top of existing tools that you already love, or more likely to be something that replaces existing workflows (if any)? We ask because the task clearly states that the consortium will coalesce around a common set of tools, and that will require changes for all of you!
1. Run management. How do you currently manage model run inputs and outputs? Think about the different contexts such as model development, calibration/validation, and project-based model application work.
MWCOG currently manages model run inputs and outputs in file folders. The ActivitySim-based Gen3 Model that is currently under development will likely adopt this system. Cube does provide a scenario manager but MWCOG staff intentionally do not use it because of the added complexities.
2. Visualization now: which tools do you currently use for visualizing model outputs? Do you have proprietary packages, agency-written scripts/tools, etc. Is any of it web-based?
Currently, MWCOG mainly uses ArcGIS to visualize outputs from our trip-based Gen2 Model. We plan to adopt a R-based ABM Visualizer developed by RSG for the ActivitySim-based Gen3 Model.
3. Analysis needs: what do you feel is currently missing from your analyst visualization toolkit? Are there things you can't do at all that you need? Are there things you can do, but they are onerous or annoying or difficult?
Gen3 Model is currently at the Phase 1 Model Deployment stage. We will further evaluate ABM Visualizer when we gain more hands-on experiences with the model and the visualization tool.
4. Outward facing visualization: Stepping away from an internal analyst role and thinking about outward presentation, what would you need to help convey model outputs? What tools do you use to get from model outputs to Board, TRB presentations, and to the public?
ABM Visualizer was mainly designed for internal use. We are interested in visualization tools that can convey model outputs (e.g., outputs from a race-related equity analysis) in less technical and more straightforward terms to the Board or the public.
5. At your agency, is a common viz toolkit something that would be added on top of existing tools that you already love, or more likely to be something that replaces existing workflows (if any)? We ask because the task clearly states that the consortium will coalesce around a common set of tools, and that will require changes for all of you!
At the early stage of Gen3 Model development, our agency is open to either option.
1. Run management. How do you currently manage model run inputs and outputs? Think about the different contexts such as model development, calibration/validation, and project-based model application work.
2. Visualization now: which tools do you currently use for visualizing model outputs? Do you have proprietary packages, agency-written scripts/tools, etc. Is any of it web-based?
3. Analysis needs: what do you feel is currently missing from your analyst visualization toolkit? Are there things you can't do at all that you need? Are there things you can do, but they are onerous or annoying or difficult?
4. Outward facing visualization: Stepping away from an internal analyst role and thinking about outward presentation, what would you need to help convey model outputs? What tools do you use to get from model outputs to Board, TRB presentations, and to the public?
5. At your agency, is a common viz toolkit something that would be added on top of existing tools that you already love, or more likely to be something that replaces existing workflows (if any)? We ask because the task clearly states that the consortium will coalesce around a common set of tools, and that will require changes for all of you!
1. Run management. How do you currently manage model run inputs and outputs? Think about the different contexts such as model development, calibration/validation, and project-based model application work.
2. Visualization now: which tools do you currently use for visualizing model outputs? Do you have proprietary packages, agency-written scripts/tools, etc. Is any of it web-based?
3. Analysis needs: what do you feel is currently missing from your analyst visualization toolkit? Are there things you can't do at all that you need? Are there things you can do, but they are onerous or annoying or difficult?
4. Outward facing visualization: Stepping away from an internal analyst role and thinking about outward presentation, what would you need to help convey model outputs? What tools do you use to get from model outputs to Board, TRB presentations, and to the public?
5. At your agency, is a common viz toolkit something that would be added on top of existing tools that you already love, or more likely to be something that replaces existing workflows (if any)? We ask because the task clearly states that the consortium will coalesce around a common set of tools, and that will require changes for all of you!
1. Run management. How do you currently manage model run inputs and outputs? Think about the different contexts such as model development, calibration/validation, and project-based model application work.
Inputs that may vary by scenario, which are typically land use and network files, are stored on a network drive in folders that are uniquely named for a given scenario. Land use and networks are stored as unique scenarios so that a given network can be run with any land use and vise versa (constrained by forecast year). The model config file has five parameters that must be set: inputs_dir, forecast_year, base_year, landuse_inputs & network_inputs. During a model run, these files are automatically copied to the model run location. Static files that do not change for a given scenario are stored in a sqlite database, which is accessed by the model throughout a run.
Outputs are stored in folders within the model run.
2. Visualization now: which tools do you currently use for visualizing model outputs? Do you have proprietary packages, agency-written scripts/tools, etc. Is any of it web-based?
We create pre-aggregated .csv outputs that are controlled by an expression file. These files are then used to create automated Jupyter notebook html files. So when a run is complete, there should be a bunch of html pages for various summaries, including base year validation comparisons. We can also visualize a run using our Plotly Dash dashboard, which uses the same .csv files and runs on an aws linux instance. The dashboard is a fully interactive and web-based app, so it offers a centralized, easily accessible place to examine/compare various scenarios.
3. Analysis needs: what do you feel is currently missing from your analyst visualization toolkit? Are there things you can't do at all that you need? Are there things you can do, but they are onerous or annoying or difficult?
Visually tracing/mapping a person/household tours and trips is something that is missing from our visualization toolkit. We can do it, but it is not automated so it is pretty onerous.
4. Outward facing visualization: Stepping away from an internal analyst role and thinking about outward presentation, what would you need to help convey model outputs? What tools do you use to get from model outputs to Board, TRB presentations, and to the public?
5. At your agency, is a common viz toolkit something that would be added on top of existing tools that you already love, or more likely to be something that replaces existing workflows (if any)? We ask because the task clearly states that the consortium will coalesce around a common set of tools, and that will require changes for all of you!
Change is fine, but I think it is important that a run automatically creates summaries that are immediately accessible and I think Jupyter notebooks--> html outputs makes a lot of sense for this. I would say the majority of our runs are looked at quickly and then deleted, so they often do not make it to our dashboard. We like Plotly Dash for web-based dashboards and have a lot of experience with it, but we are very open to other frameworks. Shiny is used quite heavily by our organization and I see a lot of positives around it.
1. Run management. How do you currently manage model run inputs and outputs? Think about the different contexts such as model development, calibration/validation, and project-based model application work.
2. Visualization now: which tools do you currently use for visualizing model outputs? Do you have proprietary packages, agency-written scripts/tools, etc. Is any of it web-based?
3. Analysis needs: what do you feel is currently missing from your analyst visualization toolkit? Are there things you can't do at all that you need? Are there things you can do, but they are onerous or annoying or difficult?
4. Outward facing visualization: Stepping away from an internal analyst role and thinking about outward presentation, what would you need to help convey model outputs? What tools do you use to get from model outputs to Board, TRB presentations, and to the public?
5. At your agency, is a common viz toolkit something that would be added on top of existing tools that you already love, or more likely to be something that replaces existing workflows (if any)? We ask because the task clearly states that the consortium will coalesce around a common set of tools, and that will require changes for all of you!
1. Run management. How do you currently manage model run inputs and outputs? Think about the different contexts such as model development, calibration/validation, and project-based model application work.
2. Visualization now: which tools do you currently use for visualizing model outputs? Do you have proprietary packages, agency-written scripts/tools, etc. Is any of it web-based?
3. Analysis needs: what do you feel is currently missing from your analyst visualization toolkit? Are there things you can't do at all that you need? Are there things you can do, but they are onerous or annoying or difficult?
4. Outward facing visualization: Stepping away from an internal analyst role and thinking about outward presentation, what would you need to help convey model outputs? What tools do you use to get from model outputs to Board, TRB presentations, and to the public?
5. At your agency, is a common viz toolkit something that would be added on top of existing tools that you already love, or more likely to be something that replaces existing workflows (if any)? We ask because the task clearly states that the consortium will coalesce around a common set of tools, and that will require changes for all of you!
1. Run management. How do you currently manage model run inputs and outputs? Think about the different contexts such as model development, calibration/validation, and project-based model application work.
2. Visualization now: which tools do you currently use for visualizing model outputs? Do you have proprietary packages, agency-written scripts/tools, etc. Is any of it web-based?
3. Analysis needs: what do you feel is currently missing from your analyst visualization toolkit? Are there things you can't do at all that you need? Are there things you can do, but they are onerous or annoying or difficult?
4. Outward facing visualization: Stepping away from an internal analyst role and thinking about outward presentation, what would you need to help convey model outputs? What tools do you use to get from model outputs to Board, TRB presentations, and to the public?
5. At your agency, is a common viz toolkit something that would be added on top of existing tools that you already love, or more likely to be something that replaces existing workflows (if any)? We ask because the task clearly states that the consortium will coalesce around a common set of tools, and that will require changes for all of you!