Project Meeting 2025.03.06

Agenda

• Technical
  • Pandas 2.0 update
  • PopulationSim scope
  • Sub-Model Preprocessing scope

Technical

• Pandas 2.0 update
  • Update from last year made software slower
  • Spent last week researching why it is slower
  • Run time for processing a lot of columns takes a lot of extra time causing long run times
  • Next release of Pandas will have a fix for part of the slowdown, but not all of it
  • Jeff is creating a script to band-aid (sub-optimal solution) the fix (sub-optimal solution), but may break if underlying data structures change
    • Code may need to constantly updated
    • Desirable change to move to PyArrow for better compatibility with Pandas, but it will be a high-effort change
  • Documentation also requires use of Pandas 2.0
  • Next step - spend a few hours to see if there is any better solution
• Population Sim
  • Issue: Right now PopulationSim uses ActivitySim as dependency, but only version that works is 1.1.3 which is out of date
    • Clarification: Purpose of task is create one install
      • Do we keep them decouple them or make PopulationSim a dependency of ActivitySim; Should it be an optional install?
        • If PopulationSim is a dependency of ActivitySim then it has to always remain consistent with ActivitySim
      • Decision: Keep them separate
  • Three tasks: Implement CI testing; Decouple PopulationSim from ActivitySim; Update package dependencies
  • For these tasks, can move forward with making PopulationSim stand-up on its alone and then decision can be made later on about how it interacts with ActivitySim
• Sub-Model Preprocessing scope
  • Not all models in ActivitySim have a preprocessor or postprocessor
    • pre-processor is useful for calculations that are going to be repeated numerous times or are complex, clarity, and better efficiency in Sharrow
    • Annotate post-processor is useful to derive and append attributes based on the model's output
    • Lack of consistency between models may cause confusion and impede implementing some features
  • Goal is to create a consistent framework that provides preprocessing and annotate postprocessing functionality to all sub-models in the ActivitySim code system
  • First step: List of all sub-models and note whether they have a preprocessor and annotate postprocessor
• Tuesday's meeting is cancelled