Re-estimation (#3)

* pydantic for estimation settings * allow df as type in config * fix table_info * auto ownership * repair for pydantic * update for ruff * updated for simple models * repair for Pydantic * simple simulate and location choice * df is attribute * scheduling * stop freq * test locations * cdap * nonmand_and_joint_tour_dest_choice * nonmand_tour_freq * fix ci to stop using mamba * test updates * use larch6 from pip * use numba for stop freq * fix for pandas 1.5 * fix stop freq test for numba * Sharrow Cache Dir Setting (ActivitySim#893) * setting necessary filesystem changes from settings file * set for multiprocessing * repair github actions * github action updates (ActivitySim#903) * script to make data * unified script for making data * remove older * bug * doc note * load from parquet if available * add original alt ids to EDB output when using compact * fix MP race * script arg to skip to EDB * clean up CDAP and blacken * refactor model_estimation_table_types change to estimation_table_types, to avoid pydantic namespace clash * repair drop_dupes * blacken * location choice with compact * choice_def for compact * spec changes for simple-simulate * re-estimation demo for auto ownership * clean up status messages * change name to stop pydantic warnings * edit configs * default estimation sample size is same as regular sample size * allow location alts not in cv format * dummy zones for location choice * update scheduling model estimation * various cleanup * stop freq * tidy build script * update 02 school location for larger example * update notebook 04 * editable model re-estimation for location choice * fix test names * update notebooks * cdap print filenames as loading * notebook 07 * tests thru 07 * notebooks 08 09 * build the data first * runnable script * change larch version dependency * keep pandas<2 * notebooks 10 11 * notebook 12 * remove odd print * add matplotlib * notebook 13 14 * test all the notebooks * add xlsxwriter to tests * notebook 15 * CDAP revise model spec demo * notebook 16 * notebook 17 * longer timeout * notebook 18 * notebook 19 * notebook 20 * smaller notebook 15 * configurable est mode setup * notebook 21 * notebook 22 * config sample size in GA * notebook 23 * updates for larch and graphviz * change default to compact * compare model 03 * test updates * rename test targets * repair_av_zq * move doctor up * add another repair * oops --------- Co-authored-by: David Hensle <[email protected]>
RSGInc · Dec 9, 2024 · 3b4974c · 3b4974c
1 parent 87b414f
commit 3b4974c
Show file tree

Hide file tree

Showing 127 changed files with 56,077 additions and 42,296 deletions.
diff --git a/.github/workflows/branch-docs.yml b/.github/workflows/branch-docs.yml
@@ -15,7 +15,7 @@ jobs:
       run:
         shell: bash -l {0}
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
         with:
           fetch-depth: 0  # get all tags, lets setuptools_scm do its thing
 

diff --git a/.github/workflows/build_installer.yml b/.github/workflows/build_installer.yml
@@ -26,7 +26,7 @@ jobs:
 
     steps:
     - name: Checkout code
-      uses: actions/checkout@v3
+      uses: actions/checkout@v4
 
     - uses: conda-incubator/setup-miniconda@v3
       with:
@@ -47,7 +47,7 @@ jobs:
       shell: bash
 
     - name: Upload installer to Github artifact
-      uses: actions/upload-artifact@v3
+      uses: actions/upload-artifact@v4
       with:
         path: build/Activitysim-${{ inputs.version }}-${{ matrix.OS_NAME }}-${{ matrix.ARCH }}*
         name: Activitysim-${{ inputs.version }}-${{ matrix.OS_NAME }}-${{ matrix.ARCH }}

diff --git a/.github/workflows/core_tests.yml b/.github/workflows/core_tests.yml
@@ -26,13 +26,11 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
-      - name: Setup Mambaforge
+      - name: Setup Miniforge
         uses: conda-incubator/setup-miniconda@v3
         with:
-            miniforge-variant: Mambaforge
             miniforge-version: latest
             activate-environment: asim-test
-            use-mamba: true
             python-version: ${{ matrix.python-version }}
 
       - name: Set cache date for year and month
@@ -46,7 +44,7 @@ jobs:
 
       - name: Update environment
         run: |
-          mamba env update -n asim-test -f conda-environments/github-actions-tests.yml
+          conda env update -n asim-test -f conda-environments/github-actions-tests.yml
         if: steps.cache.outputs.cache-hit != 'true'
 
       - name: Install activitysim
@@ -58,8 +56,8 @@ jobs:
 
       - name: Conda checkup
         run: |
-          mamba info -a
-          mamba list
+          conda info -a
+          conda list
 
       - name: Lint with Black
         run: |
@@ -111,13 +109,11 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
-      - name: Setup Mambaforge
+      - name: Setup Miniforge
         uses: conda-incubator/setup-miniconda@v3
         with:
-            miniforge-variant: Mambaforge
             miniforge-version: latest
             activate-environment: asim-test
-            use-mamba: true
             python-version: ${{ matrix.python-version }}
 
       - name: Set cache date for year and month
@@ -131,7 +127,7 @@ jobs:
 
       - name: Update environment
         run: |
-          mamba env update -n asim-test -f conda-environments/github-actions-tests.yml
+          conda env update -n asim-test -f conda-environments/github-actions-tests.yml
         if: steps.cache.outputs.cache-hit != 'true'
 
       - name: Install activitysim
@@ -143,8 +139,8 @@ jobs:
 
       - name: Conda checkup
         run: |
-          mamba info -a
-          mamba list
+          conda info -a
+          conda list
 
       - name: Lint with Black
         run: |
@@ -194,13 +190,11 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
-      - name: Setup Mambaforge
+      - name: Setup Miniforge
         uses: conda-incubator/setup-miniconda@v3
         with:
-          miniforge-variant: Mambaforge
           miniforge-version: latest
           activate-environment: asim-test
-          use-mamba: true
           python-version: ${{ env.python-version }}
 
       - name: Set cache date for year and month
@@ -214,7 +208,7 @@ jobs:
 
       - name: Update environment
         run: |
-          mamba env update -n asim-test -f conda-environments/github-actions-tests.yml
+          conda env update -n asim-test -f conda-environments/github-actions-tests.yml
         if: steps.cache.outputs.cache-hit != 'true'
 
       - name: Install activitysim
@@ -226,8 +220,8 @@ jobs:
 
       - name: Conda checkup
         run: |
-          mamba info -a
-          mamba list
+          conda info -a
+          conda list
 
       # TODO: Cache sharrow compiled flows?  The contents of __pycache__ appear to
       #       be ignored, so this is not working as expected right now
@@ -282,13 +276,11 @@ jobs:
       - name: Checkout ActivitySim
         uses: actions/checkout@v4
 
-      - name: Setup Mambaforge
+      - name: Setup Miniforge
         uses: conda-incubator/setup-miniconda@v3
         with:
-          miniforge-variant: Mambaforge
           miniforge-version: latest
           activate-environment: asim-test
-          use-mamba: true
           python-version: ${{ env.python-version }}
 
       - name: Set cache date for year and month
@@ -304,7 +296,7 @@ jobs:
 
       - name: Update environment
         run: |
-          mamba env update -n asim-test -f conda-environments/github-actions-tests.yml
+          conda env update -n asim-test -f conda-environments/github-actions-tests.yml
         if: steps.cache.outputs.cache-hit != 'true'
 
       - name: Install activitysim
@@ -316,8 +308,8 @@ jobs:
 
       - name: Conda checkup
         run: |
-          mamba info -a
-          mamba list
+          conda info -a
+          conda list
 
       - name: Checkout Example
         uses: actions/checkout@v4
@@ -345,13 +337,11 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
-      - name: Setup Mambaforge
+      - name: Setup Miniforge
         uses: conda-incubator/setup-miniconda@v3
         with:
-          miniforge-variant: Mambaforge
           miniforge-version: latest
           activate-environment: asim-test
-          use-mamba: true
           python-version: ${{ env.python-version }}
 
       - name: Set cache date for year and month
@@ -365,7 +355,7 @@ jobs:
 
       - name: Update environment
         run: |
-          mamba env update -n asim-test -f conda-environments/github-actions-tests.yml
+          conda env update -n asim-test -f conda-environments/github-actions-tests.yml
         if: steps.cache.outputs.cache-hit != 'true'
 
       - name: Install activitysim
@@ -377,8 +367,8 @@ jobs:
 
       - name: Conda checkup
         run: |
-          mamba info -a
-          mamba list
+          conda info -a
+          conda list
 
       - name: Test Random Seed Generation
         run: |
@@ -392,18 +382,16 @@ jobs:
     defaults:
       run:
         shell: bash -l {0}
-    name: estimation_mode_test
+    name: Estimation Mode Unit Tests
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
 
-      - name: Setup Mambaforge
+      - name: Setup Miniforge
         uses: conda-incubator/setup-miniconda@v3
         with:
-          miniforge-variant: Mambaforge
           miniforge-version: latest
           activate-environment: asim-test
-          use-mamba: true
           python-version: ${{ env.python-version }}
 
       - name: Set cache date for year and month
@@ -417,11 +405,11 @@ jobs:
 
       - name: Update environment
         run: |
-          mamba env update -n asim-test -f conda-environments/github-actions-tests.yml
+          conda env update -n asim-test -f conda-environments/github-actions-tests.yml
         if: steps.cache.outputs.cache-hit != 'true'
 
-      - name: Install Larch
-        run: mamba install "larch>=5.7.1"
+      - name: Install Larch v6
+        run: python -m pip install larch6
 
       - name: Install activitysim
         # installing without dependencies is faster, we trust that all needed dependencies
@@ -432,14 +420,75 @@ jobs:
 
       - name: Conda checkup
         run: |
-          mamba info -a
-          mamba list
+          conda info -a
+          conda list
 
       - name: Test Estimation Mode
         run: |
           python -m pytest activitysim/estimation/test/test_larch_estimation.py --durations=0
-  
-
+
+  estimation_notebooks:
+    needs: foundation
+    env:
+      python-version: "3.10"
+      label: linux-64
+    defaults:
+      run:
+        shell: bash -l {0}
+    name: Estimation Notebooks Test
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Setup Miniforge
+        uses: conda-incubator/setup-miniconda@v3
+        with:
+          miniforge-version: latest
+          activate-environment: asim-test
+          python-version: ${{ env.python-version }}
+
+      - name: Set cache date for year and month
+        run: echo "DATE=$(date +'%Y%m')" >> $GITHUB_ENV
+
+      - uses: actions/cache@v4
+        with:
+          path: ${{ env.CONDA }}/envs
+          key: ${{ env.label }}-conda-${{ hashFiles('conda-environments/github-actions-tests.yml') }}-${{ env.DATE }}-${{ env.CACHE_NUMBER }}
+        id: cache
+
+      - name: Update environment
+        run: |
+          conda env update -n asim-test -f conda-environments/github-actions-tests.yml
+        if: steps.cache.outputs.cache-hit != 'true'
+
+      - name: Install Graphviz
+        uses: ts-graphviz/setup-graphviz@v2
+
+      - name: Install Larch v6
+        run: python -m pip install larch6 "pandas<2" pydot
+
+      - name: Install activitysim
+        # installing without dependencies is faster, we trust that all needed dependencies
+        # are in the conda environment defined above.  Also, this avoids pip getting
+        # confused and reinstalling tables (pytables).
+        run: |
+          python -m pip install -e . --no-deps
+
+      - name: Conda checkup
+        run: |
+          conda info -a
+          conda list
+
+      - name: Create Estimation Data
+        run: |
+          python activitysim/examples/example_estimation/notebooks/est_mode_setup.py --household_sample_size 5000
+
+      - name: Test Estimation Notebooks
+        run: |
+          python -m pytest activitysim/examples/example_estimation/notebooks/*.ipynb \
+            --nbmake-timeout=3000 \
+            --ignore=activitysim/examples/example_estimation/notebooks/01_estimation_mode.ipynb
+
   estimation_edb_creation:
     needs: foundation
     env:
@@ -453,13 +502,11 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
-      - name: Setup Mambaforge
+      - name: Setup Miniforge
         uses: conda-incubator/setup-miniconda@v3
         with:
-          miniforge-variant: Mambaforge
           miniforge-version: latest
           activate-environment: asim-test
-          use-mamba: true
           python-version: ${{ env.python-version }}
 
       - name: Set cache date for year and month
@@ -473,7 +520,7 @@ jobs:
 
       - name: Update environment
         run: |
-          mamba env update -n asim-test -f conda-environments/github-actions-tests.yml
+          conda env update -n asim-test -f conda-environments/github-actions-tests.yml
         if: steps.cache.outputs.cache-hit != 'true'
 
       - name: Install activitysim
@@ -485,8 +532,8 @@ jobs:
 
       - name: Conda checkup
         run: |
-          mamba info -a
-          mamba list
+          conda info -a
+          conda list
 
       - name: Test Estimation EDB Creation
         run: |
@@ -505,15 +552,13 @@ jobs:
         with:
           fetch-depth: 0  # get all tags, lets setuptools_scm do its thing
       - name: Set up Python 3.10
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v5
         with:
           python-version: "3.10"
       - name: Install dependencies
         uses: conda-incubator/setup-miniconda@v3
         with:
-          miniforge-variant: Mambaforge
           miniforge-version: latest
-          use-mamba: true
           environment-file: conda-environments/docbuild.yml
           python-version: "3.10"
           activate-environment: docbuild