V1 Refactor RC

.github/.github/pull_request_template.md

@@ -1,27 +1,25 @@
-## What existing problem does the pull request solve and why should we include it?
+# Pull Request
 
+## What existing problem does the pull request solve and why should we include it?
 
 ## What is the testing plan?
 
 *Demonstrate the code is solid by discussing how results are verified and covered by tests*
 
- - [ ] Code for this PR is covered in tests 
- - [ ] Code passes all existing tests
+- [ ] Code for this PR is covered in tests
+- [ ] Code passes all existing tests
 
 ## Code formatting
+
 *Code should be PEP8 compliant before merging by running a package like [`black`](https://pypi.org/project/black/)*
 
- - [ ] Code linted
+- [ ] Code linted
 
 ## Applicable Issues
+
 *Please do not create a Pull Request without creating an issue first.*
 
 *Put `closes #XXXX` in your comment to auto-close the issue that your PR fixes.*
 
-
-#### Issues List
-
- -  closes...
- -  closes...
-
-
+- closes...
+- closes...

.github/ISSUE_TEMPLATE/bug_report.md

@@ -8,30 +8,33 @@ assignees: ''
 ---
 
 ## Describe the bug
-A clear and concise description of what the bug is or the error code you got. e.g. 
-```python 
+
+A clear and concise description of what the bug is or the error code you got. e.g.
+
+```python
 KeyError: 'Passing list-likes to .loc or [] with any missing labels is no longer supported, see https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#deprecate-loc-reindex-listlike'
 ```
 
 ## To Reproduce
+
 Steps to reproduce the behavior:
+
 1. Go to '...'
 2. Click on '....'
 3. Scroll down to '....'
 4. See error
 
 ### Failing tests
+
 - [ ] No applicable test failed, need to create.
-- [ ] 
+- [ ]
 
 ### Triggering line of code
 
-
 ### Thoughts on resolution
 
 ### Full stack trace
 
-
 ### Environment
 
 Operating system:

.github/ISSUE_TEMPLATE/feature_request.md

@@ -8,25 +8,30 @@ assignees: ''
 ---
 
 ### User Story
+
 *As a ...insert type of user... I'd like to ...insert desired feature or behavior...*
 
-### Priority 
+### Priority
 
 ### Level of Effort
 
 ### Resolution Ideas
 
 ### Project
+
 *Is there a funder or project associated with this feature?*
 
 ### Who should be involved?
-Implementer: 
-Commenters: 
-Users: 
-Reviewers: 
+
+Implementer:
+Commenters:
+Users:
+Reviewers:
 
 ### Risk
+
 *Will this potentially break anything?*
 
 #### Tests
+
 *What are relevant tests or what tests need to be created in order to determine that this issue is complete?*

.github/workflows/docs.yml

@@ -8,7 +8,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: [3.8]
+        python-version: [3.9]
 
     steps:
     - uses: actions/checkout@v2

.github/workflows/publish.yml

@@ -0,0 +1,79 @@
+# This workflows will upload a Python Package using Twine when a release is created
+# For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries
+
+name: Publish Python Package 📦 to PyPI and TestPyPI
+on:
+  push:
+    branches: [main]
+  release:
+      types: [created]
+  workflow_dispatch:  # Manual trigger for both PyPI and TestPyPI
+    inputs:
+      publish_target:
+        description: 'Publish Target (pypi/testpypi)'
+        required: true
+        default: 'testpypi'
+jobs:
+  build:
+    name: Build distribution 📦
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python
+      uses: actions/setup-python@v4
+      with:
+        python-version: "3.x"
+    - name: Install pypa/build
+      run: >-
+        python3 -m
+        pip install
+        build
+        --user
+    - name: Build a binary wheel and a source tarball
+      run: python3 -m build
+    - name: Store the distribution packages
+      uses: actions/upload-artifact@v3
+      with:
+        name: python-package-distributions
+        path: dist/
+  publish-to-pypi:
+    name: >-
+      Publish Python 🐍 distribution 📦 to PyPI
+    if: github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.publish_target == 'pypi')
+    needs:
+    - build
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/p/network_wrangler
+    permissions:
+      id-token: write  # IMPORTANT: mandatory for trusted publishing
+    steps:
+    - name: Download all the dists
+      uses: actions/download-artifact@v3
+      with:
+        name: python-package-distributions
+        path: dist/
+    - name: Publish distribution 📦 to PyPI
+      uses: pypa/gh-action-pypi-publish@release/v1
+  publish-to-testpypi:
+    name: Publish Python 🐍 distribution 📦 to TestPyPI
+    if: github.event_name == 'push' || (github.event_name == 'workflow_dispatch' && github.event.inputs.publish_target == 'testpypi')
+    needs:
+    - build
+    runs-on: ubuntu-latest
+    environment:
+      name: testpypi
+      url: https://test.pypi.org/p/gmnspy
+    permissions:
+      id-token: write  # IMPORTANT: mandatory for trusted publishing
+    steps:
+    - name: Download all the dists
+      uses: actions/download-artifact@v3
+      with:
+        name: python-package-distributions
+        path: dist/
+    - name: Publish distribution 📦 to TestPyPI
+      uses: pypa/gh-action-pypi-publish@release/v1
+      with:
+        repository-url: https://test.pypi.org/legacy/

.github/workflows/push.yml

@@ -8,37 +8,73 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: [3.8]
+        python-version: [3.9]
 
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v4
       with:
         fetch-depth: 0
     - name: Document branch
       run: echo ${{ github.ref_name }}
     - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v2
+      uses: actions/setup-python@v5
       with:
         python-version: ${{ matrix.python-version }}
+    - name: Configure Git user
+      run: |
+          git config --local user.email "github-actions[bot]@users.noreply.github.com"
+          git config --local user.name "github-actions[bot]"
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
         pip install -r requirements.txt
         pip install -r requirements.tests.txt
-    - name: Lint
-      run: |
-        # stop the build if there are Python syntax errors or undefined names
-        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
-        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
-        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
     - name: Install package
       run: |
         pip install -e .[test]
-    - name: Test with pytest
+    - name: Run Ruff
+      run: ruff check --output-format=github .
+    - name: Run tests with coverage and benchmarking
       run: |
-        pytest -s -m "not skipci"
-    - name: Configure Git user
+          pytest --junitxml=pytest.xml --benchmark-save=pr_benchmark --benchmark-json=pr_benchmark.json --cov-report=term-missing:skip-covered --cov=network_wrangler tests/ | tee pytest-coverage.txt
+    - name: Pytest coverage comment
+      if: github.event_name == 'pull_request'
+      uses: MishaKav/pytest-coverage-comment@main
+      with:
+        pytest-coverage-path: ./pytest-coverage.txt
+        junitxml-path: ./pytest.xml
+    - name: If a pull request, add a comment about coverage
+      if: github.event_name == 'pull_request'
+      uses: coroo/[email protected]
+    - name: Retrieve benchmark results from base branch
+      if: github.event_name == 'pull_request'
+      id: compare
+      run: |
+        git checkout ${{ github.event.pull_request.base.ref }}
+        if [ -f "tests/.benchmarks/benchmark_results.json" ]; then
+          cp tests/.benchmarks/benchmark_results.json base_benchmark.json
+          echo "::set-output name=benchmark_exists::true"
+        else
+          echo "::set-output name=benchmark_exists::false"
+        fi
+    - name: Compare benchmarks
+      if: steps.compare.outputs.benchmark_exists == 'true'
       run: |
-        git config --local user.email "github-actions[bot]@users.noreply.github.com"
-        git config --local user.name "github-actions[bot]"
+        pytest-benchmark compare base_benchmark.json pr_benchmark.json --csv=comparison.csv --json=comparison.json --histogram=histogram.png
+
+    - name: Post comparison comment on pull request
+      if: steps.compare.outputs.benchmark_exists == 'true'
+      uses: marocchino/sticky-pull-request-comment@v2
+      with:
+        header: 'Benchmark Comparison'
+        message: |
+          ## Benchmark Results
+          ![Benchmark Comparison](./histogram.png)
 
+          | Test name | PR Benchmark | Base Benchmark | Difference |
+          | --------- | ------------ | -------------- | ---------- |
+          ```bash
+          pytest-benchmark compare base_benchmark.json pr_benchmark.json --csv --diff
+          ```
+      env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.gitignore

@@ -3,16 +3,24 @@
 
 # Temp and test cache
 scratch/*
-tests/temp/
+tests/temp/*
 scratch/*
 *.mbtiles
 *.geojson
 .temp
 .pytest_cache
 *.bak
 .cache
+cache
 tests/t_*
 tests/*.txt
+tests/out/**
+tests/out/**/**
+wranglertest/
+.coverage
+coverage.xml
+prof/*
+
 # Logs
 *.log
 scenario_log.txt
@@ -65,4 +73,5 @@ env/
 .project
 .pydevproject
 .settings
+.vscode
 *.hubstorinfo

.markdownlint.yaml

@@ -10,3 +10,9 @@ MD013: false
 
 # Remove "no emphasis as heading"
 MD036: false
+
+# Remove "no first line must by h1" b/c sometimes importing from other sources
+MD041: false
+
+# Code block style - want to be able to use fenced code blocks to apply coloring
+MD046: false

.pre-commit-config.yaml

@@ -1,12 +1,15 @@
 repos:
-  - repo: https://github.com/psf/black
-    rev: 22.8.0
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    # Ruff version.
+    rev: v0.4.8
     hooks:
-      - id: black
-  - repo: https://github.com/pycqa/flake8
-    rev: 5.0.4
-    hooks:
-    - id: flake8
+      # Run the linter.
+      - id: ruff
+        types_or: [ python ]
+        args: [ --fix ]
+      # Run the formatter.
+      - id: ruff-format
+        types_or: [ python, pyi, jupyter ]
   - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v4.3.0
     hooks:
@@ -20,6 +23,12 @@ repos:
       - id: requirements-txt-fixer
       - id: mixed-line-ending
   - repo: https://github.com/igorshubovych/markdownlint-cli
-    rev: v0.32.2
+    rev: v0.34.0
+    hooks:
+      - id: markdownlint
+  - repo: https://github.com/pre-commit/mirrors-mypy
+    rev: v1.10.0
+    additional_dependencies: [pyyaml]
     hooks:
-      - id: markdownlint
+    -   id: mypy
+        exclude: 'tests/.*'

CONTRIBUTING.md

@@ -1,12 +1,9 @@
 # Contributing to Network Wrangler
 
-
 ## Roles
 
-
 ## How to Contribute
 
-
 ### Setup
 
 1. Make sure you have a [GitHub](https://github.com/) account.