Add automatic linting to commits (#61)

* Add various linting funtions to pre-commit 1. black - jupyter 2. pydocstyle: docstrings valid 3. check-toml 4. sort requirements.txt 5. check executable shebangs 6. import sorting 7. json schema for github actions/workflows * Add pre-commit to testing on push * Lint files to pass pre-commit.
BayAreaMetro · May 3, 2022 · 96ed615 · 96ed615
1 parent e0980da
commit 96ed615
Show file tree

Hide file tree

Showing 64 changed files with 740 additions and 432 deletions.
diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -8,30 +8,33 @@ assignees: ''
 ---
 
 ## Describe the bug
-A clear and concise description of what the bug is or the error code you got. e.g. 
-```python 
+
+A clear and concise description of what the bug is or the error code you got. e.g.
+
+```python
 KeyError: 'Passing list-likes to .loc or [] with any missing labels is no longer supported, see https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#deprecate-loc-reindex-listlike'
 ```
 
 ## To Reproduce
+
 Steps to reproduce the behavior:
+
 1. Go to '...'
 2. Click on '....'
 3. Scroll down to '....'
 4. See error
 
 ### Failing tests
+
 - [ ] No applicable test failed, need to create.
-- [ ] 
+- [ ]
 
 ### Triggering line of code
 
-
 ### Thoughts on resolution
 
 ### Full stack trace
 
-
 ### Environment
 
 Operating system:

diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md
@@ -8,25 +8,30 @@ assignees: ''
 ---
 
 ### User Story
+
 *As a ...insert type of user... I'd like to ...insert desired feature or behavior...*
 
-### Priority 
+### Priority
 
 ### Level of Effort
 
 ### Resolution Ideas
 
 ### Project
+
 *Is there a funder or project associated with this feature?*
 
 ### Who should be involved?
-Implementer: 
-Commenters: 
-Users: 
-Reviewers: 
+
+Implementer:
+Commenters:
+Users:
+Reviewers:
 
 ### Risk
+
 *Will this potentially break anything?*
 
 #### Tests
+
 *What are relevant tests or what tests need to be created in order to determine that this issue is complete?*
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
@@ -1,27 +1,23 @@
 ## What existing problem does the pull request solve and why should we include it?
 
-
 ## What is the testing plan?
 
 *Demonstrate the code is solid by discussing how results are verified and covered by tests*
 
- - [ ] Code for this PR is covered in tests 
- - [ ] Code passes all existing tests
+- [ ] Code for this PR is covered in tests
+- [ ] Code passes all existing tests
 
 ## Code formatting
+
 *Code should be PEP8 compliant before merging by running a package like [`black`](https://pypi.org/project/black/)*
 
- - [ ] Code linted
+- [ ] Code linted
 
 ## Applicable Issues
+
 *Please do not create a Pull Request without creating an issue first.*
 
 *Put `closes #XXXX` in your comment to auto-close the issue that your PR fixes.*
 
-
-#### Issues List
-
- -  closes...
- -  closes...
-
-
+- closes...
+- closes...
diff --git a/.github/workflows/flake.yml b/.github/workflows/flake.yml
@@ -0,0 +1,14 @@
+name: pre-commit
+
+on:
+  pull_request:
+  push:
+    branches: [main, develop]
+
+jobs:
+  pre-commit:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - uses: actions/setup-python@v2
+    - uses: pre-commit/[email protected]
diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml
@@ -8,7 +8,7 @@ on:
     types: [created]
 
 jobs:
-  
+
   deploy:
 
     runs-on: ubuntu-latest
@@ -29,4 +29,4 @@ jobs:
         TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
       run: |
         python setup.py sdist bdist_wheel
-        twine upload dist/*
+        twine upload dist/*
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -21,12 +21,6 @@ jobs:
         python -m pip install --upgrade pip
         pip install -r requirements.txt
         pip install -r dev-requirements.txt
-    - name: Lint with flake8
-      run: |
-        # stop the build if there are Python syntax errors or undefined names
-        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
-        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
-        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
     - name: Install package
       run: |
         pip install -e .[test]

diff --git a/.gitignore b/.gitignore
@@ -82,7 +82,7 @@ coverage.xml
 # mkdocs documentation
 site/*
 
-# Jupyter 
+# Jupyter
 .ipynb_checkpoints
 profile_default/
 ipython_config.py

diff --git a/.markdownlint.yaml b/.markdownlint.yaml
@@ -5,5 +5,14 @@ default: true
 MD007:
   indent: 4
 
-# Remove line length limit 
-MD013: false
+# Remove line length limit
+MD013: false
+
+# Allow us to skip levels of headings
+MD001: false
+
+# Allow us to use emphasis on whole lines.
+MD036: false
+
+# Allow us to use lower-level headings to start files
+MD041: false
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -0,0 +1,46 @@
+default_stages: [commit]
+repos:
+  - repo: https://github.com/psf/black
+    rev: 22.3.0
+    hooks:
+      - id: black
+        language_version: python3
+      - id: black-jupyter
+  - repo: https://github.com/PyCQA/pydocstyle
+    rev: 6.1.1
+    hooks:
+      - id: pydocstyle
+        stages: [manual]
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v3.4.0
+    hooks:
+      - id: trailing-whitespace
+        args: [--markdown-linebreak-ext=md]
+      - id: end-of-file-fixer
+      - id: mixed-line-ending
+      - id: check-added-large-files
+      - id: check-json
+      - id: check-toml
+      - id: check-yaml
+      - id: requirements-txt-fixer
+      - id: check-executables-have-shebangs
+  - repo: https://github.com/igorshubovych/markdownlint-cli
+    rev: v0.27.1
+    hooks:
+      - id: markdownlint
+        stages: [manual]
+  - repo: https://github.com/pycqa/isort
+    rev: 5.10.1
+    hooks:
+    - id: isort
+      args: ["--profile", "black"]
+  - repo: https://github.com/python-jsonschema/check-jsonschema
+    rev: 0.14.3
+    hooks:
+    - id: check-github-workflows
+    - id: check-github-actions
+  - repo: http://github.com/pycqa/flake8
+    rev: 4.0.1
+    hooks:
+    - id: flake8
+      stages: [manual]
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,6 +1,7 @@
 # Changelog
 
 ## Version (date)
+
 - a list
 - of things
-- that have changed
+- that have changed
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -1,2 +1 @@
 # Contribution Guide
-
diff --git a/Dockerfile b/Dockerfile
@@ -1,5 +1,5 @@
 # docker build -t mypackage .
-# docker run --rm -v "$PWD":/home/jovyan/work mypackage /bin/bash 
+# docker run --rm -v "$PWD":/home/jovyan/work mypackage /bin/bash
 FROM jupyter/minimal-notebook
 
 COPY ../requirements.txt /tmp/requirements.txt
@@ -28,4 +28,4 @@ RUN conda clean --all --yes && \
 WORKDIR /home/jovyan/work
 
 # set default command to launch when container is run
-CMD ["jupyter", "lab", "--ip='0.0.0.0'", "--port=8888", "--no-browser", "--NotebookApp.token=''", "--NotebookApp.password=''"]
+CMD ["jupyter", "lab", "--ip='0.0.0.0'", "--port=8888", "--no-browser", "--NotebookApp.token=''", "--NotebookApp.password=''"]
diff --git a/README.md b/README.md
@@ -24,6 +24,7 @@ pip install tm2py
 ```
 
 #### Bleeding Edge
+
 If you want to install a more up-to-date or development version, you can do so by installing it from the `develop` branch as follows:
 
 ```bash
@@ -33,8 +34,8 @@ pip install git+https://github.com/bayareametro/tm2py@develop
 ```
 
 #### Developers (from clone)
-If you are going to be working on Lasso locally, you might want to clone it to your local machine and install it from the clone.  The -e will install it in [editable mode](https://pip.pypa.io/en/stable/reference/pip_install/?highlight=editable#editable-installs).
 
+If you are going to be working on Lasso locally, you might want to clone it to your local machine and install it from the clone.  The -e will install it in [editable mode](https://pip.pypa.io/en/stable/reference/pip_install/?highlight=editable#editable-installs).
 
 ```bash
 conda env create -f environment.yml
@@ -48,6 +49,7 @@ Note that you'll also need to install Emme's python packages into this conda env
 Following these instructions from an INRO community forum post: In the Emme Desktop application, open Tools->Application Options->Modeller, change your Python path as desired and click the "Install Modeller Package" button.
 
 If this is successful, the following packages will be visible in your environment when you type `pip list`:
+
 * inro-dynameq
 * inro-emme
 * inro-emme-agent
@@ -70,4 +72,4 @@ tm2py -s scenario.toml -m model.toml
 
 ## Contributing
 
-Details can be found in [CONTRIBUTING]
+Details can be found in [CONTRIBUTING]
diff --git a/bin/get_test_data.bat b/bin/get_test_data.bat
@@ -11,10 +11,10 @@ ECHO "Retreiving %TEST_DATA_NAME% data from %TEST_DATA_LOCATION%
 
 SET OUTDIR=%1
 if "%OUTDIR%"=="" SET OUTDIR=DEFAULT_DIRECTORY
-if not exist %OUTDIR% mkdir %OUTDIR%  
+if not exist %OUTDIR% mkdir %OUTDIR%
 CD %OUTDIR%
 ECHO "Writing to %CD%"
 
 curl -i -X GET %TEST_DATA_LOCATION% -L -o test_data.zip
 
-CD %CWD%
+CD %CWD%
diff --git a/bin/tm2py b/bin/tm2py
@@ -20,4 +20,4 @@ def run():
     controller.run()
 
 if __name__ == "__main__":
-    run()
+    run()
diff --git a/bin/update_docs b/bin/update_docs
@@ -5,7 +5,7 @@ import os
 import tm2py
 
 # High-level settings
-MODULE_CLASS_DOC_LIST = [ 
+MODULE_CLASS_DOC_LIST = [
     ("classes_components.md",[("## Components", tm2py.components, 1)]),
     ("classes_basic.md", [("## Basic", tm2py, 1)]),
     ("classes_config.md", [("## Config", tm2py.config, 1)]),
@@ -32,4 +32,4 @@ for _class_diagram_md,_module_list in MODULE_CLASS_DOC_LIST:
     class_diagram_outfile = os.path.join(docs_dir,"includes","class_diagrams",_class_diagram_md)
     with open(class_diagram_outfile,'w') as f:
         f.write(class_diagram_str)
-    logger.info(f"Updated class diagrams in:\n{class_diagram_outfile}")
+    logger.info(f"Updated class diagrams in:\n{class_diagram_outfile}")
-Original file line number
+Diff line change
@@ Expand Up / @@ -20,4 +20,4 @@ def run(): @@
         controller.run()
     if __name__ == "__main__":
-        run()
+        run()