V1 Refactor RC #327

e-lo · 2023-04-13T20:51:10Z

This PR contains refactors, bug fixes, performance enhancement, and features for v1 of network_wrangler documented in notebook/releaseoverviewv1.ipynb including:

Features

I/O and Mutation Speed
Implicit and fast validation
Flexible serialization formats
Improved stability
Selection flexibility for transit based on any feature or by link or node
More flexible and fast transition to geodataframes for (almost) any data part
Easy data clipping to geographic bounds
Scripts for actions you might want to execute from command line (i.e. data conversion/clipping/etc)
Logging + Error descriptions
Additional project card types: add/delete transit
Scripts to create example data from OSM...anywhere

Tech Overhead Investment

Separation of causes prevents circular reference collisions, limits import bloat, and improves legibility/organization.
- Project card functionality in project card repo
- Separate modules for separate functionality
Reliability achieved through more testable code and expanding test coverage with more test cases and also anti-patterns
- Testable code
- Test coverage
Explicit data models that make complex data structures obvious and easy to validate
- Legible
- Self-documenting
- Flexible
- Validatable
Clean code principles that make code easier to test and maintain
- more functions that do a single thing...and do it well
- classes that are small - functions that access or manipulate them
Documentation
- consistent and detailed functional documentation
- consistent type hints
- usage for modules and classes
- overall documentation leverages the flexible and less bloated MkDocs package
Removes less-well-maintained dependencies
- Replaces Partridge with internal functionality

Speed

Makes the heavy shapes.geojson optional and doesn't read it in for operations that don't directly involve it.
Leverages pandera for speedy, vector-based data model validation for dataframes
Replaces row-based calculations for blank geographic values with vector-based calculations
Provides flexibility for I/O serialization formats that are faster – like Parquet.

WIP for showing

- removed project card.py - refactored selection, graph, subnet, segment from roadway network - refactored project changes apply from roadway network - refactored shapes, links and nodes specs from roadway network

+ varia bug fixes

transit selection tests passing

+ linting

- generalize transit property change projects to be any attribute in table

- update required link attr - pep8 - refactor time --> timespan

…ference IDS

…n formats - Remove LocationReferences from required part of schema that we fill in. Jump straight to geometries. - Make link geometry creation a vector operation —> speeds up significantly 30-100 times as fast - Added - Accessor for link true-shapes - Add clipping functionality for roadway and transit

- Accidentally deleted 'ref' from st paul roadway links b/c wasn't in data model even though it is used. - Added to data model since it is a key OSM field as optional and updated example network to include it. - Updated error checking for the dict query to make sure it is explicitly telling you the field doesn't exist in the df you are querying

Also: - try to fix benchmarking/coverage reports #394 - Fix bug in build osm roadnet.py

Fixes #395 and tests it as well

Also: - link to notebooks that have additional details/info - add info about using included conda env and installing optional dependencies - change name of quickstart to intro since it is actually fairly in depth

… docs.

- include docs build as part of release and push to avoid a pull clash - update checkout/python action versions for 4 and 5 - separate prerelease and release workflows - prerelease workflow pushes on any main push and wont collide with versions - fix typo referring to other repo - make pyproject.toml dependencies recursive so tests also includes docs and viz

- using coverage.xml

e-lo added 3 commits April 10, 2023 16:28

intitial separation

4913036

initial refactor

247d90f

Refactor selection, segments, apply

8eb9cc0

WIP for showing

e-lo self-assigned this Apr 13, 2023

e-lo marked this pull request as draft April 13, 2023 20:52

e-lo added 20 commits April 27, 2023 15:02

Factored out major parts or RoadwayNetwork to other classes

6351925

- removed project card.py - refactored selection, graph, subnet, segment from roadway network - refactored project changes apply from roadway network - refactored shapes, links and nodes specs from roadway network

Merge branch 'develop' into selection-refactor

34c0ddb

debugging

a1ca5c9

update project card formats

5208330

+ varia bug fixes

update selection tests to have answers

396676b

selections tests passing

4f943b1

test_properties passs

6402ef4

Passing almost all roadway tests now

f0310d1

Format updates, linting

58d27f8

Refactor transit selection

80cad3c

transit selection tests passing

transit tests working

3ffc565

+ linting

refactor transit project application

ff21855

- generalize transit property change projects to be any attribute in table

Update project cards and API to be consistent with projectcard structure

fdfa5b2

rename to timespans

7ef4580

- update required link attr - pep8 - refactor time --> timespan

Add roadway network clipping feature

c4048fa

Fix roadway single node example with incorrect ..referenceID --> ..re…

b55414b

…ference IDS

Update transit timespan --> timespans per project card v1 spec

50652d6

Update notebooks and tests to use updatated API

e70b32a

Update organization of pydantic and pandera data models

867753d

e-lo added 23 commits October 17, 2024 11:20

Try again to re-add link.json

7483fd6

Also: - try to fix benchmarking/coverage reports #394 - Fix bug in build osm roadnet.py

Keep trying to get tests to give better errors

1ee2822

maybe its the pytest config in pyproject that it doesn't like

1640a8e

try removing profiling

dc209ff

try moving coverage

bc50f94

still testing pytest args

9c90a1f

remove all pytest config options

8e078bc

maybe it is python 3.9 that pytest doesn't like?

cabee68

maybe pytest doesn't like python 3.9??

7a00ce5

Github actions interpreted python 3.10 as 3.1...trying quotes

6850aeb

remove all pytest extensions from requirements.tests.txt

a5d626a

ok - that worked! now going to slowly add back in requirements.

2870cde

add back benchmark commands

3a185d2

trying to add back in profiling

5877489

Didn't like profiling. Trying coverage.

bb8a71d

Order output tables based on datamodel

63a1963

Fixes #395 and tests it as well

Update formatting for some docs

3d16af5

Also: - link to notebooks that have additional details/info - add info about using included conda env and installing optional dependencies - change name of quickstart to intro since it is actually fairly in depth

Update data model documentation to be stored in modules instead of in…

4626a0c

… docs.

Lint

8fdf83c

Try coverage settings

67cb628

- using coverage.xml

Update action step names [skip ci]

005b9af

e-lo had a problem deploying to pypi October 21, 2024 19:55 — with GitHub Actions Failure

e-lo deployed to pypi October 21, 2024 20:33 — with GitHub Actions View deployment

e-lo added 3 commits October 21, 2024 13:49

fix: fetch depth 0 necessary for building docs

6b6e4ce

Update issue templates. skip ci

653cad2

lint line endings skip-ci

fd8d0ad

e-lo merged commit 6e87f8c into develop Oct 28, 2024
1 check passed

e-lo deleted the selection-refactor branch October 28, 2024 19:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

V1 Refactor RC #327

V1 Refactor RC #327

e-lo commented Apr 13, 2023 •

edited

Loading

V1 Refactor RC #327

V1 Refactor RC #327

Conversation

e-lo commented Apr 13, 2023 • edited Loading

Features

Tech Overhead Investment

Speed

e-lo commented Apr 13, 2023 •

edited

Loading