Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can we simplify large GeoJSONs? #12

Open
sgoodm opened this issue Aug 2, 2021 · 7 comments
Open

Can we simplify large GeoJSONs? #12

sgoodm opened this issue Aug 2, 2021 · 7 comments
Labels
code related to code for processing data data related to data

Comments

@sgoodm
Copy link
Member

sgoodm commented Aug 2, 2021

Currently there are 3 GeoJSONs that are about 7x larger than the next largest GeoJSONs.

48M	latest/geojsons/178.geojson
48M	latest/geojsons/56959.geojson
48M	latest/geojsons/695.geojson

These 3 make up about 30% of the combined GeoJSON's total size.

Possible solution is to simplify these features, but when do we decide to simplify and how much do we simplify?

@sgoodm sgoodm added code related to code for processing data data related to data labels Aug 2, 2021
@sgoodm sgoodm added this to the China data launch milestone Aug 2, 2021
@sgoodm
Copy link
Member Author

sgoodm commented Aug 2, 2021

Worth noting that all 3 of those are the same feature (a lake with very precise mapping of edges, and tributaries, creeks, etc around it)

@sgoodm
Copy link
Member Author

sgoodm commented Aug 2, 2021

Also realized that the JSON write includes indents which we can drop to reduce file size

@sgoodm
Copy link
Member Author

sgoodm commented Aug 2, 2021

JSON write updated in 9dc4f1a

@sgoodm
Copy link
Member Author

sgoodm commented Aug 28, 2021

Confirming this is still an issue and these are pushing us close to GitHub max file size for combined geojson zip

@sgoodm
Copy link
Member Author

sgoodm commented Aug 30, 2021

Moving this off of launch milestone

@sgoodm sgoodm removed this from the China data launch milestone Aug 30, 2021
@sgoodm
Copy link
Member Author

sgoodm commented Sep 29, 2021

Note that these trigger errors based on GDAL's OGR_GEOJSON_MAX_OBJ_SIZE env var, which defaults to 200MB

https://gdal.org/drivers/vector/geojson.html
https://gis.stackexchange.com/questions/300419/ogr2ogr-import-fails-with-error-1-geojson-object-too-complex

@sgoodm
Copy link
Member Author

sgoodm commented Sep 29, 2021

One more note: considering that this project is for a hydro power plant, we can likely get a better feature than the entire lake.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
code related to code for processing data data related to data
Projects
None yet
Development

No branches or pull requests

1 participant