diff --git a/README.md b/README.md index 705d174e..8d062267 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,7 @@ workflow can be used to analyse environmental risks to infrastructure networks using global open data. It is a work in progress. Goals: + - Automated pipeline for reproducible analysis anywhere in the world - Maps per-country and of larger areas - Charts/stats of exposure per admin region, per hazard type, scenario, epoch @@ -20,6 +21,7 @@ Goals: - Estimate indirect effects of disruption - people affected, economic activity disrupted Non-goals: + - Using closed data, which may be appropriate for other projects or use-cases - Detailed operational/engineering level simulation - Long-term planning @@ -27,6 +29,7 @@ Non-goals: ## Installation Install `open-gira` by cloning the repository: + ```bash git clone https://github.com/nismod/open-gira.git ``` @@ -39,11 +42,13 @@ manage these `conda` packages. Having installed one of the suggested package managers, to create the `open-gira` conda environment: + ```bash micromamba create -f environment.yml -y ``` And to activate the environment: + ```bash micromamba activate open-gira ``` @@ -52,9 +57,10 @@ micromamba activate open-gira Some rules use the `wget` utility to download files. -On Linux or MacOS, you may already have the `wget` utility available. If not, -it should be possible to install with your usual package manager (e.g. apt, +On Linux or MacOS, you may already have the `wget` utility available. If not, +it should be possible to install with your usual package manager (e.g. apt, MacPorts, brew), or else using micromamba: + ```bash micromamba install wget ``` @@ -80,6 +86,7 @@ Note that all subsequent commands given in the documentation assume that the Workflow steps are tested using small sample datasets. To run the tests: + ```bash python -m pytest tests ``` @@ -164,5 +171,5 @@ Programme. The views expressed here do not necessarily reflect the UK government's official policies. This research has also been supported by funding from the World Bank -Group, and the UK Natural Environment Research Council (NERC) through +Group, and the UK Natural Environment Research Council (NERC) through the UK Centre for Greening Finance and Investment (CGFI). diff --git a/docs/src/user-guide/usage/network-creation/rail.md b/docs/src/user-guide/usage/network-creation/rail.md index fd95cf8d..be57be35 100644 --- a/docs/src/user-guide/usage/network-creation/rail.md +++ b/docs/src/user-guide/usage/network-creation/rail.md @@ -21,32 +21,34 @@ from user-supplied sources (e.g. rehabilitation costs). ## Configuration To specify a desired network: + - Review and amend the spreadsheets in `bundled_data/transport`, these supply information that is used to gap-fill or extend what can be determined from OSM alone. - Review and amend `config/config.yaml`: - - The `infrastructure_datasets` map should contain a key pointing to an `.osm.pbf` - file URL for desired area. There are currently entries for the planet, - for (some definition of) continents and several countries. We use - the [geofabrik](http://download.geofabrik.de/) service for continent and - country-level OSM extracts. - - Check the OSM filter file pointed to by `network_filters.rail`. - This file specifies which [elements](https://wiki.openstreetmap.org/wiki/Elements) - (nodes, ways or relations) to keep (or reject) from the multitude of data - in an OSM file. See the filter expressions section - [here](https://docs.osmcode.org/osmium/latest/osmium-tags-filter.html) - for more information on the syntax of these files. - - Check and amend `keep_tags.rail`. This list of strings specifies which - `tags` (attributes) to retain on the filtered elements we extract from - the `.osm.pbf` file. - - Review `slice_count`. This controls the degree of parallelism possible. - With it set to 1, there is no spatial slicing (we create the network in - a single chunk). To speed network creation for large domains, it can be - set to a larger square number. The first square number greater than your - number of available CPUs is a good heuristic. + - The `infrastructure_datasets` map should contain a key pointing to an `.osm.pbf` + file URL for desired area. There are currently entries for the planet, + for (some definition of) continents and several countries. We use + the [geofabrik](http://download.geofabrik.de/) service for continent and + country-level OSM extracts. + - Check the OSM filter file pointed to by `network_filters.rail`. + This file specifies which [elements](https://wiki.openstreetmap.org/wiki/Elements) + (nodes, ways or relations) to keep (or reject) from the multitude of data + in an OSM file. See the filter expressions section + [here](https://docs.osmcode.org/osmium/latest/osmium-tags-filter.html) + for more information on the syntax of these files. + - Check and amend `keep_tags.rail`. This list of strings specifies which + `tags` (attributes) to retain on the filtered elements we extract from + the `.osm.pbf` file. + - Review `slice_count`. This controls the degree of parallelism possible. + With it set to 1, there is no spatial slicing (we create the network in + a single chunk). To speed network creation for large domains, it can be + set to a larger square number. The first square number greater than your + number of available CPUs is a good heuristic. ## Creation And to create the network, by way of example: + ```bash snakemake --cores all -- results/egypt-latest_filter-rail/edges.gpq ``` diff --git a/docs/src/user-guide/usage/network-creation/road.md b/docs/src/user-guide/usage/network-creation/road.md index eaf40fe7..3ae54d7d 100644 --- a/docs/src/user-guide/usage/network-creation/road.md +++ b/docs/src/user-guide/usage/network-creation/road.md @@ -20,32 +20,34 @@ sources (e.g. rehabilitation costs). ## Configuration To specify a desired network: + - Review and amend the spreadsheets in `bundled_data/transport`, these supply information that is used to gap-fill or extend what can be determined from OSM alone. - Review and amend `config/config.yaml`: - - The `infrastructure_datasets` map should contain a key pointing to an `.osm.pbf` - file URL for desired area. There are currently entries for the planet, - for (some definition of) continents and several countries. We use - the [geofabrik](http://download.geofabrik.de/) service for continent and - country-level OSM extracts. - - Check the OSM filter file pointed to by `network_filters.road`. - This file specifies which [elements](https://wiki.openstreetmap.org/wiki/Elements) - (nodes, ways or relations) to keep (or reject) from the multitude of data - in an OSM file. See the filter expressions section - [here](https://docs.osmcode.org/osmium/latest/osmium-tags-filter.html) - for more information on the syntax of these files. - - Check and amend `keep_tags.road`. This list of strings specifies which - `tags` (attributes) to retain on the filtered elements we extract from - the `.osm.pbf` file. - - Review `slice_count`. This controls the degree of parallelism possible. - With it set to 1, there is no spatial slicing (we create the network in - a single chunk). To speed network creation for large domains, it can be - set to a larger square number. The first square number greater than your - number of available CPUs is a good heuristic. + - The `infrastructure_datasets` map should contain a key pointing to an `.osm.pbf` + file URL for desired area. There are currently entries for the planet, + for (some definition of) continents and several countries. We use + the [geofabrik](http://download.geofabrik.de/) service for continent and + country-level OSM extracts. + - Check the OSM filter file pointed to by `network_filters.road`. + This file specifies which [elements](https://wiki.openstreetmap.org/wiki/Elements) + (nodes, ways or relations) to keep (or reject) from the multitude of data + in an OSM file. See the filter expressions section + [here](https://docs.osmcode.org/osmium/latest/osmium-tags-filter.html) + for more information on the syntax of these files. + - Check and amend `keep_tags.road`. This list of strings specifies which + `tags` (attributes) to retain on the filtered elements we extract from + the `.osm.pbf` file. + - Review `slice_count`. This controls the degree of parallelism possible. + With it set to 1, there is no spatial slicing (we create the network in + a single chunk). To speed network creation for large domains, it can be + set to a larger square number. The first square number greater than your + number of available CPUs is a good heuristic. ## Creation And to create the network, by way of example: + ```bash snakemake --cores all -- results/egypt-latest_filter-road-primary/edges.gpq ``` diff --git a/docs/src/user-guide/usage/utilities.md b/docs/src/user-guide/usage/utilities.md index 26d10d23..27d68047 100644 --- a/docs/src/user-guide/usage/utilities.md +++ b/docs/src/user-guide/usage/utilities.md @@ -5,6 +5,7 @@ ## Removing intermediate files You can remove intermediate files by running the `clean` rule. + ```bash snakemake --cores 1 -R clean ``` @@ -26,6 +27,7 @@ This will write `.gpkg` files beside their source `.geoparquet`. `matplotlib` plots can be interactive (zoom, pan, etc.), but not as static images. Some rules produce pickled plot files. To view these, use: + ``` python workflow/scripts/unpickle_plot.py ``` @@ -34,6 +36,7 @@ python workflow/scripts/unpickle_plot.py The bash script `archive_results.sh` can be used to back up analysis results. Here's an example usage: + ```bash ./archive_results.sh results/ /mnt/backup/open-gira ``` @@ -41,4 +44,4 @@ Here's an example usage: This will create a timestamped folder in path of the second argument, e.g. `/mnt/backup/open-gira/2023-07-24T101756+0100`, containing an archive of results (excluding input files downloaded from the internet) and a README describing the -state of the repository at the time of archive creation. \ No newline at end of file +state of the repository at the time of archive creation.