raste-tools

A collection of raster tools.

Development installation

For development, you can use a docker-compose setup:

$ docker-compose build --build-arg uid=`id -u` --build-arg gid=`id -g` lib
$ docker-compose up --no-start
$ docker-compose start
$ docker-compose exec lib bash

Create a virtualenv, install dependencies & package, run tests:

# note that Dockerfile prepends .venv/bin to $PATH
(docker)$ virtualenv --system-site-packages .venv
(docker)$ pip install -r requirements.txt --index-url https://packages.lizard.net
(docker)$ pip install -e .[test]
(docker)$ pytest

Update packages:

(docker)$ rm -rf .venv
(docker)$ virtualenv --system-site-packages .venv
(docker)$ pip install -e . --index-url https://packages.lizard.net
(docker)$ pip freeze | grep -v raster-tools > requirements.txt

Now you are ready to run the raster tools in the container.

Mapping an extra local folder

First, clone this repo and make some required directories:

$ git clone [email protected]:nens/raster-tools
$ cd raster-tools

Create a docker-compose.override.yaml to map local filesystems:

version: '3'
services:
  lib:
    volumes:
      - /some/local/path:/some/container/path

Task server installation

One time server provisioning:

ansible-playbook ansible/provision.yml -i ansible/task.yml

Deploying new versions:

ansible-playbook ansible/deploy.yml -i ansible/task.yml

Be aware that if you provision a server for the first time, the authentication needs to be setup. We may need a credentials file to access private packages on packages.lizard.net, to be put in deploy/files/nens_netrc file.

To enable users to use the raster-tools scripts, append the absolute path to .venv/bin to the PATH in /etc/environment.

Filling nodata in rasters

Use fillnodata to fill nodata regions in rasters. The input to the algorithm is the edge surrounding the region. The raster saved as the target argument only contains the complementary cells.

Rasterizing landuse tables

For rasterization of landuse tables from a postgres datasource a special wrapper command is available at bin/rasterize-landuse, use --help for args.

Creating streamlines

Run the following scripts for streamline calculation:

flow-fil  # depression filling
flow-dir  # direction calculation
flow-acc  # accumulation
flow-vec  # make shapefiles
flow-rst  # make rasters from shapefiles

Streamlines have been rasterized in the past to be able to visualize them with good performance on any zoomlevel. A number of tricks can be to make them look like a vector dataset:

• Rasterize at sufficiently high resolution so that supersampling is never needed
• (Pre)aggregate using a maximum algorithm for lower resolutions
• Use binary dilation after serving image tiles on one or more higher classes to create a 'wider stroke' effect

Multiprocessing

A number of scripts have a --part option to run the script on a subset of the source features, e.g. --part=2/3 to run on the second part of three parts. To use xargs to run some script for example on 4 processes, use:

xargs -a <(echo -e '1/4\n2/4\n3/4\n4/4') -L 1 -P 4 your_script --your_args --part