Multitemporal Land Cover Classification Network

A recurrent neural network approach to encode multi-temporal data for land cover classification.

Source code of Rußwurm & Körner (2018) PDF

If you use this repository consider citing

Rußwurm M., Körner M. (2018). Multi-Temporal Land Cover Classification with
Sequential Recurrent Encoders. ISPRS International Journal of Geo-Information, 7(4), 129.

The Tensorflow 1.7 code of the network is located at modelzoo/seqencmodel.py Further scripts for training and evaluation are provided. Additionally, Jupyter notebooks used for accuracy evaluation and extraction of internal network activiations are located in this repo. The code can be executed after downloading the demo data. After installing the dependencies the python scripts should be executable. Additionally, we provided a docker image with all dependencies already installed. The code was developed in Tensorflow 1.4 and was later ported to Tensorflow 1.7.

Network

Similar to an encoding rnn layer of sequence-to-sequence a variable-length input sequence of images is encoded to intermediate reprentation.

Encoding LSTM-RNN:

Network structure

Bidirectional rnn encoder and convolutional softmax classifier, as described in the paper.

Dependencies

Implementations of ConvGRU and ConvLSTM forked from https://github.com/carlthome/tensorflow-convlstm-cell

The code is implemented in Python 2.7 check your version via python --version

Get the LSTM/GRU definitions

git clone https://github.com/MarcCoru/tensorflow-convlstm-cell.git utils/convrnn

Create conda environment with python 2.7

conda create -n mtlcc python=2.7 pip
conda activate mtlcc

Python packages

pip install tensorflow-gpu==1.7
pip install configparser # for dataset management
pip install numpy
conda install -y gdal # for evaluation.py
pip install Pillow # for activations.py
pip install rasterio # for tfrecord2tif.py

Download demo data

Download the full dataset (40GB) via

wget https://zenodo.org/record/5712933/files/data_IJGI18.zip

download the raw shapefiles along with prediction and confidence images for both years (1.5 GB)

wget https://zenodo.org/record/5712933/files/showcase.zip

or check the Zenodo page

Convert tfrecord to tif

Convert tfrecord time series folders containing geotiffs by

python tfrecord2tif.py data_IJGI18/datasets/full/480/data16/5887.tfrecord.gz \
     --geotransforms data_IJGI18/datasets/full/480/geotransforms.csv

Jupyter notebooks

# start notebook (required dependencies)
jupyter notebook

# within docker
nvidia-docker run -ti -v $PWD/data_IJGI18:/MTLCC/data_IJGI18 -p 8888:8888 marccoru/ijgi18 \
    jupyter notebook --ip 0.0.0.0 --allow-root --no-browser

Network training and evaluation

on local machine (requires dependencies installed)

build the network graph for 24px tiles

python modelzoo/seqencmodel.py \
    --modelfolder tmp/convgru128 \
    --convrnn_filters 128 \
    --convcell gru \
    --num_classes 17 \
    --pix10m 24

train the network graph

python train.py tmp/convgru128 \
    --datadir data_IJGI18/datasets/demo/240 \
    --temporal_samples 30 \
    --epochs 30 \
    --shuffle True \
    --batchsize 4 \
    --train_on 2016 2017

build network graph for 48px tiles

python modelzoo/seqencmodel.py \
    --modelfolder tmp/convgru128_48px \
    --convrnn_filters 128 \
    --convcell gru \
    --num_classes 17 \
    --pix10m 48

initialize the network and copy weights from 24px to 48 px networks

# initialize
python init_graph.py tmp/convgru128_48px/graph.meta

# optional: compare tensor dimensions of two graphs
python compare_graphs.py tmp/convgru128 tmp/convgru128_48px

# copy network weights from source (24px) network to target (48px) network
python copy_network_weights.py tmp/convgru128 tmp/convgru128_48px

evaluate the model

(writes prediction pngs and statistics on accuracy to tmp/eval/24)

python evaluate.py tmp/convgru128 \
    --datadir data_IJGI18/datasets/demo/240 \
    --storedir tmp/eval/24 \
    --writetiles \
    --writeconfidences \
    --batchsize 1 \
    --dataset 2017

using docker image (requires nvidia-docker)

# alias for command: start nvidia-docker session and forward folders for data and models
alias dockercmd="nvidia-docker run -ti -v $PWD/data_IJGI18/datasets/demo:/data -v $PWD/tmp:/model -v $PWD/tmp:/output marccoru/ijgi18"

# create model
dockercmd python modelzoo/seqencmodel.py \
    --modelfolder /model/convgru128 \
    --convrnn_filters 128 \
    --convcell gru \
    --num_classes 17 \
    --pix10m 24

# start training
dockercmd python train.py /model/convgru128 \
    --datadir /data/240 \
    --temporal_samples 30 \
    --epochs 30 \
    --shuffle True \
    --batchsize 4 \
    -d 2016 2017

# evaluate
dockercmd python evaluate.py /model/convgru128 \
    --datadir /data/240 \
    --storedir /output \
    --writetiles \
    --writeconfidences \
    --batchsize 1 \
    --dataset 2017

Extract Activations

activations.py is a scripted version from the activations section of NetworkVisualization.ipynb

to extract internal activation images from tile 16494 as pngs to tmp/activations folder run

python activations.py \
    data_IJGI18/models/convlstm256_48px/ \
    data_IJGI18/datasets/demo/480/ \
    tmp/activations \
    --dataset 2016 \
    --partition eval \
    --tile 16494

via docker

alias dockercmd="nvidia-docker run -ti -v $PWD/data_IJGI18/datasets/demo:/data -v $PWD/data_IJGI18/models:/models -v $PWD/tmp:/output marccoru/ijgi18"

dockercmd python activations.py \
    /models/convlstm256_48px/ \
    /data/480/ \
    /output/activations \
    --dataset 2016 \
    --partition eval \
    --tile 16494

Customization

If you plan to customize this code with your data: check out SimpleTrain.ipynb

This notebook provides a simplified walkthrough from the most important components implemented in this repo

it includes

1. the creation of a custom fake dataset in the right format
2. the parsing of this dataset
3. performing one training step on this dataset