Merge branch 'dev'

.gitignore

@@ -1 +1,2 @@
 /build/
+*__pycache__*

CMakeLists.txt

@@ -27,3 +27,4 @@ if(BUILD_EXAMPLES)
 endif()
 
 install(FILES LICENSE.md DESTINATION .)
+install(DIRECTORY train DESTINATION .)

README.md

@@ -5,15 +5,15 @@
 **OpenEM** is a library that provides advanced video analytics for 
 fisheries electronic monitoring (EM) data.  It currently supports detection,
 classification, counting and measurement of fish during landing or discard.  
-This functionality is currently only available via a deployment library with
+This functionality is available via a deployment library with
 pretrained models available in our example data (see tutorial).  The base
 library is written in C++, with bindings available for both Python and C#.
 Examples are included for all three languages.
 
-There are immediate plans to develop a training library so that users
-can build their own models on their own data.  Currently builds have only 
-been tested on Windows.  We plan to support both Ubuntu and macOS in the 
-future.
+The current release includes a training library for the detection 
+model.  There are immediate plans to expand this library to include 
+all OpenEM functionality.  Currently builds have only been tested on 
+Windows.  We plan to support both Ubuntu and macOS in the future.
 
 Click the image below to see a video of OpenEM in action:
 

doc/annotation.md

@@ -1,3 +1,51 @@
-## Annotation Guidelines
+# Annotation Guidelines
 
-Guidelines coming soon...
+This document describes the data layout for building your own models with OpenEM. Training routines in OpenEM expect the following directory layout:
+
+```shell
+your-top-level-directory
+└── train
+    ├── cover.csv
+    ├── length.csv
+    ├── ruler_position.csv
+    └── videos
+        ├── 00WK7DR6FyPZ5u3A.mp4
+        ├── 01wO3HNwawJYADQw.mp4
+        ├── 02p3Yn87z0b5grhL.mp4
+```
+
+Many of the annotations require video frame numbers. It is important to point out that most video players do not have frame level accuracy, so attempting to convert timestamps in a typical video player to frame numbers will likely be inaccurate. Therefore we recommend using the frame accurate video annotator [Tator][Tator], which guarantees frame level accuracy when seeking and allows line annotations which are useful for generating frame level fish length.
+
+**videos** contains video files in mp4 format. The content of these videos should follow the [data collection guidelines][CollectionGuidelines]. We refer to the basename of each video file as the *video ID*, a unique identifier for each video.  In the directory layout above, the video ID for the videos are 00WK7DR6FyPZ5u3A, 01wO3HNwawJYADQw, and 02p3Yn87z0b5grhL.
+
+**length.csv** contains length annotations of fish in the videos. Each row corresponds to an individual fish, specifically the video frame containing the clearest view of each fish. This file is also used to train the counting algorithm, so exactly one frame should be annotated per individual fish. The columns of this file are:
+
+* *video_id*: The basename of the video.
+* *frame*: The zero-based frame number in the video.
+* *x1, y1, x2, y2*: xy-coordinates of the tip and tail of the fish in pixels.
+* *species_id*: The one-based index of the species as listed in the ini file, as described in the [tutorial][Tutorial]. If this value is zero, it indicates that no fish are present. While length.csv can be used to include no fish example frames, it is encouraged to instead include them in cover.csv. Both are used when training the detection model, but only cover.csv is used when training the classification model.
+
+![Length coordinates of a clearly visible fish.](https://user-images.githubusercontent.com/7937658/49332082-acdd5d00-f574-11e8-8a7e-23a9f9dd1f1b.png)
+
+**cover.csv** contains examples of frames that contain no fish, fish covered by a hand or other obstruction, and fish that can be clearly viewed.  The columns of this file are:
+
+* *video_id*: The basename of the video.
+* *frame*: The zero-based frame number in the video.
+* *cover*: 0 for no fish, 1 for covered fish, 2 for clear view of fish.
+
+![Example of image with no fish.](https://user-images.githubusercontent.com/7937658/49332090-c54d7780-f574-11e8-985a-87ac99c56d8c.png)
+
+![Example of image with covered fish.](https://user-images.githubusercontent.com/7937658/49332093-d4342a00-f574-11e8-8e52-6b2988aced75.png)
+
+![Example of image with clear fish.](https://user-images.githubusercontent.com/7937658/49332096-e3b37300-f574-11e8-9e36-64ba90b0e17e.png)
+
+**ruler_position.csv** contains the ruler position in each video.  The columns of this file are:
+
+* *video_id*: The basename of the video.
+* *x1, y1, x2, y2*: xy-coordinates of the ends of the ruler in pixels.
+
+![Ruler coordinates for a video.](https://user-images.githubusercontent.com/7937658/49332099-f6c64300-f574-11e8-89b2-b95e85d26b6e.png)
+
+[Tator]: https://github.com/cvisionai/Tator/releases
+[CollectionGuidelines]: ./data_collection.md
+[Tutorial]: ./tutorial.md

doc/build.md

@@ -1,4 +1,4 @@
-## Build and Test Instructions
+# Build and Test Instructions
 
 Choose your operating system:
 

doc/build/windows/build.md

@@ -1,4 +1,4 @@
-## Build and Test Instructions for Windows
+## Build and Test Instructions for Deployment Library on Windows
 
 ### Install build tools
 

doc/data_collection.md

@@ -1,6 +1,6 @@
-## Data Collection Guidelines
+# Data Collection Guidelines
 
-# Region of interest
+## Region of interest
 
 The region of interest (ROI) is the part of the video frame that may contain
 fish.  Although the ROI may be the entire video frame, typically the ROI is only
@@ -9,37 +9,37 @@ if the input images are cropped to the ROI.  Therefore, many of the data
 collection guidelines are driven by recommendations for the ROI, not the entire
 video frame.
 
-# Environment
+## Environment
 
 The example data is all taken during the daytime.  Algorithms in OpenEM can work
 under other conditions, such as with artificial lighting at night or at
 dawn/dusk for users who wish to train models on their own data.  Keep in mind,
 however, that generally speaking lower variability in appearance will lead to
 better algorithm performance.
 
-# Camera characteristics
+## Camera characteristics
 
 Video data is expected to have three channels of color (RGB).  Camera
 resolution is driven by the resolution of the region of interest.  The
 resolution of the ROI should be near 720 x 360 pixels, but lower resolution may
 still yield acceptable results.
 
-# Camera placement
+## Camera placement
 
 Video should be taken from the overhead perspective, perpendicular to the
 broadside of any fish in the field of view.  We recommend that the camera be
 aligned with perpendicular within 20 degrees.  If possible, the region of
 interest should be located near the center of the camera field of view to
 minimize lens distortion.
 
-# Use of rulers
+## Use of rulers
 
 OpenEM has functionality that allows for automatic determination of the region
 of interest.  This functionality requires the use of a ruler that will span the
 region of interest whenever a fish may require detection.  See figure 1 for
 examples of a region of interest spanned by a ruler.
 
-# Fish movement
+## Fish movement
 
 Each fish that is to be detected, counted, classified, or measured should be
 moved through the ROI in the following way:

doc/training_environment.md

@@ -0,0 +1,37 @@
+# Python Environment for Training Library
+
+This document describes how to set up a python environment to use the training library.  It specifies specific module versions that were used to test the training library, but more recent versions of these modules may also work.  The training library has been tested on Windows 10 x64 and Ubuntu 18.04 LTS.
+
+## Installing Miniconda
+
+[Miniconda][Miniconda] is a cross platform distribution of python that includes a utility for managing packages called conda.  It allows for maintenance of multiple python environments that each have different modules or libraries installed.  Download the version for python 3 and install it for your operating system of choice.
+
+## Creating a new environment (optional)
+
+To create a new environment, use this command:
+
+```shell
+conda create --name openem
+```
+
+This will create a new environment called openem.  To start using it:
+
+```shell
+source activate openem
+```
+
+If you do not do these steps then modules will be installed in the base environment, which is activated by default. This is fine if you do not plan to use Miniconda for other purposes than OpenEM training.
+
+## Install modules
+
+Versions are included here for reference, different versions may work but have not been tested.
+
+```shell
+conda install keras-gpu==2.2.4
+conda install pandas==0.23.4
+conda install opencv==3.4.2
+conda install scikit-image==0.14.0
+conda install scikit-learn==0.20.1
+```
+
+[Miniconda]: https://conda.io/miniconda.html

doc/tutorial.md

@@ -1,11 +1,13 @@
-## Tutorial
+# Tutorial
 
 This tutorial will guide you through some examples.  Before you get 
 started, make sure you have built the library, including its examples by
 following the [build instructions](build.md) for your operating
 system.  You can also download the library from the 
 [release section][Releases] if available for your operating system.
 
+## Deployment Library
+
 First, you will need to download the [OpenEM example data][ExampleData].  The
 example data includes model files for the deployment library in protobuf 
 format, as well as input data that can be used to run the examples.  You can
@@ -44,9 +46,55 @@ built the software.
 Once you are able to run the examples, you are encouraged to inspect the 
 source code for the language that you plan to use for your application.
 
+## Training Library
+
+To build your own models, you will first need to create an appropriate python environment according to the training environment [instructions][TrainingEnvironment]. You will then need to modify the configuration file included with this repository at train/train.ini. This file is included as an example but you will need to modify some paths in it to get it working. Start by making a copy of this file and modify the paths section as follows:
+
+```shell
+[Paths]
+# Path to directory that contains training data.
+TrainDir=<Path to OpenEM example data>/train
+# Path to directory for storing intermediate outputs.
+WorkDir=<Path where you want to store working files>
+# Path to directory for storing final model outputs.
+ModelDir=<Path where you want to store models>
+```
+
+TrainDir is the path to the example training data.  WorkDir is where temporary files are stored during training. ModelDir contains model outputs that can be used directly by the deployment library.  Once you have modified your copy of train.ini to use the right paths on your system, you can do the following:
+
+```shell
+python train.py train.ini preprocess
+```
+
+Where train.ini is your modified copy. This command will go through the videos and annotations and start dumping images into the working directory. It will take a few hours to complete.
+
+Next, you can do some training. To train the detection model you can do the following command:
+
+```shell
+python train.py train.ini detect
+```
+
+This will start training the detection model.  This will take longer, potentially a couple days.  If you want to monitor the training outside of the command line, you can use Tensorboard. This is a program that serves a webpage for monitoring losses during training runs. Use the following command:
+
+```shell
+tensorboard --logdir <path to WorkDir>/tensorboard --port 10000
+```
+
+Then you can open a web browser on the same machine and go to 127.0.0.1:10000. This will display a live view of the training results. You can also use a different machine on the same network and modify the IP address accordingly.
+
+Once training completes, a new model will be converted to protobuf format at the location specified in train.ini as the ModelDir, subdirectory detect. The file here, detect.pb, is the same format used in the example data for the deployment library.
+
+
+# Building Datasets
+
+Now that you have done training using the example data, you can try doing the same with your own data.  Follow the [data collection][DataCollection] and [annotation][Annotation] guidelines to build your own training set. Once you have a dataset, you can modify the train.ini file's Data section to include new species to match your data, then repeat the same training process you went through with the example data.
+
 [Releases]: https://github.com/openem-team/openem/releases
 [ExampleData]:https://drive.google.com/drive/folders/18silAFzXaP27VHLS0texHJz1ZxSMGhjx?usp=sharing
 [ExampleSources]: ../examples/deploy
 [Anaconda]: https://www.anaconda.com/download/
 [RunAll]: ../examples/deploy/run_all.py
+[TrainingEnvironment]: ./training_environment.md
+[DataCollection]: ./data_collection.md
+[Annotation]: ./annotation.md
 

train/openem_train/detect.py

@@ -0,0 +1,121 @@
+"""Functions for training detection algorithm.
+"""
+
+import os
+import glob
+import numpy as np
+from keras.optimizers import Adam
+from keras.callbacks import ModelCheckpoint
+from keras.callbacks import TensorBoard
+from openem_train.ssd import ssd
+from openem_train.ssd.ssd_training import MultiboxLoss
+from openem_train.ssd.ssd_utils import BBoxUtility
+from openem_train.ssd.ssd_dataset import SSDDataset
+from openem_train.util.model_utils import keras_to_tensorflow
+
+def _save_model(config, model):
+    """Loads best weights and converts to protobuf file.
+
+    # Arguments
+        config: ConfigInterface object.
+        model: Keras Model object.
+    """
+    best = glob.glob(os.path.join(config.checkpoints_dir(), '*best*'))
+    latest = max(best, key=os.path.getctime)
+    model.load_weights(latest)
+    os.makedirs(config.detect_model_dir(), exist_ok=True)
+    keras_to_tensorflow(model, ['output_node0'], config.detect_model_path())
+
+def train(config):
+    """Trains detection model.
+
+    # Arguments
+        config: ConfigInterface object.
+    """
+
+    # Create tensorboard and checkpoints directories.
+    os.makedirs(config.checkpoints_dir(), exist_ok=True)
+    os.makedirs(config.tensorboard_dir(), exist_ok=True)
+
+    # Build the ssd model.
+    model = ssd.ssd_model(
+        input_shape=(config.detect_height(), config.detect_width(), 3),
+        num_classes=config.num_classes())
+
+    # Set trainable layers.
+    for layer in model.layers:
+        layer.trainable = True
+
+    # Set up loss and optimizer.
+    loss_obj = MultiboxLoss(
+        config.num_classes(),
+        neg_pos_ratio=2.0,
+        pos_cost_multiplier=1.1)
+    adam = Adam(lr=3e-5)
+
+    # Compile the model.
+    model.compile(loss=loss_obj.compute_loss, optimizer=adam)
+    model.summary()
+
+    # Get prior box layers from model.
+    prior_box_names = [
+        'conv4_3_norm_mbox_priorbox',
+        'fc7_mbox_priorbox',
+        'conv6_2_mbox_priorbox',
+        'conv7_2_mbox_priorbox',
+        'conv8_2_mbox_priorbox',
+        'pool6_mbox_priorbox']
+    priors = []
+    for prior_box_name in prior_box_names:
+        layer = model.get_layer(prior_box_name)
+        if layer is not None:
+            priors.append(layer.prior_boxes)
+    priors = np.vstack(priors)
+
+    # Set up bounding box utility.
+    bbox_util = BBoxUtility(config.num_classes(), priors)
+
+    # Set up dataset interface.
+    dataset = SSDDataset(
+        config,
+        bbox_util=bbox_util,
+        preproc=lambda x: x)
+
+    # Set up keras callbacks.
+    checkpoint_best = ModelCheckpoint(
+        config.checkpoint_best(),
+        verbose=1,
+        save_weights_only=False,
+        save_best_only=True)
+
+    checkpoint_periodic = ModelCheckpoint(
+        config.checkpoint_periodic(),
+        verbose=1,
+        save_weights_only=False,
+        period=1)
+
+    tensorboard = TensorBoard(
+        config.tensorboard_dir(),
+        histogram_freq=0,
+        write_graph=True,
+        write_images=True)
+
+    # Fit the model.
+    batch_size = config.detect_batch_size()
+    val_batch_size = config.detect_val_batch_size()
+    model.fit_generator(
+        dataset.generate_ssd(
+            batch_size=batch_size,
+            is_training=True),
+        steps_per_epoch=dataset.nb_train_samples // batch_size,
+        epochs=config.detect_num_epochs(),
+        verbose=1,
+        callbacks=[checkpoint_best, checkpoint_periodic, tensorboard],
+        validation_data=dataset.generate_ssd(
+            batch_size=val_batch_size,
+            is_training=False),
+        validation_steps=dataset.nb_test_samples // val_batch_size,
+        initial_epoch=0)
+
+    # Load weights of the best model.
+    _save_model(config, model)