Merge branch 'development'

.gitignore

@@ -14,3 +14,7 @@ tests/data/*
 tierpsy.egg-info/
 _site/
 Gemfile*
+
+*.o
+
+tierpsy/extras/unet_RMSprop-5-04999-0.3997.h5

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2017 Medical Research Council
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,8 +1,13 @@
 # Tierpsy Tracker
+
 Tierpsy Tracker is a multi-animal tracker developed in the [MRC-LMS](http://lms.mrc.ac.uk/) [Behavioural Genomics Group](http://behave.csc.mrc.ac.uk/) at Imperial College London. The project combines the throughput of multiworm tracking with the resolution of single worm tracking, which means you can extract detailed phenotypic fingerprints from more animals.  A large field of view also makes it possible to analyse collective behaviours that depend on animal interactions.  The code is open source and extensible and we have ongoing projects using the tracker to analyse fish, *Drosophila* larvae, and fluorescently labelled worms.
 
+*Double-click executables are available for Windows (7 or latest) and OSX (Yosemite or latest) can be found in the [releases page](https://github.com/ver228/tierpsy-tracker/releases).*
+
+
 <img src="https://cloud.githubusercontent.com/assets/8364368/26658216/8d5599b2-4660-11e7-911b-c390330a15ee.gif" width="480">
 
 ## [Installation Instructions](docs/INSTALLATION.md)
 ## [How to Use](docs/HOWTO.md)
 ## [Algorithm Explanation](docs/EXPLANATION.md)
+## [Output Files](docs/OUTPUTS.md)

create_binaries/issues.txt

@@ -23,4 +23,6 @@ hiddenimports=['h5py.defs', 'h5py.utils', 'h5py.h5ac', 'h5py._proxy',
 	conda remove mkl mkl-service
 
 * I recieve an error of a missing package 'future'. Just use:
-	conda install futures
+	conda install futures
+
+* OSX the ffprobe of conda has relative links to the libraries and it is complicated to include in the binary. It is better to use the one of homebrew.

docs/EXPLANATION.md

@@ -69,11 +69,11 @@ This step identifies skeletons that are likely to have been produced by an inacc
 This step orients the skeleton head and tail by movement. The [SKE_CREATE](#ske_create) step produces a skeleton but does not determine which extreme of the curve is the head and which the tail. Since the head in one skeleton will not suddenly jump to the other side of the worm within a small amout of time, it is possible to assign "blocks" of skeletons with the same orientation as long as there is not a gap of missing skeletons larger than a few frames (set by `max_gap_allowed_block`). For each of these blocks we estimate the motility of each of the skeleton extremes as the standard deviation of their angular speed. The part corresponding to the head should have a larger motility than the one corresponding to the tail. This approach is able to correct most of the skeletons but further improvements can be archieve using the [image intensity](#int_ske_orient).
 
 ### STAGE_ALIGMENT
-Only used in for data from [Worm Tracker 2.0](http://www.mrc-lmb.cam.ac.uk/wormtracker/) (`SINGLE_WORM_SHAFER`). In Worm Tracker 2.0, a single worm is follow by a camera around the plate. Analysis of these videos requires the extra step of shifting skeletons coordinates from the image frame of reference to the stage frame of reference. In this step the recorded stage positions saved as a `.log.csv` file are aligned to the the video time. This step requires MATLAB to run the corresponding function from the original [segWorm](https://github.com/openworm/SegWorm) code. Please raise an [issue](https://github.com/ver228/tierpsy-tracker/issues) if you are interested in the development of a python only implementation.
+Only used in for data from [Worm Tracker 2.0](http://www.mrc-lmb.cam.ac.uk/wormtracker/) (`WT2`). In Worm Tracker 2.0, a single worm is follow by a camera around the plate. Analysis of these videos requires the extra step of shifting skeletons coordinates from the image frame of reference to the stage frame of reference. In this step the recorded stage positions saved as a `.log.csv` file are aligned to the the video time. This step requires MATLAB to run the corresponding function from the original [segWorm](https://github.com/openworm/SegWorm) code. Please raise an [issue](https://github.com/ver228/tierpsy-tracker/issues) if you are interested in the development of a python only implementation.
 
 
 ### CONTOUR_ORIENT
-Only used in `SINGLE_WORM_SHAFER`. This step switches the dorsal and ventral contours to match the `ventral_side` orientation specified in the `/experiment_info` field or in the JSON parameters file.
+Only used in `WT2`. This step switches the dorsal and ventral contours to match the `ventral_side` orientation specified in the `/experiment_info` field or in the JSON parameters file.
 
 ### INT_PROFILE
 This step uses worm intensity profiles to improve head-tail orientation detection.  Worms are first straightened using the previously extracted skeleton (see below A) and interpolating it into a 2D map (see below B top). The intensity map can be further smoothed by averaging data along the worm to obtain an intensity profile as shown below in B bottom.  
@@ -94,5 +94,5 @@ This step uses the [open worm analysis toolbox](https://github.com/openworm/open
 Same as [FEAT_CREATE](#feat_create) but it will only use the indexes that were manually identified as worms using the [Tierpsy Tracker Viewer](HOWTO.md#tierpsy-tracker-viewer). The results will be saved as `basename_feat_manual.hdf5`.
 
 ### WCON_EXPORT
-Currently only used in `SINGLE_WORM_SHAFER`. Export skeletons data in [`basename_features.hdf5`](OUTPUTS.md/#basename_features.hdf5) using the [WCON format](https://github.com/openworm/tracker-commons). In the future this step should be available in the default analysis sequence.
+Currently only used in `WT2`. Export skeletons data in [`basename_features.hdf5`](OUTPUTS.md/#basename_features.hdf5) using the [WCON format](https://github.com/openworm/tracker-commons). In the future this step should be available in the default analysis sequence.
 

docs/HOWTO.md

@@ -62,6 +62,16 @@ Tierpsy Tracker will analyse of the video files that are present in `Original Vi
 
 Alternatively, one can create a text file with the list of files to be analysed (one file per line). The path to this file can then be set in `Individual File List`. 
 
+### Parameters Files
+Parameters files created using the [Set Parameters](#set-parameters) widget can be select in the `Parameter Files` box. You can also select some previously created files using the drop-down list. If no file is selected the [default values](https://github.com/ver228/tierpsy-tracker/blob/dev/tierpsy/helper/params/docs_tracker_param.py) will be used. 
+
+#### Worm Tracker 2.0 Option
+
+You can analyse videos created by the [Worm Tracker 2.0](http://www.mrc-lmb.cam.ac.uk/wormtracker/) by selecting the parameters files [WT2_clockwise.json](https://github.com/ver228/tierpsy-tracker/blob/development/tierpsy/extras/param_files/WT2_clockwise.json) or
+[WT2_anticlockwise.json](https://github.com/ver228/tierpsy-tracker/blob/development/tierpsy/extras/param_files/WT2_anticlockwise.json). Use the former if the ventral side in the videos is located in the clockwise direction from the worm head, and the later if it is in the anticlockwise direction. To select a subset of files with a particular orientation you can save each subset in a different root directory or include the orientation information in the file name and use use the `Pattern include` option . If you need to fine-tune the parameters you can edit the .json files either with a text editor or with `Set Parameters`.
+
+Note that each of the video files `.avi` must have an additional pair of files with the extensions `.info.xml` and `.log.csv`. Additionally, keep in mind that if the stage aligment step fails, an error will be risen and the analysis of that video will be stopped. If you do not want to see the error messages untick the option `Print debug information`.
+
 
 ### Analysis Progress 
 
@@ -70,16 +80,14 @@ Tierpsy Tracker will determine which analysis steps have already been completed
 * To see only a summary of the files to be analysed without starting the analysis tick `Only Display Progress Summary`.
 
 * You can start or end the analysis at specific points by using the `Analysis Start Point` and `Analysis End Point` drop-down menus. 
-* If you want to re-analyse a file from an earlier step, delete or rename the output files that were created during the previous run. If you only want to overwrite a particular step, you have to delete the corresponding step in the `/provenance_tracking` node in the corresponding file. 
+
+* If you want to re-analyse a file from an earlier step, delete or rename the output files that were created during the previous run. If you only want to overwrite a particular step, you have to delete the corresponding step in the `/provenance_tracking` node in the corresponding file.
 
 ### Directory to Save the Output Files
 The masked videos created in the [compression step](EXPLANATION.md/#video-compression) are stored in `Masked Videos Dir`. The rest of the tracker results are stored in `Tracking Results Dir`. In both cases the subdirectory tree structure in `Original Videos Dir` is recreated. 
 
 The reason for creating the parallel directory structure is to make it easy to delete the analysis outputs to re-run with different parameter values.  It also makes it easy to delete the original videos to save space once you've arrived at good parameter values. If you prefer to have the output files in the same directory as the original videos you can set `Masked Videos Dir` and `Tracking Results Dir` to the same value.
 
-### Parameters Files
-Parameters files created using the [Set Parameters](#set-parameters) widget can be select in the `Parameter Files` box. You can also select some previously created files using the drop-down list. If no file is selected the [default values](https://github.com/ver228/tierpsy-tracker/blob/dev/tierpsy/helper/params/docs_tracker_param.py) will be used. 
-
 ### Temporary directory
 By default, Tierpsy Tracker creates files in the `Temporary Dir` and only moves them to the `Masked Videos Dir` or the `Tracking Results Dir` when the analysis has finished. The reasons to use a temporary directory are:
 
@@ -143,7 +151,7 @@ You can manually correct the trajectories as shown below. Once you have finished
 
 `Right key` : Decrease the frame by step size.
 
-## Single Worm Viewer
-This is simplified version of the [Tierpsy Tracker Viewer](#tierpsy-tracker-viewer) created specifically to view files created using [Worm Tracker 2.0](http://www.mrc-lmb.cam.ac.uk/wormtracker/) (the `SINGLE_WORM_SHAFER` case). It can be used as shown below.
+## Worm Tracker 2.0 Viewer
+This is simplified version of the [Tierpsy Tracker Viewer](#tierpsy-tracker-viewer) created specifically to view files created using [Worm Tracker 2.0](http://www.mrc-lmb.cam.ac.uk/wormtracker/) (the `WT2` case). [Above](#worm-tracker-20-option) is described how to analyse this type of files.
 
 ![SWTrackerViewer](https://cloud.githubusercontent.com/assets/8364368/26412826/e608bfea-40a1-11e7-9d3e-d0b8bf482db2.gif) 

docs/INSTALLATION.md

@@ -1,36 +1,48 @@
-# Installation
+*The instructions below are to install Tierpsy Tracker from the source code. I would recommend to do this only if you are using Linux or want to run the development version, otherwise use the double-click executables available for Windows (7 or latest) and OSX (Yosemite or latest) in the [releases page](https://github.com/ver228/tierpsy-tracker/releases).*
+
+# System Requirements 
+- Freshly installed [miniconda] (https://conda.io/miniconda.html) or at least setup up a new enviroment.
+- Optional [ffmpeg](https://ffmpeg.org/download.html): ffprobe must be accessible from the command line to calculate the video timestamps.
+- [C compiler compatible with cython](http://cython.readthedocs.io/en/latest/src/quickstart/install.html). In Windows, you can use [Visual C++ 2015 Build Tools](http://landinghub.visualstudio.com/visual-cpp-build-tools). In OSX, if you install [homebrew](https://brew.sh/) it will setup the C compiler without the need to download XCode from the appstore. 
+- [Git](https://git-scm.com/). [Here](https://gist.github.com/derhuerst/1b15ff4652a867391f03) are some instructions to install it.
 
-To install Tierpsy tracker, download the latest version to your local machine.  This can be done manually [here](https://github.com/ver228/tierpsy-tracker/archive/master.zip).  If you use Git or [Github Desktop](https://desktop.github.com/), then open a new terminal (in Windows open Git Shell) and run: 
+# Installation
 
+1. Clone this repository either using the [Github Desktop](https://desktop.github.com/) or from the command line as:
 ```bash
 git clone https://github.com/ver228/tierpsy-tracker
 ```
-## Installation for OSX
-Open a terminal and run `bash installation/installation_script.sh`.
-
-## Installation for Linux
-For linux run
-``` bash
-bash installation/installation_script.sh --linux_dependencies #You need sudo permissions
-bash installation/installation_script.sh
+
+2. Install the conda dependencies from the conda-forge channel:
+```bash
+conda config --add channels conda-forge 
+
+conda install -y numpy matplotlib pytables pandas gitpython pyqt=5 \
+h5py scipy scikit-learn scikit-image seaborn xlrd cython statsmodels
+
+conda install -y -c conda-forge keras opencv
+pip install tensorflow 
 ```
-I have tested the script in Centos7. If you have other versions take a look to the dependencies in the [installation_script.sh](https://github.com/ver228/tierpsy-tracker/blob/master/installation/installation_script.sh) on function `ubuntu_dependencies` or `redhat_dependencies` and adjust them according to your distribution.
 
-## Installation for Windows
+3. Install the rest of the modules:
+On the tierpsy-tracker root folder (the folder with the cloned repository) type:
+```bash
+bash installation/installation_script.sh #OSX or Linux
 
-- Download and install [miniconda](https://conda.io/miniconda.html).
-- Install [ffmpeg](https://ffmpeg.org/download.html). Friendly installation instructions can be found [here](http://adaptivesamples.com/how-to-install-ffmpeg-on-windows/).
-- Install [Visual C++ 2015 Build Tools](http://landinghub.visualstudio.com/visual-cpp-build-tools).
-- Run `./tierpsy-tracker/installation/installation_script.bat`.
+installation/installation_script.bat #Windows
+```
 
 ## Possible Issues
-Most of the problems that can occur during installation are due to missing/conflicting dependencies, especially if there were older versions of miniconda installed. Try updating miniconda and re-running the scripts. Many of these problems can be solved by searching for error messages online, but if problems persist, please raise an issue on Github [project page](https://github.com/ver228/tierpsy-tracker/issues).
+- The most common problem in the installation is OpenCV (error in import cv2). Try a fresh miniconda installation (or a fresh enviroment) and make sure your are using the [conda-forge](https://conda-forge.org/) packages. It this does not work I am afraid you would have to solve the problem by yourself (Google is your friend).
+
+- I known bug occurs if you install the [Open Worm Analysis Toolbox](https://github.com/openworm/open-worm-analysis-toolbox) using pip. The installer is broken and it will create a corrupt [.egg](https://stackoverflow.com/questions/2051192/what-is-a-python-egg) in your packages folder. Manually delete the .egg (use the error traceback to find the its location) and re-run `installation_script.sh`.
+
 
-## Test Examples
+# Test Examples
 On Mac OSX or Linux, some test examples can be downloaded by running 
 
 ```bash
-installation/instalation_script.sh --tests
+installation/instalation_script.sh --download_examples
 ```
 
 The tests can also be manually downloaded using [this link](https://imperiallondon-my.sharepoint.com/personal/ajaver_ic_ac_uk/_layouts/15/guestaccess.aspx?guestaccesstoken=ldZ18fLY%2bzlu7XuO9mbKVdyiKoH4naiesqiLXWU4vGQ%3d&docid=0cec4e52f4ccf4d5b8bb3a737020fc12f&rev=1). Uncompress the data and save it in the main repository folder `tests/data` .