examples/notebooks/stats/index.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Predicting inter-cell statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "This tutorial describes how to predict inter-cell statistics such as the mean methylation rate or variance across cells."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Initialization\n",
    "We first initialize some variables that will be used throughout the tutorial. `test_mode=1` should be used for testing purposes, which speeds up computations by only using a subset of the data. For real applications, `test_mode=0` should be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "function run {\n",
    "  local cmd=$@\n",
    "  echo\n",
    "  echo \"#################################\"\n",
    "  echo $cmd\n",
    "  echo \"#################################\"\n",
    "  eval $cmd\n",
    "}\n",
    "\n",
    "test_mode=1 # Set to 1 for testing and 0 otherwise.\n",
    "example_dir=\"../../data\" # Directory with example data.\n",
    "cpg_dir=\"$example_dir/cpg\" # Directory with CpG profiles.\n",
    "dna_dir=\"$example_dir/dna/mm10\" # Directory with DNA sequences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Creating DeepCpG data files\n",
    "`dcpg_data.py` provides the arguments `--cpg_stats` and `--win_stats` to compute statistics across cells for single CpG sites or in windows of lengths `--win_stats_wlen` centred on CpG sites, respectively. Supported statistics are described in the [documentation](http://deepcpg.readthedocs.io/en/latest/data.html#predicting-statistics) and include the mean methylation rate (`mean`), variance (`var`), and if a CpG site is differentially methylated (`diff`). With `--cpg_stats_cov`, per-CpG statistics will be computed only for CpG sites that are covered by at least the specified number of cells. If this number of too low, estimated statistics might be unreliable in lowly covered regions. We will compute the mean methylation rate and variance across cells in windows of different lengths, and if CpG sites with at least three observations are differentially methylated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "#################################\n",
      "dcpg_data.py --cpg_profiles ../../data/cpg/BS27_1_SER.tsv ../../data/cpg/BS27_3_SER.tsv ../../data/cpg/BS27_5_SER.tsv ../../data/cpg/BS27_6_SER.tsv ../../data/cpg/BS27_8_SER.tsv --dna_files ../../data/dna/mm10 --out_dir ./data --dna_wlen 1001 --cpg_wlen 50 --cpg_stats diff --cpg_stats_cov 3 --win_stats mean var --win_stats_wlen 1001 2001 3001 4001 5001 --chromo 1 13 --nb_sample_chromo 10000\n",
      "#################################\n",
      "INFO (2017-04-11 10:12:34,952): Reading CpG profiles ...\n",
      "INFO (2017-04-11 10:12:34,952): ../../data/cpg/BS27_1_SER.tsv\n",
      "INFO (2017-04-11 10:12:41,466): ../../data/cpg/BS27_3_SER.tsv\n",
      "INFO (2017-04-11 10:12:46,144): ../../data/cpg/BS27_5_SER.tsv\n",
      "INFO (2017-04-11 10:12:53,149): ../../data/cpg/BS27_6_SER.tsv\n",
      "INFO (2017-04-11 10:12:58,623): ../../data/cpg/BS27_8_SER.tsv\n",
      "INFO (2017-04-11 10:13:04,171): 20000 samples\n",
      "INFO (2017-04-11 10:13:04,172): --------------------------------------------------------------------------------\n",
      "INFO (2017-04-11 10:13:04,172): Chromosome 1 ...\n",
      "INFO (2017-04-11 10:13:04,225): 10000 / 10000 (100.0%) sites matched minimum coverage filter\n",
      "INFO (2017-04-11 10:13:08,288): Chunk \t1 / 1\n",
      "INFO (2017-04-11 10:13:08,296): Computing per CpG statistics ...\n",
      "INFO (2017-04-11 10:13:08,300): Extracting DNA sequence windows ...\n",
      "INFO (2017-04-11 10:13:10,457): Extracting CpG neighbors ...\n",
      "INFO (2017-04-11 10:13:11,602): Computing window-based statistics ...\n",
      "INFO (2017-04-11 10:13:12,226): --------------------------------------------------------------------------------\n",
      "INFO (2017-04-11 10:13:12,226): Chromosome 13 ...\n",
      "INFO (2017-04-11 10:13:12,276): 10000 / 10000 (100.0%) sites matched minimum coverage filter\n",
      "INFO (2017-04-11 10:13:14,456): Chunk \t1 / 1\n",
      "INFO (2017-04-11 10:13:14,462): Computing per CpG statistics ...\n",
      "INFO (2017-04-11 10:13:14,465): Extracting DNA sequence windows ...\n",
      "INFO (2017-04-11 10:13:16,464): Extracting CpG neighbors ...\n",
      "INFO (2017-04-11 10:13:17,588): Computing window-based statistics ...\n",
      "INFO (2017-04-11 10:13:18,211): Done!\n"
     ]
    }
   ],
   "source": [
    "data_dir=\"./data\"\n",
    "cmd=\"dcpg_data.py\n",
    "    --cpg_profiles $cpg_dir/*.tsv\n",
    "    --dna_files $dna_dir\n",
    "    --out_dir $data_dir\n",
    "    --dna_wlen 1001\n",
    "    --cpg_wlen 50\n",
    "    --cpg_stats diff\n",
    "    --cpg_stats_cov 3\n",
    "    --win_stats mean var\n",
    "    --win_stats_wlen 1001 2001 3001 4001 5001\n",
    "\"\n",
    "if [[ $test_mode -eq 1 ]]; then\n",
    "    cmd=\"$cmd\n",
    "        --chromo 1 13\n",
    "        --nb_sample_chromo 10000\n",
    "        \"\n",
    "fi\n",
    "run $cmd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Model training \n",
    "We will train a DNA model to predict mean methylation rates, cell-to-cell variance, and differentially methylated CpG sites from the DNA sequence alone. However, you could train a CpG model or Joint model to also use neighboring CpG sites for making predictions. To predict all per-CpG and window-based statistics computed by `dcpg_data.py` instead of methylation states, we are running `dcpg_train.py` with `--output_names 'cpg_stats/.*' 'win_stats/.*'`. You could use `--output_names '.*'` to predict both methylation states and statistics. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training files:\n",
      "./data/c1_000000-010000.h5\n",
      "\n",
      "Validation files:\n",
      "./data/c13_000000-010000.h5\n"
     ]
    }
   ],
   "source": [
    "train_files=$(ls $data_dir/c{1,3,5,7,9}_*.h5 2> /dev/null)\n",
    "echo \"Training files:\"\n",
    "echo $train_files\n",
    "echo\n",
    "\n",
    "val_files=$(ls $data_dir/c{13,14,15,16,17}_*.h5 2> /dev/null)\n",
    "echo \"Validation files:\"\n",
    "echo $val_files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "#################################\n",
      "dcpg_train.py ./data/c1_000000-010000.h5 --val_files ./data/c13_000000-010000.h5 --dna_model CnnL2h128 --out_dir ./model --output_names 'cpg_stats/.*' 'win_stats/.*' --nb_epoch 1 --nb_train_sample 10000 --nb_val_sample 10000\n",
      "#################################\n",
      "Using TensorFlow backend.\n",
      "INFO (2017-04-11 10:13:23,483): Building model ...\n",
      "INFO (2017-04-11 10:13:23,488): Building DNA model ...\n",
      "____________________________________________________________________________________________________\n",
      "Layer (type)                     Output Shape          Param #     Connected to                     \n",
      "====================================================================================================\n",
      "dna (InputLayer)                 (None, 1001, 4)       0                                            \n",
      "____________________________________________________________________________________________________\n",
      "dna/convolution1d_1 (Convolution (None, 991, 128)      5760        dna[0][0]                        \n",
      "____________________________________________________________________________________________________\n",
      "dna/activation_1 (Activation)    (None, 991, 128)      0           dna/convolution1d_1[0][0]        \n",
      "____________________________________________________________________________________________________\n",
      "dna/maxpooling1d_1 (MaxPooling1D (None, 247, 128)      0           dna/activation_1[0][0]           \n",
      "____________________________________________________________________________________________________\n",
      "dna/convolution1d_2 (Convolution (None, 245, 256)      98560       dna/maxpooling1d_1[0][0]         \n",
      "____________________________________________________________________________________________________\n",
      "dna/activation_2 (Activation)    (None, 245, 256)      0           dna/convolution1d_2[0][0]        \n",
      "____________________________________________________________________________________________________\n",
      "dna/maxpooling1d_2 (MaxPooling1D (None, 122, 256)      0           dna/activation_2[0][0]           \n",
      "____________________________________________________________________________________________________\n",
      "dna/flatten_1 (Flatten)          (None, 31232)         0           dna/maxpooling1d_2[0][0]         \n",
      "____________________________________________________________________________________________________\n",
      "dna/dense_1 (Dense)              (None, 128)           3997824     dna/flatten_1[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "dna/activation_3 (Activation)    (None, 128)           0           dna/dense_1[0][0]                \n",
      "____________________________________________________________________________________________________\n",
      "dna/dropout_1 (Dropout)          (None, 128)           0           dna/activation_3[0][0]           \n",
      "____________________________________________________________________________________________________\n",
      "dense_2 (Dense)                  (None, 1)             129         dna/dropout_1[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "dense_3 (Dense)                  (None, 1)             129         dna/dropout_1[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "dense_4 (Dense)                  (None, 1)             129         dna/dropout_1[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "dense_5 (Dense)                  (None, 1)             129         dna/dropout_1[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "dense_6 (Dense)                  (None, 1)             129         dna/dropout_1[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "cpg_stats/diff (Dense)           (None, 1)             129         dna/dropout_1[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "win_stats/1001/mean (Dense)      (None, 1)             129         dna/dropout_1[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "win_stats/1001/var (ScaledSigmoi (None, 1)             0           dense_2[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "win_stats/2001/mean (Dense)      (None, 1)             129         dna/dropout_1[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "win_stats/2001/var (ScaledSigmoi (None, 1)             0           dense_3[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "win_stats/3001/mean (Dense)      (None, 1)             129         dna/dropout_1[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "win_stats/3001/var (ScaledSigmoi (None, 1)             0           dense_4[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "win_stats/4001/mean (Dense)      (None, 1)             129         dna/dropout_1[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "win_stats/4001/var (ScaledSigmoi (None, 1)             0           dense_5[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "win_stats/5001/mean (Dense)      (None, 1)             129         dna/dropout_1[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "win_stats/5001/var (ScaledSigmoi (None, 1)             0           dense_6[0][0]                    \n",
      "====================================================================================================\n",
      "Total params: 4,103,563\n",
      "Trainable params: 4,103,563\n",
      "Non-trainable params: 0\n",
      "____________________________________________________________________________________________________\n",
      "INFO (2017-04-11 10:13:23,808): Computing output statistics ...\n",
      "Output statistics:\n",
      "               name | nb_tot | nb_obs | frac_obs | mean |  var\n",
      "--------------------------------------------------------------\n",
      "     cpg_stats/diff |  10000 |      3 |     0.00 | 0.00 | 0.00\n",
      "win_stats/1001/mean |  10000 |  10000 |     1.00 | 0.74 | 0.17\n",
      " win_stats/1001/var |  10000 |  10000 |     1.00 | 0.02 | 0.00\n",
      "win_stats/2001/mean |  10000 |  10000 |     1.00 | 0.74 | 0.16\n",
      " win_stats/2001/var |  10000 |  10000 |     1.00 | 0.03 | 0.00\n",
      "win_stats/3001/mean |  10000 |  10000 |     1.00 | 0.74 | 0.15\n",
      " win_stats/3001/var |  10000 |  10000 |     1.00 | 0.03 | 0.01\n",
      "win_stats/4001/mean |  10000 |  10000 |     1.00 | 0.74 | 0.14\n",
      " win_stats/4001/var |  10000 |  10000 |     1.00 | 0.04 | 0.01\n",
      "win_stats/5001/mean |  10000 |  10000 |     1.00 | 0.74 | 0.13\n",
      " win_stats/5001/var |  10000 |  10000 |     1.00 | 0.04 | 0.01\n",
      "\n",
      "Class weights:\n",
      "cpg_stats/diff\n",
      "--------------\n",
      "        0=0.00\n",
      "        1=1.00\n",
      "\n",
      "INFO (2017-04-11 10:13:24,689): Loading data ...\n",
      "INFO (2017-04-11 10:13:24,692): Initializing callbacks ...\n",
      "INFO (2017-04-11 10:13:24,693): Training model ...\n",
      "\n",
      "Training samples: 10000\n",
      "Validation samples: 10000\n",
      "Epochs: 1\n",
      "Learning rate: 0.0001\n",
      "====================================================================================================\n",
      "Epoch 1/1\n",
      "====================================================================================================\n",
      "done (%) | time |   loss |    acc |    mse |    mae | cpg_stats/diff_loss | win_stats/1001/mean_loss | win_stats/1001/var_loss | win_stats/2001/mean_loss | win_stats/2001/var_loss | win_stats/3001/mean_loss | win_stats/3001/var_loss | win_stats/4001/mean_loss | win_stats/4001/var_loss | win_stats/5001/mean_loss | win_stats/5001/var_loss | cpg_stats/diff_acc | win_stats/1001/mean_acc | win_stats/2001/mean_acc | win_stats/3001/mean_acc | win_stats/4001/mean_acc | win_stats/5001/mean_acc | win_stats/1001/mean_mse | win_stats/1001/var_mse | win_stats/2001/mean_mse | win_stats/2001/var_mse | win_stats/3001/mean_mse | win_stats/3001/var_mse | win_stats/4001/mean_mse | win_stats/4001/var_mse | win_stats/5001/mean_mse | win_stats/5001/var_mse | win_stats/1001/mean_mae | win_stats/1001/var_mae | win_stats/2001/mean_mae | win_stats/2001/var_mae | win_stats/3001/mean_mae | win_stats/3001/var_mae | win_stats/4001/mean_mae | win_stats/4001/var_mae | win_stats/5001/mean_mae | win_stats/5001/var_mae\n",
      "---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "     1.3 |  0.0 | 3.9137 | 0.5938 | 0.1080 | 0.2687 |              0.0000 |                   0.2227 |                  0.0153 |                   0.2121 |                  0.0150 |                   0.2018 |                  0.0147 |                   0.1842 |                  0.0141 |                   0.1871 |                  0.0134 |                nan |                  0.5469 |                  0.7578 |                  0.7422 |                  0.6875 |                  0.2344 |                  0.2227 |                 0.0153 |                  0.2121 |                 0.0150 |                  0.2018 |                 0.0147 |                  0.1842 |                 0.0141 |                  0.1871 |                 0.0134 |                  0.4516 |                 0.1233 |                  0.4341 |                 0.1220 |                  0.4161 |                 0.1206 |                  0.3883 |                 0.1168 |                  0.4015 |                 0.1132\n",
      "    11.5 |  0.2 | 3.6974 | 0.7092 | 0.0998 | 0.2607 |              0.0000 |                   0.2065 |                  0.0148 |                   0.1912 |                  0.0149 |                   0.1792 |                  0.0130 |                   0.1800 |                  0.0141 |                   0.1726 |                  0.0120 |                nan |                  0.7066 |                  0.7283 |                  0.7222 |                  0.7222 |                  0.6667 |                  0.2065 |                 0.0148 |                  0.1912 |                 0.0149 |                  0.1792 |                 0.0130 |                  0.1800 |                 0.0141 |                  0.1726 |                 0.0120 |                  0.4366 |                 0.1213 |                  0.4139 |                 0.1210 |                  0.3958 |                 0.1122 |                  0.3967 |                 0.1165 |                  0.3862 |                 0.1066\n",
      "    21.8 |  0.3 | 3.4828 | 0.7534 | 0.0910 | 0.2385 |              0.0000 |                   0.1897 |                  0.0137 |                   0.1770 |                  0.0147 |                   0.1657 |                  0.0107 |                   0.1612 |                  0.0139 |                   0.1533 |                  0.0098 |             1.0000 |                  0.7040 |                  0.7146 |                  0.7123 |                  0.7082 |                  0.6811 |                  0.1897 |                 0.0137 |                  0.1770 |                 0.0147 |                  0.1657 |                 0.0107 |                  0.1612 |                 0.0139 |                  0.1533 |                 0.0098 |                  0.3983 |                 0.1163 |                  0.3744 |                 0.1200 |                  0.3587 |                 0.0994 |                  0.3617 |                 0.1155 |                  0.3480 |                 0.0922\n",
      "    32.0 |  0.5 | 3.3104 | 0.7589 | 0.0862 | 0.2257 |              0.0000 |                   0.1801 |                  0.0125 |                   0.1687 |                  0.0136 |                   0.1579 |                  0.0092 |                   0.1527 |                  0.0131 |                   0.1453 |                  0.0086 |             1.0000 |                  0.7087 |                  0.7166 |                  0.7156 |                  0.7153 |                  0.6972 |                  0.1801 |                 0.0125 |                  0.1687 |                 0.0136 |                  0.1579 |                 0.0092 |                  0.1527 |                 0.0131 |                  0.1453 |                 0.0086 |                  0.3792 |                 0.1107 |                  0.3548 |                 0.1149 |                  0.3451 |                 0.0898 |                  0.3410 |                 0.1123 |                  0.3257 |                 0.0833\n",
      "    42.2 |  0.6 | 3.1614 | 0.7601 | 0.0834 | 0.2211 |             -0.0000 |                   0.1763 |                  0.0118 |                   0.1641 |                  0.0124 |                   0.1535 |                  0.0087 |                   0.1474 |                  0.0122 |                   0.1398 |                  0.0082 |             1.0000 |                  0.7088 |                  0.7164 |                  0.7154 |                  0.7161 |                  0.7036 |                  0.1763 |                 0.0118 |                  0.1641 |                 0.0124 |                  0.1535 |                 0.0087 |                  0.1474 |                 0.0122 |                  0.1398 |                 0.0082 |                  0.3753 |                 0.1074 |                  0.3502 |                 0.1088 |                  0.3408 |                 0.0862 |                  0.3352 |                 0.1075 |                  0.3192 |                 0.0807\n",
      "    52.5 |  0.8 | 3.0189 | 0.7611 | 0.0809 | 0.2166 |             -0.0000 |                   0.1716 |                  0.0112 |                   0.1595 |                  0.0114 |                   0.1493 |                  0.0083 |                   0.1430 |                  0.0114 |                   0.1355 |                  0.0079 |             1.0000 |                  0.7128 |                  0.7178 |                  0.7149 |                  0.7155 |                  0.7058 |                  0.1716 |                 0.0112 |                  0.1595 |                 0.0114 |                  0.1493 |                 0.0083 |                  0.1430 |                 0.0114 |                  0.1355 |                 0.0079 |                  0.3682 |                 0.1038 |                  0.3447 |                 0.1033 |                  0.3348 |                 0.0835 |                  0.3299 |                 0.1031 |                  0.3154 |                 0.0789\n",
      "    62.7 |  0.9 | 2.8936 | 0.7622 | 0.0797 | 0.2124 |             -0.0000 |                   0.1702 |                  0.0104 |                   0.1581 |                  0.0105 |                   0.1477 |                  0.0079 |                   0.1406 |                  0.0106 |                   0.1330 |                  0.0076 |             1.0000 |                  0.7124 |                  0.7172 |                  0.7152 |                  0.7175 |                  0.7108 |                  0.1702 |                 0.0104 |                  0.1581 |                 0.0105 |                  0.1477 |                 0.0079 |                  0.1406 |                 0.0106 |                  0.1330 |                 0.0076 |                  0.3640 |                 0.0992 |                  0.3418 |                 0.0977 |                  0.3318 |                 0.0801 |                  0.3248 |                 0.0983 |                  0.3103 |                 0.0760\n",
      "    73.0 |  1.1 | 2.7751 | 0.7619 | 0.0787 | 0.2099 |             -0.0000 |                   0.1690 |                  0.0097 |                   0.1570 |                  0.0098 |                   0.1461 |                  0.0076 |                   0.1390 |                  0.0100 |                   0.1312 |                  0.0073 |             1.0000 |                  0.7116 |                  0.7149 |                  0.7150 |                  0.7171 |                  0.7129 |                  0.1690 |                 0.0097 |                  0.1570 |                 0.0098 |                  0.1461 |                 0.0076 |                  0.1390 |                 0.0100 |                  0.1312 |                 0.0073 |                  0.3628 |                 0.0950 |                  0.3413 |                 0.0932 |                  0.3298 |                 0.0779 |                  0.3229 |                 0.0943 |                  0.3079 |                 0.0741\n",
      "    83.2 |  1.2 | 2.6678 | 0.7590 | 0.0783 | 0.2083 |             -0.0000 |                   0.1693 |                  0.0091 |                   0.1570 |                  0.0092 |                   0.1457 |                  0.0074 |                   0.1381 |                  0.0096 |                   0.1302 |                  0.0073 |             1.0000 |                  0.7089 |                  0.7109 |                  0.7101 |                  0.7132 |                  0.7107 |                  0.1693 |                 0.0091 |                  0.1570 |                 0.0092 |                  0.1457 |                 0.0074 |                  0.1381 |                 0.0096 |                  0.1302 |                 0.0073 |                  0.3622 |                 0.0915 |                  0.3413 |                 0.0895 |                  0.3292 |                 0.0764 |                  0.3212 |                 0.0916 |                  0.3067 |                 0.0729\n",
      "    93.4 |  1.4 | 2.5577 | 0.7602 | 0.0772 | 0.2061 |             -0.0000 |                   0.1675 |                  0.0087 |                   0.1550 |                  0.0088 |                   0.1437 |                  0.0072 |                   0.1361 |                  0.0093 |                   0.1284 |                  0.0071 |             1.0000 |                  0.7097 |                  0.7119 |                  0.7120 |                  0.7149 |                  0.7125 |                  0.1675 |                 0.0087 |                  0.1550 |                 0.0088 |                  0.1437 |                 0.0072 |                  0.1361 |                 0.0093 |                  0.1284 |                 0.0071 |                  0.3604 |                 0.0882 |                  0.3395 |                 0.0866 |                  0.3269 |                 0.0750 |                  0.3189 |                 0.0893 |                  0.3047 |                 0.0716\n",
      "   100.0 |  1.5 | 2.4949 | 0.7592 | 0.0769 | 0.2051 |             -0.0000 |                   0.1671 |                  0.0084 |                   0.1545 |                  0.0086 |                   0.1435 |                  0.0071 |                   0.1359 |                  0.0091 |                   0.1280 |                  0.0071 |             1.0000 |                  0.7088 |                  0.7102 |                  0.7102 |                  0.7136 |                  0.7123 |                  0.1671 |                 0.0084 |                  0.1545 |                 0.0086 |                  0.1435 |                 0.0071 |                  0.1359 |                 0.0091 |                  0.1280 |                 0.0071 |                  0.3590 |                 0.0864 |                  0.3386 |                 0.0851 |                  0.3261 |                 0.0742 |                  0.3181 |                 0.0880 |                  0.3042 |                 0.0709\n",
      "Epoch 00000: val_loss improved from inf to 1.44030, saving model to ./model/model_weights_val.h5\n",
      "\n",
      " split |   loss |    acc |    mse |    mae | cpg_stats/diff_loss | win_stats/1001/mean_loss | win_stats/1001/var_loss | win_stats/2001/mean_loss | win_stats/2001/var_loss | win_stats/3001/mean_loss | win_stats/3001/var_loss | win_stats/4001/mean_loss | win_stats/4001/var_loss | win_stats/5001/mean_loss | win_stats/5001/var_loss | cpg_stats/diff_acc | win_stats/1001/mean_acc | win_stats/2001/mean_acc | win_stats/3001/mean_acc | win_stats/4001/mean_acc | win_stats/5001/mean_acc | win_stats/1001/mean_mse | win_stats/1001/var_mse | win_stats/2001/mean_mse | win_stats/2001/var_mse | win_stats/3001/mean_mse | win_stats/3001/var_mse | win_stats/4001/mean_mse | win_stats/4001/var_mse | win_stats/5001/mean_mse | win_stats/5001/var_mse | win_stats/1001/mean_mae | win_stats/1001/var_mae | win_stats/2001/mean_mae | win_stats/2001/var_mae | win_stats/3001/mean_mae | win_stats/3001/var_mae | win_stats/4001/mean_mae | win_stats/4001/var_mae | win_stats/5001/mean_mae | win_stats/5001/var_mae\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      " train | 2.4949 | 0.7110 | 0.0769 | 0.2051 |             -0.0000 |                   0.1671 |                  0.0084 |                   0.1545 |                  0.0086 |                   0.1435 |                  0.0071 |                   0.1359 |                  0.0091 |                   0.1280 |                  0.0071 |                nan |                  0.7088 |                  0.7102 |                  0.7102 |                  0.7136 |                  0.7123 |                  0.1671 |                 0.0084 |                  0.1545 |                 0.0086 |                  0.1435 |                 0.0071 |                  0.1359 |                 0.0091 |                  0.1280 |                 0.0071 |                  0.3590 |                 0.0864 |                  0.3386 |                 0.0851 |                  0.3261 |                 0.0742 |                  0.3181 |                 0.0880 |                  0.3042 |                 0.0709\n",
      "   val | 1.4403 | 0.7171 | 0.0643 | 0.1832 |              0.0006 |                   0.1455 |                  0.0051 |                   0.1305 |                  0.0059 |                   0.1201 |                  0.0060 |                   0.1120 |                  0.0065 |                   0.1051 |                  0.0061 |                nan |                  0.7106 |                  0.7121 |                  0.7169 |                  0.7201 |                  0.7259 |                  0.1455 |                 0.0051 |                  0.1305 |                 0.0059 |                  0.1201 |                 0.0060 |                  0.1120 |                 0.0065 |                  0.1051 |                 0.0061 |                  0.3304 |                 0.0621 |                  0.3158 |                 0.0649 |                  0.3030 |                 0.0624 |                  0.2851 |                 0.0693 |                  0.2777 |                 0.0608\n",
      "====================================================================================================\n",
      "\n",
      "Training set performance:\n",
      "  loss |    acc |    mse |    mae | cpg_stats/diff_loss | win_stats/1001/mean_loss | win_stats/1001/var_loss | win_stats/2001/mean_loss | win_stats/2001/var_loss | win_stats/3001/mean_loss | win_stats/3001/var_loss | win_stats/4001/mean_loss | win_stats/4001/var_loss | win_stats/5001/mean_loss | win_stats/5001/var_loss | cpg_stats/diff_acc | win_stats/1001/mean_acc | win_stats/2001/mean_acc | win_stats/3001/mean_acc | win_stats/4001/mean_acc | win_stats/5001/mean_acc | win_stats/1001/mean_mse | win_stats/1001/var_mse | win_stats/2001/mean_mse | win_stats/2001/var_mse | win_stats/3001/mean_mse | win_stats/3001/var_mse | win_stats/4001/mean_mse | win_stats/4001/var_mse | win_stats/5001/mean_mse | win_stats/5001/var_mse | win_stats/1001/mean_mae | win_stats/1001/var_mae | win_stats/2001/mean_mae | win_stats/2001/var_mae | win_stats/3001/mean_mae | win_stats/3001/var_mae | win_stats/4001/mean_mae | win_stats/4001/var_mae | win_stats/5001/mean_mae | win_stats/5001/var_mae\n",
      "---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "2.4949 | 0.7110 | 0.0769 | 0.2051 |             -0.0000 |                   0.1671 |                  0.0084 |                   0.1545 |                  0.0086 |                   0.1435 |                  0.0071 |                   0.1359 |                  0.0091 |                   0.1280 |                  0.0071 |                nan |                  0.7088 |                  0.7102 |                  0.7102 |                  0.7136 |                  0.7123 |                  0.1671 |                 0.0084 |                  0.1545 |                 0.0086 |                  0.1435 |                 0.0071 |                  0.1359 |                 0.0091 |                  0.1280 |                 0.0071 |                  0.3590 |                 0.0864 |                  0.3386 |                 0.0851 |                  0.3261 |                 0.0742 |                  0.3181 |                 0.0880 |                  0.3042 |                 0.0709\n",
      "\n",
      "Validation set performance:\n",
      "  loss |    acc |    mse |    mae | cpg_stats/diff_loss | win_stats/1001/mean_loss | win_stats/1001/var_loss | win_stats/2001/mean_loss | win_stats/2001/var_loss | win_stats/3001/mean_loss | win_stats/3001/var_loss | win_stats/4001/mean_loss | win_stats/4001/var_loss | win_stats/5001/mean_loss | win_stats/5001/var_loss | cpg_stats/diff_acc | win_stats/1001/mean_acc | win_stats/2001/mean_acc | win_stats/3001/mean_acc | win_stats/4001/mean_acc | win_stats/5001/mean_acc | win_stats/1001/mean_mse | win_stats/1001/var_mse | win_stats/2001/mean_mse | win_stats/2001/var_mse | win_stats/3001/mean_mse | win_stats/3001/var_mse | win_stats/4001/mean_mse | win_stats/4001/var_mse | win_stats/5001/mean_mse | win_stats/5001/var_mse | win_stats/1001/mean_mae | win_stats/1001/var_mae | win_stats/2001/mean_mae | win_stats/2001/var_mae | win_stats/3001/mean_mae | win_stats/3001/var_mae | win_stats/4001/mean_mae | win_stats/4001/var_mae | win_stats/5001/mean_mae | win_stats/5001/var_mae\n",
      "---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "1.4403 | 0.7171 | 0.0643 | 0.1832 |              0.0006 |                   0.1455 |                  0.0051 |                   0.1305 |                  0.0059 |                   0.1201 |                  0.0060 |                   0.1120 |                  0.0065 |                   0.1051 |                  0.0061 |                nan |                  0.7106 |                  0.7121 |                  0.7169 |                  0.7201 |                  0.7259 |                  0.1455 |                 0.0051 |                  0.1305 |                 0.0059 |                  0.1201 |                 0.0060 |                  0.1120 |                 0.0065 |                  0.1051 |                 0.0061 |                  0.3304 |                 0.0621 |                  0.3158 |                 0.0649 |                  0.3030 |                 0.0624 |                  0.2851 |                 0.0693 |                  0.2777 |                 0.0608\n",
      "INFO (2017-04-11 10:15:33,542): Done!\n"
     ]
    }
   ],
   "source": [
    "model_dir=\"./model\"\n",
    "\n",
    "cmd=\"dcpg_train.py\n",
    "    $train_files\n",
    "    --val_files $val_files\n",
    "    --dna_model CnnL2h128\n",
    "    --out_dir $model_dir\n",
    "    --output_names 'cpg_stats/.*' 'win_stats/.*'\n",
    "    \"\n",
    "if [[ $test_mode -eq 1 ]]; then\n",
    "    cmd=\"$cmd\n",
    "        --nb_epoch 1\n",
    "        --nb_train_sample 10000\n",
    "        --nb_val_sample 10000\n",
    "    \"\n",
    "else\n",
    "    cmd=\"$cmd\n",
    "        --nb_epoch 30\n",
    "        \"\n",
    "fi\n",
    "run $cmd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Model evaluation "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Finally, we use `dcpg_eval.py` for predicting statistics and evaluating predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "#################################\n",
      "dcpg_eval.py ./data/c13_000000-010000.h5 ./data/c1_000000-010000.h5 --model_files ./model --out_data ./eval/data.h5 --out_report ./eval/report.csv\n",
      "#################################\n",
      "Using TensorFlow backend.\n",
      "INFO (2017-04-11 10:15:37,313): Loading model ...\n",
      "INFO (2017-04-11 10:15:37,888): Loading data ...\n",
      "INFO (2017-04-11 10:15:37,892): Predicting ...\n",
      "INFO (2017-04-11 10:15:37,921):   128/20000 (0.6%)\n",
      "INFO (2017-04-11 10:15:43,866):  2176/20000 (10.9%)\n",
      "INFO (2017-04-11 10:15:49,779):  4224/20000 (21.1%)\n",
      "INFO (2017-04-11 10:15:55,775):  6272/20000 (31.4%)\n",
      "INFO (2017-04-11 10:16:01,727):  8320/20000 (41.6%)\n",
      "INFO (2017-04-11 10:16:07,841): 10384/20000 (51.9%)\n",
      "INFO (2017-04-11 10:16:14,312): 12432/20000 (62.2%)\n",
      "INFO (2017-04-11 10:16:20,352): 14480/20000 (72.4%)\n",
      "INFO (2017-04-11 10:16:26,508): 16528/20000 (82.6%)\n",
      "INFO (2017-04-11 10:16:32,556): 18576/20000 (92.9%)\n",
      "INFO (2017-04-11 10:16:37,182): 20000/20000 (100.0%)\n",
      "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/metrics/classification.py:1113: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/metrics/classification.py:516: RuntimeWarning: invalid value encountered in double_scalars\n",
      "  mcc = cov_ytyp / np.sqrt(var_yt * var_yp)\n",
      "                 output       auc       acc  tpr  tnr        f1  mcc       mse       mad       cor   kendall        n\n",
      "1   win_stats/1001/mean  0.756945  0.711100  1.0  0.0  0.831161  0.0  0.148198  0.334125  0.563734  0.349139  20000.0\n",
      "3   win_stats/2001/mean  0.744821  0.711150  1.0  0.0  0.831195  0.0  0.134849  0.322004  0.570356  0.345430  20000.0\n",
      "5   win_stats/3001/mean  0.735982  0.713600  1.0  0.0  0.832866  0.0  0.124574  0.309773  0.572156  0.339815  20000.0\n",
      "7   win_stats/4001/mean  0.731676  0.717100  1.0  0.0  0.835245  0.0  0.116372  0.291607  0.573338  0.337945  20000.0\n",
      "9   win_stats/5001/mean  0.728083  0.722550  1.0  0.0  0.838931  0.0  0.109144  0.283824  0.572712  0.333378  20000.0\n",
      "0        cpg_stats/diff  0.409091  0.846154  0.0  1.0  0.000000  0.0       NaN       NaN       NaN       NaN     13.0\n",
      "2    win_stats/1001/var       NaN       NaN  NaN  NaN       NaN  NaN  0.005030  0.061953  0.050728  0.157179  20000.0\n",
      "4    win_stats/2001/var       NaN       NaN  NaN  NaN       NaN  NaN  0.005725  0.064554  0.047284  0.195577  20000.0\n",
      "6    win_stats/3001/var       NaN       NaN  NaN  NaN       NaN  NaN  0.005922  0.062232  0.032245  0.195001  20000.0\n",
      "8    win_stats/4001/var       NaN       NaN  NaN  NaN       NaN  NaN  0.006529  0.069937  0.038506  0.195562  20000.0\n",
      "10   win_stats/5001/var       NaN       NaN  NaN  NaN       NaN  NaN  0.006164  0.061418  0.033803  0.189928  20000.0\n",
      "INFO (2017-04-11 10:16:37,580): Done!\n"
     ]
    }
   ],
   "source": [
    "eval_dir=\"./eval\"\n",
    "mkdir -p $eval_dir\n",
    "\n",
    "cmd=\"dcpg_eval.py\n",
    "    $data_dir/c*.h5\n",
    "    --model_files $model_dir\n",
    "    --out_data $eval_dir/data.h5\n",
    "    --out_report $eval_dir/report.csv\n",
    "    \"\n",
    "run $cmd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "metric\toutput\tvalue\n",
      "acc\tcpg_stats/diff\t0.8461538461538461\n",
      "acc\twin_stats/1001/mean\t0.7111\n",
      "acc\twin_stats/2001/mean\t0.71115\n",
      "acc\twin_stats/3001/mean\t0.7136\n",
      "acc\twin_stats/4001/mean\t0.7171\n",
      "acc\twin_stats/5001/mean\t0.72255\n",
      "auc\tcpg_stats/diff\t0.40909090909090906\n",
      "auc\twin_stats/1001/mean\t0.756944739547381\n",
      "auc\twin_stats/2001/mean\t0.7448212600520717\n",
      "auc\twin_stats/3001/mean\t0.7359817580953135\n",
      "auc\twin_stats/4001/mean\t0.7316764903156783\n",
      "auc\twin_stats/5001/mean\t0.7280825544788481\n",
      "cor\twin_stats/1001/mean\t0.5637344620627303\n",
      "cor\twin_stats/1001/var\t0.05072792865652165\n",
      "cor\twin_stats/2001/mean\t0.5703561966718008\n",
      "cor\twin_stats/2001/var\t0.047283538321920414\n",
      "cor\twin_stats/3001/mean\t0.5721557471438453\n",
      "cor\twin_stats/3001/var\t0.032244822313482496\n",
      "cor\twin_stats/4001/mean\t0.5733384973383953\n",
      "cor\twin_stats/4001/var\t0.03850585916397397\n",
      "cor\twin_stats/5001/mean\t0.5727119458145952\n",
      "cor\twin_stats/5001/var\t0.03380326117980489\n",
      "f1\tcpg_stats/diff\t0.0\n",
      "f1\twin_stats/1001/mean\t0.8311612413067616\n",
      "f1\twin_stats/2001/mean\t0.831195394909856\n",
      "f1\twin_stats/3001/mean\t0.8328664799253035\n",
      "f1\twin_stats/4001/mean\t0.8352454720167725\n",
      "f1\twin_stats/5001/mean\t0.8389306551333778\n",
      "kendall\twin_stats/1001/mean\t0.349138862941582\n",
      "kendall\twin_stats/1001/var\t0.15717908755052484\n",
      "kendall\twin_stats/2001/mean\t0.3454297291357703\n",
      "kendall\twin_stats/2001/var\t0.1955773076945715\n",
      "kendall\twin_stats/3001/mean\t0.33981474815990087\n",
      "kendall\twin_stats/3001/var\t0.19500092727838456\n",
      "kendall\twin_stats/4001/mean\t0.33794530361995456\n",
      "kendall\twin_stats/4001/var\t0.19556158224596462\n",
      "kendall\twin_stats/5001/mean\t0.3333784826810524\n",
      "kendall\twin_stats/5001/var\t0.18992752930247883\n",
      "mad\twin_stats/1001/mean\t0.33412474393844604\n",
      "mad\twin_stats/1001/var\t0.06195313110947609\n",
      "mad\twin_stats/2001/mean\t0.3220037519931793\n",
      "mad\twin_stats/2001/var\t0.0645541101694107\n",
      "mad\twin_stats/3001/mean\t0.30977344512939453\n",
      "mad\twin_stats/3001/var\t0.06223225221037865\n",
      "mad\twin_stats/4001/mean\t0.2916070818901062\n",
      "mad\twin_stats/4001/var\t0.06993748992681503\n",
      "mad\twin_stats/5001/mean\t0.28382399678230286\n",
      "mad\twin_stats/5001/var\t0.06141769513487816\n",
      "mcc\tcpg_stats/diff\t0.0\n",
      "mcc\twin_stats/1001/mean\t0.0\n",
      "mcc\twin_stats/2001/mean\t0.0\n",
      "mcc\twin_stats/3001/mean\t0.0\n",
      "mcc\twin_stats/4001/mean\t0.0\n",
      "mcc\twin_stats/5001/mean\t0.0\n",
      "mse\twin_stats/1001/mean\t0.14819768071174622\n",
      "mse\twin_stats/1001/var\t0.005029818043112755\n",
      "mse\twin_stats/2001/mean\t0.13484877347946167\n",
      "mse\twin_stats/2001/var\t0.005724579095840454\n",
      "mse\twin_stats/3001/mean\t0.12457415461540222\n",
      "mse\twin_stats/3001/var\t0.005922282114624977\n",
      "mse\twin_stats/4001/mean\t0.11637239903211594\n",
      "mse\twin_stats/4001/var\t0.006528749130666256\n",
      "mse\twin_stats/5001/mean\t0.10914421081542969\n",
      "mse\twin_stats/5001/var\t0.0061644334346055984\n",
      "n\tcpg_stats/diff\t13.0\n",
      "n\twin_stats/1001/mean\t20000.0\n",
      "n\twin_stats/1001/var\t20000.0\n",
      "n\twin_stats/2001/mean\t20000.0\n",
      "n\twin_stats/2001/var\t20000.0\n",
      "n\twin_stats/3001/mean\t20000.0\n",
      "n\twin_stats/3001/var\t20000.0\n",
      "n\twin_stats/4001/mean\t20000.0\n",
      "n\twin_stats/4001/var\t20000.0\n",
      "n\twin_stats/5001/mean\t20000.0\n",
      "n\twin_stats/5001/var\t20000.0\n",
      "tnr\tcpg_stats/diff\t1.0\n",
      "tnr\twin_stats/1001/mean\t0.0\n",
      "tnr\twin_stats/2001/mean\t0.0\n",
      "tnr\twin_stats/3001/mean\t0.0\n",
      "tnr\twin_stats/4001/mean\t0.0\n",
      "tnr\twin_stats/5001/mean\t0.0\n",
      "tpr\tcpg_stats/diff\t0.0\n",
      "tpr\twin_stats/1001/mean\t1.0\n",
      "tpr\twin_stats/2001/mean\t1.0\n",
      "tpr\twin_stats/3001/mean\t1.0\n",
      "tpr\twin_stats/4001/mean\t1.0\n",
      "tpr\twin_stats/5001/mean\t1.0\n"
     ]
    }
   ],
   "source": [
    "cat $eval_dir/report.csv "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}