demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building an image classification model using very little data  \n",
    "\n",
    "Based on the tutorial by Francois Chollet @fchollet https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html and the workbook by Guillaume Dominici https://github.com/gggdominici/keras-workshop\n",
    "\n",
    "This tutorial presents several ways to build an image classifier using keras from just a few hundred or thousand pictures from each class you want to be able to recognize.\n",
    "\n",
    "We will go over the following options:  \n",
    "\n",
    "- training a small network from scratch (as a baseline)  \n",
    "- using the bottleneck features of a pre-trained network  \n",
    "- fine-tuning the top layers of a pre-trained network  \n",
    "  \n",
    "This will lead us to cover the following Keras features:   \n",
    "  \n",
    "- fit_generator for training Keras a model using Python data generators  \n",
    "- ImageDataGenerator for real-time data augmentation  \n",
    "- layer freezing and model fine-tuning  \n",
    "- ...and more.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data can be downloaded at:\n",
    "https://www.kaggle.com/c/dogs-vs-cats/data  \n",
    "All you need is the train set  \n",
    "The recommended folder structure is:  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Folder structure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "data/\n",
    "    train/\n",
    "        dogs/ ### 1024 pictures\n",
    "            dog001.jpg\n",
    "            dog002.jpg\n",
    "            ...\n",
    "        cats/ ### 1024 pictures\n",
    "            cat001.jpg\n",
    "            cat002.jpg\n",
    "            ...\n",
    "    validation/\n",
    "        dogs/ ### 416 pictures\n",
    "            dog001.jpg\n",
    "            dog002.jpg\n",
    "            ...\n",
    "        cats/ ### 416 pictures\n",
    "            cat001.jpg\n",
    "            cat002.jpg\n",
    "            ...\n",
    "```\n",
    "Note : for this example we only consider 2x1000 training images and 2x400 testing images among the 2x12500 available."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The github repo includes about 1500 images for this model. The original Kaggle dataset is much larger. The purpose of this demo is to show how you can build models with smaller size datasets. You should be able to improve this model by using more data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data loading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting pillow\n",
      "  Downloading Pillow-3.4.2-cp27-cp27mu-manylinux1_x86_64.whl (5.6MB)\n",
      "\u001b[K    100% |################################| 5.6MB 321kB/s \n",
      "\u001b[?25hInstalling collected packages: pillow\n",
      "Successfully installed pillow-3.4.2\n",
      "\u001b[33mYou are using pip version 8.1.1, however version 9.0.1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n",
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "##This notebook is built around using tensorflow as the backend for keras\n",
    "!pip install pillow\n",
    "!KERAS_BACKEND=tensorflow python -c \"from keras import backend\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Activation, Dropout, Flatten, Dense\n",
    "from keras.preprocessing.image import ImageDataGenerator\n",
    "from keras.layers import Convolution2D, MaxPooling2D, ZeroPadding2D\n",
    "from keras import optimizers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# dimensions of our images.\n",
    "img_width, img_height = 150, 150\n",
    "\n",
    "train_data_dir = 'data/train'\n",
    "validation_data_dir = 'data/validation'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 2048 images belonging to 2 classes.\n",
      "Found 832 images belonging to 2 classes.\n"
     ]
    }
   ],
   "source": [
    "# used to rescale the pixel values from [0, 255] to [0, 1] interval\n",
    "datagen = ImageDataGenerator(rescale=1./255)\n",
    "\n",
    "# automagically retrieve images and their classes for train and validation sets\n",
    "train_generator = datagen.flow_from_directory(\n",
    "        train_data_dir,\n",
    "        target_size=(img_width, img_height),\n",
    "        batch_size=16,\n",
    "        class_mode='binary')\n",
    "\n",
    "validation_generator = datagen.flow_from_directory(\n",
    "        validation_data_dir,\n",
    "        target_size=(img_width, img_height),\n",
    "        batch_size=32,\n",
    "        class_mode='binary')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Small Conv Net"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model architecture definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Convolution2D(32, 3, 3, input_shape=(img_width, img_height,3)))\n",
    "model.add(Activation('relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "\n",
    "model.add(Convolution2D(32, 3, 3))\n",
    "model.add(Activation('relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "\n",
    "model.add(Convolution2D(64, 3, 3))\n",
    "model.add(Activation('relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "\n",
    "model.add(Flatten())\n",
    "model.add(Dense(64))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(1))\n",
    "model.add(Activation('sigmoid'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer='rmsprop',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nb_epoch = 30\n",
    "nb_train_samples = 2048\n",
    "nb_validation_samples = 832"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "2048/2048 [==============================] - 38s - loss: 0.6943 - acc: 0.5269 - val_loss: 0.6779 - val_acc: 0.5769\n",
      "Epoch 2/30\n",
      "2048/2048 [==============================] - 39s - loss: 0.6795 - acc: 0.5674 - val_loss: 0.6578 - val_acc: 0.6154\n",
      "Epoch 3/30\n",
      "2048/2048 [==============================] - 39s - loss: 0.6624 - acc: 0.5981 - val_loss: 0.6664 - val_acc: 0.5216\n",
      "Epoch 4/30\n",
      "2048/2048 [==============================] - 42s - loss: 0.6346 - acc: 0.6543 - val_loss: 0.6188 - val_acc: 0.6430\n",
      "Epoch 5/30\n",
      "2048/2048 [==============================] - 40s - loss: 0.6050 - acc: 0.6724 - val_loss: 0.6060 - val_acc: 0.6671\n",
      "Epoch 6/30\n",
      "2048/2048 [==============================] - 42s - loss: 0.5843 - acc: 0.6875 - val_loss: 0.5939 - val_acc: 0.6779\n",
      "Epoch 7/30\n",
      "2048/2048 [==============================] - 41s - loss: 0.5624 - acc: 0.7178 - val_loss: 0.5784 - val_acc: 0.6851\n",
      "Epoch 8/30\n",
      "2048/2048 [==============================] - 42s - loss: 0.5269 - acc: 0.7456 - val_loss: 0.5734 - val_acc: 0.6911\n",
      "Epoch 9/30\n",
      "2048/2048 [==============================] - 41s - loss: 0.4919 - acc: 0.7646 - val_loss: 0.5773 - val_acc: 0.7067\n",
      "Epoch 10/30\n",
      "2048/2048 [==============================] - 43s - loss: 0.4695 - acc: 0.7773 - val_loss: 0.5626 - val_acc: 0.7200\n",
      "Epoch 11/30\n",
      "2048/2048 [==============================] - 43s - loss: 0.4294 - acc: 0.8047 - val_loss: 0.6142 - val_acc: 0.6983\n",
      "Epoch 12/30\n",
      "2048/2048 [==============================] - 42s - loss: 0.4107 - acc: 0.8198 - val_loss: 0.5890 - val_acc: 0.7163\n",
      "Epoch 13/30\n",
      "2048/2048 [==============================] - 42s - loss: 0.3741 - acc: 0.8384 - val_loss: 0.5833 - val_acc: 0.7007\n",
      "Epoch 14/30\n",
      "2048/2048 [==============================] - 42s - loss: 0.3381 - acc: 0.8618 - val_loss: 0.7326 - val_acc: 0.7055\n",
      "Epoch 15/30\n",
      "2048/2048 [==============================] - 43s - loss: 0.3178 - acc: 0.8721 - val_loss: 0.7492 - val_acc: 0.7151\n",
      "Epoch 16/30\n",
      "2048/2048 [==============================] - 42s - loss: 0.2731 - acc: 0.8862 - val_loss: 0.6983 - val_acc: 0.6947\n",
      "Epoch 17/30\n",
      "2048/2048 [==============================] - 42s - loss: 0.2751 - acc: 0.8945 - val_loss: 0.8021 - val_acc: 0.7067\n",
      "Epoch 18/30\n",
      "2048/2048 [==============================] - 41s - loss: 0.2449 - acc: 0.9038 - val_loss: 0.6808 - val_acc: 0.7103\n",
      "Epoch 19/30\n",
      "2048/2048 [==============================] - 43s - loss: 0.2164 - acc: 0.9170 - val_loss: 0.6530 - val_acc: 0.6959\n",
      "Epoch 20/30\n",
      "2048/2048 [==============================] - 43s - loss: 0.2181 - acc: 0.9219 - val_loss: 0.8731 - val_acc: 0.7115\n",
      "Epoch 21/30\n",
      "2048/2048 [==============================] - 41s - loss: 0.1855 - acc: 0.9248 - val_loss: 0.9053 - val_acc: 0.7296\n",
      "Epoch 22/30\n",
      "2048/2048 [==============================] - 43s - loss: 0.1854 - acc: 0.9351 - val_loss: 1.0213 - val_acc: 0.7127\n",
      "Epoch 23/30\n",
      "2048/2048 [==============================] - 43s - loss: 0.1866 - acc: 0.9419 - val_loss: 1.2990 - val_acc: 0.7067\n",
      "Epoch 24/30\n",
      "2048/2048 [==============================] - 42s - loss: 0.1605 - acc: 0.9414 - val_loss: 1.0048 - val_acc: 0.7019\n",
      "Epoch 25/30\n",
      "2048/2048 [==============================] - 43s - loss: 0.1768 - acc: 0.9448 - val_loss: 1.0695 - val_acc: 0.7103\n",
      "Epoch 26/30\n",
      "2048/2048 [==============================] - 43s - loss: 0.1310 - acc: 0.9526 - val_loss: 1.2172 - val_acc: 0.7091\n",
      "Epoch 27/30\n",
      "2048/2048 [==============================] - 43s - loss: 0.1636 - acc: 0.9419 - val_loss: 1.1059 - val_acc: 0.7019\n",
      "Epoch 28/30\n",
      "2048/2048 [==============================] - 41s - loss: 0.1512 - acc: 0.9517 - val_loss: 1.2849 - val_acc: 0.7103\n",
      "Epoch 29/30\n",
      "2048/2048 [==============================] - 43s - loss: 0.1461 - acc: 0.9517 - val_loss: 1.1104 - val_acc: 0.7007\n",
      "Epoch 30/30\n",
      "2048/2048 [==============================] - 43s - loss: 0.1437 - acc: 0.9463 - val_loss: 1.7152 - val_acc: 0.7188\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f857451b650>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit_generator(\n",
    "        train_generator,\n",
    "        samples_per_epoch=nb_train_samples,\n",
    "        nb_epoch=nb_epoch,\n",
    "        validation_data=validation_generator,\n",
    "        nb_val_samples=nb_validation_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model.save_weights('models/basic_cnn_20_epochs.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#model.load_weights('models_trained/basic_cnn_20_epochs.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If your model successfully runs at one epoch, go back and it for 30 epochs by changing nb_epoch above.  I was able to get to an val_acc of 0.71 at 30 epochs.\n",
    "A copy of a pretrained network is available in the pretrained folder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating on validation set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Computing loss and accuracy :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.90210915643435263, 0.72956730769230771]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate_generator(validation_generator, nb_validation_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evolution of accuracy on training (blue) and validation (green) sets for 1 to 32 epochs :"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Accuracy evolution](pictures/scores_no_dataaugmentation.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**After ~10 epochs the neural network reach ~70% accuracy. We can witness overfitting, no progress is made over validation set in the next epochs**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data augmentation for improving the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By applying random transformation to our train set, we artificially enhance our dataset with new unseen images.  \n",
    "This will hopefully reduce overfitting and allows better generalization capability for our network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example of data augmentation applied to a picture:\n",
    "![Example of data augmentation applied to a picture](pictures/cat_data_augmentation.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 2048 images belonging to 2 classes.\n"
     ]
    }
   ],
   "source": [
    "train_datagen_augmented = ImageDataGenerator(\n",
    "        rescale=1./255,        # normalize pixel values to [0,1]\n",
    "        shear_range=0.2,       # randomly applies shearing transformation\n",
    "        zoom_range=0.2,        # randomly applies shearing transformation\n",
    "        horizontal_flip=True)  # randomly flip the images\n",
    "\n",
    "# same code as before\n",
    "train_generator_augmented = train_datagen_augmented.flow_from_directory(\n",
    "        train_data_dir,\n",
    "        target_size=(img_width, img_height),\n",
    "        batch_size=32,\n",
    "        class_mode='binary')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "2048/2048 [==============================] - 10s - loss: 0.5532 - acc: 0.7285 - val_loss: 0.6922 - val_acc: 0.7127\n",
      "Epoch 2/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.5568 - acc: 0.7275 - val_loss: 0.5770 - val_acc: 0.7115\n",
      "Epoch 3/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.5271 - acc: 0.7451 - val_loss: 0.6073 - val_acc: 0.7440\n",
      "Epoch 4/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.5038 - acc: 0.7642 - val_loss: 0.5640 - val_acc: 0.7512\n",
      "Epoch 5/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.5324 - acc: 0.7510 - val_loss: 0.5619 - val_acc: 0.6887\n",
      "Epoch 6/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.5079 - acc: 0.7671 - val_loss: 0.5678 - val_acc: 0.7127\n",
      "Epoch 7/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.5016 - acc: 0.7695 - val_loss: 0.5821 - val_acc: 0.7452\n",
      "Epoch 8/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.4631 - acc: 0.7847 - val_loss: 0.6128 - val_acc: 0.7476\n",
      "Epoch 9/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.4762 - acc: 0.7778 - val_loss: 0.5489 - val_acc: 0.7416\n",
      "Epoch 10/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.4841 - acc: 0.7769 - val_loss: 0.5276 - val_acc: 0.7584\n",
      "Epoch 11/30\n",
      "2048/2048 [==============================] - 10s - loss: 0.4541 - acc: 0.7974 - val_loss: 0.5752 - val_acc: 0.7512\n",
      "Epoch 12/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.4770 - acc: 0.7886 - val_loss: 0.5560 - val_acc: 0.7716\n",
      "Epoch 13/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.4706 - acc: 0.7935 - val_loss: 0.5922 - val_acc: 0.7188\n",
      "Epoch 14/30\n",
      "2048/2048 [==============================] - 10s - loss: 0.4693 - acc: 0.7847 - val_loss: 0.5398 - val_acc: 0.7500\n",
      "Epoch 15/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.4419 - acc: 0.7988 - val_loss: 0.5946 - val_acc: 0.7692\n",
      "Epoch 16/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.4532 - acc: 0.8003 - val_loss: 0.5396 - val_acc: 0.7656\n",
      "Epoch 17/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.4529 - acc: 0.7983 - val_loss: 0.5153 - val_acc: 0.7656\n",
      "Epoch 18/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.4253 - acc: 0.8135 - val_loss: 0.8682 - val_acc: 0.7404\n",
      "Epoch 19/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.4090 - acc: 0.8169 - val_loss: 0.6031 - val_acc: 0.7127\n",
      "Epoch 20/30\n",
      "2048/2048 [==============================] - 10s - loss: 0.4442 - acc: 0.8110 - val_loss: 0.5669 - val_acc: 0.7740\n",
      "Epoch 21/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.4253 - acc: 0.8169 - val_loss: 0.5418 - val_acc: 0.7764\n",
      "Epoch 22/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.4352 - acc: 0.8076 - val_loss: 0.5494 - val_acc: 0.7692\n",
      "Epoch 23/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.4277 - acc: 0.8022 - val_loss: 0.5858 - val_acc: 0.7776\n",
      "Epoch 24/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.4232 - acc: 0.8115 - val_loss: 0.6827 - val_acc: 0.7224\n",
      "Epoch 25/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.4312 - acc: 0.8081 - val_loss: 0.5408 - val_acc: 0.7764\n",
      "Epoch 26/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.4178 - acc: 0.8130 - val_loss: 0.5425 - val_acc: 0.7632\n",
      "Epoch 27/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.3949 - acc: 0.8262 - val_loss: 0.6941 - val_acc: 0.7668\n",
      "Epoch 28/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.4196 - acc: 0.8149 - val_loss: 0.8530 - val_acc: 0.7284\n",
      "Epoch 29/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.4228 - acc: 0.8232 - val_loss: 0.5460 - val_acc: 0.7752\n",
      "Epoch 30/30\n",
      "2048/2048 [==============================] - 9s - loss: 0.3989 - acc: 0.8242 - val_loss: 0.6910 - val_acc: 0.7680\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fb0bc268390>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit_generator(\n",
    "        train_generator_augmented,\n",
    "        samples_per_epoch=nb_train_samples,\n",
    "        nb_epoch=nb_epoch,\n",
    "        validation_data=validation_generator,\n",
    "        nb_val_samples=nb_validation_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model.save_weights('models/augmented_30_epochs.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#model.load_weights('models_trained/augmented_30_epochs.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating on validation set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Computing loss and accuracy :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.69096329235113585, 0.76802884615384615]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate_generator(validation_generator, nb_validation_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evolution of accuracy on training (blue) and validation (green) sets for 1 to 100 epochs :"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Accuracy evolution](pictures/scores_with_dataaugmentation.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Thanks to data-augmentation, the accuracy on the validation set improved to ~80%**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using a pre-trained model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The process of training a convolutionnal neural network can be very time-consuming and require a lot of datas.  \n",
    "\n",
    "We can go beyond the previous models in terms of performance and efficiency by using a general-purpose, pre-trained image classifier.  This example uses VGG16, a model trained on the ImageNet dataset - which contains millions of images classified in 1000 categories. \n",
    "\n",
    "On top of it, we add a small multi-layer perceptron and we train it on our dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### VGG16 + small MLP\n",
    "![VGG16 + Dense layers Schema](pictures/vgg16_original.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### VGG16 model architecture definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model_vgg = Sequential()\n",
    "model_vgg.add(ZeroPadding2D((1, 1), input_shape=(img_width, img_height,3)))\n",
    "model_vgg.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_1'))\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_2'))\n",
    "model_vgg.add(MaxPooling2D((2, 2), strides=(2, 2)))\n",
    "\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_1'))\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_2'))\n",
    "model_vgg.add(MaxPooling2D((2, 2), strides=(2, 2)))\n",
    "\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_1'))\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_2'))\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_3'))\n",
    "model_vgg.add(MaxPooling2D((2, 2), strides=(2, 2)))\n",
    "\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_1'))\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_2'))\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_3'))\n",
    "model_vgg.add(MaxPooling2D((2, 2), strides=(2, 2)))\n",
    "\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_1'))\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_2'))\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_3'))\n",
    "model_vgg.add(MaxPooling2D((2, 2), strides=(2, 2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loading VGG16 weights\n",
    "This part is a bit complicated because the structure of our model is not exactly the same as the one used when training weights.  \n",
    "Otherwise, we would use the `model.load_weights()` method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note : the VGG16 weights file (~500MB) is not included in this repository. You can download from here :  \n",
    "https://gist.github.com/baraldilorenzo/07d7802847aaad0a35d3*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import h5py\n",
    "f = h5py.File('models/vgg/vgg16_weights.h5')\n",
    "for k in range(f.attrs['nb_layers']):\n",
    "    if k >= len(model_vgg.layers) - 1:\n",
    "        # we don't look at the last two layers in the savefile (fully-connected and activation)\n",
    "        break\n",
    "    g = f['layer_{}'.format(k)]\n",
    "    weights = [g['param_{}'.format(p)] for p in range(g.attrs['nb_params'])]\n",
    "    layer = model_vgg.layers[k]\n",
    "\n",
    "    if layer.__class__.__name__ in ['Convolution1D', 'Convolution2D', 'Convolution3D', 'AtrousConvolution2D']:\n",
    "        weights[0] = np.transpose(weights[0], (2, 3, 1, 0))\n",
    "\n",
    "    layer.set_weights(weights)\n",
    "\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the VGG16 model to process samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 2048 images belonging to 2 classes.\n",
      "Found 832 images belonging to 2 classes.\n"
     ]
    }
   ],
   "source": [
    "train_generator_bottleneck = datagen.flow_from_directory(\n",
    "        train_data_dir,\n",
    "        target_size=(img_width, img_height),\n",
    "        batch_size=32,\n",
    "        class_mode=None,\n",
    "        shuffle=False)\n",
    "\n",
    "validation_generator_bottleneck = datagen.flow_from_directory(\n",
    "        validation_data_dir,\n",
    "        target_size=(img_width, img_height),\n",
    "        batch_size=32,\n",
    "        class_mode=None,\n",
    "        shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a long process, so we save the output of the VGG16 once and for all.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "bottleneck_features_train = model_vgg.predict_generator(train_generator_bottleneck, nb_train_samples)\n",
    "np.save(open('models/bottleneck_features_train.npy', 'wb'), bottleneck_features_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "bottleneck_features_validation = model_vgg.predict_generator(validation_generator_bottleneck, nb_validation_samples)\n",
    "np.save(open('models/bottleneck_features_validation.npy', 'wb'), bottleneck_features_validation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can load it..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_data = np.load(open('models/bottleneck_features_train.npy', 'rb'))\n",
    "train_labels = np.array([0] * (nb_train_samples // 2) + [1] * (nb_train_samples // 2))\n",
    "\n",
    "validation_data = np.load(open('models/bottleneck_features_validation.npy', 'rb'))\n",
    "validation_labels = np.array([0] * (nb_validation_samples // 2) + [1] * (nb_validation_samples // 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And define and train the custom fully connected neural network :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model_top = Sequential()\n",
    "model_top.add(Flatten(input_shape=train_data.shape[1:]))\n",
    "model_top.add(Dense(256, activation='relu'))\n",
    "model_top.add(Dropout(0.5))\n",
    "model_top.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "model_top.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 2048 samples, validate on 832 samples\n",
      "Epoch 1/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.9784 - acc: 0.6855 - val_loss: 0.4140 - val_acc: 0.8077\n",
      "Epoch 2/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.4831 - acc: 0.7891 - val_loss: 0.3646 - val_acc: 0.8365\n",
      "Epoch 3/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.3974 - acc: 0.8271 - val_loss: 0.3369 - val_acc: 0.8558\n",
      "Epoch 4/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.3438 - acc: 0.8525 - val_loss: 0.3136 - val_acc: 0.8594\n",
      "Epoch 5/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.3131 - acc: 0.8647 - val_loss: 0.5777 - val_acc: 0.7873\n",
      "Epoch 6/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.2883 - acc: 0.8823 - val_loss: 0.2913 - val_acc: 0.8786\n",
      "Epoch 7/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.2321 - acc: 0.9048 - val_loss: 0.3724 - val_acc: 0.8462\n",
      "Epoch 8/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.2057 - acc: 0.9092 - val_loss: 0.4276 - val_acc: 0.8438\n",
      "Epoch 9/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.2000 - acc: 0.9141 - val_loss: 0.3741 - val_acc: 0.8618\n",
      "Epoch 10/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.1811 - acc: 0.9248 - val_loss: 0.3447 - val_acc: 0.8726\n",
      "Epoch 11/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.1540 - acc: 0.9395 - val_loss: 0.3352 - val_acc: 0.8714\n",
      "Epoch 12/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.1511 - acc: 0.9414 - val_loss: 0.3706 - val_acc: 0.8738\n",
      "Epoch 13/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.1362 - acc: 0.9399 - val_loss: 0.3913 - val_acc: 0.8798\n",
      "Epoch 14/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.1134 - acc: 0.9497 - val_loss: 0.3854 - val_acc: 0.8726\n",
      "Epoch 15/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.1117 - acc: 0.9609 - val_loss: 0.5750 - val_acc: 0.8534\n",
      "Epoch 16/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0998 - acc: 0.9565 - val_loss: 0.4339 - val_acc: 0.8870\n",
      "Epoch 17/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0678 - acc: 0.9736 - val_loss: 0.4881 - val_acc: 0.8762\n",
      "Epoch 18/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0852 - acc: 0.9658 - val_loss: 0.4335 - val_acc: 0.8870\n",
      "Epoch 19/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0733 - acc: 0.9692 - val_loss: 0.4505 - val_acc: 0.8750\n",
      "Epoch 20/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0616 - acc: 0.9756 - val_loss: 0.5825 - val_acc: 0.8606\n",
      "Epoch 21/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0585 - acc: 0.9771 - val_loss: 0.6856 - val_acc: 0.8510\n",
      "Epoch 22/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0494 - acc: 0.9800 - val_loss: 0.5624 - val_acc: 0.8750\n",
      "Epoch 23/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0464 - acc: 0.9839 - val_loss: 0.8560 - val_acc: 0.8401\n",
      "Epoch 24/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0626 - acc: 0.9727 - val_loss: 0.5666 - val_acc: 0.8738\n",
      "Epoch 25/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0285 - acc: 0.9912 - val_loss: 0.5543 - val_acc: 0.8786\n",
      "Epoch 26/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0463 - acc: 0.9805 - val_loss: 0.6137 - val_acc: 0.8738\n",
      "Epoch 27/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0260 - acc: 0.9917 - val_loss: 0.6608 - val_acc: 0.8882\n",
      "Epoch 28/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0414 - acc: 0.9824 - val_loss: 0.6423 - val_acc: 0.8702\n",
      "Epoch 29/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0308 - acc: 0.9878 - val_loss: 0.6941 - val_acc: 0.8726\n",
      "Epoch 30/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0273 - acc: 0.9897 - val_loss: 0.6677 - val_acc: 0.8678\n",
      "Epoch 31/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0315 - acc: 0.9927 - val_loss: 0.6830 - val_acc: 0.8738\n",
      "Epoch 32/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0258 - acc: 0.9922 - val_loss: 0.7593 - val_acc: 0.8798\n",
      "Epoch 33/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0266 - acc: 0.9922 - val_loss: 0.8203 - val_acc: 0.8546\n",
      "Epoch 34/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0198 - acc: 0.9941 - val_loss: 0.9712 - val_acc: 0.8450\n",
      "Epoch 35/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0291 - acc: 0.9902 - val_loss: 0.8274 - val_acc: 0.8594\n",
      "Epoch 36/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0168 - acc: 0.9941 - val_loss: 0.8353 - val_acc: 0.8738\n",
      "Epoch 37/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0324 - acc: 0.9912 - val_loss: 0.7481 - val_acc: 0.8702\n",
      "Epoch 38/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0191 - acc: 0.9932 - val_loss: 0.8011 - val_acc: 0.8750\n",
      "Epoch 39/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0254 - acc: 0.9897 - val_loss: 0.7558 - val_acc: 0.8702\n",
      "Epoch 40/40\n",
      "2048/2048 [==============================] - 0s - loss: 0.0267 - acc: 0.9897 - val_loss: 0.7828 - val_acc: 0.8798\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fb0b60feb90>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb_epoch=40\n",
    "model_top.fit(train_data, train_labels,\n",
    "          nb_epoch=nb_epoch, batch_size=32,\n",
    "          validation_data=(validation_data, validation_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training process of this small neural network is very fast : ~2s per epoch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model_top.save_weights('models/bottleneck_40_epochs.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bottleneck model evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#model_top.load_weights('models/with-bottleneck/1000-samples--100-epochs.h5')\n",
    "#model_top.load_weights('/notebook/Data1/Code/keras-workshop/models/with-bottleneck/1000-samples--100-epochs.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loss and accuracy :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "704/832 [========================>.....] - ETA: 0s"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.782802758165277, 0.87980769230769229]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_top.evaluate(validation_data, validation_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evolution of accuracy on training (blue) and validation (green) sets for 1 to 32 epochs :"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Accuracy evolution](pictures/scores_with_bottleneck.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**We reached a 90% accuracy on the validation after ~1m of training (~20 epochs) and 8% of the samples originally available on the Kaggle competition !**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "##Fine-tuning the top layers of a a pre-trained network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Start by instantiating the VGG base and loading its weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_vgg = Sequential()\n",
    "model_vgg.add(ZeroPadding2D((1, 1), input_shape=(img_width, img_height,3)))\n",
    "model_vgg.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_1'))\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_2'))\n",
    "model_vgg.add(MaxPooling2D((2, 2), strides=(2, 2)))\n",
    "\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_1'))\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_2'))\n",
    "model_vgg.add(MaxPooling2D((2, 2), strides=(2, 2)))\n",
    "\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_1'))\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_2'))\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_3'))\n",
    "model_vgg.add(MaxPooling2D((2, 2), strides=(2, 2)))\n",
    "\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_1'))\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_2'))\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_3'))\n",
    "model_vgg.add(MaxPooling2D((2, 2), strides=(2, 2)))\n",
    "\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_1'))\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_2'))\n",
    "model_vgg.add(ZeroPadding2D((1, 1)))\n",
    "model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_3'))\n",
    "model_vgg.add(MaxPooling2D((2, 2), strides=(2, 2)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import h5py\n",
    "f = h5py.File('models/vgg/vgg16_weights.h5')\n",
    "for k in range(f.attrs['nb_layers']):\n",
    "    if k >= len(model_vgg.layers) - 1:\n",
    "        # we don't look at the last two layers in the savefile (fully-connected and activation)\n",
    "        break\n",
    "    g = f['layer_{}'.format(k)]\n",
    "    weights = [g['param_{}'.format(p)] for p in range(g.attrs['nb_params'])]\n",
    "    layer = model_vgg.layers[k]\n",
    "\n",
    "    if layer.__class__.__name__ in ['Convolution1D', 'Convolution2D', 'Convolution3D', 'AtrousConvolution2D']:\n",
    "        weights[0] = np.transpose(weights[0], (2, 3, 1, 0))\n",
    "\n",
    "    layer.set_weights(weights)\n",
    "\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build a classifier model to put on top of the convolutional model. For the fine tuning, we start with a fully trained-classifer. We will use the weights from the earlier model. And then we will add this model on top of the convolutional base."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "top_model = Sequential()\n",
    "top_model.add(Flatten(input_shape=model_vgg.output_shape[1:]))\n",
    "top_model.add(Dense(256, activation='relu'))\n",
    "top_model.add(Dropout(0.5))\n",
    "top_model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "top_model.load_weights('models/bottleneck_40_epochs.h5')\n",
    "\n",
    "model_vgg.add(top_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For fine turning, we only want to train a few layers.  This line will set the first 25 layers (up to the conv block) to non-trainable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for layer in model_vgg.layers[:25]:\n",
    "    layer.trainable = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# compile the model with a SGD/momentum optimizer\n",
    "# and a very slow learning rate.\n",
    "model_vgg.compile(loss='binary_crossentropy',\n",
    "              optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 2048 images belonging to 2 classes.\n",
      "Found 832 images belonging to 2 classes.\n"
     ]
    }
   ],
   "source": [
    "# prepare data augmentation configuration  . . . do we need this?\n",
    "train_datagen = ImageDataGenerator(\n",
    "        rescale=1./255,\n",
    "        shear_range=0.2,\n",
    "        zoom_range=0.2,\n",
    "        horizontal_flip=True)\n",
    "\n",
    "test_datagen = ImageDataGenerator(rescale=1./255)\n",
    "\n",
    "train_generator = train_datagen.flow_from_directory(\n",
    "        train_data_dir,\n",
    "        target_size=(img_height, img_width),\n",
    "        batch_size=32,\n",
    "        class_mode='binary')\n",
    "\n",
    "validation_generator = test_datagen.flow_from_directory(\n",
    "        validation_data_dir,\n",
    "        target_size=(img_height, img_width),\n",
    "        batch_size=32,\n",
    "        class_mode='binary')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/40\n",
      "2048/2048 [==============================] - 39s - loss: 0.3593 - acc: 0.8862 - val_loss: 0.3669 - val_acc: 0.8738\n",
      "Epoch 2/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.2559 - acc: 0.9077 - val_loss: 0.3483 - val_acc: 0.8594\n",
      "Epoch 3/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.1942 - acc: 0.9292 - val_loss: 0.3380 - val_acc: 0.8618\n",
      "Epoch 4/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.1900 - acc: 0.9258 - val_loss: 0.3268 - val_acc: 0.8858\n",
      "Epoch 5/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.1638 - acc: 0.9395 - val_loss: 0.3094 - val_acc: 0.8894\n",
      "Epoch 6/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.1307 - acc: 0.9502 - val_loss: 0.3038 - val_acc: 0.8930\n",
      "Epoch 7/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.1223 - acc: 0.9556 - val_loss: 0.3267 - val_acc: 0.8990\n",
      "Epoch 8/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.1233 - acc: 0.9570 - val_loss: 0.2864 - val_acc: 0.8930\n",
      "Epoch 9/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.1147 - acc: 0.9648 - val_loss: 0.3657 - val_acc: 0.8990\n",
      "Epoch 10/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.1070 - acc: 0.9590 - val_loss: 0.3192 - val_acc: 0.8990\n",
      "Epoch 11/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0825 - acc: 0.9683 - val_loss: 0.4118 - val_acc: 0.8990\n",
      "Epoch 12/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0953 - acc: 0.9663 - val_loss: 0.3561 - val_acc: 0.9002\n",
      "Epoch 13/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0605 - acc: 0.9756 - val_loss: 0.3643 - val_acc: 0.9026\n",
      "Epoch 14/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0826 - acc: 0.9736 - val_loss: 0.3549 - val_acc: 0.8990\n",
      "Epoch 15/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0758 - acc: 0.9707 - val_loss: 0.3911 - val_acc: 0.9014\n",
      "Epoch 16/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0547 - acc: 0.9756 - val_loss: 0.4181 - val_acc: 0.9002\n",
      "Epoch 17/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0573 - acc: 0.9795 - val_loss: 0.3912 - val_acc: 0.9038\n",
      "Epoch 18/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0499 - acc: 0.9810 - val_loss: 0.4097 - val_acc: 0.9075\n",
      "Epoch 19/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0504 - acc: 0.9790 - val_loss: 0.4063 - val_acc: 0.9159\n",
      "Epoch 20/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0398 - acc: 0.9858 - val_loss: 0.4282 - val_acc: 0.9099\n",
      "Epoch 21/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0477 - acc: 0.9824 - val_loss: 0.4364 - val_acc: 0.8930\n",
      "Epoch 22/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0401 - acc: 0.9873 - val_loss: 0.4338 - val_acc: 0.9038\n",
      "Epoch 23/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0357 - acc: 0.9897 - val_loss: 0.4830 - val_acc: 0.9026\n",
      "Epoch 24/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0397 - acc: 0.9878 - val_loss: 0.4229 - val_acc: 0.9111\n",
      "Epoch 25/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0353 - acc: 0.9863 - val_loss: 0.4715 - val_acc: 0.9171\n",
      "Epoch 26/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0308 - acc: 0.9888 - val_loss: 0.4751 - val_acc: 0.9159\n",
      "Epoch 27/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0432 - acc: 0.9834 - val_loss: 0.4557 - val_acc: 0.9099\n",
      "Epoch 28/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0385 - acc: 0.9868 - val_loss: 0.4859 - val_acc: 0.9038\n",
      "Epoch 29/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0270 - acc: 0.9893 - val_loss: 0.4334 - val_acc: 0.9062\n",
      "Epoch 30/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0356 - acc: 0.9897 - val_loss: 0.4468 - val_acc: 0.9038\n",
      "Epoch 31/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0344 - acc: 0.9888 - val_loss: 0.4405 - val_acc: 0.9087\n",
      "Epoch 32/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0273 - acc: 0.9897 - val_loss: 0.5071 - val_acc: 0.9050\n",
      "Epoch 33/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0179 - acc: 0.9937 - val_loss: 0.5163 - val_acc: 0.9111\n",
      "Epoch 34/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0397 - acc: 0.9868 - val_loss: 0.4168 - val_acc: 0.9123\n",
      "Epoch 35/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0240 - acc: 0.9941 - val_loss: 0.4653 - val_acc: 0.9111\n",
      "Epoch 36/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0206 - acc: 0.9932 - val_loss: 0.5174 - val_acc: 0.9062\n",
      "Epoch 37/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0308 - acc: 0.9902 - val_loss: 0.5016 - val_acc: 0.9087\n",
      "Epoch 38/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0159 - acc: 0.9966 - val_loss: 0.4876 - val_acc: 0.9111\n",
      "Epoch 39/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0335 - acc: 0.9883 - val_loss: 0.4280 - val_acc: 0.9050\n",
      "Epoch 40/40\n",
      "2048/2048 [==============================] - 35s - loss: 0.0198 - acc: 0.9922 - val_loss: 0.4693 - val_acc: 0.9038\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fb059495c50>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fine-tune the model\n",
    "model_vgg.fit_generator(\n",
    "        train_generator,\n",
    "        samples_per_epoch=nb_train_samples,\n",
    "        nb_epoch=nb_epoch,\n",
    "        validation_data=validation_generator,\n",
    "        nb_val_samples=nb_validation_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_vgg.save_weights('models/finetuning_20epochs_vgg.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_vgg.load_weights('models/finetuning_20epochs_vgg.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating on validation set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Computing loss and accuracy :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.46926221093879295, 0.90384615384615385]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_vgg.evaluate_generator(validation_generator, nb_validation_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.69096328031558252, 0.76802884615384615]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate_generator(validation_generator, nb_validation_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "704/832 [========================>.....] - ETA: 0s"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.782802758165277, 0.87980769230769229]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_top.evaluate(validation_data, validation_labels)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}