Neural_Nets_Gesture_Recognition_Final_Notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gesture Recognition\n",
    "In this group project, you are going to build a 3D Conv model that will be able to predict the 5 gestures correctly. Please import the following libraries to get started."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import os\n",
    "from scipy.misc import imread, imresize\n",
    "import datetime\n",
    "import os\n",
    "import cv2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We set the random seed so that the results don't vary drastically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(30)\n",
    "import random as rn\n",
    "rn.seed(30)\n",
    "from keras import backend as K\n",
    "import tensorflow as tf\n",
    "tf.set_random_seed(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this block, you read the folder names for training and validation. We will set the initial `batch_size` here. Note that we will experimentally set the batch size in such a way that you are able to use the GPU in full capacity. Keep increasing the batch size until the machine throws an error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_doc = np.random.permutation(open('./storage/Demo-gesture/Project_data/train.csv').readlines())\n",
    "val_doc = np.random.permutation(open('./storage/Demo-gesture/Project_data/val.csv').readlines())\n",
    "batch_size = 10 #experiment with the batch size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Note*** in our implementaiton true **batch_size** is always 2 fold as we agument our data. if if batch size is 10, true batch size at the run time is 20, as twice the number of data set loaded are added to the batch\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generator\n",
    "This is one of the most important part of the code. The overall structure of the generator is broken down into modules. In the generator, we are going to preprocess the images as we have images of 2 different dimensions as well as create a batch of video frames. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parameters initialization\n",
    "nb_rows = 120   # X dimension of the image\n",
    "nb_cols = 120   # Y dimesnion of the image\n",
    "#total_frames = 30\n",
    "nb_frames = 30  # lenght of the video frames\n",
    "nb_channel = 3 # numbe rof channels in images 3 for color(RGB) and 1 for Gray"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Helper function to generate a random affine transform on the iamge\n",
    "def get_random_affine():\n",
    "    dx, dy = np.random.randint(-1.7, 1.8, 2)\n",
    "    M = np.float32([[1, 0, dx], [0, 1, dy]])\n",
    "    return M"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Helper function to normalise the data\n",
    "def normalize_data(data):\n",
    "    return data/127.5-1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Helper function to initialize all the batch image data and labels\n",
    "def init_batch_data(batch_size):\n",
    "    batch_data = np.zeros((batch_size, nb_frames, nb_rows, nb_cols, nb_channel)) \n",
    "    batch_labels = np.zeros((batch_size,5)) # batch_labels is the one hot representation of the output\n",
    "    return batch_data, batch_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_batch_images(source_path, folder_list, batch_num, batch_size, t,validation):\n",
    "    \n",
    "    batch_data,batch_labels = init_batch_data(batch_size)\n",
    "    \n",
    "    # We will also build an augumented batch data with affine transformation\n",
    "    batch_data_aug,batch_labels_aug = init_batch_data(batch_size)\n",
    "    \n",
    "    # We will also build an augmented batch data with horizontal flip\n",
    "    batch_data_flip,batch_labels_flip = init_batch_data(batch_size)\n",
    "    \n",
    "    #create a list of image numbers you want to use for a particular video using full frames\n",
    "    img_idx = [x for x in range(0, nb_frames)] \n",
    "\n",
    "    for folder in range(batch_size): # iterate over the batch_size\n",
    "        # read all the images in the folder\n",
    "        imgs = sorted(os.listdir(source_path+'/'+ t[folder + (batch_num*batch_size)].split(';')[0])) \n",
    "        # Generate a random affine to be used in image transformation for buidling agumented data set\n",
    "        M = get_random_affine()\n",
    "        \n",
    "        #  Iterate over the frames/images of a folder to read them in\n",
    "        for idx, item in enumerate(img_idx): \n",
    "            ## image = imread(source_path+'/'+ t[folder + (batch_num*batch_size)].strip().split(';')[0]+'/'+imgs[item]).astype(np.float32)\n",
    "            image = cv2.imread(source_path+'/'+ t[folder + (batch_num*batch_size)].strip().split(';')[0]+'/'+imgs[item], cv2.IMREAD_COLOR)\n",
    "            image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "            \n",
    "            # Cropping non symmetric frames\n",
    "            if image.shape[0] != image.shape[1]:\n",
    "                image=image[0:120,20:140]\n",
    "            \n",
    "            #crop the images and resize them. Note that the images are of 2 different shape \n",
    "            #and the conv3D will throw error if the inputs in a batch have different shapes   \n",
    "            resized = cv2.resize(image, (nb_rows,nb_cols), interpolation = cv2.INTER_AREA)\n",
    "            #Normal data\n",
    "            batch_data[folder,idx] = (resized)\n",
    "            \n",
    "            #Data with affine transformation\n",
    "            batch_data_aug[folder,idx] = (cv2.warpAffine(resized, M, (resized.shape[0], resized.shape[1])))\n",
    "            \n",
    "            # Data with horizontal flip\n",
    "            batch_data_flip[folder,idx]= np.flip(resized,1)\n",
    "\n",
    "        batch_labels[folder, int(t[folder + (batch_num*batch_size)].strip().split(';')[2])] = 1\n",
    "        batch_labels_aug[folder, int(t[folder + (batch_num*batch_size)].strip().split(';')[2])] = 1\n",
    "        \n",
    "        # Labeling data with horizobtal flip, right swipe becomes left swipe and viceversa\n",
    "        if int(t[folder + (batch_num*batch_size)].strip().split(';')[2])==0:\n",
    "                    batch_labels_flip[folder, 1] = 1\n",
    "        elif int(t[folder + (batch_num*batch_size)].strip().split(';')[2])==1:\n",
    "                    batch_labels_flip[folder, 0] = 1\n",
    "                    \n",
    "        else:\n",
    "                    batch_labels_flip[folder, int(t[folder + (batch_num*batch_size)].strip().split(';')[2])] = 1\n",
    "                  \n",
    "    \n",
    "    batch_data_final = np.append(batch_data, batch_data_aug, axis = 0)\n",
    "    batch_data_final = np.append(batch_data_final, batch_data_flip, axis = 0)\n",
    "\n",
    "    batch_labels_final = np.append(batch_labels, batch_labels_aug, axis = 0) \n",
    "    batch_labels_final = np.append(batch_labels_final, batch_labels_flip, axis = 0)\n",
    "    \n",
    "    if validation:\n",
    "        batch_data_final=batch_data\n",
    "        batch_labels_final= batch_labels\n",
    "        \n",
    "    return batch_data_final,batch_labels_final"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generator(source_path, folder_list, batch_size, validation=False):\n",
    "    print( 'Source path = ', source_path, '; batch size =', batch_size)\n",
    "    while True:\n",
    "        t = np.random.permutation(folder_list)\n",
    "        num_batches = len(folder_list)//batch_size # calculate the number of batches\n",
    "        for batch in range(num_batches): # we iterate over the number of batches\n",
    "            # you yield the batch_data and the batch_labels, remember what does yield do\n",
    "            yield load_batch_images(source_path, folder_list, batch, batch_size, t,validation)\n",
    "            \n",
    "\n",
    "        \n",
    "        # Code for the remaining data points which are left after full batches\n",
    "        if (len(folder_list) != batch_size*num_batches):\n",
    "            batch_size = len(folder_list) - (batch_size*num_batches)\n",
    "            yield load_batch_images(source_path, folder_list, batch, batch_size, t,validation)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note here that a video is represented above in the generator as (number of images, height, width, number of channels). Take this into consideration while creating the model architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# training sequences = 663\n",
      "# validation sequences = 100\n",
      "# epochs = 10\n"
     ]
    }
   ],
   "source": [
    "curr_dt_time = datetime.datetime.now()\n",
    "train_path = './storage/Demo-gesture/Project_data/train'\n",
    "val_path = './storage/Demo-gesture/Project_data/val'\n",
    "num_train_sequences = len(train_doc)\n",
    "print('# training sequences =', num_train_sequences)\n",
    "num_val_sequences = len(val_doc)\n",
    "print('# validation sequences =', num_val_sequences)\n",
    "num_epochs = 10 # choose the number of epochs\n",
    "print ('# epochs =', num_epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Final Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential, Model\n",
    "from keras.layers import Dense, GRU, Flatten, TimeDistributed, Flatten, BatchNormalization, Activation, Dropout\n",
    "from keras.layers.convolutional import Conv3D, MaxPooling3D\n",
    "from keras.callbacks import ModelCheckpoint, ReduceLROnPlateau\n",
    "from keras.losses import categorical_crossentropy\n",
    "from keras.optimizers import Adam\n",
    "\n",
    "nb_filters = [8,16,32,64]\n",
    "nb_dense = [256, 128, 5]\n",
    "\n",
    "# Input\n",
    "input_shape=(nb_frames,nb_rows,nb_cols,nb_channel)\n",
    "\n",
    "# Define model\n",
    "model = Sequential()\n",
    "\n",
    "model.add(Conv3D(nb_filters[0], \n",
    "                 kernel_size=(3,3,3), \n",
    "                 input_shape=input_shape,\n",
    "                 padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model.add(Conv3D(nb_filters[1], \n",
    "                 kernel_size=(3,3,3), \n",
    "                 padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model.add(Conv3D(nb_filters[2], \n",
    "                 kernel_size=(1,3,3), \n",
    "                 padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model.add(Conv3D(nb_filters[3], \n",
    "                 kernel_size=(1,3,3), \n",
    "                 padding='same'))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Dropout(0.25))\n",
    "\n",
    "model.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "#Flatten Layers\n",
    "model.add(Flatten())\n",
    "\n",
    "model.add(Dense(nb_dense[0], activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "\n",
    "model.add(Dense(nb_dense[1], activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "\n",
    "#softmax layer\n",
    "model.add(Dense(nb_dense[2], activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have written the model, the next step is to `compile` the model. When you print the `summary` of the model, you'll see the total number of parameters you have to train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv3d_33 (Conv3D)           (None, 30, 120, 120, 8)   656       \n",
      "_________________________________________________________________\n",
      "batch_normalization_25 (Batc (None, 30, 120, 120, 8)   32        \n",
      "_________________________________________________________________\n",
      "activation_33 (Activation)   (None, 30, 120, 120, 8)   0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_33 (MaxPooling (None, 15, 60, 60, 8)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_34 (Conv3D)           (None, 15, 60, 60, 16)    3472      \n",
      "_________________________________________________________________\n",
      "batch_normalization_26 (Batc (None, 15, 60, 60, 16)    64        \n",
      "_________________________________________________________________\n",
      "activation_34 (Activation)   (None, 15, 60, 60, 16)    0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_34 (MaxPooling (None, 7, 30, 30, 16)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_35 (Conv3D)           (None, 7, 30, 30, 32)     4640      \n",
      "_________________________________________________________________\n",
      "batch_normalization_27 (Batc (None, 7, 30, 30, 32)     128       \n",
      "_________________________________________________________________\n",
      "activation_35 (Activation)   (None, 7, 30, 30, 32)     0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_35 (MaxPooling (None, 3, 15, 15, 32)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_36 (Conv3D)           (None, 3, 15, 15, 64)     18496     \n",
      "_________________________________________________________________\n",
      "activation_36 (Activation)   (None, 3, 15, 15, 64)     0         \n",
      "_________________________________________________________________\n",
      "dropout_21 (Dropout)         (None, 3, 15, 15, 64)     0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_36 (MaxPooling (None, 1, 7, 7, 64)       0         \n",
      "_________________________________________________________________\n",
      "flatten_7 (Flatten)          (None, 3136)              0         \n",
      "_________________________________________________________________\n",
      "dense_19 (Dense)             (None, 256)               803072    \n",
      "_________________________________________________________________\n",
      "dropout_22 (Dropout)         (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "dense_20 (Dense)             (None, 128)               32896     \n",
      "_________________________________________________________________\n",
      "dropout_23 (Dropout)         (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "dense_21 (Dense)             (None, 5)                 645       \n",
      "=================================================================\n",
      "Total params: 864,101\n",
      "Trainable params: 863,989\n",
      "Non-trainable params: 112\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "optimiser = Adam() #write your optimizer\n",
    "model.compile(optimizer=optimiser, loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
    "print (model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us create the `train_generator` and the `val_generator` which will be used in `.fit_generator`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_generator = generator(train_path, train_doc, batch_size)\n",
    "val_generator = generator(val_path, val_doc, batch_size,validation=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = 'model_init' + '_' + str(curr_dt_time).replace(' ','').replace(':','_') + '/'\n",
    "    \n",
    "if not os.path.exists(model_name):\n",
    "    os.mkdir(model_name)\n",
    "        \n",
    "filepath = model_name + 'model-{epoch:05d}-{loss:.5f}-{categorical_accuracy:.5f}-{val_loss:.5f}-{val_categorical_accuracy:.5f}.h5'\n",
    "\n",
    "checkpoint = ModelCheckpoint(filepath, monitor='val_loss', verbose=1, save_best_only=False, save_weights_only=False, mode='auto', period=1)\n",
    "\n",
    "# write the Reducelronplateau code here\n",
    "LR = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=2, cooldown=1, verbose=1)\n",
    "callbacks_list = [checkpoint, LR]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `steps_per_epoch` and `validation_steps` are used by `fit_generator` to decide the number of next() calls it need to make."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "if (num_train_sequences%batch_size) == 0:\n",
    "    steps_per_epoch = int(num_train_sequences/batch_size)\n",
    "else:\n",
    "    steps_per_epoch = (num_train_sequences//batch_size) + 1\n",
    "\n",
    "if (num_val_sequences%batch_size) == 0:\n",
    "    validation_steps = int(num_val_sequences/batch_size)\n",
    "else:\n",
    "    validation_steps = (num_val_sequences//batch_size) + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now fit the model. This will start training the model and with the help of the checkpoints, you'll be able to save the model at the end of each epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "Source path =  ./storage/Demo-gesture/Project_data/train ; batch size = 10\n",
      "Source path =  ./storage/Demo-gesture/Project_data/val ; batch size = 10\n",
      "67/67 [==============================] - 115s 2s/step - loss: 1.6766 - categorical_accuracy: 0.2972 - val_loss: 1.3920 - val_categorical_accuracy: 0.4700\n",
      "\n",
      "Epoch 00001: saving model to model_init_2018-10-2118_02_29.588384/model-00001-1.67796-0.29915-1.39202-0.47000.h5\n",
      "Epoch 2/20\n",
      "67/67 [==============================] - 33s 486ms/step - loss: 1.5612 - categorical_accuracy: 0.3665 - val_loss: 1.4123 - val_categorical_accuracy: 0.3800\n",
      "\n",
      "Epoch 00002: saving model to model_init_2018-10-2118_02_29.588384/model-00002-1.56120-0.36650-1.41231-0.38000.h5\n",
      "Epoch 3/20\n",
      "67/67 [==============================] - 34s 502ms/step - loss: 1.3966 - categorical_accuracy: 0.4295 - val_loss: 1.4822 - val_categorical_accuracy: 0.4600\n",
      "\n",
      "Epoch 00003: saving model to model_init_2018-10-2118_02_29.588384/model-00003-1.39658-0.42952-1.48217-0.46000.h5\n",
      "\n",
      "Epoch 00003: ReduceLROnPlateau reducing learning rate to 0.0005000000237487257.\n",
      "Epoch 4/20\n",
      "67/67 [==============================] - 32s 477ms/step - loss: 1.3741 - categorical_accuracy: 0.4328 - val_loss: 1.2293 - val_categorical_accuracy: 0.5300\n",
      "\n",
      "Epoch 00004: saving model to model_init_2018-10-2118_02_29.588384/model-00004-1.37410-0.43284-1.22931-0.53000.h5\n",
      "Epoch 5/20\n",
      "67/67 [==============================] - 33s 497ms/step - loss: 1.2576 - categorical_accuracy: 0.4577 - val_loss: 1.1498 - val_categorical_accuracy: 0.6300\n",
      "\n",
      "Epoch 00005: saving model to model_init_2018-10-2118_02_29.588384/model-00005-1.25759-0.45771-1.14977-0.63000.h5\n",
      "Epoch 6/20\n",
      "67/67 [==============================] - 33s 488ms/step - loss: 1.1050 - categorical_accuracy: 0.5672 - val_loss: 0.9732 - val_categorical_accuracy: 0.6700\n",
      "\n",
      "Epoch 00006: saving model to model_init_2018-10-2118_02_29.588384/model-00006-1.10502-0.56716-0.97318-0.67000.h5\n",
      "Epoch 7/20\n",
      "67/67 [==============================] - 33s 497ms/step - loss: 1.0169 - categorical_accuracy: 0.5854 - val_loss: 1.0151 - val_categorical_accuracy: 0.6900\n",
      "\n",
      "Epoch 00007: saving model to model_init_2018-10-2118_02_29.588384/model-00007-1.01691-0.58541-1.01510-0.69000.h5\n",
      "Epoch 8/20\n",
      "67/67 [==============================] - 32s 480ms/step - loss: 1.0137 - categorical_accuracy: 0.5970 - val_loss: 0.8777 - val_categorical_accuracy: 0.6600\n",
      "\n",
      "Epoch 00008: saving model to model_init_2018-10-2118_02_29.588384/model-00008-1.01374-0.59701-0.87768-0.66000.h5\n",
      "Epoch 9/20\n",
      "67/67 [==============================] - 33s 489ms/step - loss: 0.9708 - categorical_accuracy: 0.5987 - val_loss: 0.7682 - val_categorical_accuracy: 0.7400\n",
      "\n",
      "Epoch 00009: saving model to model_init_2018-10-2118_02_29.588384/model-00009-0.97082-0.59867-0.76819-0.74000.h5\n",
      "Epoch 10/20\n",
      "67/67 [==============================] - 33s 490ms/step - loss: 0.8585 - categorical_accuracy: 0.6567 - val_loss: 0.8484 - val_categorical_accuracy: 0.6700\n",
      "\n",
      "Epoch 00010: saving model to model_init_2018-10-2118_02_29.588384/model-00010-0.85848-0.65672-0.84838-0.67000.h5\n",
      "Epoch 11/20\n",
      "67/67 [==============================] - 33s 494ms/step - loss: 0.8390 - categorical_accuracy: 0.6766 - val_loss: 0.9478 - val_categorical_accuracy: 0.7300\n",
      "\n",
      "Epoch 00011: saving model to model_init_2018-10-2118_02_29.588384/model-00011-0.83900-0.67662-0.94784-0.73000.h5\n",
      "\n",
      "Epoch 00011: ReduceLROnPlateau reducing learning rate to 0.0002500000118743628.\n",
      "Epoch 12/20\n",
      "67/67 [==============================] - 32s 484ms/step - loss: 0.6513 - categorical_accuracy: 0.7512 - val_loss: 0.7291 - val_categorical_accuracy: 0.7100\n",
      "\n",
      "Epoch 00012: saving model to model_init_2018-10-2118_02_29.588384/model-00012-0.65128-0.75124-0.72911-0.71000.h5\n",
      "Epoch 13/20\n",
      "67/67 [==============================] - 32s 482ms/step - loss: 0.6478 - categorical_accuracy: 0.7280 - val_loss: 0.6481 - val_categorical_accuracy: 0.7500\n",
      "\n",
      "Epoch 00013: saving model to model_init_2018-10-2118_02_29.588384/model-00013-0.64778-0.72803-0.64809-0.75000.h5\n",
      "Epoch 14/20\n",
      "67/67 [==============================] - 34s 503ms/step - loss: 0.6001 - categorical_accuracy: 0.7745 - val_loss: 0.6912 - val_categorical_accuracy: 0.6900\n",
      "\n",
      "Epoch 00014: saving model to model_init_2018-10-2118_02_29.588384/model-00014-0.60013-0.77446-0.69118-0.69000.h5\n",
      "Epoch 15/20\n",
      "67/67 [==============================] - 33s 488ms/step - loss: 0.5834 - categorical_accuracy: 0.7562 - val_loss: 0.8076 - val_categorical_accuracy: 0.7000\n",
      "\n",
      "Epoch 00015: saving model to model_init_2018-10-2118_02_29.588384/model-00015-0.58336-0.75622-0.80763-0.70000.h5\n",
      "\n",
      "Epoch 00015: ReduceLROnPlateau reducing learning rate to 0.0001250000059371814.\n",
      "Epoch 16/20\n",
      "67/67 [==============================] - 32s 482ms/step - loss: 0.5462 - categorical_accuracy: 0.7944 - val_loss: 0.6183 - val_categorical_accuracy: 0.7400\n",
      "\n",
      "Epoch 00016: saving model to model_init_2018-10-2118_02_29.588384/model-00016-0.54619-0.79436-0.61834-0.74000.h5\n",
      "Epoch 17/20\n",
      "67/67 [==============================] - 34s 504ms/step - loss: 0.5750 - categorical_accuracy: 0.7711 - val_loss: 0.5670 - val_categorical_accuracy: 0.7700\n",
      "\n",
      "Epoch 00017: saving model to model_init_2018-10-2118_02_29.588384/model-00017-0.57498-0.77114-0.56700-0.77000.h5\n",
      "Epoch 18/20\n",
      "67/67 [==============================] - 32s 478ms/step - loss: 0.5183 - categorical_accuracy: 0.7910 - val_loss: 0.5089 - val_categorical_accuracy: 0.7800\n",
      "\n",
      "Epoch 00018: saving model to model_init_2018-10-2118_02_29.588384/model-00018-0.51826-0.79104-0.50885-0.78000.h5\n",
      "Epoch 19/20\n",
      "67/67 [==============================] - 33s 486ms/step - loss: 0.4489 - categorical_accuracy: 0.8192 - val_loss: 0.5587 - val_categorical_accuracy: 0.7900\n",
      "\n",
      "Epoch 00019: saving model to model_init_2018-10-2118_02_29.588384/model-00019-0.44894-0.81924-0.55868-0.79000.h5\n",
      "Epoch 20/20\n",
      "67/67 [==============================] - 33s 485ms/step - loss: 0.5012 - categorical_accuracy: 0.8159 - val_loss: 0.5473 - val_categorical_accuracy: 0.7900\n",
      "\n",
      "Epoch 00020: saving model to model_init_2018-10-2118_02_29.588384/model-00020-0.50118-0.81592-0.54727-0.79000.h5\n",
      "\n",
      "Epoch 00020: ReduceLROnPlateau reducing learning rate to 6.25000029685907e-05.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fd968138dd8>"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = 10\n",
    "num_epochs = 20\n",
    "model.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "34/34 [==============================] - 19s 547ms/step - loss: 0.3854 - categorical_accuracy: 0.8464 - val_loss: 0.5860 - val_categorical_accuracy: 0.7400\n",
      "\n",
      "Epoch 00001: saving model to model_init_2018-10-2118_02_29.588384/model-00001-0.38542-0.84641-0.58598-0.74000.h5\n",
      "Epoch 2/20\n",
      "34/34 [==============================] - 17s 505ms/step - loss: 0.3897 - categorical_accuracy: 0.8693 - val_loss: 0.5580 - val_categorical_accuracy: 0.8000\n",
      "\n",
      "Epoch 00002: saving model to model_init_2018-10-2118_02_29.588384/model-00002-0.38967-0.86928-0.55803-0.80000.h5\n",
      "Epoch 3/20\n",
      "34/34 [==============================] - 17s 492ms/step - loss: 0.4773 - categorical_accuracy: 0.7974 - val_loss: 0.3119 - val_categorical_accuracy: 0.8600\n",
      "\n",
      "Epoch 00003: saving model to model_init_2018-10-2118_02_29.588384/model-00003-0.47731-0.79739-0.31192-0.86000.h5\n",
      "Epoch 4/20\n",
      "34/34 [==============================] - 17s 498ms/step - loss: 0.4562 - categorical_accuracy: 0.8497 - val_loss: 0.4582 - val_categorical_accuracy: 0.8000\n",
      "\n",
      "Epoch 00004: saving model to model_init_2018-10-2118_02_29.588384/model-00004-0.45622-0.84967-0.45824-0.80000.h5\n",
      "Epoch 5/20\n",
      "34/34 [==============================] - 17s 504ms/step - loss: 0.3790 - categorical_accuracy: 0.8595 - val_loss: 0.4840 - val_categorical_accuracy: 0.8000\n",
      "\n",
      "Epoch 00005: saving model to model_init_2018-10-2118_02_29.588384/model-00005-0.37903-0.85948-0.48397-0.80000.h5\n",
      "\n",
      "Epoch 00005: ReduceLROnPlateau reducing learning rate to 3.125000148429535e-05.\n",
      "Epoch 6/20\n",
      "34/34 [==============================] - 17s 492ms/step - loss: 0.4398 - categorical_accuracy: 0.8464 - val_loss: 0.4722 - val_categorical_accuracy: 0.8400\n",
      "\n",
      "Epoch 00006: saving model to model_init_2018-10-2118_02_29.588384/model-00006-0.43976-0.84641-0.47223-0.84000.h5\n",
      "Epoch 7/20\n",
      "34/34 [==============================] - 17s 495ms/step - loss: 0.3248 - categorical_accuracy: 0.8856 - val_loss: 0.5560 - val_categorical_accuracy: 0.7600\n",
      "\n",
      "Epoch 00007: saving model to model_init_2018-10-2118_02_29.588384/model-00007-0.32481-0.88562-0.55600-0.76000.h5\n",
      "\n",
      "Epoch 00007: ReduceLROnPlateau reducing learning rate to 1.5625000742147677e-05.\n",
      "Epoch 8/20\n",
      "34/34 [==============================] - 16s 479ms/step - loss: 0.4114 - categorical_accuracy: 0.8399 - val_loss: 0.5010 - val_categorical_accuracy: 0.8200\n",
      "\n",
      "Epoch 00008: saving model to model_init_2018-10-2118_02_29.588384/model-00008-0.41142-0.83987-0.50103-0.82000.h5\n",
      "Epoch 9/20\n",
      "34/34 [==============================] - 16s 484ms/step - loss: 0.3683 - categorical_accuracy: 0.8693 - val_loss: 0.4484 - val_categorical_accuracy: 0.7600\n",
      "\n",
      "Epoch 00009: saving model to model_init_2018-10-2118_02_29.588384/model-00009-0.36833-0.86928-0.44843-0.76000.h5\n",
      "\n",
      "Epoch 00009: ReduceLROnPlateau reducing learning rate to 7.812500371073838e-06.\n",
      "Epoch 10/20\n",
      "34/34 [==============================] - 16s 479ms/step - loss: 0.3935 - categorical_accuracy: 0.8529 - val_loss: 0.3130 - val_categorical_accuracy: 0.8800\n",
      "\n",
      "Epoch 00010: saving model to model_init_2018-10-2118_02_29.588384/model-00010-0.39346-0.85294-0.31305-0.88000.h5\n",
      "Epoch 11/20\n",
      "34/34 [==============================] - 17s 487ms/step - loss: 0.4121 - categorical_accuracy: 0.8333 - val_loss: 0.5551 - val_categorical_accuracy: 0.7800\n",
      "\n",
      "Epoch 00011: saving model to model_init_2018-10-2118_02_29.588384/model-00011-0.41212-0.83333-0.55512-0.78000.h5\n",
      "\n",
      "Epoch 00011: ReduceLROnPlateau reducing learning rate to 3.906250185536919e-06.\n",
      "Epoch 12/20\n",
      "34/34 [==============================] - 17s 505ms/step - loss: 0.3696 - categorical_accuracy: 0.8529 - val_loss: 0.5074 - val_categorical_accuracy: 0.7800\n",
      "\n",
      "Epoch 00012: saving model to model_init_2018-10-2118_02_29.588384/model-00012-0.36964-0.85294-0.50735-0.78000.h5\n",
      "Epoch 13/20\n",
      "34/34 [==============================] - 17s 497ms/step - loss: 0.4278 - categorical_accuracy: 0.8497 - val_loss: 0.4602 - val_categorical_accuracy: 0.8400\n",
      "\n",
      "Epoch 00013: saving model to model_init_2018-10-2118_02_29.588384/model-00013-0.42783-0.84967-0.46023-0.84000.h5\n",
      "\n",
      "Epoch 00013: ReduceLROnPlateau reducing learning rate to 1.9531250927684596e-06.\n",
      "Epoch 14/20\n",
      "34/34 [==============================] - 17s 491ms/step - loss: 0.3789 - categorical_accuracy: 0.8464 - val_loss: 0.4324 - val_categorical_accuracy: 0.7800\n",
      "\n",
      "Epoch 00014: saving model to model_init_2018-10-2118_02_29.588384/model-00014-0.37891-0.84641-0.43243-0.78000.h5\n",
      "Epoch 15/20\n",
      "34/34 [==============================] - 17s 495ms/step - loss: 0.4161 - categorical_accuracy: 0.8268 - val_loss: 0.4482 - val_categorical_accuracy: 0.8000\n",
      "\n",
      "Epoch 00015: saving model to model_init_2018-10-2118_02_29.588384/model-00015-0.41609-0.82680-0.44819-0.80000.h5\n",
      "\n",
      "Epoch 00015: ReduceLROnPlateau reducing learning rate to 9.765625463842298e-07.\n",
      "Epoch 16/20\n",
      "34/34 [==============================] - 16s 460ms/step - loss: 0.4387 - categorical_accuracy: 0.8039 - val_loss: 0.5061 - val_categorical_accuracy: 0.8000\n",
      "\n",
      "Epoch 00016: saving model to model_init_2018-10-2118_02_29.588384/model-00016-0.43871-0.80392-0.50610-0.80000.h5\n",
      "Epoch 17/20\n",
      "34/34 [==============================] - 18s 525ms/step - loss: 0.3719 - categorical_accuracy: 0.8529 - val_loss: 0.5178 - val_categorical_accuracy: 0.8400\n",
      "\n",
      "Epoch 00017: saving model to model_init_2018-10-2118_02_29.588384/model-00017-0.37187-0.85294-0.51785-0.84000.h5\n",
      "\n",
      "Epoch 00017: ReduceLROnPlateau reducing learning rate to 4.882812731921149e-07.\n",
      "Epoch 18/20\n",
      "34/34 [==============================] - 17s 501ms/step - loss: 0.4002 - categorical_accuracy: 0.8660 - val_loss: 0.3957 - val_categorical_accuracy: 0.8000\n",
      "\n",
      "Epoch 00018: saving model to model_init_2018-10-2118_02_29.588384/model-00018-0.40021-0.86601-0.39573-0.80000.h5\n",
      "Epoch 19/20\n",
      "34/34 [==============================] - 17s 501ms/step - loss: 0.3555 - categorical_accuracy: 0.8725 - val_loss: 0.5425 - val_categorical_accuracy: 0.7600\n",
      "\n",
      "Epoch 00019: saving model to model_init_2018-10-2118_02_29.588384/model-00019-0.35551-0.87255-0.54246-0.76000.h5\n",
      "\n",
      "Epoch 00019: ReduceLROnPlateau reducing learning rate to 2.4414063659605745e-07.\n",
      "Epoch 20/20\n",
      "34/34 [==============================] - 16s 483ms/step - loss: 0.4065 - categorical_accuracy: 0.8497 - val_loss: 0.4677 - val_categorical_accuracy: 0.8200\n",
      "\n",
      "Epoch 00020: saving model to model_init_2018-10-2118_02_29.588384/model-00020-0.40645-0.84967-0.46770-0.82000.h5\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fd9684f6518>"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = 20\n",
    "num_epochs = 20\n",
    "\n",
    "if (num_train_sequences%batch_size) == 0:\n",
    "    steps_per_epoch = int(num_train_sequences/batch_size)\n",
    "else:\n",
    "    steps_per_epoch = (num_train_sequences//batch_size) + 1\n",
    "\n",
    "if (num_val_sequences%batch_size) == 0:\n",
    "    validation_steps = int(num_val_sequences/batch_size)\n",
    "else:\n",
    "    validation_steps = (num_val_sequences//batch_size) + 1\n",
    "    \n",
    "model.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "batch_size = 30\n",
    "num_epochs = 30\n",
    "\n",
    "if (num_train_sequences%batch_size) == 0:\n",
    "    steps_per_epoch = int(num_train_sequences/batch_size)\n",
    "else:\n",
    "    steps_per_epoch = (num_train_sequences//batch_size) + 1\n",
    "\n",
    "if (num_val_sequences%batch_size) == 0:\n",
    "    validation_steps = int(num_val_sequences/batch_size)\n",
    "else:\n",
    "    validation_steps = (num_val_sequences//batch_size) + 1\n",
    "    \n",
    "model.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "batch_size = 50\n",
    "num_epochs = 30\n",
    "\n",
    "if (num_train_sequences%batch_size) == 0:\n",
    "    steps_per_epoch = int(num_train_sequences/batch_size)\n",
    "else:\n",
    "    steps_per_epoch = (num_train_sequences//batch_size) + 1\n",
    "\n",
    "if (num_val_sequences%batch_size) == 0:\n",
    "    validation_steps = int(num_val_sequences/batch_size)\n",
    "else:\n",
    "    validation_steps = (num_val_sequences//batch_size) + 1\n",
    "    \n",
    "model.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "batch_size = 90\n",
    "num_epochs = 30\n",
    "\n",
    "if (num_train_sequences%batch_size) == 0:\n",
    "    steps_per_epoch = int(num_train_sequences/batch_size)\n",
    "else:\n",
    "    steps_per_epoch = (num_train_sequences//batch_size) + 1\n",
    "\n",
    "if (num_val_sequences%batch_size) == 0:\n",
    "    validation_steps = int(num_val_sequences/batch_size)\n",
    "else:\n",
    "    validation_steps = (num_val_sequences//batch_size) + 1\n",
    "    \n",
    "model.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "batch_size = 90\n",
    "num_epochs = 20\n",
    "\n",
    "if (num_train_sequences%batch_size) == 0:\n",
    "    steps_per_epoch = int(num_train_sequences/batch_size)\n",
    "else:\n",
    "    steps_per_epoch = (num_train_sequences//batch_size) + 1\n",
    "\n",
    "if (num_val_sequences%batch_size) == 0:\n",
    "    validation_steps = int(num_val_sequences/batch_size)\n",
    "else:\n",
    "    validation_steps = (num_val_sequences//batch_size) + 1\n",
    "    \n",
    "\n",
    "model.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "batch_size = 90\n",
    "num_epochs = 20\n",
    "\n",
    "if (num_train_sequences%batch_size) == 0:\n",
    "    steps_per_epoch = int(num_train_sequences/batch_size)\n",
    "else:\n",
    "    steps_per_epoch = (num_train_sequences//batch_size) + 1\n",
    "\n",
    "if (num_val_sequences%batch_size) == 0:\n",
    "    validation_steps = int(num_val_sequences/batch_size)\n",
    "else:\n",
    "    validation_steps = (num_val_sequences//batch_size) + 1\n",
    "    \n",
    "\n",
    "model.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 90\n",
    "num_epochs = 20\n",
    "\n",
    "if (num_train_sequences%batch_size) == 0:\n",
    "    steps_per_epoch = int(num_train_sequences/batch_size)\n",
    "else:\n",
    "    steps_per_epoch = (num_train_sequences//batch_size) + 1\n",
    "\n",
    "if (num_val_sequences%batch_size) == 0:\n",
    "    validation_steps = int(num_val_sequences/batch_size)\n",
    "else:\n",
    "    validation_steps = (num_val_sequences//batch_size) + 1\n",
    "    \n",
    "\n",
    "model.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}