mnist_nn_analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this assignment, you'll implement an L-layered deep neural network and train it on the MNIST dataset. The MNIST dataset contains scanned images of handwritten digits, along with their correct classification labels (between 0-9). MNIST's name comes from the fact that it is a modified subset of two data sets collected by NIST, the United States' National Institute of Standards and Technology.<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pickle\n",
    "import gzip\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import h5py\n",
    "import sklearn\n",
    "import sklearn.datasets\n",
    "import scipy\n",
    "from PIL import Image\n",
    "from scipy import ndimage\n",
    "\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MNIST dataset we use here is 'mnist.pkl.gz' which is divided into training, validation and test data. The following function <i> load_data() </i> unpacks the file and extracts the training, validation and test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_data():\n",
    "    f = gzip.open('mnist.pkl.gz', 'rb')\n",
    "    f.seek(0)\n",
    "    training_data, validation_data, test_data = pickle.load(f, encoding='latin1')\n",
    "    f.close()\n",
    "    return (training_data, validation_data, test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how the data looks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_data, validation_data, test_data = load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]], dtype=float32),\n",
       " array([5, 0, 4, ..., 8, 4, 8], dtype=int64))"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(50000, 784)\n",
      "(50000,)\n"
     ]
    }
   ],
   "source": [
    "# shape of data\n",
    "print(training_data[0].shape)\n",
    "print(training_data[1].shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The feature dataset is:[[0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " ...\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]]\n",
      "The target dataset is:[5 0 4 ... 8 4 8]\n",
      "The number of examples in the training dataset is:50000\n",
      "The number of points in a single input is:784\n"
     ]
    }
   ],
   "source": [
    "print(\"The feature dataset is:\" + str(training_data[0]))\n",
    "print(\"The target dataset is:\" + str(training_data[1]))\n",
    "print(\"The number of examples in the training dataset is:\" + str(len(training_data[0])))\n",
    "print(\"The number of points in a single input is:\" + str(len(training_data[0][1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, as discussed earlier in the lectures, the target variable is converted to a one hot matrix. We use the function <i> one_hot </i> to convert the target dataset to one hot encoding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def one_hot(j):\n",
    "    # input is the target dataset of shape (m,) where m is the number of data points\n",
    "    # returns a 2 dimensional array of shape (10, m) where each target value is converted to a one hot encoding\n",
    "    # Look at the next block of code for a better understanding of one hot encoding\n",
    "    n = j.shape[0]\n",
    "    new_array = np.zeros((10, n))\n",
    "    index = 0\n",
    "    for res in j:\n",
    "        new_array[res][index] = 1.0\n",
    "        index = index + 1\n",
    "    return new_array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10,)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       [0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],\n",
       "       [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],\n",
       "       [0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],\n",
       "       [0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],\n",
       "       [0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],\n",
       "       [0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],\n",
       "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])\n",
    "print(data.shape)\n",
    "one_hot(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function data_wrapper() will convert the dataset into the desired shape and also convert the ground truth labels to one_hot matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def data_wrapper():\n",
    "    tr_d, va_d, te_d = load_data()\n",
    "    \n",
    "    training_inputs = np.array(tr_d[0][:]).T\n",
    "    training_results = np.array(tr_d[1][:])\n",
    "    train_set_y = one_hot(training_results)\n",
    "    \n",
    "    validation_inputs = np.array(va_d[0][:]).T\n",
    "    validation_results = np.array(va_d[1][:])\n",
    "    validation_set_y = one_hot(validation_results)\n",
    "    \n",
    "    test_inputs = np.array(te_d[0][:]).T\n",
    "    test_results = np.array(te_d[1][:])\n",
    "    test_set_y = one_hot(test_results)\n",
    "    \n",
    "    return (training_inputs, train_set_y, test_inputs, test_set_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_set_x, train_set_y, test_set_x, test_set_y = data_wrapper()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train_set_x shape: (784, 50000)\n",
      "train_set_y shape: (10, 50000)\n",
      "test_set_x shape: (784, 10000)\n",
      "test_set_y shape: (10, 10000)\n"
     ]
    }
   ],
   "source": [
    "print (\"train_set_x shape: \" + str(train_set_x.shape))\n",
    "print (\"train_set_y shape: \" + str(train_set_y.shape))\n",
    "print (\"test_set_x shape: \" + str(test_set_x.shape))\n",
    "print (\"test_set_y shape: \" + str(test_set_y.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the data_wrapper has converted the training and validation data into numpy array of desired shapes. Let's convert the actual labels into a dataframe to see if the one hot conversions are correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = pd.DataFrame(train_set_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The target dataset is:[5 0 4 ... 8 4 8]\n",
      "The one hot encoding dataset is:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>49990</th>\n",
       "      <th>49991</th>\n",
       "      <th>49992</th>\n",
       "      <th>49993</th>\n",
       "      <th>49994</th>\n",
       "      <th>49995</th>\n",
       "      <th>49996</th>\n",
       "      <th>49997</th>\n",
       "      <th>49998</th>\n",
       "      <th>49999</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 50000 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   0      1      2      3      4      5      6      7      8      9      \\\n",
       "0    0.0    1.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "1    0.0    0.0    0.0    1.0    0.0    0.0    1.0    0.0    1.0    0.0   \n",
       "2    0.0    0.0    0.0    0.0    0.0    1.0    0.0    0.0    0.0    0.0   \n",
       "3    0.0    0.0    0.0    0.0    0.0    0.0    0.0    1.0    0.0    0.0   \n",
       "4    0.0    0.0    1.0    0.0    0.0    0.0    0.0    0.0    0.0    1.0   \n",
       "5    1.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "6    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "7    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "8    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "9    0.0    0.0    0.0    0.0    1.0    0.0    0.0    0.0    0.0    0.0   \n",
       "\n",
       "   ...    49990  49991  49992  49993  49994  49995  49996  49997  49998  49999  \n",
       "0  ...      0.0    0.0    0.0    0.0    0.0    0.0    1.0    0.0    0.0    0.0  \n",
       "1  ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0  \n",
       "2  ...      0.0    0.0    0.0    1.0    0.0    0.0    0.0    0.0    0.0    0.0  \n",
       "3  ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0  \n",
       "4  ...      0.0    0.0    0.0    0.0    1.0    0.0    0.0    0.0    1.0    0.0  \n",
       "5  ...      0.0    1.0    1.0    0.0    0.0    1.0    0.0    0.0    0.0    0.0  \n",
       "6  ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0  \n",
       "7  ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0  \n",
       "8  ...      1.0    0.0    0.0    0.0    0.0    0.0    0.0    1.0    0.0    1.0  \n",
       "9  ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0  \n",
       "\n",
       "[10 rows x 50000 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"The target dataset is:\" + str(training_data[1]))\n",
    "print(\"The one hot encoding dataset is:\")\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us visualise the dataset. Feel free to change the index to see if the training data has been correctly tagged."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x18fb816fb00>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAEICAYAAACQ6CLfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAEJJJREFUeJzt3X2QVfV9x/H3x5WYBrABKQ+uGBJER9saUxiaKUxKTJNSx45mRA01lY5Y0jaMzRi16uhI01ohk0Ts1MmU1AfABMSIyhinieNoTOxIXRgJRBqDDCphZQVU0Oog+O0f92xnXe89d/c+nbv7+7xmdu7d8z0PX+7w2XPOPefenyICM0vPMUU3YGbFcPjNEuXwmyXK4TdLlMNvliiH3yxRDn8iJD0h6fJGLyvpekn/UV93VgSHf4iRtEvSnxTdR6+I+JeIGPQfFUljJT0g6S1JL0r6i2b0Z5UdW3QDlqzbgcPABOAs4EeStkTEL4ttKx3e8w8TksZIeljSq5Jey56f1G+2qZL+W9Ibkh6SNLbP8p+W9F+SXpe0RdKcAW53iaR7sucflnSPpP3Zep6RNKHMMiOBC4AbI+LNiPg5sAH4y1r//TZ4Dv/wcQxwF/Ax4GTgbeDf+s1zKXAZcCJwBPhXAEmdwI+AfwbGAlcB90v6nUH2sAD4bWAycALwN1kf/Z0KHI2I5/tM2wL87iC3Z3Vw+IeJiNgfEfdHxP9GxCHgZuCP+822OiK2RcRbwI3ARZI6gC8Dj0TEIxHxXkQ8CnQB5wyyjXcphf6UiDgaEZsi4mCZ+UYBb/Sb9gYwepDbszo4/MOEpI9I+vfszbODwJPAR7Nw93q5z/MXgRHAOEpHCxdmh+qvS3odmA1MGmQbq4EfA2sl7ZH0TUkjysz3JnB8v2nHA4cGuT2rg8M/fHwdOA34w4g4HvhMNl195pnc5/nJlPbU+yj9UVgdER/t8zMyIpYOpoGIeDci/jEizgD+CDiX0qlGf88Dx0qa1mfaJwG/2ddCDv/QNCJ7c63351hKh8xvA69nb+TdVGa5L0s6Q9JHgG8AP4yIo8A9wJ9L+lNJHdk655R5wzCXpM9K+v3saOMgpT8uR/vPl512rAe+IWmkpFnAeZSOHKxFHP6h6RFKQe/9WQIsB36L0p78aeA/yyy3GrgbeAX4MHAFQES8TCl81wOvUjoSuJrB//+YCPyQUvC3Az+l9IelnL/L+u0B1gB/68t8rSV/mYdZmrznN0uUw2+WKIffLFEOv1miWvrBHkl+d9GsySJC1eeqc88vaa6kX0naIenaetZlZq1V86W+7EaO54HPA7uBZ4D5EfFczjLe85s1WSv2/DOBHRGxMyIOA2sp3ShiZkNAPeHv5P0fFNmdTXsfSYskdUnqqmNbZtZg9bzhV+7Q4gOH9RGxAlgBPuw3ayf17Pl38/5PiZ0E7KmvHTNrlXrC/wwwTdLHJX0I+BKlr2IysyGg5sP+iDgiaTGlL2/oAO70p7LMho6WfqrP5/xmzdeSm3zMbOhy+M0S5fCbJcrhN0uUw2+WKIffLFEOv1miHH6zRDn8Zoly+M0S5fCbJcrhN0uUw2+WKIffLFEOv1miHH6zRDn8Zoly+M0S5fCbJcrhN0uUw2+WKIffLFEOv1miHH6zRDn8Zoly+M0S5fCbJcrhN0uUw2+WqJqH6LY0nHLKKbn1K664Ire+ePHiijUpfzDZI0eO5NYvv/zy3PqaNWsq1g4fPpy7bArqCr+kXcAh4ChwJCJmNKIpM2u+Ruz5PxsR+xqwHjNrIZ/zmyWq3vAH8BNJmyQtKjeDpEWSuiR11bktM2ugeg/7Z0XEHknjgUcl/U9EPNl3hohYAawAkBR1bs/MGqSuPX9E7Mkee4AHgJmNaMrMmq/m8EsaKWl073PgC8C2RjVmZs2liNqOxCV9gtLeHkqnDz+IiJurLOPD/hbr6OjIrV966aW59WXLluXWx40bN+ieevX09OTWx48fX/O6AaZNm1ax9sILL9S17nYWEfk3UGRqPuePiJ3AJ2td3syK5Ut9Zoly+M0S5fCbJcrhN0uUw2+WqJov9dW0MV/qa4r58+dXrE2fPj132SuvvLKubT/44IO59dtvv71irdrltrVr1+bWZ87Mv6fsiSeeqFg7++yzc5cdygZ6qc97frNEOfxmiXL4zRLl8JslyuE3S5TDb5Yoh98sUb7OPwTkff01wG233VaxVu3rsffv359bnzt3bm598+bNufV6/n+NGjUqt37w4MGatz1r1qzcZZ9++uncejvzdX4zy+XwmyXK4TdLlMNvliiH3yxRDr9Zohx+s0R5iO42UO16drXr/HnX8t96663cZc8999zc+qZNm3LrzVRtGO3t27fn1k8//fRGtjPseM9vliiH3yxRDr9Zohx+s0Q5/GaJcvjNEuXwmyXK1/nbwOjRo3Prp556as3rXr58eW5948aNNa+72apd59+6dWtu3df581Xd80u6U1KPpG19po2V9KikX2ePY5rbppk12kAO++8G+n+dy7XAYxExDXgs+93MhpCq4Y+IJ4ED/SafB6zMnq8Ezm9wX2bWZLWe80+IiG6AiOiWNL7SjJIWAYtq3I6ZNUnT3/CLiBXACvAXeJq1k1ov9e2VNAkge+xpXEtm1gq1hn8DsCB7vgB4qDHtmFmrVD3sl7QGmAOMk7QbuAlYCqyTtBB4CbiwmU0OdyeccEJdy+d9Zv+uu+6qa902fFUNf0TMr1D6XIN7MbMW8u29Zoly+M0S5fCbJcrhN0uUw2+WKH+ktw3MmzevruXXrVtXsbZz58661m3Dl/f8Zoly+M0S5fCbJcrhN0uUw2+WKIffLFEOv1mifJ2/Bap9ZHfhwoV1rb+rq6uu5dvVcccdl1ufNWtWizoZnrznN0uUw2+WKIffLFEOv1miHH6zRDn8Zoly+M0S5ev8LXDaaafl1js7O+ta/4ED/YdSHB46Ojpy69Vet3feeadi7e23366pp+HEe36zRDn8Zoly+M0S5fCbJcrhN0uUw2+WKIffLFG+zj8MbNiwoegW2tKOHTsq1rZs2dLCTtpT1T2/pDsl9Uja1mfaEkm/kfRs9nNOc9s0s0YbyGH/3cDcMtNvjYizsp9HGtuWmTVb1fBHxJPA8Lx/1Cxh9bzht1jSL7LTgjGVZpK0SFKXpOH5RXNmQ1St4f8uMBU4C+gGvl1pxohYEREzImJGjdsysyaoKfwRsTcijkbEe8D3gJmNbcvMmq2m8Eua1OfXLwLbKs1rZu2p6nV+SWuAOcA4SbuBm4A5ks4CAtgFfKWJPVqiFixYUNfyy5Yta1Anw1PV8EfE/DKT72hCL2bWQr691yxRDr9Zohx+s0Q5/GaJcvjNEqWIaN3GpNZtrI2MGDEit/7cc8/l1qdOnZpbHzlyZMVaO39F9cSJE3Prmzdvrmv5E088sWLtlVdeyV12KIsIDWQ+7/nNEuXwmyXK4TdLlMNvliiH3yxRDr9Zohx+s0T5q7tb4N13382tHz16tEWdtJfZs2fn1qtdx6/2urXyHpahyHt+s0Q5/GaJcvjNEuXwmyXK4TdLlMNvliiH3yxRvs4/DHR2dlas5Q1T3Qrjx4+vWLvhhhtyl612HX/hwoW59b179+bWU+c9v1miHH6zRDn8Zoly+M0S5fCbJcrhN0uUw2+WqIEM0T0ZWAVMBN4DVkTEbZLGAvcCUygN031RRLzWvFaHr3vvvTe3fuONN+bW582bV7G2dOnSmnoaqI6Ojtz6NddcU7F25pln5i7b3d2dW1+1alVu3fINZM9/BPh6RJwOfBr4qqQzgGuBxyJiGvBY9ruZDRFVwx8R3RGxOXt+CNgOdALnASuz2VYC5zerSTNrvEGd80uaAnwK2AhMiIhuKP2BACrfx2lmbWfA9/ZLGgXcD3wtIg5KAxoODEmLgEW1tWdmzTKgPb+kEZSC//2IWJ9N3itpUlafBPSUWzYiVkTEjIiY0YiGzawxqoZfpV38HcD2iPhOn9IGYEH2fAHwUOPbM7NmqTpEt6TZwM+ArZQu9QFcT+m8fx1wMvAScGFEHKiyLn+XchkXXHBBbv2+++7Lre/atatibfr06bnLvvZafVdnL7nkktz66tWrK9YOHMj978LcuXNz611dXbn1VA10iO6q5/wR8XOg0so+N5imzKx9+A4/s0Q5/GaJcvjNEuXwmyXK4TdLlMNvlih/dXcbePzxx3Pr+/fvz61PmTKlYu3qq6/OXfbWW2/NrV922WW59byP7FazfPny3Lqv4zeX9/xmiXL4zRLl8JslyuE3S5TDb5Yoh98sUQ6/WaKqfp6/oRvz5/lrMmNG/pcgPfXUUxVrI0aMyF123759ufWxY8fm1o85Jn//sX79+oq1iy++OHfZakN0W3kD/Ty/9/xmiXL4zRLl8JslyuE3S5TDb5Yoh98sUQ6/WaJ8nX8YuOqqqyrWrrvuutxlx4wZU9e2b7nlltx63vcFVLvHwGrj6/xmlsvhN0uUw2+WKIffLFEOv1miHH6zRDn8Zomqep1f0mRgFTAReA9YERG3SVoC/DXwajbr9RHxSJV1+Tq/WZMN9Dr/QMI/CZgUEZsljQY2AecDFwFvRsS3BtqUw2/WfAMNf9UReyKiG+jOnh+StB3orK89MyvaoM75JU0BPgVszCYtlvQLSXdKKnufqKRFkrokeewlszYy4Hv7JY0CfgrcHBHrJU0A9gEB/BOlU4Pcgd182G/WfA075weQNAJ4GPhxRHynTH0K8HBE/F6V9Tj8Zk3WsA/2SBJwB7C9b/CzNwJ7fRHYNtgmzaw4A3m3fzbwM2ArpUt9ANcD84GzKB327wK+kr05mLcu7/nNmqyhh/2N4vCbNZ8/z29muRx+s0Q5/GaJcvjNEuXwmyXK4TdLlMNvliiH3yxRDr9Zohx+s0Q5/GaJcvjNEuXwmyXK4TdLVNUv8GywfcCLfX4fl01rR+3aW7v2Be6tVo3s7WMDnbGln+f/wMalroiYUVgDOdq1t3btC9xbrYrqzYf9Zoly+M0SVXT4VxS8/Tzt2lu79gXurVaF9FboOb+ZFafoPb+ZFcThN0tUIeGXNFfSryTtkHRtET1UImmXpK2Sni16fMFsDMQeSdv6TBsr6VFJv84ey46RWFBvSyT9JnvtnpV0TkG9TZb0uKTtkn4p6e+z6YW+djl9FfK6tfycX1IH8DzweWA38AwwPyKea2kjFUjaBcyIiMJvCJH0GeBNYFXvUGiSvgkciIil2R/OMRHxD23S2xIGOWx7k3qrNKz8X1Hga9fI4e4boYg9/0xgR0TsjIjDwFrgvAL6aHsR8SRwoN/k84CV2fOVlP7ztFyF3tpCRHRHxObs+SGgd1j5Ql+7nL4KUUT4O4GX+/y+mwJfgDIC+ImkTZIWFd1MGRN6h0XLHscX3E9/VYdtb6V+w8q3zWtXy3D3jVZE+MsNJdRO1xtnRcQfAH8GfDU7vLWB+S4wldIYjt3At4tsJhtW/n7gaxFxsMhe+irTVyGvWxHh3w1M7vP7ScCeAvooKyL2ZI89wAOUTlPayd7eEZKzx56C+/l/EbE3Io5GxHvA9yjwtcuGlb8f+H5ErM8mF/7aleurqNetiPA/A0yT9HFJHwK+BGwooI8PkDQyeyMGSSOBL9B+Q49vABZkzxcADxXYy/u0y7DtlYaVp+DXrt2Guy/kDr/sUsZyoAO4MyJubnkTZUj6BKW9PZQ+7vyDInuTtAaYQ+kjn3uBm4AHgXXAycBLwIUR0fI33ir0NodBDtvepN4qDSu/kQJfu0YOd9+Qfnx7r1mafIefWaIcfrNEOfxmiXL4zRLl8JslyuE3S5TDb5ao/wPMtvAoDfUVWwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "index  = 1000\n",
    "k = train_set_x[:,index]\n",
    "k = k.reshape((28, 28))\n",
    "plt.title('Label is {label}'.format(label= training_data[1][index]))\n",
    "plt.imshow(k, cmap='gray')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feedforward"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### sigmoid\n",
    "This is one of the activation functions. It takes the cumulative input to the layer, the matrix **Z**, as the input. Upon application of the **`sigmoid`** function, the output matrix **H** is calculated. Also, **Z** is stored as the variable **sigmoid_memory** since it will be later used in backpropagation.You use _[np.exp()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.exp.html)_ here in the following way. The exponential gets applied to all the elements of Z."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(Z):\n",
    "    \n",
    "    # Z is numpy array of shape (n, m) where n is number of neurons in the layer and m is the number of samples \n",
    "    # sigmoid_memory is stored as it is used later on in backpropagation\n",
    "    \n",
    "    H = 1/(1+np.exp(-Z))\n",
    "    sigmoid_memory = Z\n",
    "    \n",
    "    return H, sigmoid_memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sigmoid(Z) = (array([[0.5       , 0.73105858],\n",
      "       [0.88079708, 0.95257413],\n",
      "       [0.98201379, 0.99330715],\n",
      "       [0.99752738, 0.99908895]]), array([[0, 1],\n",
      "       [2, 3],\n",
      "       [4, 5],\n",
      "       [6, 7]]))\n"
     ]
    }
   ],
   "source": [
    "Z = np.arange(8).reshape(4,2)\n",
    "print (\"sigmoid(Z) = \" + str(sigmoid(Z)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### relu\n",
    "This is one of the activation functions. It takes the cumulative input to the layer, matrix **Z** as the input. Upon application of the **`relu`** function, matrix **H** which is the output matrix is calculated. Also, **Z** is stored as **relu_memory** which will be later used in backpropagation. You use _[np.maximum()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.maximum.html)_ here in the following way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def relu(Z):\n",
    "    # Z is numpy array of shape (n, m) where n is number of neurons in the layer and m is the number of samples \n",
    "    # relu_memory is stored as it is used later on in backpropagation\n",
    "    \n",
    "    H = np.maximum(0,Z)\n",
    "    \n",
    "    assert(H.shape == Z.shape)\n",
    "    \n",
    "    relu_memory = Z \n",
    "    return H, relu_memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "relu(Z) = (array([[ 1,  3],\n",
      "       [ 0,  0],\n",
      "       [ 0,  7],\n",
      "       [ 9, 18]]), array([[ 1,  3],\n",
      "       [-1, -4],\n",
      "       [-5,  7],\n",
      "       [ 9, 18]]))\n"
     ]
    }
   ],
   "source": [
    "Z = np.array([1, 3, -1, -4, -5, 7, 9, 18]).reshape(4,2)\n",
    "print (\"relu(Z) = \" + str(relu(Z)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### softmax\n",
    "This is the activation of the last layer. It takes the cumulative input to the layer, matrix **Z** as the input. Upon application of the **`softmax`** function, the output matrix **H** is calculated. Also, **Z** is stored as **softmax_memory** which will be later used in backpropagation. You use _[np.exp()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.exp.html)_ and _[np.sum()](https://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.sum.html)_ here in the following way. The exponential gets applied to all the elements of Z."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax(Z):\n",
    "    # Z is numpy array of shape (n, m) where n is number of neurons in the layer and m is the number of samples \n",
    "    # softmax_memory is stored as it is used later on in backpropagation\n",
    "   \n",
    "    Z_exp = np.exp(Z)\n",
    "\n",
    "    Z_sum = np.sum(Z_exp,axis = 0, keepdims = True)\n",
    "    \n",
    "    H = Z_exp/Z_sum  #normalising step\n",
    "    softmax_memory = Z\n",
    "    \n",
    "    return H, softmax_memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "Z = np.array([[11,19,10], [12, 21, 23]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[2.68941421e-01 1.19202922e-01 2.26032430e-06]\n",
      " [7.31058579e-01 8.80797078e-01 9.99997740e-01]]\n",
      "[[11 19 10]\n",
      " [12 21 23]]\n"
     ]
    }
   ],
   "source": [
    "#Z = np.array(np.arange(30)).reshape(10,3)\n",
    "H, softmax_memory = softmax(Z)\n",
    "print(H)\n",
    "print(softmax_memory)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### initialize_parameters\n",
    "Let's now create a function **`initialize_parameters`** which initializes the weights and biases of the various layers. One way to initialise is to set all the parameters to 0. This is not a considered a good strategy as all the neurons will behave the same way and it'll defeat the purpose of deep networks. Hence, we initialize the weights randomly to very small values but not zeros. The biases are initialized to 0. Note that the **`initialize_parameters`** function initializes the parameters for all the layers in one `for` loop. \n",
    "\n",
    "The inputs to this function is a list named `dimensions`. The length of the list is the number layers in the network + 1 (the plus one is for the input layer, rest are hidden + output). The first element of this list is the dimensionality or length of the input (784 for the MNIST dataset). The rest of the list contains the number of neurons in the corresponding (hidden and output) layers.\n",
    "\n",
    "For example `dimensions = [784, 3, 7, 10]` specifies a network for the MNIST dataset with two hidden layers and a 10-dimensional softmax output.\n",
    "\n",
    "Also, notice that the parameters are returned in a dictionary. This will help you in implementing the feedforward through the layer and the backprop throught the layer at once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def initialize_parameters(dimensions):\n",
    "\n",
    "    # dimensions is a list containing the number of neuron in each layer in the network\n",
    "    # It returns parameters which is a python dictionary containing the parameters \"W1\", \"b1\", ..., \"WL\", \"bL\":\n",
    "\n",
    "    np.random.seed(2)\n",
    "    parameters = {}\n",
    "    L = len(dimensions)            # number of layers in the network + 1\n",
    "\n",
    "    for l in range(1, L): \n",
    "        parameters['W' + str(l)] = np.random.randn(dimensions[l], dimensions[l-1]) * 0.1\n",
    "        parameters['b' + str(l)] = np.zeros((dimensions[l], 1)) \n",
    "        \n",
    "        assert(parameters['W' + str(l)].shape == (dimensions[l], dimensions[l-1]))\n",
    "        assert(parameters['b' + str(l)].shape == (dimensions[l], 1))\n",
    "\n",
    "        \n",
    "    return parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "W1 = [[-0.04167578 -0.00562668 -0.21361961 ... -0.06168445  0.03213358\n",
      "  -0.09464469]\n",
      " [-0.05301394 -0.1259207   0.16775441 ... -0.03284246 -0.05623108\n",
      "   0.01179136]\n",
      " [ 0.07386378 -0.15872956  0.01532001 ... -0.08428557  0.10040469\n",
      "   0.00545832]]\n",
      "b1 = [[0.]\n",
      " [0.]\n",
      " [0.]]\n",
      "W2 = [[ 0.06650944 -0.19626047  0.2112715 ]\n",
      " [-0.28074571 -0.13967752  0.02641189]\n",
      " [ 0.10925169  0.06646016  0.08565535]\n",
      " [-0.11058228  0.03715795  0.13440124]\n",
      " [-0.16421272 -0.1153127   0.02013163]\n",
      " [ 0.13985659  0.07228733 -0.10717236]\n",
      " [-0.05673344 -0.03663499 -0.15460347]]\n",
      "b2 = [[0.]\n",
      " [0.]\n",
      " [0.]\n",
      " [0.]\n",
      " [0.]\n",
      " [0.]\n",
      " [0.]]\n"
     ]
    }
   ],
   "source": [
    "dimensions  = [784, 3,7,10]\n",
    "parameters = initialize_parameters(dimensions)\n",
    "print(\"W1 = \" + str(parameters[\"W1\"]))\n",
    "print(\"b1 = \" + str(parameters[\"b1\"]))\n",
    "print(\"W2 = \" + str(parameters[\"W2\"]))\n",
    "print(\"b2 = \" + str(parameters[\"b2\"]))\n",
    "# print(\"W3 = \" + str(parameters[\"W3\"]))\n",
    "# print(\"b3 = \" + str(parameters[\"b3\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### layer_forward\n",
    "\n",
    "The function **`layer_forward`** implements the forward propagation for a certain layer 'l'. It calculates the cumulative input into the layer **Z** and uses it to calculate the output of the layer **H**. It takes **H_prev, W, b and the activation function** as inputs and stores the **linear_memory, activation_memory** in the variable **memory** which will be used later in backpropagation. \n",
    "\n",
    "<br> You have to first calculate the **Z**(using the forward propagation equation), **linear_memory**(H_prev, W, b) and then calculate **H, activation_memory**(Z) by applying activation functions - **`sigmoid`**, **`relu`** and **`softmax`** on **Z**.\n",
    "\n",
    "<br> Note that $$H^{L-1}$$ is referred here as H_prev. You might want to use _[np.dot()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.dot.html)_ to carry out the matrix multiplication."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Graded\n",
    "\n",
    "def layer_forward(H_prev, W, b, activation = 'relu'):\n",
    "\n",
    "    # H_prev is of shape (size of previous layer, number of examples)\n",
    "    # W is weights matrix of shape (size of current layer, size of previous layer)\n",
    "    # b is bias vector of shape (size of the current layer, 1)\n",
    "    # activation is the activation to be used for forward propagation : \"softmax\", \"relu\", \"sigmoid\"\n",
    "\n",
    "    # H is the output of the activation function \n",
    "    # memory is a python dictionary containing \"linear_memory\" and \"activation_memory\"\n",
    "    \n",
    "    if activation == \"sigmoid\":\n",
    "        Z = np.dot(W,H_prev)+b\n",
    "        linear_memory = (H_prev, W, b)\n",
    "        H, activation_memory = sigmoid(Z)\n",
    " \n",
    "    elif activation == \"softmax\":\n",
    "        Z = np.dot(W,H_prev)+b\n",
    "        linear_memory = (H_prev, W, b)\n",
    "        H, activation_memory = softmax(Z)\n",
    "    \n",
    "    elif activation == \"relu\":\n",
    "        Z = np.dot(W,H_prev)+b\n",
    "        linear_memory = (H_prev, W, b)\n",
    "        H, activation_memory = relu(Z)\n",
    "        \n",
    "    assert (H.shape == (W.shape[0], H_prev.shape[1]))\n",
    "    memory = (linear_memory, activation_memory)\n",
    "\n",
    "    return H, memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.        , 1.        , 1.        , 1.        , 1.        ],\n",
       "       [0.99908895, 0.99330715, 0.99999969, 1.        , 0.99987661],\n",
       "       [0.73105858, 0.5       , 0.99330715, 0.9999546 , 0.88079708]])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# verify\n",
    "# l-1 has two neurons, l has three, m = 5\n",
    "# H_prev is (l-1, m)\n",
    "# W is (l, l-1)\n",
    "# b is (l, 1)\n",
    "# H should be (l, m)\n",
    "H_prev = np.array([[1,0, 5, 10, 2], [2, 5, 3, 10, 2]])\n",
    "W_sample = np.array([[10, 5], [2, 0], [1, 0]])\n",
    "b_sample = np.array([10, 5, 0]).reshape((3, 1))\n",
    "\n",
    "H = layer_forward(H_prev, W_sample, b_sample, activation=\"sigmoid\")[0]\n",
    "H"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should get:<br>\n",
    "    array([[1.        , 1.        , 1.        , 1.        , 1.        ],<br>\n",
    "      [0.99908895, 0.99330715, 0.99999969, 1.        , 0.99987661],<br>\n",
    "       [0.73105858, 0.5       , 0.99330715, 0.9999546 , 0.88079708]])\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### L_layer_forward\n",
    "**`L_layer_forward`** performs one forward pass through the whole network for all the training samples (note that we are feeding all training examples in one single batch). Use the **`layer_forward`** you have created above here to perform the feedforward for layers 1 to 'L-1' in the for loop with the activation **`relu`**. The last layer having a different activation **`softmax`** is calculated outside the loop. Notice that the **memory** is appended to **memories** for all the layers. These will be used in the backward order during backpropagation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Graded\n",
    "\n",
    "def L_layer_forward(X, parameters):\n",
    "\n",
    "    # X is input data of shape (input size, number of examples)\n",
    "    # parameters is output of initialize_parameters()\n",
    "    \n",
    "    # HL is the last layer's post-activation value\n",
    "    # memories is the list of memory containing (for a relu activation, for example):\n",
    "    # - every memory of relu forward (there are L-1 of them, indexed from 1 to L-1), \n",
    "    # - the memory of softmax forward (there is one, indexed L) \n",
    "\n",
    "    memories = []\n",
    "    H = X\n",
    "    L = len(parameters) // 2                  # number of layers in the neural network\n",
    "    \n",
    "    # Implement relu layer (L-1) times as the Lth layer is the softmax layer\n",
    "    for l in range(1, L):\n",
    "        H_prev =H\n",
    "        \n",
    "        H, memory = layer_forward(H_prev,parameters['W'+str(l)],parameters['b'+str(l)],activation='relu')\n",
    "        \n",
    "        memories.append(memory)\n",
    "    \n",
    "    # Implement the final softmax layer\n",
    "    # HL here is the final prediction P as specified in the lectures\n",
    "    HL, memory = layer_forward(H,parameters['W'+str(L)],parameters['b'+str(L)],activation='softmax')\n",
    "    \n",
    "    memories.append(memory)\n",
    "\n",
    "    assert(HL.shape == (10, X.shape[1]))\n",
    "            \n",
    "    return HL, memories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(784, 10)\n",
      "[[0.10106734 0.10045152 0.09927757 0.10216656 0.1       ]\n",
      " [0.10567625 0.10230873 0.10170271 0.11250099 0.1       ]\n",
      " [0.09824287 0.0992886  0.09967128 0.09609693 0.1       ]\n",
      " [0.10028288 0.10013048 0.09998149 0.10046076 0.1       ]\n",
      " [0.09883601 0.09953443 0.09931419 0.097355   0.1       ]\n",
      " [0.10668575 0.10270912 0.10180736 0.11483609 0.1       ]\n",
      " [0.09832513 0.09932275 0.09954792 0.09627089 0.1       ]\n",
      " [0.09747092 0.09896735 0.0995387  0.09447277 0.1       ]\n",
      " [0.09489069 0.09788255 0.09929998 0.08915178 0.1       ]\n",
      " [0.09852217 0.09940447 0.09985881 0.09668824 0.1       ]]\n"
     ]
    }
   ],
   "source": [
    "# verify\n",
    "# X is (784, 10)\n",
    "# parameters is a dict\n",
    "# HL should be (10, 10)\n",
    "x_sample = train_set_x[:, 10:20]\n",
    "print(x_sample.shape)\n",
    "HL = L_layer_forward(x_sample, parameters=parameters)[0]\n",
    "print(HL[:, :5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should get:\n",
    "\n",
    "(784, 10)<br>\n",
    "[[0.10106734 0.10045152 0.09927757 0.10216656 0.1       ]<br>\n",
    " [0.10567625 0.10230873 0.10170271 0.11250099 0.1       ]<br>\n",
    " [0.09824287 0.0992886  0.09967128 0.09609693 0.1       ]<br>\n",
    " [0.10028288 0.10013048 0.09998149 0.10046076 0.1       ]<br>\n",
    " [0.09883601 0.09953443 0.09931419 0.097355   0.1       ]<br>\n",
    " [0.10668575 0.10270912 0.10180736 0.11483609 0.1       ]<br>\n",
    " [0.09832513 0.09932275 0.09954792 0.09627089 0.1       ]<br>\n",
    " [0.09747092 0.09896735 0.0995387  0.09447277 0.1       ]<br>\n",
    " [0.09489069 0.09788255 0.09929998 0.08915178 0.1       ]<br>\n",
    " [0.09852217 0.09940447 0.09985881 0.09668824 0.1       ]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loss\n",
    "\n",
    "### compute_loss\n",
    "The next step is to compute the loss function after every forward pass to keep checking whether it is decreasing with training.<br> **`compute_loss`** here calculates the cross-entropy loss. You may want to use _[np.log()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.log.html)_, _[np.sum()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.log.html)_, _[np.multiply()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.multiply.html)_ here. Do not forget that it is the average loss across all the data points in the batch. It takes the output of the last layer **HL** and the ground truth label **Y** as input and returns the **loss**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Graded\n",
    "\n",
    "def compute_loss(HL, Y):\n",
    "\n",
    "\n",
    "    # HL is probability matrix of shape (10, number of examples)\n",
    "    # Y is true \"label\" vector shape (10, number of examples)\n",
    "\n",
    "    # loss is the cross-entropy loss\n",
    "\n",
    "    m = Y.shape[1]\n",
    "\n",
    "    loss = -1.0/HL.shape[1]*(np.sum(np.multiply(Y,np.log(HL))))\n",
    "    \n",
    "    loss = np.squeeze(loss)      # To make sure that the loss's shape is what we expect (e.g. this turns [[17]] into 17).\n",
    "    assert(loss.shape == ())\n",
    "    \n",
    "    return loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.4359949  0.02592623 0.54966248 0.43532239 0.4203678 ]\n",
      " [0.33033482 0.20464863 0.61927097 0.29965467 0.26682728]\n",
      " [0.62113383 0.52914209 0.13457995 0.51357812 0.18443987]\n",
      " [0.78533515 0.85397529 0.49423684 0.84656149 0.07964548]\n",
      " [0.50524609 0.0652865  0.42812233 0.09653092 0.12715997]\n",
      " [0.59674531 0.226012   0.10694568 0.22030621 0.34982629]\n",
      " [0.46778748 0.20174323 0.64040673 0.48306984 0.50523672]\n",
      " [0.38689265 0.79363745 0.58000418 0.1622986  0.70075235]\n",
      " [0.96455108 0.50000836 0.88952006 0.34161365 0.56714413]\n",
      " [0.42754596 0.43674726 0.77655918 0.53560417 0.95374223]]\n",
      "[[0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 1.]\n",
      " [0. 0. 0. 0. 0.]\n",
      " [1. 0. 1. 0. 0.]\n",
      " [0. 0. 0. 0. 0.]\n",
      " [0. 1. 0. 0. 0.]\n",
      " [0. 0. 0. 1. 0.]\n",
      " [0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0.]]\n",
      "0.8964600261334037\n"
     ]
    }
   ],
   "source": [
    "# sample\n",
    "# HL is (10, 5), Y is (10, 5)\n",
    "np.random.seed(2)\n",
    "HL_sample = np.random.rand(10,5)\n",
    "Y_sample = train_set_y[:, 10:15]\n",
    "print(HL_sample)\n",
    "print(Y_sample)\n",
    "\n",
    "print(compute_loss(HL_sample, Y_sample))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should get:<br>\n",
    "    \n",
    "[[0.4359949  0.02592623 0.54966248 0.43532239 0.4203678 ]<br>\n",
    " [0.33033482 0.20464863 0.61927097 0.29965467 0.26682728]<br>\n",
    " [0.62113383 0.52914209 0.13457995 0.51357812 0.18443987]<br>\n",
    " [0.78533515 0.85397529 0.49423684 0.84656149 0.07964548]<br>\n",
    " [0.50524609 0.0652865  0.42812233 0.09653092 0.12715997]<br>\n",
    " [0.59674531 0.226012   0.10694568 0.22030621 0.34982629]<br>\n",
    " [0.46778748 0.20174323 0.64040673 0.48306984 0.50523672]<br>\n",
    " [0.38689265 0.79363745 0.58000418 0.1622986  0.70075235]<br>\n",
    " [0.96455108 0.50000836 0.88952006 0.34161365 0.56714413]<br>\n",
    " [0.42754596 0.43674726 0.77655918 0.53560417 0.95374223]]<br>\n",
    "[[0. 0. 0. 0. 0.]<br>\n",
    " [0. 0. 0. 0. 1.]<br>\n",
    " [0. 0. 0. 0. 0.]<br>\n",
    " [1. 0. 1. 0. 0.]<br>\n",
    " [0. 0. 0. 0. 0.]<br>\n",
    " [0. 1. 0. 0. 0.]<br>\n",
    " [0. 0. 0. 1. 0.]<br>\n",
    " [0. 0. 0. 0. 0.]<br>\n",
    " [0. 0. 0. 0. 0.]<br>\n",
    " [0. 0. 0. 0. 0.]]<br>\n",
    "0.8964600261334037"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Backpropagation\n",
    "Let's now get to the next step - backpropagation. Let's start with sigmoid_backward.\n",
    "\n",
    "### sigmoid-backward\n",
    "You might remember that we had created **`sigmoid`** function that calculated the activation for forward propagation. Now, we need the activation backward, which helps in calculating **dZ** from **dH**. Notice that it takes input **dH** and **sigmoid_memory** as input. **sigmoid_memory** is the **Z** which we had calculated during forward propagation. You use _[np.exp()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.exp.html)_ here the following way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid_backward(dH, sigmoid_memory):\n",
    "    \n",
    "    # Implement the backpropagation of a sigmoid function\n",
    "    # dH is gradient of the sigmoid activated activation of shape same as H or Z in the same layer    \n",
    "    # sigmoid_memory is the memory stored in the sigmoid(Z) calculation\n",
    "    \n",
    "    Z = sigmoid_memory\n",
    "    \n",
    "    H = 1/(1+np.exp(-Z))\n",
    "    dZ = dH * H * (1-H)\n",
    "    \n",
    "    assert (dZ.shape == Z.shape)\n",
    "    \n",
    "    return dZ"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### relu-backward\n",
    "You might remember that we had created **`relu`** function that calculated the activation for forward propagation. Now, we need the activation backward, which helps in calculating **dZ** from **dH**. Notice that it takes input **dH** and **relu_memory** as input. **relu_memory** is the **Z** which we calculated uring forward propagation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def relu_backward(dH, relu_memory):\n",
    "    \n",
    "    # Implement the backpropagation of a relu function\n",
    "    # dH is gradient of the relu activated activation of shape same as H or Z in the same layer    \n",
    "    # relu_memory is the memory stored in the sigmoid(Z) calculation\n",
    "    \n",
    "    Z = relu_memory\n",
    "    dZ = np.array(dH, copy=True) # dZ will be the same as dA wherever the elements of A weren't 0\n",
    "    \n",
    "    dZ[Z <= 0] = 0\n",
    "    \n",
    "    assert (dZ.shape == Z.shape)\n",
    "    \n",
    "    return dZ"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### layer_backward\n",
    "\n",
    "**`layer_backward`** is a complimentary function of **`layer_forward`**. Like **`layer_forward`** calculates **H** using **W**, **H_prev** and **b**, **`layer_backward`** uses **dH** to calculate **dW**, **dH_prev** and **db**. You have already studied the formulae in backpropogation. To calculate **dZ**, use the **`sigmoid_backward`** and **`relu_backward`** function. You might need to use _[np.dot()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.dot.html)_, _[np.sum()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.sum.html)_ for the rest. Remember to choose the axis correctly in db. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Graded\n",
    "\n",
    "def layer_backward(dH, memory, activation = 'relu'):\n",
    "    \n",
    "    # takes dH and the memory calculated in layer_forward and activation as input to calculate the dH_prev, dW, db\n",
    "    # performs the backprop depending upon the activation function\n",
    "    \n",
    "\n",
    "    linear_memory, activation_memory = memory\n",
    "    \n",
    "    if activation == \"relu\":\n",
    "        dZ = relu_backward(dH,activation_memory)\n",
    "        H_prev, W, b = linear_memory\n",
    "        m = H_prev.shape[1]\n",
    "        dW = (1.0/m)*(np.dot(dZ,H_prev.T))\n",
    "        db = (1.0/m)*(np.sum(dZ,keepdims=True,axis=1))\n",
    "        dH_prev = np.dot(W.T,dZ)\n",
    "        \n",
    "    elif activation == \"sigmoid\":\n",
    "        dZ = sigmoid_backward(dH,activation_memory)\n",
    "        H_prev, W, b = linear_memory\n",
    "        m = H_prev.shape[1]\n",
    "        dW = (1.0/m)*(np.dot(dZ,H_prev.T))\n",
    "        db = (1.0/m)*(np.sum(dZ,keepdims=True,axis=1))\n",
    "        dH_prev = np.dot(W.T,dZ)\n",
    "    \n",
    "    return dH_prev, dW, db"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dH_prev is \n",
      " [[5.6417525  0.66855959 6.86974666 5.46611139 4.92177244]\n",
      " [2.17997451 0.12963116 2.74831239 2.17661196 2.10183901]]\n",
      "dW is \n",
      " [[1.67565336 1.56891359]\n",
      " [1.39137819 1.4143854 ]\n",
      " [1.3597389  1.43013369]]\n",
      "db is \n",
      " [[0.37345476]\n",
      " [0.34414727]\n",
      " [0.29074635]]\n"
     ]
    }
   ],
   "source": [
    "# verify\n",
    "# l-1 has two neurons, l has three, m = 5\n",
    "# H_prev is (l-1, m)\n",
    "# W is (l, l-1)\n",
    "# b is (l, 1)\n",
    "# H should be (l, m)\n",
    "H_prev = np.array([[1,0, 5, 10, 2], [2, 5, 3, 10, 2]])\n",
    "W_sample = np.array([[10, 5], [2, 0], [1, 0]])\n",
    "b_sample = np.array([10, 5, 0]).reshape((3, 1))\n",
    "\n",
    "H, memory = layer_forward(H_prev, W_sample, b_sample, activation=\"relu\")\n",
    "np.random.seed(2)\n",
    "dH = np.random.rand(3,5)\n",
    "dH_prev, dW, db = layer_backward(dH, memory, activation = 'relu')\n",
    "print('dH_prev is \\n' , dH_prev)\n",
    "print('dW is \\n' ,dW)\n",
    "print('db is \\n', db)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should get:<br>\n",
    "dH_prev is <br>\n",
    " [[5.6417525  0.66855959 6.86974666 5.46611139 4.92177244]<br>\n",
    " [2.17997451 0.12963116 2.74831239 2.17661196 2.10183901]]<br>\n",
    "dW is <br>\n",
    " [[1.67565336 1.56891359]<br>\n",
    " [1.39137819 1.4143854 ]<br>\n",
    " [1.3597389  1.43013369]]<br>\n",
    "db is <br>\n",
    " [[0.37345476]<br>\n",
    " [0.34414727]<br>\n",
    " [0.29074635]]<br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### L_layer_backward\n",
    "\n",
    "**`L_layer_backward`** performs backpropagation for the whole network. Recall that the backpropagation for the last layer, i.e. the softmax layer, is different from the rest, hence it is outside the reversed `for` loop. You need to use the function **`layer_backward`** here in the loop with the activation function as **`relu`**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Graded\n",
    "\n",
    "def L_layer_backward(HL, Y, memories):\n",
    "    \n",
    "    # Takes the predicted value HL and the true target value Y and the \n",
    "    # memories calculated by L_layer_forward as input\n",
    "    \n",
    "    # returns the gradients calulated for all the layers as a dict\n",
    "\n",
    "    gradients = {}\n",
    "    L = len(memories) # the number of layers\n",
    "    m = HL.shape[1]\n",
    "    Y = Y.reshape(HL.shape) # after this line, Y is the same shape as AL\n",
    "    \n",
    "    # Perform the backprop for the last layer that is the softmax layer\n",
    "    current_memory = memories[-1]\n",
    "    linear_memory, activation_memory = current_memory\n",
    "    dZ = HL - Y\n",
    "    H_prev, W, b = linear_memory\n",
    "    # Use the expressions you have used in 'layer_backward'\n",
    "    gradients[\"dH\" + str(L-1)] = np.dot(W.T,dZ)\n",
    "    gradients[\"dW\" + str(L)] = (1.0/m)*(np.dot(dZ,H_prev.T))\n",
    "    gradients[\"db\" + str(L)] = (1.0/m)*(np.sum(dZ,keepdims=True,axis=1))\n",
    "    \n",
    "    # Perform the backpropagation l-1 times\n",
    "    for l in reversed(range(L-1)):\n",
    "        # Lth layer gradients: \"gradients[\"dH\" + str(l + 1)] \", gradients[\"dW\" + str(l + 2)] , gradients[\"db\" + str(l + 2)]\n",
    "        current_memory = memories[l]\n",
    "        \n",
    "        dH_prev_temp, dW_temp, db_temp = layer_backward(gradients[\"dH\" + str(l + 1)],current_memory,activation='relu')\n",
    "        gradients[\"dH\" + str(l)] = dH_prev_temp\n",
    "        gradients[\"dW\" + str(l + 1)] =  dW_temp\n",
    "        gradients[\"db\" + str(l + 1)] = db_temp\n",
    "\n",
    "\n",
    "    return gradients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dW3 is \n",
      " [[ 0.02003701  0.0019043   0.01011729  0.0145757   0.00146444  0.00059863\n",
      "   0.        ]\n",
      " [ 0.02154547  0.00203519  0.01085648  0.01567075  0.00156469  0.00060533\n",
      "   0.        ]\n",
      " [-0.01718407 -0.00273711 -0.00499101 -0.00912135 -0.00207365  0.00059996\n",
      "   0.        ]\n",
      " [-0.01141498 -0.00158622 -0.00607049 -0.00924709 -0.00119619  0.00060381\n",
      "   0.        ]\n",
      " [ 0.01943173  0.0018421   0.00984543  0.01416368  0.00141676  0.00059682\n",
      "   0.        ]\n",
      " [ 0.01045447  0.00063974  0.00637621  0.00863306  0.00050118  0.00060441\n",
      "   0.        ]\n",
      " [-0.06338911 -0.00747251 -0.0242169  -0.03835708 -0.00581131  0.0006034\n",
      "   0.        ]\n",
      " [ 0.01911373  0.001805    0.00703101  0.0120636   0.00138836 -0.00140535\n",
      "   0.        ]\n",
      " [-0.01801603  0.0017357  -0.01489228 -0.02026076  0.00133528  0.00060264\n",
      "   0.        ]\n",
      " [ 0.0194218   0.00183381  0.00594427  0.01187949  0.00141043 -0.00340965\n",
      "   0.        ]]\n",
      "db3 is \n",
      " [[ 0.10031756]\n",
      " [ 0.00460183]\n",
      " [-0.00142942]\n",
      " [-0.0997827 ]\n",
      " [ 0.09872663]\n",
      " [ 0.00536378]\n",
      " [-0.10124784]\n",
      " [-0.00191121]\n",
      " [-0.00359044]\n",
      " [-0.00104818]]\n",
      "dW2 is \n",
      " [[ 4.94428956e-05  1.13215514e-02  5.44180380e-02]\n",
      " [-4.81267081e-05 -2.96999448e-05 -1.81899582e-02]\n",
      " [ 5.63424333e-05  4.77190073e-03  4.04810232e-02]\n",
      " [ 1.49767478e-04 -1.89780927e-03 -7.91231369e-03]\n",
      " [ 1.97866094e-04  1.22107085e-04  2.64140566e-02]\n",
      " [ 0.00000000e+00 -3.75805770e-04  1.63906102e-05]\n",
      " [ 0.00000000e+00  0.00000000e+00  0.00000000e+00]]\n",
      "db2 is \n",
      " [[ 0.013979  ]\n",
      " [-0.01329383]\n",
      " [ 0.01275707]\n",
      " [-0.01052957]\n",
      " [ 0.03179224]\n",
      " [-0.00039877]\n",
      " [ 0.        ]]\n"
     ]
    }
   ],
   "source": [
    "# verify\n",
    "# X is (784, 10)\n",
    "# parameters is a dict\n",
    "# HL should be (10, 10)\n",
    "x_sample = train_set_x[:, 10:20]\n",
    "y_sample = train_set_y[:, 10:20]\n",
    "\n",
    "HL, memories = L_layer_forward(x_sample, parameters=parameters)\n",
    "gradients  = L_layer_backward(HL, y_sample, memories)\n",
    "print('dW3 is \\n', gradients['dW3'])\n",
    "print('db3 is \\n', gradients['db3'])\n",
    "print('dW2 is \\n', gradients['dW2'])\n",
    "print('db2 is \\n', gradients['db2'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should get:<br>\n",
    "\n",
    "dW3 is <br>\n",
    " [[ 0.02003701  0.0019043   0.01011729  0.0145757   0.00146444  0.00059863  0.        ]<br>\n",
    " [ 0.02154547  0.00203519  0.01085648  0.01567075  0.00156469  0.00060533   0.        ]<br>\n",
    " [-0.01718407 -0.00273711 -0.00499101 -0.00912135 -0.00207365  0.00059996   0.        ]<br>\n",
    " [-0.01141498 -0.00158622 -0.00607049 -0.00924709 -0.00119619  0.00060381   0.        ]<br>\n",
    " [ 0.01943173  0.0018421   0.00984543  0.01416368  0.00141676  0.00059682   0.        ]<br>\n",
    " [ 0.01045447  0.00063974  0.00637621  0.00863306  0.00050118  0.00060441   0.        ]<br>\n",
    " [-0.06338911 -0.00747251 -0.0242169  -0.03835708 -0.00581131  0.0006034   0.        ]<br>\n",
    " [ 0.01911373  0.001805    0.00703101  0.0120636   0.00138836 -0.00140535   0.        ]<br>\n",
    " [-0.01801603  0.0017357  -0.01489228 -0.02026076  0.00133528  0.00060264   0.        ]<br>\n",
    " [ 0.0194218   0.00183381  0.00594427  0.01187949  0.00141043 -0.00340965    0.        ]]<br>\n",
    "db3 is <br>\n",
    " [[ 0.10031756]<br>\n",
    " [ 0.00460183]<br>\n",
    " [-0.00142942]<br>\n",
    " [-0.0997827 ]<br>\n",
    " [ 0.09872663]<br>\n",
    " [ 0.00536378]<br>\n",
    " [-0.10124784]<br>\n",
    " [-0.00191121]<br>\n",
    " [-0.00359044]<br>\n",
    " [-0.00104818]]<br>\n",
    "dW2 is <br>\n",
    " [[ 4.94428956e-05  1.13215514e-02  5.44180380e-02]<br>\n",
    " [-4.81267081e-05 -2.96999448e-05 -1.81899582e-02]<br>\n",
    " [ 5.63424333e-05  4.77190073e-03  4.04810232e-02]<br>\n",
    " [ 1.49767478e-04 -1.89780927e-03 -7.91231369e-03]<br>\n",
    " [ 1.97866094e-04  1.22107085e-04  2.64140566e-02]<br>\n",
    " [ 0.00000000e+00 -3.75805770e-04  1.63906102e-05]<br>\n",
    " [ 0.00000000e+00  0.00000000e+00  0.00000000e+00]]<br>\n",
    "db2 is <br>\n",
    " [[ 0.013979  ]<br>\n",
    " [-0.01329383]<br>\n",
    " [ 0.01275707]<br>\n",
    " [-0.01052957]<br>\n",
    " [ 0.03179224]<br>\n",
    " [-0.00039877]<br>\n",
    " [ 0.        ]]<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parameter Updates\n",
    "\n",
    "Now that we have calculated the gradients. let's do the last step which is updating the weights and biases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Graded\n",
    "\n",
    "def update_parameters(parameters, gradients, learning_rate):\n",
    "\n",
    "    # parameters is the python dictionary containing the parameters W and b for all the layers\n",
    "    # gradients is the python dictionary containing your gradients, output of L_model_backward\n",
    "    \n",
    "    # returns updated weights after applying the gradient descent update\n",
    "\n",
    "    \n",
    "    L = len(parameters) // 2 # number of layers in the neural network\n",
    "\n",
    "    for l in range(L):\n",
    "        parameters[\"W\" + str(l+1)] = parameters[\"W\" + str(l+1)] - learning_rate*(gradients[\"dW\" + str(l+1)])\n",
    "        parameters[\"b\" + str(l+1)] = parameters[\"b\" + str(l+1)] - learning_rate*(gradients[\"db\" + str(l+1)])\n",
    "\n",
    "        \n",
    "    return parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having defined the bits and pieces of the feedforward and the backpropagation, let's now combine all that to form a model. The list `dimensions` has the number of neurons in each layer specified in it. For a neural network with 1 hidden layer with 45 neurons, you would specify the dimensions as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "dimensions = [784, 45, 10] #  three-layer model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model\n",
    "\n",
    "### L_layer_model\n",
    "\n",
    "This is a composite function which takes the training data as input **X**, ground truth label **Y**, the **dimensions** as stated above, **learning_rate**, the number of iterations **num_iterations** and if you want to print the loss, **print_loss**. You need to use the final functions we have written for feedforward, computing the loss, backpropagation and updating the parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Graded\n",
    "\n",
    "def L_layer_model(X, Y, dimensions, learning_rate = 0.0075, num_iterations = 3000, print_loss=False):\n",
    "    \n",
    "    # X and Y are the input training datasets\n",
    "    # learning_rate, num_iterations are gradient descent optimization parameters\n",
    "    # returns updated parameters\n",
    "\n",
    "    np.random.seed(2)\n",
    "    losses = []                         # keep track of loss\n",
    "    \n",
    "    # Parameters initialization\n",
    "    parameters = initialize_parameters(dimensions)\n",
    " \n",
    "    for i in range(0, num_iterations):\n",
    "\n",
    "        # Forward propagation\n",
    "        HL, memories = L_layer_forward(X, parameters)\n",
    "        \n",
    "        # Compute loss\n",
    "        loss = compute_loss(HL, Y)\n",
    "    \n",
    "        # Backward propagation\n",
    "        gradients = L_layer_backward(HL, Y, memories)\n",
    " \n",
    "        # Update parameters.\n",
    "        parameters = update_parameters(parameters, gradients, learning_rate)\n",
    "                \n",
    "        # Printing the loss every 100 training example\n",
    "        if print_loss and i % 100 == 0:\n",
    "            print (\"Loss after iteration %i: %f\" %(i, loss))\n",
    "            losses.append(loss)\n",
    "            \n",
    "    # plotting the loss\n",
    "    plt.plot(np.squeeze(losses))\n",
    "    plt.ylabel('loss')\n",
    "    plt.xlabel('iterations (per tens)')\n",
    "    plt.title(\"Learning rate =\" + str(learning_rate))\n",
    "    plt.show()\n",
    "    \n",
    "    return parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since, it'll take a lot of time to train the model on 50,000 data points, we take a subset of 5,000 images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(784, 5000)"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set_x_new = train_set_x[:,0:5000]\n",
    "train_set_y_new = train_set_y[:,0:5000]\n",
    "train_set_x_new.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's call the function L_layer_model on the dataset we have created.This will take 10-20 mins to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss after iteration 0: 2.422624\n",
      "Loss after iteration 100: 2.129232\n",
      "Loss after iteration 200: 1.876095\n",
      "Loss after iteration 300: 1.604213\n",
      "Loss after iteration 400: 1.350205\n",
      "Loss after iteration 500: 1.144823\n",
      "Loss after iteration 600: 0.990554\n",
      "Loss after iteration 700: 0.876603\n",
      "Loss after iteration 800: 0.791154\n",
      "Loss after iteration 900: 0.725441\n",
      "Loss after iteration 1000: 0.673485\n",
      "Loss after iteration 1100: 0.631386\n",
      "Loss after iteration 1200: 0.596598\n",
      "Loss after iteration 1300: 0.567342\n",
      "Loss after iteration 1400: 0.542346\n",
      "Loss after iteration 1500: 0.520746\n",
      "Loss after iteration 1600: 0.501865\n",
      "Loss after iteration 1700: 0.485205\n",
      "Loss after iteration 1800: 0.470368\n",
      "Loss after iteration 1900: 0.457054\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VPW5x/HPNwsJhCQQEvawKSCgoJiyqHVpFcG6tFVbrVutLWL1tnp7W7XtrdZuVrtca2sVl7pb27pU64LWugtiQBAQ2fc1JEBIgEDguX+cExzjBAbI5EyS5/16nVdmzvmdOc+cJPOds/2OzAznnHNuX9KiLsA551zz4IHhnHMuIR4YzjnnEuKB4ZxzLiEeGM455xLigeGccy4hHhiuxZP0gqRLoq7DuebOA8MljaSlkk6Oug4zG2dmD0RdB4Ck1yR9swmWkyXpPkmVktZK+u99tL8mbLc5nC8rZlofSa9K2irpo9jfqaQ7JVXFDDWStsRMf03S9pjp85Lzjl1T8MBwzZqkjKhrqJNKtQA3Av2B3sBJwA8kjY3XUNKpwHXA54E+QD/gpzFNHgPeBzoBPwL+IakIwMwmmFn7uiFs+/d6i7gqps3ARnp/LgIeGC4Skk6XNEPSJknvSBoaM+06SYskbZH0oaQvxUz7uqS3Jf1eUgVwYzjuLUm/kbRR0hJJ42Lm2fOtPoG2fSW9ES7735L+JOnhBt7DiZJWSrpW0lrgL5I6SvqXpLLw9f8lqWfY/hfAZ4E/ht+2/xiOP0zSy5IqJM2T9JVGWMUXAz8zs41mNhe4G/h6A20vAe41szlmthH4WV1bSQOA4cANZrbNzJ4AZgFnx1kfOeH4lNiac43PA8M1OUnDgfuAywm+td4FPBOzG2QRwQdrPsE33YcldYt5iZHAYqAz8IuYcfOAQuAW4F5JaqCEvbV9FJga1nUjcNE+3k5XoIDgm/x4gv+pv4TPewHbgD8CmNmPgDf5+Bv3VeGH7MvhcjsD5wN3SBoSb2GS7ghDNt7wQdimI9AdmBkz60wg7muG4+u37SKpUzhtsZltqTc93mudDZQBb9Qb/ytJG8KgP7GBGlwz4IHhovAt4C4ze9fMdoXHF2qAUQBm9nczW21mu83scWABMCJm/tVmdruZ1ZrZtnDcMjO728x2EXzD7QZ0aWD5cdtK6gV8BviJme0ws7eAZ/bxXnYTfPuuCb+Bl5vZE2a2NfyQ/QVwwl7mPx1YamZ/Cd/PdOAJ4Jx4jc3s22bWoYGhbiutffhzc8ysm4HcBmpoH6ctYfv60/b2WpcAD9onO6i7lmAXVw9gIvCspEMaqMOlOA8MF4XewPdivx0DxQTfipF0cczuqk3A4QRbA3VWxHnNtXUPzGxr+LB9nHZ7a9sdqIgZ19CyYpWZ2fa6J5LaSbpL0jJJlQTftjtISm9g/t7AyHrr4gKCLZcDVRX+zIsZlwdsidO2rn39toTt60+L+1qSigmC8cHY8eGXgi1hoD4AvA2cluD7cCnGA8NFYQXwi3rfjtuZ2WOSehPsb78K6GRmHYDZQOzupWR1sbwGKJDULmZc8T7mqV/L94CBwEgzywOOD8ergfYrgNfrrYv2ZnZFvIXFOSspdpgDEB6HWAMMi5l1GDCngfcwJ07bdWZWHk7rJym33vT6r3Ux8I6ZLW5gGXWMT/4uXTPigeGSLVNSdsyQQRAIEySNVCBH0hfCD6Ucgg+VMgBJlxJsYSSdmS0DSgkOpLeRNBo4Yz9fJpfguMUmSQXADfWmryPYRVPnX8AASRdJygyHz0ga1ECNnzgrqd4Qe1zhQeDH4UH4wwh2A97fQM0PApdJGhwe//hxXVszmw/MAG4If39fAoYS7DaLdXH915fUQdKpdb93SRcQBOikBupwKc4DwyXb8wQfoHXDjWZWSvAB9kdgI7CQ8KwcM/sQ+C0wmeDD9QiC3RhN5QJgNFAO/Bx4nOD4SqL+D2gLbACmAC/Wm34bcE54BtUfwuMcY4DzgNUEu8t+DWRxcG4gOHlgGfA6cKuZvQggqVe4RdILIBx/C/Bq2H4Znwy684ASgt/VzcA5ZlZWNzEM1p58+nTaTIJ1WEawPv4L+KKZ+bUYzZT8BkrONUzS48BHZlZ/S8G5Vse3MJyLEe4OOkRSmoIL3c4Cno66LudSQSpdmepcKugKPElwHcZK4Aozez/akpxLDb5LyjnnXEKStktKUrGCDsvmSpoj6btx2pyooLOzGeHwk5hpY8NuEhZKui5ZdTrnnEtMMndJ1QLfM7Pp4emS0yS9HJ4FE+tNMzs9dkR4kdOfgFMIdgu8J+mZOPN+QmFhofXp06fx3oFzzrVw06ZN22BmRYm0TVpgmNkagouHMLMtkuYSdA+w1w/90AhgYd1FQJL+SnDwca/z9unTh9LS0oOq2znnWhNJyxJt2yRnSUnqAxwFvBtn8mhJMxXc5KbuwqMefLJLhpXhuHivPV5SqaTSsrKyeE2cc841gqQHhqT2BFeFXm1mlfUmTwd6m9kw4HY+Pn0xXtcBcY/Om9lEMysxs5KiooS2qpxzzh2ApAaGpEyCsHjEzJ6sP93MKs2sKnz8PEE3EoUEWxSxffj0JLgK1jnnXESSeZaUgHuBuWb2uwbadK27D4GkEWE95cB7QH8FN7NpQ9A1wb66mXbOOZdEyTxL6liCm8/MkjQjHPdDgpvKYGZ3EvT5f4WkWoJ+hs4L+9KvlXQVQSdl6cB9ZtZQT5vOOeeaQIu6cK+kpMT8LCnnnEucpGlmVpJIW+9LyjnnXEJafWDU1O5i4huLeG9pRdSlOOdcSmv1gWEGf3l7Kb94bi4tafecc841tlYfGNmZ6VxzygBmrNjEC7PX7nsG55xrpVp9YACcPbwnA7vkcuukeezctTvqcpxzLiV5YADpaeLacQNZsqGav05dHnU5zjmXkjwwQicN7MzIvgXc9soCqmpqoy7HOedSjgdGSBLXjTuMDVU7uPuNxVGX45xzKccDI8ZRvTpy2hFdufvNxazfsj3qcpxzLqV4YNTz/VMPY0ftbv7wyoKoS3HOuZTigVFP38Iczh/Ri8emrmBxWVXU5TjnXMrwwIjjO5/vT3ZGGrdOmhd1Kc45lzI8MOIoys3iW8f344XZa5m+fGPU5TjnXErwwGjAtz7bj8L2Wdz8/EfeZYhzzuGB0aCcrAy+e3J/pi6t4JW566MuxznnIueBsRfnfaaYfoU5/PrFj6j1LkOcc61cMm/RWizpVUlzJc2R9N04bS6Q9EE4vCNpWMy0pZJmSZohKZK7ImWmp/H9UweyYH0VT0xfGUUJzjmXMpK5hVELfM/MBgGjgCslDa7XZglwgpkNBX4GTKw3/SQzOzLRu0Elw9jDu3JUrw787uX5bNuxK6oynHMuckkLDDNbY2bTw8dbgLlAj3pt3jGzutOQpgA9k1XPgZLE9eMGsa6yhvveXhJ1Oc45F5kmOYYhqQ9wFPDuXppdBrwQ89yAlyRNkzQ+edXt24i+BZw8qDN3vraIiuodUZbinHORSXpgSGoPPAFcbWaVDbQ5iSAwro0ZfayZDQfGEezOOr6BecdLKpVUWlZW1sjVf+zasYdRvaOWP/5nYdKW4ZxzqSypgSEpkyAsHjGzJxtoMxS4BzjLzMrrxpvZ6vDneuApYES8+c1sopmVmFlJUVFRY7+FPfp3yeXco4t5aMpSVlRsTdpynHMuVSXzLCkB9wJzzex3DbTpBTwJXGRm82PG50jKrXsMjAFmJ6vWRF1zygDS08RvXvIuQ5xzrU8ytzCOBS4CPheeGjtD0mmSJkiaELb5CdAJuKPe6bNdgLckzQSmAs+Z2YtJrDUhXfOz+caxffnnjNXMXrU56nKcc65JqSV1e1FSUmKlpcm9ZKNy+05OuOVVhnTP5+FvjkzqspxzLtkkTUv00gW/0ns/5WVnctXn+vPWwg28MT95B9mdcy7VeGAcgAtH9aJnx7bc/MJH7N7dcrbQnHNubzwwDkBWRjrfP3UgH66p5J8zV0VdjnPONQkPjAN0xtDuHN4jj99Mms/2nd5liHOu5fPAOEBpaeK6sYNYtWkbD09ZFnU5zjmXdB4YB+G4/oV8tn8hf3x1IZu37Yy6HOecSyoPjIN03bjD2LxtJ7e/siDqUpxzLqk8MA7SkO75fLWkmL+8s5Q5q/1iPudcy+WB0QiuHzeIju0yuf7JWezy02ydcy2UB0YjyG+Xyf+ePpgPVm7mwclLoy7HOeeSwgOjkZw5rDvHDyjiN5PmsXrTtqjLcc65RueB0Ugk8YsvHs4uM254Zk7U5TjnXKPzwGhExQXtuPrkAbz84TpenL026nKcc65ReWA0ssuO68thXXO58Zk5bNnu12Y451oOD4xGlpmexs1nD2Xdlu38ZpLfaMk513J4YCTBkcUduGR0Hx6csozpyzdGXY5zzjUKD4wk+d6YAXTJzeaHT85i567dUZfjnHMHLZn39C6W9KqkuZLmSPpunDaS9AdJCyV9IGl4zLRLJC0Ih0uSVWey5GZn8tOzhvDR2i3c8+aSqMtxzrmDlswtjFrge2Y2CBgFXClpcL0244D+4TAe+DOApALgBmAkMAK4QVLHJNaaFKcO6cqYwV247ZX5LC/fGnU5zjl3UJIWGGa2xsymh4+3AHOBHvWanQU8aIEpQAdJ3YBTgZfNrMLMNgIvA2OTVWsy/fSsIWSkpfGjp2fRku6f7pxrfZrkGIakPsBRwLv1JvUAVsQ8XxmOa2h8vNceL6lUUmlZWerdY7tbflv+Z8wA3lywgWdmro66HOecO2BJDwxJ7YEngKvNrLL+5Diz2F7Gf3qk2UQzKzGzkqKiooMrNkkuGt2HYcUduOnZD9m0dUfU5Tjn3AFJamBIyiQIi0fM7Mk4TVYCxTHPewKr9zK+WUpPE7/60hFs2raTXz3/UdTlOOfcAUnmWVIC7gXmmtnvGmj2DHBxeLbUKGCzma0BJgFjJHUMD3aPCcc1W4O75/HN4/ryeOkK3l1cHnU5zjm335K5hXEscBHwOUkzwuE0SRMkTQjbPA8sBhYCdwPfBjCzCuBnwHvhcFM4rln77sn9KS5oy/VPzaKmdlfU5Tjn3H5RSzpzp6SkxEpLS6MuY69en1/GJfdN5eqT+3P1yQOiLsc518pJmmZmJYm09Su9m9gJA4o4c1h37nh1EQvXV0VdjnPOJcwDIwL/e/pgsjPT+OFTs9jtt3R1zjUTHhgRKMrN4oenDWLqkgr+Pm3FvmdwzrkU4IERka+UFDOiTwG/fP4jNlTVRF2Oc87tkwdGRNLSxC+/fDhbd9Tys399GHU5zjm3Tx4YETq0cy5XnHgo/5yxmtfnp163Js45F8sDI2LfPvEQ+hXmcOMzc9hR6/fNcM6lLg+MiGVnpvOTMwazZEM1f3nb75vhnEtdHhgp4MSBnTl5UGf+8MoC1lduj7oc55yLywMjRfz4C4PZucu4+UXvnNA5l5o8MFJEn8IcvvnZvjw5fRXTlm2MuhznnPsUD4wUcuVJh9IlL4sbn5njV4A751KOB0YKycnK4IenDWLWqs1+BbhzLuV4YKSYM4d1p6R3R255cR6bt+2MuhznnNvDAyPFSOLGM4dQsXUHt/17QdTlOOfcHh4YKejwHvmcP6IXD0xeyoJ1W6IuxznngOTeovU+SeslzW5g+vdj7sQ3W9IuSQXhtKWSZoXTUvuOSEnyP2MGktMmnRufnUNLusmVc675SuYWxv3A2IYmmtmtZnakmR0JXA+8Xu82rCeF0xO6E1RLU5DThu+NGcjbC8uZNGdd1OU451zyAsPM3gASvQ/3+cBjyaqlubpgZC8Gdsnl5899yPadfg9w51y0Ij+GIakdwZbIEzGjDXhJ0jRJ4/cx/3hJpZJKy8paVo+vGelp3HDmYFZu3MbENxZHXY5zrpWLPDCAM4C36+2OOtbMhgPjgCslHd/QzGY20cxKzKykqKgo2bU2uWMOKeQLR3TjjtcWsmrTtqjLcc61YqkQGOdRb3eUma0Of64HngJGRFBXyrj+tMMA+OXzcyOuxDnXmkUaGJLygROAf8aMy5GUW/cYGAPEPdOqtejZsR1XnHAoz32whsmLyqMuxznXSiXztNrHgMnAQEkrJV0maYKkCTHNvgS8ZGbVMeO6AG9JmglMBZ4zsxeTVWdzcfkJ/ejRoS0/fXYOtbv8RkvOuaaXkawXNrPzE2hzP8Hpt7HjFgPDklNV85Wdmc7/nj6ICQ9P59Gpy7l4dJ+oS3LOtTKpcAzDJejUIV059tBO/Pal+VRU74i6HOdcK+OB0YxI4oYzhlBVU8tvX5oXdTnOuVbGA6OZGdAll4tH9+bRqcuZvWpz1OU451oRD4xm6OqTB9CxXRt+6v1MOeeakAdGM5TfNpMfnDqQ95Zu5JmZq6MuxznXSnhgNFPnlhRzRI98fvX8R1TX1EZdjnOuFfDAaKbS08SNZw5mbeV27nhtYdTlOOdaAQ+MZuzo3gV8+age3P3GEpaVV+97BuecOwgeGM3cteMOIzNd3PiMHwB3ziWXB0Yz1yUvm/8eM5BX55Xx/Ky1UZfjnGvBPDBagEtG9+bwHnnc+OwcKrfvjLoc51wL5YHRAmSkp/GrLw2lvKqGW178KOpynHMtlAdGC3FEz3y+fkxfHnl3OdOWbYy6HOdcC+SB0YJ8b8wAuuVl88MnZ7HTu0B3zjUyD4wWJCcrg5+edTjz1m3hnjeXRF2Oc66F8cBoYU4Z3IWxQ7py2yvzWV6+NepynHMtSDLvuHefpPWS4t5eVdKJkjZLmhEOP4mZNlbSPEkLJV2XrBpbqhvPHEJGWho/enqWX5vhnGs0ydzCuB8Yu482b5rZkeFwE4CkdOBPwDhgMHC+pMFJrLPF6ZqfzfdPHcibCzZ454TOuUaTtMAwszeAigOYdQSw0MwWm9kO4K/AWY1aXCtw4ajeDCvuwM/+9SGbtvrd+ZxzBy+hwJD0XUl5CtwrabqkMY2w/NGSZkp6QdKQcFwPYEVMm5XhuIZqGy+pVFJpWVlZI5TUMqSniV9+6XA2bt3Jr/3aDOdcI0h0C+MbZlYJjAGKgEuBmw9y2dOB3mY2DLgdeDocrzhtG9wRb2YTzazEzEqKiooOsqSWZUj3fC47ri+PTV3B1CUHsrHnnHMfSzQw6j7ETwP+YmYzif/BnjAzqzSzqvDx80CmpEKCLYrimKY9Ad8Rf4CuPrk/PTq05YdPzWJHrV+b4Zw7cIkGxjRJLxEExiRJucBBffpI6ipJ4eMRYS3lwHtAf0l9JbUBzgOeOZhltWbt2mTw8y8ezsL1Vdz1+qKoy3HONWMZCba7DDgSWGxmWyUVEOyWapCkx4ATgUJJK4EbgEwAM7sTOAe4QlItsA04z4JzQGslXQVMAtKB+8xszn6/M7fHSYd15gtDu3H7qws5fVh3+hbmRF2Sc64ZUiLn6Us6FphhZtWSLgSGA7eZ2bJkF7g/SkpKrLS0NOoyUtL6yu18/nevc0SPfB755kjCjTvnXCsnaZqZlSTSNtFdUn8GtkoaBvwAWAY8eID1uQh0zsvm2rGH8c6icp56f1XU5TjnmqFEA6M23F10FsGWxW1AbvLKcsnwtRG9GN6rAz9/bi4V1X5thnNu/yQaGFskXQ9cBDwXXo2dmbyyXDKkpYlffvkIKrft5FfPz426HOdcM5NoYHwVqCG4HmMtwYV0tyatKpc0h3XN41vH9+Pv01YyeVF51OU455qRhAIjDIlHgHxJpwPbzcyPYTRT3/lcf3oVtONHT82ipnZX1OU455qJRLsG+QowFTgX+ArwrqRzklmYS562bdL5+RcPZ/GGau541a/NcM4lJtHrMH4EfMbM1gNIKgL+DfwjWYW55Dp+QBFnHdmdP7+2iDOGdefQzu2jLsk5l+ISPYaRVhcWofL9mNelqB9/YTDZmWn86Cm/b4Zzbt8S/dB/UdIkSV+X9HXgOeD55JXlmkJRbhY/PG0Q7y6p4PH3Vux7Budcq5boQe/vAxOBocAwYKKZXZvMwlzT+EpJMaP7deLGZ+fw0drKqMtxzqWwhHcrmdkTZvbfZnaNmT2VzKJc00lLE7edfyR52Zlc8fB0KrfvjLok51yK2mtgSNoiqTLOsEWSfx1tITrnZvOnC4azvGIr3//7TD+e4ZyLa6+BYWa5ZpYXZ8g1s7ymKtIl32f6FHD9uMOYNGcdE99YHHU5zrkU5Gc6uT0uO64vXziiG79+8SO/Ctw59ykeGG4PSfz6nKH0Lczhvx6bztrN26MuyTmXQjww3Ce0z8rgzguPZuuOXVz56HR27vLbujrnAkkLDEn3SVovaXYD0y+Q9EE4vBPea6Nu2lJJsyTNkOR3RGpi/bvk8uuzhzJt2UZ+6b3aOudCydzCuB8Yu5fpS4ATzGwo8DOC6zxinWRmRyZ6JyjXuM4Y1p1Lj+3DX95eyrMzV0ddjnMuBSQtMMzsDaBiL9PfMbON4dMpQM9k1eIOzA9PG0RJ745c+8QHLFi3JepynHMRS5VjGJcBL8Q8N+AlSdMkjd/bjJLGSyqVVFpWVpbUIlubzPQ0/vi14bRrk86Eh6dRVVMbdUnOuQhFHhiSTiIIjNiuRo41s+HAOOBKScc3NL+ZTTSzEjMrKSoqSnK1rU/X/GxuP384SzZUc+0/PvCL+pxrxSINDElDgXuAs8xsz4n/ZrY6/LkeeAoYEU2FDmD0IZ34wdjDeG7WGu59a0nU5TjnIhJZYEjqBTwJXGRm82PG50jKrXsMjAHinmnlms7lx/djzOAu/OqFj5i6pMFDU865FiyZp9U+BkwGBkpaKekySRMkTQib/AToBNxR7/TZLsBbkmYS3OXvOTN7MVl1usRI4jdfGUavgnZc9eh01m/xi/qca23UkvZJl5SUWGmpX7aRTB+treSLf3qboT078Og3R5KRHvlhMOfcQZA0LdHLF/y/3e2Xw7rmcfOXhzJ1SQW3TJoXdTnOuSbkgeH22xeP6sFFo3oz8Y3FvDh7TdTlOOeaiAeGOyA/Pn0QRxZ34H/+/gGLyqqiLsc51wQ8MNwBycpI544LhtMmI40rHp7G1h1+UZ9zLZ0Hhjtg3Tu05Q/nHcWC9VV857H3qandFXVJzrkk8sBwB+W4/oXcdOYQ/j13PZc/NI3tOz00nGupPDDcQbtodB9+9eUjeH1+GZc98J7vnnKuhfLAcI3i/BG9+M05w5i8qJyv3/ceW7bvjLok51wj88Bwjebso3ty23lHMW35Ri66dyqbt3loONeSeGC4RnXGsO7cccFw5qzezAX3TGFj9Y6oS3LONRIPDNfoTh3SlYkXlTB/XRXn3z2FDVU1UZfknGsEHhguKU46rDP3XfIZlpZX89W7JrOu0jsrdK6588BwSXNc/0IeuHQEazdv5yt3TWbVpm1Rl+ScOwgeGC6pRvbrxEPfHElF9Q6+etdkVlRsjbok59wB8sBwSTe8V0ce/eYoqmpqOffOySz2vqeca5Y8MFyTOKJnPo99axQ7d+3mK3dNYf66LVGX5JzbT0kNDEn3SVovKe4tVhX4g6SFkj6QNDxm2iWSFoTDJcms0zWNQd3yePzyUaQJzps4hQ9XV0ZdknNuPyR7C+N+YOxepo8D+ofDeODPAJIKgBuAkcAI4AZJHZNaqWsSh3bO5W+XjyY7I43z757CzBWboi7JOZegpAaGmb0BVOylyVnAgxaYAnSQ1A04FXjZzCrMbCPwMnsPHteM9CnM4fHLR5PXNoML73mXacv29ifinEsVUR/D6AGsiHm+MhzX0PhPkTReUqmk0rKysqQV6hpXcUE7/nb5aApzs7jo3qm8tWBD1CU55/Yh6sBQnHG2l/GfHmk20cxKzKykqKioUYtzydUtvy2Pjx9Fz45tuei+d7l10kfs3LU76rKccw2IOjBWAsUxz3sCq/cy3rUwnfOyeerbx3Lu0T3506uLOOfOySzdUB11Wc65OKIOjGeAi8OzpUYBm81sDTAJGCOpY3iwe0w4zrVAOVkZ3HLOMO64YDhLyqr4wh/e5O+lKzCLu1HpnItIRjJfXNJjwIlAoaSVBGc+ZQKY2Z3A88BpwEJgK3BpOK1C0s+A98KXusnM/MhoC3faEd04srgD1zw+g+//4wNem1/GL794BPntMqMuzTkHqCV9iyspKbHS0tKoy3AHaddu487XF/H7l+fTOTeL33/1SEb26xR1Wc61SJKmmVlJIm2j3iXl3Kekp4krTzqUJ644hjbh9Rq/fWmeHxB3LmIeGC5lDSvuwHPf+SxnD+/J7f9ZyLl3TmZZuR8Qdy4qHhgupeVkZXDrucP409eGs7isitNue5Mnpq30A+LORcADwzULXxjajReuPp4hPfL53t9n8p2/zvB7hjvXxDwwXLPRo0NbHvvWKL5/6kCen7WG0257k/eW+slzzjUVDwzXrMQeEM9IF1+9azK/e2ketX5A3Lmk88BwzdKR4QHxLw/vyR/+s5DTb3+LV+et92MbziWRB4ZrttpnZfCbc4dx54XD2bZzF5f+5T3OmziF95dvjLo051okDwzX7I09vBsvX3MCN501hEVlVXzpjneY8NA0Fq73W8E615j8Sm/XolTX1HLvW0u46/VFbK/dzblH9+TqkwfQNT876tKcS0n7c6W3B4Zrkcqravjjqwt5eMoy0iQuPbYvV5xwiPdL5Vw9HhjOhVZUbOV3L8/n6RmryMvO5NsnHsIlx/QhOzM96tKcSwkeGM7V8+HqSm6Z9BGvzSujW34215w8gC8P70FGuh/Gc62bdz7oXD2Du+dx/6Uj+Ov4UXTJy+YHT3zA2NveZNKctX4qrnMJ8sBwrcqofp146tvHcOeFR7PbjMsfmsbZf36HN+aXsXu3B4dze+O7pFyrVbtrN/+YtpLf/3s+6ypr6FuYwwUje3Hu0cV+cNy1GilzDEPSWOA2IB24x8xurjf998BJ4dN2QGcz6xBO2wXMCqctN7Mz97U8Dwx3ILbv3MWLs9fy0JRlTFu2kayMNM4c1p0LR/VmWHGHqMtzLqlSIjAkpQPzgVOAlQS3Wz3fzD5soP1/AUeZ2TfC51Vm1n5/lumB4Q7Wh6srefjdZTz9/iq27tjF0J75XDiqN2cM7U7bNn5mlWt5UiUwRgM3mtmp4fPrAczsVw0yNc/8AAASOElEQVS0fwe4wcxeDp97YLjIVG7fydPvr+KhyctYsL6KvOwMzi0p5oKRvehXtF9/ls6ltP0JjIwk1tEDWBHzfCUwMl5DSb2BvsB/YkZnSyoFaoGbzezpBuYdD4wH6NWrVyOU7RzkZWdy8eg+XDSqN1OXVPDQlGU88M5S7n1rCccdWsiFo3pz8qDOflqua1WSGRiKM66hzZnzgH+Y2a6Ycb3MbLWkfsB/JM0ys0WfekGzicBECLYwDrZo52JJYmS/Tozs14n1W7bzt/dW8Oi7y5nw8DS65mVz/ohenD+imM553vWIa/mSGRgrgeKY5z2B1Q20PQ+4MnaEma0Ofy6W9BpwFPCpwHCuqXTOzeaqz/VnwgmH8Oq8Mh6asozf/3s+t/9nAScO7MxpR3Tl84O6kN/Wz7ByLVMyA+M9oL+kvsAqglD4Wv1GkgYCHYHJMeM6AlvNrEZSIXAscEsSa3UuYRnpaZwyuAunDO7C0g3VPDp1Oc/OXM2/564jM10ce2gh4w7vyimDu1KQ0ybqcp1rNMk+rfY04P8ITqu9z8x+IekmoNTMngnb3Ahkm9l1MfMdA9wF7Ca4uPD/zOzefS3PD3q7qOzebcxcuYkXZ6/l+dlrWFGxjfQ0MapfAeMO78aYIV3onOu7rVzqSYmzpKLggeFSgZkxZ3UlL8xewwuz1rJ4QzUSfKZPAeMO78rYw7vSLb9t1GU6B3hgRF2Gc3uYGfPXVe0Jj3nrtgBwVK8OjDu8K+MO70ZxQbuIq3StmQeGcylqUVlVsNtq1hrmrK4E4PAeeZwyqCvH9e/E0J4dyPRTdV0T8sBwrhlYXr6VF+es4flZa5m5chNmkNMmnVH9OnHsoYUc17+Q/p3bI8U7Q925xuGB4Vwzs7F6B5MXl/P2wg28vXADS8u3AlCUm8WxhwQBcuyhhXTv4Mc+XOPywHCumVu5cSvvLCznrYUbeGfRBjZU7QCgX2FOGB6dGN2v0HvVdQfNA8O5FsTMmLduC28t2MA7i8qZsricrTt2kSY4okc+xxxayKh+nTiyZwcPELffPDCca8F21O5m5spNe3Zfvb98E7XhzZ8OKcrhqF4dOapXB44q7siALu29vyu3Vx4YzrUi1TW1zFyxifdXbOL95Rt5f/kmyquDXVjt2qQztGd+ECLFHTiyVwe/gNB9Qqr0VuucawI5WRkcc2ghxxxaCAS7sFZUbOP9FUF4vL9iE/e8uZidu4Ivhz06tA22QMItkSHd88jK8Ht9uH3zwHCuhZFEr07t6NWpHWcd2QMI7io4Z3VlsAWyYhPvL9/Evz5YA0Cb9DQGds1lULdcBnXL2zN4J4quPg8M51qB7Mx0ju7dkaN7d9wzbl3l9nALZCNzVlXyytz1/K105Z7pPTq0ZVC3PAbHBEmvgnakpfl1Ia2VB4ZzrVSXvGzGhn1bQbArq2xLDR+uqWTumi3MXVPJ3DWVvDpvPbvCg+o5bdLDrZGPt0QO65pLTpZ/lLQGftDbObdX23fuYsG6KuauqQzDJBgqt9cCIEFxx3YcUpTDIUXtOaRzew7t3J5Ditp79+7NgB/0ds41muzMdI7omc8RPfP3jDMzVm/eztzVQXjMX1/FovVVTF5czvadu/e069guMwiRojBEOgeh0rNjO9J911az41sYzrlGs3u3sWrTNhaVVbGorJpFZVUsXF/F4rKqPVerA7TJSKNvpxwO6ZzDoUXt6VuUQ6+CdvQqyKGwfRvvP6sJ+RaGcy4SaWmiuKAdxQXtOHHgJ6dt2rojCJH1VWGgVDF3zRZenL2W3THfW9u1SQ/Dox29OwU/e3XKoXdBO3p0bOu9+UYoqYEhaSxwG8Ed9+4xs5vrTf86cCvBLVwB/mhm94TTLgF+HI7/uZk9kMxanXPJ1aFdG47u3eYTZ2oB1NTuYkXFNpZXVLO8fCvLKrayvHwrSzZU8/r8MmpqP97FlSbo3qHtx0FSkEPvTu0o7hiEScd2mb51kkRJCwxJ6cCfgFOAlcB7kp4xsw/rNX3czK6qN28BcANQAhgwLZx3Y7Lqdc5FIysjnUPDA+X17d5trN9Sw/KKrSwrr2Z5xdbw8VYmzVlHRfWOT7TPzkyje4e29AiH7vV+ds3Ppk2Gb6EcqGRuYYwAFprZYgBJfwXOAuoHRjynAi+bWUU478vAWOCxJNXqnEtBaWmia342XfOzGdG34FPTt2zfyfKKrazcuI1VG7exetM2Vm0Kfs5ds4UNVTWfaC9B59ysT4RIj45t6Zbfli55WXTNy6ZT+yw/IN+AZAZGD2BFzPOVwMg47c6WdDwwH7jGzFY0MG+PZBXqnGuecrMzGdI9nyHd8+NO375zF2s2bw+CZGMQJnWBMnvVZl6as44du3Z/Yp70NNE5N4suedl7QqRLfjZd84Khc14QYO1b4bUnyXzH8SK6/ilZzwKPmVmNpAnAA8DnEpw3WIg0HhgP0KtXrwOv1jnX4mRnptO3MIe+hTlxp+/ebWyormHNpu2srdzO+srg59rNNayr3M6ismreWVTOlvCak1jtszKCQMnPpktuNkW5WRTlZlHYPmvP46L2WXRoQcdVkhkYK4HimOc9gdWxDcysPObp3cCvY+Y9sd68r8VbiJlNBCZCcFrtwRTsnGtd0tJE59xsOudmM2wv7aprallXWRcqNWGobGddZTC8u6SCsqoadtTu/tS8menaEyKF7YMQ2RMoMSFTkNOGvOyMlA6XZAbGe0B/SX0JzoI6D/habANJ3cxsTfj0TGBu+HgS8EtJdadTjAGuT2KtzjnXoJysDPoVtadf0acPzNcxM7bU1FK2peYTw4aq8HFVsNUye9Vmyqt37OluJVZmuijIaUNBThadctrQqX0bCnLahI+DUClsH0yPImCSFhhmVivpKoIP/3TgPjObI+kmoNTMngG+I+lMoBaoAL4ezlsh6WcEoQNwU90BcOecS0WSyMvOJC87uLp9b3bvNjZu3UFZ1cehUl61g/LqHVRU7aC8uoby6h2sWLGV8qodVNV8epcYfBwwvQty+NuE0cl4W5/gV3o751yK275zFxXVO6io3sGGqpqYxzuoqK4hTeLms4ce0Gv7ld7OOdeCZGem0z08FThKfgWLc865hHhgOOecS4gHhnPOuYR4YDjnnEuIB4ZzzrmEeGA455xLiAeGc865hHhgOOecS0iLutJbUhmw7ABnLwQ2NGI5jc3rOzhe38Hx+g5OKtfX28yKEmnYogLjYEgqTfTy+Ch4fQfH6zs4Xt/BSfX6EuW7pJxzziXEA8M551xCPDA+NjHqAvbB6zs4Xt/B8foOTqrXlxA/huGccy4hvoXhnHMuIR4YzjnnEtLqAkPSWEnzJC2UdF2c6VmSHg+nvyupTxPWVizpVUlzJc2R9N04bU6UtFnSjHD4SVPVFy5/qaRZ4bI/dXtDBf4Qrr8PJA1vwtoGxqyXGZIqJV1dr02Trj9J90laL2l2zLgCSS9LWhD+7NjAvJeEbRZIuqQJ67tV0kfh7+8pSR0amHevfwtJrO9GSatifoenNTDvXv/Xk1jf4zG1LZU0o4F5k77+Gp2ZtZqB4N7ii4B+QBtgJjC4XptvA3eGj88DHm/C+roBw8PHucD8OPWdCPwrwnW4FCjcy/TTgBcAAaOAdyP8Xa8luCgpsvUHHA8MB2bHjLsFuC58fB3w6zjzFQCLw58dw8cdm6i+MUBG+PjX8epL5G8hifXdCPxPAr//vf6vJ6u+etN/C/wkqvXX2ENr28IYASw0s8VmtgP4K3BWvTZnAQ+Ej/8BfF6SmqI4M1tjZtPDx1uAuUCPplh2IzoLeNACU4AOkrpFUMfngUVmdqBX/jcKM3sDqKg3OvZv7AHgi3FmPRV42cwqzGwj8DIwtinqM7OXzKw2fDoF6NnYy01UA+svEYn8rx+0vdUXfm58BXissZcbldYWGD2AFTHPV/LpD+Q9bcJ/ms1ApyapLka4K+wo4N04k0dLminpBUlDmrQwMOAlSdMkjY8zPZF13BTOo+F/1CjXH0AXM1sDwZcEoHOcNqmyHr9BsMUYz77+FpLpqnCX2X0N7NJLhfX3WWCdmS1oYHqU6++AtLbAiLelUP+84kTaJJWk9sATwNVmVllv8nSC3SzDgNuBp5uyNuBYMxsOjAOulHR8vempsP7aAGcCf48zOer1l6hUWI8/AmqBRxposq+/hWT5M3AIcCSwhmC3T32Rrz/gfPa+dRHV+jtgrS0wVgLFMc97AqsbaiMpA8jnwDaJD4ikTIKweMTMnqw/3cwqzawqfPw8kCmpsKnqM7PV4c/1wFMEm/6xElnHyTYOmG5m6+pPiHr9hdbV7aYLf66P0ybS9RgeZD8duMDCHe71JfC3kBRmts7MdpnZbuDuBpYb9frLAL4MPN5Qm6jW38FobYHxHtBfUt/wW+h5wDP12jwD1J2Rcg7wn4b+YRpbuM/zXmCumf2ugTZd646pSBpB8Dssb6L6ciTl1j0mODg6u16zZ4CLw7OlRgGb63a/NKEGv9lFuf5ixP6NXQL8M06bScAYSR3DXS5jwnFJJ2kscC1wppltbaBNIn8Lyaov9pjYlxpYbiL/68l0MvCRma2MNzHK9XdQoj7q3tQDwVk88wnOoPhROO4mgn8OgGyCXRkLgalAvyas7TiCzeYPgBnhcBowAZgQtrkKmENw1scU4JgmrK9fuNyZYQ116y+2PgF/CtfvLKCkiX+/7QgCID9mXGTrjyC41gA7Cb71XkZwTOwVYEH4syBsWwLcEzPvN8K/w4XApU1Y30KC/f91f4N1Zw12B57f299CE9X3UPi39QFBCHSrX1/4/FP/601RXzj+/rq/uZi2Tb7+GnvwrkGcc84lpLXtknLOOXeAPDCcc84lxAPDOedcQjwwnHPOJcQDwznnXEI8MFzKk/RO+LOPpK818mv/MN6ykkXSF5PVQ27999JIr3mEpPsb+3Vd8+Sn1bpmQ9KJBL2Unr4f86Sb2a69TK8ys/aNUV+C9bxDcM3PhoN8nU+9r2S9F0n/Br5hZssb+7Vd8+JbGC7lSaoKH94MfDa8f8A1ktLDeze8F3ZEd3nY/kQF9xV5lOACLyQ9HXbyNqeuozdJNwNtw9d7JHZZ4ZXqt0qaHd6z4Ksxr/2apH8ouGfEIzFXjt8s6cOwlt/EeR8DgJq6sJB0v6Q7Jb0pab6k08PxCb+vmNeO914ulDQ1HHeXpPS69yjpFwo6YJwiqUs4/tzw/c6U9EbMyz9LcKW0a+2ivnLQBx/2NQBV4c8TibmXBTAe+HH4OAsoBfqG7aqBvjFt666mbkvQBUOn2NeOs6yzCboUTwe6AMsJ7ldyIkEPxj0JvnBNJrhCvwCYx8db7R3ivI9Lgd/GPL8feDF8nf4EVwpn78/7ild7+HgQwQd9Zvj8DuDi8LEBZ4SPb4lZ1iygR/36gWOBZ6P+O/Ah+iEj0WBxLgWNAYZKOid8nk/wwbsDmGpmS2LafkfSl8LHxWG7vfUhdRzwmAW7fdZJeh34DFAZvvZKAAV3U+tD0M3IduAeSc8B/4rzmt2Asnrj/mZBJ3oLJC0GDtvP99WQzwNHA++FG0Bt+biTwx0x9U0DTgkfvw3cL+lvQGzHl+sJurVwrZwHhmvOBPyXmX2iU77wWEd1vecnA6PNbKuk1wi+ye/rtRtSE/N4F8Hd6WrDzgw/T7D75irgc/Xm20bw4R+r/kFEI8H3tQ8CHjCz6+NM22lmdcvdRfg5YGYTJI0EvgDMkHSkmZUTrKttCS7XtWB+DMM1J1sIbl1bZxJwhYIu4ZE0IOz5s758YGMYFocR3Dq2zs66+et5A/hqeDyhiOBWnFMbKkzBPUzyLegy/WqCezXUNxc4tN64cyWlSTqEoEO6efvxvuqLfS+vAOdI6hy+RoGk3nubWdIhZvaumf0E2MDH3YMPoDn0pOqSzrcwXHPyAVAraSbB/v/bCHYHTQ8PPJcR/3anLwITJH1A8IE8JWbaROADSdPN7IKY8U8Bowl6EzXgB2a2NgyceHKBf0rKJvh2f02cNm8Av5WkmG/484DXCY6TTDCz7ZLuSfB91feJ9yLpxwR3dEsj6E31SmBvt6y9VVL/sP5XwvcOcBLwXALLdy2cn1brXBOSdBvBAeR/h9c3/MvM/hFxWQ2SlEUQaMfZx/f5dq2U75Jyrmn9kuCeHc1FL+A6DwsHvoXhnHMuQb6F4ZxzLiEeGM455xLigeGccy4hHhjOOecS4oHhnHMuIf8PPMpdctfwvEAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "parameters = L_layer_model(train_set_x_new, train_set_y_new, dimensions, num_iterations = 2000, print_loss = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(X, y, parameters):\n",
    "    \n",
    "    # Performs forward propogation using the trained parameters and calculates the accuracy\n",
    "    \n",
    "    m = X.shape[1]\n",
    "    n = len(parameters) // 2 # number of layers in the neural network\n",
    "    \n",
    "    # Forward propagation\n",
    "    probas, caches = L_layer_forward(X, parameters)\n",
    "    \n",
    "    p = np.argmax(probas, axis = 0)\n",
    "    act = np.argmax(y, axis = 0)\n",
    "\n",
    "    print(\"Accuracy: \"  + str(np.sum((p == act)/m)))\n",
    "        \n",
    "    return p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see the accuray we get on the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.8774000000000002\n"
     ]
    }
   ],
   "source": [
    "pred_train = predict(train_set_x_new, train_set_y_new, parameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get ~ 88% accuracy on the training data. Let's see the accuray on the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.8674000000000002\n"
     ]
    }
   ],
   "source": [
    "pred_test = predict(test_set_x, test_set_y, parameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is ~87%. You can train the model even longer and get better result. You can also try to change the network structure. \n",
    "<br>Below, you can see which all numbers are incorrectly identified by the neural network by changing the index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x18fb86967f0>"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAEICAYAAACQ6CLfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAESxJREFUeJzt3X2sVPWdx/H3R0otVSsgSkFQWutuSiprlagRbK/pVsVd14dEU2IV063XTWq2JrUuwbjYVVvTbF13o9FeHwFbXVulIkrrwyq63ZQVSRexrK3Rq6IXbgEtsMW0wnf/mHPtgDNn5s7TGfh9XsnNnZnvefg68rnnzHmYnyICM0vPPkU3YGbFcPjNEuXwmyXK4TdLlMNvliiH3yxRDv8eStLTkr7a6nklzZN0e4PL/Y6kyxqZd5jr+RtJ97V7PXs7h79gkvol/WXRfQyJiG9HxLD/qEg6GLgQ+H72/ARJj0vaLOm3kn4kaUKdyzpE0r2S3pL0O0k/l3R8WY9LgM9ImjbcPu1PHH5rlYuARyNie/Z8DNAHTAEOB7YCd9W5rP2B54BjgbHAAuARSfuXTXMv0Nt01wlz+LuUpDGSlmZbzbezx5N2m+wISf+dbR0fkjS2bP4TJP2XpHck/Y+knjrXe7Wke7LHH5F0j6RN2XKekzS+yqyzgOVDTyJiWUT8KCK2RMTvgZuAGfX0EBGvRMQNETEQETsiog/4MPDnZZM9DfxVPcuzyhz+7rUPpS3l4cBhwHZKASp3IfAVYCLwHvBvAJIOBR4BrqW05bwceCDbNR+OOcCBwGTgIODvsj4qOQp4KWdZnwNeHOb6AZB0NKXwv1z28lpgiqSPNbJMc/i7VkRsiogHIuL3EbEVuA74/G6TLYqINRHxf8BVwHmSRgBfprQL/mhE7IyIx4GVwOnDbOOPlEL/qWwL/HxEbKky7WhKu/YfkH02/0fgm8NcP1m4FwHfiojflZWG1jV6uMu0Eoe/S0n6qKTvS3pN0hbgGWB0Fu4hb5Q9fg0YCYyjtLdwbrar/o6kd4CZQF0H3MosAn4G3JcdfPuupJFVpn0bOKDCf8engGXA1yPi2eGsXNIo4GHgFxHxnd3KQ+t6ZzjLtD9x+LvXNyh9xj0+Ij5GabcZQGXTTC57fBilLfVGSn8UFkXE6LKf/SLi+uE0EBF/jIhvRcRU4ETgryl91KhkNfBn5S9IOhx4ArgmIhYNZ92S9gV+ArwJXFJhkk8D/Tl7IlaDw98dRmYH14Z+PkRpy7YdeCc7kDe/wnxfljRV0keBfwJ+HBE7gHuAMySdKmlEtsyeCgcMc0k6WdJR2d7GFkp/XHZUmfxRyj6WZMcd/gO4OSJurbDsiyT1V1nvSODHlP77L4yInRUm+zylPQprkMPfHR6l9A996Odq4EZgFKUt+S+An1aYbxFwN7Ae+Ajw9wAR8QZwJjAP+C2lPYFvMvz/3x+nFMItlA6wLaf0h6WShcDp2a46wFeBTwLzJW0b+imbfjLw8yrLGtrLOIXSH7+h+U8qm2Y22TUF1hj5yzysVSR9GxiMiBvrmPYxSscB1jawnjOACyLivAbatIzDb5Yo7/abJcrhN0uUw2+WqA91cmWSfIDBrM0iQrWnanLLL+k0SS9JelnS3GaWZWad1fDR/uzCj18DXwTWUboFc3ZE/CpnHm/5zdqsE1v+44CXs9sv/wDcR+nCEjPbAzQT/kPZ9caSddlru5DUK2mlpJVNrMvMWqyZA36Vdi0+sFuffRFDH3i336ybNLPlX8eud5VNAt5qrh0z65Rmwv8ccKSkT0j6MPAlYElr2jKzdmt4tz8i3pN0KaUvexgB3BkRDX1Nk5l1Xkdv7PFnfrP268hFPma253L4zRLl8JslyuE3S5TDb5Yoh98sUQ6/WaIcfrNEOfxmiXL4zRLl8JslyuE3S5TDb5Yoh98sUQ6/WaIcfrNEOfxmiXL4zRLl8JslyuE3S5TDb5aojg7Rbe1x8MEHV63deuutufOec845ufX169fn1i+44ILc+hNPPJFbt+J4y2+WKIffLFEOv1miHH6zRDn8Zoly+M0S5fCbJcrn+fcCt99+e9XaMccckzvvySefnFufNGlSbv2RRx7JrZ911llVa8uWLcud19qrqfBL6ge2AjuA9yJieiuaMrP2a8WW/+SI2NiC5ZhZB/kzv1mimg1/AI9Jel5Sb6UJJPVKWilpZZPrMrMWana3f0ZEvCXpEOBxSf8bEc+UTxARfUAfgKRocn1m1iJNbfkj4q3s9yCwGDiuFU2ZWfs1HH5J+0k6YOgxcAqwplWNmVl7NbPbPx5YLGloOT+MiJ+2pCvbxcSJE3PrJ5xwQtVab2/FQzHve/rppxtp6X0nnnhibj3vGoRp06blzrtp06aGerL6NBz+iHgF+IsW9mJmHeRTfWaJcvjNEuXwmyXK4TdLlMNvlihFdO6iO1/h15gXX3wxt75t27aqtVqn4nbs2NFQT0MmT56cW+/v769ay7vdF+Dhhx9upKXkRYTqmc5bfrNEOfxmiXL4zRLl8JslyuE3S5TDb5Yoh98sUf7q7j1ArVt6zzjjjKq1Zs/j17J169aG5z377LNz6z7P317e8pslyuE3S5TDb5Yoh98sUQ6/WaIcfrNEOfxmifJ5fmvKli1bcutLly6tWps1a1buvKNGjcqtb9++Pbdu+bzlN0uUw2+WKIffLFEOv1miHH6zRDn8Zoly+M0S5fP81pSdO3fm1t99992qtfHjx+fO29PTk1tftmxZbt3y1dzyS7pT0qCkNWWvjZX0uKTfZL/HtLdNM2u1enb77wZO2+21ucCTEXEk8GT23Mz2IDXDHxHPAJt3e/lMYEH2eAGQP+6SmXWdRj/zj4+IAYCIGJB0SLUJJfUCvQ2ux8zapO0H/CKiD+gDD9Rp1k0aPdW3QdIEgOz3YOtaMrNOaDT8S4A52eM5wEOtacfMOqXmbr+ke4EeYJykdcB84Hrgfkl/C7wOnNvOJlN31VVX5dZfffXVDnVie5Oa4Y+I2VVKX2hxL2bWQb681yxRDr9Zohx+s0Q5/GaJcvjNEuVbevcAN910U9EttMXGjRtz6ytWrOhQJ2nylt8sUQ6/WaIcfrNEOfxmiXL4zRLl8JslyuE3S5QiOvflOv4mn73PgQcemFvftGlT1drAwEDuvJMnT26op9RFhOqZzlt+s0Q5/GaJcvjNEuXwmyXK4TdLlMNvliiH3yxRvp/fmiLln1KuVbfieMtvliiH3yxRDr9Zohx+s0Q5/GaJcvjNEuXwmyXK5/mtMIODg0W3kLSaW35Jd0oalLSm7LWrJb0p6ZfZz+ntbdPMWq2e3f67gdMqvP4vEXF09vNoa9sys3arGf6IeAbY3IFezKyDmjngd6mk1dnHgjHVJpLUK2mlpJVNrMvMWqzR8N8CHAEcDQwA36s2YUT0RcT0iJje4LrMrA0aCn9EbIiIHRGxE7gNOK61bZlZuzUUfkkTyp6eDaypNq2Zdaea5/kl3Qv0AOMkrQPmAz2SjgYC6AcuaWOPtpdavHhx0S0krWb4I2J2hZfvaEMvZtZBvrzXLFEOv1miHH6zRDn8Zoly+M0S5Vt693JTp07NrU+aNKmp5R977LENz7tw4cKm1m3N8ZbfLFEOv1miHH6zRDn8Zoly+M0S5fCbJcrhN0uUz/N3wIgRI3Lr48ePz60ff/zxufW5c+dWrU2cODF33lr23Xff3Pq4ceNy6xFRtTZq1KiGerLW8JbfLFEOv1miHH6zRDn8Zoly+M0S5fCbJcrhN0uU8s7DtnxlUudW1kWuueaa3Pq8efNy69u2bcut33LLLQ3VAF577bXc+syZM3Pry5cvz63nWb16dW591qxZufX169c3vO69WUSonum85TdLlMNvliiH3yxRDr9Zohx+s0Q5/GaJcvjNElXPEN2TgYXAx4GdQF9E/KukscC/A1MoDdN9XkS83b5WizVmzJiqtcsvvzx33osvvji3fvPNN+fWr7322tz64OBgbr0ZBxxwQG59586dufXzzz+/am3KlCm5865atSq33tfXl1vPu8Zhw4YNufOmoJ4t/3vANyLi08AJwNckTQXmAk9GxJHAk9lzM9tD1Ax/RAxExKrs8VZgLXAocCawIJtsAXBWu5o0s9Yb1md+SVOAzwIrgPERMQClPxDAIa1uzszap+7v8JO0P/AAcFlEbJHqunwYSb1Ab2PtmVm71LXllzSSUvB/EBEPZi9vkDQhq08AKh51ioi+iJgeEdNb0bCZtUbN8Ku0ib8DWBsRN5SVlgBzssdzgIda356ZtUvNW3olzQSeBV6gdKoPYB6lz/33A4cBrwPnRsTmGsvaY2/pve2226rWag2DnXe6C6C/v7+Rllpin33y//4/9dRTufWjjjoqtz527Nhh9zRk2rRpufWDDjootz569OiqtcWLFzfU056g3lt6a37mj4j/BKot7AvDacrMuoev8DNLlMNvliiH3yxRDr9Zohx+s0Q5/GaJ8hDdmbvuuiu33tPTU7V26qmn5s5b5Hn8Wq644orc+kknnZRbnz9/fivb2UWtr/a25njLb5Yoh98sUQ6/WaIcfrNEOfxmiXL4zRLl8JslykN0ZxYtWpRbf/3116vWrrzyyla30zLjxo3Lra9cuTK3vnlz7lc0MGPGjNz69u3bc+vWeh6i28xyOfxmiXL4zRLl8JslyuE3S5TDb5Yoh98sUT7Pb7aX8Xl+M8vl8JslyuE3S5TDb5Yoh98sUQ6/WaIcfrNE1Qy/pMmSnpK0VtKLkr6evX61pDcl/TL7Ob397ZpZq9S8yEfSBGBCRKySdADwPHAWcB6wLSL+ue6V+SIfs7ar9yKfmiP2RMQAMJA93ippLXBoc+2ZWdGG9Zlf0hTgs8CK7KVLJa2WdKekMVXm6ZW0UlL+90WZWUfVfW2/pP2B5cB1EfGgpPHARiCAayh9NPhKjWV4t9+szerd7a8r/JJGAkuBn0XEDRXqU4ClEfGZGstx+M3arGU39kgScAewtjz42YHAIWcDa4bbpJkVp56j/TOBZ4EXgJ3Zy/OA2cDRlHb7+4FLsoODecvylt+szVq6298qDr9Z+/l+fjPL5fCbJcrhN0uUw2+WKIffLFEOv1miHH6zRDn8Zoly+M0S5fCbJcrhN0uUw2+WKIffLFEOv1mian6BZ4ttBF4rez4ue60bdWtv3doXuLdGtbK3w+udsKP3839g5dLKiJheWAM5urW3bu0L3FujiurNu/1miXL4zRJVdPj7Cl5/nm7trVv7AvfWqEJ6K/Qzv5kVp+gtv5kVxOE3S1Qh4Zd0mqSXJL0saW4RPVQjqV/SC9mw44WOL5iNgTgoaU3Za2MlPS7pN9nvimMkFtRbVwzbnjOsfKHvXbcNd9/xz/ySRgC/Br4IrAOeA2ZHxK862kgVkvqB6RFR+AUhkj4HbAMWDg2FJum7wOaIuD77wzkmIv6hS3q7mmEO296m3qoNK38RBb53rRzuvhWK2PIfB7wcEa9ExB+A+4AzC+ij60XEM8Dm3V4+E1iQPV5A6R9Px1XprStExEBErMoebwWGhpUv9L3L6asQRYT/UOCNsufrKPANqCCAxyQ9L6m36GYqGD80LFr2+5CC+9ldzWHbO2m3YeW75r1rZLj7Visi/JWGEuqm840zIuIYYBbwtWz31upzC3AEpTEcB4DvFdlMNqz8A8BlEbGlyF7KVeirkPetiPCvAyaXPZ8EvFVAHxVFxFvZ70FgMaWPKd1kw9AIydnvwYL7eV9EbIiIHRGxE7iNAt+7bFj5B4AfRMSD2cuFv3eV+irqfSsi/M8BR0r6hKQPA18ClhTQxwdI2i87EIOk/YBT6L6hx5cAc7LHc4CHCuxlF90ybHu1YeUp+L3rtuHuC7nCLzuVcSMwArgzIq7reBMVSPokpa09lG53/mGRvUm6F+ihdMvnBmA+8BPgfuAw4HXg3Ijo+IG3Kr31MMxh29vUW7Vh5VdQ4HvXyuHuW9KPL+81S5Ov8DNLlMNvliiH3yxRDr9Zohx+s0Q5/GaJcvjNEvX/gjYb6jSOG3AAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "index  = 3474\n",
    "k = test_set_x[:,index]\n",
    "k = k.reshape((28, 28))\n",
    "plt.title('Label is {label}'.format(label=(pred_test[index], np.argmax(test_set_y, axis = 0)[index])))\n",
    "plt.imshow(k, cmap='gray')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}