TensorFlow2.0__00.01_basic_Classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Network Classifier\n",
    "\n",
    "#### Author: Ivan Bongiorni, Data Scientist at GfK.\n",
    "\n",
    "[LinkedIn profile](https://www.linkedin.com/in/ivan-bongiorni-b8a583164/)\n",
    "\n",
    "This model is a **feed forward Neural Network Classifier** based on the University of Wisconsin's **breast cancer dataset**.\n",
    "\n",
    "It represents a tutorial on basic TensorFlow 2.0 (alpha). The 1.x version of the same model can be found in [this Notebook](https://github.com/IvanBongiorni/TensorFlow_Tutorial/blob/master/TensorFlow_1_Classification_BatchGD.ipynb).\n",
    "\n",
    "Summary:\n",
    "\n",
    "1. Import data + quick dataprep,\n",
    "2. Neural Network architecture,\n",
    "3. Implementation of full-Batch Gradient Descent,\n",
    "4. Visualization,\n",
    "5. Wrap the Training process with a function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.0.0-alpha0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "pd.options.display.max_columns = None  # print all columns\n",
    "\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Import data + dataprep\n",
    "\n",
    "The purpose of this dataset is to classify breast cancer cases between **malignant** (M), and **benign** (B). Therefore, my classification network will have two output nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(569, 32)\n"
     ]
    }
   ],
   "source": [
    "# load the dataset from the UCI ML repository\n",
    "df = pd.read_csv(\"https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/wdbc.data\", \n",
    "                 sep=\",\", header=None)\n",
    "print(df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>842302</td>\n",
       "      <td>M</td>\n",
       "      <td>17.99</td>\n",
       "      <td>10.38</td>\n",
       "      <td>122.80</td>\n",
       "      <td>1001.0</td>\n",
       "      <td>0.11840</td>\n",
       "      <td>0.27760</td>\n",
       "      <td>0.3001</td>\n",
       "      <td>0.14710</td>\n",
       "      <td>0.2419</td>\n",
       "      <td>0.07871</td>\n",
       "      <td>1.0950</td>\n",
       "      <td>0.9053</td>\n",
       "      <td>8.589</td>\n",
       "      <td>153.40</td>\n",
       "      <td>0.006399</td>\n",
       "      <td>0.04904</td>\n",
       "      <td>0.05373</td>\n",
       "      <td>0.01587</td>\n",
       "      <td>0.03003</td>\n",
       "      <td>0.006193</td>\n",
       "      <td>25.38</td>\n",
       "      <td>17.33</td>\n",
       "      <td>184.60</td>\n",
       "      <td>2019.0</td>\n",
       "      <td>0.1622</td>\n",
       "      <td>0.6656</td>\n",
       "      <td>0.7119</td>\n",
       "      <td>0.2654</td>\n",
       "      <td>0.4601</td>\n",
       "      <td>0.11890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>842517</td>\n",
       "      <td>M</td>\n",
       "      <td>20.57</td>\n",
       "      <td>17.77</td>\n",
       "      <td>132.90</td>\n",
       "      <td>1326.0</td>\n",
       "      <td>0.08474</td>\n",
       "      <td>0.07864</td>\n",
       "      <td>0.0869</td>\n",
       "      <td>0.07017</td>\n",
       "      <td>0.1812</td>\n",
       "      <td>0.05667</td>\n",
       "      <td>0.5435</td>\n",
       "      <td>0.7339</td>\n",
       "      <td>3.398</td>\n",
       "      <td>74.08</td>\n",
       "      <td>0.005225</td>\n",
       "      <td>0.01308</td>\n",
       "      <td>0.01860</td>\n",
       "      <td>0.01340</td>\n",
       "      <td>0.01389</td>\n",
       "      <td>0.003532</td>\n",
       "      <td>24.99</td>\n",
       "      <td>23.41</td>\n",
       "      <td>158.80</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>0.1238</td>\n",
       "      <td>0.1866</td>\n",
       "      <td>0.2416</td>\n",
       "      <td>0.1860</td>\n",
       "      <td>0.2750</td>\n",
       "      <td>0.08902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>84300903</td>\n",
       "      <td>M</td>\n",
       "      <td>19.69</td>\n",
       "      <td>21.25</td>\n",
       "      <td>130.00</td>\n",
       "      <td>1203.0</td>\n",
       "      <td>0.10960</td>\n",
       "      <td>0.15990</td>\n",
       "      <td>0.1974</td>\n",
       "      <td>0.12790</td>\n",
       "      <td>0.2069</td>\n",
       "      <td>0.05999</td>\n",
       "      <td>0.7456</td>\n",
       "      <td>0.7869</td>\n",
       "      <td>4.585</td>\n",
       "      <td>94.03</td>\n",
       "      <td>0.006150</td>\n",
       "      <td>0.04006</td>\n",
       "      <td>0.03832</td>\n",
       "      <td>0.02058</td>\n",
       "      <td>0.02250</td>\n",
       "      <td>0.004571</td>\n",
       "      <td>23.57</td>\n",
       "      <td>25.53</td>\n",
       "      <td>152.50</td>\n",
       "      <td>1709.0</td>\n",
       "      <td>0.1444</td>\n",
       "      <td>0.4245</td>\n",
       "      <td>0.4504</td>\n",
       "      <td>0.2430</td>\n",
       "      <td>0.3613</td>\n",
       "      <td>0.08758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>84348301</td>\n",
       "      <td>M</td>\n",
       "      <td>11.42</td>\n",
       "      <td>20.38</td>\n",
       "      <td>77.58</td>\n",
       "      <td>386.1</td>\n",
       "      <td>0.14250</td>\n",
       "      <td>0.28390</td>\n",
       "      <td>0.2414</td>\n",
       "      <td>0.10520</td>\n",
       "      <td>0.2597</td>\n",
       "      <td>0.09744</td>\n",
       "      <td>0.4956</td>\n",
       "      <td>1.1560</td>\n",
       "      <td>3.445</td>\n",
       "      <td>27.23</td>\n",
       "      <td>0.009110</td>\n",
       "      <td>0.07458</td>\n",
       "      <td>0.05661</td>\n",
       "      <td>0.01867</td>\n",
       "      <td>0.05963</td>\n",
       "      <td>0.009208</td>\n",
       "      <td>14.91</td>\n",
       "      <td>26.50</td>\n",
       "      <td>98.87</td>\n",
       "      <td>567.7</td>\n",
       "      <td>0.2098</td>\n",
       "      <td>0.8663</td>\n",
       "      <td>0.6869</td>\n",
       "      <td>0.2575</td>\n",
       "      <td>0.6638</td>\n",
       "      <td>0.17300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>84358402</td>\n",
       "      <td>M</td>\n",
       "      <td>20.29</td>\n",
       "      <td>14.34</td>\n",
       "      <td>135.10</td>\n",
       "      <td>1297.0</td>\n",
       "      <td>0.10030</td>\n",
       "      <td>0.13280</td>\n",
       "      <td>0.1980</td>\n",
       "      <td>0.10430</td>\n",
       "      <td>0.1809</td>\n",
       "      <td>0.05883</td>\n",
       "      <td>0.7572</td>\n",
       "      <td>0.7813</td>\n",
       "      <td>5.438</td>\n",
       "      <td>94.44</td>\n",
       "      <td>0.011490</td>\n",
       "      <td>0.02461</td>\n",
       "      <td>0.05688</td>\n",
       "      <td>0.01885</td>\n",
       "      <td>0.01756</td>\n",
       "      <td>0.005115</td>\n",
       "      <td>22.54</td>\n",
       "      <td>16.67</td>\n",
       "      <td>152.20</td>\n",
       "      <td>1575.0</td>\n",
       "      <td>0.1374</td>\n",
       "      <td>0.2050</td>\n",
       "      <td>0.4000</td>\n",
       "      <td>0.1625</td>\n",
       "      <td>0.2364</td>\n",
       "      <td>0.07678</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         0  1      2      3       4       5        6        7       8   \\\n",
       "0    842302  M  17.99  10.38  122.80  1001.0  0.11840  0.27760  0.3001   \n",
       "1    842517  M  20.57  17.77  132.90  1326.0  0.08474  0.07864  0.0869   \n",
       "2  84300903  M  19.69  21.25  130.00  1203.0  0.10960  0.15990  0.1974   \n",
       "3  84348301  M  11.42  20.38   77.58   386.1  0.14250  0.28390  0.2414   \n",
       "4  84358402  M  20.29  14.34  135.10  1297.0  0.10030  0.13280  0.1980   \n",
       "\n",
       "        9       10       11      12      13     14      15        16       17  \\\n",
       "0  0.14710  0.2419  0.07871  1.0950  0.9053  8.589  153.40  0.006399  0.04904   \n",
       "1  0.07017  0.1812  0.05667  0.5435  0.7339  3.398   74.08  0.005225  0.01308   \n",
       "2  0.12790  0.2069  0.05999  0.7456  0.7869  4.585   94.03  0.006150  0.04006   \n",
       "3  0.10520  0.2597  0.09744  0.4956  1.1560  3.445   27.23  0.009110  0.07458   \n",
       "4  0.10430  0.1809  0.05883  0.7572  0.7813  5.438   94.44  0.011490  0.02461   \n",
       "\n",
       "        18       19       20        21     22     23      24      25      26  \\\n",
       "0  0.05373  0.01587  0.03003  0.006193  25.38  17.33  184.60  2019.0  0.1622   \n",
       "1  0.01860  0.01340  0.01389  0.003532  24.99  23.41  158.80  1956.0  0.1238   \n",
       "2  0.03832  0.02058  0.02250  0.004571  23.57  25.53  152.50  1709.0  0.1444   \n",
       "3  0.05661  0.01867  0.05963  0.009208  14.91  26.50   98.87   567.7  0.2098   \n",
       "4  0.05688  0.01885  0.01756  0.005115  22.54  16.67  152.20  1575.0  0.1374   \n",
       "\n",
       "       27      28      29      30       31  \n",
       "0  0.6656  0.7119  0.2654  0.4601  0.11890  \n",
       "1  0.1866  0.2416  0.1860  0.2750  0.08902  \n",
       "2  0.4245  0.4504  0.2430  0.3613  0.08758  \n",
       "3  0.8663  0.6869  0.2575  0.6638  0.17300  \n",
       "4  0.2050  0.4000  0.1625  0.2364  0.07678  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>5.690000e+02</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>569.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>3.037183e+07</td>\n",
       "      <td>14.127292</td>\n",
       "      <td>19.289649</td>\n",
       "      <td>91.969033</td>\n",
       "      <td>654.889104</td>\n",
       "      <td>0.096360</td>\n",
       "      <td>0.104341</td>\n",
       "      <td>0.088799</td>\n",
       "      <td>0.048919</td>\n",
       "      <td>0.181162</td>\n",
       "      <td>0.062798</td>\n",
       "      <td>0.405172</td>\n",
       "      <td>1.216853</td>\n",
       "      <td>2.866059</td>\n",
       "      <td>40.337079</td>\n",
       "      <td>0.007041</td>\n",
       "      <td>0.025478</td>\n",
       "      <td>0.031894</td>\n",
       "      <td>0.011796</td>\n",
       "      <td>0.020542</td>\n",
       "      <td>0.003795</td>\n",
       "      <td>16.269190</td>\n",
       "      <td>25.677223</td>\n",
       "      <td>107.261213</td>\n",
       "      <td>880.583128</td>\n",
       "      <td>0.132369</td>\n",
       "      <td>0.254265</td>\n",
       "      <td>0.272188</td>\n",
       "      <td>0.114606</td>\n",
       "      <td>0.290076</td>\n",
       "      <td>0.083946</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.250206e+08</td>\n",
       "      <td>3.524049</td>\n",
       "      <td>4.301036</td>\n",
       "      <td>24.298981</td>\n",
       "      <td>351.914129</td>\n",
       "      <td>0.014064</td>\n",
       "      <td>0.052813</td>\n",
       "      <td>0.079720</td>\n",
       "      <td>0.038803</td>\n",
       "      <td>0.027414</td>\n",
       "      <td>0.007060</td>\n",
       "      <td>0.277313</td>\n",
       "      <td>0.551648</td>\n",
       "      <td>2.021855</td>\n",
       "      <td>45.491006</td>\n",
       "      <td>0.003003</td>\n",
       "      <td>0.017908</td>\n",
       "      <td>0.030186</td>\n",
       "      <td>0.006170</td>\n",
       "      <td>0.008266</td>\n",
       "      <td>0.002646</td>\n",
       "      <td>4.833242</td>\n",
       "      <td>6.146258</td>\n",
       "      <td>33.602542</td>\n",
       "      <td>569.356993</td>\n",
       "      <td>0.022832</td>\n",
       "      <td>0.157336</td>\n",
       "      <td>0.208624</td>\n",
       "      <td>0.065732</td>\n",
       "      <td>0.061867</td>\n",
       "      <td>0.018061</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>8.670000e+03</td>\n",
       "      <td>6.981000</td>\n",
       "      <td>9.710000</td>\n",
       "      <td>43.790000</td>\n",
       "      <td>143.500000</td>\n",
       "      <td>0.052630</td>\n",
       "      <td>0.019380</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.106000</td>\n",
       "      <td>0.049960</td>\n",
       "      <td>0.111500</td>\n",
       "      <td>0.360200</td>\n",
       "      <td>0.757000</td>\n",
       "      <td>6.802000</td>\n",
       "      <td>0.001713</td>\n",
       "      <td>0.002252</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.007882</td>\n",
       "      <td>0.000895</td>\n",
       "      <td>7.930000</td>\n",
       "      <td>12.020000</td>\n",
       "      <td>50.410000</td>\n",
       "      <td>185.200000</td>\n",
       "      <td>0.071170</td>\n",
       "      <td>0.027290</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.156500</td>\n",
       "      <td>0.055040</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>8.692180e+05</td>\n",
       "      <td>11.700000</td>\n",
       "      <td>16.170000</td>\n",
       "      <td>75.170000</td>\n",
       "      <td>420.300000</td>\n",
       "      <td>0.086370</td>\n",
       "      <td>0.064920</td>\n",
       "      <td>0.029560</td>\n",
       "      <td>0.020310</td>\n",
       "      <td>0.161900</td>\n",
       "      <td>0.057700</td>\n",
       "      <td>0.232400</td>\n",
       "      <td>0.833900</td>\n",
       "      <td>1.606000</td>\n",
       "      <td>17.850000</td>\n",
       "      <td>0.005169</td>\n",
       "      <td>0.013080</td>\n",
       "      <td>0.015090</td>\n",
       "      <td>0.007638</td>\n",
       "      <td>0.015160</td>\n",
       "      <td>0.002248</td>\n",
       "      <td>13.010000</td>\n",
       "      <td>21.080000</td>\n",
       "      <td>84.110000</td>\n",
       "      <td>515.300000</td>\n",
       "      <td>0.116600</td>\n",
       "      <td>0.147200</td>\n",
       "      <td>0.114500</td>\n",
       "      <td>0.064930</td>\n",
       "      <td>0.250400</td>\n",
       "      <td>0.071460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>9.060240e+05</td>\n",
       "      <td>13.370000</td>\n",
       "      <td>18.840000</td>\n",
       "      <td>86.240000</td>\n",
       "      <td>551.100000</td>\n",
       "      <td>0.095870</td>\n",
       "      <td>0.092630</td>\n",
       "      <td>0.061540</td>\n",
       "      <td>0.033500</td>\n",
       "      <td>0.179200</td>\n",
       "      <td>0.061540</td>\n",
       "      <td>0.324200</td>\n",
       "      <td>1.108000</td>\n",
       "      <td>2.287000</td>\n",
       "      <td>24.530000</td>\n",
       "      <td>0.006380</td>\n",
       "      <td>0.020450</td>\n",
       "      <td>0.025890</td>\n",
       "      <td>0.010930</td>\n",
       "      <td>0.018730</td>\n",
       "      <td>0.003187</td>\n",
       "      <td>14.970000</td>\n",
       "      <td>25.410000</td>\n",
       "      <td>97.660000</td>\n",
       "      <td>686.500000</td>\n",
       "      <td>0.131300</td>\n",
       "      <td>0.211900</td>\n",
       "      <td>0.226700</td>\n",
       "      <td>0.099930</td>\n",
       "      <td>0.282200</td>\n",
       "      <td>0.080040</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>8.813129e+06</td>\n",
       "      <td>15.780000</td>\n",
       "      <td>21.800000</td>\n",
       "      <td>104.100000</td>\n",
       "      <td>782.700000</td>\n",
       "      <td>0.105300</td>\n",
       "      <td>0.130400</td>\n",
       "      <td>0.130700</td>\n",
       "      <td>0.074000</td>\n",
       "      <td>0.195700</td>\n",
       "      <td>0.066120</td>\n",
       "      <td>0.478900</td>\n",
       "      <td>1.474000</td>\n",
       "      <td>3.357000</td>\n",
       "      <td>45.190000</td>\n",
       "      <td>0.008146</td>\n",
       "      <td>0.032450</td>\n",
       "      <td>0.042050</td>\n",
       "      <td>0.014710</td>\n",
       "      <td>0.023480</td>\n",
       "      <td>0.004558</td>\n",
       "      <td>18.790000</td>\n",
       "      <td>29.720000</td>\n",
       "      <td>125.400000</td>\n",
       "      <td>1084.000000</td>\n",
       "      <td>0.146000</td>\n",
       "      <td>0.339100</td>\n",
       "      <td>0.382900</td>\n",
       "      <td>0.161400</td>\n",
       "      <td>0.317900</td>\n",
       "      <td>0.092080</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>9.113205e+08</td>\n",
       "      <td>28.110000</td>\n",
       "      <td>39.280000</td>\n",
       "      <td>188.500000</td>\n",
       "      <td>2501.000000</td>\n",
       "      <td>0.163400</td>\n",
       "      <td>0.345400</td>\n",
       "      <td>0.426800</td>\n",
       "      <td>0.201200</td>\n",
       "      <td>0.304000</td>\n",
       "      <td>0.097440</td>\n",
       "      <td>2.873000</td>\n",
       "      <td>4.885000</td>\n",
       "      <td>21.980000</td>\n",
       "      <td>542.200000</td>\n",
       "      <td>0.031130</td>\n",
       "      <td>0.135400</td>\n",
       "      <td>0.396000</td>\n",
       "      <td>0.052790</td>\n",
       "      <td>0.078950</td>\n",
       "      <td>0.029840</td>\n",
       "      <td>36.040000</td>\n",
       "      <td>49.540000</td>\n",
       "      <td>251.200000</td>\n",
       "      <td>4254.000000</td>\n",
       "      <td>0.222600</td>\n",
       "      <td>1.058000</td>\n",
       "      <td>1.252000</td>\n",
       "      <td>0.291000</td>\n",
       "      <td>0.663800</td>\n",
       "      <td>0.207500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 0           2           3           4            5   \\\n",
       "count  5.690000e+02  569.000000  569.000000  569.000000   569.000000   \n",
       "mean   3.037183e+07   14.127292   19.289649   91.969033   654.889104   \n",
       "std    1.250206e+08    3.524049    4.301036   24.298981   351.914129   \n",
       "min    8.670000e+03    6.981000    9.710000   43.790000   143.500000   \n",
       "25%    8.692180e+05   11.700000   16.170000   75.170000   420.300000   \n",
       "50%    9.060240e+05   13.370000   18.840000   86.240000   551.100000   \n",
       "75%    8.813129e+06   15.780000   21.800000  104.100000   782.700000   \n",
       "max    9.113205e+08   28.110000   39.280000  188.500000  2501.000000   \n",
       "\n",
       "               6           7           8           9           10          11  \\\n",
       "count  569.000000  569.000000  569.000000  569.000000  569.000000  569.000000   \n",
       "mean     0.096360    0.104341    0.088799    0.048919    0.181162    0.062798   \n",
       "std      0.014064    0.052813    0.079720    0.038803    0.027414    0.007060   \n",
       "min      0.052630    0.019380    0.000000    0.000000    0.106000    0.049960   \n",
       "25%      0.086370    0.064920    0.029560    0.020310    0.161900    0.057700   \n",
       "50%      0.095870    0.092630    0.061540    0.033500    0.179200    0.061540   \n",
       "75%      0.105300    0.130400    0.130700    0.074000    0.195700    0.066120   \n",
       "max      0.163400    0.345400    0.426800    0.201200    0.304000    0.097440   \n",
       "\n",
       "               12          13          14          15          16          17  \\\n",
       "count  569.000000  569.000000  569.000000  569.000000  569.000000  569.000000   \n",
       "mean     0.405172    1.216853    2.866059   40.337079    0.007041    0.025478   \n",
       "std      0.277313    0.551648    2.021855   45.491006    0.003003    0.017908   \n",
       "min      0.111500    0.360200    0.757000    6.802000    0.001713    0.002252   \n",
       "25%      0.232400    0.833900    1.606000   17.850000    0.005169    0.013080   \n",
       "50%      0.324200    1.108000    2.287000   24.530000    0.006380    0.020450   \n",
       "75%      0.478900    1.474000    3.357000   45.190000    0.008146    0.032450   \n",
       "max      2.873000    4.885000   21.980000  542.200000    0.031130    0.135400   \n",
       "\n",
       "               18          19          20          21          22          23  \\\n",
       "count  569.000000  569.000000  569.000000  569.000000  569.000000  569.000000   \n",
       "mean     0.031894    0.011796    0.020542    0.003795   16.269190   25.677223   \n",
       "std      0.030186    0.006170    0.008266    0.002646    4.833242    6.146258   \n",
       "min      0.000000    0.000000    0.007882    0.000895    7.930000   12.020000   \n",
       "25%      0.015090    0.007638    0.015160    0.002248   13.010000   21.080000   \n",
       "50%      0.025890    0.010930    0.018730    0.003187   14.970000   25.410000   \n",
       "75%      0.042050    0.014710    0.023480    0.004558   18.790000   29.720000   \n",
       "max      0.396000    0.052790    0.078950    0.029840   36.040000   49.540000   \n",
       "\n",
       "               24           25          26          27          28  \\\n",
       "count  569.000000   569.000000  569.000000  569.000000  569.000000   \n",
       "mean   107.261213   880.583128    0.132369    0.254265    0.272188   \n",
       "std     33.602542   569.356993    0.022832    0.157336    0.208624   \n",
       "min     50.410000   185.200000    0.071170    0.027290    0.000000   \n",
       "25%     84.110000   515.300000    0.116600    0.147200    0.114500   \n",
       "50%     97.660000   686.500000    0.131300    0.211900    0.226700   \n",
       "75%    125.400000  1084.000000    0.146000    0.339100    0.382900   \n",
       "max    251.200000  4254.000000    0.222600    1.058000    1.252000   \n",
       "\n",
       "               29          30          31  \n",
       "count  569.000000  569.000000  569.000000  \n",
       "mean     0.114606    0.290076    0.083946  \n",
       "std      0.065732    0.061867    0.018061  \n",
       "min      0.000000    0.156500    0.055040  \n",
       "25%      0.064930    0.250400    0.071460  \n",
       "50%      0.099930    0.282200    0.080040  \n",
       "75%      0.161400    0.317900    0.092080  \n",
       "max      0.291000    0.663800    0.207500  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore the target variable\n",
    "\n",
    "Let's explore the dependent variable. Its a string column, referring to the 'M' and 'B' targt categories. Before feeding the data in a Neural Network, it will require one-hot encoding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'M', 'B'}\n",
      "no. classes: 2\n"
     ]
    }
   ],
   "source": [
    "print(set(df[df.columns[1]]))\n",
    "print('no. classes: ' + str(len(set(df[df.columns[1]]))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(569, 2)\n"
     ]
    }
   ],
   "source": [
    "# One-hot encoding\n",
    "classification = pd.get_dummies(df[df.columns[1]])\n",
    "print(classification.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I obtained a target dataframe, called `classification`, that contains the one-hot encoded version of my dependent binary variable. Now I can isolate the explanatory variables in my dataframe `df`. In order to do that I'll drop columns 0 and 1: the first is an index, the second is the target variable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>17.99</td>\n",
       "      <td>10.38</td>\n",
       "      <td>122.80</td>\n",
       "      <td>1001.0</td>\n",
       "      <td>0.11840</td>\n",
       "      <td>0.27760</td>\n",
       "      <td>0.3001</td>\n",
       "      <td>0.14710</td>\n",
       "      <td>0.2419</td>\n",
       "      <td>0.07871</td>\n",
       "      <td>1.0950</td>\n",
       "      <td>0.9053</td>\n",
       "      <td>8.589</td>\n",
       "      <td>153.40</td>\n",
       "      <td>0.006399</td>\n",
       "      <td>0.04904</td>\n",
       "      <td>0.05373</td>\n",
       "      <td>0.01587</td>\n",
       "      <td>0.03003</td>\n",
       "      <td>0.006193</td>\n",
       "      <td>25.38</td>\n",
       "      <td>17.33</td>\n",
       "      <td>184.60</td>\n",
       "      <td>2019.0</td>\n",
       "      <td>0.1622</td>\n",
       "      <td>0.6656</td>\n",
       "      <td>0.7119</td>\n",
       "      <td>0.2654</td>\n",
       "      <td>0.4601</td>\n",
       "      <td>0.11890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20.57</td>\n",
       "      <td>17.77</td>\n",
       "      <td>132.90</td>\n",
       "      <td>1326.0</td>\n",
       "      <td>0.08474</td>\n",
       "      <td>0.07864</td>\n",
       "      <td>0.0869</td>\n",
       "      <td>0.07017</td>\n",
       "      <td>0.1812</td>\n",
       "      <td>0.05667</td>\n",
       "      <td>0.5435</td>\n",
       "      <td>0.7339</td>\n",
       "      <td>3.398</td>\n",
       "      <td>74.08</td>\n",
       "      <td>0.005225</td>\n",
       "      <td>0.01308</td>\n",
       "      <td>0.01860</td>\n",
       "      <td>0.01340</td>\n",
       "      <td>0.01389</td>\n",
       "      <td>0.003532</td>\n",
       "      <td>24.99</td>\n",
       "      <td>23.41</td>\n",
       "      <td>158.80</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>0.1238</td>\n",
       "      <td>0.1866</td>\n",
       "      <td>0.2416</td>\n",
       "      <td>0.1860</td>\n",
       "      <td>0.2750</td>\n",
       "      <td>0.08902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>19.69</td>\n",
       "      <td>21.25</td>\n",
       "      <td>130.00</td>\n",
       "      <td>1203.0</td>\n",
       "      <td>0.10960</td>\n",
       "      <td>0.15990</td>\n",
       "      <td>0.1974</td>\n",
       "      <td>0.12790</td>\n",
       "      <td>0.2069</td>\n",
       "      <td>0.05999</td>\n",
       "      <td>0.7456</td>\n",
       "      <td>0.7869</td>\n",
       "      <td>4.585</td>\n",
       "      <td>94.03</td>\n",
       "      <td>0.006150</td>\n",
       "      <td>0.04006</td>\n",
       "      <td>0.03832</td>\n",
       "      <td>0.02058</td>\n",
       "      <td>0.02250</td>\n",
       "      <td>0.004571</td>\n",
       "      <td>23.57</td>\n",
       "      <td>25.53</td>\n",
       "      <td>152.50</td>\n",
       "      <td>1709.0</td>\n",
       "      <td>0.1444</td>\n",
       "      <td>0.4245</td>\n",
       "      <td>0.4504</td>\n",
       "      <td>0.2430</td>\n",
       "      <td>0.3613</td>\n",
       "      <td>0.08758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11.42</td>\n",
       "      <td>20.38</td>\n",
       "      <td>77.58</td>\n",
       "      <td>386.1</td>\n",
       "      <td>0.14250</td>\n",
       "      <td>0.28390</td>\n",
       "      <td>0.2414</td>\n",
       "      <td>0.10520</td>\n",
       "      <td>0.2597</td>\n",
       "      <td>0.09744</td>\n",
       "      <td>0.4956</td>\n",
       "      <td>1.1560</td>\n",
       "      <td>3.445</td>\n",
       "      <td>27.23</td>\n",
       "      <td>0.009110</td>\n",
       "      <td>0.07458</td>\n",
       "      <td>0.05661</td>\n",
       "      <td>0.01867</td>\n",
       "      <td>0.05963</td>\n",
       "      <td>0.009208</td>\n",
       "      <td>14.91</td>\n",
       "      <td>26.50</td>\n",
       "      <td>98.87</td>\n",
       "      <td>567.7</td>\n",
       "      <td>0.2098</td>\n",
       "      <td>0.8663</td>\n",
       "      <td>0.6869</td>\n",
       "      <td>0.2575</td>\n",
       "      <td>0.6638</td>\n",
       "      <td>0.17300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>20.29</td>\n",
       "      <td>14.34</td>\n",
       "      <td>135.10</td>\n",
       "      <td>1297.0</td>\n",
       "      <td>0.10030</td>\n",
       "      <td>0.13280</td>\n",
       "      <td>0.1980</td>\n",
       "      <td>0.10430</td>\n",
       "      <td>0.1809</td>\n",
       "      <td>0.05883</td>\n",
       "      <td>0.7572</td>\n",
       "      <td>0.7813</td>\n",
       "      <td>5.438</td>\n",
       "      <td>94.44</td>\n",
       "      <td>0.011490</td>\n",
       "      <td>0.02461</td>\n",
       "      <td>0.05688</td>\n",
       "      <td>0.01885</td>\n",
       "      <td>0.01756</td>\n",
       "      <td>0.005115</td>\n",
       "      <td>22.54</td>\n",
       "      <td>16.67</td>\n",
       "      <td>152.20</td>\n",
       "      <td>1575.0</td>\n",
       "      <td>0.1374</td>\n",
       "      <td>0.2050</td>\n",
       "      <td>0.4000</td>\n",
       "      <td>0.1625</td>\n",
       "      <td>0.2364</td>\n",
       "      <td>0.07678</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      2      3       4       5        6        7       8        9       10  \\\n",
       "0  17.99  10.38  122.80  1001.0  0.11840  0.27760  0.3001  0.14710  0.2419   \n",
       "1  20.57  17.77  132.90  1326.0  0.08474  0.07864  0.0869  0.07017  0.1812   \n",
       "2  19.69  21.25  130.00  1203.0  0.10960  0.15990  0.1974  0.12790  0.2069   \n",
       "3  11.42  20.38   77.58   386.1  0.14250  0.28390  0.2414  0.10520  0.2597   \n",
       "4  20.29  14.34  135.10  1297.0  0.10030  0.13280  0.1980  0.10430  0.1809   \n",
       "\n",
       "        11      12      13     14      15        16       17       18  \\\n",
       "0  0.07871  1.0950  0.9053  8.589  153.40  0.006399  0.04904  0.05373   \n",
       "1  0.05667  0.5435  0.7339  3.398   74.08  0.005225  0.01308  0.01860   \n",
       "2  0.05999  0.7456  0.7869  4.585   94.03  0.006150  0.04006  0.03832   \n",
       "3  0.09744  0.4956  1.1560  3.445   27.23  0.009110  0.07458  0.05661   \n",
       "4  0.05883  0.7572  0.7813  5.438   94.44  0.011490  0.02461  0.05688   \n",
       "\n",
       "        19       20        21     22     23      24      25      26      27  \\\n",
       "0  0.01587  0.03003  0.006193  25.38  17.33  184.60  2019.0  0.1622  0.6656   \n",
       "1  0.01340  0.01389  0.003532  24.99  23.41  158.80  1956.0  0.1238  0.1866   \n",
       "2  0.02058  0.02250  0.004571  23.57  25.53  152.50  1709.0  0.1444  0.4245   \n",
       "3  0.01867  0.05963  0.009208  14.91  26.50   98.87   567.7  0.2098  0.8663   \n",
       "4  0.01885  0.01756  0.005115  22.54  16.67  152.20  1575.0  0.1374  0.2050   \n",
       "\n",
       "       28      29      30       31  \n",
       "0  0.7119  0.2654  0.4601  0.11890  \n",
       "1  0.2416  0.1860  0.2750  0.08902  \n",
       "2  0.4504  0.2430  0.3613  0.08758  \n",
       "3  0.6869  0.2575  0.6638  0.17300  \n",
       "4  0.4000  0.1625  0.2364  0.07678  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.drop([df.columns[0], df.columns[1]], axis=1)   # drop the target variable from df\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(569, 30)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now the shape is:\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before training the model, I turn both explanatory and target data into numpy objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.values\n",
    "classification = classification.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# in this case, I uniform the datatypes to float64\n",
    "classification = classification.astype(np.float64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train-Test split\n",
    "\n",
    "In an actual ML job, you would split your dataset in **Train**, **Validation** and **Test**sets. However, this is just an example on how to implement and run a Neural Network, so I'll skip that part and will split the data in train and test only."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train shape: (426, 30)\n",
      "y_train shape: (426, 2)\n",
      "\n",
      "X_test shape: (143, 30)\n",
      "y_test shape: (143, 2)\n"
     ]
    }
   ],
   "source": [
    "## TRAIN-TEST SPLIT\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(df, classification, test_size=0.25, random_state=173)\n",
    "\n",
    "print('X_train shape: ' + str(X_train.shape) + '\\ny_train shape: ' + str(y_train.shape))\n",
    "print('\\nX_test shape: ' + str(X_test.shape) + '\\ny_test shape: ' + str(y_test.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scaling the variables must happen after the train-test split. That is because the test set must be scaled using the parameters of the training set: in real world cases you don't know what data you'll get from training, therefore this is the only way to truly understand the performance of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Scale the variables using Z-scores\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "scaler = StandardScaler()\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Neural Network architecture\n",
    "\n",
    "Since the network is not very deep, and the number of parameters is relatively small, I can employ more \"demanding\" (and performing) activation functions. In this case, I choose **ELU** (**Exponential Linear Unit**) activations. A **softmax** function is then applied at the end, so that the attribution of classes (M/B) is shrinked into probabilities.\n",
    "\n",
    "Additionally, I apply **dropout** in order to prevent overfitting. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import Dense, Dropout\n",
    "from tensorflow.keras.activations import elu, softmax\n",
    "\n",
    "# Architecture\n",
    "n_input = X_train.shape[1]\n",
    "n_hidden1 = 30\n",
    "n_hidden2 = 20\n",
    "n_hidden3 = 15\n",
    "n_output = y_train.shape[1]\n",
    "\n",
    "# set dropout probability\n",
    "dropout_prob = 0.1\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The definition of a model in TensorFlow 2.0 follows the syntax of Keras' `Sequential()` models.\n",
    "\n",
    "Each layer is defined by the `Dense()` function, taking as inputs: the previous layer, the number of nodes, and the activation function (it can actually take a lor of additional arguments, but I'll not review them here). The input layer also requires a definition of the input data shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = tf.keras.models.Sequential([\n",
    "    Dense(n_input, input_shape = (n_input,), activation = elu),   # Input layer\n",
    "    \n",
    "    Dense(n_hidden1, activation = elu), # hidden layer 1\n",
    "    Dropout(dropout_prob),     \n",
    "    \n",
    "    Dense(n_hidden2, activation = elu), # hidden layer 2\n",
    "    Dropout(dropout_prob), \n",
    "    \n",
    "    Dense(n_hidden3, activation = elu), # hidden layer 3\n",
    "    Dropout(dropout_prob), \n",
    "    \n",
    "    Dense(n_output, activation = softmax)  # Output layer\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 30)                930       \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 30)                930       \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 30)                0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 20)                620       \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 20)                0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 15)                315       \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 15)                0         \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 2)                 32        \n",
      "=================================================================\n",
      "Total params: 2,827\n",
      "Trainable params: 2,827\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Implementation of full-Batch Gradient Descent\n",
    "\n",
    "This is Gradient Descent in its simplest form, in which the whole bunch (erhm, batch) of training data is fed into the network at each iteration.\n",
    "\n",
    "(In a following Notebook I will show the implementation of a more powerful technique: **Mini-Batch Gradient Descent**.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to train the model, I neeed to define a **loss function** (that Gradient Descent will minimize), an **accuracy metrics** (in oreder to monitor the model's performance through the epochs) and an **optimization algorithm**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loss: Binary cross-entropy is made for \n",
    "bce_loss = tf.keras.losses.BinaryCrossentropy()\n",
    "\n",
    "# Binary Accuracy (expressed in the [0,1] interval)\n",
    "accuracy = tf.keras.metrics.BinaryAccuracy()\n",
    "\n",
    "# Adam Optimizer (what you'll need 99.99% of the time)\n",
    "optimizer = tf.optimizers.Adam(learning_rate = 0.001)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are at least two ways to train a Neural Network in TensorFlow 2.0. The first is using Keras, by calling:\n",
    "\n",
    "    model.compile(optimizer, loss, metrics)\n",
    "\n",
    "The other is to use pure TensorFlow's **eager execution method**, which is what I'll do here. In TensorFlow 1.x what you had to do was to create a computational graph (with placeholders instead of actual data), and then running it in a tf.Session(). Eager execution instead is the TensorFlow's variant of **imperative programming**: operations are evaluated immediately without building graphs, and run instantly. Eager execution makes code is easier to debug, and your whole script much less verbose and easier to read (now it looks pretty much like canonical Python). Another reason why Google's engineers changed TensorFlow so profoudly is that symbolic programming was not so popular (and they want TensorFlow to keep competing against pyTorch, which gained a lot in popularity recently).\n",
    "\n",
    "The reason why I employ eager execution is that I think it gives you more control on model training, and a better control of the output. Also, it forces you to better understand how training a Neural Network works."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The fundamental element of training a Network in eager execution is represented by `tf.GradientTape()`. This object calculates and stores the gradient of the loss function at each iteration of the training operation. Once you generate a GradientTape, you can call the `.gradient()` argument to get the actual gradient (i.e. the first derivative of the loss function). Later, you feed this values into an `optimizer` using the `.apply_gradients` argument that updates the Network's trainable variables (the very act of \"learning\")."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100.\tTraining Loss: 0.3144478499889374,\tAccuracy: 0.9882629\n",
      "200.\tTraining Loss: 0.18320266902446747,\tAccuracy: 0.9906103\n",
      "300.\tTraining Loss: 0.1586928367614746,\tAccuracy: 0.9906103\n",
      "400.\tTraining Loss: 0.15164142847061157,\tAccuracy: 0.9906103\n",
      "500.\tTraining Loss: 0.12006960064172745,\tAccuracy: 0.9929578\n",
      "600.\tTraining Loss: 0.11340983211994171,\tAccuracy: 0.9929578\n",
      "700.\tTraining Loss: 0.1114405170083046,\tAccuracy: 0.9929578\n",
      "800.\tTraining Loss: 0.11051212251186371,\tAccuracy: 0.9929578\n",
      "900.\tTraining Loss: 0.10998599976301193,\tAccuracy: 0.9929578\n",
      "1000.\tTraining Loss: 0.10965467244386673,\tAccuracy: 0.9929578\n",
      "\n",
      "Training complete.\n"
     ]
    }
   ],
   "source": [
    "###  TRAINING\n",
    "\n",
    "# save loss and accuracy improvements through epochs\n",
    "loss_history = []\n",
    "accuracy_history = []\n",
    "\n",
    "# iterate for 1000 epochs\n",
    "for epoch in range(1000):\n",
    "    \n",
    "    # GratientTape is what I need in order to calculate gradients of the loss function\n",
    "    with tf.GradientTape() as tape:\n",
    "        # take current binary cross-entropy (bce_loss)\n",
    "        current_loss = bce_loss(model(X_train), y_train)\n",
    "    \n",
    "    # HERE THE ACTUAL TRAINING HAPPENS:\n",
    "    # Update weights based on the gradient of the loss function\n",
    "    gradients = tape.gradient(current_loss, model.trainable_variables)    # get the gradient of the loss function\n",
    "    optimizer.apply_gradients(zip(gradients, model.trainable_variables))  # update the weights\n",
    "    \n",
    "    # save current loss in its history vector\n",
    "    loss_history.append(current_loss.numpy())\n",
    "    \n",
    "    # save current accuracy in its history vector\n",
    "    accuracy.update_state(y_train, model(X_train))  # this computes the accuracy and stores it\n",
    "    current_accuracy = accuracy.result().numpy()  # save its result as numpy object\n",
    "    accuracy_history.append(current_accuracy)\n",
    "    \n",
    "    # In order to monitor progress, I will print loss and accuracy scores every 100 epochs\n",
    "    if (epoch+1) % 100 == 0:\n",
    "        print(str(epoch+1) + '.\\tTraining Loss: ' + str(current_loss.numpy()) + ',\\tAccuracy: ' + str(current_accuracy))\n",
    "    \n",
    "    accuracy.reset_states()  # reset the state of accuracy object for next iteration\n",
    "#\n",
    "print('\\nTraining complete.')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Visualization\n",
    "\n",
    "Once the training is done, let's check the model's improvement visually:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3IAAAEWCAYAAAAjLaWEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmYZGV58P/v3Xv3bMzAsM0AAwroSFR0VFyj4gLG4BoDcddI8nvdlxh5YzQak5jEJGpCjIhEVBSNKzHEDbfXBWRARBbRAQWGdYDZmKWnu+v+/VGnZ2p6arqrp6u6pk5/P9dVV9c59dQ5d50p6uE+zxaZiSRJkiSpc3S1OwBJkiRJ0vSYyEmSJElShzGRkyRJkqQOYyInSZIkSR3GRE6SJEmSOoyJnCRJkiR1GBM5aT8SES+OiG+2Ow5JksoiIu6PiGPaHYfUbCZymnMi4rcR8bQ2nPcVEfHDyeLJzAsy8xkNHOsTEfG+VsQpSSqHiPheRKyPiP52x9IqEZER8cAJ+/4qIj49vp2Z8zPzpimO8+SIWNuqOKVWMJGT5qCI6G53DJKk1omIFcATgQROm+Vz98zm+fYHc/Ezq/1M5KQaEfGaiFgTEfdFxEURcXixPyLiXyLi7ojYGBFXR8QJxWvPiojrImJzRNwWEW+bwfl3ttrt7ZwRcSbwYuDtRXeR/y7KP7i4+7ohIq6NiNNqjvuJiPhIRFwcEVuAt0TEXbUVT0S8ICKu2tfYJUn7lZcBlwKfAF5e+0JEDEbEP0XEzUX98sOIGCxee0JE/LioS26NiFcU+78XEX9cc4zdepkULWOvjYhfA78u9n2oOMamiLgiIp5YU747Iv5vRNxY1J9XRMQREXF2RPzThHj/OyLetK8XorbVrl6dHRHzgP8FDi/q1fsj4vCI6I+ID0bE7cXjg+Otm+MteBHx5xFxJ/CfEXFNRPx+zXl7I+KeiHj4vsYuTcZETipExFOBvwNeBBwG3AxcWLz8DOBJwHHAAcAfAvcWr30c+JPMXACcAHynSSHVPWdmngNcAPxD0V3k9yOiF/hv4JvAwcDrgQsi4via4/0R8DfAAuBfi/ifXvP6S4BPNSl2SVJ7vYxqXXEB8MyIOKTmtQ8AjwQeBywB3g5UIuJIqgnNvwJLgYcD07nB91zgMcDKYvvy4hhLgM8A/xURA8VrbwHOAJ4FLAReBWwFzgfOiIgugIg4CDgZ+Ow04pjMHnV2Zm4BTgVuL+rV+Zl5O/AXwEnFZ3gY8GjgnTXHOrT4bEcBZwKfpFqXjnsWcEdmepNULWEiJ+3yYuC8zLwyM4eBs4DHFt1TRqgmQA8CIjOvz8w7iveNACsjYmFmrs/MKyc5x0nFXc6dD+DIvZSd7Jx7HBeYD7w/M3dk5neAr1GtJMd9NTN/lJmVzNxOtbJ8CUBELAGeSbWilSR1sIh4AtXk4vOZeQVwI9WbeRQJ0quAN2bmbZk5lpk/Luq9FwPfzszPZuZIZt47zSTk7zLzvszcBpCZny6OMZqZ/wT0A+M3GP8YeGdm3pBVPy/K/hTYSDV5Azgd+F5m3jXJea+cUK++Y5Ky06mzXwy8NzPvzsx1wHuAl9a8XgHenZnDxWf+NPCsiFhYvP5SvEGqFjKRk3Y5nGorHACZeT/VVqtlRWL0b8DZwF0RcU7ND/ULqN51uzkivh8Rj53kHJdm5gG1D+CWegWnOGe92G/NzErNvpuBZTXbt054z6eB34+I+VRbIf/fJImiJKlzvBz4ZmbeU2x/hl3dKw8CBqgmdxMdsZf9jdqtnomIt0bE9UX3zQ3AouL8U51r541GGust8ogJ9er7Jyk7nTp7t/8vKJ4fXrO9rrgxCkDRivcj4AURcQDVVr4Lpohd2mcmctIut1O9gwlA0Wf+QOA2gMz8cGY+EngI1e6Of1bsvzwzn0O1S+NXgM83K6C9nZPq4PWJsR8x3hWlcOR47PXek5m3AT8Bnod3DSWpFIqxbi8Cfjci7izGb70ZeFhEPAy4B9gOPKDO22/dy36ALcBQzfahdcrsrGeK8XB/XsSyuEiwNgLRwLk+DTyniPfBVOvWppikzp5Yr8KE/y+gWq/eXnu4Ou8ZT0L/APhJUddKLWEip7mqNyIGah49VO9YvjIiHl4MZv5b4LLM/G1EPCoiHlOMRdtCtRIci4i+qK79tigzR4BNwFgzAtzbOYuX7wJq18S5rCjz9mJw9ZOB32fXGL+9+STVsRG/A3y5GXFLktrquVTripVUx3Y9nGoy9P+AlxU9N84D/rmY0KM7Ih5b1HsXAE+LiBdFRE9EHFgzUcdVwPMjYqiYOOTVU8SxABgF1gE9EfEuqmPhxp0L/HVEHBtVD42IAwEycy3V8XWfAr443lVzpqaos+8CDoyIRTVv+SzwzohYWozVexfVJHMyXwEeAbyRah0rtYyJnOaqi4FtNY+/ysxLgL8EvgjcQfVO4elF+YXAx4D1VLtW3Et1sDhUW7N+GxGbgD9l94HOMzHZOT9OtY//hoj4SmbuoDq99KlU77b+O9UK+5dTnOPLVO82frkY7C1J6mwvB/4zM2/JzDvHH1S76r+4uHH5NuAXVJOl+4C/B7oy8xaq3Q7fWuy/iuokHwD/AuygmvCcz9RdBr9BdeKUX1Gtw7aze9fLf6baGvZNqgnVx4HBmtfPp3qTsdm9RerW2UV9+VngpqJuPRx4H7AauJrq9bqy2LdXRdL5ReBo4EtNjl3aTWTWaxWWNFdExI1UZ/D6drtjkSQJICKeRLX1a8WE8d/7vaL18bjMbNaNXakuFy+U5rCIeAHVPv7NWjJBkqQZKYYUvBE4twOTuCVUu52+dKqy0kzZtVKaoyLie8BHgNd2WkUpSSqniHgwsIHqeq4fbHM40xIRr6HaffR/M/MH7Y5H5WfXSkmSJEnqMLbISZIkSVKH2a/GyB100EG5YsWKdochSWqxK6644p7MXNruODqF9aMkzR2N1pH7VSK3YsUKVq9e3e4wJEktFhE3tzuGTmL9KElzR6N1pF0rJUmSJKnDmMhJkiRJUocxkZMkSZKkDmMiJ0lSk0TEeRFxd0Rcs5fXIyI+HBFrIuLqiHjEbMcoSSoHEzlJkprnE8Apk7x+KnBs8TgT+MgsxCRJKiETOUmSmiQzfwDcN0mR5wCfzKpLgQMi4rDZiU6SVCYmcpIkzZ5lwK0122uLfXuIiDMjYnVErF63bt2sBCdJ6hz71TpyM/WxH9zEEUuGOOWEQ9sdiiRJ9USdfVmvYGaeA5wDsGrVqrplpP3VWCX5zx/9hk3bRtodijSrXn/ysfR2z05bWakSuU/8+Lc85pglJnKSpP3VWuCImu3lwO1timW/lpn8zf9cz2/u2dLuULQPfnPPFm4q/u2i3u0LqaT+z1MeSG/37JyrVIncQG8XwyOVdochSdLeXAS8LiIuBB4DbMzMO9ocU1Ns3THK1h1ju+1bu34bb7rwZ2wbGdvLu/Zu8/bq8RYM9HDUgUPNClOzZKi/m9/7ncP48Bkn0t1lJie1QqkSucG+7n2qLCRJaoaI+CzwZOCgiFgLvBvoBcjM/wAuBp4FrAG2Aq9sT6TNc+Ut6/n1XZs560u/oLKXDqCnPexwhvqmf4t6wUAPb33G8QzM1u1tSeogpUrkBnq62W4iJ0lqk8w8Y4rXE3jtLIXTEtt2jHHBZTezdv021m0e5n9+satB8fVPfSAHL+jfrfwxS+fz+AceNNthSlLplSqRG+zr5v7h0XaHIUlSKWUmrz7/cn5847079x0w1Mu/v/gRHL5okBUHzWtjdJI0t5Qqkevv6eae+3e0OwxJkkrnx2vu4Y2fu4p1m4d53AMO5LBFgzz66MX8wSOPoMsxUJI060qVyA322bVSkqRm2jFa4QPfvIFzfnATAC896Sj+4vce7Lg1SWqzUiVyAz1dJnKSJDXJFTffx/v+53p+dssG5vV186k/fgyPOHJxu8OSJFGyRM5ZKyVJap7XfeZn3LFxOw874gA+/vJVHDS/f+o3SZJmRakSuYFeu1ZKktQMX73qNu7YuJ13nPog/uRJxxCu6ixJ+5WudgfQTNVErkJ1dmdJkrSvPvHj3wLwvBOXmcRJ0n6oZC1y1bx0eLTiIGxJkvbB2vVbOfu7N/KzWzZw2sMO55CFA+0OSZJUR6kSucEiedu2Y8xETpKkffD2L1zNj2+8l6G+bt757Ae3OxxJ0l6UM5EbGcM5tSRJmp4LLruZH994L288+Vj+vyc/wJuikrQfK90YOcAJTyRJ2gffuPYuAF71+KNN4iRpP1fKRM4lCCRJmr7b1m/l1BMOZdFQb7tDkSRNoaWJXES8OSKujYhrIuKzEdHSEdPjk51sH6m08jSSJJVOZnLbhm0sO2Cw3aFIkhrQskQuIpYBbwBWZeYJQDdweqvOB7vGyNm1UpKk6bl3yw62j1RYtthETpI6Qau7VvYAgxHRAwwBt7fyZI6RkyRp39y2fhsAyxcPtTkSSVIjWpbIZeZtwAeAW4A7gI2Z+c2J5SLizIhYHRGr161bN6NzDvY5Rk6SpH1xw52bAVhui5wkdYRWdq1cDDwHOBo4HJgXES+ZWC4zz8nMVZm5aunSpTM650DPeIucY+QkSZqOC356C8csncfxhyxodyiSpAa0smvl04DfZOa6zBwBvgQ8roXnY6Cv+nFskZMkqXFX3bqBn9+6gVc8bgVdXdHucCRJDWhlIncLcFJEDEVEACcD17fwfDvHyA2byEmS1LCLf3EHEfDcE5e1OxRJUoNaOUbuMuALwJXAL4pzndOq88GuWSu37TCRkySpEZu2j3DOD27i6IPmsXDA9eMkqVP0tPLgmflu4N2tPEet3u4uuruC7aMmcpIkTSUzeddXrgHgpScd1eZoJEnT0erlB2bdYG8323Y42YkkSVO59b5tfOWq6spAL3zk8jZHI0majtIlcgO9XU52IklSA27fWF077tOvfgwL7FYpSR2lhIlct5OdSJLUgLs2bQfg0EX9bY5EkjRdpUvkBnu7bZGTJKkBt963FYBDF7kIuCR1mtIlcgO93Ww3kZMktUlEnBIRN0TEmoh4R53Xj4qISyLi6oj4XkS0ZXDa6FiFD39nDcccNI/5/S2d+0yS1AKlS+RskZMktUtEdANnA6cCK4EzImLlhGIfAD6ZmQ8F3gv83exGWXXTPVvYMVrhuEMWtOP0kqQZKl0i19/bxfYRZ62UJLXFo4E1mXlTZu4ALgSeM6HMSuCS4vl367w+KzZtGwHgxScd2Y7TS5JmqHSJ3KBdKyVJ7bMMuLVme22xr9bPgRcUz58HLIiIAyceKCLOjIjVEbF63bp1TQ900/ZqIuci4JLUmUqXyDlGTpLURlFnX07YfhvwuxHxM+B3gduA0T3elHlOZq7KzFVLly5teqCbtlVPuWDA8XGS1IlK9+vtGDlJUhutBY6o2V4O3F5bIDNvB54PEBHzgRdk5sZZi7Cws0Vu0BY5SepEJWyRc4ycJKltLgeOjYijI6IPOB24qLZARBwUEeP171nAebMcIwB3bxoGbJGTpE5VvkSuzxY5SVJ7ZOYo8DrgG8D1wOcz89qIeG9EnFYUezJwQ0T8CjgE+Jt2xPrt6+9i1VGL6e/pbsfpJUkzVLrbcAM93ewYrVCpJF1d9YYqSJLUOpl5MXDxhH3vqnn+BeALsx3XRGvXb+OFj2zLEnaSpCYoXYvcYF/1zuL2UVvlJEmqZ/P2Ee4fHuXQRQPtDkWStI9Kl8gN9FQ/kuPkJEmq765ifNyhC03kJKlTlS6RG2+Rc5ycJEn1bdi6A4Al8/raHIkkaV+VLpEb6C0SuR0mcpIk1ePSA5LU+UqbyLkouCRJ9W3e7mLgktTpSpfIDZrISZI0qU3biha5AVvkJKlTlS6R29Ui52QnkiTVs8kWOUnqeKVL5MZb5JzsRJKk+jZuG6G/p2vnzU9JUucpXSI30Du+/ICJnCRJ9dy5cbtryElShythImeLnCRJk7lz43YOcQ05SepopU3khk3kJEmq667NJnKS1OmmTOQiYnVEvDYiFs9GQDPlguCSJE1u07YRDnANOUnqaI20yJ0OHA5cHhEXRsQzIyJaHNc+G+gZHyPnrJWSJE2Umdw/PMp8Z6yUpI42ZSKXmWsy8y+A44DPAOcBt0TEeyJiSasDnK6e7i56u8MWOUmS6hgerTAylszvN5GTpE7W0Bi5iHgo8E/APwJfBF4IbAK+07rQ9t1AT7ezVkqSVMeWYdeQk6QymPJXPCKuADYAHwfekZnDxUuXRcTjWxncvhroM5GTJKme+4tEzhY5SepsjfyK/0Fm3lTvhcx8fpPjaYqB3i7HyEmSVMfm7SZyklQGjXSt3BgRH46IKyPiioj4UEQc2PLIZmCwt5utO0bbHYYkSfud8THk47M8S5I6UyOJ3IXAOuAFVMfGrQM+18qgZmqwt5tttshJkrSH4aJ+HF93VZLUmRrpV7EkM/+6Zvt9EfHcVgXUDIN93WyzRU6SpD0Mj1Zb5Pp7GprvTJK0n2rkV/y7EXF6RHQVjxcB/9PqwGZiXl8PW3c42YkkSRMNj1Zb5Pp7bJGTpE7WSCL3J1TXj9tRPC4E3hIRmyNiUyuD21fVFjkTOUmSJrJFTpLKYcqulZm5YDYCaaahvm5b5CRJqsMxcpJUDg3NPRwRpwFPKja/l5lfa/B9BwDnAicACbwqM3+yL4FOx1Bfj7NWSpJUx66ulbbISVIna2RB8PcDjwIuKHa9MSKekJnvaOD4HwK+npkvjIg+YGjfQ23coC1ykiTVtbNrZa+JnCR1skZ+xZ8FPD0zz8vM84BTin2TioiFVFvxPg6QmTsyc8NMgm3UvL5uRivJjlGXIJAkza6IOCUiboiINRGxx03PiDgyIr4bET+LiKsjYso6tZnGu1b2dZvISVIna/RX/ICa54safM8xVNec+8+isjo3IuZNLBQRZ0bE6ohYvW7dugYPPbnBvmpDoxOeSJJmU0R0A2cDpwIrgTMiYuWEYu8EPp+ZJwKnA/8+mzEOj1bo6Qp6TOQkqaM18iv+d8DPIuITEXE+cAXwtw28rwd4BPCRorLaAuxxZzIzz8nMVZm5aunSpdMIfe+G+qoDuLeOOE5OkjSrHg2sycybMnN8pufnTCiTwMLi+SLg9lmMj+HRMcfHSVIJTDpGLiIC+CFwEtVxcgH8eWbe2cCx1wJrM/OyYvsL1EnkWmFnImeLnCRpdi0Dbq3ZXgs8ZkKZvwK+GRGvB+YBT5ud0KqGRyv0O2OlJHW8SW/JZWYCX8nMOzLzosz8aoNJHEW5WyPi+GLXycB1Mwu3MYNFBbV12EROkjSros6+nLB9BvCJzFxOdcz5pyJij/q4FUMPoDpGzhY5Sep8jfySXxoRj9rH478euCAirgYeTmNdMmdsXn+1odElCCRJs2wtcETN9nL27Dr5auDzAMWSPAPAQRMP1IqhB2DXSkkqi0bWkXsK8CcRcTPVcW5BtbHuoVO9MTOvAlbNLMTpG9w5Rs4WOUnSrLocODYijgZuozqZyR9NKHML1V4qn4iIB1NN5JrX5DaF4dEK/T12rZSkTtdIIndqy6NosvExcs5aKUmaTZk5GhGvA74BdAPnZea1EfFeYHVmXgS8FfhYRLyZarfLVxRDGWbF9pEx15CTpBJoJJF7X2a+tHZHRHwKeOleyrfdUO9410oTOUnS7MrMi4GLJ+x7V83z64DHz3Zc46otciZyktTpGvklf0jtRrFGziNbE05zDPWPz1rpGDlJkmrZtVKSymGviVxEnBURm4GHRsSm4rEZuBv46qxFuA9cfkCSpPqc7ESSymGvv+SZ+XeZuQD4x8xcWDwWZOaBmXnWLMY4bQM9JnKSJNUzPFJxjJwklcCUY+Qy86yIWAYcVVs+M3/QysBmoqsrGOztZptdKyVJ2o1dKyWpHKZM5CLi/VSnT74OGG/iSmC/TeSg2r3SFjlJknZn10pJKodGZq18HnB8Zg63OphmGurvdvkBSZImcNZKSSqHRn7JbwJ6Wx1Isw319rDFrpWSJO2mOkbOrpWS1OkaaZHbClwVEZcAO1vlMvMNLYuqCQbtWilJ2kfFot4XZOb6dsfSTJlp10pJKolGErmLikdHGeqza6UkaZ8dClweEVcC5wHfyMxsc0wzNlpJKomJnCSVwF4TuYhYmJmbMvP8Oq8d2dqwZm6or5sNW0faHYYkqQNl5jsj4i+BZwCvBP4tIj4PfDwzb2xvdPtueLQC4KyVklQCk92S+974k6JbZa2vtCSaJhrq62HbiC1ykqR9U7TA3Vk8RoHFwBci4h/aGtgMDBf1ouvISVLnm6xrZdQ8XzLJa/ulob5utgw72Ykkafoi4g3Ay4F7gHOBP8vMkYjoAn4NvL2d8e2rXS1yJnKS1OkmS+RyL8/rbe93Bh0jJ0nadwcBz8/Mm2t3ZmYlIp7dpphmzK6VklQekyVyB0fEW6i2vo0/p9he2vLIZmior5utI2NkJhH7fQOiJGn/cjFw3/hGRCwAVmbmZZl5ffvCmpnh0aJrpS1yktTxJvsl/xiwAJhf83x8+9zWhzYzQ309jFWSHWOVdociSeo8HwHur9neUuzraMMjRYucY+QkqePttUUuM98zm4E021BftdvI1uExu5BIkqYrapcbKLpUNrJkz37NrpWSVB7TuiVXrKfTEeb1V+vbLTuc8ESSNG03RcQbIqK3eLwRuKndQc2UXSslqTym+0veMYPN5heJ3P3OXClJmr4/BR4H3AasBR4DnNnWiJpgZ9dKW+QkqeNNt5vI/7QkihYYT+RcgkCSNF2ZeTdwervjaLadXSsdIydJHW/KRC4i5gHbMrMCfDIiTgP+NzNHWh7dDIx3rdy83UROkjQ9ETEAvBp4CDAwvj8zX9W2oJrArpWSVB6N/JL/ABiIiGXAJcArgU+0Mqhm2NUi51pykqRp+xRwKPBM4PvAcmBzWyNqAic7kaTyaCSRi8zcCjwf+NfMfB6wsrVhzdz8AbtWSpL22QMz8y+BLZl5PvB7wO+0OaYZGx6xRU6SyqKhRC4iHgu8mF1j5Pb7KZjn9xVdK03kJEnTNz58YENEnAAsAla0L5zmcIycJJVHIwnZm4CzgC9n5rURcQzw3daGNXPz+qvdRmyRkyTtg3MiYjHwTuAiYD7wl+0Naea2O2ulJJXGlIlcZn6f6vgAIqILuCcz39DqwGaqp7uLgd4ulx+QJE1LUddtysz1VMeJH9PmkJpmeHSM3u6gu6tjVhOSJO3FlH0rIuIzEbGwmL3yOuCGiPiz1oc2c/P7e0zkJEnTUszS/Lp2x9EKw6MVW+MkqSQa6SS/MjM3Ac8FLgaOBF7a0qiaZH5/j10rJUn74lsR8baIOCIilow/GnljRJwSETdExJqIeEed1/8lIq4qHr+KiA3ND7++4dExJzqRpJJoZIxcb0T0Uk3k/i0zRyIiWxxXU8zr7+F+15GTJE3f+Hpxr63Zl0zRzTIiuoGzgacDa4HLI+KizLxu50Ey31xT/vXAic0KeirDIxUTOUkqiUYSuY8CvwV+DvwgIo4CNrUyqGaxa6UkaV9k5tH7+NZHA2sy8yaAiLgQeA7VoQn1nAG8ex/PNW3DoxX6e+1aKUll0MhkJx8GPlyz6+aIeErrQmqe+f093Llpe7vDkCR1mIh4Wb39mfnJKd66DLi1Znst8Ji9nOMo4GjgO3t5/UzgTIAjjzxyitM2xq6VklQeUyZyEbGI6t3CJxW7vg+8F9jYwriaYp5j5CRJ++ZRNc8HgJOBK4GpErl600HubTjC6cAXMnOs3ouZeQ5wDsCqVauaMqShOtmJiZwklUEjXSvPA64BXlRsvxT4T+D5rQqqWeYP2LVSkjR9mfn62u3ipuanGnjrWuCImu3lwO17KXs6u4/Ba7nqGDm7VkpSGTSSyD0gM19Qs/2eiLiqVQE1k2PkJElNshU4toFylwPHRsTRwG1Uk7U/mlgoIo4HFgM/aWaQUxkeHWNefyNVvyRpf9fIr/m2iHhCZv4QICIeD2xrbVjNMb+/h+0jFUbHKvR025VEktSYiPhvdnWJ7AJWAp+f6n2ZORoRrwO+AXQD52XmtRHxXmB1Zl5UFD0DuDAzZ3UW6OHRCkvmWR9KUhk0ksj9KfDJolsJwHrg5Y2eoJiKeTVwW2Y+e/oh7rvxu45bhsdYNGTFJUlq2Adqno8CN2fm2kbemJkXU113tXbfuyZs/9VMA9wXLgguSeUxaSIXEV3A8Zn5sIhYCFAsDj4dbwSuBxbuW4j7bkGRyG0eHmHRUO9sn16S1LluAe7IzO0AETEYESsy87ftDWtmnLVSkspj0l/zzKwAryueb5puEhcRy4HfA87d5whnYOFgkci5KLgkaXr+C6jUbI8V+zra8EiF/l4TOUkqg0Z+zb8VEW+LiCMiYsn4o8HjfxB4O7tXhrNm4UC1FW7TtpF2nF6S1Ll6MnPH+EbxvK+N8TSFXSslqTwaGSP3quJv7RTJCRwz2Zsi4tnA3Zl5RUQ8eZJyTV/wdNzCwWoit9FETpI0Pesi4rTxyUki4jnAPW2OacbsWilJ5TFlIpeZR+/jsR8PnBYRz6K6mOrCiPh0Zr5kwvGbvuDpuJ0tcnatlCRNz58CF0TEvxXba4GXtTGeGctMFwSXpBLZayIXES8BIjM/NWH/a4AtmfmZyQ6cmWcBZxXveTLwtolJXKuNj5Gza6UkaToy80bgpIiYT7Uu3NzumGZqZCzJhP5eu1ZKUhlMdlvurcBX6uz/XPHafm/BzhY5EzlJUuMi4m8j4oDMvD8zN0fE4oh4X7vjmonh0TEAW+QkqSQm+zXvrncHspi5clpz+Wfm92Z7DTmA7q5gQX8Pm7bZtVKSNC2nZuaG8Y3MXA88q43xzNjwaHXeMRM5SSqHyX7NeyNi3sSdEbGADpq5a+Fgr5OdSJKmqzsi+sc3ImIQ6J+k/H5vVyJn10pJKoPJErmPA1+IiBXjO4rnFxavdYQFAz12rZQkTdengUsi4tUR8WrgW8D5bY5pRoZHiq6VriMnSaWw18lOMvMDEXE/8P1isHcCW4D3Z+ZHZivAmVo42OtkJ5KkacnMf4iIq4GnAQF8HTiqvVHNjF0rJalcJl1+IDP/A/iPTp61a+FAL7c3pxGhAAAaGklEQVRt2NbuMCRJnedOoAK8CPgN8MX2hjMzdq2UpHJpZEFwMvP+VgfSKosGe7n+jk3tDkOS1AEi4jjgdOAM4F6qMzVHZj6lrYE1wc6ulbbISVIpNJTIdbKFgz12rZQkNeqXwP8Dfj8z1wBExJvbG1Jz7GyRc4ycJJVC6X/NFw70snl4lLFKtjsUSdL+7wVUu1R+NyI+FhEnUx0j1/HGE7m+brtWSlIZTJnIRcTqiHhtRCyejYCabeFgdcm7+7e7lpwkaXKZ+eXM/EPgQcD3gDcDh0TERyLiGW0NboZGx6qJXE93KfJSSZrzGmmROx04HLg8Ii6MiGdGRMfUAgsHqr1HXUtOktSozNySmRdk5rOB5cBVwDvaHNaMjBQ9U3pN5CSpFKZM5DJzTWb+BXAc8BngPOCWiHhPRCxpdYAzdcBQde3yDdt2tDkSSVInysz7MvOjmfnUdscyEztb5LpKP6pCkuaEhn7NI+KhwD8B/0h1+uUXApuA77QutOZYMq/atfK+LSZykqS5a3Ss2iJn10pJKocpZ62MiCuADcDHgXdk5nDx0mUR8fhWBtcMi4sWufVbTeQkSXPXSKXaItfbbYucJJXBpIlcRHQBX8zMv633emY+vyVRNdGB8/oBuG+LY+QkSXPXzha5LlvkJKkMJr0tl5kV4JRZiqUlFgz00N0VrLdrpSRpDhvZOWulLXKSVAaN/Jp/KyLeFhFHRMSS8UfLI2uSrq5g8VAv99m1UpI0h406a6UklcqUY+SAVxV/X1uzL4Fjmh9Oaywe6rNFTpI0pzlrpSSVy5SJXGYePRuBtNLieX3OWilJmtNGxmyRk6QyaXT5gRMi4kUR8bLxR6sDa6YlQ33OWilJmhURcUpE3BARayKi7iLiRZ16XURcGxGfmY24RisVuruCCBM5SSqDRpYfeDfwZGAlcDFwKvBD4JMtjayJFs/r476bnbVSktRaEdENnA08HVgLXB4RF2XmdTVljgXOAh6fmesj4uDZiG10LJ2xUpJKpJEWuRcCJwN3ZuYrgYcB/S2NqsmWzOtl/dYdVIqB3pIktcijgTWZeVNm7gAuBJ4zocxrgLMzcz1AZt49G4GNjKVryElSiTTyi76tWIZgNCIWAnfTQROdQHWyk7FKsnn7aLtDkSSV2zLg1prttcW+WscBx0XEjyLi0oiou8xPRJwZEasjYvW6detmHNhopUKP4+MkqTQaSeRWR8QBwMeAK4ArgZ+2NKomWzKvD8AlCCRJrVYvU5rYHaQHOJbqsIUzgHOLenb3N2Wek5mrMnPV0qVLZxzYyFg6Y6UklUgjs1b+n+Lpf0TE14GFmXl1a8NqrvFE7t77hzn6oHltjkaSVGJrgSNqtpcDt9cpc2lmjgC/iYgbqCZ2l7cysNGxijNWSlKJNDpr5bKIeBxwJHBARDyptWE118ELBgC4e/NwmyORJJXc5cCxEXF0RPQBpwMXTSjzFeApABFxENWulje1OrDRStq1UpJKpJFZK/8e+EPgOmCs2J3AD1oYV1MdsrA6N8tdm7a3ORJJUpll5mhEvA74BtANnJeZ10bEe4HVmXlR8dozImK8Xv2zzLy31bGNVuxaKUllMmUiBzwXOD4zO7Y5a/FQHz1dYYucJKnlMvNiqsv11O57V83zBN5SPGZNpZK4+oAklUcjt+ZuAnpbHUgrdXUFSxf0c/cmEzlJ0tw0Vkm6zeQkqTQaaZHbClwVEZcAOzOhzHxDy6JqgYMX9HP3ZrtWSpLmprFMusJETpLKopFE7iL2HKjdcQ5eOMAt925tdxiSJLVFxRY5SSqVRpYfOH82Amm1gxf0s/q397U7DEmS2mIsTeQkqUz2mshFxOcz80UR8Qv2XMyUzHxoSyNrsoMXDLB+6wjDo2P093S3OxxJkmbVWMWulZJUJpO1yL2x+Pvs2Qik1caXIFi3eZjli4faHI0kSbOrYoucJJXKXhO5zLyj+Hvz+L5i4dJ7i6mTO8rBO9eSM5GTJM09Y5Wk2xY5SSqNvS4/EBEnRcT3IuJLEXFiRFwDXAPcFRGnzF6IzXH4AYMA3L5hW5sjkSRp9lUq4HrgklQek3Wt/Dfg/wKLgO8Ap2bmpRHxIOCzwNdnIb6mWVYkcmvXm8hJkuaesUx6zeQkqTQm+0XvycxvZuZ/AXdm5qUAmfnL2QmtuRYM9HLAUC9r17sEgSRp7nGyE0kql8kSuUrN84nNWB03Rg5g+eJBW+QkSXOSk51IUrlM1rXyYRGxCQhgsHhOsT0w1YEj4gjgk8ChVJPCczLzQzOMd0aOWDzEr+++v50hSJLUFk52IknlMtmslTNdbG0UeGtmXhkRC4ArIuJbmXndDI+7z5YvHuS7N9xNZhJWZpKkOWSsknTZIidJpdGyUc+ZeUdmXlk83wxcDyxr1fkasXzxENtHKty7ZUc7w5AkadZV0hY5SSqTWZm+KiJWACcCl9V57cyIWB0Rq9etW9fSOJYvrs5ceet9TngiSZpbxiqOkZOkMml5IhcR84EvAm/KzE0TX8/MczJzVWauWrp0aUtjOerA6kLgN99rIidJmlsqiV0rJalEWprIRUQv1STugsz8UivP1Ygjl8yjuyu4cZ0TnkiS5pbqZCftjkKS1CwtS+SiOpvIx4HrM/OfW3We6ejr6eKoJUMmcpKkOcfJTiSpXFrZIvd44KXAUyPiquLxrBaeryHHLJ3PjXdvaXcYkiTNKic7kaRymWwduRnJzB9SXXNuv/KAg+fxg1+tc9C3JGlOsd6TpHKZlVkr9ycPWDqfHWMVZ66UJM0plbRrpSSVyZxM5ADW3O04OUnS3FGd7MRETpLKYs4lcscfuoAIuO6OPVZCkCSptOxaKUnlMucSufn9PRx94DyuuW1ju0ORJGnWVBK6bJGTpNKYc4kcwEOWLeLa222RkyTNHaOVCt1zstaXpHKakz/pDzl8Ibdt2Mb6LTvaHYokSbOiUsHJTiSpROZkInfC4YsAuOZ2u1dKkporIk6JiBsiYk1EvKPO66+IiHU1a6z+8WzENeY6cpJUKnMykfudZYuIgJ/dsqHdoUiSSiQiuoGzgVOBlcAZEbGyTtHPZebDi8e5sxGbk51IUrnMyURu0VAvxx+ygJ/+5r52hyJJKpdHA2sy86bM3AFcCDynzTFRqSTgZCeSVCZzMpEDeMzRS7jylvWMjFXaHYokqTyWAbfWbK8t9k30goi4OiK+EBFH1DtQRJwZEasjYvW6detmFNRYVhM5W+QkqTzmbCL3qKOXsHXHmLNXSpKaqV6mlBO2/xtYkZkPBb4NnF/vQJl5TmauysxVS5cunVFQYxUTOUkqmzmbyD366CUA/GjNPW2ORJJUImuB2ha25cDttQUy897MHC42PwY8stVBVdKulZJUNnM2kTt4wQAPXb6Ib19/V7tDkSSVx+XAsRFxdET0AacDF9UWiIjDajZPA65vdVC7WuRafSZJ0myZ0z/pJz/oEK66dQPrNg9PXViSpClk5ijwOuAbVBO0z2fmtRHx3og4rSj2hoi4NiJ+DrwBeEWr46oUw8FtkZOk8pjTidzTVh5MJlxiq5wkqUky8+LMPC4zH5CZf1Pse1dmXlQ8PyszH5KZD8vMp2TmL1sdk5OdSFL5zOlEbuVhCznmoHl86We3tTsUSZJaxslOJKl85nQiFxG84JHL+elv7uPme7e0OxxJklrCyU4kqXzmdCIH8LwTlxEBF15+69SFJUnqQLbISVL5zPlE7vADBjnlIYfy6Utv5v7h0XaHI0lS0+1M5GyRk6TSmPOJHMCZTzqGzdtH+fSlN7c7FEmSmm5n10pb5CSpNEzkgBOPXMxTjl/K2d9Zwz33uxSBJKlcXEdOksrHn/TCO5+9km0jY/zj129odyiSJDWVk51IUvmYyBUesHQ+r37C0Xxu9a186zrXlZMklcdYsSC4k51IUnmYyNV4yzOO44RlC3nbf/3c5QgkSaXhZCeSVD4mcjX6e7o5+48eQVfAi8+9jDs2bmt3SJIkzZiTnUhS+ZjITXDUgfM4/1WPZsPWEV74kZ/wyzs3tTskSZJmxBY5SSofE7k6Hrr8AD77mpMYrVR4wb//mC//bC1Z3M2UJKnTjKULgktS2ZjI7cXvLF/EV1/7BB582ELe/Lmf85pPrua39zhuTpLUeSoVu1ZKUtmYyE3i0EUDfO5PHss7f+/B/GjNvTztn7/PO7/yCydCkSR1FLtWSlL59LQ7gP1dd1fwx088htMefjj/eskaPvvTW7jgslt46vEH84ePOoInHbeUgd7udocpSdJeje2c7KTNgUiSmsZErkEHLxjgr597Aq976gO54NKb+cxPb+GSX97NUF83T33QwTzpuKU89pgDOWLJULtDlSRpN5XxdeRskZOk0jCRm6ZDFg7wlmccz+tPPpZLb7qXi39xJ9+67k6+dvUdACw7YJBHrVjMQw5fxMrDF7LysIUsntfX5qglSXOZk51IUvmYyO2j3u4unnjsUp547FL+9nkn8Ou77+cnN97LT268l8t+cx9fuer2nWUPXtDPioPmcdSSIY46cIijDpzH8sWDHLJwgKUL+unttq+LJKl1nOxEksrHRK4JIoLjDlnAcYcs4OWPWwHAfVt2cN3tm7jujo3ccOf93HLfFr73q3Ws2zy8x/sPnNfHwQsHOHhBPwcv6GfxvD4WDfayeKiPA4Z6q4/BPhbP62XRYC+Dvd2E3WMkSQ1yshNJKh8TuRZZMq+PJxx7EE849qDd9m/dMcot921l7X3buHvzMHdv3s5dm4ZZV/y94c7NbNi2g+0jlb0eOwKGersZ6u9hXl83Q309zOvf/W9/Txd9xaO/u4v+3m76unftq33eX2x3dQU9XbHzb3fxqD7vojuC7u6o/t1L2e4IIjDRlKT9iF0rJal8TORm2VBfDw86dCEPOnThpOW2j4yxYesI67fuYMPWETZu28H6rSNs3DbC1uFRtuwYY+uOUbYM7/q7YesObtswxtbhUYZHK+wYrTA8Vv3bDl1FQtcVEIwneNAVQVD8rS1T/IXx7WqZriIp7OqqHme8bAQ7jzPxWF2xe7muCefoqtk/8T3j5++KqJ5z/PjFZ6p9T093FwcM9bJkqNqKumCgh4Hebvp7uujv7aKnq4ue7qC3u4ve7i56uoKe7l3H2yM+guiqd41qPieYLEualp1dK/3dkKTSaGkiFxGnAB8CuoFzM/P9rTxfmQz0dnPoom4OXTQw42NlJiNjyY4iqRseHWPHeKI3WmHHWIWR0QpjmVQqMFqpMFbJnY/RSlLJZHSs2JfVfWNjFcYSxiqVapmibCZkcd5MqGSSFH+zur+S7HyNmtfGy+723qRant3fX1u2Uqm+Pn7capmachP+jn+mys5Ycuc5KpXdy06MrTJ+vqxe0w1bRxgt/idpNk1MZAl2JoO1ZXY+37kv9thHnXK1ZesdZ7fXJ8Q1sfTe3z++r/GY65Zr8P9Ng8b/J7bxYzZ6vGmcu8kFp/O/7o3E+YrHreAlJx01jaOq3WyRk6TyaVkiFxHdwNnA04G1wOURcVFmXteqc6q+iKCvJ+jr6YL+dkdTPpnJ5uFRNm0bYfP2amvo8MgY20crjI5VGBmrMDKWjFaqf8cqeyatlcp4kjh1ArxHQszu+2vj2vW8+Ltb3OP79iw38fPtfD6N92edfdQpu/t7ph8zdWKuZzqpdta7EDM4ZoOHm+YxmxvjdAof6Ey8HeewRQM863cOZeGgHXEkqSxa+Yv+aGBNZt4EEBEXAs8BTORUKhHBwoFeFg70tjsUSarrkUct4ZFHLWl3GJKkJmrlvPfLgFtrttcW+3YTEWdGxOqIWL1u3boWhiNJkiRJ5dDKRK5eR/w9Ou5k5jmZuSozVy1durSF4UiS1HoRcUpE3BARayLiHZOUe2FEZESsms34JEnl0MpEbi1wRM32cuD2vZSVJKnj1YwPPxVYCZwRESvrlFsAvAG4bHYjlCSVRSsTucuBYyPi6IjoA04HLmrh+SRJared48MzcwcwPj58or8G/gHYPpvBSZLKo2WJXGaOAq8DvgFcD3w+M69t1fkkSdoPTDk+PCJOBI7IzK9NdiDHkEuSJtPSeYgz82Lg4laeQ5Kk/cik48Mjogv4F+AVUx0oM88BzgFYtWrV7C9WKUnar7Wya6UkSXPNVOPDFwAnAN+LiN8CJwEXOeGJJGm6TOQkSWqeSceHZ+bGzDwoM1dk5grgUuC0zFzdnnAlSZ2qpV0rp+uKK664JyJunuFhDgLuaUY8JeI12ZPXpD6vy568JntqxjU5qhmB7G8yczQixseHdwPnZea1EfFeYHVm7tOkX9aPLeV12ZPXZE9ekz15TeqbtToyMsvV7T4iVmemXVRqeE325DWpz+uyJ6/Jnrwmncl/t/q8LnvymuzJa7Inr0l9s3ld7FopSZIkSR3GRE6SJEmSOkwZE7lz2h3AfshrsievSX1elz15TfbkNelM/rvV53XZk9dkT16TPXlN6pu161K6MXKSJEmSVHZlbJGTJEmSpFIzkZMkSZKkDlOaRC4iTomIGyJiTUS8o93xzJaIOCIivhsR10fEtRHxxmL/koj4VkT8uvi7uNgfEfHh4jpdHRGPaO8naJ2I6I6In0XE14rtoyPisuKafK5YrJeI6C+21xSvr2hn3K0UEQdExBci4pfFd+axc/27EhFvLv7buSYiPhsRA3PxuxIR50XE3RFxTc2+aX83IuLlRflfR8TL2/FZtCfrSOvIiawjd2f9WJ915P5dP5YikYuIbuBs4FRgJXBGRKxsb1SzZhR4a2Y+GDgJeG3x2d8BXJKZxwKXFNtQvUbHFo8zgY/Mfsiz5o3A9TXbfw/8S3FN1gOvLva/GlifmQ8E/qUoV1YfAr6emQ8CHkb1+szZ70pELAPeAKzKzBOoLuB8OnPzu/IJ4JQJ+6b13YiIJcC7gccAjwbePV65qX2sI60j98I6cnfWjxNYR+70CfbX+jEzO/4BPBb4Rs32WcBZ7Y6rTdfiq8DTgRuAw4p9hwE3FM8/CpxRU35nuTI9gOXFf1hPBb4GBHAP0DPxOwN8A3hs8bynKBft/gwtuCYLgd9M/Gxz+bsCLANuBZYU//ZfA545V78rwArgmn39bgBnAB+t2b9bOR9t+3e1jtz12a0j0zqyzvWwfqx/Xawjd12L/bJ+LEWLHLu+aOPWFvvmlKIJ+0TgMuCQzLwDoPh7cFFsrlyrDwJvByrF9oHAhswcLbZrP/fOa1K8vrEoXzbHAOuA/yy605wbEfOYw9+VzLwN+ABwC3AH1X/7K/C7Mm66343Sf2c6lP8uWEdOYB25O+vHOqwjJ7Vf1I9lSeSizr45ta5CRMwHvgi8KTM3TVa0zr5SXauIeDZwd2ZeUbu7TtFs4LUy6QEeAXwkM08EtrCrK0A9pb8uRbeG5wBHA4cD86h2i5horn1XprK36+D12T/N+X8X68hdrCPrsn6swzpyn8xq/ViWRG4tcETN9nLg9jbFMusiopdqBXVBZn6p2H1XRBxWvH4YcHexfy5cq8cDp0XEb4ELqXYd+SBwQET0FGVqP/fOa1K8vgi4bzYDniVrgbWZeVmx/QWqFddc/q48DfhNZq7LzBHgS8Dj8LsybrrfjbnwnelEc/rfxTpyD9aRe7J+rM86cu/2i/qxLInc5cCxxSw6fVQHYl7U5phmRUQE8HHg+sz855qXLgLGZ8R5OdVxAeP7X1bMqnMSsHG8abgsMvOszFyemSuofhe+k5kvBr4LvLAoNvGajF+rFxblS3cHKTPvBG6NiOOLXScD1zGHvytUu4ucFBFDxX9L49dkTn9Xakz3u/EN4BkRsbi4k/uMYp/ayzrSOnIn68g9WT/ulXXk3u0f9WO7Bw826wE8C/gVcCPwF+2OZxY/9xOoNs1eDVxVPJ5FtU/yJcCvi79LivJBdfayG4FfUJ2JqO2fo4XX58nA14rnxwA/BdYA/wX0F/sHiu01xevHtDvuFl6PhwOri+/LV4DFc/27ArwH+CVwDfApoH8ufleAz1IdAzFC9c7hq/fluwG8qrg+a4BXtvtz+dj572IdaR1Z7/pYR+66FtaP9a/LnK8j9+f6MYoDS5IkSZI6RFm6VkqSJEnSnGEiJ0mSJEkdxkROkiRJkjqMiZwkSZIkdRgTOUmSJEnqMCZy0n4qIp4cEV9rdxySJO1PrB+lKhM5SZIkSeowJnLSDEXESyLipxFxVUR8NCK6I+L+iPiniLgyIi6JiKVF2YdHxKURcXVEfDkiFhf7HxgR346InxfveUBx+PkR8YWI+GVEXBARUZR/f0RcVxznA2366JIk7ZX1o9RaJnLSDETEg4E/BB6fmQ8HxoAXA/OAKzPzEcD3gXcXb/kk8OeZ+VDgFzX7LwDOzsyHAY8D7ij2nwi8CVgJHAM8PiKWAM8DHlIc532t/ZSSJE2P9aPUeiZy0sycDDwSuDwiriq2jwEqwOeKMp8GnhARi4ADMvP7xf7zgSdFxAJgWWZ+GSAzt2fm1qLMTzNzbWZWgKuAFcAmYDtwbkQ8HxgvK0nS/sL6UWoxEzlpZgI4PzMfXjyOz8y/qlMupzjG3gzXPB8DejJzFHg08EXgucDXpxmzJEmtZv0otZiJnDQzlwAvjIiDASJiSUQcRfW/rRcWZf4I+GFmbgTWR8QTi/0vBb6fmZuAtRHx3OIY/RExtLcTRsR8YFFmXky1W8nDW/HBJEmaAetHqcV62h2A1Mky87qIeCfwzYjoAkaA1wJbgIdExBXARqrjBABeDvxHURHdBLyy2P9S4KMR8d7iGH8wyWkXAF+NiAGqdyvf3OSPJUnSjFg/Sq0XmZO1aEvaFxFxf2bOb3cckiTtT6wfpeaxa6UkSZIkdRhb5CRJkiSpw9giJ0mSJEkdxkROkiRJkjqMiZwkSZIkdRgTOUmSJEnqMCZykiRJktRh/n9rTLXId3nm5gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1080x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# using plt.subplot I can show paired loss and accuracy\n",
    "\n",
    "plt.figure(figsize = (15, 4))  # adjust figures size\n",
    "plt.subplots_adjust(wspace=0.2)  # adjust distance between plots\n",
    "\n",
    "# loss plot\n",
    "plt.subplot(1, 2, 1)\n",
    "plt.plot(loss_history)\n",
    "plt.title('Loss History')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Binary Cross-Entropy')\n",
    "\n",
    "# accuracy plot\n",
    "plt.subplot(1, 2, 2)\n",
    "plt.plot(accuracy_history)\n",
    "plt.title('Accuracy History')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Accuracy')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Accuracy scores reached .97+ levels after less than 200 epochs, which is good.\n",
    "\n",
    "At this point, we can make a prediction on the test set, and check the Network's ability to generalize beyond the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction = model.predict(X_test)    # (Keras syntax)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# outputs vector containing the value of the class with the highest predicted probability\n",
    "prediction = np.argmax(prediction, axis=1)\n",
    "\n",
    "# do it also for test data, i.e. reverse one-hot encoding and get a vector of 0-1 values\n",
    "testdata = np.argmax(y_test, axis=1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[87  1]\n",
      " [ 2 53]]\n"
     ]
    }
   ],
   "source": [
    "# Now I can plot the Confusion Matrix\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "CM = confusion_matrix(prediction, testdata)\n",
    "print(CM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Confusion Matrix looks very good: oservations on the matrix diagonal are correct predictions. Luckily, only three datapoints have been misclassified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Accuracy: 0.9790209790209791\n"
     ]
    }
   ],
   "source": [
    "# Accuracy = sum of the diagonal / sum of the whole matrix\n",
    "\n",
    "print('Test Accuracy: ' + str(np.sum(np.diag(CM)) / np.sum(CM)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I like to visualize Confusion Matrices using a heatmap. IMHO, `seaborn` is the best Python library for that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1d0489acc88>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVoAAAD8CAYAAAA2Y2wxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAD8JJREFUeJzt3X+wXGV9x/H3lxuSwKVKaGgIiUVFJKKFUAE1KKBg5Ycj4K+KUilFgzPFKXSoKJ1SoXQKLZTiyNBeIBAKJkEcRBiBMYBAx0qgEiGSOCBVSQwJCOFHEgJ399s/sjC35JLda/a5Z+/J+8U8k7tn7z77hcl85stznnNOZCaSpHK2qboASao7g1aSCjNoJakwg1aSCjNoJakwg1aSCjNoJakwg1aSCjNoJamwcaW/4OWnHvPSM21iu10/UHUJ6kGDL62ILZ1jJJmz7eS3bvH3dcKOVpIKK97RStKoajaqrmATBq2kemkMVl3BJgxaSbWS2ay6hE0YtJLqpWnQSlJZdrSSVJgnwySpMDtaSSor3XUgSYV5MkySCnPpQJIK82SYJBVmRytJhXkyTJIK82SYJJWV6RqtJJXlGq0kFebSgSQVZkcrSYU1Xq66gk0YtJLqxaUDSSrMpQNJKsyOVpIKM2glqaz0ZJgkFeYarSQV1qWlg4jYE1gw5NBbgbOAHYEvAk+2jp+Zmd/f3FwGraR66VJHm5k/B2YCREQfsAK4ATgRuCgzL+h0LoNWUr2UORl2KPCLzPxVRIz4w9t0vx5JqlA2Ox+d+wwwb8jrUyLiwYiYExGT2n3YoJVUL4ODHY+ImB0R9w8Zs187XUSMBz4GfLt16FJgdzYuK6wELmxXkksHkuplBJ1qZg4AA21+7QjgJ5m5qvWZVa+8ERGXATe3+x6DVlK9dH+N9jiGLBtExNTMXNl6eSywpN0EBq2keuniPtqI2B74MHDykMP/HBEzgQR++Zr3hmXQSqqXLna0mbkO+P3XHPuzkc5j0EqqF68Mk6TCBn3cuCSVlVl1BZswaCXVi7dJlKTCDFpJKsyTYZJUWKNRdQWbMGgl1YtLB5JUmEErSYW5RitJZWXTfbSSVJZLB5JUmLsOJKkwO9qtx9Xzb+A7N91KRLDH7m/m3DP/mi+eeiZr160H4Oln1vBHe+3JN847q+JKVZXLBi7kqCMPY/WTTzFz30OrLqc+ejBofWZYAauefIprr7+RBXO+wXev+XeazSa3LLyLqy+9gO/MvYTvzL2Efd71Dg49eFbVpapCV199HUd99HNVl1E/mZ2PUdK2o42IGcDRwDQ23lH8N8D3MnNp4drGtMFGgw0bXmJc3zjWv7iBnSfv9Op7a9euY9FPfsq5f3tahRWqavf8173sttv0qsuon7HW0UbEGcB8IIBFwH2tn+dFxFfLlzc2Tdl5Mn9+3Cc47OOf54NHf5bf69+eA9/z7lffX3j3j3jPu/dhh/7+CquUaqqZnY9R0m7p4CRg/8w8LzOvaY3zgANa72kYzz73PHfe82Nu+/aV3HHjtax/cQM33XbHq+/fsvAujjzskOoKlOqs0eh8jJJ2QdsEdh3m+NTWe8Ma+qz0y6+e93q/Vls/vn8x03adwk6TdmTbceM49OBZLH7oYQDWPPscDz38cw6adUDFVUr1lM1mx2O0tFujPRW4PSIeAR5vHftD4G3AKa/3oaHPSn/5qcd67zKNwqZO2ZkHlyxj/YsvMnHCBO69fzHvnLEHALfdcQ8HzzqACRPGV1ylVFNj7cqwzLw1It7OxqWCaWxcn10O3JeZvbcruEfs/c4ZfPiD7+fTJ36Zvr4+Zrx9dz519BEA3HL7XXzh+E9XXKF6wTX/eQkHH/Q+Jk/eiV8+dj9nn3MBV141v+qyxr4evNdBZOEtDltjR6v2ttv1A1WXoB40+NKK2NI51p7zuY4zp/+sa7f4+zrhBQuS6mWw9/5n26CVVC89uHRg0Eqql7F2MkySxprR3LbVKYNWUr3Y0UpSYT0YtN69S1K9dPES3IjYMSKuj4hlEbE0It4XETtFxA8i4pHWn5PazWPQSqqVbGbHowMXA7dm5gxgH2Ap8FXg9szcA7i99XqzDFpJ9dKlu3dFxBuAg4ArADLzpcxcw8bbxs5t/dpc4Jh2JRm0kuql2ex8bN5bgSeBKyPigYi4PCL6gSmZuRKg9ecftJvIoJVULyPoaIfeabA1Zg+ZaRzwx8ClmbkvsJYOlgmG464DSfUygl0HQ+80OIzlwPLMvLf1+no2Bu2qiJiamSsjYiqwut332NFKqpVsNDsem50n8wng8YjYs3XoUOBh4HvACa1jJwA3tqvJjlZSvXR3H+2XgWsjYjzwGHAiGxvU6yLiJODXwKfaTWLQSqqVDrdtdTZX5mJgv2HeGtHz4Q1aSfXSg1eGGbSS6qX37ilj0EqqlxzsvaQ1aCXVS+/lrEErqV66eTKsWwxaSfViRytJZdnRSlJpdrSSVFYOVl3BpgxaSbXSg08bN2gl1YxBK0ll2dFKUmEGrSQVlo2ouoRNGLSSasWOVpIKy6YdrSQVZUcrSYVl2tFKUlF2tJJUWNNdB5JUlifDJKkwg1aSCsveux2tQSupXuxoJakwt3dJUmENdx1IUll2tJJUmGu0klRYL+462KbqAiSpm7IZHY9ORERfRDwQETe3Xl8VEf8bEYtbY2a7OexoJdVKo9n1/vGvgKXAG4Yc+5vMvL7TCexoJdVKZuejnYiYDhwFXL4lNRm0kmqlmdHx6MC/AV9h02fr/mNEPBgRF0XEhHaTGLSSaiUzOh4RMTsi7h8yZr8yT0R8FFidmf/zmq/4GjAD2B/YCTijXU2u0UqqlZHsOsjMAWDgdd4+EPhYRBwJTATeEBHXZObxrfc3RMSVwOntvqd40PZPO6j0V2gMWnHg26ouQTXV4ZJAW5n5NTZ2r0TEIcDpmXl8REzNzJUREcAxwJJ2c9nRSqqVArsOXuvaiNgZCGAx8KV2HzBoJdVKiesVMvOHwA9bP39opJ83aCXVSreWDrrJoJVUK95URpIK68GH4Bq0kuolsaOVpKIGXTqQpLLsaCWpMNdoJakwO1pJKsyOVpIKa9jRSlJZPfhsRoNWUr007WglqawefAiuQSupXjwZJkmFNcOlA0kqqlF1AcMwaCXVirsOJKkwdx1IUmHuOpCkwlw6kKTC3N4lSYU17GglqSw7WkkqzKCVpMJ68JFhBq2kerGjlaTCvARXkgpzH60kFdaLSwfbVF2AJHVTcwRjcyJiYkQsioifRsTPIuLs1vG3RMS9EfFIRCyIiPHtajJoJdVKjmC0sQH4UGbuA8wEDo+I9wLnAxdl5h7AM8BJ7SYyaCXVSjM6H5uTG73QerltayTwIeD61vG5wDHtajJoJdVKYwQjImZHxP1Dxuyhc0VEX0QsBlYDPwB+AazJzMHWrywHprWryZNhkmqlOYIbJWbmADCwmfcbwMyI2BG4AXjHcL/W7nsMWkm1UmLXQWauiYgfAu8FdoyIca2udjrwm3afd+lAUq1062RYROzc6mSJiO2Aw4ClwJ3AJ1u/dgJwY7ua7Ggl1UoXO9qpwNyI6GNjU3pdZt4cEQ8D8yPiXOAB4Ip2Exm0kmplMLrzMJvMfBDYd5jjjwEHjGQug1ZSrfjMMEkqrBcvwTVoJdXKSLZ3jRaDVlKt9F7MGrSSasalA0kqrNGDPa1BK6lW7GglqbC0o5Wksuxot1LTp09lzhUXs8suO9NsNrn8im/xzW+2vWpPNTV53nya69ZDswGNBk9/6WT6T/wLJhz4fsgmzWfW8Nz5/0Tzt7+tutQxye1dW6nBwQZfOeMcFi9ewg479HPvj2/h9oV3s3TZI1WXpoo8c9qp5HPPvvp63YL5rL1yDgDbffwT9H/+BJ6/6F+rKm9M672YNWhHxRNPrOaJJ1YD8MILa1m27BF2nbaLQatX5bp1r/4cEyf2ZlqMEYM9+B/vdw7aiDgxM6/sZjFbg912m84++7yLRYseqLoUVSVh0r9cACTrb7qJ9TffBED/SV9guz/5CLn2BZ4+7dRqaxzDevFk2Jbcj/bs13tj6OMhmo21W/AV9dLfvz0L5g9w+ulf5/nnX2j/AdXS01/+S54++Ys8c8ZX2O6YY9h2770BWHvF5Tz1p59i/cKFbH/sxyuucuzq1lNwu2mzQRsRD77OeAiY8nqfy8yBzNwvM/fbpq+/60WPRePGjWPBggHmzb+B7954S9XlqEKvnOTKNWvYcM89bDvj/z8d5cXbFzLxoIOqKK0WcgT/jJZ2SwdTgI+w8ZG6QwXwoyIV1dTAf1zAsmWPcvHFl1Vdiqo0cSIRQa5fDxMnMn6//Vl79Vz6pk2jsWIFABNmHcjgr39dcaFj11jc3nUzsENmLn7tG63n56gDs2btz/HHf5KHHlrKfYtuA+DvzjqfW2+9o+LKNNr6Jk3ijf9wLgDR18eLCxfy0n2LeOPZ5zDuTW8im0lz1Sqeu+jCiisduxrZe2u0kYWLGj9heu/9W6tyj8/aveoS1IOm3HlXbOkcn93t2I4z51u/umGLv68Tbu+SVCu9uOvAoJVUK2NxjVaSxhQvwZWkwlw6kKTCenHXgUErqVZcOpCkwjwZJkmFuUYrSYW5dCBJhZW+2vV3YdBKqpVefNz4ltyPVpJ6TpPseLQTEXMiYnVELBly7OsRsSIiFrfGke3mMWgl1Upmdjw6cBVw+DDHL8rMma3x/XaTuHQgqVa6eTIsM++OiDdv6Tx2tJJqZSRPWBj62K3WmN3h15zSetrMnIiY1O6XDVpJtdLI7HgMfexWawx08BWXArsDM4GVQNu7tLt0IKlWSu+jzcxVr/wcEZex8Uk0m2XQSqqV0kEbEVMzc2Xr5bHAks39Phi0kmqmmxcsRMQ84BBgckQsB/4eOCQiZgIJ/BI4ud08Bq2kWunyroPjhjl8xUjnMWgl1Yo3lZGkwhrZezdKNGgl1Yo3lZGkwrxNoiQV5hqtJBXWdOlAksqyo5Wkwtx1IEmFuXQgSYW5dCBJhdnRSlJhdrSSVFgjG1WXsAmDVlKteAmuJBXmJbiSVJgdrSQV5q4DSSrMXQeSVJiX4EpSYa7RSlJhrtFKUmF2tJJUmPtoJakwO1pJKsxdB5JUmCfDJKkwlw4kqTCvDJOkwuxoJamwXlyjjV5M/7qKiNmZOVB1Heot/r2ov22qLmArM7vqAtST/HtRcwatJBVm0EpSYQbt6HIdTsPx70XNeTJMkgqzo5WkwgzaURIRh0fEzyPi0Yj4atX1qHoRMSciVkfEkqprUVkG7SiIiD7gEuAIYC/guIjYq9qq1AOuAg6vugiVZ9COjgOARzPzscx8CZgPHF1xTapYZt4NPF11HSrPoB0d04DHh7xe3jomaStg0I6OGOaY2z2krYRBOzqWA28a8no68JuKapE0ygza0XEfsEdEvCUixgOfAb5XcU2SRolBOwoycxA4BbgNWApcl5k/q7YqVS0i5gH/DewZEcsj4qSqa1IZXhkmSYXZ0UpSYQatJBVm0EpSYQatJBVm0EpSYQatJBVm0EpSYQatJBX2fx1p9Ff62GKwAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn\n",
    "\n",
    "seaborn.heatmap(CM, annot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Wrap the training process with a function\n",
    "\n",
    "Working with Machine (and Deep) Learning requires testing a whole lot of models. Sometimes, writing a block of code for training each Neural Network you want to try might take too much time. For that reason, it becomes useful to define a \"training function\" with parameters you can tune that you can call whenever you need.\n",
    "\n",
    "This is just a wrapper for the above code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_model(model, train, target, loss, optimizer, metrics, n_epochs, verbose=True, plot=True):\n",
    "    '''\n",
    "    Trains a TensorFlow 2.0 model\n",
    "    \n",
    "    Returns:\n",
    "        prediction: The model's prediction - numpy\n",
    "        loss_history: list\n",
    "        metrics_history: list\n",
    "    \n",
    "    Arguments:\n",
    "        model: a Sequential() model\n",
    "        train: Train dataset (numpy)\n",
    "        target: Target dataset - True outcome (numpy)\n",
    "        loss: loss object or function\n",
    "        optimizer: optimizer object (learnin rate is specified within)\n",
    "        metrics: an evaluation metrics of the model's goodness\n",
    "        n_epochs: number of iterations of training\n",
    "        verbose: print training progress yes/no\n",
    "        plot = plot loss and accuracy histories at the end of training y/n\n",
    "    '''\n",
    "    import numpy as np\n",
    "    import tensorflow as tf\n",
    "    import matplotlib.pyplot as plt\n",
    "    \n",
    "    \n",
    "    loss_history = []\n",
    "    metrics_history = []\n",
    "    \n",
    "    for epoch in range(n_epochs):\n",
    "    \n",
    "        with tf.GradientTape() as tape:\n",
    "            current_loss = loss(model(train), target)\n",
    "        \n",
    "        gradients = tape.gradient(current_loss, model.trainable_variables)\n",
    "        optimizer.apply_gradients(zip(gradients, model.trainable_variables))\n",
    "        \n",
    "        loss_history.append(current_loss.numpy())\n",
    "        metrics.update_state(target, model(train))\n",
    "        current_metrics = metrics.result().numpy()\n",
    "        metrics_history.append(current_metrics)\n",
    "        \n",
    "        if verbose:            \n",
    "            if (epoch+1) % 100 == 0:\n",
    "                print(str(epoch+1) \n",
    "                      + '.\\tTraining Loss: ' + str(current_loss.numpy()) \n",
    "                      + ',\\tAccuracy: ' + str(current_metrics))\n",
    "    \n",
    "        accuracy.reset_states()\n",
    "    \n",
    "    print('\\nTraining complete.')\n",
    "    \n",
    "    if plot:\n",
    "        plt.figure(figsize = (15, 4))\n",
    "        plt.subplots_adjust(wspace=0.2)  # this distances the plots\n",
    "\n",
    "        plt.subplot(1, 2, 1)\n",
    "        plt.plot(loss_history)\n",
    "        plt.title('Loss History')\n",
    "        plt.xlabel('Epochs')\n",
    "        plt.ylabel('Loss')\n",
    "\n",
    "        plt.subplot(1, 2, 2)\n",
    "        plt.plot(metrics_history)\n",
    "        plt.title('Metrics History')\n",
    "        plt.xlabel('Epochs')\n",
    "        plt.ylabel('Metrics')\n",
    "        \n",
    "        plt.show()\n",
    "    \n",
    "    return prediction, loss_history, metrics_history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a very bare training function that I could have called as follows:\n",
    "\n",
    "    preds, loss_history, accuracy_history = train_model(model, X_train, y_train, \n",
    "                                                        loss = tf.keras.losses.BinaryCrossentropy(), \n",
    "                                                        optimizer = tf.optimizers.Adam(learning_rate=0.001), \n",
    "                                                        accuracy = tf.keras.metrics.BinaryAccuracy(), \n",
    "                                                        n_epochs = 1000)\n",
    "\n",
    "With it, you can train this or a similar model with a single line of code. You can also update it with lots of fancier features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, this is all for now. Thanks for take the time to read my Notebook.\n",
    "\n",
    "Call me crazy, but I think I'm going to miss the good ol' `tf.Session()` of TensorFlow 1.x. However, I must admit that once you've learned the TensorFlow 2.0 eager execution you'll be able to implement Deep Learning model much quicker that usual."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}