White-box_attack_on_MNIST.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Aqf-ZLThgk0P"
   },
   "source": [
    "# Study of first order Frank Wolfe algorithms to solve constrained non-convex problems in the context of white box adversarial attacks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7-g4bgEOgrHX"
   },
   "source": [
    "Authors: Laia Porcar, Stefanija Galevska, Philippe Robert, Luis M. López\n",
    "\n",
    "Objectives:\n",
    "\n",
    "- Implement three Frank Wolfe algorithms for non convex contrained problem.\n",
    "\n",
    "- Test the algorithms with the MNIST and ImageNet dataset on a whitebox adversarial attack problem. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jbtpI9ssgvqY"
   },
   "source": [
    "# Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "isecGizugxRL"
   },
   "outputs": [],
   "source": [
    "from keras.datasets import mnist\n",
    "from matplotlib import pyplot as plt\n",
    "from numpy import mean\n",
    "import numpy as np\n",
    "from numpy import std\n",
    "from sklearn.model_selection import KFold\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Conv2D\n",
    "from tensorflow.keras.layers import MaxPooling2D\n",
    "from tensorflow.keras.layers import Dense\n",
    "from tensorflow.keras.layers import Flatten\n",
    "from tensorflow.keras.optimizers import SGD\n",
    "from numpy import random\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras import models\n",
    "import keras\n",
    "from tensorflow.keras import backend as K\n",
    "from tensorflow.keras import losses\n",
    "from tensorflow.keras.preprocessing.image import img_to_array\n",
    "from tensorflow.keras.applications.imagenet_utils import decode_predictions\n",
    "\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i0ZG284ohK4H"
   },
   "source": [
    "# Dataset and training the neural network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 393
    },
    "executionInfo": {
     "elapsed": 1382,
     "status": "ok",
     "timestamp": 1654688270120,
     "user": {
      "displayName": "Philippe Robert",
      "userId": "13287346637647799893"
     },
     "user_tz": -120
    },
    "id": "6M1HXUNEhMek",
    "outputId": "e22e306b-5831-4f28-dd69-03e98a6a0bcb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train: (60000, 28, 28)\n",
      "Y_train: (60000,)\n",
      "X_test:  (10000, 28, 28)\n",
      "Y_test:  (10000,)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU4AAAD7CAYAAAAFI30bAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABWlklEQVR4nO39WWyk2XnYjf9O7ftOFpfi2uxm79M9rZmeGc2MImmkyLIdWUlsWM4iJwGMwAkQA0FgJTe5+gO+Cj4D33cjwI6U2FBixJY8SSRZ1mjpmdFo1NM9o97YC5tkk8Uq1kLWvi/v/6L5HpO9DdlNVhXJ8wMIksUi69T78Dzvc55VaJqGQqFQKLaOodsLUCgUir2GUpwKhUKxTZTiVCgUim2iFKdCoVBsE6U4FQqFYpsoxalQKBTb5JkUpxDiC0KIW0KIWSHE13ZqUYruouS6f1Gy3RnE0+ZxCiGMwG3gc0AUuAh8RdO0Gzu3PEWnUXLdvyjZ7hymZ/jdF4FZTdPmAIQQ/wP4EvBYIQghDnq2fVrTtL5uL+JjUHLdPntBrrBN2Sq5Pl6uz3JUHwaWNnwfXX9M8XjudXsBW0DJdfvsBbmCku12eaxcn8Xi3BJCiN8Dfm+3X0fRWZRc9ydKrlvjWRTnMjCy4fvI+mOb0DTt68DXQZn+ewQl1/3Lx8pWyXVrPMtR/SJwWAgxIYSwAL8NvLkzy1J0ESXX/YuS7Q7x1BanpmlNIcS/Bf4GMAJ/qmna9R1bmaIrKLnuX5Rsd46nTkd6qhdTpv8lTdM+0e1F7DRKrkqu+5THylVVDikUCsU22fWoukLRKYxGI0ajEbPZjNVqxWg0YrVaAajVarRaLWq1GvV6nXa7TavV6vKKFXsVpTgV+wKDwUBfXx8ej4dDhw5x4sQJ+vr6OH78OADXr18nnU5z/fp17ty5Q6FQIJVK0W63u7xyxV7kQCtOIYT8ePDxdrtNu91GCIHBYHjk8zRNkx9qA3YXg8GAy+XC7/czNjbGiRMnGB4e5uWXXwbAYrEQi8XI5/Osrq7SbrdJp9NdXrViN9D3qr5vW63Wju/Pfa04hRDy6Pag0gMYGhpiYmICg8GAyWTCaDTidrsxmUxcu3aNO3fuEIlEOH36NG63m/HxcWw2m1Sqi4uL3L17l1QqxbVr16jVal14lwcbs9mMz+fD7XbzxS9+kVOnTjE0NMTIyAgulwuz2QzA4cOHGRwcJBQK8YlPfIL33nuPb33rW+q4vs9wu9309/cTDAZ55ZVXsNvtXLhwgdu3b1MulymVSjvyOvtWcep3HJPJhM1me6TiHBsb4/z585hMJqxWK2azmcHBQSwWC61Wi3g8zuTkJJ///OcZHBzkpZdewufz0Wg0aLVavP/++/z0pz/lzp073L59WynOLmAymQgEAoRCIV5//XU+85nPYLVapW9TZ3R0FICJiQnq9TqtVov/9b/+F9VqtRvLVuwSTqeTSCTC5OQk/+Sf/BP8fj/ZbJZUKgVwsBWn7vjXTXKLxYLb7cZsNuPxeDCbzZjNZoxGIx6Ph3A4jNFofOjvjI2NcfjwYYxGIyaTCSEEDocDg8HAkSNHKBQKHD9+nImJCfx+PwaDgXq9TqFQoFKpkE6nSSaTZLNZdVTvMFarFafTSSAQ4OWXX2ZwcJChoSEpd7jvStEtyo3ysVgsuFwugsEgJpOJYrG4K8e5/Y7BYMDj8WCxWOSJrVQqkclk6Nb0XKvVSiAQwO12UywWAWg0Go80nJ6FPac4hRA4nU4cDodUnj6fj0OHDuFyuZiamsLr9WKz2bBarUQiEZ577jlMpoffqh6F1f9us9kkn89TrVZ59dVXmZiYYGxsjLNnz2I2m2k0GlQqFaks5+bmuHPnDslkkmaz2elLcaBxOp2Mjo4yMTHBP/2n/5SxsTH6+vpwOBwA0u+sR9CbzSaapmGz2bDb7fh8PiYmJkilUiwuLlKpVKS/WrE1TCYTw8PD+Hw+7HY7NpuNaDRKPp/v2n5wOp2MjIwQDAZJp9NkMpkdszI3smcUp25Zms1mRkdH6evrk5ai2+1mdHRUXjS3243FYsFqtcpI66MszkeRy+XkR7VaJZfLEY/HEUJQqVRoNBrEYjEKhQLxeJxsNku5XFbWyi6jny7MZjMmk4lQKMTExATj4+MEAgFp+Wx8frvdlulHa2trNBoNhoaGpOI8cuSIdL0UCgUymQyVSqWL73JvoQfkPB4Pbrcbh8NBNpvdcetuu2vSlbjJZMJg2J1U9T2jOC0WC4ODgwQCAb761a/y0ksvSeWo5+sZDIZNx/iNQZ+PQ9M0SqUSFy5cYHZ2llKpRLVaxWKx8NZbb9FsNsnlcjQaDalU4/E40WiUZrOpLM5dRj+C9/f34/f7eeGFF/jN3/xNAoEA4+PjOByOh04V9XqdVCpFJpPh7bffJpFI8Gu/9mt86lOf4tSpU/yH//AfiMfjfPe73yUWi/HOO+9w9+7dLr3DvYduxIyOjsqbl6ZpfPDBBzQajY6vR3e1hUIhgsEgoVAIAJvNtuOvtWcUp25x2u12hoaGmJqawmazbeuitNvtTccxg8Eg70iaptFsNkkkEiwtLVGv1zcpw0ajQTablcf1er1OJpORfhTF7mEwGLDZbFgsFvx+PwMDAwwNDTE2NobH48HhcMjo+UZ0H2e9XiedThOPxykUCjSbTRwOBxMTE/L/qdls7soG28/oFmcgEMDn8+HxeLDb7V2xOHVDyWw243K55I1UPwm2Wq0ddcPsGcXZarXI5XIYDAby+TylUkluqK1Qr9dJJBLy6NZsNgmFQvT398sNViwWuXHjBu+//75MOdLRNG2Tv0z3nyl2H4fDwWc/+1lGR0c5duwYExMT9PX10d/fj8VieeyJwmazMTQ0hNVqle6btbU17ty5g8/nY2BgALPZTDgcptFoYLfbO/zO9jZWq5Xjx49z5swZCoXCrvgSt4p+PB8dHeXll1/GaDQSj8fJZDLEYjESicSO7tc9ozjb7TaVSgWr1Uq1WqVWq23rH73ZbJLNZimVSpTLZRqNBiaTiWAwKBVnrVYjHo+zsLCwe29EsW2sVitHjx7l5MmTnDlzhunp6Yeeo2naQ5aOyWTC6/XSarWkC6dUKpFMJgFktoXb7cbr9W7ykSo+HqPRyMDAAOPj4ywuLnZVcZrNZux2O6FQiMnJSRqNBvPz82QyGXK5HIVCYUdfb08pznq9TrFY5Nq1a1itVsLhMCMjIzQaDfL5PGazmePHj+P3++Xv1Wo1CoUCKysrfPe73yWZTFKtVmk0GgwODjIyMoLH4yESiZDJZCiXy118l4qNWCwWmU42NjbG5OQkHo9nk4JsNBqsrq5Sq9VIp9MUi0VGRkaYmJiQwaFiscjq6irJZJI7d+5QrVY5evQohw8f7mogY6+jX7tuX0ODwcD09DQnTpzg9OnTmEwmqtUqhUKBbDa7KyfDj1WcQog/BX4NSGqadnL9sQDwP4FxYAH4LU3TMju+ug1omka1WqXVanHp0iWSySSTk5McPXqUYrHI4uIiLpdLBg90yuUyKysr3Lp1i7/4i79gbm6OWq0mj+rhcJjR0VHeeOMNuckOAr0i1ydhtVoZGBggEokwNTXFkSNHHjpl1Ot1YrEY2WyWa9euEYvFeP311xkfH6fValEqlcjn8ySTSZaXlymXyywtLWE2m/nsZz/72LLbvUwnZauXNXYTg8HA6dOn+dKXvkQkEsFkMskT5traWncUJ/AN4P8F/tuGx74GvKVp2h+tz2b+GvCHO766R9But8lmszLVQNM0KpUKKysruN1uotEoTqcTr9eL0+mkVCqxuLjI8vIyxWKRWq1Go9GQR/9sNovFYmFmZoZms9nV40aH+QY9JNeN6N2NwuEwZ86cYWhoiEAggNlslsG8Wq1GqVRidXWVq1evsra2RiKRoFAokEgkmJ2dpVKpsLS0RCqVkoEh/f8ll8vRarUwGo34fD7q9Tperxe32029Xt/rVWDfYBdlazQasdlsOBwOGbTbarrfbqFXCOqFLK1WSyrO3agO+1jFqWnaBSHE+AMPfwn4e+tffxP4CR3aYK1Wi/n5eRYXF7l58yZOp1MqPJ/Px8jICKlUiueee47p6WlisRg//OEPiUajZDIZ6vW6jK4Vi0XK5bI8wmmadmCO6r0m143ox/PnnnuOf/Nv/g3hcJhQKLQpYpvJZLh9+zbz8/P8yZ/8CfF4nEgkgt/vlyeSRCLBu+++K1015XJZWkjHjh2j1WrhcDg4cuQI4XCYI0eOsLS0RDqdZmVlpdNve8fYbdnabDYGBweJRCIyDenBEtdOYzabZdELQLVaZXZ2ltu3b5PP53f89Z7WxxnWNC2+/vUKEH7cE3djal6j0ZD14nrdca1Wk+VzxWJxk3muK8oHK0P0yHmz2VQ1y/fpqlz19DC9lFKPnAeDQSwWi6wGarfblMtlVldXSafTpFIp0uk0LpcLo9EogzzJZJKVlRWKxSKlUmlTepl+A9UzMxqNBj6fj2AwSLlcRgix36qItiTbrcjVaDTicrlkpoJu5XUDg8EgC2E25nC3222q1epDct8pnjk4pGma9qQW+7s5NU9PC9I3lO4HrVQq8mL19/fzyU9+krt373Lx4sWu1tHuJTotV7101uFw8Nxzz/HSSy8xMTGBz+fDarXKRsSlUolarcbNmzd57733iMViMqg3Pz/P8vKyPOrXajWy2SzNZvOxXZD0aiSHw8GZM2ew2WxcuHCBhYWFfft/8iTZbkWubrebY8eOMTo6is/n29QfoJMYDAYcDgd2u102enG5XJvKp7sWHHoMCSHEoKZpcSHEIJDcyUVtlY1NHPTvm82mDP5omiYTnAuFAhaLBYPBIJWs4iG6JlchBHa7XQb4xsfHGRgYkDJrNBoyeKf7NmOxGMlkUsr7aVNO9AqzUCgklUG3Ax67wI7J1mw2EwwGCQQCsmIPHj7R7TZCCKxWq1SeDodj08lkYzxjp3laxfkm8FXgj9Y///WOregZqFarXLp0SUbYA4EARqOR0dFRGo0Ghw8fBpBBBMVDdEWuesns2bNnZb7m6dOncTqdAGSzWX7yk58QjUZJpVKsra2xtrbG4uIixWJxR8r7hBC4XC5p9e5Ddky2FouFQCAgA3aADMrl8/mO9W2w2Wy8+OKLjI2NMT09jdPppNVqsba2JvsO6CeVnWYr6Ujf4r5TOSSEiAL/mfsX/y+EEP8KuAf81o6v7Cmo1WrcuHEDq9XK9PQ009PThMNhDh8+TL1eZ2RkhGq1SrFYPPCKs1fkqpfK6VUoeleqo0ePSquhUCjw/vvvc/XqVaLRKIlEQjaobjQaO+LDEkJgs9keahayF9lt2erNo71eL2azWfZ5WF1dpVgsdszqtNlsnDlzhpMnT3Lo0CGcTieFQmFTk56uWZyapn3lMT/67A6v5ZnRj+561cDPf/5zjh8/zujoKFarlVOnThEMBvH5fMTjcekPLZVKrKysHKhGHb0iV6PRiNfrlSWQepsyIQSZTIZf/vKXrKysMDs7SywWI5fLUavV5PFd9dF8mG7IVg/W6SlfO4nRaJRBQ703gd7IY3p6mtHRUdxuN3A/2+L69evcvXuXQqEgy6R3mj1TObRVdAvk4sWLzM3N8YUvfIHz58/jdDr54he/SLVa5datWyQSCZLJJLFYjMXFRbLZ7IFJfu8lLBYLAwMD9PX1cfjwYU6cOCF9ZtFolP/+3/87y8vLXLt2jbW1tUf60ZS/urtomkYmk+HevXuk0+kdV1R6Y3K9EGJwcJAXX3yRQCDAc889RzAYlGlIsViMv/3bv2V5eZl0Or1rbQL3neIENiU5641q3W43TqcTl8tFX1+fTF/Qndu3b9/GbrdLn9mDTT4Uu4PZbJbdjvSjX71ep1KpkM/nZbqRXjW2k+gVQ0rxdgaTySRb/+nX3mazyYbiG5uKG41GnE4nJpNJpjwNDw/LFnYulwu73S77s+rBvHq9TjabJZfL7eoJcl8qTrif3F6pVLh06RL/9b/+V0ZGRviH//AfEg6H5cjYZrNJvV5nbm4Oq9VKIpHgww8/JJ1OUyqVVFPbDuD3+/niF7/I1NQUY2NjwP3jVjQa5ebNm9y8eVN2tdpJHiy13G9ll51GH2XyuPlecD+Nqa+vb9PIm0OHDuH1emUrOD0v0+PxcObMGTwej8zv1X9WrVZZW1uTFULNZlP6yguFgrR8d7N72b5VnK1WS5Zdzc/PA5DP56WQdEvTYDBQrVYZGRnBZDIxPz8vncp6UryySHYe3aqw2WzS4tSj2Xrn/VwuJ2+AnUCdMrbGg+4SPS1I7wTvdrsfynQQQuD3+6XihPut4IaHh/H7/Xg8HpxOp1SAPp+Pqakp2RwZkOlFer627pbb2Ge32WzKisDdlOW+VZw6mUyGq1evsry8TK1Wo7+/n5dffpnJyUnZ5KO/v5/Pf/7zZLNZvF4vy8vLfPTRR9y6dYt6vX6Q6tc7hl4ee/jwYcbHxxkeHsbhcKBpGqlUiqtXrzI/P79rncT1jaZ/tFotVldXpb9b8Xja7fam6j0hBGfPniUQCLC2tkY0Gn2kW6Wvr0+28tMVpF59pH+vu9ja7TbXr1+XcikWi6TTaRKJhCyT9vv9/P7v/z5erxeTySRLplOplCx82C32veIsl8uUy2VZ7O/z+fD7/TJZtr+/H5fLxfHjxykWi6ytreHz+WQneP1vKKtzZ3E4HAwNDclxKD6fT/q4CoWCbM6xm3PPN1pOekpNNpuVg9sUj0c/0el9UCORCP39/XLK5aOsvVAoxODg4Ca3iP439HxLvVl5oVCQkfHFxUXS6TTLy8vyRJjP5xkcHOS3f/u3ZWaFpmlyftRuGzv7XnHq6G2marUa7733HtFolImJCY4cOUJfXx+nTp3CbDYzOTlJIBCg2WwSDAaZnZ3l3Xff7coMlf3MxhpjPd1Et2ASiQS3b98mkUjs2HXX3TJjY2MMDg5y9OhRTCYTjUaDVCpFoVDg0qVLfPTRR8zNzSnF+QTW1tZ4++236e/vRwjByMiI7JRUr9cf2yhH70imT4rVc6r1MTT5fJ5KpSK7mOl9VvVJldlslnw+j91u5/Dhw0QiEYLBIHa7nVarJcd2d0J2B0pxptNpDAYDmUwGq9XKoUOHOHbsGCdOnODQoUP4/X6OHTtGu90mGAxy7Ngx3nrrLX7xi18oxbnDbJxYqSvOWq1GtVolGo1y9epVSqXSjl13XUkfO3aM8+fPc+rUKUwmE7VajWg0SjKZ5Gc/+xlvv/32pg5aiodJJpP84Ac/wOPx0Gq1GB8fZ3R0lMHBwSf+nj4BM5VK8eGHH1IoFIjFYhSLRZaWlojH7/cg2XjtN54I9M8ul4tTp04xMjIiR0Lrnd47dTo8MIpTR/fPaJom/TH9/f3ySKj7Wlwul0yy9fv9mEwmSqXSrh4dDyr6sW3jcLVqtbojUVFdQQ8MDOB2u5mcnOTQoUMEg0E5USAajRKPx8nlcg8N6VM8Gn0iw/LysrT2ttqKL5vNcu/ePcrlMul0ettdjMxmsyz51Ku89JLPXC6nFOduUa/Xqdfr3Lt3j1gshsVioVKp0G63peO6r68Pv99PNBrl6NGjpFIpZmdnD0y/zm6gp5boUfVnLZcTQmAymXC73bzyyiuMj4/z6U9/mhdffFEO54tGo1y4cIF79+6xuLhIrVZT1uYWqVQqvP/++/LUsNUOSbrS1W+UejR8qzidTg4fPixdBJqmsbS0xEcffcT8/HxHjJsDozj19Bd9hOhGYev5YxvRfWIqt68z6AGCjY0ZnlZp6v5Ti8UiG3cMDg4yPDwsI7DFYpFYLCaP6aurqyootE30QpNOo6c/6Z2zABkA7lSt/IFRnGazmf7+fhwOB4cOHSIcDuP3+2ULM6/Xu+mOmcvl5JHi9u3bskZasTu0Wi0WFhbkmJNn+ef3eDwEg0EGBgZ48cUX6e/v59VXX2VwcJB8Ps+HH37Ihx9+yHe+8x1yuZzsW6DSzvYm7XabaDTK5cuXWVxcVBbnTqD7LPWJiW63m7GxMUZGRmRTCb3DuI5u/ZRKJTkpr5NdXw4iGzvsPK3lp6e52O122TTk+PHj9Pf3Mzk5STAY5Pr168Tjce7cucPPf/5zVR22T9DHPiuL8xnRa1n7+/uZmprC6/Vy/PhxvF6vrHd1OBy43W7sdrusodWboOq5YrrvUynNnWdjPp/RaGR4eBiz2cz169e37SIxGo0cPnyYcDjMsWPHOHPmDIFAgCNHjmA2m1lZWWFxcZEf/ehH/PznP2dpaUllSiiemq304xzh/rS8MKABX9c07Y97bZTsg+jt9A8dOsSnPvUp+vr6+MQnPoHP58Pj8WCz2R75e3pStF4xVK1W96XS7LZcNypG3f88MDCAx+MhEAhsW3GaTCbZx/OVV17h05/+NBaLBYfDQaVS4b333mN5eZm3336b73//+zv9dnqGbsv1oLAVi7MJ/HtN0y4LIdzAJSHE3wK/Sw+MkgVkwMdsNhMKhXA4HExOThKJRBgdHeXo0aO43W6pMB+M/jUaDakkl5aWKBQK3Lx5k3v37nH9+vX9mp7SM3LVb0z62IPR0VE+8YlPUCgUSCaTm2qSnU6nHLERCoUwm82yucTZs2eJRCIMDAzIBi5ra2sUCgXu3LkjJ53uc3pGrp1AP7XotfKdag25lUbGcSC+/nVBCDEDDNMjo2QBmXLicrl44YUXGBwc5Pz585w+fRq3201/f7+MtD6qC061WmV5eZnV1VW+973vsbCwwI0bN5ibm5MbcL/RbblurBPXsdvtWK1WnnvuOf7xP/7HLC8v87Of/WzTlEpdMY6Pj3Pu3Dk8Hg+RSASHw0EwGMTlclEulymVSuRyORYWFlhdXeWdd95haWmJaDS602+lp+i2XLuBEELu82KxiMFg2PUA0bZ8nOuzms8C77OD40a3i54qpPf3c7vdjIyM4PV6mZycJBwOMzAwIOfH6E0EdPQkZ32DZbNZZmdnWVtbY2lpiZWVFbLZ7IHJ2ewVuW5s/BCJRDAajSSTSUqlEsVikVarRSQSIRwOE4lEGBwcxOl0yka2JpOJVqslU1PS6TR3796VX+sltweFXpFrJ9iYbtgJtqw4hRAu4C+BP9A0Lb9RET3ruNHtIITA4XBgtVplw46pqSn+wT/4BwSDQcbGxnC73XLy3YO5mO12m0QiQTab5erVq1y+fJlkMskvf/lLyuUy+XyeWq12YDZYr8h1/e8ghGB8fJxwOEy5XOaVV16RjY1brRb9/f2yYsRut8vZQ5qmsbKyQi6XY3l5mYWFBRYWFvje974n29PV6/UDExDqJbnuNvqppZM511tSnEIIM/eF8Oeapv3V+sMdGyW7sfGpyWTC4/Fgt9sJBoOEw2GGh4dlGV04HN40pXBjdYJuaeoWyPLyMnfv3iWZTDI3N3fgEqC7KVddLnq1UKvVkjc5u92O3W6XGQ96NVGr1SIYDOL1euXf0P9Os9kkl8uRTCZJJBKsrKwQi8Wkz/og0e392kn0/aqfQDs1330rUXUB/Akwo2naf9nwo46MktVHXASDQT71qU8RCoVk9NXv98uNND4+LkdhbKRYLJJMJuXgr7W1NW7duiU3WCKRkNUqB0xpdlWuxWKR+fl5jEYj9+7dQwhBKBTC5XLJ5xiNRtxuN61WC5fLhaZpUr7NZlPOWdfdLG+99RY3b96kXC5TLBZlOtlBotty7QZCCILBIBMTE7It3W6zFYvzk8A/A64KIT5af+w/0aFRsnrpnN/v59y5c4yOjjIyMoLf78fr9RIIBJ74+9VqlXQ6TTwe5xe/+AXxeJzr16/LqZb7NGK+Fboq11qtJi3/dDotixMeVJwbLYiNN7ZWqyVHB8/Pz7OyssLPfvYzLl++vBvL3Ut0Va7dwul0yhtvJ47sW4mqvwM8biU7Om5Ub5lvs9kYGRkhHA7jdrsJhUKEQiGZ1Ozz+bDb7Y/MxdRrnJeXl0kmkywsLHDp0iVWV1e5ceMGuVyOfD4v01sOKp2U66NoNBqyXvytt97i2rVrnDp1iqGhIYaHhxkZGXloA2iaJuWaSqWYm5sjm81y48YNstksiURit5fd83Rbrp2mW7OieqpyyGQy0d/fj8/n4/XXX+fcuXMy4GOz2WTeHjzaEbxxrvqdO3f46KOPuHbtGn/zN38jG6iqKqDeQB+9UKlUePPNN3E6nbz22mtMTU1x/vx5IpHIQ7+jaRoLCwt8+OGH3L59m3fffZd8Pi+HuSm5Hlw6rTx7SnEajUZ8Pp9MJxocHMTj8eByubBYLDIPs1ar0Ww2qVQqlEoluWEajQaJRIJisciVK1e4ffs2y8vLsrejUpq9h17eCvfnqGuahslkotlsbsqI0Ethr1+/zuzsLNFolFwuR6VSeeb2c4q9hR7gdTqdXcux7inFaTabOXLkCFNTUzz//POcPXtWRsr0HL92u002m6VQKLCwsMDs7KxUiIVCgR//+MdEo1Hy+byckb5fyyb3A3oT3GKxyHvvvYfJZOK73/3uQ0E+HX3SoR4c0k8ZioNDsVhkZmaGYrHIiy+++FAhRSfoKcUJyONbPp8nnU4/ZIK3Wi05I2ZpaYmlpaVNijMajcqJlvux4mc/oluLB6XgQPFs6DOK7HY70WgUi8VCIpFgbW1t0wl0NxGd1NQfl1Cr+zgdDgc+nw+32/3I5+mNbkul0qbcy2azKQc86bmbPcYlTdM+0e1F7DR7JVF6F1Fy7SAWiwWfz4fVamVoaAiHwyFPmLrPe4dcN4+Va09ZnM1mk1gs1u1lKBSKHqZer5NM3s/f10d4d5rOFHYqFArFPkIpToVCodgmSnEqFArFNlGKU6FQKLaJUpwKhUKxTTodVU8DpfXPe40Qz77usZ1YSA+i5Lo/UXJ9DB3N4wQQQnywF3Pe9uq6O8VevT57dd2dYq9en91etzqqKxQKxTZRilOhUCi2STcU59e78Jo7wV5dd6fYq9dnr667U+zV67Or6+64j1OhUCj2OuqorlAoFNtEKU6FQqHYJh1TnEKILwghbgkhZoUQX+vU624XIcSIEOLHQogbQojrQoh/t/54QAjxt0KIO+uf/d1ea6+wF2Sr5Lp9lFyf8LodafophBG4DXwOiAIXga9omnZj1198m6zPnB7UNO2yEMINXAJ+A/hdYE3TtD9a/yfya5r2h91baW+wV2Sr5Lo9lFyfTKcszheBWU3T5jRNqwP/A/hSh157W2iaFtc07fL61wVgBhjm/nq/uf60b3JfOIo9Ilsl122j5PoEnklxbsOUHwY2dhyNrj/W0wghxoGzwPtAWNO0+PqPVoBwt9a122zziLbnZHtQ5Qr7e892Uq5PrTjXTfn/D/gV4DjwFSHE8Z1aWLcRQriAvwT+QNO0/Mafaff9G/syj0vJdX/KFfa3bDsuV31C3HY/gJeBv9nw/X8E/uOTnru++IP8kXra692pj+3IdcPzu31du/3R83J9yj3b7eva7Y/HyvVZuiM9ypQ//+CThBC/B/wecOoZXmu/cK/bC9gC25WrYm/IFbYgWyXXTTxWrrseHNI07eva/S4lX97t11J0Dl2u2h7snKN4PEquW+NZFOcyMLLh+8j6Y49E07TvPsNrKTrHtuSq2FMo2e4Qz6I4LwKHhRATQggL8NvAmzuzLEUXUXLdvyjZ7hBP7ePUNK0phPi33A/6GIE/1TTt+o6tTNEVlFz3L0q2O0dHuyMJITr3Yr3Jpf3oO1JyVXLdpzxWrqrJh0KhUGwTpTgVCoVimyjFqVAoFNuk0+OBex6DwSA/CyE2/azVam2srFAoFF1ACPHQ3uz0vlSKcwM2m40TJ04QDAaZnp5mbGxMCimVSvHWW2+RTCZJp9MUi8VuL1ehOBCYTCaMRqP8cLlcDA0NYTabsdlsCCGYn58nGo3Sbrdpt9u7v6Zdf4U9hMVi4dixYxw6dIg33niDV155RSrO2dlZVlZWMJvNlMtlpTgVig5hMpkwm82YzWYsFgvBYJCjR4/idDrxeDwYjUZqtRrx+P1mSEpxdgi73U5fXx+hUIhjx44xNTVFKBQC7h8BNh4L1FFdodg9TCYTVqsVm81GJBLBbrfj9/txOBw4nU6cTider5fR0VH5PCEExWKRdrtNOp1mfn6eVqu1u+vc1b++R/B6vTz//PNEIhE++9nPcuTIEaxWq1SQ7XZbKUyFogNYrVYCgQDhcJi///f/PuFwmPHxcYLBIKFQiP7+fgwGw6ZYRLvdJhQKMTY2xuXLl1laWlKKsxMYjUZ5N7PZbNhsNoxG40PWpqL7bDyyORwOTCYTFosFk8kkfWE6tVqNbDZLs9mUgT2LxYLZbKZer1Mul2m1WtRqNXVT7BJms1nuP7vdjs/nY2hoiL6+PkZHR6Wy9Pv9eDwebDYbcN+YEUJgMt1XYT6fj4GBAfx+P3a7HU3TaDQauyZXpTi5Lzy/308gEMBut2MymZTC7EGEEIRCIYLBICMjI5w8eRKPx8P4+Dgul4tQKITb7ZbPv3fvHt///vfJZrMUi0UajQbDw8P09/ezvLzMlStXKBaLRKNRqtVqF9/ZwcRoNBIMBnE6nZw9e5Zjx44xMjLCmTNnsNvtBAIBLBYLVqtV+jEzmQzNZpN6vY7RaCQUCmG32xkbG8Pr9VIsFrlw4QKZTIZUKkWj0diVtR9oxSmEwGg0YrVacblc0oIRQsijeavVol6vy42nWy6KzmIymTAYDHg8Hvr6+hgYGGBiYgKfz8fhw4dxuVz09/dLxSmEwGazMTMzg8fjIZvNUq/XGR0dZWhoCIDl5WUMBsMmK1Wx+xiNRnlq8Pl8eL1ehoeHmZiYYHx8nCNHjsiTgW7AaJpGsViUirNWq2E2m/H5fMD9OIUQArfbjd1up1QqyeP8bnCgFWcwGCQcDnPixAlee+01wuEwfv/9KaKFQoFisciNGzf44Q9/SDKZ5NKlS+RyOUqlUpdXfrCwWCxMTU3h9/t57bXXeP755/H7/QwNDWG1WmVkNZvNEovFsFqtWK1WTCYTn//85+XNr91uyxvk6OgobrebxcVFotGokmkH0KPj4XCYl19+mUAgwMmTJwmFQgwMDNDf34/L5cJms8k8ak3TyOfzVKtV3nnnHX74wx/SbDaB+/v3q1/9KseOHZNWqdvtxuVyUalUlOLcLZxOJwMDAwwPD3P48GFCoRAOhwOAarVKPp/nzp07/N//+3/J5/Pk83kajcaumf+KR2MymQiHwwwNDXHmzBlef/11bDYbTqdTpos1m01isRjJZBK73Y7T6cTlcnH8+HHpB9vofrFYLBSLRQwGg/SbKXYXg8GAxWIhFApx5swZhoaGePHFFwmHw1itVsxm80O/o2ma3Iu3b9/mrbfeot1uYzKZGBoa4stfvt8fXc/x1G+aG63V3eDAKU4hBD6fD4fDwZkzZ3j11VcZHR3F4/HIi91ut1lcXGRmZoZbt26RzWYpl8vSalFH9d1HCIHVasXr9eL3+/nkJz/J1NQUU1NT2O126vU6sViMYrHI7Ows+Xye2dlZksmk3DyRSITXX38dr9dLMBiUN0WASqVCIpFgdXVV3Qg7hN/vJxKJMD09zalTp+jv78fn82GxWB6yDlutFuVymUqlwoULF7hz5w4ffPABuVxOJsF3k49VnEKIPwV+DUhqmnZy/bEA8D+BcWAB+C1N0zK7t8ydw2Aw0NfXR19fHy+99BL/6B/9IxwOBz6fT/q6Wq0Wc3NzXLhwgbm5OdbW1qjX611e+c7Sy3IVQmAwGHA4HAwPDzM8PMwbb7zBmTNnsFgsWCwWSqUSi4uLLC8v89d//desrKwwOztLKpWS0fVTp04RCAQYHh7G6XRuUpzlcpl4PE4ymdx3irNXZavnSZ88eZJz584RCAQeaxW2Wi1yuRyZTIYf/OAH/PjHPyafz5PJZLDb7Ztk2Q224gT4BvCFBx77GvCWpmmHgbfWv98TGAwG/H4/w8PD0grRfSobHdCJRILl5WXW1tZ2PSesS3yDHpWry+VicHCQ8fFxzp07x5kzZwgEApjNZmq1GmtraywtLfHRRx9x/fp14vE4qVSKcrlMo9GQWRLBYJBAIIDX68VkMqFpmgws5PN5ebTXfWb7iG/Qg7J1Op0MDg4SDAYfOko3m00qlQqFQoFkMkk0GuXKlSt8+OGHxONxSqWSTBvTb6oOh2NX/ZhP4mMtTk3TLqwPet/Il4C/t/71N4GfAH+4kwvbLYxGI9PT05w7d47p6Wn8fr9MqK3X60SjUVZXV/nwww959913ZQ7gfqOX5To0NMSpU6c4duwYv/M7v0MgEMDtdmMymVheXmZ5eZn33nuPP/uzP5NWSKPRoNls0m636evr48SJE5w+fZqTJ08SDAaln7NcLlMul5mfn+e9994jk8nsu8BQr8p2YGCAc+fOyTrzjZRKJQqFAplMhmg0SiwW4zvf+Q6xWIxoNEo2m5WllFarlf7+fvr7+7FarZ18C5Kn9XGGNU2Lr3+9AoQf98ReGjeqJ8x6vV6ZumI0GmX0rtFokM1mSaVS0q95wOiqXB0OB2azmb6+PiKRiMy59Hg8VKtVqtUqqVSKaDQqj9nlcplqtbqpPtnhcBAOhwkGg9jtdiwWC/B3KS2rq6tkMhkKhQLlcrkjtc09wJZku5v7tdFoUK1WpZKs1+syxW9tbY18Pi9PE/F4nHg8TiKRkIUKOgaDAavV+kjfaKd45uCQpmnak1rsa5r2deDr0N1W/Lp57/P5OHnyJK+++qp0MDebTarVKmtra7zzzjvcvHmTubm5bi21J+i0XC0WCy+88AKTk5O88MILvPrqq7jdbpxOJ9VqlQ8//JBEIsGFCxe4dOkS6XSafD4vrcyNTE1N8au/+quEw2Hsdrt8vN1uc/nyZd555x1mZmZYW1ujVqvtyxPFk3iSbHdzv16/fp1vfvObDA0NMT8/j8lkYn5+XirMbDZLqVRibW2NarVKMpmkVqs95IM2mUy43W5p+HSDp1WcCSHEoKZpcSHEIJDcyUXtBkajEZvNhsPhIBQKySTojUnu5XKZ5eVlFhYWyOVyXV5xV+iKXA0GA2azmaGhIY4cOcL09DTHjh2TdcjlcplEIsG9e/e4ffs2V69epdFoPBSw01OTvF4v4+Pj0rcJ95Vmq9UikUhw8+ZNotEotVptP/o3H0fX9+zq6ir1ep1CoUB/fz8mk4lr166RyWRIp9Pkcjmq1SrFYvGJmSv6/8tetDjfBL4K/NH657/esRXtMHpNcygU4o033mB4eJhDhw5tek4ul+Ojjz4iHo8zMzPD3bt3D6ri7Lhc3W43J06coL+/n09/+tOcPn2acDhMu92mWCyysrJCKpXiRz/6EXfu3GFubu4hK1HP4QuHw7Jzjs/nw+l0YjAYqNVqXL9+nUQiwcWLF7lx4waFQuGgWZpd37O60lxaWuLtt98GIJ1OU6vVqFQqUq4fl+6nd04aHh7uWg7uVtKRvsV9p3JICBEF/jP3L/5fCCH+FXAP+K3dXOSzYDab8Xg8RCIRPv/5zzM5OcnIyMim5xQKBa5fv87S0hJ3794lGo12abWdo1fk6nA4eO655xgbG+P8+fOcPHlSWofFYpF79+6xvLzMxYsXuX79ugwCbXgfGI1GLBaLTJIfHBzE6/VKi6Rer8sb4tWrV7l79+5uv62u0iuyfRC9eKRUKpFIJJ7671itVsLhMP39/dJ/3Wm2ElX/ymN+9NkdXsuuYLfbGRgYkGkQPp9PRvSKxSL5fJ6lpSVu3bpFPB4/MAGhbstVrxF3uVxMTEwwOTkp68z1lLDFxUXeeecdVlZWSKfTm/yZJpMJm82G2+3m6NGj+Hw+jhw5Qn9/P1NTU5hMJmq1GslkktXVVW7evMmdO3dYW1vrxNvrKt2W7bPg8XjkMV73Xz7YD/fQoUMEg0G8Xq/cy6VSiUqlQiqVksHd3TxR7PvKIZ/Px9GjR6WlOTg4KAWSyWS4e/cuV65c4cKFC6RSKQqFQpdXfDAwGo3Y7XaCwSAvvPACR44cwefzoWka6XSaO3fu8NFHH/GNb3yDdDotq7Z0bDab7MH4L/7Fv2B8fJzR0VGZemQ2m8nlcly7do3l5WV++tOfMjMzc2BujHuV/v5+XnvtNWw2G1arVRZD6LTbbQYGBhgbG8Pn88m+uaurq6ysrDA3N8fc3BylUmlX/df7VnHqFz4UCjEyMsLAwIBMdNd9Kclkkvn5eZaXlykWiw+ltSh2D31D6Jaj3pkKIJ/Pc+/ePVZWVqRMdOvCZrNhsVikZRKJRBgcHCQUCuHxeDZF0ev1Oqurq3JGVKVSOUjBoJ5GD+Tp/Tj1rkaHDh1icnISm82G2WzeNDRR9336/X68Xq+sHmo0GqTTae7evUsikZDunN0sjd6XilMIQSQSYWRkhPPnz/Obv/mbeL1evF4v7XablZUVMpkMP/zhD/nOd75DNpuVET+lODuD7ps0m82yIYduWdy4cYM///M/J5vNyhLZ559/nv7+fnly8Hq9DA4OYrfbGRoakhttI9lslg8++IDFxUVSqZRqWNwj6FFxs9ksOyKdP3+e06dPMzIywnPPPffQUX1jq0eTySRvkHqPzp/+9Kf8n//zf0gkEh2R875TnPqG9Hq90rc5ODgok6v19JZCoUAqlWJpaWlTAw9F59A3hF65pVsWev/TZrOJx+PB6XQyPDzM4OAghw4dYmJiAo/HQzgcxmw2Y7VaNx3n9A1Wq9VYXV2V+ZpKaXYXXdZmsxm32y1PhB6Ph9HRUaampuR+NRqNcj/qWRMPjq9pt9tUq1UajQa5XI5EIkEul1PD2raLwWCQJv+nP/1pfv3Xf112Bdcb4bbbbUqlEul0mkwmQz6fV0qzC+j5s3qUtVQqYbPZMJlMvPrqq4RCIVqtlmwh1tfXh8PhwOPxyMIFvfKkVqshhJA3R/2xVCrFjRs3pCtG0R30m6LH48Hv9zM2NsYXv/hFfD6f7OAeCoXw+/2y3V+9XmdlZYVms0kkEsHv90sFqqOPtjEajQwPD3PmzBkWFxdlCa4KDm0RIQR2ux2Xy8X09DSvvvrqI59XrVZluV21Wj1o+Xw9gaZpMu2oXq9TrVYxm82YTCbZPm7jcx+kXC6Ty+Xk5tF7PZrNZjlHqFgsyrI9RffY2O0qEAgwNTXF5z73uU1t5XS/pP6/UCwWWV5eptFo4PV65c3ywYR3/W/7/X5GR0cplUqYTCZ501Uzh7aA1Wrl6NGjDA4OMjAw8Mhha41Gg9nZWen7Use37tBut6VV+MMf/pA7d+5w/vx5Jicn5eC1ZrNJqVSiXq+TTCZl/l86naZSqVAsFnG5XLIjvN1ux263s7q6ysLCAouLi/uuZdxeQnfB6H7Ms2fPcv78eSKRCH19fZhMJm7evEmhUJAll4VCQSbF5/N5mX3hcDhwuVyb/NhCCCwWi2zco/vKl5eXyeVyxONxarXarijQfac4T58+zbFjxxgeHn7kxWo0Gly/fp2f/OQnrK2tqSN6l2i1WrRaLVZWVvjf//t/09fXRyAQYHBwUI69qNfrcjN9+OGHJJNJLl++zI0bN6jVapTLZVklNDk5SSAQACCVSnH16lXm5uaU4uwSG08BelHCG2+8we/8zu/IwE8ul+PKlSvMzc3JwpNEIsH8/Dxwvw2dx+Ph0KFDDA0NSVfcxtfQg0QnT57kxIkTOJ1OZmdnWVlZIZfLyei6UpyPQK9D93g8BINB+vr65AXVL1iz2aRQKJDNZsnn8xSLRWq1WjeXreC+XLLZLAAfffQR7XZbTjbUI6aVSoXZ2VkymQyxWExG2/X0JK/Xi8fjkVUkut+0Wq2qE0UX0Lv3DwwM4PF4OHHiBKOjo0QiEelK2dgJaWFhgVgsRjqdli3+HA4HR44ckYP5XC4XFotFjtJIp9Pyf0XPzNDzgk+ePCnLdrPZLMlkkkKhQLvd3mQo6cGlp3HV7QvFabPZZNT16NGjnDhxglAotOk5lUqFmZkZEokEc3NzrKysKGuzB9B7oMbjcRYXF2XSs55+ov+z685+vWwvGAwyPj7O0NAQhw8fZnJyUrYI1CtINvZwVHQGPVgTCAR44403iEQifOYzn2F6ehq73Y7BYCCXy3Hz5k1isRgXLlzgxo0blEolyuUyJpMJp9PJ6Ogo//Jf/ksmJiaYnp4mHA7LgGIikeBHP/oRtVqNwcFBXC4Xhw8fZmxsjMOHDzM0NEQ+n+fatWusrq5y4cIFbt++Lf2n+s200WiwvLz8VP1Y94Xi1O9ug4ODsrmD7gtpt9vSV5ZIJIjFYhQKBZUI3SPoXdn1zuzb+T2TySRHaejWiKZpsplEpVJRirPDWCwWXC4XgUCAoaEhhoeHpRumWq2SzWZJp9PEYjHi8bjMbNH3o91up6+vT6YlDQwMyNOj3iE+Ho/L7latVguXy4Xb7ZZFFBaLBafTSTgclg1BqtUqtVpNKk49LXGjlbsd9oXiHB8f51//63/N8PAwU1NTcvAaIDvsLCws8K1vfYt79+6xsLDQ3QUrnhndf6ZXG+lKU9M0YrEYv/jFL+QsdcXuo58SxsbGeOWVVxgZGeFXf/VXNzWivnr1Kh988AHxeJyLFy+SzWaZm5ujUqnIPhLHjh3js5/9LOFwmFOnTuH1eqlWqyQSCT744APefvttkskkH374IfV6XY4TnpiYkCXV09PTcvzK4OAgQ0NDMkjUbrfljXVlZYU//uM/fqr+BftCceqtySKRyKbSPUAGGFKpFHfu3GF+fl75NvcBerbEg+V4gDxdHKDu7l1Hj6DrwZyRkRHGxsYIBoNUq1Xq9TqpVIqZmRlWVlaYmZmRjTn0II/f72dkZITTp08TCARkXqc+mntxcZGPPvqI1dVVZmdnaTQa2O12zGaz7OyfzWZxOp2EQiE5vbavr29TNL5SqZBOp3G5XE899G1fKE6DwSBTUR7sCJ1KpfjFL37BvXv3pAWi8jb3Pvo4YJvNpm6EXUYIwcjICMPDw5w7d47XX39dNuAol8tcvHiRhYUFrl69ypUrV2g0GvI4rpfPbmzCMz4+TqPR4PLlyxQKBWZmZojH49y6dYu7d+9SqVRk3856vU6z2WRlZWXT6TIQCMjPExMTMuMCYGVlhffee0+O53gattKPcwT4b9yfUaIBX9c07Y97Ydyojh5V1zulbLQ+MpkMV65cIR6PUygUVHrKOntBrk9CrzYKBALqOL6BbshVCMHQ0BCnT5/mzJkzPP/88zIrolwuc+XKFX7xi1+wsLDAnTt3cLvdHDp0CL/fz2uvvcbw8DDT09OMj4/LzlaJRIIbN26wtLTEe++9x9zcnMyK2YjuG02n06TTaQCuXLmC3+8nl8vR399PqVRidHRU/s6dO3d48803SafTJJNP1wh/KxZnE/j3mqZdFkK4gUtCiL8Ffpf740b/SAjxNe6PG+3o1LyhoSFGR0c5fvy4jMY+iF6vnMlklKW5mZ6V61aw2+14vd6HjmGKzstVCMHg4CCnTp1iZGQEk8m0qftVJBKhWCwSDoeZmJjA7XYzMjKCx+ORk2b1MSe5XI6VlRXi8ThXr14lFouRSCRkIcRWqdVqLC8vyzaRG2eI6alPxWLxqXXCVhoZx4H4+tcFIcQMMEwPjBs9fvw4X/rSlxgfH8fhcDxScRaLRRYWFmT3I8V9elmuW8Hr9TI9Pc3k5GTXRsT2It2Qq8Fg4NixY3zxi1/cFGPQc211harXmutTSPWWcrp/1GAwsLKywg9+8AMWFxf53ve+RzKZlKlo28nJLZfL3LhxAyEEH3zwwaZSTT2t7VkS47fl41yf1XwWeJ8ujhvVa5r1Mb8+n0/6NvWLoTues9msdE6rQMGj6RW5bgeTyYTD4ZA9VhUP00m5blR+G/4WRqMRp9Mp08f0vav7J/WUIn388+zsLIuLi8TjcUqlkoyGPw36MX433HNbVpxCCBfwl8AfaJqWf6CdfcfGjRoMBkKhEF6vl8OHD3Py5EnZ/UhvGtFsNpmZmeHmzZtcunSJtbU1SqWSUpyPoFfkul1sNht9fX2y47tiM52Uq6ZpVCoVOeDQ4/HI0ShGo5G+vj78fr/skqQnwOszpXK5HHfv3uXevXtkMhmi0ajM+ezVPbul/zghhJn7QvhzTdP+av3hrowb1cu5bDYbTqdTJr7q/xi68tTLufSxC9s19Q8CvSTXp2FjH0+90gh4pMvmINENueqnO7ivOPUj+Ea56EZNsVgkmUzKFKNMJiNTBfW56vo+7lW2ElUXwJ8AM5qm/ZcNP+rauFG9MsDlcskaZSGEHCmrp0B8+9vfJpvNbkpfUNynF+W6HYrFInNzc5jNZur1+iPzOQ8i3ZBrq9XiJz/5CdFoVJY96yO59aF5jUaDpaUlOQ9I79Suu9Ly+bys6NsLe3UrFucngX8GXBVCfLT+2H+ii+NGdaezbnnqUTy9VVmpVGJhYYGPPvroY//WAabn5Lod9EYP/f39myyTg25t0gW5aprGnTt3WFhYIBKJkMlk8Hg8jI+PY7VaKRQKVKtVrl27xgcffEC9Xpeus92eDbRbbCWq/g7wuP/Gnh83qng0e12u1WpVDmLTezna7XZMJhNWqxWv14vBYDhwvu1uyVUPxKytrXHz5k2sVitLS0ubLE494KO3FNzNRsO7jfKqK/YkehTW7/eTSqXo6+uTR0ObzUYwGATuV44pdh9dGSaTSVKp1CMt/93oi9kt9qTi1Id56ZUEdrsdp9PZ7WUpukC9Xmd5eVmmJjmdTvx+P1NTUzidTtLptByRcpAsz26ynxTk49hzilNvTtpoNGQJVzAYZHJyUuXzHUDy+Tw//elPmZubk5Mvjxw5wpe//GVu3brF2toa6XSaRCJBpVLp9nIV+4Q9pzjhfkKrnue1vLwsG6AaDAbW1tYoFotP1WNPsfdoNpusra1htVrJ5/NUKhUsFgv9/f1kMhlCoRDtdptMJqM6wit2jD2nOPVk21qtxk9/+lNmZmawWCyyPZQ+LW9xcbHLK1V0Ar20bnl5mTNnzuB2u/H5fBw+fBin00mxWCQWi/Htb3+barUq010UimdhzylO+LsIXiwWIxaLdXk1im6iW5yNRoNEIkEikcBut+PxeKjX60xOTsqu5BuryxSKZ2FPKk6FQmfjqIy3336bhYUFzpw5w+rqKjabjfHxcdxuN4ODg8TjcXK5nGotqHhmlOJU7Gk0TZMD3C5dusSVK1fI5/PYbDbGxsY4duyYnH7qdrtVgEixIyjFqdg36C6c5eVlLl68yPz8PMlkUo4Xzufzqlu8YkdQilOxb9Atz2vXrnHz5k2EELKFmd6+TEXVFTuBUpyKfYc+blih2C06rTjTQGn9814jxLOve2wnFtKDKLnuT5RcH4Po9NFFCPGBpmmf6OiL7gB7dd2dYq9en7267k6xV6/Pbq9b1SgqFArFNlGKU6FQKLZJNxTn17vwmjvBXl13p9ir12evrrtT7NXrs6vr7riPU6FQKPY66qiuUCgU20QpToVCodgmHVOcQogvCCFuCSFmhRBf69TrbhchxIgQ4sdCiBtCiOtCiH+3/nhACPG3Qog765/93V5rr7AXZKvkun2UXJ/wup3wcQohjMBt4HNAFLgIfEXTtBu7/uLbZH3m9KCmaZeFEG7gEvAbwO8Ca5qm/dH6P5Ff07Q/7N5Ke4O9Ilsl1+2h5PpkOmVxvgjMapo2p2laHfgfwJc69NrbQtO0uKZpl9e/LgAzwDD31/vN9ad9k/vCUewR2Sq5bhsl1yfwTIpzG6b8MLC04fvo+mM9jRBiHDgLvA+ENU2Lr/9oBQh3a127zTaPaHtOtgdVrrC/92wn5frUinPdlP//gF8BjgNfEUIc36mFdRshhAv4S+APNE3Lb/yZdt+/sS/zuJRc96dcYX/LtuNy1Ud5bvcDeBn4mw3f/0fgPz7pueuLP8gfqae93p362I5cNzy/29e12x89L9en3LPdvq7d/nisXJ+lO9KjTPnzDz5JCPF7wO8Bp57htfYL97q9gC2wXbkq9oZcYQuyVXLdxGPluuvBIU3Tvq7d71Ly5d1+LUXn0OWq7cHOOYrHo+S6NZ5FcS4DIxu+j6w/9kg0TfvuM7yWonNsS66KPYWS7Q7xLIrzInBYCDEhhLAAvw28uTPLUnQRJdf9i5LtDvHUPk5N05pCiH/L/aCPEfhTTdOu79jKFF1ByXX/omS7c3S0O5IQonMv1ptc2o++IyVXJdd9ymPlqpp8KBQKxTbZt1Mu7XY7NpsNk8mEzWYD7k8/bLfbFItFqtUq7XabTlrcCoVif7AvFafBYOD555/nzJkzjI6OcubMGdrtNqlUilwux5tvvskvf/lLyuUyxWKx28tVKBR7jH2rOPv7+zl69CjT09N86lOfot1us7y8zOrqKh988AG3b9+m0Wh0e6mKp0AIIT8LITAYns7jpFeBtFqtnVye4hl5UK6tVqvnTob7SnEaDAacTid2u52pqSnOnDlDf38/BoMBg8FAIBDAaDTS19dHX18frVaLbDbbc0JRbMZgMOB2u7FYLHJTud1uQqEQfr+fF198Ea/Xu7Fc8GPJ5XLEYjHS6TTvvvsumUxGuW66iC5Xl8tFIBDA5/Nx5swZLBYLP/nJT7h9+3a3l7iJfaU4hRA4HA7cbjcjIyMcP34ci8Ui71xerxej0UggECAQCJDP5z/mLyp6AV1xOhwODAYDRqOR/v5+pqamGB8f55//839OJBLZluJcWlriypUrzM7Ocu3aNXK53LZ+X7GzCCEwGo04nU6Gh4cZGRnh13/913G5XMzNzSnFuZtomkaj0aBarZLNZkkmk3g8Hmw220PHOf24p+g9rFYrNpsNu91Of38/DoeDQ4cO4fF4Np0eIpEIfX19OJ3Obb+GvkHb7Tbnzp2jr6+PhYUFVldXabVatNvtXXhnisdhNBoxm82MjIzwmc98hr6+PnladDgc2Gw2ms0mzWaz20sF9qHirFQqtNttYrEYd+7cIRKJEAqFntoPpug8TqeTvr4+hoaGeOmllwiFQpw7d47+/n55pHM4HPj9foxGIxaLZdvWos/nw+12Mzg4SKVSIRaL8e1vf5tSqUS9XqdWq+3iO1Q8iNlsxm63c/r0aX7/938fh8NBuVwml8vh8/nweDw9FczdV4oT7juSm80mtVqNSqVCvV7v9pIU28TtdhOJRBgaGmJkZIRAIEAwGMTn88mTgs1m23SS0JXmVpWnEEKmqvX396NpGj6fD7vdjqZpSnF2GP2GaDKZcDgcOBwOWq0WFotFfvSSTPaV4tSP6pqmUSgUSKfTBAIB5bfaY0xPT/PlL3+ZgYEBzp49i8PhwG63YzKZpOI0GAw74m5xOp08//zz5HI5Ll26RDweZ2VlhVKppP5vuojRaMTtdssbmtfrpdFokMvlur00YJ8pTticYtJut5Wvag9it9sJBoOEQiGCwSB2u33TzzVNo91uSwtEV6APKjo9pWXj5wfRN6gQAqfTic1mw2w279I7U2wV3fo0m83ys9Fo7PayJPtKcQohsFgsWK1WQqEQw8PDBAIBFQjaY6yurnLz5k0ajQYnT57c9LNarUatViORSHDnzh2azeZjFafD4SAcDmO32xkaGsLhcHTsPSj2N/tKcQKYTCYsFgsulwu/34/T6VSKc49RLBaJx+P4/f5NyekbsyZSqRQzMzPSh20wGB46Xfh8Ptrttsz53IriVMfz3qJX9+6+Upy6ea87mD0eD3a7fdPFNxgM+Hw+BgYGKBQKJBIJGo0GlUpFbZoeIZVKcfXqVdLpNLVabZPCq1ar1Ot1EokEt2/ffqjqZ6MMHQ4HN27cIBgM4nQ6MZvNWCyWTUfxer3OysoKmUyGpaUlEokEhUJB/S/0CJqmYTQaMZlM0t3SC7L5WMUphPhT4NeApKZpJ9cfCwD/ExgHFoDf0jQts3vL3Bq64rRYLDidTrxeLw6H46G7lp4DqFePlMtlarXagSq962W5xuNxkskkNpuN999/f1Mqme63LpfL5PP5J/qwTSYTdrud4eFhzp49S19fHx6PZ5PirFarzM3NEY/HWVhYYHl5uWdyBZ+WXpbtVtE0Te7bjYqzV9jKSr4BfOGBx74GvKVp2mHgrfXvu45+lKvVamQyGeLxuCyl0zGZTITDYQ4dOsTo6Cj9/f0ysfqA8Q16VK56cK9er1MqlR75UavVZLerx30YDAY8Hg8ejweLxfLIzddut6lUKlSrVRqNxn5Jfv8GPSrb7aArT7vdLgtZeoWPtTg1TbuwPuh9I18C/t76198EfgL84U4u7GnQW8ZVKhVmZmYIBAKcOHGCsbExGZEzm8184hOf4LnnnsPv91OtVllcXCQWix2oph+9LFc9M6Jer5PJZDadGLaTr+lyuTh69CjDw8OEQiF5XN9Iq9WiUCiQyWSk8tzr9LJst4vBYCAUCjE+Pk4ul9s7R/XHENY0Lb7+9QoQftwTOz1uVLcWstks8XhcltVtxG63Y7fbZUMQq9Xas07oDtNTcn3a2vGNAcJgMIjf78dqtWI0Gh+Sc7PZJJfLScW5j9mSbLs1HvhRs8v1pHir1YrD4eipNLFnDg5pmqY9qcW+pmlfB74OnWvF3263uXHjBtFoFIPBwJe+9KVOvOy+ohflulUGBwc5dOgQw8PDvPbaa7Ibltlsfkhx5nI53nrrLW7evEk0Gu3SijvLk2TbLbnqubn1ep1KpYLZbMZmsyGEwOPx0N/fj9vt7tRyPpanVZwJIcSgpmlxIcQgkNzJRe0Ea2trrK2tkUqlNlmcj8v5UwB7QK6PQrdM9ER3r9fL4OAgw8PDDA8PyyT6B/2bemllNBrl3r17VCqVLr2DjtDTstUVp+6iaTab0uq0WCyycqxXeNqVvAl8Ffij9c9/vWMr2mE2bqqNylJ/TLGJPSNXHbvdztjYGB6Ph6NHjzIwMMDw8DCjo6N4PB5GR0dlp6WNVCoV2UGrWq32ZLPcHaanZauXSq+trXHv3j3K5TJjY2M9G7TdSjrSt7jvVA4JIaLAf+b+xf8LIcS/Au4Bv7Wbi3xWNirJfb45tsx+kCvcb/YxMTHBwMAAv/Irv8KxY8fw+XwEg0GMRuNjy/Sq1SrpdJpMJkOj0dgPkXTJXpRtq9Wi1WqRz+eJxWJomsbQ0BBWq7XbS3skW4mqf+UxP/rsDq9l19jocFbcp5flqieq2+12QqEQNpsNr9eLxWJ56Ller5dz584RDAYZHR3F5/PhcDgeCgTpGzMWi5FMJkmlUszPz7O0tEQ+n99XFmcvy3a/0DtOg13icUpTKdLexWaz4fP5GBwc5MUXXyQQCHD06FF8Pt9Dz3W5XExPT+NyuTAajbK65MEUJr3H5gcffMB7773H8vIyN27coFgskkwm90UakqJz7HvF+SBKWfYuegec/v5+RkdHGRgYYHx8HK/Xy8DAAB6P56HfcTgcuFwumRz9KPnqSe7lcplUKkU0GiWRSJDNZqV/U9G76ONSesnfeeAUp6I3MRgMBINBXC4Xn/vc5/iN3/gNGR23WCzYbLZH+iv1qOuTaDabLC0tkUqluHjxIj/+8Y9lsxBN0/Z8ieV+RgiBzWbD5XL1VL61UpyKnkHP3evr62NyclJOPNxKGoqeNfGojdVut2k0GrKEU69x308Bof3Mk26c3UIpTkVPsxULY+MR7lFHdavVyuTkJOFwmJMnT3L37l3W1taIRqNKefY4RqORkZER3G43ly5dUhanQrFTPM7S1DEajQSDQbxeL8PDwwwODqJpGrFYTCnOHkW/AeoTTd1uN16vt8ur+jsOhOLcmAC/cYN5PB4ikQjFYrGnqhIOIpqmkc/naTQaXLp0CavVSiAQYGpqCqPRSDqdlhNMH2VVbixwCIVCDAwM4PV6GRsb66kaZ8WTKRaLsvS1lzMd9r222Fg5BJuPcl6vl/HxcbLZbE/5Tw4imqaRy+XI5XK8++67zMzMMDg4yMsvv4wQgmvXrslk9Y+Lgp88eZJz584xNjbGwMCAVJwqBa33KRaLLCwsIITo6Qm1+15xPgl9Jk0oFMLtdlOtVqlWq+r41mXq9TrFYpHV1VXu3r2L0WgkmUySy+W2FNSJx+PMzs5iMplkzbNib9BsNmV/VP0GqRs++uz1Wq3WdaW67xXnxuYBD26gcDhMIBCgXq8zMjKCEIJEIrHfmz30PJVKhVqtRjab3XRs0xXmxynCTCbDrVu3WFtbk2lNir1Bo9FgbW2NYDAo5W0wGDAYDDgcDoLBIMVikUwm09X8232vOJvNpvRhWq3WTRFYfT6RPmajUCiwtramFGeX0TvA613gt4ueeqTyM/ce7XZbyl6/QepdrwwGAyaTqSfcavteca6srPD2228zMDDA2bNnH9nTz+/389JLL7G8vEwmkyGfz3dhpYqdQs8DHR8fV4GhPYZ+03wwCCiEkLOHlOLsAKVSiVgshsFgeGyUzmq1Mjg4SLPZ/NgqFMXOs1Odq3RfmNPpJBwO4/P5emKTKbaO7lrTLc6N/xMbLc5u53Nupa3cCPDfuN9qXwO+rmnaH++VqXl6+zCHw6GObhvotlx1JWez2WTkO5lMUiwWn2pgmtFoZHx8nGAwyIsvvsjrr7/OwMBATw346gTdluuzksvlmJmZQdM04vE4TqcTh8OB1WpldHSUT37ykywuLrK2ttbV/byVqvkm8O81TTsOvAT8GyHEcfbI1LxGoyGP36qZwya6LleDwYDNZiMSichmHk/bzMFoNBKJRDhx4gTnz5/nc5/7HOfOnTtwipMekOuzoKcjLSwskEqlyGaztFotTCYTg4ODnD17lqmpqa67YLbSjzMOxNe/LgghZoBh9sjUvHw+z+3bt2m322QyGTweD1arddOFdzqdTE1NYbVaCYVCrKysUKvVejoB91npllxdLhcOhwO/38/Q0BA+n49Tp05hNpspFouyY9HHWRP6sc1sNuP3+3G5XJw6dYpjx44xOjr60JFuY9Bh41Fwv7HX96tOo9EglUpJ2drtdmKxGJcvX2ZxcXFvpSOtjxw9C7zPNiYidpNEIkE6nWZ1dZUvf/nLuFwuQqHQJsUZCAR45ZVXiMVifPe732VlZYW1tbV9rTg30im5CiFkVc+JEyd44403CAaDnDx5Ek3TiEajLCwsoGnax06cNJlMOBwOPB4PJ06cIBwO8yu/8iucO3dOTjHV0TsgNRoNGXE/CLm6e3G/6lSrVebm5tA0DZ/Ph8vl4saNG/zVX/0VlUql65kvW1acQggX8JfAH2ialn+gUexjp+Z1a9yojp4wXavV5AV/0JrR+/3p/SD1ZrgHgU7KVQiBz+djfHycSCRCOByWFkW73SYQCDAwMECxWMThcDzxbzkcDnw+Hx6Ph8OHDxMKhQgGg5vGyLZaLXlyyGQysh9noVCgXC7vS4tTZ6/u1420Wi2azaa8yTUaDSqVCvV6veuy25LiFEKYuS+EP9c07a/WH97S1LxeGSPbaDRIJBJ4PB48Hs+mpGhdCI+rg96vdFquBoOBF154gS9/+csyZUhvJddoNDh79ixms1ne5J5EX18fU1NTuFwupqamZC6uPlIW7mdUzM/Pk81mef/994nH41y8eJGZmZlNG3K/sR/264PoaUrNZrMnYhVbiaoL4E+AGU3T/suGH/X01LwHabValEolCoXCQ2V4D07APAh0S64ej0f6Nt1utwwECSEIBAJEIhFpJT5JJgMDA1Jx6pMs9fQV3YdZKpVIp9Ok02nu3bvH8vIyyWSSUqm0k2+pp9gv+/VBNuZx9kIv1a1YnJ8E/hlwVQjx0fpj/4ken5r3IKVSicuXL5NMJolEIoyMjHR7Sd2mp+RqMpk4duwYkUhEKr4nYbfb8Xg8mM1meTSvVCryZBGLxZifn+d73/sea2trxGIxSqUS2Wy2A++mq/SUXJ8VvcO/zWZjaGiI5557jtXVVebm5roag9hKVP0d4HEOvz0zNa9Wq7G0tESj0aBYLHZ7OV2nm3J9lCVpMBgIh8OEw9uLWWxMntfHYaTTae7evcvMzAzvvPMO2Wx2340Afhz7Zb/C38lWjz/o/VQNBgOLi4u9rTj3C61Wi2w2i9lsZnV1lbW1NWw228cGIRQ7h6Zp3Lx5k+9///sMDw9z4sQJXC4XkUhky/Oz9QYgus9LbwpRrVa5e/euHMY2Pz9PMpmkXC6rDkl7EKvVyvj4OIcPH8bpdNJqtSiXy6TT6Z7IyT4wilPfYJqmkUwmSSQShEIhpTg7SLvd5sqVKyQSCaanp6lUKgwMDBAMBresOMvlMrlcTqYXFYtFbt++TSaT4d1335XKc2VlRfo7FXsPm83G4cOHOX78OE6nUzbrSaVSsr1gNzkwilPPDSwUCszMzGA2m/F4PHJWtxCC1dVVVlZWHpmypHh2NE2jXC6TyWSIxWLMzMywurpKIBAgGAzS19cnA0Z6b4F8Pi83Tb1eJx6Pk0ql5LG8UqkQjUYpFArE43Gy2SzlcrnrFoni2Wi32xQKBXK5HJqmYbPZqNVqPZMNcWAUZ7PZJJvNks/n+bM/+zPZYm5jeV+73SaXy1Gr1XpCOPsRvfx1ZWWF69evEwgEWFhYYHBwkDfeeIPjx49jsViw2+3k83muXbtGLpfj1q1bZDIZbt68ydzcHNVqVda165uplzaW4tmoVqvcu3dP+r7dbrf0VfeC6+XAKE74u2T4AxBZ7Vl0GWzstRmLxWg0GkSjUdxuN1arFavVytraGktLS+RyOaLRqIyOJxIJqTi7vYEUu4OeHWG1Wmk0Grhcri2PTukEopP/eL2UUNslLmma9oluL2KneVq56uMQfD4fFouFUCiEy+WS1Vv1el3m3ZZKJRqNBqVSiUqlIuvOewQl1x3GZrMxPDyM3W7HYrFgMplIJpOsrKzIoGAHeKxcD5TFqegtNE2jXq+TTN4vYtHHZCgUepZEr7L9/l0KhUJxwFGKU6FQKLaJUpwKhUKxTZTiVCgUim2iFKdCoVBsk05H1dNAaf3zXiPEs697bCcW0oMoue5PlFwfQ0fzOAGEEB/sxZy3vbruTrFXr89eXXen2KvXZ7fXrY7qCoVCsU2U4lQoFIpt0g3F+fUuvOZOsFfX3Sn26vXZq+vuFHv1+uzqujvu41QoFIq9jjqqKxQKxTbpmOIUQnxBCHFLCDErhPhap153uwghRoQQPxZC3BBCXBdC/Lv1xwNCiL8VQtxZ/+zv9lp7hb0gWyXX7aPk+oTX7cRRXQhhBG4DnwOiwEXgK5qm3dj1F98m6zOnBzVNuyyEcAOXgN8AfhdY0zTtj9b/ifyapv1h91baG+wV2Sq5bg8l1yfTKYvzRWBW07Q5TdPqwP8AvtSh194WmqbFNU27vP51AZgBhrm/3m+uP+2b3BeOYo/IVsl12yi5PoFOKc5hYGnD99H1x3oaIcQ4cBZ4HwhrmhZf/9EKsL05tvuXPSdbJdctoeT6BFRw6DEIIVzAXwJ/oGlafuPPtPv+DZWOsAdRct2fdFqunVKcy8DIhu8j64/1JEIIM/eF8Oeapv3V+sOJdX+K7ldJdmt9Pcaeka2S67ZQcn0CnVKcF4HDQogJIYQF+G3gzQ699rYQQgjgT4AZTdP+y4YfvQl8df3rrwJ/3em19Sh7QrZKrttGyfVJr9upBHghxBeB/wcwAn+qadr/ryMvvE2EEK8CbwNXAX3O7H/ivt/kL4BR4B7wW5qmrXVlkT3GXpCtkuv2UXJ9wuuqyiGFQqHYHio4pFAoFNtEKU6FQqHYJkpxKhQKxTZRilOhUCi2iVKcCoVCsU2U4lQoFIptohSnQqFQbBOlOBUKhWKb/P8Bb3Tt8VAj+FMAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 9 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "(X_train, y_train), (X_test, y_test) = mnist.load_data()\n",
    "\n",
    "print('X_train: ' + str(X_train.shape))\n",
    "print('Y_train: ' + str(y_train.shape))\n",
    "print('X_test:  '  + str(X_test.shape))\n",
    "print('Y_test:  '  + str(y_test.shape))\n",
    "\n",
    "# The training set contains 60k images and the testing set contains 10k images.\n",
    "# The training input vector is of the dimension [60000 X 28 X 28].\n",
    "# The training output vector is of the dimension [60000 X 1].\n",
    "# Each individual input vector is of the dimension [28 X 28].\n",
    "# Each individual output vector is of the dimension [1].\n",
    "\n",
    "# plot first few images\n",
    "for i in range(9):\n",
    "\t# define subplot\n",
    "\tplt.subplot(330 + 1 + i)\n",
    "\t# plot raw pixel data\n",
    "\tplt.imshow(X_train[i], cmap=plt.get_cmap('gray'))\n",
    "# show the figure\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "mEzjRsXE8Xur"
   },
   "outputs": [],
   "source": [
    "# Reshape data\n",
    "X_train = X_train.reshape((X_train.shape[0], 28, 28, 1))\n",
    "X_test = X_test.reshape((X_test.shape[0], 28, 28, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "p8Ify9q99QVb"
   },
   "outputs": [],
   "source": [
    "# one hot encoding of target values\n",
    "y_train_cat = to_categorical(y_train)\n",
    "y_test_cat = to_categorical(y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "UBARy2Gr9rDR"
   },
   "outputs": [],
   "source": [
    "def prep_pixels(train, test):\n",
    "\t# convert from integers to floats\n",
    "\ttrain_norm = train.astype('float32')\n",
    "\ttest_norm = test.astype('float32')\n",
    "\t# normalize to range 0-1\n",
    "\ttrain_norm = train_norm / 255.0\n",
    "\ttest_norm = test_norm / 255.0\n",
    "\t# return normalized images\n",
    "\treturn train_norm, test_norm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "Gg3Mmu6h9rLq"
   },
   "outputs": [],
   "source": [
    "def define_model():\n",
    "  model = Sequential()\n",
    "  model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))\n",
    "  model.add(Conv2D(32, (3, 3), activation='relu'))\n",
    "  model.add(MaxPooling2D((2, 2)))\n",
    "  model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "  model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "  model.add(MaxPooling2D((2, 2)))\n",
    "\n",
    "  model.add(Flatten())\n",
    "  \n",
    "  model.add(Dense(200, activation='relu'))\n",
    "  model.add(Dense(200, activation='relu'))\n",
    "  model.add(Dense(10, activation='softmax'))\n",
    "  \n",
    "  # compile model\n",
    "  opt = SGD(learning_rate=0.01, momentum=0.9)\n",
    "  model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "  return model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "GAfYdJheR2jB"
   },
   "outputs": [],
   "source": [
    "def plot_loss(history):\n",
    "  plt.figure(figsize=(10,6))\n",
    "  plt.plot(history.epoch,history.history['loss'], label='loss')\n",
    "  plt.plot(history.epoch,history.history['val_loss'],label='val_loss')\n",
    "  plt.title('Loss')\n",
    "  plt.legend(loc=1, prop={'size': 12})\n",
    "  \n",
    "def plot_accuracy(history):\n",
    "  plt.figure(figsize=(10,6))\n",
    "  plt.plot(history.epoch,history.history['accuracy'],label='accuracy')\n",
    "  plt.plot(history.epoch,history.history['val_accuracy'],label='val_accuracy')\n",
    "  plt.title('Accuracy')\n",
    "  plt.legend(loc=4, prop={'size': 12})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 74216,
     "status": "ok",
     "timestamp": 1654684929849,
     "user": {
      "displayName": "Stefanija Galevska",
      "userId": "16800269765637290243"
     },
     "user_tz": -120
    },
    "id": "8zKOTvPVMhc4",
    "outputId": "2c6fa5a6-41e9-494b-dcc8-ca82568acb9e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: total: 1h 3min 52s\n",
      "Wall time: 7min 33s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "X_train, X_test = prep_pixels(X_train, X_test)\n",
    "model = define_model()\n",
    "history = model.fit(X_train, y_train_cat, epochs=10, batch_size=32, validation_split = 0.20, verbose=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 798
    },
    "executionInfo": {
     "elapsed": 1934,
     "status": "ok",
     "timestamp": 1654684931779,
     "user": {
      "displayName": "Stefanija Galevska",
      "userId": "16800269765637290243"
     },
     "user_tz": -120
    },
    "id": "J7RpDxrMR8pF",
    "outputId": "1c6619bc-55f2-4b12-a21a-49b7d4621abb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "313/313 - 2s - loss: 0.0355 - accuracy: 0.9905 - 2s/epoch - 6ms/step\n",
      " accuracy: 99.05%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlkAAAF1CAYAAADbfv+XAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAzI0lEQVR4nO3de3yc1X3v++9vRpfRzfLoYnwRHhlMwAZTbA8EkpCEQrk1hTZN8gJv2iRtD9knZZ+QpDm4p5xAHdKEZENJCulOTkt2dwphU07odogJCRByQhOIZSAY2zgYYxnZMtbNtmxJo9s6fzyPpNF4ZI2kGc3t834xr3kua2aWJFv+stZ6np855wQAAID0CmS7AwAAAIWIkAUAAJABhCwAAIAMIGQBAABkACELAAAgAwhZAAAAGUDIAgAAyABCFoC8ZGb7zOyKbPcDAKZCyAIAAMgAQhaAgmFm5WZ2n5kd9B/3mVm5f67BzJ4wsyNm1m1mvzCzgH/uNjM7YGa9ZrbbzC7P7lcCoBCUZLsDAJBGfyPpYkkXSHKS/pek2yX935I+L6lNUqPf9mJJzszOlnSLpAudcwfNrFlScH67DaAQMZIFoJD8J0mbnHOHnXMdkv5W0p/454YkLZEUcc4NOed+4bzirSOSyiWtNrNS59w+59ybWek9gIJCyAJQSJZKao3bb/WPSdLXJe2R9BMz22tmGyXJObdH0q2S7pR02MweMbOlAoA5ImQBKCQHJUXi9pf7x+Sc63XOfd45d4ak6yR9bmztlXPuYefc+/zXOkl3z2+3ARQiQhaAfFZqZqGxh6TvS7rdzBrNrEHSFyX9qySZ2YfMbKWZmaSj8qYJR83sbDP7XX+B/ICkfkmj2flyABQSQhaAfLZFXigae4QktUh6VdJ2SS9Justve5akpyUdl/QrSd9yzv1M3nqsr0rqlHRI0iJJfz1/XwKAQmXeuk8AAACkEyNZAAAAGUDIAgAAyABCFgAAQAYQsgAAADKAkAUAAJABOVe7sKGhwTU3N2e7GwAAANPatm1bp3OuMdm5nAtZzc3NamlpyXY3AAAApmVmrVOdY7oQAAAgAwhZAAAAGUDIAgAAyABCFgAAQAbk3MJ3AACQuqGhIbW1tWlgYCDbXSlYoVBITU1NKi0tndHrUgpZZna1pG9ICkr6J+fcVxPOf07SX0galtQh6c+cc63+uRFJ2/2m+51z182ohwAAYEptbW2qqalRc3OzzCzb3Sk4zjl1dXWpra1NK1asmNFrp50uNLOgpAckXSNptaQbzWx1QrOXJUWdc+dLekzS1+LO9TvnLvAfBCwAANJoYGBA9fX1BKwMMTPV19fPaqQwlTVZF0na45zb65wblPSIpOvjGzjnfuac6/N3X5DUNOOeAACAWSFgZdZsv7+phKxlkt6O22/zj03lzyU9GbcfMrMWM3vBzP5w5l0EAADIP2m9utDMbpIUlfT1uMMR51xU0gZJ95nZmUled7MfxFo6OjrS2SUAAJAlzc3Nevrpp7PdjaxJJWQdkHR63H6Tf2wSM7tC0t9Ius45Fxs77pw74D/vlfScpLWJr3XOfcc5F3XORRsbk5b/AQAAyCuphKytks4ysxVmVibpBkmb4xuY2VpJ35YXsA7HHQ+bWbm/3SDpvZJ2pqvzszEy6rR1X7d++05vNrsBAAAK3LQhyzk3LOkWSU9J2iXpUefcDjPbZGZjVwt+XVK1pH8zs1fMbCyErZLUYma/kfQzSV91zmU1ZEnSn313q/7ll/uy3Q0AAIpCLBbTrbfeqqVLl2rp0qW69dZbFYt5k16dnZ360Ic+pIULF6qurk6XXnqpRkdHJUl33323li1bppqaGp199tl65plnsvllzFhK98lyzm2RtCXh2Bfjtq+Y4nW/lLRmLh1Mt2DAdMHyhdrW2pPtrgAAkHZ/+8Md2nnwWEY/Y/XSBbrjD85Nuf2Xv/xlvfDCC3rllVdkZrr++ut111136Utf+pLuueceNTU1aWxN9gsvvCAz0+7du3X//fdr69atWrp0qfbt26eRkZFMfUkZUZRldaKROu1+p1dH+4ey3RUAAAreQw89pC9+8YtatGiRGhsbdccdd+h73/ueJKm0tFTt7e1qbW1VaWmpLr30UpmZgsGgYrGYdu7cqaGhITU3N+vMM0+6di6nFWVZnWhzWM5JL+/v0QfPXpTt7gAAkDYzGWGaLwcPHlQkEhnfj0QiOnjwoCTpC1/4gu68805deeWVkqSbb75ZGzdu1MqVK3Xffffpzjvv1I4dO3TVVVfp3nvv1dKlS7PyNcxGUY5kXXD6QgUDxpQhAADzYOnSpWptbR3f379//3hYqqmp0T333KO9e/dq8+bNuvfee8fXXm3YsEHPP/+8WltbZWa67bbbstL/2SrKkFVVXqLVSxaoZR8hCwCATLvxxht11113qaOjQ52dndq0aZNuuukmSdITTzyhPXv2yDmn2tpaBYNBBQIB7d69W88++6xisZhCoZAqKioUCORXbMmv3qbR+khYr7x9REMjo9nuCgAABe32229XNBrV+eefrzVr1mjdunW6/fbbJUlvvPGGrrjiClVXV+uSSy7Rpz/9aV122WWKxWLauHGjGhoatHjxYh0+fFhf+cpXsvyVzIw557Ldh0mi0ahraWnJ+Oc88epB3fLwy9p8y3t1ftPCjH8eAACZsGvXLq1atSrb3Sh4U32fzWybX9nmJEU7khWN1EmStjJlCAAAMqBoQ9bi2pCWLazQttbubHcFAAAUoKINWZJ3K4eWfT3KtSlTAACQ/4o7ZEXCOtwbU1tPf7a7AgAACkxRh6z1/rqsFqYMAQBAmhV1yDp7cY1qyku4XxYAAEi7og5ZFIsGAACZUtQhS6JYNAAAyAxCVlyxaAAAgHQp+pBFsWgAAHLLc889p6ampmnbNTc36+mnn56HHs1O0YesqvISrVpSw+J3AACQVkUfsiRvXRbFogEAQDqVZLsDuWB9JKz//st92tV+jGLRAID89uRG6dD2zH7G4jXSNV+dttndd9+trVu36rHHHhs/9pnPfEbOOa1du1Zf+9rX1NbWpsbGRt1222361Kc+NesuxWIx3XbbbXr00UclSR/72Md09913q7y8XJ2dnfrEJz6h559/XoFAQOeee65+/vOfKxAI6O6779Y3v/lNHTt2TEuXLtW3vvUtXX755bPuRzxGsuQtfpfElCEAAGl0ww03aMuWLert7ZUkjYyM6NFHH9WGDRu0aNEiPfHEEzp27Ji++93v6rOf/axeeumlWX/Wl7/8Zb3wwgt65ZVX9Jvf/Ea//vWvddddd0mS7rnnHjU1Namjo0PvvPOO/u7v/k5mpt27d+v+++/X1q1b1dvbq6eeekrNzc3p+NIlMZIlSVpSW+EXi+7Rn71vRba7AwDA7KUwwjRfIpGI1q1bp8cff1x/+qd/qmeffVaVlZW6+OKLJ7X7wAc+oCuvvFK/+MUvtG7dull91kMPPaR/+Id/0KJFiyRJd9xxhz71qU/pS1/6kkpLS9Xe3q7W1latXLlSl156qSQpGAwqFotp586damxsTGvAkhjJGhdtDqultZti0QAApNGGDRv0/e9/X5L08MMPa8OGDZKkJ598UhdffLHq6uq0cOFCbdmyRZ2dnbP+nIMHDyoSiYzvRyIRHTx4UJL0hS98QStXrtSVV16pM844Q1/9qhdEV65cqfvuu0933nmnFi1apBtuuGH8NelAyPJFI2G9c4xi0QAApNNHP/pRPffcc2pra9Pjjz+uDRs2KBaL6Y//+I/1V3/1V3rnnXd05MgRXXvttXMa6Fi6dKlaW1vH9/fv36+lS5dKkmpqanTPPfdo79692rx5s+69914988wzkrwQ+Pzzz6u1tVVmpttuu21uX3AcQpZvrFg098sCACB9Ghsb9cEPflCf/OQntWLFCq1atUqDg4OKxWJqbGxUSUmJnnzySf3kJz+Z0+fceOONuuuuu9TR0aHOzk5t2rRJN910kyTpiSee0J49e+ScU21trYLBoAKBgHbv3q1nn31WsVhMoVBIFRUVCgTSF40IWb6xYtFb93VnuysAABSUDRs26Omnnx6fKqypqdE3v/lNfexjH1M4HNbDDz+s6667bk6fcfvttysajer888/XmjVrtG7dOt1+++2SpDfeeENXXHGFqqurdckll+jTn/60LrvsMsViMW3cuFENDQ1avHixDh8+rK985Stz/nrHWK6tQYpGo66lpSUrn/0n//yiOnpj+vGt78/K5wMAMFO7du3SqlWrst2NgjfV99nMtjnnoslew0hWHIpFAwCAdCFkxaFYNAAAuWP//v2qrq5O+ti/f3+2uzct7pMVJ75Y9AfPXpTt7gAAUNSWL1+u48ePZ7sbs8ZIVhyKRQMA8lGura8uNLP9/hKyElAsGgCQT0KhkLq6ughaGeKcU1dXl0Kh0Ixfy3RhAopFAwDySVNTk9ra2tTR0ZHtrhSsUCikpqamGb+OkJUgvlg0IQsAkOtKS0u1YgV1d3MR04UJ4otFAwAAzBYhK4n1EYpFAwCAuSFkJRFtplg0AACYG0JWEusj3rospgwBAMBsEbKSOGfxAlWXl6illWLRAABgdghZSQQDprXLF3JTUgAAMGuErCmMFYs+NkCxaAAAMHOErClMFIs+ku2uAACAPETImsJ4seh9rMsCAAAzR8iawlix6K2sywIAALNAyDoFikUDAIDZImSdwvpIWP1DI9rVfizbXQEAAHmGkHUK8cWiAQAAZoKQdQoUiwYAALNFyJoGxaIBAMBsELKmQbFoAAAwG4SsaVAsGgAAzAYhaxoUiwYAALNByJoGxaIBAMBsELJSsD4Splg0AACYkZRClpldbWa7zWyPmW1Mcv5zZrbTzF41s2fMLBJ37uNm9ob/+Hg6Oz9fopE6ikUDAIAZmTZkmVlQ0gOSrpG0WtKNZrY6odnLkqLOufMlPSbpa/5r6yTdIendki6SdIeZhdPX/flxwfKFCpgoFg0AAFKWykjWRZL2OOf2OucGJT0i6fr4Bs65nznn+vzdFyQ1+dtXSfqpc67bOdcj6aeSrk5P1+dPdXmJVi1ZoBauMAQAAClKJWQtk/R23H6bf2wqfy7pyZm81sxuNrMWM2vp6OhIoUvzLxoJ65W3j2iYYtEAACAFaV34bmY3SYpK+vpMXuec+45zLuqcizY2NqazS2mzvrlOfYMj2tXem+2uAACAPJBKyDog6fS4/Sb/2CRmdoWkv5F0nXMuNpPX5oMLx4pFc78sAACQglRC1lZJZ5nZCjMrk3SDpM3xDcxsraRvywtYh+NOPSXpSjML+wver/SP5Z2xYtGsywIAAKkoma6Bc27YzG6RF46Ckh50zu0ws02SWpxzm+VND1ZL+jczk6T9zrnrnHPdZvYleUFNkjY55/J2KGh9JKwX3+qSc07+1wkAAJDUtCFLkpxzWyRtSTj2xbjtK07x2gclPTjbDuaSaHNYm39zUG09/Tq9rjLb3QEAADmMO77PAMWiAQBAqghZM0CxaAAAkCpC1gxQLBoAAKSKkDVDFIsGAACpIGTNEMWiAQBAKghZM0SxaAAAkApC1gxRLBoAAKSCkDULFIsGAADTIWTNAsWiAQDAdAhZsxCNUCwaAACcGiFrFpYurNDS2hDrsgAAwJQIWbO0vrlO2/b1yDmX7a4AAIAcRMiapWgkrEPHBnTgSH+2uwIAAHIQIWuWKBYNAABOhZA1S+csrlFVWZA6hgAAIClC1iyVBANaFwmz+B0AACRFyJqD9ZGwXj90jGLRAADgJISsOaBYNAAAmAohaw4oFg0AAKZCyJoDikUDAICpELLmiGLRAAAgGULWHFEsGgAAJEPImiOKRQMAgGQIWXNEsWgAAJAMISsNKBYNAAASEbLSgGLRAAAgESErDSgWDQAAEhGy0oBi0QAAIBEhKw1KggGtXU6xaAAAMIGQlSbrI2HtPnRMvRSLBgAAImSlTbQ5rFGKRQMAAB8hK03WLg8rYGLKEAAASCJkpU11eYnOWbxA27jzOwAAECErraLNYb28n2LRAACAkJVWUYpFAwAAHyErjSgWDQAAxhCy0ohi0QAAYAwhK80oFg0AACRCVtpRLBoAAEiErLSjWDQAAJAIWWlHsWgAACARstKOYtEAAEAiZGUExaIBAAAhKwMoFg0AAAhZGUCxaAAAQMjKAIpFAwAAQlaGUCwaAIDiRsjKkPWRsPoGR/T6IYpFAwBQjAhZGRJtrpMktexjyhAAgGJEyMqQZQsrtIRi0QAAFC1CVgatj4QprwMAQJEiZGVQNBJW+1GKRQMAUIxSCllmdrWZ7TazPWa2Mcn595vZS2Y2bGYfSTg3Ymav+I/N6ep4PmBdFgAAxWvakGVmQUkPSLpG0mpJN5rZ6oRm+yV9QtLDSd6i3zl3gf+4bo79zSsUiwYAoHiVpNDmIkl7nHN7JcnMHpF0vaSdYw2cc/v8c9wUKg7FogEAKF6pTBcuk/R23H6bfyxVITNrMbMXzOwPZ9K5QkCxaAAAitN8LHyPOOeikjZIus/MzkxsYGY3+0GspaOjYx66NH8oFg0AQHFKJWQdkHR63H6TfywlzrkD/vNeSc9JWpukzXecc1HnXLSxsTHVt84LFIsGAKA4pRKytko6y8xWmFmZpBskpXSVoJmFzazc326Q9F7FreUqBhSLBgCgOE0bspxzw5JukfSUpF2SHnXO7TCzTWZ2nSSZ2YVm1ibpo5K+bWY7/JevktRiZr+R9DNJX3XOFVXIkigWDQBAMUrl6kI557ZI2pJw7Itx21vlTSMmvu6XktbMsY95b30krP/xq1a9fqhX5y2rzXZ3AADAPOCO7/OAm5ICAFB8CFnzgGLRAAAUH0LWPKFYNAAAxYWQNU8oFg0AQHEhZM0T1mUBAFBcCFnz5JzFNaosCzJlCABAkSBkzROvWPRCtewjZAEAUAwIWfNofaROrx86puOx4Wx3BQAAZBghax5FI2PFohnNAgCg0BGy5tHa5QsVMGkrU4YAABQ8QtY8qgmVUiwaAIAiQciaZxSLBgCgOBCy5tn6SFh9gyN6/VBvtrsCAAAyiJA1z7gpKQAAxYGQNc8oFg0AQHEgZGUBxaIBACh8hKwsoFg0AACFj5CVBazLAgCg8BGysoBi0QAAFD5CVhZQLBoAgMJHyMoSikUDAFDYCFlZQrFoAAAKGyErS8aKRTNlCABAYSJkZUlNqFRnL17A4ncAAAoUISuLopGwXt7fQ7FoAAAKECEri6LNYZ2gWDQAAAWJkJVF6yNhSWLKEACAAkTIyqJlCyu0eEFIW7nzOwAABYeQlUVmpvXNFIsGAKAQEbKy7EKKRQMAUJAIWVlGsWgAAAoTISvLKBYNAEBhImRlGcWiAQAoTISsHECxaAAACg8hKwdQLBoAgMJDyMoBFIsGAKDwELJyAMWiAQAoPISsHEGxaAAACgshK0dQLBoAgMJCyMoRFIsGAKCwELJyxFix6BZCFgAABYGQlSPGi0VTXgcAgIJAyMoh0UhYB48O6CDFogEAyHuErBwSjfjFopkyBAAg7xGycsiqJX6xaKYMAQDIe4SsHFISDOiC0xcykgUAQAEgZOWYaCSsXe0UiwYAIN8RsnJMtLmOYtEAABQAQlaOoVg0AACFgZCVYygWDQBAYSBk5SCKRQMAkP8IWTmIYtEAAOQ/QlYOolg0AAD5L6WQZWZXm9luM9tjZhuTnH+/mb1kZsNm9pGEcx83szf8x8fT1fFCRrFoAADy37Qhy8yCkh6QdI2k1ZJuNLPVCc32S/qEpIcTXlsn6Q5J75Z0kaQ7zCw8924XNopFAwCQ/1IZybpI0h7n3F7n3KCkRyRdH9/AObfPOfeqpMSV2ldJ+qlzrts51yPpp5KuTkO/Cx7FogEAyG+phKxlkt6O22/zj6ViLq8tahSLBgAgv+XEwnczu9nMWsyspaOjI9vdyQkUiwYAIL+lErIOSDo9br/JP5aKlF7rnPuOcy7qnIs2Njam+NaFjWLRAADkt1RC1lZJZ5nZCjMrk3SDpM0pvv9Tkq40s7C/4P1K/xhSQLFoAADy17Qhyzk3LOkWeeFol6RHnXM7zGyTmV0nSWZ2oZm1SfqopG+b2Q7/td2SviQvqG2VtMk/hhSs94tFv7L/SLa7AgAAZqgklUbOuS2StiQc+2Lc9lZ5U4HJXvugpAfn0MeitXb5QplJLa3det9ZDdnuDgAAmIGcWPiO5BaESnX2aTXc+R0AgDxEyMpx0eawXmqlWDQAAPmGkJXjopE6ikUDAJCHCFk5jmLRAADkJ0JWjmsKUywaAIB8RMjKcRSLBgAgPxGy8gDFogEAyD+ErDxAsWgAAPIPISsPUCwaAID8Q8jKAxSLBgAg/xCy8gTFogEAyC+ErDxBsWgAAPILIStPxBeLBgAAuY+QlScoFg0AQH4hZOWRaHNYL+8/opFRl+2uAACAaRCy8kg0UqfjsWG9fuhYtrsCAACmQcjKIxSLBgAgfxCy8khTuEKnLShXyz5CFgAAuY6QlUfMTNFIHSNZAADkAUJWnlkfCevAkX6KRQMAkOMIWXkm2uyty6LEDgAAuY2QlWdWLVmgilKKRQMAkOsIWXmmlGLRAADkBUJWHrqwmWLRAADkOkJWHqJYNAAAuY+QlYcoFg0AQO4jZOUhikUDAJD7CFl5imLRAADkNkJWnqJYNAAAuY2QlacoFg0AQG4jZOUpikUDAJDbCFl5imLRAADkNkJWHhsrFt1+lGLRAADkGkJWHhsvFs2UIQAAOYeQlcfGi0UzZQgAQM4hZOWxiWLR3PkdAIBcQ8jKc9HmsHa19+oExaIBAMgphKw8tz4S1sio0ytvH8l2VwAAQBxCVp5bFwnLTNq6jylDAAByCSErz1EsGgCA3ETIKgDrIxSLBgAg1xRnyHKFFUaizWGKRQMAkGOKM2Q9eJX0+P8uvfFTaXgw272Zs2ikThLFogEAyCXFF7KGBqS6M6XXfyQ99BHpv54l/a9bpD3PSCND2e7drFAsGgCA3FOS7Q7Mu9KQ9Ef/KA3HpDeflXY8Lu34d+nl70kVddLq66Rz/0iKvE8K5se3h2LRAADknvxIEZlQUi6dfY33GBqQ9jztBa5X/03a9t+lqkZp1Vjgeo8UCGa7x6e0PhLWj7a3q/1ov5bUVmS7OwAAFL3iDVnxSkPSqg95j8E+ac9PvcD1m+9LLf8sVZ8mrb5eOvfD0unvlgK5N8saXyz6D36HkAUAQLYRshKVVXqBavX10uAJ6bdPSTt+IL30P6Rff0eqWSKt/kPpvA9Ly6I5E7jii0X/we8szXZ3AAAoeoSsUymr8sLUeR+WYr1e4HrtB97o1ov/KC1oks79Q2+Ea9k6ySxrXaVYNAAAuYWQlaryGmnNR7zHwFFp95PelOKL35Z+db+0cLm3fuvcP5KWXJCVwBVtDutbz72pE7FhVZXzowUAIJv4l3g2QrXS79zgPfqPeLeD2PG49KsHpP/4hhRe4YWt8z4snXbevAWu+GLR713ZMC+fCQAAkiNkzVXFQmntf/Iefd3S6094ges/viE9f69Uv9If4fqwtGhVRgPXWLHoln09hCwAALKMkJVOlXXSuj/1Hic6pV0/9BbN/+Ie6f/7utRwtje6de4fSY1np/3jx4pFsy4LAIDsS+nSODO72sx2m9keM9uY5Hy5mf1P//yLZtbsH282s34ze8V//Lc09z93VTVI0U9KH/+h9Pnd0u/f491767mvSg9cJH3rPdLPvy517knrx1IsGgCA3DBtyDKzoKQHJF0jabWkG81sdUKzP5fU45xbKenvJd0dd+5N59wF/uM/p6nf+aV6kXThX0if/JH0+dela74mhRZIP7tLun+99N/e5412de+d80eNFYvefag3DR0HAACzlcpI1kWS9jjn9jrnBiU9Iun6hDbXS/oXf/sxSZebZfF+BrmsZrH07k9Jf/Zj6bM7pau+IpVUSM9skr65Vvr2B6Tn75N6Wmf19hPFopkyBAAgm1IJWcskvR233+YfS9rGOTcs6aikev/cCjN72cx+bmaXJvsAM7vZzFrMrKWjo2NGX0Beq10mXfJp6S9+Kt36mnTlXV75nqfvkL5xvvT//K70y/ulo20pv2VTuEKLasq1lWLRAABkVaZvV94uablzbq2kz0l62MwWJDZyzn3HORd1zkUbGxsz3KUctfB06T3/RfrfnpU+8xvpir+VRkekn/yN9PfnSv/0e9IL/ygdO3jKtzEzXdhcp2d2vaNNP9ypba3dGmV9FgAA8y6VqwsPSDo9br/JP5asTZuZlUiqldTlnHOSYpLknNtmZm9Kepeklrl2vKCFm6X33eo9ut6Udv679Nrj0o83Sj/+a2n5Jd4Viquv86YfE3zuyncpNjyif32hVQ/+x1tavCCkq89brGvXLFE0ElYgwEwuABSV4UGp5y2p8w2p6w3voqvuvZKcVFrplZQrrfKfK72KJ/HPpRVxx+Lb+s8loaxWPclV5uWgUzTwQtNvJV0uL0xtlbTBObcjrs1fSlrjnPvPZnaDpA875z5mZo2Sup1zI2Z2hqRf+O2mXDAUjUZdSwsZLKnON6Qd/+7dFuLwTkkmNb/PK+2z6nqpevIo4LGBIT2767C2bG/Xc7/t0ODwqBbVlI8Hrgub6xQkcAFAYXBOOn7YD1FvSF17JkJVT6vkRibaVp8m1Z3pLVEZ6pMG+6ShE/6z/5gJC/hhLC6ElVacIrwlCWrj5ytODnqBYHq/V2lkZtucc9Gk56YLWf4bXCvpPklBSQ86575sZpsktTjnNptZSNL3JK2V1C3pBufcXjP7Y0mbJA1JGpV0h3Puh6f6LEJWig6/7t30dMcPpM7fen/Amy/1RrhWXSdV1U9qfjw2rGdfP6wtr7brZ7sPKzY8qobqcl193mm69rwlumhFnUqCuVHsGgBwCkP93izH2IhUfKiKHZtoVxLybohdv1JqOEuqP0tq8PdDtaf+jNFRaXjAD2AnkgSxuEA26XyyY/Ft/f3R4Zl9zcHyaQJbZfIRuerTvFmfDJpzyJpPhKwZcs4b1drxuFe8uvtNyYLS0rXeH67KOqmy3rtvV2W9VFmv/pJa/eqQ9KM9g9ryxgn1D42qvqpMV567WNeuWaxLzqgncAFANjknHTtw8ohU5x7p6NuS4v7tXtDkh6ez/DDlh6oFTVIgR3+XDw/Gha/+FMJb4vH+qdsM9098TuM50l++mNEvhZBVLJyTDm33RrfaWqT+Hu/O831d0uhQ8pcEShUrW6iu0Rq9HatQ52i1+oK1qmtcosjyiFYsP10l1RMBTZX13lAuAGDuYr1+iIofkXrDG6mKn7Irq04yInWWVH+mN6KDCaOjEyNqo8PSgqUZ/bhThSzK6hQSM2nJ+d4jnnPeX+S+Lq++Yl+X1OeFL+vrUqivS8v6urXkRKf6jhyW+nap8nCvAh1O2pbkc0qr/MBVNzl8VdZ705SJxyrqpCB/1AAUqdER6cj+hBEpf4Sqtz2uoUkLl0sN7/KWf8SHqprFLCxPVSAglVd7jyzjX75iYObdYT60QKpbMWWzgKSxP5IDsUG9uPNN/Wr7b7XzzX0KDfVoSVmfLlo0qvMWDqupvF/BAT+wdb/phbf4tQCJQgtPDl+VdZOmMScdL6/N3WFuAEimv+fkEamxq/hGYhPtQrVecDrjg5ODVN0ZUmkoa91H+jFdiGnFhkf0H3s6tWX7If1kxyEdGxhWTXmJLl+1SNeuWaL3v6tRodKgNByLGymb7tHtTWXG/+KJZ8GTg9f4aFmDNzpWVundLb+0wvvFVOpfRlxaObEfLJ3fbxaAwjYyJPXsO3lEqvMNb4ZgTKDEux1P/NTeWJiqamBUqoCwJgtpMzg8ql++2akntx/SUzsP6UjfkKrKgrp81Wm6ds1iffDsRV7gSoVz3qLF+OA1KYh1Jj/uRlPvsAX9EFYxTSAbOz9FWEvadux94wIdvzhRyEZHpdhR/+9kt3f1WaKkfweSHDupXSptkrRL6+cl7A8NeAEq/iq+nn2Tr4yrbJi82HwsTIWb+Z+8IkHIQkYMjYzqhb1d2rK9XU/teEfdJwZVWRbUZecs0rXnLdFl5zSqsizNM9Kjo9LAEW9YfmjsqhT/Mdw/xb5/GfLwwDTn+rz9+CtTZmLsPjGpBLJk5xL3yxdMjOKFFjJ9ivQaGfb+Lo39j0x/d5LtHu+5v3vi+Ez+J6dQBMu8e0oljkg1rJQqwtnuHbKMkIWMGx4Z1YtvdfuB65A6jw8qVBrQZWd7U4q/e84iVZXnyRJA5yYHsikDXJKAlnLYm2Ggs4D3yzz+YoJJFx4kXIRQESaYFZPhwSlCUlw4GhsRHtseODL1+wXLEv6c1Z38566iLsmVxkn+PUn6b4ybeZuk7ZK1SfJWs/28QIm3Tmrh8py+GSayi5CFeTUy6vRrP3D9eMchdfTGVF4S0Afe1ajfP98LXDUhhtElnRzoxgNYvzRwbGKKtL87YVo1bn+K23N4wawuIYTF/2OZGNC44CAnDPZNDkN9Xd7IbWJIit8ePD71+5VWTQSlxJA0vh2e+DNQUefdEoCpbyAlhCxkzcio07bWHm3Z3q4nX2vXO8diKisJ6P1nNeraNYt1xerTtIDANXtjt+eYNHKRsI6tPyGU9XWfIpgFJ4+YVSaOmNWfHNxCtfyDnEz8rVP6uxOm3pJMy41tJ1vnNKa8VqoMJx9VmhSk4gITV6sBGUXIQk4YHXV6+e0e/ejVQ3rytXa1Hx1QadB06VmNunbNEv3eqtNUW0ngyrjE+6ZNGiVLCGrx56Yqg2HB5KMjU01jThXMnPOu3Bod8p+Hvcf4seG4cyNx23Hnxtsnvm548vuOnRsdOcVnDiecG5niPYYT+hZ3bsr1S+Z/LxJHFsMnh6Tx7TALqYEcRMhCzhkddXql7Yie3N6uLdsP6cCRfpUGTe9d2aBrz1ui31t9msJVZdnuJsY4590HLXGq8qSA1jP53FTBLFDiTUmNh5zhycVrM80CUqDU60ewxNsOlvrHghPbk86V+O0Tzk35HmVemEwWmEK1rPEBCgQhCznNOadX245qy/Z2bXmtXW9396skYLrkzHpdu2aJrjp3seoIXPlnPJglWUfW1+WtI5ourAT9YBMfdJKGocQQVJJwLpgQglh3BiA9CFnIG845vXbgmLa81q4t29vV2tWnYMB08Rl144Grobo8290EAEASIQt5yjmnne3H9OT2Q9qyvV17O08oYNK7V9TrmjWL9Z4z63VmY7WMRdcAgCwhZCHvOee0+51ebfED157D3iXrCytLtX55WOubw4pG6nR+U23qd5wHAGCOCFkoOG91ntDWt7rV0tqtltYe7e04IUkqDZrOW1arC5vrtD4S1vpImOlFAEDGELJQ8LpPDGpba49aWru1bV+PXm07qsER7/L5FQ1VWh8JKxoJK9ocZooRAJA2hCwUndjwiF47cFQt+3q0dV+PXtrfo+4Tg5KYYgQApA8hC0XPOae9nSe0bV8PU4wAgLQhZAFJMMUIAJgrQhaQgoEhf4qxtUctTDECAFJwqpBVMt+dAXJVqDSoaHOdos110geSTzE+8/phSUwxAgCmx0gWMANMMQIA4jFdCGRI4hTjttZu9fQNSWKKEQCKAdOFQIYwxQgAmAojWUCGMcUIAIWL6UIgh6Q6xbh6yQJF6qvUFK5QaTCQ5V4DAJJhuhDIITOZYpSkYMC0dGFIkboqLa+vVKSuUpH6Si2vq1KkvlJV5fw1BoBcxG9nIMvMTGc2VuvMxmp97MLTJXlTjHsOH1dr1wnt7+5Ta1efWrv79OT29vFRrzEN1eWK+OFref1EAGuur1RdVRnTjwCQJYQsIAfVVZXpohV1umhF3UnnjvYPaX9Xn1q7T6i1q298+4W9XXr8lQOKXwFQXV6i5WMjX/WVivijX5H6Si2prVAwQAADgEwhZAF5praiVGuaarWmqfakcwNDI2rr8Ue+uvr8UbAT2n2oV0/vekdDIxMJrDRoOj1cOT4Fuby+SpG6SjU3VKopXMntJgBgjghZQAEJlQa1clGNVi6qOencyKhT+9F+f+RrLISd0L7OPrXs69Hx2PB4WzNp8YLQ+ChYpL5Ky+sq1VzvrQurrSidzy8LAPISIQsoEsGAqSnsjVK9J+Gcc07dJwbV2u1PP3Z5I2Ct3X169vUOdR5vm9R+YWXppNGv5fVeAIvUV2pRTTnrwABAhCwA8hbf11eXq766XOuWh086fyI2PL4Af3/3Ce3z14K98naPfvTqQY3GrQMLlQa0vG7i6sexkbBIXaWWcTsKAEWEkAVgWlXlJVq1ZIFWLVlw0rmhkVEd6On3R8G8ADYWxp7f06GBodHxtvG3ozi9rkKNNSEtqilXY025FtWUa9GCkBqqy1RewnowAPmPkAVgTkqDATU3VKm5oUpS46Rzzjkd7o2ptatP+7pOjK8H2991Qj/ZcUxdJwaTvufCylI1Vpdr0YJy/zk0se8HssaakBaESpiaBJCzCFkAMsbMdNqCkE5bEEp6O4qhkVF1HR9UR29Mh3sH/OfY+P7h3phaWnt0uDemweHRk15fXhJQY0188CrXoprQxMiYv91QXaYSpikBzDNCFoCsKQ0GtLg2pMW1IUkn35JijHNOxwaG1eEHr47xIDYRyPZ2nNCLb3XrSMLNWiXvasm6yrK4QBYXxOJHy2rKVc0d9AGkCb9NAOQ8M1NtRalqK0qT3p4iXmx4RJ3HB3X4WOLI2Fg4G9Cbh4+r43hs0n3DxlSWBZOOjCUeq6sq42auAE6JkAWgoJSXBLVsYYWWLaw4ZbvRUacj/UNTTFV6Yez1Q736xRud6h0YPun1wYCpvqrspPA1NjLWWFOuBRWlqi4vUXWoRFVlJYQyoMgQsgAUpUDAVFdVprqqMp29+NSjY/2DI94o2PEBHT4WU8fxmPc8FtCOx7Tj4DF1Ho9Nup1FosqyoKrKS1QTF7yqQyVeECufvF1Vnvx4dahElaVBBQhsQM4jZAHANCrKglru1388lZFR76auXiCLqXdgSMcHhnU85j8GhnVicFi9A8M64R97u7tPJwaHx9slm8JMZCYvoJWXqKo8qOpQqarLg34Q87dDJSkFuorSIFdoAhlCyAKANAkGbHz91mzFhke8MBYbUW9sSCdiIzoeG9Lx2IgfxCa2x4La2KOzt2/S/siphtV8AdPkMBY/gpZkdK0mNLFdWRZUqDSo8pLA+HN5iffMSBtAyAKAnFJeElR5dVD11XN7H+ecYsOjk0bNEkfTjse8cye1iQ3r0NEB75zfJoW8NklZMKDyUi90hUoD4wEsVJr8ubw0cHJgGw9uJ5+LbxOKe+ZWHcglhCwAKEBmplCpN9I0l5E1yQts/UMjJ0199g+NaGBoVLHhiefY8KgGhpI/x8afR9V1YlCxoVENDI+c9OxmGOjiBQM2HrqSBbZQKsHN3x57VJQFVVkWVIW/P7ZdUea1Z7oVUyFkAQBOycxUWVaiyrISLcrwZznnNDTiJgW3k4LcVIEuSWhLbHOsf0gDQyMajA9//vbwTIfr5K2Pq/CDV6g0OHm77OTjXjgrUUVpQBVlY9sToa0i4XWVZUHqfeYxQhYAIGeYmcpKTGUlAdWE5vezh0dGJ4WugaERf7RuRH2DI+of9PbHn8e2kxzvGxxR94lBtfV4x8ffY2hkxv0qCdhECCtLCGQJ4Wxse/LIW4kqygIT23GvCwZMAZNMJpm3Rs9s4piZFyTHtgNmMvnHGMGbFiELAABJJUFvTVfV3GZXT2lsrVzfYJKQNjSi/sHh8ZCWGM6Shb2j/UMnhbtkJagyJT54Bbw0Nr5tcduyk4+NBTVLeB+ziXB30rFJ7z35PQIB71jAf0OTtLyuUt+8ce28fT8SEbIAAJgn8WvlMmVk1I0HsoG48JUY2voHvStQnaRR5wVASRp1Ts75x+Rtu6mOSf7xsfdxkvefRkcnn49/byfnf+bEe086FvcZo/77TWo3OtHGm+V1Cf3w2leWZe77nApCFgAABSQYMFX5t9lAdrGaDgAAIAMIWQAAABlAyAIAAMiAlEKWmV1tZrvNbI+ZbUxyvtzM/qd//kUza44799f+8d1mdlUa+w4AAJCzpg1ZZhaU9ICkayStlnSjma1OaPbnknqccysl/b2ku/3XrpZ0g6RzJV0t6Vv++wEAABS0VEayLpK0xzm31zk3KOkRSdcntLle0r/4249Juty8u5RdL+kR51zMOfeWpD3++wEAABS0VELWMklvx+23+ceStnHODUs6Kqk+xdfKzG42sxYza+no6Ei99wAAADkqJxa+O+e+45yLOueijY2N2e4OAADAnKUSsg5IOj1uv8k/lrSNmZVIqpXUleJrAQAACk4qIWurpLPMbIWZlclbyL45oc1mSR/3tz8i6Vnn3Z9/s6Qb/KsPV0g6S9Kv09N1AACA3DXtPfedc8NmdoukpyQFJT3onNthZpsktTjnNkv6Z0nfM7M9krrlBTH57R6VtFPSsKS/dM7NvAQ5AABAnrGxgpC5IhqNupaWlmx3AwAAYFpmts05F012LicWvgMAABSanBvJMrMOSa3z8FENkjrn4XOQOfwM8xs/v/zHzzD/8TOcu4hzLumtEXIuZM0XM2uZangP+YGfYX7j55f/+BnmP36GmcV0IQAAQAYQsgAAADKgmEPWd7LdAcwZP8P8xs8v//EzzH/8DDOoaNdkAQAAZFIxj2QBAABkTNGFLDO72sx2m9keM9uY7f5gZszsdDP7mZntNLMdZvaZbPcJs2NmQTN72cyeyHZfMHNmttDMHjOz181sl5ldku0+IXVm9ln/d+hrZvZ9Mwtlu0+FqKhClpkFJT0g6RpJqyXdaGars9srzNCwpM8751ZLuljSX/IzzFufkbQr253ArH1D0o+dc+dI+h3xs8wbZrZM0v8hKeqcO09eybwbsturwlRUIUvSRZL2OOf2OucGJT0i6fos9wkz4Jxrd8695G/3yvvFviy7vcJMmVmTpN+X9E/Z7gtmzsxqJb1fXt1aOecGnXNHstopzFSJpAozK5FUKelglvtTkIotZC2T9Hbcfpv4BzpvmVmzpLWSXsxyVzBz90n6PyWNZrkfmJ0Vkjokfdef8v0nM6vKdqeQGufcAUn/VdJ+Se2SjjrnfpLdXhWmYgtZKBBmVi3p/5V0q3PuWLb7g9SZ2YckHXbObct2XzBrJZLWSfpH59xaSSckscY1T5hZWN4szgpJSyVVmdlN2e1VYSq2kHVA0ulx+03+MeQRMyuVF7Aecs79INv9wYy9V9J1ZrZP3pT975rZv2a3S5ihNkltzrmxUeTH5IUu5IcrJL3lnOtwzg1J+oGk92S5TwWp2ELWVklnmdkKMyuTt9Bvc5b7hBkwM5O3DmSXc+7ebPcHM+ec+2vnXJNzrlne38FnnXP8X3Qecc4dkvS2mZ3tH7pc0s4sdgkzs1/SxWZW6f9OvVxcuJARJdnuwHxyzg2b2S2SnpJ3NcWDzrkdWe4WZua9kv5E0nYze8U/9n8557Zkr0tAUfovkh7y/4d1r6RPZrk/SJFz7kUze0zSS/Ku2H5Z3Pk9I7jjOwAAQAYU23QhAADAvCBkAQAAZAAhCwAAIAMIWQAAABlAyAIAAMgAQhYAAEAGELIAAAAygJAFAACQAf8/5IH+xG9xCroAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlkAAAF1CAYAAADbfv+XAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAA+bElEQVR4nO3deXRc9Z3n/fdXUkmyNq+yvEiWF4xtGRxjyw4EErtJk2BCQwLN5mHLRp6Z5ulMJkyGDDlpQieH9IRmunOSJ082nIY0WWAaQvJgoMMy0JmkoWQ222DHEJV3I1slW4u1VX2fP+pKLsuSVZIlVZX0eZ2jU7fu/d2qb6kAffjd3/39zN0RERERkZGVk+4CRERERMYjhSwRERGRUaCQJSIiIjIKFLJERERERoFCloiIiMgoUMgSERERGQUKWSIiIiKjQCFLRDKCmb1gZlEzK0h3LSIiI0EhS0TSzszmAx8EHLhiDN83b6zeS0QmHoUsEckENwN/AH4C3NKz08yqzOxfzKzBzI6Y2XeSjn3WzN4ys2Yz225mq4L9bmZnJbX7iZl9Pdheb2Z7zey/mdlBYJOZTTWz3wTvEQ22K5POn2Zmm8xsf3D88WD/VjP7i6R2ITM7bGbnjdYvSUSyi0KWiGSCm4F/Dn4+amYVZpYL/AaIAPOBucDPAczsGuDu4LwyEr1fR1J8r1nANKAauI3Efwc3Bc/nAceB7yS1fwgoApYDM4H/Gex/ELgxqd1lwAF3fzXFOkRknDOtXSgi6WRmFwHPA7Pd/bCZvQ18n0TP1hPB/u4+5zwNPOnu/9jP6zmw2N13Bc9/Aux196+Y2XrgGaDM3dsHqGcl8Ly7TzWz2cA+YLq7R/u0mwPsAOa6+zEzexR42d3/xzB/FSIyzqgnS0TS7RbgGXc/HDx/ONhXBUT6BqxAFfDOMN+vITlgmVmRmX3fzCJmdgx4EZgS9KRVAY19AxaAu+8HfgdcbWZTgA0keuJERADQoE8RSRszmwRcC+QGY6QACoApwCFgnpnl9RO09gCLBnjZNhKX93rMAvYmPe/bff9FYAnwfnc/GPRkvQpY8D7TzGyKuzf1817/BHyGxH9Lf+/u+waoSUQmIPVkiUg6fRyIATXAyuBnGfBScOwA8E0zKzazQjO7MDjvR8AdZrbaEs4ys+rg2GvARjPLNbNLgXWD1FBKYhxWk5lNA/6m54C7HwA2A/9PMEA+ZGYfSjr3cWAV8HkSY7RERHopZIlIOt0CbHL33e5+sOeHxMDzG4C/AM4CdpPojboOwN0fAb5B4tJiM4mwMy14zc8H5zUB/yE4djr/AEwCDpMYB/ZUn+M3AV3A28B7wH/uOeDux4H/BSwA/iX1jy0iE4EGvouInAEz+ypwtrvfOGhjEZlQNCZLRGSYgsuLnybR2yUichJdLhQRGQYz+yyJgfGb3f3FdNcjIplHlwtFRERERoF6skRERERGgUKWiIiIyCjIuIHvM2bM8Pnz56e7DBEREZFB1dXVHXb38v6OZVzImj9/PuFwON1liIiIiAzKzCIDHdPlQhEREZFRMGjIMrMHzOw9M9s6wHEzs2+b2S4ze8PMViUdu8XM/hj83DKShYuIiIhkslR6sn4CXHqa4xuAxcHPbcD3oHeSvr8B3g+sBf7GzKaeSbEiIiIi2WLQkBVMstd4miZXAg96wh+AKWY2G/go8K/u3ujuUeBfOX1YExERERk3RmJM1lwSsx732BvsG2j/KczsNjMLm1m4oaFhBEoSERERSa+MGPju7j9w91p3ry0v7/cuSBEREZGsMhIhax9QlfS8Mtg30H4RERGRcW8kQtYTwM3BXYbnA0fd/QDwNPARM5saDHj/SLBPREREZNwbdDJSM/sZsB6YYWZ7SdwxGAJw9/8XeBK4DNgFtAGfDI41mtnfAq8EL3WPu59uAL2IiIjIuDFoyHL3GwY57sBfDXDsAeCB4ZUmIiIikr0yYuC7iIiIyHiTcWsXioiIyPjk7sTiTswdd05sxyHmTtydeLAv7iS248H+YF8s2OeeOCex3dMO4n7inEmhXN6/cHraPq9CloiISBbpjsU53hXjeFeMjq5guzPxvD34SeyL9263B23au0/e3x1PhJq4DxBc3InFSQoxJ4JOf4EoPkDY6TlnrJ01s4Tf/pd1Y/6+PRSyREREzpC70xXzfgJNcviJDxiGEoEp1huejncmndMTlILX7IoNPayYwaRQLpNCuRSGcikM5VAYyiUvN4dcgxwzcnKM3BwjlGPkWGI7x3p+SDzvOWYkbRs5OZxyTm7OidfNMYJ2fV878boWvGbvdk7idS3Yd6JN4pycnIHfN7nWSaHcUfi2U6eQJSIi45q709Edp62zJ8B0c7wzTltn90m9QG1BsDnRLjkIxWjvjtPeeSIItScFquNdMYbTUZMbBIHCUC6T8nOSQlAuU4ryg+c5TMoP2oSSHvNzTxxP2leYl8uk4NikUC4FoRwK8nIws5H/5cppKWSJiEhaxeIehJxu2jvjtHV1J4JLUvjpPwx1B71Aifb9haSeQORDDEChXOsNMz2BpSCUy6RQDjNK8ntDT2+boF1BXs5JAae3TZ9A1BOQQrm6/2w8U8gSEZFBxeNOW1eM5vYumtu7aW7vpq2z++RgkxRuTuo16ttL1Cc8dXbHh1xPT1gpys+jMJRDUX4ek4Len9mTcynKTwSZop6AE4SaoiAc9bRP3p/cTuFHRoJClojIONfZHaelo/ukgNSzfdL+jsSxlqR2LR3dHGvvoqWjO+XeoBwjCD9BeEkKM7PKQgOGn0S7vD5hKOnSV36wPy+XnBxd+pLMp5AlIpKh3J3WzkTvUUt7N8eCcNTSG4YSz4/1CUs9x3ued6TQU5Sfl0NZYR4lBXmUFoYoKcijenoRpYUhSgvzen9KChLPS4K2yeGnZzs/V+N/REAhS0RkVHTF4icFneY+Aam5vau35+hMeo/MoCS/JwSFKCnMY1pxPvOmJQLSieB04nhpYR6lBSfCU0lhHgV56b0LS2Q8UsgSERlEe1eMprYuGls7aWrrpLGtk2hbF9HWTqJtncFjV2K7rZNoayIgDaZv71Fp4el7j078BL1JBXkU5+fp0plIhlLIEpEJ5XhnjGhbZxCYumhsC4JTz/Oe4BSEpWhbJ22dsQFfr7QgjynFIaYV5TO1KJ9F5SVMKQoxtSifsj69R2XBZTj1HolMDApZIpKV3J22zli/YemkXqaksBRt66S9a+DxSWWFeUwtToSlmaWFnF1RmghPwb6pRaET28UhpkzKJz9Pd6GJSP8UskQk7dydlo7u3jDUmHwJboCwFG3tojPWf2Ayg8mTEr1LU4pCzJlSSM2cMqYVJ54n9uczrfhEcJoyKUSebtsXkRGkkCUio87daWjuYMehZnYcbOaPh1qoP9IaBKZEkOoeYLrsHIOpQViaVpxP1bQi3lc55aRLdFOLT+5lmjwpRK7GKYlImilkiciIamrrZOehFnYcambnwebE46Fmmtq6etvMKMlnwYxiFs4oYWpxKLgUd2pYmlaUT2mhBnaLSHZSyBKRYWnt6OaP77WcFKR2HGzmveaO3jalhXmcXVHKhnNms6SihLNnlXJ2RSkzSgrSWLmIyNhQyBKR0+rojvHOe63sDILUzkOJULWn8Xhvm8JQDotnlvLBxeUsmVXC2RWlLJlVyqyyQk1KKSITlkKWiADQHYsTaWw7pWeq/kgbsWC8VF6Osai8hJVVU7l2dRVnzyplSUUpVdOKNAZKRKQPhSyRCSYed/Y1He/tkUqEqhbeaWjpXajXDKqnFXF2RSmXnTu7t2dq/vRiTVkgIpIihSyRcaq/O/p2HGrmj4eaaU2aXHPO5ELOnlXKhxbP6A1Ti8pLmJSviTJFRM6EQpbIOJDqHX2LZ5ZyTW1VEKZKWFxRSllhKI2Vi4iMXwpZIlmkvzv6dh5q5tCxpDv6CvI4e5bu6BMRSTeFLJEM1NEd492G1t7B56e7o++is8p7e6WWVJQye7Lu6BMRyQQKWSJpdrSti+0HjrH9wDG27T/K9v3H2PVeS+8M6Hk5xsLyYt5XOUV39ImIZBGFLJEx4u7sP9rO9v3H2L4/CFQHjrE3eqJ3qry0gOVzyrh46UyWzi5jqe7oExHJWgpZIqOgOxbnnYZWth84GgSqRE9Vz0B0M1gwo5iVVVPY+P55LJ8zmWWzS5lZWpjmykVEZKQoZImcodaObt4+mOidSlzyO8bbB5t755wqyMth6axSNpwzi5rZZdTMmczSWaUUF+hfPxGR8Uz/lRcZgobmjt7LfD2X/f50pBVPDJ9iSlGI5XPKuOWCamrmlLF8zmQWzigmL1eX+0REJhqFLJF+xONOpLGtdyB6z+W+hqTFjyunTmL5nDKuXDk3CFRlurNPRER6pRSyzOxS4B+BXOBH7v7NPsergQeAcqARuNHd9wbH/g74WND0b939FyNUu8iIaO+K8cdDLb09VNv2H+OtA8doC2ZFz8sxzppZwocWl/eGqWWzy5g8SZN4ioiMCHeIdUF3O8Q6E4/dHUnbnace6+6AWMeJ7VOet0NxOfz536TtYw0asswsF/gucAmwF3jFzJ5w9+1Jze4DHnT3fzKzi4F7gZvM7GPAKmAlUAC8YGab3f3YCH8OkZQ0tXWedKlv2/5j7Gpo6V0AuaQgj2WzS7m2tioYP1XG4ooSCvK0xIyIjEPuwwwy/bXveX664HOa1xsJOSHIK0j85BZA+dkj87rDlEpP1lpgl7u/C2BmPweuBJJDVg3wX4Lt54HHk/a/6O7dQLeZvQFcCvzyzEsXGZh7YhHk5Et92/cfY1/TiekSKsoKWD5nMpfUVPT2UFVNLSJHc0+JZB936GqDzlbobAke+24P8VjX8cHfN5u5Q7xr8HapyAlBXiHk5Scec/NPfV5UcnIAyisY4PkA5+cVDnx+XmFiX05mjX9NJWTNBfYkPd8LvL9Pm9eBq0hcUvwEUGpm04P9f2Nmfw8UAX/GyeFM5Ix1xeK809DCtn0nwtT2A8c4evzEdAkLZxSzunoqN11Q3dtDpWVmRNIk1g1dqQSfoQSkVsBTe3/LgfwSyC9O+ilJXFqaOv/E87zCxH9AxrPc/BRDzmlCUQaGm0wxUgPf7wC+Y2a3Ai8C+4CYuz9jZmuA/wM0AL8HYn1PNrPbgNsA5s2bN0IlyXi1r+k4z739Hlv3JsZQ7TjUZ7qE2WV8bMXs3jC1dFYpRfm6x0OkX+4QjyUu+cQ6E+NiUt3ubh9eKBrKpaHc/BOhJzkUTa46OSD1DUyn254I4UkyQip/efYBVUnPK4N9vdx9P4meLMysBLja3ZuCY98AvhEcexjY2fcN3P0HwA8AamtrU/xfEZlIIkda2bz1IJvfPMDre48CMLUoxPI5k/nkB+ZTM6eMmtllLNB0CZIp4vHhBZchb49A21R7gAYTKjo10BSUQumswYNPf89DxYmeFJEslUrIegVYbGYLSISr64GNyQ3MbAbQ6O5x4Msk7jTsGTQ/xd2PmNkKYAXwzAjWL+PYOw0tbH7zAE++eZDtBxL3SqyonMx/u3Qpl54zi/nTi8bXdAnxGLS8B8374diBxB+/3PzgJzS0bXXdn6z3zqXj0NU+jMf2xPiclB6D82Kdo/NZckIp/nMQgtDkYfzzM4S2eYVQEISiUBHk6AYRkWSDhix37zaz24GnSUzh8IC7bzOze4Cwuz8BrAfuNTMncbnwr4LTQ8BLwR/CYySmduge+Y8h44G7s/NQC0++eYDNWw+w81ALAKvmTeErH1vGR5fPompaUZqrHKaOFmg+AMf2D/B4AFoOgZ9yNX14LHcYfzyHGOTO9A+35QwxvAwx5PR99Pgwf5c5kDcJQoX9PBZCyczEY2jSyY89g3RH6vfVsz2e/sdCZJwz98y6OldbW+vhcDjdZcgYcXe27T/G5q0H2PzmQd493IoZrJk/jcvOmcVHz5nF7MmT0l3mwOIxaG3oPzT19Eg1H4COfmYtKZgMZbOhdDaUzQkeZ0PpnMRj3qQMuNSUtG+kAuCZys0/OeT0DTcDPvYXklJ4VLARkdMwszp3r+3vmEYDy5hzd17fezRxKXDrAfY0Hic3xzh/4TQ+ddECPrK8IjMWSu5s7ROW+j4egOaDp4YPy02MQSmdnZijZeH6k8NTz2N+cVo+1rDFY8McBzTAtseGEXoKdUlKRLKGQpaMiXjc2bI7ypNvHuSprQfYf7SdUK7xgUUzuP3PzuKSmllMKx6jAa7xeKL3qb/QlNwT1XH01HMLyk70OM1Y109P1JzEbeDjMQjk5CZ+QhkQgEVEsoBCloyaWNx5+U+NbN56gKe2HuS95g7y83L40OIZfPEjS/jzZRVMLhrhpWk621IY+3QQ4n2GBloulFQkQtP0s2DBh04NT6WzE4N8RUREUqCQJSOqKxbn9+8cYfPWgzyz7SBHWjspDOXwZ0tmcuk5s7h46UxKC4cRrOJxaDs8yNin/dDeT+9TfumJHqcFH+x/DFTJzPHZ+yQiImmjkCVnrKM7xu92HWbzmwd5Zvshjh7vojg/l4uXVbDhnFmsX1J++slAu44PHJqSxz71Xf7BchK9T6WzYfoimH9R/2OfCkpH9xcgIiLSD4UsGZb2rhgv7mxg89aD/Hb7IZo7uiktzOOSZRVces4sPnR2OYW5Bm1H4Mi2gcPTsf3Q3nTqG+SXnOhpqr6w//BUPBNy9Y+wiIhkJv2FkpS1dXbz/NsNbN56gH97ey+lXYc5q7CZL1d1s3ZGB/Pzj5LXchD+cACeCULUKYuP2omxT1MXQPUH+h/7VFiWls8oIiIyUhSy5GTu0NbY29vU3riH+j/t4vD+emLH9rPQG/l6TpQpOc1QQGI1jr3BT6j4xNin6gv6D08lFep9EhGRCUF/7SaS7o7+J8o8aTD5QYh19J5SCJztxgybTEdRBZOmn03ZrOoBxj6VadJGERGRgELWeNPSAHv+AIe2nTqYvO3Iqe3zJkHZbLqKZ7GvaDnbYu+nLjqJ/fGpxIorOGfZMi46r4bzqmeSk6MAJSIikiqFrGzmDo3vwu7fBz9/gCO7ThwvLk9cops8FyprT5m2oCFnGk/tauepbQf5wzuNxOLOvGlFbLhwFv/xnNmsqJw8vhZgFhERGUMKWdkk1gUH30iEqZ5Q1dqQODZpGsw7H1bdDPMugNnvSyxO28eBo8d5autBNv/2IK9E/oQ7LCwv5j+uW8SGc2dRM7tMwUpERGQEKGRlso5m2PvKiVC1NwxdbYljU+fDWX+eCFbzLoDpiyEnp9+X2dPYlghWWw+wZXcTAEsqSvn8hxdz2bmzWTyzRMFKRERkhClkZZLmgyd6qHb/Hg6+CR5PTLo569ygl+p8qDo/ccnvNP50uLV3OZs39iZmQT9nbhn/9aNLuPScWSwq1/IwIiIio0khK13c4fDOk0NVtD5xLFSUGEP1of+aCFWVa4Y0a/mDv6/nq7/aBsDKqil8ecNSNpwzm3nTi0bhg4iIiEh/FLLGSncnHHgtKVT9AY43Jo4VlyfC1NrbEo+zVkDu8BdO/tVr+1k6q5Qf37qGuVMmjUz9IiIiMiQKWaOl/SjseeXEnX/76qC7PXFs+lmw9LLEWKp5F8C0hSM2v1R7V4w39x7l1gvnK2CJiIikkULWSDm67+ReqkNbAYecvMSdfms+c2I8VUn5qJWxbf9ROmNxVs2bOmrvISIiIoNTyBqOeBwa3j45VB3dnTiWXwJVa6HmvydC1dzVkF88ZqWF66MArK5WyBIREUknhaxUdLXD/ldPhKo9f0hcDgQomZVYp+8DtydC1czlaV2bLxyJMn96EeWlp86RJSIiImNHIas/bY2w5+UToWr/Foh1Jo6VL4XlnwjGU50PU6ozZr0+d2dLJMr6JTPTXYqIiMiEp5DlDk27T55FveGtxLGcEMxdBef/x0Soqno/FE1Lb72n8afDrRxp7aR2vi4VioiIpNvEC1nxeGJQenKoat6fOFYwGea9H1ZckwhVc86DUPbcoReOJMZj1Wo8loiISNpNvJDlcdi0ATpboKwS5l94Ymma8mUDLk2TDbZEopQV5mk2dxERkQww8UJWbh5c/88wbRFMqUp3NSMqHImyunoqOTmZMUZMRERkIsvebpszsXD9uAtYTW2d7Hqvhdr5mTtmTEREZCKZmCFrHKqLaH4sERGRTKKQNU6EI1Hycoz3VU5JdykiIiKCQta4UVcfZfncyUzKz013KSIiIoJC1rjQ2R3n9b1NmrpBREQkgyhkjQPb9h+lozuu8VgiIiIZJKWQZWaXmtkOM9tlZnf2c7zazJ41szfM7AUzq0w69j/MbJuZvWVm3zbLkDVoxpE6TUIqIiKScQYNWWaWC3wX2ADUADeYWU2fZvcBD7r7CuAe4N7g3A8AFwIrgHOANcC6EateAAjXR6maNomZZYXpLkVEREQCqfRkrQV2ufu77t4J/By4sk+bGuC5YPv5pOMOFAL5QAEQAg6dadFygrsTjkSprdb8WCIiIpkklZA1F9iT9HxvsC/Z68BVwfYngFIzm+7uvycRug4EP0+7+1t938DMbjOzsJmFGxoahvoZJrTdjW0cbunQeCwREZEMM1ID3+8A1pnZqyQuB+4DYmZ2FrAMqCQRzC42sw/2Pdndf+Dute5eW15ePkIlTQzh+mA81nyFLBERkUySytqF+4DkNWgqg3293H0/QU+WmZUAV7t7k5l9FviDu7cExzYDFwAvjUDtAtTtjlJakMfimaXpLkVERESSpNKT9Qqw2MwWmFk+cD3wRHIDM5thZj2v9WXggWB7N4kerjwzC5Ho5TrlcqEMX119lPOqp5KrRaFFREQyyqAhy927gduBp0kEpF+6+zYzu8fMrgiarQd2mNlOoAL4RrD/UeAd4E0S47Zed/dfj+xHmLiOHu9i53vNmrpBREQkA6VyuRB3fxJ4ss++ryZtP0oiUPU9LwZ87gxrlAFs2R3FXfNjiYiIZCLN+J7F6uqj5OYYK+dNSXcpIiIi0odCVhYLRxqpmV1GUX5KHZIiIiIyhhSyslRXLM7re45qfiwREZEMpZCVpd46cIzjXTGFLBERkQylkJWlNAmpiIhIZlPIylJ1kShzp0xi9uRJ6S5FRERE+qGQlYUSi0I36lKhiIhIBlPIykJ7o8c5dKxDlwpFREQymEJWFtqyOzEeSz1ZIiIimUshKwuF66MU5+eypEKLQouIiGQqhawsFI5EOW/eVPJy9fWJiIhkKv2VzjLN7V3sOHhMlwpFREQynEJWlnl1dxNx1/xYIiIimU4hK8uEI1FyDM6bp5AlIiKSyRSyssyWSJSls8ooKdCi0CIiIplMISuLdMfivLo7qvFYIiIiWUAhK4u8fbCZ1s6YxmOJiIhkAYWsLFIX0SSkIiIi2UIhK4uEI1FmlRUyd4oWhRYREcl0CllZpK6+kdXzp2Jm6S5FREREBqGQlSX2Nx1n/9F2anWpUEREJCsoZGUJjccSERHJLgpZWaIuEmVSKJdls8vSXYqIiIikQCErS4QjjaysmkJIi0KLiIhkBf3FzgKtHd28daBZ82OJiIhkEYWsLPDaniZicdd4LBERkSyikJUFwvVRzGCVQpaIiEjWUMjKAnW7oyypKKWsMJTuUkRERCRFClkZLhZ3Xo1E1YslIiKSZRSyMtzOQ800d3RrElIREZEso5CV4cLBJKS11dPSXImIiIgMRUohy8wuNbMdZrbLzO7s53i1mT1rZm+Y2QtmVhns/zMzey3pp93MPj7Cn2Fcq6tvpLy0gKppWhRaREQkmwwasswsF/gusAGoAW4ws5o+ze4DHnT3FcA9wL0A7v68u69095XAxUAb8MzIlT/+hSNRaqu1KLSIiEi2SaUnay2wy93fdfdO4OfAlX3a1ADPBdvP93Mc4C+Bze7eNtxiJ5pDx9rZGz2u+bFERESyUCohay6wJ+n53mBfsteBq4LtTwClZja9T5vrgZ/19wZmdpuZhc0s3NDQkEJJE4MWhRYREcleIzXw/Q5gnZm9CqwD9gGxnoNmNhs4F3i6v5Pd/QfuXuvuteXl5SNUUvYL10cpyMth+ZzJ6S5FREREhigvhTb7gKqk55XBvl7uvp+gJ8vMSoCr3b0pqcm1wGPu3nVG1U4wdZFG3lc1hfw83QQqIiKSbVL56/0KsNjMFphZPonLfk8kNzCzGWbW81pfBh7o8xo3MMClQunf8c4Y2/Yf0/xYIiIiWWrQkOXu3cDtJC71vQX80t23mdk9ZnZF0Gw9sMPMdgIVwDd6zjez+SR6wv73yJY+vr22p4nuuFM7XyFLREQkG6VyuRB3fxJ4ss++ryZtPwo8OsC59Zw6UF4GsWV3YtD7qnkKWSIiItlIg30yVLi+kbNmljClKD/dpYiIiMgwKGRloHjcqQsmIRUREZHspJCVgXY1tHCsvVvzY4mIiGQxhawMFK4PFoWer0WhRUREspVCVgYKRxqZXpzP/OlF6S5FREREhkkhKwPVRaKs1qLQIiIiWU0hK8M0NHcQOdKm+bFERESynEJWhtGi0CIiIuODQlaGqYs0kp+XwzlztSi0iIhINlPIyjDhSJQVcydTkJeb7lJERETkDChkZZD2rhhb9x1ltcZjiYiIZD2FrAzyxt6jdMWc2mrNjyUiIpLtFLIyiAa9i4iIjB8KWRmkLtLIwhnFTCvWotAiIiLZTiErQ7h77ySkIiIikv0UsjLEOw2tRNu6NAmpiIjIOKGQlSHqIo0ArNagdxERkXFBIStDhOujTC0Ksai8ON2liIiIyAhQyMoQdbu1KLSIiMh4opCVARpbO3m3oZVVGvQuIiIybihkZYCe+bE0CamIiMj4oZCVAcKRRkK5xopKLQotIiIyXihkZYC6+ijnzJ1MYUiLQouIiIwXCllp1tEd4419R6nVeCwREZFxRSErzbbuO0pnd1zzY4mIiIwzCllppkWhRURExieFrDQL10epnl5EeWlBuksRERGREaSQlUZaFFpERGT8UshKo/ojbRxp7dT8WCIiIuOQQlYahesTi0LXzldPloiIyHiTUsgys0vNbIeZ7TKzO/s5Xm1mz5rZG2b2gplVJh2bZ2bPmNlbZrbdzOaPYP1ZrS4Spawwj7PKS9JdioiIiIywQUOWmeUC3wU2ADXADWZW06fZfcCD7r4CuAe4N+nYg8C33H0ZsBZ4byQKHw96xmPl5GhRaBERkfEmlZ6stcAud3/X3TuBnwNX9mlTAzwXbD/fczwIY3nu/q8A7t7i7m0jUnmWa2rr5I/vtWjQu4iIyDiVSsiaC+xJer432JfsdeCqYPsTQKmZTQfOBprM7F/M7FUz+1bQMzbhbdndMz+WBr2LiIiMRyM18P0OYJ2ZvQqsA/YBMSAP+GBwfA2wELi178lmdpuZhc0s3NDQMEIlZbZwfZS8HGNl1ZR0lyIiIiKjIJWQtQ+oSnpeGezr5e773f0qdz8PuCvY10Si1+u14FJjN/A4sKrvG7j7D9y91t1ry8vLh/VBsk04EmX5nDIm5atjT0REZDxKJWS9Aiw2swVmlg9cDzyR3MDMZphZz2t9GXgg6dwpZtaTnC4Gtp952dmtszvO63uadKlQRERkHBs0ZAU9ULcDTwNvAb90921mdo+ZXRE0Ww/sMLOdQAXwjeDcGIlLhc+a2ZuAAT8c8U+RZbYfOEZHd1zzY4mIiIxjeak0cvcngSf77Ptq0vajwKMDnPuvwIozqHHc6ZmEVHcWioiIjF+a8T0N6iJRKqdOoqKsMN2liIiIyChRyBpj7k44EqVWvVgiIiLjmkLWGNvTeJyG5g5Wz9egdxERkfFMIWuMhSPBotDqyRIRERnXFLLGWF0kSmlBHmdXlKa7FBERERlFClljrC4S5bzqqeRqUWgREZFxTSFrDB093sWOQ82snqdLhSIiIuOdQtYYenV3FHc0CamIiMgEoJA1huoiUXK1KLSIiMiEoJA1hsL1UZbNLqW4IKWJ9kVERCSLKWSNka5YnNf2NFGrRaFFREQmBIWsMfL2gWaOd8W0XqGIiMgEoZA1RnomIVXIEhERmRgUssZIOBJlzuRC5kyZlO5SREREZAwoZI0Bd6euPqr1CkVERCYQhawxsK/pOAePtWu9QhERkQlEIWsM1EWigMZjiYiITCQKWWOgLhKlOD+XpbO0KLSIiMhEoZA1BsL1UVbOm0Jern7dIiIiE4X+6o+ylo5u3j54jNWahFRERGRCUcgaZa/ujhJ3NOhdRERkglHIGmXh+ig5BufNm5LuUkRERGQMKWSNsrpIlCWzyigtDKW7FBERERlDClmjKBZ3Xt0d1aVCERGRCUghaxS9ffAYrZ0xaucrZImIiEw0ClmjqGcS0lXzFLJEREQmGoWsURSuj1JRVkDlVC0KLSIiMtEoZI2iukiU2uppmFm6SxEREZExppA1Sg4cPc6+puNar1BERGSCUsgaJT3jsTToXUREZGJSyBol4fook0K5LJtdlu5SREREJA1SCllmdqmZ7TCzXWZ2Zz/Hq83sWTN7w8xeMLPKpGMxM3st+HliJIvPZHWRKO+rmkxIi0KLiIhMSIMmADPLBb4LbABqgBvMrKZPs/uAB919BXAPcG/SsePuvjL4uWKE6s5orR3dbD9wjFotCi0iIjJhpdLNshbY5e7vunsn8HPgyj5taoDngu3n+zk+oby+p4lY3Fmt8VgiIiITViohay6wJ+n53mBfsteBq4LtTwClZjY9eF5oZmEz+4OZfby/NzCz24I24YaGhtSrz1DhSBQzTUIqIiIykY3UgKE7gHVm9iqwDtgHxIJj1e5eC2wE/sHMFvU92d1/4O617l5bXl4+QiWlTzgS5eyZpUyepEWhRUREJqpUQtY+oCrpeWWwr5e773f3q9z9POCuYF9T8LgveHwXeAE474yrzmDxuPNqJKpLhSIiIhNcKiHrFWCxmS0ws3zgeuCkuwTNbIaZ9bzWl4EHgv1Tzaygpw1wIbB9pIrPRDvfa6a5o5vVulQoIiIyoQ0asty9G7gdeBp4C/ilu28zs3vMrOduwfXADjPbCVQA3wj2LwPCZvY6iQHx33T3cR2ywvWahFREREQgL5VG7v4k8GSffV9N2n4UeLSf8/4PcO4Z1phV6iJRZpQUMG9aUbpLERERkTTSTJkjLBxppLZ6qhaFFhERmeAUskbQe8fa2dN4XJcKRURERCFrJPUsCr26WiFLRERkolPIGkHhSJSCvByWz5mc7lJEREQkzRSyRlA4EuV9lVPIz9OvVUREZKJTGhghxztjbNt3VJOQioiICKCQNWJe39tEd9yp1XgsERERQSFrxGjQu4iIiCRTyBohdZEoZ80sYUpRfrpLERERkQygkDUC4nGnLhLVpUIRERHppZA1At5paOHo8S5WKWSJiIhIQCFrBISD8VjqyRIREZEeClkjIFwfZXpxPgtmFKe7FBEREckQClkjoC7SyCotCi0iIiJJFLLO0OGWDuqPtOlSoYiIiJxEIesM9cyPVauZ3kVERCSJQtYZqotEyc/VotAiIiJyMoWsMxSub+TcyskUhnLTXYqIiIhkEIWsM9DeFWPrvmMajyUiIiKnUMg6A2/uO0pnLK71CkVEROQUCllnIFyvRaFFRESkfwpZZ6AuEmXhjGKmlxSkuxQRERHJMApZw+TubNkd1XqFIiIi0i+FrGF693Arja2dGvQuIiIi/VLIGqa6ek1CKiIiIgNTyBqmcKSRKUUhFs4oSXcpIiIikoEUsoYpHImyet5UcnK0KLSIiIicSiFrGBpbO3m3oZXVulQoIiIiA1DIGoYtPYtCV09LcyUiIiKSqRSyhiEciRLKNVZUalFoERER6Z9C1jDURRpZPkeLQouIiMjAUgpZZnapme0ws11mdmc/x6vN7Fkze8PMXjCzyj7Hy8xsr5l9Z6QKT5eO7hiv7z2q+bFERETktAYNWWaWC3wX2ADUADeYWU2fZvcBD7r7CuAe4N4+x/8WePHMy02/rfuO0dkd1/xYIiIiclqp9GStBXa5+7vu3gn8HLiyT5sa4Llg+/nk42a2GqgAnjnzctOvZ9D7ag16FxERkdNIJWTNBfYkPd8b7Ev2OnBVsP0JoNTMpptZDvD3wB2newMzu83MwmYWbmhoSK3yNAlHGqmeXkR5qRaFFhERkYGN1MD3O4B1ZvYqsA7YB8SA/wQ86e57T3eyu//A3Wvdvba8vHyEShp57k5dMAmpiIiIyOnkpdBmH1CV9Lwy2NfL3fcT9GSZWQlwtbs3mdkFwAfN7D8BJUC+mbW4+ymD57NB5Egbh1s6NQmpiIiIDCqVkPUKsNjMFpAIV9cDG5MbmNkMoNHd48CXgQcA3P0/JLW5FajN1oAFifmxQJOQioiIyOAGvVzo7t3A7cDTwFvAL919m5ndY2ZXBM3WAzvMbCeJQe7fGKV606ou0khZYR6LZ2pRaBERETm9VHqycPcngSf77Ptq0vajwKODvMZPgJ8MucIMEq6Psqpai0KLiIjI4DTje4qOtnXxx/daNAmpiIiIpEQhK0VbdifGY61SyBIREZEUKGSlKBxpJDfHWFk1Jd2liIiISBZQyEpRuD7K8jllFOWnNIxNREREJjiFrBR0xeK8vreJ1bpUKCIiIilSyErBtv3HaO+Ka34sERERSZlCVgrqeiYh1UzvIiIikiKFrBTURRqpnDqJirLCdJciIiIiWUIhaxDuTrg+qvFYIiIiMiQKWYPYGz3Oe80dmoRUREREhkQhaxDhSCMAqzXoXURERIZAIWsQ4foopQV5LJlVmu5SREREJIsoZA2iLhJl5bwp5GpRaBERERkChazTONbexY5DzZofS0RERIZMIes0Xt3dhDu6s1BERESGTCHrNOrqG8kxWDlvSrpLERERkSyjkHUa4UiUZbPLKCnQotAiIiIyNApZA+iOxXltT5PmxxIREZFhUcgawNsHm2nrjLF6vga9i4iIyNApZA0gXJ+YhFQ9WSIiIjIcClkDCEeizJ5cyJwpk9JdioiIiGQhhawB1EW0KLSIiIgMn0JWP/Y1HefA0XZdKhQREZFhU8jqR+94LA16FxERkWFSyOpHXSRKUX4uS7UotIiIiAyTQlY/6iJRzps3hbxc/XpERERkeJQi+mjp6OatA8dYrUWhRURE5AwoZPXx2u4m4loUWkRERM6QFuXrIxxpxAzO06LQIiKSong8zt69e2ltbU13KTIKiouLqaysJCdnaH1TCll91EWiLKkopawwlO5SREQkSxw+fBgzY8mSJUP+QyyZLR6Ps2/fPg4fPszMmTOHdG5K/ySY2aVmtsPMdpnZnf0crzazZ83sDTN7wcwqk/ZvMbPXzGybmf1fQ6pujMXizqu7m6idr0uFIiKSuqamJioqKhSwxqGcnBwqKio4evTo0M8drIGZ5QLfBTYANcANZlbTp9l9wIPuvgK4B7g32H8AuMDdVwLvB+40szlDrnKM7DjYTEtHN7Ua9C4iIkMQi8UIhXQFZLwKhUJ0d3cP+bxUIvdaYJe7v+vuncDPgSv7tKkBngu2n+857u6d7t4R7C9I8f3Spi6SmIRUg95FRGSozCzdJcgoGe53m0romQvsSXq+N9iX7HXgqmD7E0CpmU0PCqsyszeC1/g7d98/rErHQDgSZWZpAZVTtSi0iIiInJmR6lm6A1hnZq8C64B9QAzA3fcElxHPAm4xs4q+J5vZbWYWNrNwQ0PDCJU0dOH6KLXzp+r/RkREROSMpRKy9gFVSc8rg3293H2/u1/l7ucBdwX7mvq2AbYCH+z7Bu7+A3evdffa8vLyoX2CEXLwaDv7mo5rElIREREZEamErFeAxWa2wMzygeuBJ5IbmNkMM+t5rS8DDwT7K81sUrA9FbgI2DFSxY+kcDAeq1bjsUREREaEuxOPx9NdRtoMGrLcvRu4HXgaeAv4pbtvM7N7zOyKoNl6YIeZ7QQqgG8E+5cB/25mrwP/G7jP3d8c4c8wIuoiUSaFcqmZU5buUkREREbUN7/5TRYtWkRpaSk1NTU89thjvcd++MMfsmzZst5jW7ZsAWDPnj1cddVVlJeXM336dG6//XYA7r77bm688cbe8+vr6zGz3rvv1q9fz1133cWFF15IUVER7777Lps2bep9j4ULF/L973//pPp+9atfsXLlSsrKyli0aBFPPfUUjzzyCKtXrz6p3f3338+VV/a99y5zpTQZqbs/CTzZZ99Xk7YfBR7t57x/BVacYY1joi4S5X1VkwlpUWgRETlDX/v1NrbvPzaq71Ezp4y/+YvlKbVdtGgRL730ErNmzeKRRx7hxhtvZNeuXfzbv/0bd999N48//ji1tbW88847hEIhYrEYl19+ORdffDEPPfQQubm5hMPhlGt76KGH2Lx5M0uWLMHd2bFjB7/5zW9YuHAhL774Ihs2bGDNmjWsWrWKl19+mZtvvplHH32UD3/4wxw4cIDm5mYWLFjA5z73Od566y2WLVvW+7pf+cpXhvX7SgclCqCts5tt+49p6gYRERmXrrnmGubMmUNOTg7XXXcdixcv5uWXX+ZHP/oRX/rSl1izZg1mxllnnUV1dTUvv/wy+/fv51vf+hbFxcUUFhZy0UUXpfx+t956K8uXLycvL49QKMTHPvYxFi1ahJmxbt06PvKRj/DSSy8B8OMf/5hPfepTXHLJJeTk5DB37lyWLl1KQUEB1113HT/96U8B2LZtG/X19Vx++eWj8jsaDVpWB3htTxOxuGsSUhERGRGp9jCNlQcffJD777+f+vp6AFpaWjh8+DB79uxh0aJFp7Tfs2cP1dXV5OUNLyZUVVWd9Hzz5s187WtfY+fOncTjcdra2jj33HN73+uyyy7r93VuueUWbrjhBr7+9a/z0EMPce2111JQUDCsmtJBPVlAXX0UgFXz1JMlIiLjSyQS4bOf/Szf+c53OHLkCE1NTZxzzjm4O1VVVbzzzjunnFNVVcXu3bv7neW8uLiYtra23ucHDx48pU3yVEgdHR1cffXV3HHHHRw6dIimpiYuu+wy3L33vfqrAeD8888nPz+fl156iYcffpibbrppyJ8/nRSySExCenZFCZOLtCSCiIiML62trZgZPVMkbdq0ia1btwLwmc98hvvuu4+6ujrcnV27dhGJRFi7di2zZ8/mzjvvpLW1lfb2dn73u98BsHLlSl588UV2797N0aNHuffeewd8b4DOzk46OjooLy8nLy+PzZs388wzz/Qe//SnP82mTZt49tlnexdjfvvtt3uP33zzzdx+++2EQqEhXbLMBBM+ZMXjzpbdUc2PJSIi41JNTQ1f/OIXueCCC6ioqODNN9/kwgsvBBJjte666y42btxIaWkpH//4x2lsbCQ3N5df//rX7Nq1i3nz5lFZWckvfvELAC655BKuu+46VqxYwerVqwcdI1VaWsq3v/1trr32WqZOncrDDz/MFVdc0Xt87dq1bNq0iS984QtMnjyZdevWEYlEeo/fdNNNbN269aQ7GrOF9XTXZYra2lofyh0MZ2rHwWY++g8v8vfXvI+rV1eO2fuKiMj4kXwHnIys48ePM3PmTLZs2cLixYvTVsdA37GZ1bl7bX/nTPierN5JSOdrPJaIiEim+d73vseaNWvSGrCGa8LfXVhXH2VGST7zphWluxQRERFJMn/+fNydxx9/PN2lDMuED1nhSJTV1VoUWkREJNP0TDmRrSb05cL3mtvZ3dim+bFERERkxE3okNUzP9ZqjccSERGRETaxQ1YkSkFeDufMmZzuUkRERGScmdAhKxyJ8r7KKeTnTehfg4iIiIyCCZsu2rtibNt/lFVaFFpERERGwYQNWa/vaaIr5tQqZImIiJzihRdeoLJSk3SfiQkbssKRYNC7QpaIiIiMggkbsuoiURaVFzO1OD/dpYiIiMgwuDvxeDzdZQxoQoasnkWhNT+WiIiMd3/3d3/HX/7lX5607/Of/zx//dd/zaZNm1i2bBmlpaUsXLiQ73//+0N+/W9+85ssWrSI0tJSampqeOyxx046/sMf/rD3PWpqatiyZQsAe/bs4aqrrqK8vJzp06dz++23A3D33XeftBh0fX09ZkZ3dzcA69ev56677uLCCy+kqKiId999d9DP8atf/YqVK1dSVlbGokWLeOqpp3jkkUdYvXr1Se3uv/9+rrzyyiH/DgYyIWd8f/dwC01tXZofS0RERsfmO+Hgm6P7HrPOhQ3fHLTZ9ddfz9e+9jWam5spLS0lFovxy1/+kscee4wjR47wm9/8hoULF/Liiy+yYcMG1qxZw6pVq1IuY9GiRbz00kvMmjWLRx55hBtvvJFdu3Yxe/ZsHnnkEe6++24ef/xxamtreeeddwiFQsRiMS6//HIuvvhiHnroIXJzcwmHwym/50MPPcTmzZtZsmQJ7s6OHTsG/Bwvv/wyN998M48++igf/vCHOXDgAM3NzSxYsIDPfe5zJy38/NBDD/GVr3wl5ToGMyF7ssL1Go8lIiITQ3V1NatWrertYXruuecoKiri/PPP52Mf+xiLFi3CzFi3bh0f+chHeOmll4b0+tdccw1z5swhJyeH6667jsWLF/Pyyy8D8KMf/YgvfelLrFmzBjPjrLPOorq6mpdffpn9+/fzrW99i+LiYgoLC7noootSfs9bb72V5cuXk5eXRygUOu3n+PGPf8ynPvUpLrnkEnJycpg7dy5Lly6loKCA6667jp/+9KcAbNu2jfr6ei6//PIhff7TmZA9WeFIlGnF+SycUZzuUkREZDxKoYdpLG3cuJGf/exn3HzzzTz88MNs3LgRgM2bN/O1r32NnTt3Eo/HaWtr49xzzx3Saz/44IPcf//9vesMtrS0cPjwYSBxSXDRokWnnLNnzx6qq6vJyxteDKmqqjrp+ek+x549e7jsssv6fZ1bbrmFG264ga9//es89NBDXHvttRQUFAyrpv5MyJ6sukiUVfO0KLSIiEwM11xzDS+88AJ79+7lscceY+PGjXR0dHD11Vdzxx13cOjQIZqamrjssstw95RfNxKJ8NnPfpbvfOc7HDlyhKamJs4555ze16iqquKdd9455byqqip2797dO84qWXFxMW1tbb3PDx48eEqb5L/fg32OgWoAOP/888nPz+ell17i4Ycf5qabbkr5s6diwoWsrlicpbNKuXjpzHSXIiIiMibKy8tZv349n/zkJ1mwYAHLli2js7OTjo4OysvLycvLY/PmzTzzzDNDet3W1lbMjPLycgA2bdrE1q1be49/5jOf4b777qOurg53Z9euXUQiEdauXcvs2bO58847aW1tpb29nd/97ncArFy5khdffJHdu3dz9OhR7r333tPWMNjn+PSnP82mTZt49tlnicfj7Nu3j7fffrv3+M0338ztt99OKBQa0iXLVEy4kBXKzeF7N65m4/vnpbsUERGRMbNx40Z++9vf9l4qLC0t5dvf/jbXXnstU6dO5eGHH+aKK64Y0mvW1NTwxS9+kQsuuICKigrefPNNLrzwwt7j11xzDXfddRcbN26ktLSUj3/84zQ2NpKbm8uvf/1rdu3axbx586isrOQXv/gFAJdccgnXXXcdK1asYPXq1YOOkRrsc6xdu5ZNmzbxhS98gcmTJ7Nu3ToikUjv8ZtuuomtW7eedEfjSLGhdAuOhdraWh/KHQYiIiLplnyHmmSX48ePM3PmTLZs2cLixYsHbDfQd2xmde5e2985E64nS0RERKTH9773PdasWXPagDVcE/LuQhEREUnN7t27qamp6ffY9u3bmTcve4ffzJ8/H3fn8ccfH5XXV8gSERGRAc2bN4+WlpZ0lzEqeqadGC26XCgiIjICMm2Ms4yc4X63ClkiIiJnKDc3l66urnSXIaOkq6trWBOnKmSJiIicoSlTpnDo0CHi8Xi6S5ERFo/HOXToEJMnTx7yuSnFMjO7FPhHIBf4kbt/s8/xauABoBxoBG50971mthL4HlAGxIBvuPsvhlyliIhIBpsxYwZ79+5lx44d6S5FRkFxcTEzZswY8nmDhiwzywW+C1wC7AVeMbMn3H17UrP7gAfd/Z/M7GLgXuAmoA242d3/aGZzgDoze9rdm4ZcqYiISIbKycnJ6rvsZHSkcrlwLbDL3d91907g58CVfdrUAM8F28/3HHf3ne7+x2B7P/Aeid4uERERkXEtlZA1F9iT9HxvsC/Z68BVwfYngFIzm57cwMzWAvnAKas0mtltZhY2s3BDQ0OqtYuIiIhkrJEa+H4HsM7MXgXWAftIjMECwMxmAw8Bn3T3U0YFuvsP3L3W3Wt7FpkUERERyWapDHzfB1QlPa8M9vUKLgVeBWBmJcDVPeOuzKwM+P+Au9z9DyNQs4iIiEjGG3SBaDPLA3YCHyYRrl4BNrr7tqQ2M4BGd4+b2TeAmLt/1czygc3Ar939H1IqyKwBiAza8MzNAA6PwfvI6NF3mN30/WU/fYfZT9/hmat2934vww3ak+Xu3WZ2O/A0iSkcHnD3bWZ2DxB29yeA9cC9ZubAi8BfBadfC3wImG5mtwb7bnX3107zfmNyvdDMwgOtmi3ZQd9hdtP3l/30HWY/fYeja9CerPFK/2BlP32H2U3fX/bTd5j99B2OLs34LiIiIjIKJnLI+kG6C5Azpu8wu+n7y376DrOfvsNRNGEvF4qIiIiMponckyUiIiIyaiZcyDKzS81sh5ntMrM7012PDI2ZVZnZ82a23cy2mdnn012TDI+Z5ZrZq2b2m3TXIkNnZlPM7FEze9vM3jKzC9Jdk6TOzL4Q/Dd0q5n9zMwK013TeDShQlbSYtcbSKy3eIOZ1aS3KhmibuCL7l4DnA/8lb7DrPV54K10FyHD9o/AU+6+FHgf+i6zhpnNBf4aqHX3c0hMz3R9eqsanyZUyCK1xa4lg7n7AXffEmw3k/gPe9+1NCXDmVkl8DHgR+muRYbOzCaTmAPxxwDu3tmzyodkjTxgUjDheBGwP831jEsTLWSlsti1ZAkzmw+cB/x7mkuRofsH4EvAKWuZSlZYADQAm4JLvj8ys+J0FyWpcfd9wH3AbuAAcNTdn0lvVePTRAtZMk4Ea2T+L+A/u/uxdNcjqTOzy4H33L0u3bXIsOUBq4Dvuft5QCugMa5ZwsymkriKswCYAxSb2Y3prWp8mmgha9DFriXzmVmIRMD6Z3f/l3TXI0N2IXCFmdWTuGR/sZn9NL0lyRDtBfa6e08v8qMkQpdkhz8H/uTuDe7eBfwL8IE01zQuTbSQ9Qqw2MwWBItXXw88keaaZAjMzEiMA3nL3e9Pdz0ydO7+ZXevdPf5JP4dfM7d9X/RWcTdDwJ7zGxJsOvDwPY0liRDsxs438yKgv+mfhjduDAqBl0gejwZaLHrNJclQ3MhcBPwppm9Fuz77+7+ZPpKEpmQ/m/gn4P/YX0X+GSa65EUufu/m9mjwBYSd2y/imZ+HxWa8V1ERERkFEy0y4UiIiIiY0IhS0RERGQUKGSJiIiIjAKFLBEREZFRoJAlIiIiMgoUskRERERGgUKWiIiIyChQyBIREREZBf8/wcRtyW1gsaMAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_loss(history)\n",
    "plot_accuracy(history)\n",
    "\n",
    "scores = model.evaluate(X_test, y_test_cat, verbose=2)\n",
    "print(\" %s: %.2f%%\" % (model.metrics_names[1], scores[1]*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GfAvDPg0bTwL"
   },
   "source": [
    "# Some functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "ivnIFcge8_uM"
   },
   "outputs": [],
   "source": [
    "def run_classification(input_img, target_label):\n",
    "    \"\"\"\n",
    "    Uses the trained model to classify the image passed and return wether it\n",
    "    belongs or not to the target class.\n",
    "       \n",
    "    Parameters:\n",
    "    :input_img: (np array): Input image to be classified.\n",
    "    :target_label: (int): Target class for the attack.\n",
    "\n",
    "    Returns:\n",
    "    :target_reached: (boolean) True if the predicted target is the target class.\n",
    "    :predicted_label: (integer) class that has been predicted.\n",
    "    \"\"\"    \n",
    "    \n",
    "    # Run the model to predict based on pre-processed image\n",
    "    predictions_model = model.predict(input_img, verbose = 0)\n",
    "    \n",
    "    # Get the predicted label by finding the maximum value\n",
    "    predicted_label = np.argmax(predictions_model)\n",
    "    \n",
    "    # Evaluate if the target label has been reached by comparing        \n",
    "    target_reached = (target_label == predicted_label)\n",
    "    \n",
    "    return target_reached, predicted_label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "Yg50F9kt7dm-"
   },
   "outputs": [],
   "source": [
    "loss_object = tf.keras.losses.CategoricalCrossentropy()\n",
    "\n",
    "def evaluate_grads(input_img, tgt_class):\n",
    "  \"\"\" \n",
    "  Calculates the cross entropy loss and the gradient of the loss with\n",
    "  respect to the image\n",
    "\n",
    "  :input_img: (np array) image with respect to which the gradient is calculated.\n",
    "  :tgt_class: (np array) categorical target class of the attack.\n",
    "\n",
    "  Returns\n",
    "  :grad: (tensor) gradient of the loss with respect to the image.\n",
    "  \"\"\"\n",
    "  tgt_class = tgt_class.reshape((1, 10))\n",
    "  image = tf.Variable(input_img)\n",
    "\n",
    "  with tf.GradientTape() as gtape:\n",
    "    gtape.watch(image)\n",
    "    prediction = model(image)\n",
    "    loss = loss_object(tgt_class, prediction)\n",
    "\n",
    "  grads = gtape.gradient(loss, image)\n",
    "  return grads\n",
    "\n",
    "\n",
    "def evaluate_grads_sep(input_img, tgt_class, index): \n",
    "  \"\"\" \n",
    "  Calculates the cross entropy loss for one class and the gradient of the loss\n",
    "   with respect to the image.\n",
    "\n",
    "  :input_img: (np array) image with respect to which the gradient is calculated.\n",
    "  :tgt_class: (np array) categorical target class of the attack.\n",
    "  :index: (int) class for which the cross entropy loss is calculated.\n",
    "\n",
    "  Returns\n",
    "  :grads: (tensor) gradient of the loss with respect to the image.\n",
    "  \"\"\"\n",
    "  image = tf.Variable(input_img)\n",
    "\n",
    "  with tf.GradientTape() as gtape:\n",
    "    gtape.watch(image)\n",
    "    prediction = model(image)[0]\n",
    "    loss = -tgt_class[index] * tf.math.log(prediction[index])\n",
    "\n",
    "  grads = gtape.gradient(loss, image)\n",
    "  return grads\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "slhOznM-bTwM"
   },
   "outputs": [],
   "source": [
    "def condg(g, u_ori, step, epsilon, eta):\n",
    "  \"\"\" \n",
    "  Chooses n_attack images that are classified with an accuracy of 100% by\n",
    "  the model and randomly creates target classes for those images. \n",
    "\n",
    "  Parameters\n",
    "  :g: (np array) gradient of the loss function with respect to the input image.\n",
    "  :u_ori: (np array) image to be modified by the function.\n",
    "  :step: (float) stepsize.\n",
    "  :epsilon: (float) constraint on the distortion.\n",
    "  :eta: (float) tolerance.\n",
    "\n",
    "  Returns\n",
    "  :u: (np array)  modified image.\n",
    "  :counter: (int) number of times the linear minimisation oracle is called.\n",
    "  \"\"\"\n",
    "\n",
    "  u_original = np.squeeze(np.copy(u_ori))\n",
    "  u = np.copy(u_original)\n",
    "  grad = np.squeeze(np.copy(g))\n",
    "  inv_step = 1/step\n",
    "  counter = 0\n",
    "\n",
    "  while 1:\n",
    "    counter += 1\n",
    "    m = grad + inv_step * (u-u_original)\n",
    "    # Linear Minimisation Oracle\n",
    "    v_t = - epsilon * np.sign(m) + u_original\n",
    "    w = v_t - u\n",
    "\n",
    "    V = np.trace(np.inner(m,-w))\n",
    "\n",
    "    if V <= eta: \n",
    "      return u.reshape((1,28,28,1)), counter\n",
    "\n",
    "    k = inv_step * (u_original - u) - grad\n",
    "\n",
    "    numerator = np.trace(np.inner(k, w))\n",
    "    denominator = inv_step * (np.linalg.norm(w))**2 \n",
    "    # y2= np.sqrt(np.cumsum(np.square(x), axis=1))\n",
    "    # denominator = (1/step) * (np.cumsum(np.square(w), axis = 1))**2 \n",
    "    second_term = numerator/denominator\n",
    "    alpha = np.minimum(1, second_term)\n",
    "    u = u + alpha * (v_t-u)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "RTldJUPM8dJV"
   },
   "outputs": [],
   "source": [
    "def setup_images(n_attacks, X_test, y_test, y_test_cat):\n",
    "  \"\"\" \n",
    "  Chooses n_attack images that are classified with an accuracy of 100% by\n",
    "  the model and randomly creates target classes for those images. \n",
    "\n",
    "  Parameters\n",
    "  :n_attacks: (int) Number of images to attack.\n",
    "  :X_test: (np array) test images of MNIST.\n",
    "  :y_test: (np array) test classes of MNIST.\n",
    "  :y_test_cat: (np array) categorical test classes of MNIST.\n",
    "\n",
    "  Returns\n",
    "  :X_test_attack: (np array)  chosen images to attack\n",
    "  :random_labels: (np array) randomly generated target classes.\n",
    "  \"\"\"\n",
    "\n",
    "  chosen_index, random_labels = [],[]\n",
    "\n",
    "  # Randomly choose indices of correctly classified images to attack\n",
    "  while len(chosen_index) < n_attacks:\n",
    "    index = np.random.choice(range(len(y_test_cat)), replace = False)\n",
    "    score = model.evaluate(X_test[index:index+1], \n",
    "                           y_test_cat[index:index+1], \n",
    "                           verbose=0)\n",
    "    if score[1] == 1:\n",
    "      chosen_index.append(index)\n",
    "\n",
    "  # Extract the images to attack\n",
    "  X_test_attack = X_test[chosen_index]\n",
    "  true_labels = y_test[chosen_index]\n",
    "\n",
    "  # Create random labels\n",
    "  while len(random_labels) < n_attacks:\n",
    "    label = random.randint(10)\n",
    "    if label != true_labels[len(random_labels)]:\n",
    "      random_labels.append(label)\n",
    "\n",
    "  return np.array(random_labels), X_test_attack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "CwH3Bg0CbTwM"
   },
   "outputs": [],
   "source": [
    "def launch_attack(random_labels, n_attacks, X_test_attack, func):\n",
    "  \"\"\" \n",
    "  Launches the attack of each image at a time storing the new modified image,\n",
    "  the number of iterations, the distortion and the success boolean.\n",
    "\n",
    "  Parameters\n",
    "  :n_attacks: (int) Number of images to attack.\n",
    "  :random_labels: (np array) randomly generated target classes.\n",
    "  :X_test_attack: (np array)  chosen images to attack\n",
    "  :func: name of the algorithm that will perform the attack.\n",
    "  \"\"\"\n",
    "  x_mod, ifo_calls, lmo_calls, distortions, success = [], [], [], [], []\n",
    "  for i in range(n_attacks):\n",
    "\n",
    "    image_toattack = np.array(X_test_attack[i:i+1])\n",
    "    tgt_lab = random_labels[i]\n",
    "\n",
    "    x_new, ifo, distortion, tgt_reached, lmo = func(image_toattack, tgt_lab)\n",
    "\n",
    "    x_mod.append(x_new)\n",
    "    ifo_calls.append(ifo)\n",
    "    lmo_calls.append(lmo)\n",
    "    distortions.append(distortion)\n",
    "    success.append(tgt_reached)\n",
    "\n",
    "  x_mod = np.array(x_mod)\n",
    "  x_mod = np.squeeze(x_mod, 1)\n",
    "  success_rate = 100* sum(success)/len(success)\n",
    "\n",
    "  print(f\"Success rate: {success_rate}%\") \n",
    "  print(\"Average number of IFO/SFO calls: \", np.mean(np.array(ifo_calls)))\n",
    "  print(\"Average number of LMO calls: \", np.mean(np.array(lmo_calls)))\n",
    "  # print(\"Average number of iterations: \", np.mean(np.array(iterations)))\n",
    "  print(\"Max distorsion: \", np.mean(distortions))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Qb_wwMoHbTwM"
   },
   "source": [
    "# FW momentum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "4rsrPbKP8Iz1"
   },
   "outputs": [],
   "source": [
    "def fw_momentum(image_toattack, tgt_lab):\n",
    "  \"\"\" \n",
    "  Implements the Frank Wolfe with Momentum algorithm to minimize the loss \n",
    "  function of the target class by modifying the original image. \n",
    "\n",
    "  Parameters:\n",
    "  :image_toattack: (np array) image to be attacked.\n",
    "  :tgt_lab: (int) target class.\n",
    "\n",
    "  Returns:\n",
    "  :x: (np array) modified image.\n",
    "  :lmo_calls: (int) number of times the linar minimisation oracle is called.\n",
    "  :distortion: L_{inf} norm of the difference between original image and \n",
    "  modified image.\n",
    "  :target_reached: (boolean) True if the attack has been successful and the\n",
    "  modified image predicts the target class. \n",
    "  \"\"\"\n",
    "    \n",
    "  # Hyperparameters\n",
    "  gamma = 0.45     \n",
    "  beta = 0.5\n",
    "  epsilon = 0.3\n",
    "    \n",
    "  lmo_calls = 0\n",
    "  x_ori = np.copy(image_toattack)\n",
    "  tgt_lab_cat = to_categorical(tgt_lab, 10) \n",
    "  x = np.copy(x_ori)\n",
    "\n",
    "  m = evaluate_grads(x_ori, tgt_lab_cat)\n",
    "\n",
    "\n",
    "  for iteration in range(30):\n",
    "      lmo_calls += 1\n",
    "\n",
    "      # Classify\n",
    "      target_reached, _ = run_classification(x, tgt_lab)\n",
    "\n",
    "      if target_reached: break\n",
    "\n",
    "      # Get gradients\n",
    "      grad = evaluate_grads(x, tgt_lab_cat)\n",
    "\n",
    "      m = beta * m + (1-beta) * grad\n",
    "      v = - epsilon * np.sign(m) + x_ori\n",
    "\n",
    "      d = v - x\n",
    "      x = x + gamma * d\n",
    "\n",
    "  # Calculate distortion\n",
    "  distortion = np.max(np.abs(x - x_ori))\n",
    "    \n",
    "#   if target_reached:\n",
    "#       print('Attack successful')\n",
    "        \n",
    "  return x, lmo_calls * 10, distortion, target_reached, lmo_calls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "W7QK6i_0hDv1"
   },
   "source": [
    "# FCGS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "id": "_i4Hw5-Rz4ok"
   },
   "outputs": [],
   "source": [
    "def fcgs(image_toattack, tgt_lab):\n",
    "  \"\"\" \n",
    "  Implements the Faster first order Conditional Gradient Sliding algorithm to\n",
    "  minimize the loss function of the target class by modifying the original \n",
    "  image. \n",
    "\n",
    "  Parameters:\n",
    "  :image_toattack: (np array) image to be attacked.\n",
    "  :tgt_lab: (int) target class.\n",
    "\n",
    "  Returns:\n",
    "  :x_alpha: (np array) modified image.\n",
    "  :ifo_calls: (int) number times the incremental first order oracle is called.\n",
    "  :distortion: L_{inf} norm of the difference between original image and \n",
    "  modified image.\n",
    "  :target_reached: (boolean) True if the attack has been successful and the\n",
    "  modified image predicts the target class. \n",
    "  :lmo_calls: (int) number of times the linear minimisation oracle is called.\n",
    "  \"\"\"\n",
    "\n",
    "  # Hyperparameters\n",
    "  n = 10\n",
    "  step = 0.03\n",
    "  eta = 18\n",
    "  epsilon = 0.3\n",
    "  ifo_calls = 0\n",
    "  lmo_calls = 0\n",
    "\n",
    "\n",
    "  # Implement S2, not as a set of functions, but as the a set \n",
    "  # of the numbers corresponding to each function.\n",
    "  n_S_2 = int(np.ceil(np.sqrt(n))) \n",
    "  S_2 = np.random.choice(range(n), n_S_2, replace = False)\n",
    "  q = len(S_2)\n",
    "\n",
    "  x_ori = np.copy(image_toattack)\n",
    "  x_set = [x_ori]\n",
    "  x = np.copy(x_ori)\n",
    "  tgt_lab_cat = to_categorical(tgt_lab, 10)\n",
    "\n",
    "\n",
    "  for k in range(30):\n",
    "    \n",
    "    target_reached, _ = run_classification(x, tgt_lab)\n",
    "    if target_reached: break\n",
    "\n",
    "    if (k % q) == 0:\n",
    "      ifo_calls += 10\n",
    "      v = evaluate_grads(x, tgt_lab_cat)\n",
    "    else:\n",
    "      ifo_calls += 2*len(S_2)\n",
    "      v = (1/q) * sum([evaluate_grads_sep(x, tgt_lab_cat, i) - \n",
    "                      evaluate_grads_sep(x_set[-1], tgt_lab_cat, i) + v \n",
    "                      for i in S_2])\n",
    "\n",
    "    x, lmo_call = condg(v, x, step, epsilon, eta)\n",
    "    x = tf.cast(x, tf.float32)\n",
    "    x_set.append(x)\n",
    "    lmo_calls += lmo_call\n",
    "\n",
    "  # Calculate distortion\n",
    "  distortion = np.max(np.abs(x - x_ori))\n",
    "    \n",
    "  # if target_reached:\n",
    "  #     print('Attack successful')\n",
    "\n",
    "  # Randomly choose x_alpha from x_k and return it\n",
    "  ind = np.random.choice(range(len(x_set))) # but this those doesn't make much sense to me (Luis)\n",
    "  x_alpha = x_set[ind]\n",
    "  return x_alpha, ifo_calls, distortion, target_reached, lmo_calls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5hQJGBv6435u"
   },
   "source": [
    "# NCGS-VR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "id": "8lNJzhVy5brV"
   },
   "outputs": [],
   "source": [
    "def ncgs_vr(image_toattack, tgt_lab):  \n",
    "  \"\"\" \n",
    "  Implements the Variance Reduction Non-convex Conditional Gradient sSiding\n",
    "  algorithm to minimize the loss function of the target class by modifying the \n",
    "  original image. \n",
    "\n",
    "  Parameters:\n",
    "  :image_toattack: (np array) image to be attacked.\n",
    "  :tgt_lab: (int) target class.\n",
    "\n",
    "  Returns:\n",
    "  :x_alpha: (np array) modified image.\n",
    "  :ifo_calls: (int) number times the incremental first order oracle is called.\n",
    "  :distortion: L_{inf} norm of the difference between original image and \n",
    "  modified image.\n",
    "  :target_reached: (boolean) True if the attack has been successful and the\n",
    "  modified image predicts the target class.\n",
    "  :lmo_calls: (int) number of times the linear minimisation oracle is called. \n",
    "  \"\"\"\n",
    "\n",
    "  n = 10\n",
    "  tol = 1.1\n",
    "  step = 0.08\n",
    "  epsilon = 0.3\n",
    "  ifo_calls = 0\n",
    "  lmo_calls = 0\n",
    "  m = int(np.ceil(n**(1/3)))   #iteration length\n",
    "  b = int(np.ceil(n**(2/3)))   #mini batch size\n",
    "  T = 100\n",
    "  S = int(T/m)\n",
    "\n",
    "  tgt_lab_cat = to_categorical(tgt_lab, 10)\n",
    "  x_hat = np.copy(image_toattack)\n",
    "  x_s = np.copy(image_toattack)\n",
    "  x_set = [x_s]\n",
    "\n",
    "  for s in range(S):  \n",
    "\n",
    "    target_reached, _ = run_classification(x_s, tgt_lab)    \n",
    "    if target_reached: break\n",
    "\n",
    "    g = (1/n) * sum([evaluate_grads_sep(x_hat, tgt_lab_cat, i) \n",
    "                    for i in range(n)])\n",
    "\n",
    "    for t in range(m):\n",
    "      I_t = np.random.randint(low=0, high=n, size=b)\n",
    "      ifo_calls += 2*len(I_t)\n",
    "      v = g + (1/b) * sum([evaluate_grads_sep(x_s, tgt_lab_cat, j) - \n",
    "               evaluate_grads_sep(x_hat, tgt_lab_cat, j) \n",
    "               for j in I_t])\n",
    "\n",
    "      x_s, lmo_call = condg(v, x_s, step, epsilon, tol)  \n",
    "      x_set.append(x_s)\n",
    "      lmo_calls += lmo_call\n",
    "    \n",
    "    x_hat = x_s    \n",
    "\n",
    "  # Calculate distortion\n",
    "  distortion = np.max(np.abs(x_s - image_toattack))\n",
    "    \n",
    "  # if target_reached:\n",
    "  #     print('Attack successful')\n",
    "\n",
    "  ind = np.random.choice(range(len(x_set)))\n",
    "  # x_alpha = x_set[ind]\n",
    "  x_alpha = x_set[-1]\n",
    "  return x_alpha, ifo_calls, distortion, target_reached, lmo_calls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "osNxhknF2kxX"
   },
   "source": [
    "# **SFW**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "executionInfo": {
     "elapsed": 509,
     "status": "ok",
     "timestamp": 1654688366177,
     "user": {
      "displayName": "Stefanija Galevska",
      "userId": "16800269765637290243"
     },
     "user_tz": -120
    },
    "id": "Lg1WZck0bTwO"
   },
   "outputs": [],
   "source": [
    "def sfw(image_toattack, tgt_lab):\n",
    "  \"\"\" \n",
    "  Implements the Stochastic Frank Wolfe algorithm to minimize the loss function\n",
    "  of the target class by modifying the original image. \n",
    "\n",
    "  Parameters:\n",
    "  :image_toattack: (np array) image to be attacked.\n",
    "  :tgt_lab: (int) target class.\n",
    "\n",
    "  Returns:\n",
    "  :x_alpha: (np array) modified image.\n",
    "  :lmo_calls: (int) number of times the linear minimisation oracle is called.\n",
    "  :distortion: L_{inf} norm of the difference between original image and \n",
    "  modified image.\n",
    "  :target_reached: (boolean) True if the attack has been successful and the\n",
    "  modified image predicts the target class. \n",
    "  :ifo_calls: (int) number times the incremental first order oracle is called.\n",
    "  \"\"\"\n",
    "  # Hyperparameters\n",
    "  epsilon = 0.3    \n",
    "  step = 0.4\n",
    "  batch_size = 400 # Let's take fixed batch size\n",
    "  T = 400\n",
    "\n",
    "  lmo_calls = 0\n",
    "  sfo_calls = 0\n",
    "\n",
    "  tgt_lab_cat = to_categorical(tgt_lab, 10)\n",
    "  x_ori = np.copy(image_toattack)\n",
    "  x = np.copy(x_ori)\n",
    "  x_set = [x_ori]\n",
    "  all_tuples = [(i,j) for i in range(28) for j in range(28)]\n",
    "\n",
    "  for t in range(T):\n",
    "    lmo_calls += 1\n",
    "    sfo_calls += batch_size\n",
    "    target_reached, _ = run_classification(x, tgt_lab)\n",
    "    if target_reached: break\n",
    "\n",
    "    zetas_ind = np.random.choice(range(28*28), batch_size, replace = False)\n",
    "    zetas = [all_tuples[i] for i in zetas_ind]\n",
    "\n",
    "    m = np.zeros((1,28,28,1))\n",
    "    g = evaluate_grads(x, tgt_lab_cat)\n",
    "\n",
    "    for index,tupla in enumerate(zetas):\n",
    "      i, j = tupla\n",
    "      m[0,i,j,0] = g[0,i,j,0]\n",
    "\n",
    "  \n",
    "    m = m * (-1/batch_size)\n",
    "    v = epsilon * np.sign(m) + x_ori\n",
    "\n",
    "    d = v - x\n",
    "    x = x + step * d\n",
    "    x = tf.cast(x, tf.float32)\n",
    "    x_set.append(x) \n",
    "\n",
    "  # Calculate distortion\n",
    "  distortion = np.max(np.abs(x - x_ori))\n",
    "      \n",
    "  # if target_reached:\n",
    "  #   print('Attack successful')\n",
    "\n",
    "  ind = np.random.choice(range(len(x_set)))\n",
    "  x_alpha = x_set[ind]\n",
    "  return x_alpha, sfo_calls, distortion, target_reached, lmo_calls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b7VroYA7nJ_k"
   },
   "source": [
    "# To run the attack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "background_save": true
    },
    "id": "spZHrDwGbTwO"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: total: 39.5 s\n",
      "Wall time: 36.8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "n_attacks = 1000\n",
    "random_labels, X_test_attack = setup_images(n_attacks = n_attacks, \n",
    "                                            X_test = X_test, y_test = y_test,\n",
    "                                            y_test_cat = y_test_cat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 415
    },
    "executionInfo": {
     "elapsed": 6,
     "status": "error",
     "timestamp": 1654688692265,
     "user": {
      "displayName": "Stefanija Galevska",
      "userId": "16800269765637290243"
     },
     "user_tz": -120
    },
    "id": "d89WLPbBbTwP",
    "outputId": "6e1a76ef-ef31-4184-98be-510b14e5dc2c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success rate: 99.7%\n",
      "Average number of IFO/SFO calls:  47.76\n",
      "Average number of LMO calls:  4.776\n",
      "Max distorsion:  0.24498962\n",
      "CPU times: total: 3min 50s\n",
      "Wall time: 3min 10s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "launch_attack(random_labels, n_attacks, X_test_attack, fw_momentum)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success rate: 66.8%\n",
      "Average number of IFO/SFO calls:  170.912\n",
      "Average number of LMO calls:  91.595\n",
      "Max distorsion:  0.29316637\n",
      "CPU times: total: 41min 18s\n",
      "Wall time: 27min 51s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "launch_attack(random_labels, n_attacks, X_test_attack, fcgs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success rate: 73.8%\n",
      "Average number of IFO/SFO calls:  601.47\n",
      "Average number of LMO calls:  416.71\n",
      "Max distorsion:  0.29096627\n",
      "CPU times: total: 3h 31min\n",
      "Wall time: 2h 5min 43s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "launch_attack(random_labels, n_attacks, X_test_attack, ncgs_vr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success rate: 57.0%\n",
      "Average number of IFO/SFO calls:  74544.4\n",
      "Average number of LMO calls:  186.361\n",
      "Max distorsion:  0.28289142\n",
      "CPU times: total: 5h 36min 16s\n",
      "Wall time: 4h 59min 38s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "launch_attack(random_labels, n_attacks, X_test_attack, sfw)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Optimization_project.ipynb",
   "provenance": [
    {
     "file_id": "1plHRJXX-84jkUX-cep8fNrwKS8JKVrmp",
     "timestamp": 1653762826892
    }
   ],
   "version": ""
  },
  "interpreter": {
   "hash": "323d1892a3217eb0f4db6041660d5c1546636e2d429706ca4ab8fde40b13a00b"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}