pytorch_ipynb/gnn/gnn-basic-1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deep Learning Models -- A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.\n",
    "- Author: Sebastian Raschka\n",
    "- GitHub Repository: https://github.com/rasbt/deeplearning-models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sebastian Raschka \n",
      "\n",
      "CPython 3.7.3\n",
      "IPython 7.6.1\n",
      "\n",
      "torch 1.2.0\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -a 'Sebastian Raschka' -v -p torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Runs on CPU or GPU (if available)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Most Basic Graph Neural Network with Gaussian Filter on MNIST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implementing a very basic graph neural network (GNN) using a Gaussian filter. \n",
    "\n",
    "Here, the 28x28 image of a digit in MNIST represents the graph, where each pixel (i.e., cell in the grid) represents a particular node. The feature of that node is simply the pixel intensity in range [0, 1]. \n",
    "\n",
    "Here, the adjacency matrix of the pixels is basically just determined by their neighborhood pixels. Using a Gaussian filter, we connect pixels based on their Euclidean distance in the grid.\n",
    "\n",
    "Using this adjacency matrix $A$, we can compute the output of a layer as \n",
    "\n",
    "$$X^{(l+1)}=A X^{(l)} W^{(l)}.$$\n",
    "\n",
    "Here, $A$ is the $N \\times N$ adjacency matrix, and $X$ is the $N \\times C$ feature matrix (a  2D coordinate array, where $N$ is the total number of pixels -- $28 \\times 28 = 784$ in MNIST). $W$ is the weight matrix of shape $N \\times P$, where $P$ would represent the number of classes if we have only a single hidden layer.\n",
    "\n",
    "\n",
    "\n",
    "- Inspired by and based on Boris Knyazev's tutorial at https://medium.com/@BorisAKnyazev/tutorial-on-graph-neural-networks-for-computer-vision-and-beyond-part-1-3d9fada3b80d."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import numpy as np\n",
    "from scipy.spatial.distance import cdist\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torchvision import datasets\n",
    "from torchvision import transforms\n",
    "from torch.utils.data import DataLoader\n",
    "from torch.utils.data.dataset import Subset\n",
    "\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "    torch.backends.cudnn.deterministic = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Settings and Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "##########################\n",
    "### SETTINGS\n",
    "##########################\n",
    "\n",
    "# Device\n",
    "DEVICE = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "# Hyperparameters\n",
    "RANDOM_SEED = 1\n",
    "LEARNING_RATE = 0.05\n",
    "NUM_EPOCHS = 20\n",
    "BATCH_SIZE = 128\n",
    "IMG_SIZE = 28\n",
    "\n",
    "# Architecture\n",
    "NUM_CLASSES = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MNIST Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Image batch dimensions: torch.Size([128, 1, 28, 28])\n",
      "Image label dimensions: torch.Size([128])\n"
     ]
    }
   ],
   "source": [
    "train_indices = torch.arange(0, 59000)\n",
    "valid_indices = torch.arange(59000, 60000)\n",
    "\n",
    "custom_transform = transforms.Compose([transforms.ToTensor()])\n",
    "\n",
    "\n",
    "train_and_valid = datasets.MNIST(root='data', \n",
    "                                 train=True, \n",
    "                                 transform=custom_transform,\n",
    "                                 download=True)\n",
    "\n",
    "test_dataset = datasets.MNIST(root='data', \n",
    "                              train=False, \n",
    "                              transform=custom_transform,\n",
    "                              download=True)\n",
    "\n",
    "train_dataset = Subset(train_and_valid, train_indices)\n",
    "valid_dataset = Subset(train_and_valid, valid_indices)\n",
    "\n",
    "train_loader = DataLoader(dataset=train_dataset, \n",
    "                          batch_size=BATCH_SIZE,\n",
    "                          num_workers=4,\n",
    "                          shuffle=True)\n",
    "\n",
    "valid_loader = DataLoader(dataset=valid_dataset, \n",
    "                          batch_size=BATCH_SIZE,\n",
    "                          num_workers=4,\n",
    "                          shuffle=False)\n",
    "\n",
    "test_loader = DataLoader(dataset=test_dataset, \n",
    "                         batch_size=BATCH_SIZE,\n",
    "                         num_workers=4,\n",
    "                         shuffle=False)\n",
    "\n",
    "# Checking the dataset\n",
    "for images, labels in train_loader:  \n",
    "    print('Image batch dimensions:', images.shape)\n",
    "    print('Image label dimensions:', labels.shape)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def precompute_adjacency_matrix(img_size):\n",
    "    col, row = np.meshgrid(np.arange(img_size), np.arange(img_size))\n",
    "    \n",
    "    # N = img_size^2\n",
    "    # construct 2D coordinate array (shape N x 2) and normalize\n",
    "    # in range [0, 1]\n",
    "    coord = np.stack((col, row), axis=2).reshape(-1, 2) / img_size\n",
    "\n",
    "    # compute pairwise distance matrix (N x N)\n",
    "    dist = cdist(coord, coord, metric='euclidean')\n",
    "    \n",
    "    # Apply Gaussian filter\n",
    "    sigma = 0.05 * np.pi\n",
    "    A = np.exp(- dist / sigma ** 2)\n",
    "    A[A < 0.01] = 0\n",
    "    A = torch.from_numpy(A).float()\n",
    "\n",
    "    # Normalization as per (Kipf & Welling, ICLR 2017)\n",
    "    D = A.sum(1)  # nodes degree (N,)\n",
    "    D_hat = (D + 1e-5) ** (-0.5)\n",
    "    A_hat = D_hat.view(-1, 1) * A * D_hat.view(1, -1)  # N,N\n",
    "    \n",
    "    return A_hat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQEAAAD8CAYAAAB3lxGOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAa90lEQVR4nO2dbYxc53mer2e4/IicKjIZWSW1WsvqbiRZkvkRU5RhIWilqLDdwAkQ0x8IRQYQoB9xCxsJkMror/5okfyJHQOFUMdqbNlGbEt2GkM17CqSgiJBQ0nWklyJS2bXqveDS0uyJcuu1V1qOE9/zDvD9fK87zk7n2dm7gtYcOd8zTlazT3Pe5/nvLe5O0KI0aXS7xMQQvQXiYAQI45EQIgRRyIgxIgjERBixJEICDHidEUEzOx9ZnbWzObN7IFuvIcQojNYp/sEzGwL8E/APcAy8AzwMXc/3dE3EkJ0hG5UArcD8+7+ortfAL4K/HYX3kcI0QHGunDMa4Glda+XgUMbNzKz+4H7Abb90pZf3/7/rowe0LZtwy9c6PBpCjFa/IzXfuTuV29c3g0RsIxll4053P1zwOcArrSdfuTZDzB3qAq1i5fvXTXGrh+nurB0+TqAyhYq27ZSW12NnJGB2qPFiPO3/uhC1vJuDAeWgevWvR4HVvJ2mjtUZep4RJPcqb30CmNvvy57fe0itQtvQmVLdH8sS5uEEN0QgWeAKTN7h5ltAz4KfCt3r9pF5g6uMfXM9uzVq6tUF5eTQgBICITYJB0XAXevAv8W+C4wC3zd3V8oun9TCLI+zO5UF5fh9tuydw5CUNmxI3ZyRU9DiJGhK30C7v5td/81d/8X7v6fNrt/3tCAZ55vfWgAqgiEWEc5OwbXDw0SFUHe0CBZEUgIhADKKgKBuUNV7jr50+yV7tRe+TFbpm7IXt+oCGJICIQASiICtm1b9geydpEnb3sLy9+4JXO/2htvcHHuRcbGr80+cO1ivZKQWShElFKIgF+4wNjEeLR8H//dF1h85Lboh7l6boXKvndmH1xmoRBJSiECANWFJSrXXB39Zp74yGkWvxb5oLtTOznL2LV7stfLLBQiSmlEAOpCMDYxHh0aTByeiVcE7lRXzieFAGQWCrGRUokA1IWAg7e2XhH8+NXWKwIJgRhBSicCADw9U6wiyKC2ulqoIpAQCFGnnCLApaFB5YorMtcXGRrILBQin9KKANSFwK7959H1Ex85ne4jODkbP3heHwGoIhAjQXlEIFKeN/sAEn0E81/en70+lPbJoUFlC7Y9+6ElDQ3EKFAaEahs2xrvA1g+R2XvzdEP5OS9J5j/0r7sA7tz8Uc/TgqBv1mVRyBGltKIQHNCkMiHsXbiNGN7dke/8SePTNcrggx8bU1moRARSiMCQO6HsXpuhbE9u6OGXlMIIkJRXTlfrygS750cGggxhJRLBABqF9NDg3MrVHbtbHloUDt1pvWhAagiEENH+USA/KFBoyLIHRokKoK8oYHMQjEqlFIEgGIVgcxCIdqmvCJAqAgajwNnrT9xOv6BDBVBbM5CX1urVxSpiUkS7y2PQAwL5RGBvG/WxDh9bM/uaPk+d3CNuYcPRI9fXVym8q6b0qcWGxoIMQTkioCZ/Tcze9nMnl+3bKeZPW5mc+Hft4blZmafDRmEp8zsQOEzSZXYBYYGW351V3T/qWPTzH0x+/Yh7tRmzsosFCNLkUrgC8D7Nix7AHjC3aeAJ8JrgPcDU+HnfuDBTZ1NQgjaNQunjj4XrwhkFooRJlcE3P1/Aa9uWPzbwBfD718Efmfd8oe9zj8CV5nZ7k2dUWqsXcQsfNdNLVcEMgvFKNKqJ3CNu58HCP++LSzPyiHMnADQzO43s2fN7Nk3WcvaIPONczsLT84WqwgyUGehGEU6bQwWyiGEehahu7/b3d+9lYwyO8cjAHKHBrHyvcjQIGoWqrNQDBmtisBLjTI//PtyWN5SDmGUTpiFEaaOTac7C2UWihGhVRH4FnAs/H4M+Jt1y4+GuwR3AK83hg0tk2cWJu7lN/sAEn0Ei4/cFq8Izq2khwa1i5qzUAw8RW4R/hXwv4EbzWzZzO4D/gS4x8zmgHvCa4BvAy8C88BfAH/QkbPMKbFt61hcCBaWqNx2Y3zOwg8/HzcLIW0WguYsFAOPeQnGsFfaTj9kd+dvaBYXhMYHseEXbGDs2j1UV85H9597+ABTR5+Lvu/Ynt1Uz0VGNjnvnTxvIXrE3/qj33P3d29cXp6OwSLILBSi4wyWCECuECSHBuosFOIyBk8EICkEvhZ6DsraWShEyRhMEYDczsK8iiBlFna1sxBUEYhSMbgi0CCvIohQO3WmXhFk7lyvCGIBJ83OwtRjyCl010CUiMEXgTyzMBFN3ugDiJXvE4dn0jMULSyl5yzMm85ciBIw+CIAXTULJ+89kewj0JyFYtAZDhEAmYVCtMjwiADILBSiBYZLBBq0WBE0zcIuPoasiUlE2RhOEShxZ6GvrWloIErFcIoAlLezEDQ0EKVieEUAkt+seX0E1XMrjE2MR487dfS56HTmjYogSpE+AiF6xHCLQB6N+QBSjyEnAk7mDq5FQ1Bxp7JjR24suioC0W9GRwS6MWch9RDUmFlYW12VWShKz+iIgMxCITIZHREAmYVCZDBaIgB97yzUxCSibIyeCEBfOws1MYkoG0UmGr3OzJ4ys1kze8HMPhGWdz6PsNf0obNQkWeibBSpBKrAH7n7zcAdwMfN7J10K4+wl+SZhTnTmTduA2bR7CPI2j9MZ167M5J7EMzC5HTmQnSIIlmE5939ufD7z4BZ6tFi3csj7CU536ypoQFAZdfOeB/BoSpzX9gb3/cfTrY+nTmoIhAdYVOegJldD+wHjtNmHmFuFmEv6ZZZWLsos1CUnsIiYGa/DHwD+KS7/zS1acayy/5vzc0i7DUyC8WIUkgEzGwrdQH4irt/MyzuTR5hr+m2WditiUkkBKJFitwdMOAhYNbd/2zdqt7lEfaSIhVBhL5OTCIhEC1SpBJ4L3AvcJeZnQg/H6DXeYS9JlER2PaI609vJiaREIhOMlhZhL2m3ezDWHYhMP/l/UwemY6+b+VdN1E7ORt978q2rZcefhKiAMORRdhrcoYGeX0EtTv3JZ8+TM1HUDs5m8w1qK2ups1CIQoiEShCXokd+TBW/v5E8jHkuYNrcbMQqC4uJ/sIIGEWgoYGohASgSLILBRDjERgM8gsFEOIRGAzdKCzsFsTk4A6C0VrSAQ2S5udhV2bmESdhaJFJAKtMqiRZxICsQGJQKu0axbGSnvqFcH8lyKPGcssFB1GItAuiYogNZ15sw8g8o0/eWSaxUduix67EaueffB0D4OEQKxHItAuiQ9U3nTm1YWlpFk4cXgm3UeQMgsDMgtFHhKBTiCzUAwwEoFOIrNQDCASgU7Sx4lJZBaKVpEIdIM+zGKszkLRKhKBblDiyDOQWSh+EYlAt8gZGlS2bY2ubpqFEaaOTbP49Vuj71ubOZt8DNnfrMbPDVQRjBgSgW6Tun2YiCZv9gFEvvEnDs/UY9FjFcHCUm4susxCARKB7tNFs3Dy3hNxsxBkFopCFJlodIeZPW1mJ0MM2X8My99hZsdDDNnXzGxbWL49vJ4P66/v7iUMCDILRUkpUgmsAXe5+15gH/C+MIvwnwKfDjFkrwH3he3vA15z90ng02E7IbNQlJQiMWTu7v83vNwafhy4C3g0LN8YQ9aIJ3sUuDtMWy5yhECdhaIfFA0f2WJmJ6gHjDwOfB/4ibs3bOb1UWPNGLKw/nUgbnWPGt2KPFNnoWiRQiLg7hfdfR/1NKHbgZuzNgv/FoohK1UWYa9RZ6EoEZu6O+DuPwH+jnpE+VVm1nhofn3UWDOGLKz/FeDVjGOVK4uwRDQrggi1U2cYmxiP7OyXYtEjx66uJAKhIjkK648vIRguitwduNrMrgq//xLwm9TjyZ8CPhQ22xhD1ogn+xDwpJch4aSspMzCVB9B6AOIle/J6czDBzlpFub1EYihoUglsBt4ysxOAc8Aj7v7Y8C/B/7QzOapj/kfCts/BOwKy/8QeKDzpz1EyCwUfUYxZGWh3cizlfPR/ecePsDUsens9WaM7dkdj0yrbMG2juUOUUT5UQxZ2RlUsxBUEQw4EoGyoc5C0WMkAmVDnYWix0gEyojMQtFDJAJlJa+zMCcWPbr/+j6CrP3d6xVFYj4Cahfr06nHzlsMFBKBMpPzgUpVBEDSLJw7VGXuC3uj+9ZeeiUZi1678KYqgiFBIjAIdMMsrF1MmoW11VU9azAiSAQGgRKbhc1Y9th5i9IjERgUymoWgszCAUciMEgM8mPIorRIBAYNdRaKDiMRGFTaNQszd65XBPNfzhaCop2FyduHEoLSIREYVPLMwpw+grG3Xxf9sDZj0WNDg3Mr2P5bou9dW11VH8EAIREYZHK+WZNDg4UlKrt2Rvef+PDzyenM/cRp9REMCRKBQafPZqGeNRh8JALDQB/NQj1rMPhIBIaJPjyGrFmMBx+JwDChzkLRAhKBYUOdhWKTFBaBEEAybWaPhdfKIiwrRczCCE2zMHLcqaPP1W8fRtZXV84nbx9CTh+B6DmbqQQ+QX2q8QbKIiwzOWZhZceOZEVg+2+J3z48PJMUAp9+IekRNGPZY6gi6ClFY8jGgX8DfD68NpRFOBhE/tPXVlfrv0Q+jD79QtwspC4EUY8A0mZh49RkFpaCopXAZ4A/Bmrh9S7azCIc6RiyXlLk9mEERZ6NBkUSiH4LeNndv7d+ccamm8oiVAxZj0l4BLY9MtUYmsV4FChSCbwX+KCZ/QD4KvVhwGdoM4tQ9JhudRaiWYwHnVwRcPdPufu4u18PfJR6tuDvoSzCwUOdhSKDdvoElEU4qHS7s7BbE5NICLqCsgjFZVR27Lh09yAD238LPv1CZKWx+PVbmTg8Ez125ZqrqS4sRd58SzoePZXZKJIoi1BcTuL2YaqPoNkHEPnGnzg8E52YpLa62oxVz94gHcmuiqDzSARGmcQHKq+PIO9Zg8kj07l9BDILy4FEYNRp0yxU5NngIxEQdQZ1FmMJQdtIBEQddRaOLBIB8Yuos3DkkAiIX6QDnYWxR4WbQhBLQ5ZZ2BckAuJy8h5D3rY1urp6boXKNVdH10/9/kmmjkeGFsEsTL23v1mNrwdVBC0gERBxUrcPE/fym30AkTTkuYNrSbMQs9w+ApmFnUMiIOIMauSZhGBTSAREPn2YxVhmYe+QCIh8SjyLMcgsbBeJgCiGOguHFomA2BzqLBw6JAJic8gsHDokAqI1UhVBIha9dupM/AMZKoKpZ7K/0X1trRmrnn3wdCS7PIJsJAKiNYp8sybG6SmzMNlHAFQXl+NmYSA6NBCXIREQrVPWyDOZhZtCIiDao1uzGMss7BlFE4h+YGYzZnbCzJ4Ny3aa2eMhi/BxM3trWG5m9tmQRXjKzCKZ1mJokFk40GymEvhX7r5v3USFDwBPhCzCJ7g0q/D7ganwcz/wYKdOVpQcdRYOJO0MB9ZnDm7MInzY6/wj9ZCSSMytGCrUWTiQFBUBB/6nmX3PzO4Py65x9/MA4d+3heXNLMLA+pzCJsoiHFI6YRZGmDo2zfyX9kXfV2ZhaxQVgfe6+wHqpf7Hzew3Etsqi3DUyTMLE/fym30AkW/8ySPT9Vj0WEVwbiU9NGjEsm/yvIeZQiLg7ivh35eBvwZuB15qlPnh35fD5s0swsD6nEIxKuSU2MmKYGEpaRZOfPj5uFkIabMQqF14Ux7BOoqkEr/FzP5Z43fgXwPP84uZgxuzCI+GuwR3AK83hg1iBJFZWHqKVALXAH9vZieBp4H/4e7fAf4EuMfM5oB7wmuAbwMvAvPAXwB/0PGzFoODzMLSoyxC0RsSGYK2fXvdtItkEI5du4fqyvns/c2Y++J+po4+F33fsT27qZ6LjEgbAhTLPxyi7ENlEYr+MsidhUOORED0jkHtLISh9ggkAqL35FUEEZpmYebO9Ypg8ZHboseurpxPP4acYojNQomA6D15ZmFqOvPQBxAr35ux6LGhwcISlb03J9971IYGEgHRH7r4GPLkvSeSfQS1U2c0NFiHRED0D5mFpUAiIPqLzMK+IxEQ5aDEnYXDPjGJRECUgxJ3FjZj2WPnPeBIBER5KOuchTDUQwOJgCgXiW/WvD6C6rkVxibGo8dNTWfeqAiiFOkjGFAkAmKwaMwHkHoMee/N0W/muYNr9T6CLNyp7NiRG4s+bBWBRECUl8gHqra6Wv8lZhaenI2bhcDkkemoWVhbXR05s1AiIMqLzMKeIBEQ5UZmYdeRCIjy0+fOwmGfmEQiIAaDPnYWDvssxhIBMVj0obNw2CPPJAJisMgzC3OmM2/cBsyi2UeQtX+Yzrx2ZyT3IJiFyenMS0rRLMKrzOxRMztjZrNm9h5lEYq+kfPNmhoaAFR27Yz3ERyqMveFvfF9/+Fk69OZQykrgqKVwJ8D33H3m4C9wCzKIhT9pFtmYe3iyJmFRXIHrgR+A3gIwN0vuPtPUBah6DcyCztCkUrgBuAV4C/NbNrMPh9CSJRFKMpBt83Cbk1MUhIhKCICY8AB4EF33w/8nEulfxbKIhS9pUhFEKGvE5OURAiKiMAysOzux8PrR6mLgrIIRblIVAS2PeL6o8izXBFw9x8CS2Z2Y1h0N3AaZRGKstEJszDC1NHnkk8fVlfOp2cxhtLePix6d+DfAV8xs1PAPuA/oyxCUUZyhgZ5fQS1O/clnz5MzUdQOzmbzDWora6mzcI+oSxCMZykMgRz8geT2YfA3MMHmDo2Hc1GzMs+tK1j8QlSuph9qCxCMVrILCyMREAMNzILc5EIiOGmA2ZhtyYmgXJ0FkoExPDTZmdh1yYmKUlnoURAjA6DGnnWZSGQCIjRoV2zMFbaU68I5r8Uecy45GahRECMHomKIDWdebMPIPKNP3lkmsVHboseuxGrnn3wdA9DN4VAIiBGj8QHKm868+rCUtIsnDg8Ex8aQNosDPTaLJQIiNFEZmETiYAYbWQWSgTEiNPHiUnKYhZKBISAvsxiXJbOQomAEFDqyDPorlkoERCiQc7QoLJta3R10yyMMHVsmsWv3xp939rM2eRjyP5mNX5u0FZFIBEQYiOp24eJaPJmH0DkG3/i8Ex9YpJYRbCwlBuLnpyYpEUhkAgIsZEumoWT956Im4WQaxYmcw1aFAKJgBAxRsQslAgIEaPfZmGP5iwsEj5yo5mdWPfzUzP7pGLIxEiQIwRd7Sw8dab1oQEUrgiKzDZ81t33ufs+4NeBN4C/RjFkYlToVuRZhzoL2zULNzscuBv4vrsvoBgyMUqUuLOwXbNwsyLwUeCvwu9txZAJMUxEZw8O1E6dYWxiPLKzX4pFjxy7upKI7ojMmrz++MmU5vTelzCzbcAHgUfyNs06jYzjKYtQDCYpszDVRxD6AGJm4dzBtWQfAWZU9r0z+d7RoUFCKDZTCbwfeM7dXwqv24ohUxahGFi6aBZO3nsiOUNR7eRse2ZhBpsRgY9xaSgAiiETo0wXzcLJI9PpzsJ2zMIMComAmV1BPWrsm+sWK4ZMjDbtmoXvuqn1iuDHr3asIlAMmRDt0sXIs/kv72fyyHT0ffMiz9a/t2LIhOgWXewsLDI0SJqF5A8NJAJCdIJ+m4XjkbvwBYYGEgEhOkWeWZgTix7dP5iFd838PHt/d6rL59gydUP2eTWmM48gERCik+R4bJVtW9PfyntvjgrJk3uvZPFrkdIf8HM/jE9MkjqnTe8hhMinxVyD2onT8duHtYtMHJ6JBpzU3niD6uLypoVAIiBEN+iAWRgz9JpCEBsaLC7D7dlCkYVEQIhukSMEqaFB9dwKlV07o/tPfOR0fGjgDs88X7gikAgI0U3aiTxLdRauHxokKoIiQiAREKLbFJnFOFURJMzCvIqg9tIruUIgERCiV7RjFiZahCcOz7D8jVuix64uLjN2/UT0tCQCQvSKPLMw1UcQ+gAqV1yRuX78d19I9xH8YDF6WhIBIXpJzkw/qaHBxbkXqVwd7yx8cu+VTB0f2/QpSQSE6DXtmIULS/UZiiJm4dzBtfoMRZuYU0AiIEQ/aMcsXFiCg7dGhWTuUHVTFYFEQIh+0mJFwNMzxSqCAkgEhOgn7XQWhqFBrLOw6NBAIiBEv2mns3Bhico1V7c1NJAICFEGEtOCN4cGEaoLS8lY87mDa/XbhxFKMb2Ymf0MONvv8+givwr8qN8n0SWG+dpguK7v7e5+9caFm7+p2B3OZs19NiyY2bPDen3DfG0w/NcHGg4IMfJIBIQYccoiAp/r9wl0mWG+vmG+Nhj+6yuHMSiE6B9lqQSEEH1CIiDEiNN3ETCz95nZWTObN7MH+n0+m8XMrjOzp8xs1sxeMLNPhOU7zexxM5sL/741LDcz+2y43lNmdqC/V5CPmW0xs2kzeyy8foeZHQ/X9rUQW4+ZbQ+v58P66/t53kUws6vM7FEzOxP+hu8Zpr9dEfoqAma2Bfgv1GPP3wl8zMziE6uXkyrwR+5+M3AH8PFwDQ8AT7j7FPBEeA31a50KP/cDD/b+lDfNJ4DZda//FPh0uLbXgPvC8vuA19x9Evh02K7s/DnwHXe/CdhL/TqH6W+Xj7v37Qd4D/Ddda8/BXyqn+fUgWv6G+opzWeB3WHZbuoNUQD/FfjYuu2b25XxBxin/kG4C3gMMOoddGMb/4bAd4H3hN/HwnbW72tIXNuVwP/ZeI7D8rcr+tPv4cC1wNK618th2UASyt/9wHHgGnc/DxD+fVvYbNCu+TPAHwO18HoX8BN3r4bX68+/eW1h/eth+7JyA/AK8JdhuPN5M3sLw/O3K0S/RSDr0aeBvGdpZr8MfAP4pLv/NLVpxrJSXrOZ/Rbwsrt/b/3ijE29wLoyMgYcAB509/3Az7lU+mcxaNdXiH6LwDKw/vGncSAStl5ezGwrdQH4irt/Myx+ycx2h/W7gZfD8kG65vcCHzSzHwBfpT4k+AxwlZk1njtZf/7NawvrfwV4tZcnvEmWgWV3Px5eP0pdFIbhb1eYfovAM8BUcJu3AR8FvtXnc9oUZmbAQ8Csu//ZulXfAo6F349R9woay48Gp/kO4PVG6Vk23P1T7j7u7tdT/9s86e6/BzwFfChstvHaGtf8obB9ab8p3f2HwJKZ3RgW3Q2cZgj+dpui36YE8AHgn4DvA/+h3+fTwvnfSb0kPAWcCD8foD4WfgKYC//uDNsb9Tsi3wdmgHf3+xoKXue/BB4Lv98APA3MA48A28PyHeH1fFh/Q7/Pu8B17QOeDX+//w68ddj+dnk/ahsWYsTp93BACNFnJAJCjDgSASFGHImAECOORECIEUciIMSIIxEQYsT5/xW3l0L3C9cJAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(precompute_adjacency_matrix(28));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "##########################\n",
    "### MODEL\n",
    "##########################\n",
    "\n",
    "        \n",
    "\n",
    "class GraphNet(nn.Module):\n",
    "    def __init__(self, img_size=28, num_classes=10):\n",
    "        super(GraphNet, self).__init__()\n",
    "        \n",
    "        n_rows = img_size**2\n",
    "        self.fc = nn.Linear(n_rows, num_classes, bias=False)\n",
    "\n",
    "        A = precompute_adjacency_matrix(img_size)\n",
    "        self.register_buffer('A', A)\n",
    "\n",
    "        \n",
    "\n",
    "    def forward(self, x):\n",
    "        \n",
    "        B = x.size(0) # Batch size\n",
    "\n",
    "        ### Reshape Adjacency Matrix\n",
    "        # [N, N] Adj. matrix -> [1, N, N] Adj tensor where N = HxW\n",
    "        A_tensor = self.A.unsqueeze(0)\n",
    "        # [1, N, N] Adj tensor -> [B, N, N] tensor\n",
    "        A_tensor = self.A.expand(B, -1, -1)\n",
    "        \n",
    "        ### Reshape inputs\n",
    "        # [B, C, H, W] => [B, H*W, 1]\n",
    "        x_reshape = x.view(B, -1, 1)\n",
    "        \n",
    "        # bmm = batch matrix product to sum the neighbor features\n",
    "        # Input: [B, N, N] x [B, N, 1]\n",
    "        # Output: [B, N]\n",
    "        avg_neighbor_features = (torch.bmm(A_tensor, x_reshape).view(B, -1))\n",
    "        \n",
    "        logits = self.fc(avg_neighbor_features)\n",
    "        probas = F.softmax(logits, dim=1)\n",
    "        return logits, probas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(RANDOM_SEED)\n",
    "model = GraphNet(img_size=IMG_SIZE, num_classes=NUM_CLASSES)\n",
    "\n",
    "model = model.to(DEVICE)\n",
    "\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=LEARNING_RATE)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 001/020 | Batch 000/461 | Cost: 2.2677\n",
      "Epoch: 001/020 | Batch 150/461 | Cost: 0.8999\n",
      "Epoch: 001/020 | Batch 300/461 | Cost: 0.6701\n",
      "Epoch: 001/020 | Batch 450/461 | Cost: 0.4905\n",
      "Epoch: 001/020\n",
      "Train ACC: 87.02 | Validation ACC: 92.40\n",
      "Time elapsed: 0.08 min\n",
      "Epoch: 002/020 | Batch 000/461 | Cost: 0.5868\n",
      "Epoch: 002/020 | Batch 150/461 | Cost: 0.4526\n",
      "Epoch: 002/020 | Batch 300/461 | Cost: 0.4192\n",
      "Epoch: 002/020 | Batch 450/461 | Cost: 0.3647\n",
      "Epoch: 002/020\n",
      "Train ACC: 88.25 | Validation ACC: 92.60\n",
      "Time elapsed: 0.14 min\n",
      "Epoch: 003/020 | Batch 000/461 | Cost: 0.4316\n",
      "Epoch: 003/020 | Batch 150/461 | Cost: 0.4165\n",
      "Epoch: 003/020 | Batch 300/461 | Cost: 0.4130\n",
      "Epoch: 003/020 | Batch 450/461 | Cost: 0.3991\n",
      "Epoch: 003/020\n",
      "Train ACC: 88.80 | Validation ACC: 93.30\n",
      "Time elapsed: 0.22 min\n",
      "Epoch: 004/020 | Batch 000/461 | Cost: 0.3537\n",
      "Epoch: 004/020 | Batch 150/461 | Cost: 0.3460\n",
      "Epoch: 004/020 | Batch 300/461 | Cost: 0.4011\n",
      "Epoch: 004/020 | Batch 450/461 | Cost: 0.4666\n",
      "Epoch: 004/020\n",
      "Train ACC: 89.34 | Validation ACC: 93.40\n",
      "Time elapsed: 0.29 min\n",
      "Epoch: 005/020 | Batch 000/461 | Cost: 0.4523\n",
      "Epoch: 005/020 | Batch 150/461 | Cost: 0.4006\n",
      "Epoch: 005/020 | Batch 300/461 | Cost: 0.4396\n",
      "Epoch: 005/020 | Batch 450/461 | Cost: 0.4509\n",
      "Epoch: 005/020\n",
      "Train ACC: 89.65 | Validation ACC: 93.40\n",
      "Time elapsed: 0.36 min\n",
      "Epoch: 006/020 | Batch 000/461 | Cost: 0.3381\n",
      "Epoch: 006/020 | Batch 150/461 | Cost: 0.3627\n",
      "Epoch: 006/020 | Batch 300/461 | Cost: 0.2736\n",
      "Epoch: 006/020 | Batch 450/461 | Cost: 0.3932\n",
      "Epoch: 006/020\n",
      "Train ACC: 89.85 | Validation ACC: 93.50\n",
      "Time elapsed: 0.42 min\n",
      "Epoch: 007/020 | Batch 000/461 | Cost: 0.4984\n",
      "Epoch: 007/020 | Batch 150/461 | Cost: 0.3718\n",
      "Epoch: 007/020 | Batch 300/461 | Cost: 0.2903\n",
      "Epoch: 007/020 | Batch 450/461 | Cost: 0.4040\n",
      "Epoch: 007/020\n",
      "Train ACC: 90.02 | Validation ACC: 93.50\n",
      "Time elapsed: 0.50 min\n",
      "Epoch: 008/020 | Batch 000/461 | Cost: 0.5250\n",
      "Epoch: 008/020 | Batch 150/461 | Cost: 0.3481\n",
      "Epoch: 008/020 | Batch 300/461 | Cost: 0.3838\n",
      "Epoch: 008/020 | Batch 450/461 | Cost: 0.4789\n",
      "Epoch: 008/020\n",
      "Train ACC: 90.14 | Validation ACC: 93.90\n",
      "Time elapsed: 0.57 min\n",
      "Epoch: 009/020 | Batch 000/461 | Cost: 0.3028\n",
      "Epoch: 009/020 | Batch 150/461 | Cost: 0.3982\n",
      "Epoch: 009/020 | Batch 300/461 | Cost: 0.4042\n",
      "Epoch: 009/020 | Batch 450/461 | Cost: 0.5471\n",
      "Epoch: 009/020\n",
      "Train ACC: 90.26 | Validation ACC: 93.90\n",
      "Time elapsed: 0.64 min\n",
      "Epoch: 010/020 | Batch 000/461 | Cost: 0.2279\n",
      "Epoch: 010/020 | Batch 150/461 | Cost: 0.2992\n",
      "Epoch: 010/020 | Batch 300/461 | Cost: 0.4507\n",
      "Epoch: 010/020 | Batch 450/461 | Cost: 0.2165\n",
      "Epoch: 010/020\n",
      "Train ACC: 90.40 | Validation ACC: 93.90\n",
      "Time elapsed: 0.71 min\n",
      "Epoch: 011/020 | Batch 000/461 | Cost: 0.5089\n",
      "Epoch: 011/020 | Batch 150/461 | Cost: 0.2480\n",
      "Epoch: 011/020 | Batch 300/461 | Cost: 0.3782\n",
      "Epoch: 011/020 | Batch 450/461 | Cost: 0.3228\n",
      "Epoch: 011/020\n",
      "Train ACC: 90.47 | Validation ACC: 93.40\n",
      "Time elapsed: 0.78 min\n",
      "Epoch: 012/020 | Batch 000/461 | Cost: 0.2597\n",
      "Epoch: 012/020 | Batch 150/461 | Cost: 0.2955\n",
      "Epoch: 012/020 | Batch 300/461 | Cost: 0.2243\n",
      "Epoch: 012/020 | Batch 450/461 | Cost: 0.2967\n",
      "Epoch: 012/020\n",
      "Train ACC: 90.58 | Validation ACC: 93.60\n",
      "Time elapsed: 0.85 min\n",
      "Epoch: 013/020 | Batch 000/461 | Cost: 0.3367\n",
      "Epoch: 013/020 | Batch 150/461 | Cost: 0.3696\n",
      "Epoch: 013/020 | Batch 300/461 | Cost: 0.2744\n",
      "Epoch: 013/020 | Batch 450/461 | Cost: 0.4097\n",
      "Epoch: 013/020\n",
      "Train ACC: 90.65 | Validation ACC: 93.80\n",
      "Time elapsed: 0.92 min\n",
      "Epoch: 014/020 | Batch 000/461 | Cost: 0.2629\n",
      "Epoch: 014/020 | Batch 150/461 | Cost: 0.3282\n",
      "Epoch: 014/020 | Batch 300/461 | Cost: 0.2407\n",
      "Epoch: 014/020 | Batch 450/461 | Cost: 0.2714\n",
      "Epoch: 014/020\n",
      "Train ACC: 90.66 | Validation ACC: 93.80\n",
      "Time elapsed: 0.99 min\n",
      "Epoch: 015/020 | Batch 000/461 | Cost: 0.2497\n",
      "Epoch: 015/020 | Batch 150/461 | Cost: 0.3774\n",
      "Epoch: 015/020 | Batch 300/461 | Cost: 0.3405\n",
      "Epoch: 015/020 | Batch 450/461 | Cost: 0.4727\n",
      "Epoch: 015/020\n",
      "Train ACC: 90.81 | Validation ACC: 93.90\n",
      "Time elapsed: 1.06 min\n",
      "Epoch: 016/020 | Batch 000/461 | Cost: 0.4100\n",
      "Epoch: 016/020 | Batch 150/461 | Cost: 0.3284\n",
      "Epoch: 016/020 | Batch 300/461 | Cost: 0.3974\n",
      "Epoch: 016/020 | Batch 450/461 | Cost: 0.2978\n",
      "Epoch: 016/020\n",
      "Train ACC: 90.86 | Validation ACC: 93.90\n",
      "Time elapsed: 1.13 min\n",
      "Epoch: 017/020 | Batch 000/461 | Cost: 0.2101\n",
      "Epoch: 017/020 | Batch 150/461 | Cost: 0.3024\n",
      "Epoch: 017/020 | Batch 300/461 | Cost: 0.2714\n",
      "Epoch: 017/020 | Batch 450/461 | Cost: 0.2259\n",
      "Epoch: 017/020\n",
      "Train ACC: 90.91 | Validation ACC: 93.90\n",
      "Time elapsed: 1.20 min\n",
      "Epoch: 018/020 | Batch 000/461 | Cost: 0.3154\n",
      "Epoch: 018/020 | Batch 150/461 | Cost: 0.2534\n",
      "Epoch: 018/020 | Batch 300/461 | Cost: 0.3008\n",
      "Epoch: 018/020 | Batch 450/461 | Cost: 0.2815\n",
      "Epoch: 018/020\n",
      "Train ACC: 90.98 | Validation ACC: 93.90\n",
      "Time elapsed: 1.27 min\n",
      "Epoch: 019/020 | Batch 000/461 | Cost: 0.2850\n",
      "Epoch: 019/020 | Batch 150/461 | Cost: 0.2086\n",
      "Epoch: 019/020 | Batch 300/461 | Cost: 0.4104\n",
      "Epoch: 019/020 | Batch 450/461 | Cost: 0.2749\n",
      "Epoch: 019/020\n",
      "Train ACC: 90.94 | Validation ACC: 94.00\n",
      "Time elapsed: 1.35 min\n",
      "Epoch: 020/020 | Batch 000/461 | Cost: 0.4211\n",
      "Epoch: 020/020 | Batch 150/461 | Cost: 0.2129\n",
      "Epoch: 020/020 | Batch 300/461 | Cost: 0.2256\n",
      "Epoch: 020/020 | Batch 450/461 | Cost: 0.5096\n",
      "Epoch: 020/020\n",
      "Train ACC: 91.02 | Validation ACC: 94.30\n",
      "Time elapsed: 1.42 min\n",
      "Total Training Time: 1.42 min\n"
     ]
    }
   ],
   "source": [
    "def compute_acc(model, data_loader, device):\n",
    "    correct_pred, num_examples = 0, 0\n",
    "    for features, targets in data_loader:\n",
    "        features = features.to(device)\n",
    "        targets = targets.to(device)\n",
    "        logits, probas = model(features)\n",
    "        _, predicted_labels = torch.max(probas, 1)\n",
    "        num_examples += targets.size(0)\n",
    "        correct_pred += (predicted_labels == targets).sum()\n",
    "    return correct_pred.float()/num_examples * 100\n",
    "    \n",
    "\n",
    "start_time = time.time()\n",
    "\n",
    "cost_list = []\n",
    "train_acc_list, valid_acc_list = [], []\n",
    "\n",
    "\n",
    "for epoch in range(NUM_EPOCHS):\n",
    "    \n",
    "    model.train()\n",
    "    for batch_idx, (features, targets) in enumerate(train_loader):\n",
    "        \n",
    "        features = features.to(DEVICE)\n",
    "        targets = targets.to(DEVICE)\n",
    "            \n",
    "        ### FORWARD AND BACK PROP\n",
    "        logits, probas = model(features)\n",
    "        cost = F.cross_entropy(logits, targets)\n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        cost.backward()\n",
    "        \n",
    "        ### UPDATE MODEL PARAMETERS\n",
    "        optimizer.step()\n",
    "        \n",
    "        #################################################\n",
    "        ### CODE ONLY FOR LOGGING BEYOND THIS POINT\n",
    "        ################################################\n",
    "        cost_list.append(cost.item())\n",
    "        if not batch_idx % 150:\n",
    "            print (f'Epoch: {epoch+1:03d}/{NUM_EPOCHS:03d} | '\n",
    "                   f'Batch {batch_idx:03d}/{len(train_loader):03d} |' \n",
    "                   f' Cost: {cost:.4f}')\n",
    "\n",
    "        \n",
    "\n",
    "    model.eval()\n",
    "    with torch.set_grad_enabled(False): # save memory during inference\n",
    "        \n",
    "        train_acc = compute_acc(model, train_loader, device=DEVICE)\n",
    "        valid_acc = compute_acc(model, valid_loader, device=DEVICE)\n",
    "        \n",
    "        print(f'Epoch: {epoch+1:03d}/{NUM_EPOCHS:03d}\\n'\n",
    "              f'Train ACC: {train_acc:.2f} | Validation ACC: {valid_acc:.2f}')\n",
    "        \n",
    "        train_acc_list.append(train_acc)\n",
    "        valid_acc_list.append(valid_acc)\n",
    "        \n",
    "    elapsed = (time.time() - start_time)/60\n",
    "    print(f'Time elapsed: {elapsed:.2f} min')\n",
    "  \n",
    "elapsed = (time.time() - start_time)/60\n",
    "print(f'Total Training Time: {elapsed:.2f} min')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3dd3zTdf7A8dcn6WSVUfYqyB4FoWxEZAscoHCKcp648PypeHjq4T5x4el5iiKKA5yAJzhBFJC99y67QNlQOhhdyef3R75JkzZJ05GmJe/n49EHyXd+8iX5vr+frbTWCCGECF6mQCdACCFEYEkgEEKIICeBQAghgpwEAiGECHISCIQQIsiFBDoBBRUdHa1jYmICnQwhhChTNm/efF5rXd3dujIXCGJiYti0aVOgkyGEEGWKUuqop3VSNCSEEEFOAoEQQgQ5CQRCCBHkylwdgRCi8LKyskhMTCQ9PT3QSRF+EhERQb169QgNDfV5HwkEQgSRxMREKlasSExMDEqpQCdHFDOtNRcuXCAxMZFGjRr5vJ8UDQkRRNLT06lWrZoEgWuUUopq1aoVOMcngUCIICNB4NpWmP/foAkE+06n8Z/f93HhUkagkyKEEKVK0ASCg2cv8d4fBzl/KTPQSREiqCmluOuuuxzvs7OzqV69OkOHDgXgp59+YvLkyV6PcfLkSUaNGgXAzJkzeeSRRwqUhtdeey3fbcaOHct3331XoOMWxrZt21iwYIHfz+NN0ASCELMtu5RlsQY4JUIEt/Lly7Nr1y6uXr0KwKJFi6hbt65j/bBhw5g4caLXY9SpU6dIN2lfAkFJkUBQgkKNQJBtlRnZhAi0m2++mfnz5wMwa9Ys7rjjDsc65yf8sWPHMn78eLp3707jxo0dN/+EhATatGnj2Of48eMMGjSI5s2b89JLLzmWjxgxgo4dO9K6dWumT58OwMSJE7l69Srt27dnzJgxAHzxxRfExsbSrl07l9zKihUr8pw7N3f7Hj16lL59+xIbG0vfvn05duwYAP/73/9o06YN7dq1o1evXmRmZvLCCy8wZ84c2rdvz5w5c4p2YQspaJqPhphsMU9yBELYvPTzbvacTC3WY7aqU4kX/9Q63+1Gjx7NpEmTGDp0KDt27ODee+9l5cqVbrc9deoUq1atIj4+nmHDhjmKhJxt2LCBXbt2Ua5cOTp16sSQIUOIi4vjs88+o2rVqly9epVOnToxcuRIJk+ezPvvv8+2bdsA2L17N6+++iqrV68mOjqapKQkn8/tad9HHnmEv/71r9x999189tlnjB8/nh9++IFJkybx22+/UbduXZKTkwkLC2PSpEls2rSJ999/3+frXNyCJkdgNtlyBFbJEQgRcLGxsSQkJDBr1iwGDx7sddsRI0ZgMplo1aoVZ86ccbtN//79qVatGpGRkdx6662sWrUKgClTptCuXTu6du3K8ePHOXDgQJ59//jjD0aNGkV0dDQAVatW9fncnvZdu3Ytd955JwB33XWXIz09evRg7NixfPzxx1gsFq+fuyQFTY7AZDSpkjgghI0vT+7+NGzYMJ544gmWLVvGhQsXPG4XHh7ueK21+x9w7iaTSimWLVvG4sWLWbt2LeXKlaN3795u29drrT02uczv3N72dZe+Dz/8kPXr1zN//nzat2/vyJUEWtDkCIwMgccvkhCiZN1777288MILtG3btsjHWrRoEUlJSVy9epUffviBHj16kJKSQpUqVShXrhzx8fGsW7fOsX1oaChZWVkA9O3bl2+//dYRjJyLhvLjad/u3bsze/ZsAL7++mt69uwJwKFDh+jSpQuTJk0iOjqa48ePU7FiRdLS0op8DYoieAKBEQksEgiEKBXq1avHY489VizH6tmzJ3fddRft27dn5MiRxMXFMWjQILKzs4mNjeX555+na9euju3HjRtHbGwsY8aMoXXr1jz77LPceOONtGvXjscff9zn83rad8qUKcyYMYPY2Fi+/PJL3n33XQCefPJJ2rZtS5s2bejVqxft2rXjpptuYs+ePQGtLFZl7Qk5Li5OF2Zims1HLzJy2ho+v7czNzZzO0mPENe8vXv30rJly0AnQ/iZu/9npdRmrXWcu+2DJ0dgFA1Zy1jgE0IIfwuiQGCLBGUtBySEEP4WdIFAuhEIIYSr4AkExieVoiEhhHAVPIFAioaEEMKtoAsE0qFMCCFcBVEgsP1rkUggRECZzWbat29PmzZt+NOf/kRycrJfztO9e3e/HPdaFDyBwD7WkBQNCRFQkZGRbNu2jV27dlG1alWmTp3ql/OsWbPGL8ctDqVpnCEIpkDgqCMIcEKEEA7dunXjxIkTACxbtswxOQ3YRvCcOXMmADExMbz44ot06NCBtm3bEh8fD8C//vUv7r33Xnr37k3jxo2ZMmWKY/8KFSo4jtu7d29GjRpFixYtGDNmjKOucMGCBbRo0YKePXsyfvx4l/PbJSQkcMMNN9ChQwc6dOjgCDC33367yzwCY8eOZe7cuVgsFp588kk6depEbGwsH330kSMdN910E3feeadjWA13w2QDfPrppzRr1ozevXvzwAMPOIblPnfuHCNHjqRTp0506tSJ1atXF+Hq5wiiQeds/0rRkBCGXyfC6Z3Fe8xabeFm77OL2VksFpYsWcJ9993n0/bR0dFs2bKFDz74gLfeeotPPvkEgPj4eJYuXUpaWhrNmzfnoYceIjQ01GXfrVu3snv3burUqUOPHj1YvXo1cXFxPPjgg6xYsYJGjRq5zIngrEaNGixatIiIiAgOHDjAHXfcwaZNmxg9ejRz5sxh8ODBZGZmsmTJEqZNm8ann35KVFQUGzduJCMjgx49ejBgwAAgZ7jsRo0aAbgdJjsjI4OXX36ZLVu2ULFiRfr06UO7du0AeOyxx5gwYQI9e/bk2LFjDBw4kL179/p0/bwJokAgRUNClAb2SWESEhLo2LEj/fv392m/W2+9FYCOHTsyb948x/IhQ4YQHh5OeHg4NWrU4MyZM9SrV89l386dOzuW2c9doUIFGjdu7Lgp33HHHS5P5XZZWVk88sgjbNu2DbPZzP79+wHb5Drjx48nIyODhQsX0qtXLyIjI/n999/ZsWOHYyKblJQUDhw4QFhYGJ07d3acD2xjEn3//fcAjmGyT58+zY033ugY0vrPf/6z45yLFy9mz549jv1TU1NJS0ujYsWKPl1DT4InEJikaEgIFz4+uRc3ex1BSkoKQ4cOZerUqYwfP56QkBCs1pwen7mHjLYPCW02m8nOzs6z3N06b9v42pT8v//9LzVr1mT79u1YrVYiIiIAiIiIoHfv3vz222/MmTPHkaPQWvPee+8xcOBAl+MsW7aM8uXLu7x3N0y2t3RZrVbWrl1LZGSkT2n3VRDVEdj+lRyBEKVDVFQUU6ZM4a233iIrK4uGDRuyZ88eMjIySElJYcmSJX49f4sWLTh8+DAJCQkAHkf+TElJoXbt2phMJr788kuXit7Ro0czY8YMVq5c6bjxDxw4kGnTpjmGud6/fz+XL192e1x3w2R37tyZ5cuXc/HiRbKzs5k7d65jnwEDBrjMZFZc8xn4LRAopeorpZYqpfYqpXYrpfKMN6tspiilDiqldiilOvgrPY4hJiQQCFFqXH/99bRr147Zs2dTv359brvtNsfw0Ndff71fzx0ZGckHH3zAoEGD6NmzJzVr1iQqKirPdv/3f//H559/TteuXdm/f7/LU/2AAQNYsWIF/fr1IywsDID777+fVq1a0aFDB9q0acODDz7oNpfiaZjsunXr8swzz9ClSxf69etHq1atHOmaMmUKmzZtIjY2llatWvHhhx8Wz8XQWvvlD6gNdDBeVwT2A61ybTMY+BVQQFdgfX7H7dixoy6Ms6npuuE/f9FfrDlSqP2FuBbs2bMn0EkoVdLS0rTWWlutVv3QQw/pt99+O8ApsrGnKysrSw8dOlTPmzevQPu7+38GNmkP91W/5Qi01qe01luM12nAXqBurs2GA18Y6VwHVFZK1fZHeuxzFkurISGE3ccff0z79u1p3bo1KSkpPPjgg4FOEmBrFmvvdNeoUSNGjBjh1/OVSGWxUioGuB5Yn2tVXeC40/tEY9mpXPuPA8YBNGjQoFBpMDuKhgq1uxDiGjRhwgQmTJgQ6GTk8dZbb5Xo+fxeWayUqgDMBf6utU7NvdrNLnlu1Vrr6VrrOK11XPXqhZtdzD76qJY6AhHk5DdwbSvM/69fA4FSKhRbEPhaaz3PzSaJQH2n9/WAk/5IS858BPIjEMErIiKCCxcuSDC4RmmtuXDhgqOJq6/8VjSklFLAp8BerfXbHjb7CXhEKTUb6AKkaK1Pedi2SMwyeb0Q1KtXj8TERM6dOxfopAg/iYiIyNOhLj/+rCPoAdwF7FRK2Ru7PgM0ANBafwgswNZy6CBwBbjHX4lx9CyWHIEIYqGhoS49W4UAPwYCrfUq3NcBOG+jgYf9lQZnOa2GSuJsQghRdgRdz2IpGhJCCFdBEwiUUiglLSaEECK3oAkEYOtLIK2GhBDCVVAFApNJSdGQEELkElSBwKyUtBoSQohcgisQmJS0GhJCiFyCKhCYlMxHIIQQuQVXIDApCQRCCJFLUAUCaTUkhBB5BVUgkByBEELkFVSBQHIEQgiRV3AFApNC4oAQQrgKqkCglIw+KoQQuQVVILiUkc2Fy5mBToYQQpQqQRUIkq9ksXy/TMghhBDOgioQCCGEyEsCgRBCBDkJBEIIEeQkEAghRJCTQCCEEEFOAoEQQgS5oAoEg9vWomr5sEAnQwghSpWgCgSVIkIJNatAJ0MIIUqVoAoEZpMi2yJDTAghhLOgCgQhJkW2jDUkhBAugisQmE0yDLUQQuQSXIHApMiS2euFEMJFcAUCs0xMI4QQuQVVIDCbTGRbNVqmqxRCCIegCgShJlvTUckVCCFEjqAKBGajD4G0HBJCiBxBFQhCTBIIhBAit6AKBIv3nAXgXFpGgFMihBClR1AFgg0JSQAcOJMW4JQIIUTpEVSB4MmBzQFoVrNigFMihBClR76BQCllLomElIQ6lSMAkBoCIYTI4UuO4KBS6k2lVCu/p8bPTMpWWWyVfgRCCOHgSyCIBfYDnyil1imlximlKuW3k1LqM6XUWaXULg/reyulUpRS24y/FwqY9gJTRiCQOCCEEDnyDQRa6zSt9cda6+7AU8CLwCml1OdKqSZedp0JDMrn8Cu11u2Nv0k+p7qQjNaj0rNYCCGc+FRHoJQappT6HngX+A/QGPgZWOBpP631CiCpuBJaHHKKhgKcECGEKEVCfNjmALAUeFNrvcZp+XdKqV5FPH83pdR24CTwhNZ6t7uNlFLjgHEADRo0KPTJ7DkCqSMQQogcvgSCWK31JXcrtNbji3DuLUBDrfUlpdRg4AegqYfzTAemA8TFxRX6Lq6kslgIIfLwpbK4hlLqZ6XUeaPy90elVOOinlhrnWoPMFrrBUCoUiq6qMf1xj5N5YVLmf48jRBClCm+BIJvgG+BWkAd4H/ArKKeWClVSxmP6EqpzkZaLhT1uN4s3H0agHFfbvLnaYQQokzxpWhIaa2/dHr/lVLqkXx3UmoW0BuIVkolYmttFAqgtf4QGAU8pJTKBq4Co7Wfm/OcN8YYSs+SWcqEEMLOl0CwVCk1EZiNrVPu7cB8pVRVAK2125ZBWus7vB1Ua/0+8H7Bkls0ZnttsRBCCAdfAsHtxr8P5lp+L7bAUOT6gpIigUAIIfLKNxBorRuVREJKQogEAiGEyCPfQKCUCgUeAux9BpYBH2mts/yYLr+oUj4s0EkQQohSx5eioWnYKnk/MN7fZSy731+J8pdalSICnQQhhCh1fAkEnbTW7Zze/2H0Bi5zlJQMCSFEHr70I7Aopa6zvzE6k1n8lyT/Gdy2dqCTIIQQpY4vOYInsTUhPQwooCFwj19T5Scta+c7erYQQgQdr4FAKWXC1tmrKdAcWyCI11rL7O9CCHGN8BoItNZWpdR/tNbdgB0llCYhhBAlyJc6gt+VUiPt4wIJIYS4tvhSR/A4UB7IVkqlYyse0lrrMlngHlsvirOpUrIlhBB2vvQsrlgSCSkpVq05nZoe6GQIIUSp4ctUlUt8WVZW7DqRGugkCCFEqeIxR6CUigDKYRtGugq2IiGAStjmJSjTDp69RJMaFQKdDCGECDhvOYIHgc1AC+Nf+9+PwFT/J82/ftlxMtBJEEKIUsFjjkBr/S7wrlLqUa31eyWYJiGEECXIl8ri95RS3YEY5+211l/4MV1CCCFKiC/DUH8JXAdsI2eMIQ2U6UBwPOlqoJMghBClgi/9COKAVv6eT7ikJV/JDHQShBCiVPClZ/EuoJa/E1LSKpeTSWqEEAJ8yxFEA3uUUhsAR5dcrfUwv6WqBNzYvHqgkyCEEKWCL4HgX/5ORCDIwElCCGHjsWhIKdUCQGu9HFintV5u/8MpZ1BWfbLqSKCTIIQQpYK3OoJvnF6vzbXuA8qo6xtUBmD78eQAp0QIIUoHb4FAeXjt7n2Z8fTNLQOdBCGEKFW8BQLt4bW792WGqcyGMCGE8A9vlcX1lFJTsD39219jvK/r95T5iUyvI4QQrrwFgiedXm/KtS73+zLj2uoWJ4QQRedt0LnPSzIhJUXigBBCuPKlH8G1QWvITkdbLPlvK4QQQcSXISauDbvmwqu1CE/N6T/ww9YTAUyQEEKUDsETCMJtUy+bsy45Fi2JPxuo1AghRKnhy5zF/1ZKVVJKhSqlliilziul/lISiStWRiCoEZYz6mhGlhQTCSGELzmCAVrrVGAokAg0w7VFUdkQXglwDQRScSyEEL4FglDj38HALK11kh/T4z9GjoCMNMciaUoqhBC+BYKflVLx2CaoWaKUqg6k+zdZfuAmEAghhPAhEGitJwLdgDitdRZwGRju74QVOzeBYMuxiwFKjBBClB6+VBb/GcjWWluUUs8BXwF1fNjvM6XUWaXULg/rlVJqilLqoFJqh1KqQ4FTXxDmUAiJhIxUx6KkyzJdpRBC+FI09LzWOk0p1RMYCHwOTPNhv5nAIC/rbwaaGn/jfDxm0YRXdAkEQgghfAsE9jaWQ4BpWusfgXwn/NVarwC8VSwPB77QNuuAykqp2j6kp/DCK0odgRBC5OJLIDihlPoIuA1YoJQK93G//NQFjju9T8Tfo5pKIBBCiDx8uaHfBvwGDNJaJwNVKZ5+BO4GhHbboFMpNU4ptUkptencuXOFP6MEAiGEyMOXVkNXgEPAQKXUI0ANrfXvxXDuRKC+0/t6wEkPaZiutY7TWsdVr1698GeMrAxXpaWQEEI486XV0GPA10AN4+8rpdSjxXDun4C/Gq2HugIpWutTxXBcz8pVgysXXBYt2yfjDQkhgpsvw1DfB3TRWl8GUEq9gW0y+/e87aSUmgX0BqKVUonAixi9lLXWHwILsPVWPghcAe4p3EcogHLRcCUJhRVtxMCxMzaSMHmI308thBCllS+BQJHTcgjjdb4TPmqt78hnvQYe9uH8xad8NGgL98dV4eNNKSV6aiGEKK18CQQzgPVKqe+N9yOAT/2XJD8qFw1AZS19CYQQwi7fQKC1flsptQzoiS0ncI/Wequ/E+YX5aoCUM10CSgf2LQIIUQp4TUQKKVMwA6tdRtgS8kkyY/K23IE5bIuIoFACCFsvLYa0lpbge1KqQYllB7/MoqGymW7NiFNTc8KRGqEEKJU8KVDWW1gtzE72U/2P38nzC+MHEHtkMsui2P/9Ts/bXfbhUEIIa55vlQWv+T3VJSUkHAIr0SrSnmnU/hw2SGGtct3UFUhhLjmeAwESqkmQE2t9fJcy3sBJ/ydML+pVBeVUnaTL4QQxc1b0dA7gLuBea4Y68qmyvUh5ViexSrfnhFCCHFt8hYIYrTWO3Iv1FpvAmL8liJ/i6oPycfz304IIYKEt0AQ4WVdZHEnpMRUrg/pyZTnqstiyREIIYKVt0CwUSn1QO6FSqn7gM3+S5KfRdkGPK2rzgc4IUIIUTp4azX0d+B7pdQYcm78cdhmJ7vF3wnzm8q2LhF11Xn265xRsHedkGEnhBDByWOOQGt9RmvdHVvz0QTj7yWtdTet9emSSZ4feMkRrDl0noH/XUFGtiXPOiGEuFb5MtbQUmBpCaSlZFSoCeZwelS6xFe5piJ47oddHD53meNJV2hSo2Jg0ieEECWsOOYeLltMJqjRgpbqaJ5V59MyjFdScyyECB7BFwgAarejYeZBck+RnJqeHZj0CCFEAAVtIFBXk6iL+5ZDC3b6d8ZMIYQoTYI0EFwPQBtTgtvVby/aX4KJEUKIwArOQFCzFSgzvSt5HnFUWg4JIYJFcAaC0Eio3oLm+rDHTS5n2ALBb7tPk3D+ssfthBCirAvOQABQux3NrJ4Dgdaaoxcu8+CXm+n91rKSS5cQQpSwoA4EFbIuUJMkt6sf+moLHy73HCg+WXmYf36XZ0w+v0nPsnAlU1o1CSGKX/AGgvqdAehiine7ekNCErM25AxXvXBXTkuifafTeGX+XuZsKrlRTG/491JavfBbiZ1PCBE8gjcQ1G6HDo+ip2mnT5vvP3PJ8XrgOyv8lSqPzjk6u137Fu46xaerjgQ6GR59sOwgqw/KoIXi2hG8gcBkRjXtRx/zVkxY891809GL+W4jisffvtrCy7/sCXQyPPr3wn2M+WR9oJMhRLEJ3kAA0Hww0SqVLqa9+W66Yv85t8tjJs5nuYd1QghRFgR3IGgxhFRdjjHmJUU6zG+7i2cwVq11/hsJIcqcjGxLqe6bFNyBIDQSU9xYbjatp6Eq/M28OG7g329NpNHTCziedKXIxxKl3/bjyZxJTQ90MkQJ6TBpEc2fW8jDX28JdFLcCu5AAFzq8CDZhDA+5Pt8t42ZOJ+YifPzLNcaNh+9yI/bTgCw6sB55u8o2HhFP22z9XJ+/sddtH3xt2LLZQj3zqam88bCeKzWwOTChk9dzY1vXjujuwvvLmfacgPzS+k4ZkEfCCzlazLDMpCR5pXEqkOFOsbxi1cYOW0Nj83exvbjyfzl0/U8/M0WnvnetxZJzpbtO0daRjaTf3XfrLUoZm04xoYj7vtN5JaeZWH3yZRiT0Np8eR3O5i27BAbEny7Hv6QnpV/IwVv/vXTbtZI6yVRDII+EISHmHgv+xbO6SheCf0MMwUvx1t98ILj9fCpqx2vv1l/zN3mPvFHfcHT83Zy20drfdr2mXk7GTJlFR8tP1TsaVl/+AK/FyDHo7Xmqe+2s/7whfw39pG9vNZahutlZq5J4E5pvSSKQdAHgugK4XRvFcOLWXcTazrCfeYFfjtX8pVMnvpuO1eNbGKWxcqag+dJvFj66gW2HLM1l33913iXQOfs8LlLjJ+1lSxLwZ5sb5++jnFfbs5/Q4NVw7ebErnj43UFOk9Zc/DsJVLTs4p8nIxsC33eWnbNt2Yb/O5Kmj/3a6CTcU0I+kAA8M9BzVlg7cIiSwf+HjKPeqr4fkArD5zjjYXx3P7RWt5ZfIBvNyUyZ6Mtp/DW7/u485P19HxjKbmLqgP9nGo25czSdikjZ2iLfm8vdwzT/dR3O/hp+0m2HU8u8fTltv14MulZvufmcmcENiUkce/MjVhKoM7AU+fAfm8v5/aPih7sTianc/j8Ze7/fCMxE+ezcFfpqG/SWpN8JbPYjrfnVCoZ2TkPIVprury2mG832nr870hM5sPlhSvuDTYSCMCYn1jxQtY9lFMZ/Bz2LOW5WizHvuvTDUxbdoj1R5IcRSxnjRtB/Kk0x3a5n97sNyqtNY/P2eZY/s7inLkSrFbNjkT3N+FzaRnc//lGUq76/oT5ycrDxEycz4nkqy6BwNnBs5eYsuQAACZl28ZfN8+E85eZsuRAvkVTp1KuMnzqard1MqnpWaRc8XwNlDEt6aOztvJH/FlOGy15Br2zgtcW5N+/pDA6vbrY47q9p1KL7TxZFtt1m7clsdiOWRSfrU6g/aRFHLvgnxywVcOZ1AwmztvBubQMhr2/2qWu7UxqepFzXJsSkvzawOAf327nT++t8tvxPZFAYFj8+I2cohr/zHqAKuoSn4a9RSjFO8ibPQB8sOwQU5ce9Lqt/eaaabEyb+sJx/J3Fh9wvP5s9RGGvb+amauPkJlt5ZYPVrPOKEeftuwQi/ee5X8FGA/plfm2G9/WYxcdN3m71PSsPC2h7Jv4Ws7+j2+38/Q83wfq6/3WMt5etJ8z+QyvkWZMMboj0Va5fTzpCh8ZT4LtXvqddpN+z7NP7hTbP6896MSfTmP6Cs+DDpZmpXXG7SV7zwC2xhX+pMHt96zLa0vo95/lhT7usn1nGfXhWj5bXbThT17/1fMDxtwtiew8UfKNNCQQGJrUqMC4Xo2ZY7mJZ7Luo6tpL5NDPy7Wc+xxetp7/4+DXM7wHGjsxRzKzc/6s1VHSL6SyV4jR/Gvn/fQ7Llf2XosmdHT15FyJYvNRhn/r16KBS5cynCkYeuxnCE0LFadJxD849vtPPyNaxvonJtn3mP/EX+GP+LPuCybuyWRWRsKPlBffk9g9pTab+J3fbqe13+N5+Vf9rhNm8u+uS6vP8Z0ysy2Fstx//blZlq9sLDA++X+jJ5sP57s106N/q6Xz/kewOK9Z91uc9bL/0NqepbL7yC3E8m2UoJD5y553MYXH3kZ1ThQJBA4ubFZdQC+sfRlkaUjI80ruc1cfG29jzplia9mWbyOX3Thsq0s9b7PN+ZZN+mXPUyc67lp6saEJHYaRUabj170+OXu+MpiOr+6mLWHLnDLB2scyy1W7RK0AE5cdC0qy7JYWWvkPqxaczXTtefkvTM3ce/MTY73l7wEvfxsznWdTqVc5eDZnB+j/UZ36JxtAiF7m23ngev2nbYFTa01Hyw7SGKujnv2Yzhfh9x+3HbCpw5/m48mkelUdv34t9vo9OriIt9kF+4+zZXM4u+duubQeV75ZQ/Dp65mxuqEYj9+bu7i0pqD570W4Wmt2Wnk+E6npHO7m9ZvRY0z98zYyC0frCHbYqXZc79y92cbinhE32w5dpGYifM5lVI8xdGFIYHASY8m0Y7Xj2Q9ynldiX+Hfkx3066ApWnlAfftxM9d8vxkk3I1y6Xy2Qa4rGkAACAASURBVNvN7XKmhQNn01yW5S7zd/dE+cHSnEo4q4aWLyyk+XMLPd7s/vpp/s0cD5695Hb/vzvVkQB0e/0P+r3tOYvv7kYz8J0VXM20sObQBf69cB8nU1x79TrfhDq8vMjxOvlKpiNNj83exgin5sHubDl2kZHT1vLq/JxB834xitSKq2h589GC932wWHWe+SyyLVa01tz58Xo+MYLmFi9PxLtOpBRoToxFe84w6J0Veb5PT+eqy7mckc2dn6x3+9Bj9/yPu/jT+6v4I/4M01ccZr2P/WEAmj37q8tYYTsTU3hjYd5+OtuNRg9XsyxkZls9trpatOcsMRPn51vxbbFqYibOZ/oK7xXWX607CuCxdV5J8GsgUEoNUkrtU0odVEpNdLN+rFLqnFJqm/F3vz/TUxAZhHFzxmQAvgl7jeaq8H0CCsvbUMc7E1OY66ES8B//216g87zw426X91dztb45k5qe54neYs154l22Lycbnrvo59C5S5xNS2fLMe8ti9YcPE+/t5cz4L++D/H90/aTrDl0nty3/mQPT5aPztqSZ9RQ+55pTp8v6XLOD7z9pEUuOYsLl73/+G81gu4vO06xNN52XUxe6lIaPz2fx7/dlme5NyOnrc3Twz3pciadX13MnpOpeQL3yeR0Hvlmi8t8FpnZVpo8+yuTc90Qf9lxitcW7M3TJDg1PYuh763isdnbOHg2zacmw0/8bzvxp9NIzdVg4eiFKzz/Q87DVbZRqb3/TBp//WwDzXI1Cd18NImv1tl+f0fOF7x+IdNi5a9OT/d/en8V05YdchQ5HjiTRka2xZGjeMrDhFP2YtrzxkNYfp0z7Tnk/y464HW7gDcRxI+BQCllBqYCNwOtgDuUUq3cbDpHa93e+PvEX+kpjHNU5uaM1wH4LXwie8PHUpXia9WRH29DHWcWsO1+QeQODC/8uJtjuYpE1hzKeXpxLk7YlKunbt//LKfzq+4H9ev7n2XETJzPhiNJjie8A2c9l79aNS43wPGztnLnx+sZPT2nyeW6wxc8XhtPT1z5NX/9YdsJDpxJc7vueNIVzqamk2WxMnFuzg3kwuVM7pm5kcm/xjtyAu4CgVXDvC0n8iwvqBX7z3E2LYPBU1bmWbfzREqeuqJ04yblrrx6+orD/LLjJP/39Wb+azQVttdZLdpzhn5vr+Dxb7cz+dd4n1qMZVmt7EhMRjvd8b40noK11i7DqazYf86lWC3LYuXn7TmNFJzXFZXGFkD7/3cFT8/d6fgsnurVcgfYcV9u5o2F8ewyKnfnbUnkcAHrD+6ZscGlMYjd+Vw5/hPJVxk9fW2BWgEWRIhfjmrTGTiotW2GeKXUbGA4UHoHmndjr27IQ5mP8c+Q2cSYzrAl4m/8N2skMywDSaVCoJPnM+eevFczLUSGmYt0PE/1G7/tPu3S3NUbe5m+r72dPXH+0TgHhdzcFXFdybJwu5d9AHadSKW/m5yKu3GncnNux/7s97t468/t8t0HbMUK6VkWLmVks+14MgNb1/K47ZhP1vHnjvUd7901MLDbcCSJy5nZdGxYxev5sy2aBTtPA6cZ3LY2VcqFuqz/ebttbKxOMVXo27Im//h2O5UiQ3jxT605dO4Su06kOIrUJi+IZ97WE9SoGJ7nPF+vP8ZzP3guen3zt33MXJPgeP/GwnjCzO6fXwtaB3Mq5Sr2jO3GQhS3ga113rRlh0iYPITHv91OWIiJ/a/czB/xZ9h6LKeoyZOl+9wXP907cyM/PdLT8f79Pw6y7nASv+w4yZguDQuVVm/8GQjqAs7lBIlAFzfbjVRK9QL2AxO01nmalSilxgHjABo0aOCHpHr3q7ULv2Z24U+mNbwX9j4TQucyIXQuT2aNY4GlC5eJLPE0FZRzT95HvtnCp2M7+eU8lzMtbp9wSqt7Znguly5u321O9BgIcrcquu4Z1x7uy5/s7fG4qw9eoGujao73Ry5c9ritPeg+M7hFfsl1GPjOCjY809fturT0bNKzLI5iyh7XRfPAl5tcWgjZvw8Xc5WpHzx7KU9DgNzcNeF1l+NLvpLpMougL3q+sZTnh9oKKbwFz4LIzLby7abjHouXfHUyuWQrjv1ZR+DuyuYO2T8DMVrrWGAx8Lm7A2mtp2ut47TWcdWrVy/mZLqaemcHRrSv43bdz9buxKR/zZtZtwHwZuh0dkfcxzuh79NMldz8xUW1JP4sZ4NwCOTsAI006uzWD9xXNo+ftdXrfvm1FvrPopyOhk/6UEf07abi6WT29znb6OhUuX7/F5t8bib69qJ9Hm+/i/ac8SnHZdd+0qJC5SztM+F5amJ71em6e/v+nHC6cRc2CPgyPau/muD6MxAkAvWd3tcDTjpvoLW+oLW2Pwp9DHT0Y3p8MiS2Nu+Mvp6IUE+XRjHVMoJ26dPZam0CwAjzGn4P/yezw16mLmVjfJfOrxVtMp6yqDjLlwvLU6V5YnJOHczsDXkbJuRuvutNsg/lyPk9/+a+32R4uXaXfWzSau/pbLdg52mOOtU9pabnVNg/8MUmfFUcI7B6Gp6k5QsL+dUYOvoNLyMCzyxgJ7O1h/LWV7nrVa615ttNx/3+3fVn0dBGoKlSqhFwAhgN3Om8gVKqttbaXhM0DPBPn/5C+O5v3flibQIhZpPbUURTqMAtmZMA6KD280roDLqa9rI64jFSdSQ/W7rzYvbdZPv1EotrxfGknBv9xHl5+4jcX4Aboy83DW+V8pC3vP2DZf4ZsyetGAbZK44RWM+kem6O/dDXW3io93Ve+8J8vLJggeCOj9fRqnYlr9tkZFuYsTrBZZgMf+Vp/XaX0lpnK6UeAX4DzMBnWuvdSqlJwCat9U/AeKXUMCAbSALG+is9BdWmbhT/HmUrz81vOOktuhlDMl/lr+ZF3GTaRm/zdsaELGFMyBJmZA9kl7URv1njAKimUjmqa1J6BwIQIu9cCRfzaTZbWMVVNu9v0/wQCHN32nR2/lImfd5a7lLk5E+qrM2TGxcXpzdt8v3pqDgUpKzSRjPCtJo3QqcTrjw/ReyyxrDA0plfrV04omsXLZFCFKNyYWa/9GLOrVF0eY6c91y5LVy9PKINd3UtXKshpdRmrXWcu3XSs9gH/73dtyZ/ORQ/WHvSPOML2qVP542s0STr8my3NmajtZljqzamBJ4K/Zal4f/gMfNcAELILvbB7oQoqJIIAoAEgVJCcgQ+KniuwLsmKpG66gIDTRv5s3k5oSrnh5eqIzmo66JRbLE25QoRnNWV+cXSlZQy1HdBCFG8Xh7emru6xRRqX285AgkEPtpzMtWl12a9KpEkFqAlhzdhZPFYyFweDvmJVB1JJeXbcVda2jDb0oca6iI1VDILLZ3Yrm0tmWpwkXTCSKV8saRRCBF4EggMgQoEdrd9uJYNCUk82KsxH/l1vHpNLZKop85RUV2ltUrgidD/FegIWdrMN5Y+XCGCaqSyTTdhhbUtibo6lbhCXXWevboBLdRxEnRN0nHt+amwoqX0UIhSQwKBIdCBIOlyJmsOned0SrpjIpdAaahOM8S0nlO6KhmE8kHYFMe6hZZOhJFFH3PBBjSz79tUJXKd6RSXdAShZJNKOQ7pulQhjWey7mOzbl6cH0UI4YNG0eVZ+kTvQu0rgcAPPl5xmFf9NJVh0WicpujgnyGzqaEu8oOlJ/1Mm7nT/AcHdV126xgUVpqpRMcQGV1Nvn+e87oS0crW/O0HS3eO6prssDZmjzWGzqa9VFVp7LA2prUpgT3WhlRTqSy3tsOE5vGQ/3GZSB4Lmcd+a13idQNqqGT+nXU7W3SzfM4sRPCqGB7CzpcGFmpfb4FAejtdc5TL6zey73C8W2mN5cXse7zuPcy0mgHmzUzLHka8rk97dZCzVOa4rglAa5XAxJBviFFnHKcaYfY830F+mplO0AzbWDTzwv/lWP6HpT2tTEeppWxj0ZzUVbFoM+EqixoqGatWtMyYQQZhAJiwUoU0MgmlqUqkjrpAlLpMjDpNjDrDFcJ5O3sUkWQSZ9rHPmt9Nurm2FqyW6lBMmeoWujPIURZJoGgkHRpGETcD36y9uAnaw/H+9xFQLt1DHdlPeN4f6d5CW3UEeqoC5zTUbQ0HSNJV2S1tQ0ZhJKNmdHmpYRgIZRs6qgLzLH0Zq9uyFprK9J1GJ1M+9hsbcr34S9SW9lGgcxdpHVSR9NQnaG6sg35a1KafRFjC/TZhvsQsE7oalzUFdlljQFgm25CZS4RrVJorE450nVVh5FMBWqrJFZa2jDL0ofdOoYupr20V4fYqxvwcuhMAJJ1eX60dGextSNpuhzZmDis63CFCELIpqdpF1Fc4gJRrLK2MVLifqiucLKIJINkKhboszvrZtpNJBlssTYt0nFEAPip/50UDRXS9BWHeG1BPPf3bESDauXo3awGvd60TWtZo2I4qelZeXpnivw1UGeI4jJtTUf4wdKDK0S43e6lkBncHbIoz/LLOpxUypOkKzLTMpBD1jps0034i3kRk0I/55C1NueJIpoUrjPljHOfpCtQVRVtLtqCsmiFWXn+/V3W4VwhguoqhW3W62hvcu3dus16HUm6Io3VKdZZW3KZSMKwDdnQx7yViVkPMNK8gj6mray1tiaNcowyuw6nvdnalI4m28QpfTPe5JCui8KKwtbyLExlM8C0if7mzVTiMpeIpJNpPzutMSToWsyy9OGsrowZK/t0A8LIIpMQFJqqRg4tjXL5XovhplW8FPo5lZWtX8ElHUEFlc6IjElsM1rCeWYLkE3USW42r+eMrsJpXZX9uh5HtW347kpcoppKc+q46VyE6t3NpvX8PWQu8yw3sNjagUO6rk/7+YO/ioYkEBSSPRA8cEMjnh1iG8r2bFo6oSYTVcqH8fX6ozz7vW2c9SrlQhnWrg6frz0ayCSLfISRRXWSOUU1qpHCaPNSqqo0Duk6LLO2I1lXIIMwsgjBfiOpSRLjQuYTrVI4pmtwQVeiprrIIktHdulGZBIKaOaEvcwFXYlIMkgnjHO6MpXUZYaY1rNXN+AP6/W0UkcZYM4ZLjxNR2LB5Lg5FqfZ2b1pZDpNnNrnNRgVF6tWmJTm4czxXCacVda2hGDh3dCpDDTn/3s+aK1DY3WK01RhctYddDftZnTIskKlZb+1Ls1MtuLIczqK3yxx/M9yI5eI5LSuSk/TTt4NncpKa1v6m7d4PM7bWaP40drdEWzswsiijjpPgq6FCU1rlUANdZHzOooEXYuHQ37kNvMyllrbs9DSmT26ASm6PCY0UeoyIVg8BpsK4SHskkBQegLBR8sP8fqvroHA2Vfrjjom3Jhyx/U0qFrO43y3657uS9fXg280UFEYOU+ydTlHnGkfp3U1zlMJCyYu6XI0MZ2guTpOXXWeY7oG+6z1OUVV6qnzrLO6flejuIQJK81NiXwV+hohyspVHUakyuQXS1duMO3ghK7OsMyXaajOcF5HodA0VGc4pavxXOhXDDOv5bSu4qjPcZaoozmlq9LJtD/Putx6pL9LEhW5auQCW6sjPBjyC8PMa0nXoUQo7wPUHbdWp77J/ei/660t6GKyDd6WYK1JpMqgpvI+M53dGksrvrL0I4lK/M38MzeYdvg9eLZP/8htsZ2/AoHUERRSC2PkwLb1Krtd7/w1GdK2NiYF9/SI4dC5y3wwpgNtXsyZP7ZWVATrnu7Liv3neMqY7rCkxnoRZU1OccYJqnPCmnd+jvPWKNaR9+EkUdfIs8zeU32dtRVNMr7yembnp9RkbbtJjc96lPFZj/qU8iguMdb8G/eELMSEJo1IPskezAzLzW63360buRy/MmmEk8VQ81rKk8FCaydO6OgiTgyl6WnaRR/TVlqbEqjIVdKIZLmlHfG6PpuszV06ZdoDaTVS+GfIbEaZV2DKFRQ2WJvT2bTPZdlaSyu6mfeQYK3JUmt7vrL0I1lXoKtpL1PDprDfWhcLZs7qyl7rbtxNd1ocJEdQBEcvXKZhNfc9d1cfPM+YT9bz6i1t3E4tZx+y4seHe9Cufk4wmTh3B7M3Hmfr8/25/uVF1ImK4I1RsZQPD3FMjO7MpKAUzLciRFAzYcVaAp0vI0PN7H15UKH2lRyBn3gKAgA9mkTz29970aym97GBnIMAwOSRsUweGQvAKyPacGOz6tSvWs7jxBmP9GnKlCUHCphyIURxKokg4E8SCPyoeS3PTfNqVgr3OhkGwF+chpuNCDWz418DKB8WgtaadxYf4P2lB2ldx/vkFnbXN6jsmEy7a+OqrDtcuMm6hRDXnrIdxsqw3yfcyKp/3lSgfSpFhGI2KULMJh7v34zZ47oysHUtEiYP4cmB3od8+O9t7R2v3U0G0qBq/k38hBCBVT7c7JfjSiAIkKjIUOpVKfzN12RSdG1czfH+4Zua8NafCzpvgs3scV1Z8VTBgpIQouRVLR/ml+NKILiG1Kmct/NVC6N4qlqFnC+Qc5HVC0NbOQLKL4/2zPccO/41wOV9qFlxW1w9x/uwEPdfqTUT++RZdl/PRvmeryA+GNOhWI8nRLCQQHAN6X5dNHMf6k7HhlUcy14Z0YaEyUOoGBHqWPbM4JZMvdN202xXP8qxvE3dnNcJk4ew9fn+zH2oG//5czua1qjAkdcHUykilG/u78LtcfUd25qUrajp1g51WfZEbx7r29QlXbPHdaVOZdcmfnd3a8jzQ/M2cfTFu6Pbu13unEPyxfD2dTyu+9uN15EweQjdryvYMUujobEyDeq1wv5bK25SWXyN6diwClPv7MCsDccY1r4O11XPabW0/MnepFzNIizExJDY2nS/rj9VvGQ1q5QPo2P5qnRsWJWRHXOe+rs3iabbddXYdjyZR/s2YeX+8wB0iqlKncqRTOjfjHWHLxB/Oo2XR7Rx3KD/0b8Z/1mUf8ciZ2sm9uHLdUd5tE8TDpy5RGy9KDKybUN3hIWYyMzOGcbDU27Ek9dvbcuP2066Xdeyti3XNPOezjR77leXdb882pM3Fsaz8oDtc780rDUv/rS7QOcuKfPH96R1nSjSszaxeO+ZQCdHlFKSI7gG1YqKYEL/Zi5BAGzNXWOdOsC5CwJ1otyP7ZObUorfJvRiaGwdxwB8zs8qcx7sxvYXBzCsXc5T96N9m9K1sW2ET3uLqAn9bMNOPzUop7LbOUdRp3Ik/xzUgnJhIbSrXxnl/ESk4YeHcwbIcz7/1uf7e0z7m6Ni2f7CAMqFuT4H7XppII2iyzs+H7gGl34tazK+b1Na16nE4LY5T9mt61Tim/u7uBzLfhyAzo0KP6qpt1xLfmxpteXywkN9/6m/MbItX9zbudDnFe4NaFXT6/rxuXLS7ig/5QgkEAgXCyf0clue7439phgXUyWfLWH2uG4kTB5C05q2J+5H+jThy/s683+9cwYWm9C/Geuf6cu6p/u6PUZ4iImhsbWZeU8n2tevTGRo3pYUuYNckxq2oPjaLW35c1x9osrZisoe7NXYsU2F8BD+3s/2Y2xVO2/T3+eGtOTx/s1QKqei/uY2tejYsArdm0Q7ttv/ys30d/rRm5XimcEtHO+fHNjc600hYfIQx2uTUnkCSdMa3vum9G1Rg/t7NuJvN+Z8tqfctCq75fq6jjof51ZnretE0atZTo/lFl6aQfvLvlcGea3zWTShV6GO+9nYnP5U0RWKXvHqXETqTVRkKDUruT5kvX/n9S7vH+/veS6O9kZ/I5OfRh+VQCBcVIoIzVOen5/ezWuQMHkITWoU/IZhNiluaGq76ax48iY+N55Ea1aKoJaH3IlSivfv7OC4+d7RuQGQt2jI7PSr8dSD/unBLfnh4R7MfagbAMPb1yX+5UFuP0uM01N+o+jyJEwewrS/dMzzlJY7HW/d1o5xva5zvH/4piZM/2ucIzfkjr0eJC6mCt8+2I3Gxrnnj+/JosdvzLP9Nw/k5EimjunAc0NbueR4GlYr76gXsosINVEpIpSEyUN4+Ka8I3ze28NWmR8ZVvgmi2O6NPC47okBeT9/u/qVuS2uHmFmU57Ols7sDxIA218cwJHXB+ebloTJQ+jToiaThrcGbEWZAJ1jqrLrpYFe++Q83r8Zi52u+8sjbMOF39Ut76gBzv74h22fu7vHEGJ2/Z6Emk30aOK5Dmr7CzkNM+wPMn7KEEgdgSg9GlQrR4NqBW9S+9yQljw1qDmhZhOTb23ruBH/PqEX//l9Hwt2nqZZzYocOud+FM/2uW44EW5yGL5oFF2exItXXJY9fXML6noIrI/1a0r3JtXYlHCRzo2qMnJazhAiw9vXJbZeZWKM6/Fo3yZMmLOdxtEVHJ/ZPlXqS8Na0/26aLa/OACT8pz+IbG1efgb2+umNSrw9OCWbrcrH267LUQYxUm9mlZn94lUMi2uw6r3aFKNr+/v6hguJbdvHuhC9+ui6d28Bj9uO0FqejbPDm5JzUrhHE+6Stt6UdwWV5/Or+UMuDi4TS0evNEWNGtVyvsgsOQfNzp62f/xjxsJCzERFRmaZztvBrWuxQs/7ubu7jG8MSqW8BAT4SFm5o+/gUk/7+Gz1UcAiH95ECEmxeK9Z+jXsiYhZhO7XhqIMq7R6E71CTWbSJg8xOM1aFy9Asue6E39quW4lJHNjNUJAFSMCKF/y5oMbF2LmInzaevUUANs9VBR5UI59Npgvt96guuql+e7zYlu+wAVBwkEoswzmRQRJtvNb3TnnCfQ66pX4IMxHQF45vudgP+eqACWPH6jY7DBu7o25I/4s9xyvfex6zvFVHU8md7RuQEr9ueMnulcz3DL9fW45fqcCvv7ejbiz3H1+Wj5IUeOyJcb4oyxnbhn5kZm3tuZShGu2+9+aSAbEpIc5320T1NCTIqHb2rChP7NXG52n/w1jp5NbTmy3JX2dt2vs63v36qmS1EZQOVytmKZGrlu9sPb51wv5xzdwNY1sVi1S71X4+rei8j+0rUBX607xjODW7jkyGpUinApfnP2wp9aOQKBPaAOapNTH1QhPOeWGWrOyfntemkgWmuOJ11l8JSVLse05ySjIkPZ/uIA9p1Oo1NMFUdO0jktsx7oSt3KkY4HIrNJMapjPZIuZwIwurNvRVEFJYFABIUnBzRHa/K9MbszoV8zujTOv8LX5HTjql+1nEtRgp23ooDXb23rc5qUUkRFhvLUoBb5b+zkphY1PN4Ey4eHcFPznBFKI8PMPD4gb91C7v1X/fMmTianM2LqampUDOdsmvehU3KbNLw1L/xoa3WVuzjwhqbRrDxwno/ucjtWmlcvDWtDy9qVfC7HLwp7gGhVJ5S4hlXYdDTvkNxgCwbeGg9089BcuWr5MI//b8VBRh8VooRczbQQYlYuT5JlSZNnFtCkRgUW/t19RW3ixStUjAhlyJSVnEy+yuHXfb9xrTt8geoVw/O0dEvPspB6NStPzsGdDUeSeO6Hnew/c4llT/R2qdMpCHvOp7A33myLlaHvrXLk2koLmZhGCFFk9ntFfk0Ys426hJAyGvCKGghKKxmGWghRZL62YS+rAcDuzVGxXoeYvxZJIBBCCCelqTinpJTt0C2EEKLIJBAIIUSQk0AghBBBTgKBEEIEOQkEQggR5CQQCCFEkJNAIIQQQU4CgRBCBLkyN8SEUuoccLSQu0cD54sxOWWVXAcbuQ455FrYXMvXoaHWurq7FWUuEBSFUmqTp7E2golcBxu5DjnkWtgE63WQoiEhhAhyEgiEECLIBVsgmB7oBJQSch1s5DrkkGthE5TXIajqCIQQQuQVbDkCIYQQuUggEEKIIBc0gUApNUgptU8pdVApNTHQ6SluSqn6SqmlSqm9SqndSqnHjOVVlVKLlFIHjH+rGMuVUmqKcT12KKU6OB3rbmP7A0qpuwP1mQpLKWVWSm1VSv1ivG+klFpvfJ45SqkwY3m48f6gsT7G6RhPG8v3KaUGBuaTFI1SqrJS6julVLzxvegWpN+HCcZvYpdSapZSKiJYvxMeaa2v+T/ADBwCGgNhwHagVaDTVcyfsTbQwXhdEdgPtAL+DUw0lk8E3jBeDwZ+BRTQFVhvLK8KHDb+rWK8rhLoz1fAa/E48A3wi/H+W2C08fpD4CHj9f8BHxqvRwNzjNetjO9IONDI+O6YA/25CnEdPgfuN16HAZWD7fsA1AWOAJFO34Wxwfqd8PQXLDmCzsBBrfVhrXUmMBsYHuA0FSut9Smt9RbjdRqwF9uPYDi2GwLGvyOM18OBL7TNOqCyUqo2MBBYpLVO0lpfBBYBg0rwoxSJUqoeMAT4xHivgD7Ad8Ymua+B/dp8B/Q1th8OzNZaZ2itjwAHsX2HygylVCWgF/ApgNY6U2udTJB9HwwhQKRSKgQoB5wiCL8T3gRLIKgLHHd6n2gsuyYZ2dnrgfVATa31KbAFC6CGsZmna1LWr9U7wFOA1XhfDUjWWmcb750/j+OzGutTjO3L+jUAW+73HDDDKCb7RClVniD7PmitTwBvAcewBYAUYDPB+Z3wKFgCgXKz7JpsN6uUqgDMBf6utU71tqmbZdrL8lJPKTUUOKu13uy82M2mOp91ZfYaOAkBOgDTtNbXA5exFQV5ck1eC6MOZDi24pw6QHngZjebBsN3wqNgCQSJQH2n9/WAkwFKi98opUKxBYGvtdbzjMVnjCw+xr9njeWerklZvlY9gGFKqQRsxX99sOUQKhvFAuD6eRyf1VgfBSRRtq+BXSKQqLVeb7z/DltgCKbvA0A/4IjW+pzWOguYB3QnOL8THgVLINgINDVaCoRhqwT6KcBpKlZGOeanwF6t9dtOq34C7C097gZ+dFr+V6O1SFcgxSgq+A0YoJSqYjxNDTCWlXpa66e11vW01jHY/o//0FqPAZYCo4zNcl8D+7UZZWyvjeWjjRYkjYCmwIYS+hjFQmt9GjiulGpuLOoL7CGIvg+GY0BXpVQ54zdivw5B953wKtC11SX1h61VxH5stf3PBjo9fvh8PbFlVXcAdbyDSQAAAppJREFU24y/wdjKN5cAB4x/qxrbK2CqcT12AnFOx7oXW2XYQeCeQH+2Ql6P3uS0GmqM7Ud7EPgfEG4sjzDeHzTWN3ba/1nj2uwDbg705ynkNWgPbDK+Ez9ga/UTdN8H4CUgHtgFfImt5U9Qfic8/ckQE0IIEeSCpWhICCGEBxIIhBAiyEkgEEKIICeBQAghgpwEAiGECHISCETQUkpdMv6NUUrdWczHfibX+zXFeXwhipMEAiEgBihQIFBKmfPZxCUQaK27FzBNQpQYCQRCwGTgBqXUNmPserNS6k2l1EZjbP4HAZRSvZVtzodvsHW6Qin1g1JqszHe/Thj2WRso11uU0p9bSyz5z6UcexdSqmdSqnbnY69TOXMH/C10RNWCL8LyX8TIa55E4EntNZDAYwbeorWupNSKhxYrZT63di2M9BG24YiBrhXa52klIoENiql5mqtJyqlHtFat3dzrlux9fhtB0Qb+6ww1l0PtMY2hs1qbGMnrSr+jyuEK8kRCJHXAGzj7mzDNpR3NWxjywBscAoCAOOVUtuBddgGJWuKdz2BWVpri9b6DLAc6OR07ESttRXbECExxfJphMiH5AiEyEsBj2qtXQZXU0r1xjacs/P7fkA3rfUVpdQybGPV5HdsTzKcXluQ36coIZIjEALSsE3vafcb8JAxrDdKqWbGpC65RQEXjSDQAtsUj3ZZ9v1zWQHcbtRDVMc2i9i1M4qlKJPkiUMI2+ic2UYRz0zgXWzFMluMCttz5Exl6Gwh8Del1A5sI1Kuc1o3HdihlNqibUNh230PdMM2/60GntJanzYCiRABIaOPCiFEkJOiISGECHISCIQQIshJIBBCiCAngUAIIYKcBAIhhAhyEgiEECLISSAQQogg9/8Rwhgejp49cwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(cost_list, label='Minibatch cost')\n",
    "plt.plot(np.convolve(cost_list, \n",
    "                     np.ones(200,)/200, mode='valid'), \n",
    "         label='Running average')\n",
    "\n",
    "plt.ylabel('Cross Entropy')\n",
    "plt.xlabel('Iteration')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deXxU5dn/8c+VhewkIQt72JFNNiMqIqJYFaqi1A1xw4ViaV2e+vzkqf5s61P7c3tabfWh4t66V8WtbhQpaBEwIEE2CWCAkAAh+55Mcv3+OEMImMAAOTNJ5nq/Xnll5sw5Z64chm/u3Oc+9xFVxRhjTPAICXQBxhhj/MuC3xhjgowFvzHGBBkLfmOMCTIW/MYYE2TCAl2AL5KTk7Vv376BLsMYY9qV1atX71fVlMOXt4vg79u3LxkZGYEuwxhj2hUR2dHccuvqMcaYIGPBb4wxQcaC3xhjgowFvzHGBBkLfmOMCTIW/MYYE2Qs+I0xJshY8BtjTFtUshs+ngdVxa2+63ZxAZcxxgSNsj3wxR9g9QugCv0nwUkXtupbWPAbY0xbUL4P/v0EfP0s1NfBmJlw1t2Q2KfV38qC3xhjAqmiAJY/AaueAU81jJoBE++GLv1de0sLfmOMCYTKQvjqSVj5NNRWwMlXwNn3QPJA19/agt8YY/ypqhhWzIcV/ws1pTB8OkyaBykn+a0EV4NfRO4AbgUEeEZVH2/y2t3Ao0CKqu53sw5jjAm46lKndf/Vn6G6BIZeDJP+C7oO93sprgW/iIzACf1xQC3wiYj8Q1WzRKQ38CNgp1vvb4wxbUJNOaxaAMv/BFVFcNJUp4XffVTASnKzxT8UWKGqlQAishS4DHgE+CPwf4D3XHx/Y4wJjNoKyP8Ovl8Ky5+Eyv0w6Hwn8HueEujqXA3+9cCDIpIEVAFTgQwRuQTYraqZItLixiIyG5gNkJaW5mKZxhhznOqqYf8WyN8M+zbCvs2QvwmKdgDqrNP/HDjnV9B7XEBLbcq14FfVTSLyMLAIKAcyAQ9wL3C+D9svABYApKenq1t1GmPMUXlqoXDboeG+bxMUbgdtcNYJCYOkQdBjDIy6BlKHOv33SQMCW3szXD25q6rPAc8BiMjvgb3ATOBAa78XsEZExqnqHjdrMe3Q9n/B2tdA6wNXQ0yKM8QuKiFwNbQn9XVOGO7zBmNRdmD//U6Upxr2b4WCLGjwOMskxBljnzrUGZGTOtT56jIAwjoFtl4fuT2qJ1VV94lIGjAdOENVn2jyejaQbqN6zCFqyuGfv3auYIxOgsj4wNVSvBO2LoZr3oAu/QJXR1vTUO+E+r5NB1u/+zY73R4Ndd6VBOJ7QWh4ICs9MSFhkDQQTppyMOCTBkF4ZKArOyFuj+N/29vHXwfMVdUil9/PtHc7lsO7tzl9pGf8HM69D8KjAlfP91/AG9fCs5Ph6lch7fTA1RIIDQ1QsuuwgN/kBLyn+uB6CWmQMhQG/ehgQCYPDuy/nWmRqLb97vP09HTNyMgIdBnGTXVV8Pnv4KunnLlJLp0PfcYHuirH/q3w6pVOAE57CkZeGeiKjqy2ErI+hfXvwI5/O63z4+WpAU/Vweede0LKkIPhnjLUufAoIvbE6zatTkRWq2r64cvtyl0TeDmr4d05Tisy/Wb40QNtK0iSB8It/4Q3roN3boX9Wc4ojSOMSvO7umrYusgJ+y2fQF0lxKTC4CnQKeb49xsaDsmDDga8nevoECz4TeB4amHpw/DlHyGuG1y3EAacG+iqmhfdxanvw7tg2SNQsBUu/d/AdmV4amHb57DhHdj8EdSWOedERl4FI6ZDnzMhJDRw9Zk2y4LfBMaeb2HhHNi7HkZfCxf+PrAncX0R1gmmPen8BfDP3zhdP1e/CrGp/quhvg62L/WG/YfOpf+RCTB8mjPCpN/ZEGr/rc2R2SfE+Fe9B/79R/jXwxCVCDNed0ZMtBciMOEuZ+jeO7PhmcnOiJ+uw9x7z3oP7PjS6cbZ9AFUFUJEZxjyYyfs+09qN8MITdtgwW/8J/87p5WfuwZG/ASmPuZ0obRHwy6BhN7w6tXw3PlwxYsw6LzW239DPez8yhv270NFPnSKdX5JDp8OAydDWETrvZ8JKhb8xn0N9c4UtIv/2znRePkLTh90e9djDNz6Obx2Fbx6BVz4MJw2+/j319AAOatgw0LY8C6U74GwKBh8gXO8Bp1vwyNNq7DgN+4q2Abv/gx2rYCTfgwXP+7fPnG3xfeEWZ84o30+/k/nCs8L/p/v/eyqsHuN02e/4V0ozYHQCGc8/IjpMPjCExuVY0wzLPjNQYdfbp+/CUpyTmyf+zZBSDhc9rQz2qQtDYFsLRGxcNXLsOh+545Khd/D5c9DZOfm11eFvExv2C90rg4OCXe6bybf73TntLStMa3Agj8Y+Xq5fZd+kNDnxIYEDr8MzrnXaRl3ZCGhcMGDzpj3f/wSnr/AOemb4J1ZVhX2bvB247zj/IINCXNOzJ49zzlRa2PkjZ9Y8LdlDQ1QU3Ji+6guPXTK2H0b7XJ7N51yIyT2hTeuh2fOhR//wRv47zjHXUKg30Q48w4Yekn7Pblt2jUL/rZAFUpzfzgfSv53UFfReu8T18MJ9X4TvZfdD7PL7d3Qf5Jzpe+rV8Cb1wHiXEx12k9h6DSITQlwgSbYWfD7kyqU7zs03PdtclrkNaUH14tJdQJ67HVOa1xCjv89w6OckE8ZYl0J/pQyGG5d4kwtnXYGdO4e6IqMaWTB76bKQudP/KYhX1V48PWoLk6re+SVB1vgqUPtz/+OIrpLxxi2ajocC3631JTBSxc7UxJExDuBPuwSpy/9QD96TErHHOVijGnTLPjd0FAPb9/itPCvedO58MYC3hjTRljwu2HR/c7UuFMfc666NMaYNuQEzhqaZq1+0bmIZ9xPYdytga7GGGN+wIK/NW1f6ly8M/A8uOD3ga7GGGOa5Wrwi8gdIrJeRDaIyJ3eZf8tIutEZK2IfCYiPdyswW/2b3XGbCcNci7XtznRjTFtlGvBLyIjgFuBccAo4CIRGQQ8qqojVXU08CFwv1s1+E1loXNP1pAwuOb1tn9DEWNMUHOzxT8UWKGqlarqAZYCl6lqkyuViAHa/t3ej8RTC29ef/BuTIl9A12RMcYckZvBvx6YKCJJIhINTAV6A4jIgyKyC5hJe27xq8I//gOyv4BLnoS00wNdkTHGHJVrwa+qm4CHgUXAJ0Am4PG+dq+q9gZeAX7e3PYiMltEMkQkIz8/360yT8xXT8I3f4Oz7oZRVwW6GmOM8YmrJ3dV9TlVHauqE4FCIOuwVV4FftLCtgtUNV1V01NS2uCkVt99DJ/9Xxg2zZl22Bhj2gm3R/Wker+nAdOB17wneA+4BNjsZg2u2PMtvHUz9BgNl/4FQmxUrDGm/XB7zOHbIpIE1AFzVbVIRJ4VkZOABmAHMMflGlpX2R7nBtuR8TDjdegUHeiKjDHmmLga/Kp6VjPLmu3aaRfqquD1a5wZNm/6BOK6BboiY4w5ZnaVka8aGuDd25wbY1/1MnQfFeiKjDHmuFjw+2rpQ879Us/7LQy9KNDVGGPMcbOzkr5Y93dY+jCMvta5V6oxxrRjFvxHs2sVvDfXuWfqRX+0efWNMe2eBf+R7PnWOZnbuYfTrx/WKdAVGWPMCbM+/sMVbHPuk7vhXee2iZHxzl207D64xpgOwoIfoGiHc+J2wzuQl+ks630aXPgQDJ8OcV0DW58xxrSi4A3+khynVb/hHdi92lnWYyyc/zsYdikk9A5sfcYY45LgCv6yPbDxPVj/Duxa4SzrNhLO+w0Mv8ymVDbGBIWOH/zl+bDpPVi/EHb8G1BIHQ7n3gfDLoPkgYGu0Bhj/KpjB/+i+2H5n0EbIHkwnH0PjJgOKScFujJjjAmYjh38PcbChP9wwj51mI3BN8YYOnrwD7/U+TLGGNPILuAyxpggY8FvjDFBxoLfGGOCjAW/McYEGQt+Y4wJMhb8xhgTZFwNfhG5Q0TWi8gGEbnTu+xREdksIutEZKGIJLhZgzHGmEO5FvwiMgK4FRgHjAIuEpFBwCJghKqOBLYA/+VWDcYYY37IzRb/UGCFqlaqqgdYClymqp95nwOsAHq5WIMxxpjDuBn864GJIpIkItHAVODwuY5vAj5ubmMRmS0iGSKSkZ+f72KZxhgTXFwLflXdBDyM07XzCZAJHGjpIyL3ep+/0sL2C1Q1XVXTU1JS3CrTGGOCjqsnd1X1OVUdq6oTgUIgC0BEbgAuAmaqqrpZgzHGmEO5OkmbiKSq6j4RSQOmA2eIyIXAPcDZqlrp5vsbY4z5Ibdn53xbRJKAOmCuqhaJyJNABLBInGmSV6jqHJfrMMYY4+Vq8KvqWc0ss1teGWNMANmVu8YYE2Qs+I0xJshY8BtjTJCx4DfGmCBjwW+MMUHGgt8YY4KMBb8xxgQZC35jjAkyFvzGGBNkLPiNMSbIHDX4ReTnIpLoj2KMMca4z5cWfzfgaxF5U0QuFO/MasYYY9qnowa/qt4HDAKeA24EskTk9yIywOXajDHGuMCnPn7vzVL2eL88QCLwlog84mJtxhhjXHDUaZlF5HbgBmA/8Czwn6paJyIhOHfU+j/ulmiMMaY1+TIffzIwXVV3NF2oqg0icpE7ZRljjHGLL109H+HcLxcAEYkTkdOg8Ybqxhhj2hFfgn8+UN7keYV3mTHGmHbIl+AX78ldwOniwcdbNorIHSKyXkQ2iMid3mVXeJ83iEj68ZVtjDHmePkS4Nu9J3gPtPJ/Bmw/2kYiMgK4FRgH1AKfiMg/gPXAdODp46rYGGM6KE99A7nF1WQXVLCjsJKdBRVce3of+iTFtOr7+BL8c4A/AfcBCiwGZvuw3VBghapWAojIUuAyVX3E+/y4CjbGGLdV19WTW1zF/vJaIsNDiIsMJy4yjLjIMCLCQk943zsLK9lRUMmOggrne6HzeHdRFZ6Gxg4WIsJCOHNgsv+DX1X3AVcfx77XAw+KSBJQBUwFMnzdWERm4/0Fk5aWdhxvb4wxP+Spb2BfWQ25xVXkllSTW1xFXtPHJdUUVtS2uH2n0JDGXwJxkeHERhx8fHC58zwiLITc4qpDwn1vac0h++scGUafpBhO7hnPRSO706dLDH2SoumTFENqXAQhIa3fSPZlHH8kcDMwHIg8sFxVbzrSdqq6SUQeBhbhnBzOxLn4yyequgBYAJCenq5HWd0YE+RUldIqD/nlNewvr6GgvJb8smrySqrZ7Q303OIq9pZW03BYosRFhtEjPoruCZGM6p1Aj/hIusdHkRIXQY2ngbLqOsqqPZTXeCg98Lja07h8Z2ElZdXOa+U1HvSw/afGRdAnKZoJA1PomxRNmjfY+yZFkxDdyX8HycuXrp6/AZuBC4AHgJmAT8M4VfU5nKkeEJHfAznHV6YxJhjVNyhFlbXsL69hf5n3e3kN+8ubPnZCvqC8ltr6hh/so1NYCN3jI+kRH8UZA5LomRBFd2/IO48jiYsMb7WaVZWK2nrKquuoqq2nW3wk0Z18Gg/jN75UM1BVrxCRaar6koi8Cnzqy85FJFVV94lIGs4J3TNOpFhjTPtV62mguLKWoso6iiprKa6spbDi4OOiyrqDr1fUUlRZS0lV3Q9a5wDhoUJybARJsZ1Ijo1gSLfOJMdGkOx9nhwbQXKc8zgpppNfzymKCLERYcRGtK2wb8qXyuq834u9I3X2AH193P/b3j7+OmCuqhaJyGXAn4EU4B8islZVLzjGuo0xbVRxZS0rthfy1bb9rN1VTEFFLUUVtVTU1re4TWR4CInRnUiI7kRidDhDe3QmMTqcxOhOhwR8cmwEKbERdI4KswEiJ8CX4F/gnY//PuB9IBb4v77sXFXPambZQmDhsRRpjGm7Kmo8fJ1dyFfbCli+rYD1uSWoQlR4KGPSEhiQEtsY6Akxnehy4HF0JxJjnHCPDD+xkTLm2Bwx+L0TsZWqahGwDOjvl6qMMW1Wjaeeb3YWs3xbAcu3Oq16T4PSKTSEMWkJ3Dl5MOMHJjGqVwKdwuwmf23REYPfOxHbz4E3/VSPMaaN8dQ3sD63lH9v3c9X2wr4OruQGk8DIQIn90rg1on9GT8gifQ+XYjqZC339sCXrp5FInI38AbOPD0AqGphy5sYYwKhoUEpqKglr6SK3OIq8stqqK1XGhoUT4NS39BAfQPUNzQ4z1Wpr3dea1Dv98Z1leLKWjKyiyircUZiD+kWx8zT+jB+QBLj+nehcyuOhjH+40vwHxivP7fJMsW6fYzxu7LqOnKLq8n1Bnte08cl1eQVVzc7pLE5YSFCaIgQFiKEeL+HhoQ0Lg8NEaI7hXLx6B6MH5DE6f2TSI6NcPknNP7gy5W7/fxRiDHGUVpdx+a8MjbmlrBlX7lzhak35A+0vA8IDRG6dY6ke3wko3olcOEIZ7x6D+/49NTOEUSEhhIa6g14ORj0Jnj5cuXu9c0tV9W/tn45xgQPVSW3pJqNuaXOV14JG/NK2VVY1bhOQnQ4vRKj6JMUw/gByc6FSAlR9EhwvqfERhAWaidQzbHxpavn1CaPI4HJwBrAgt8YH9V6Gti6r5yNeU1CPreU0mqnBS8C/ZJiGNkrgatPTWNY984M69GZ1LgIG69uWp0vXT2/aPpcROJxpnEwxjShquSX17CrsIqcokp2FVayfX8Fm/LK2LqvjLp65xLUyPAQhnTrzEWjejCse2eGdu/MkG5xxLThKz1Nx3I8n7RKYFBrF2JMe1BSVceuwkpvsFexyxvwu4qcsK+uO/TEatfOznQCk05KaWzF902KIdT62E0A+dLH/wHOKB5w7tg1DBvXbzqwA33vq3cUsWF3CTsLK9lZ6AT8ga6ZA+Iiw+idGM2AlBgmDU6hd5do0rpE07tLFL0So+2KVNMm+dLif6zJYw+wQ1Vtlk3TYdR6GtiYV8rqHUWs2VHE6h1F7CmtBpyZHXslRtE7MZoxaQn0TjwQ7NH0TowmPtrGsZv2x5fg3wnkqWo1gIhEiUhfVc12tTJjXFJYUesE/E4n5DN3FVPjcbpoeiZEMa5fF07pk8gpfRIZ0i3ORs2YDseX4P87ML7J83rvslObX92YtqOhQdmWX06GtyW/ZkcR2/c7F6CHhQjDe8Yz87Q+pPdNZGxaIt3iI4+yR2PaP1+CP0xVG+9Dpqq1IuL/W8YY44M9JdVk5hSzLqeYdTklZO4qbuyX7xLTibFpiVyR3ptT+iQysle89cGboORL8OeLyCWq+j6AiEwD9rtbljFHV1RRy7rdJazbVUxmTgnrcorZV+bczzQ0RDipaxw/Htmjsdumb1K0jYk3Bt+Cfw7wiog86X2eAzR7Na8xbqmo8fDt7pLGlvy6HGe0zQH9U2I4c2AyI3vFM7JXAsN7dLbWvDEt8OUCrm3A6SISC4iqlrlflglmtZ4GvttTxtqcYjJ3OV9b88sbb2DdMyGKkb3imTEujVG94hnRK95miTTmGPgyjv/3wCOqWux9ngj8UlXvc7s40/GpKjsLK1m7q5i13pBfn1tKrXeUTVJMJ0b2imfqyd0Z1dtpzdsMkcacGF+6eqao6q8OPPHeN3cqzq0Yj0hE7gBuBQR4RlUfF5EuOHP79wWygSu9d/gyQaCwopbMnGLW7iwm09uiL6p0buscGR7CyT3jueGMPozqncCoXgn0SoyyfnljWpkvwR8qIhGqWgPOOH7gqE0u743ZbwXGAbXAJyLyD++yxar6kIjMA+YB9xzvD2DarlpPA9/uLmlsya/dVdzYLy8Cg1PjOH9YN0b1TmB07wQGd421MfPG+IEvwf8ysFhEXvA+nwW85MN2Q4EVqloJICJLgcuAacAk7zovAf/Cgr9DUFWyCypZtiWfZVvy+Wp7AZW19QB0j49kdO8ErjktjdG9ExjRM55Ym5TMmIDw5eTuIyKyDjgPp8vmE6CPD/teDzwoIklAFTAVyAC6qmqed995IpLa3MYiMhuYDZCWlubD25lAKK2uY/nW/SzL2s+yLfnkFDlzyfdJimb62J5MGJjMmLREuna2C6OMaSt8bXLtARqAK4HvgbePtoGqbhKRh4FFQDmQiTPXj09UdQGwACA9PV2Psrrxk/oGZV1OMcu27OeLrHy+2VVMfYMSGxHGGQOS+OnZA5g4KJk+STGBLtUY04IWg19EBgNXAzOAApwTsqKq5/i6c1V9DnjOu7/f41wDsFdEuntb+92BfSdQv/GD3OIqlm3J54us/Xy5dT8lVXWIwMk947nt7AFMHJzCmLQEwq1/3ph24Ugt/s3AF8DFqroVQETuOpadi0iqqu4TkTRgOnAG0A+4AXjI+/294yncuGvznlLeX5vLZxv3snVfOeDMLX/+sK6cNTiFCQOT6RJjM3cY0x4dKfh/gtPiXyIinwCv4/TxH4u3vX38dcBc71DQh4A3ReRmnJk/rziOuo0LdhZU8n7mbt7PzGXL3nJCQ4TT+3fh6lN7c9agFAZ3jbWhlcZ0AC0Gv6ouBBaKSAxwKXAX0FVE5gMLVfWzo+1cVc9qZlkBzn17TRuwr7SaD9bl8X5mLpm7igFI75PIA9OGM/Xk7naxlDEdkC+jeiqAV3Dm6+mC00KfBxw1+E3bVFxZy8fr9/D+2lxWfF+AKgzr3pl5U4Zw0cju9EqMDnSJxhgXHdNAalUtBJ72fpl2pKLGwz837eX9tbksy8qnrl7plxzDL84dxCWjejAwNTbQJRpj/MSuoOnA6uob+Nd3+by3djeLN+2jqq6e7vGR3Di+L5eM6smInp2tz96YIGTB3wE1NCgfrMvlD4u2sKOgksTocKaP7cklo3pwat8uhIRY2BsTzCz4OxBVZcl3+3j00y1syitlSLc4nr7uFM4dkmpj7I0xjSz4O4hV3xfy6Keb+Tq7iD5J0Txx9WguHtnDWvfGmB+w4G/nNuSW8Nin37Hku3xS4yJ48LIRXJne21r4xpgWWfC3U9n7K/ifRVv4IDOX+Khw5k0Zwg1n9CWqk91u0BhzZBb87cze0mqeWJzFm1/vIjw0hLnnDGD2xAHER9mtB40xvrHgbyeKK2uZ/69tvLg8mwZVZp6WxtxzB5IaZ9MdG2OOjQV/G1dR4+GFf3/P08u2U17j4bLRPbnrR4Pp3cWurjXGHB8L/jbs0w17uHfhevaX13De0K7cfcFghnTrHOiyjDHtnAV/G/XKyh3c9+56Tu4Zz9PXncIpfRIDXZIxpoOw4G9jVJUnP9/K/yzawrlDUnnqmrE2UscY06os+NuQhgblgQ838uLybKaP6cnDl4+08fjGmFZnwd9G1Hoa+OXfM/kgM5dbJvTjV1OH2lW3xhhXWPC3ARU1Hua8vJovsvYzb8oQfjqxv82aaYxxjQV/gBVW1DLrxa/5NqeYR34ykitP7R3okowxHZyrHcgicpeIbBCR9SLymohEisi5IrLGu+wlEQnaXz67i6u4/C/L2ZxXyl+uPcVC3xjjF64Fv4j0BG4H0lV1BBAKXAO8BFztXbYDuMGtGtqyrL1lXD5/OfllNfz1pnGcP7xboEsyxgQJt4eMhAFR3lZ9NFAB1KjqFu/ri4CfuFxDm7NmZxFXPP0VngbljdlncFr/pECXZIwJIq4Fv6ruBh4DdgJ5QAnwJhAuIune1S4Hmu3fEJHZIpIhIhn5+flulel3S77bx8xnVhIfFc7bc8YzrIddiWuM8S83u3oSgWlAP6AHEAPMBK4G/igiq4AywNPc9qq6QFXTVTU9JSXFrTL96t1vdnPrSxn0S47hrTnjSUuy+XaMMf7n5onV84DvVTUfQETeAcar6svAWd5l5wODXayhzXj+y+954MONnN6/CwuuT6dzpE2jbIwJDDf7+HcCp4tItDiD0icDm0QkFUBEIoB7gL+4WEPAqSqPfrqZBz7cyIXDu/HirHEW+saYgHKtxa+qK0XkLWANTnfON8AC4HcichHOL535qvq5WzUEmqe+gfveXc/rX+9ixrg0fnfpCELtalxjTICJqga6hqNKT0/XjIyMQJdxTKrr6rnj9W/4dMNefnHuQP7jR4PtalxjjF+JyGpVTT98edBePOWmsuo6Zv91NV9tL+D+i4Zx04R+gS7JGGMaWfC3sv3lNdz4wio255Xx+FWjuXRMz0CXZIwxh7Dgb0U5RZVc99wq8kqqeOb6dM4Zkhrokowx5gcs+FvJlr1lXPfcSqpq63n55tNI79sl0CUZY0yzLPhbweodRdz04tdEhIXw5pwz7L64xpg2zYL/BC3dks+cv60mtXMEL998Gr272NW4xpi2zYL/BLyfmcsv31zLoNQ4XrppHClxEYEuyRhjjsqC/zj99atsfv3+Bk7t24Vnb7ApGIwx7YcF/zFSVR7/ZxZPLM7ivKFdefKaMUSGhwa6LGOM8ZkF/zFoaFB+88EG/vrVDi4/pRcPTT+ZsFC3b2lgjDGty4LfR7WeBu7+eybvZ+Yye2J//mvKEJuCwRjTLlnw+6Cy1sOcl9ewbEs+86YMYc7ZAwJdkjHGHDcL/qMorqxl1otfk7mrmEd+MtJuiG6Mafcs+I9gb2k11z23kuyCSuZfewoX2A3RjTEdgAX/Edz37npyiqp4cdapjB+QHOhyjDGmVdiQlBZs3lPKoo17mT2xv4W+MaZDseBvwfx/bSOmUyg3ju8b6FKMMaZVWfA3I3t/BR9k5nLt6X1IiO4U6HKMMaZVuRr8InKXiGwQkfUi8pqIRIrIZBFZIyJrReRLERnoZg3H4+ll2wgLDeFmu3OWMaYDci34RaQncDuQrqojgFDgamA+MFNVRwOvAve5VcPx2FNSzVurc7gqvTepnSMDXY4xxrQ6t0f1hAFRIlIHRAO5gAIHJqyP9y5rMxYs206DwuyJ/QNdijEdTl1dHTk5OVRXVwe6lA4lMjKSXr16ER7u22SRrgW/qu4WkceAnUAV8JmqfiYitwAfiUgVUAqc3tz2IjIbmA2QlpbmVpmHKCiv4bVVO7l0dE+bV98YF+Tk5BAXF0ffvn1typNWoqoUFBSQk5NDv36+dU+72dWTCEwD+gE9gBgRuRa4C5iqqr2AF4A/NLe9qi5Q1XRVTU9JSXGrzEO88O9sqj313DbJWvvGuKG6upqkpCQL/VYkIrZvFt4AAA/xSURBVCQlJR3TX1Funtw9D/heVfNVtQ54BzgTGKWqK73rvAGMd7EGn5VW1/HSV9lcOLwbA1PjAl2OMR2WhX7rO9Zj6mbw7wROF5FocaqaDGwE4kVksHedHwGbXKzBZy+v2EFZtYe557S5QUbGGNOq3OzjXykibwFrAA/wDbAAyAHeFpEGoAi4ya0afFVVW89zX3zP2YNTGNEzPtDlGGNcUlBQwOTJkwHYs2cPoaGhHOhKXrVqFZ06Hf26nVmzZjFv3jxOOumkFtd56qmnSEhIYObMma1TeCtzdVSPqv4a+PVhixd6v9qMN77eSUFFLT8/11r7xnRkSUlJrF27FoDf/OY3xMbGcvfddx+yjqqiqoSENN8h8sILLxz1febOnXvixboo6Cdpq/U08PSy7Yzr24VT+3YJdDnGBI3ffrCBjbmlrbrPYT068+uLhx/zdlu3buXSSy9lwoQJrFy5kg8//JDf/va3rFmzhqqqKq666iruv/9+ACZMmMCTTz7JiBEjSE5OZs6cOXz88cdER0fz3nvvkZqayn333UdycjJ33nknEyZMYMKECXz++eeUlJTwwgsvMH78eCoqKrj++uvZunUrw4YNIysri2effZbRo0e36jFpTtBP2fDuN7vJK6nmZ+fYzVWMCWYbN27k5ptv5ptvvqFnz5489NBDZGRkkJmZyaJFi9i4ceMPtikpKeHss88mMzOTM844g+eff77Zfasqq1at4tFHH+WBBx4A4M9//jPdunUjMzOTefPm8c0337j68zUV1C3++gZl/tJtjOjZmbMH+2fIqDHGcTwtczcNGDCAU089tfH5a6+9xnPPPYfH4yE3N5eNGzcybNiwQ7aJiopiypQpAJxyyil88cUXze57+vTpjetkZ2cD8OWXX3LPPfcAMGrUKIYP99/xCOrg/+jbPL7fX8H8mWNtiJkxQS4mJqbxcVZWFk888QSrVq0iISGBa6+9ttlx8k1PBoeGhuLxeJrdd0RExA/WUdXWLP+YBG1Xj6ry1JKtDEiJsTtrGWMOUVpaSlxcHJ07dyYvL49PP/201d9jwoQJvPnmmwB8++23zXYluSVoW/yfb97H5j1l/M8VowgJsda+MeagsWPHMmzYMEaMGEH//v0588wzW/09fvGLX3D99dczcuRIxo4dy4gRI4iP989wcgnknxu+Sk9P14yMjFbbn6oyff5y8stqWHL3JMJDg/YPH2P8atOmTQwdOjTQZbQJHo8Hj8dDZGQkWVlZnH/++WRlZREWdnzt8eaOrYisVtX0w9cNyhb/iu2FfLOzmP++dISFvjEmIMrLy5k8eTIejwdV5emnnz7u0D9WQRn8Ty3ZSnJsBFec0ivQpRhjglRCQgKrV68OyHsHXXN37a5ivty6n1vP6kdkeGigyzHGGL8LuuB/aslW4qPCmXl6n0CXYowxARFUwf/dnjIWbdzLjeP7EhsRlL1cxhgTXME//19bie4Uyqwz+wa6FGOMCZigCf4dBRW8n5nLtaf3ISH66FOvGmM6nkmTJv3gYqzHH3+cn/3sZy1uExsbC0Bubi6XX355i/s92pDzxx9/nMrKysbnU6dOpbi42NfSW1XQBP9flm4nLDSEWyb4dk9KY0zHM2PGDF5//fVDlr3++uvMmDHjqNv26NGDt95667jf+/Dg/+ijj0hISDju/Z2IoOjo3lNSzdurc7jy1F6kdo4MdDnGGICP58Geb1t3n91OhikPtfjy5Zdfzn333UdNTQ0RERFkZ2eTm5vL6NGjmTx5MkVFRdTV1fG73/2OadOmHbJtdnY2F110EevXr6eqqopZs2axceNGhg4dSlVVVeN6t912G19//TVVVVVcfvnl/Pa3v+VPf/oTubm5nHPOOSQnJ7NkyRL69u1LRkYGycnJ/OEPf2ic2fOWW27hzjvvJDs7mylTpjBhwgSWL19Oz549ee+994iKijrhwxQULf5nvthOvSo/nWhTLxsTzJKSkhg3bhyffPIJ4LT2r7rqKqKioli4cCFr1qxhyZIl/PKXvzziJGrz588nOjqadevWce+99x4yHv/BBx8kIyODdevWsXTpUtatW8ftt99Ojx49WLJkCUuWLDlkX6tXr+aFF15g5cqVrFixgmeeeaZxiuasrCzmzp3Lhg0bSEhI4O23326V49DhW/yFFbW8unIn00b3oHeX6ECXY4w54Agtczcd6O6ZNm0ar7/+Os8//zyqyq9+9SuWLVtGSEgIu3fvZu/evXTr1vwEjsuWLeP2228HYOTIkYwcObLxtTfffJMFCxbg8XjIy8tj48aNh7x+uC+//JLLLruscXbQ6dOn88UXX3DJJZfQr1+/xhuzNJ3S+US52uIXkbtEZIOIrBeR10QkUkS+EJG13q9cEXnXzRpe+Pf3VHvq+dkka+0bY+DSSy9l8eLFjXfXGjt2LK+88gr5+fmsXr2atWvX0rVr12anYW6quancv//+ex577DEWL17MunXr+PGPf3zU/RzpL4sD0znDkad9PlauBb+I9ARuB9JVdQQQClytqmep6mhVHQ18BbzjVg1l1XW8uDybC4Z1Y2BqnFtvY4xpR2JjY5k0aRI33XRT40ndkpISUlNTCQ8PZ8mSJezYseOI+5g4cSKvvPIKAOvXr2fdunWAM51zTEwM8fHx7N27l48//rhxm7i4OMrKyprd17vvvktlZSUVFRUsXLiQs846q7V+3Ga53dUTBkSJSB0QDeQeeEFE4oBzgVluvfnfVuygrNrD3HPsJurGmINmzJjB9OnTG0f4zJw5k4svvpj09HRGjx7NkCFDjrj9bbfdxqxZsxg5ciSjR49m3LhxgHMnrTFjxjB8+PAfTOc8e/ZspkyZQvfu3Q/p5x87diw33nhj4z5uueUWxowZ02rdOs1xdVpmEbkDeBCoAj5T1ZlNXrseuERVmx0YKyKzgdkAaWlppxztN3Bz/p6xi6+zC3nk8lHHU74xppXZtMzuOZZpmd3s6kkEpgH9gB5AjIhc22SVGcBrLW2vqgtUNV1V01NSju9+uFek97bQN8aYw7h5cvc84HtVzVfVOpy+/PEAIpIEjAP+4eL7G2OMaYabwb8TOF1EosU5/T0Z2OR97QrgQ1U98uluY0yH0x7u+tfeHOsxdS34VXUl8BawBvjW+14LvC9fzRG6eYwxHVNkZCQFBQUW/q1IVSkoKCAy0vdZCVwd1aOqvwZ+3czySW6+rzGmberVqxc5OTnk5+cHupQOJTIykl69fL+jYIe/ctcY03aEh4fTr59NlBhoQTFXjzHGmIMs+I0xJshY8BtjTJBx9crd1iIi+cCxX7rrH8nA/kAXcQRW34mx+k6M1XfiTqTGPqr6gytg20Xwt2UiktHcJdFthdV3Yqy+E2P1nTg3arSuHmOMCTIW/MYYE2Qs+E/cgqOvElBW34mx+k6M1XfiWr1G6+M3xpggYy1+Y4wJMhb8xhgTZCz4fSAivUVkiYhs8t48/o5m1pkkIiVNbiR/v59rzBaRb73vndHM6yIifxKRrSKyTkTG+rG2k5ocl7UiUioidx62jl+Pn4g8LyL7RGR9k2VdRGSRiGR5vye2sO0N3nWyROQGP9b3qIhs9v77LRSRhBa2PeJnwcX6fiMiu5v8G05tYdsLReQ772dxnh/re6NJbdkisraFbf1x/JrNFL99BlXVvo7yBXQHxnofxwFbgGGHrTMJ5x4DgaoxG0g+wutTgY8BAU4HVgaozlBgD86FJQE7fsBEYCywvsmyR4B53sfzgIeb2a4LsN37PdH7ONFP9Z0PhHkfP9xcfb58Flys7zfA3T78+28D+gOdgMzD/y+5Vd9hr/8PcH8Aj1+zmeKvz6C1+H2gqnmqusb7uAznhjI9A1vVMZsG/FUdK4AEEekegDomA9tUNaBXYqvqMqDwsMXTgJe8j18CLm1m0wuARapaqKpFwCLgQn/Up6qfqarH+3QF4Ps8vK2shePni3HAVlXdrqq1wOs4x71VHak+742hriSA9wQ5Qqb45TNowX+MRKQvMAZY2czLZ4hIpoh8LCLD/VoYKPCZiKz23qj+cD2BXU2e5xCYX15HuglPII8fQFdVzQPnPyaQ2sw6beU43oTzF1xzjvZZcNPPvV1Rz7fQTdEWjt9ZwF5VzWrhdb8ev8MyxS+fQQv+YyAiscDbwJ2qWnrYy2twui9GAX8G3vVzeWeq6lhgCjBXRCYe9ro0s41fx/KKSCfgEuDvzbwc6OPnq7ZwHO8FPMArLaxytM+CW+YDA4DRQB5Od8rhAn78gBkcubXvt+N3lExpcbNmlh3TMbTg95GIhOP8A72iqu8c/rqqlqpquffxR0C4iCT7qz5VzfV+3wcsxPmTuqkcoHeT572AXP9U12gKsEZV9x7+QqCPn9feA91f3u/7mlknoMfReyLvImCmejt8D+fDZ8EVqrpXVetVtQF4poX3DfTxCwOmA2+0tI6/jl8LmeKXz6AFvw+8fYLPAZtU9Q8trNPNux4iMg7n2Bb4qb4YEYk78BjnJOD6w1Z7H7jeO7rndKDkwJ+UftRiSyuQx6+J94EDIyRuAN5rZp1PgfNFJNHblXG+d5nrRORC4B7gElWtbGEdXz4LbtXX9JzRZS2879fAIBHp5/0L8Gqc4+4v5wGbVTWnuRf9dfyOkCn++Qy6eea6o3wBE3D+lFoHrPV+TQXmAHO86/wc2IAzSmEFMN6P9fX3vm+mt4Z7vcub1ifAUzgjKr4F0v18DKNxgjy+ybKAHT+cX0B5QB1OC+pmIAlYDGR5v3fxrpsOPNtk25uArd6vWX6sbytO3+6Bz+BfvOv2AD460mfBT/X9zfvZWocTYN0Pr8/7fCrOKJZt/qzPu/zFA5+5JusG4vi1lCl++QzalA3GGBNkrKvHGGOCjAW/McYEGQt+Y4wJMhb8xhgTZCz4jTEmyFjwGwOISL0cOoNoq80aKSJ9m84SaUyghQW6AGPaiCpVHR3oIozxB2vxG3ME3rnZHxaRVd6vgd7lfURksXdCssUikuZd3lWcufIzvV/jvbsKFZFnvHOvfyYiUQH7oUzQs+A3xhF1WFfPVU1eK1XVccCTwOPeZU/iTHM9EmeytD95l/8JWKrOZHNjca7+BBgEPKWqw4Fi4Ccu/zzGtMiu3DUGEJFyVY1tZnk2cK6qbvdOqrVHVZNEZD/OlAR13uV5qposIvlAL1WtabKPvjjzpw/yPr8HCFfV37n/kxnzQ9biN+botIXHLa3TnJomj+ux82smgCz4jTm6q5p8/8r7eDnOzJIAM4EvvY8XA7cBiEioiHT2V5HG+MpaHcY4ouTQm29/oqoHhnRGiMhKnIbSDO+y24HnReQ/gXxglnf5HcACEbkZp2V/G84skca0GdbHb8wRePv401V1f6BrMaa1WFePMcYEGWvxG2NMkLEWvzHGBBkLfmOMCTIW/MYYE2Qs+I0xJshY8BtjTJD5/1CgvZIM4ZTSAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.arange(1, NUM_EPOCHS+1), train_acc_list, label='Training')\n",
    "plt.plot(np.arange(1, NUM_EPOCHS+1), valid_acc_list, label='Validation')\n",
    "\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation ACC: 94.30%\n",
      "Test ACC: 91.63%\n"
     ]
    }
   ],
   "source": [
    "with torch.set_grad_enabled(False):\n",
    "    test_acc = compute_acc(model=model,\n",
    "                           data_loader=test_loader,\n",
    "                           device=DEVICE)\n",
    "    \n",
    "    valid_acc = compute_acc(model=model,\n",
    "                            data_loader=valid_loader,\n",
    "                            device=DEVICE)\n",
    "    \n",
    "\n",
    "print(f'Validation ACC: {valid_acc:.2f}%')\n",
    "print(f'Test ACC: {test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torchvision 0.4.0a0+6b959ee\n",
      "torch       1.2.0\n",
      "numpy       1.16.4\n",
      "matplotlib  3.1.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%watermark -iv"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}