pytorch-lightning_ipynb/cnn/cnn-lenet5-cifar10.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1212b481",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch            : 1.10.1\n",
      "pytorch_lightning: 1.6.0.dev0\n",
      "torchmetrics     : 0.6.2\n",
      "matplotlib       : 3.3.4\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -p torch,pytorch_lightning,torchmetrics,matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4775e637",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext pycodestyle_magic\n",
    "%flake8_on --ignore W291,W293,E703"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "074f80cd",
   "metadata": {},
   "source": [
    "<a href=\"https://pytorch.org\"><img src=\"https://raw.githubusercontent.com/pytorch/pytorch/master/docs/source/_static/img/pytorch-logo-dark.svg\" width=\"90\"/></a> &nbsp; &nbsp;&nbsp;&nbsp;<a href=\"https://www.pytorchlightning.ai\"><img src=\"https://raw.githubusercontent.com/PyTorchLightning/pytorch-lightning/master/docs/source/_static/images/logo.svg\" width=\"150\"/></a>\n",
    "\n",
    "# Model Zoo -- LeNet-5 Trained on CIFAR-10"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14838d26",
   "metadata": {},
   "source": [
    "This notebook implements the classic LeNet-5 convolutional network [1] and applies it to MNIST digit classification. The basic architecture is shown in the figure below:\n",
    "\n",
    "![](../../pytorch_ipynb/images/lenet/lenet-5_1.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "daea9947",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "LeNet-5 is commonly regarded as the pioneer of convolutional neural networks, consisting of a very simple architecture (by modern standards). In total, LeNet-5 consists of only 7 layers. 3 out of these 7 layers are convolutional layers (C1, C3, C5), which are connected by two average pooling layers (S2 & S4). The penultimate layer is a fully connexted layer (F6), which is followed by the final output layer. The additional details are summarized below:\n",
    "\n",
    "- All convolutional layers use 5x5 kernels with stride 1.\n",
    "- The two average pooling (subsampling) layers are 2x2 pixels wide with stride 1.\n",
    "- Throughrout the network, tanh sigmoid activation functions are used. (**In this notebook, we replace these with ReLU activations**)\n",
    "- The output layer uses 10 custom Euclidean Radial Basis Function neurons for the output layer. (**In this notebook, we replace these with softmax activations**)\n",
    "\n",
    "### References\n",
    "\n",
    "- [1] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. [Gradient-based learning applied to document recognition](https://ieeexplore.ieee.org/document/726791). Proceedings of the IEEE, november 1998."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab60f759",
   "metadata": {},
   "source": [
    "## General settings and hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dd68213",
   "metadata": {},
   "source": [
    "- Here, we specify some general hyperparameter values and general settings\n",
    "- Note that for small datatsets, it is not necessary and better not to use multiple workers as it can sometimes cause issues with too many open files in PyTorch. So, if you have problems with the data loader later, try setting `NUM_WORKERS = 0` instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1484a931",
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 128\n",
    "NUM_EPOCHS = 20\n",
    "LEARNING_RATE = 0.001\n",
    "NUM_WORKERS = 4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d84a1eb",
   "metadata": {},
   "source": [
    "## Implementing a Neural Network using PyTorch Lightning's `LightningModule`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18717c08",
   "metadata": {},
   "source": [
    "- In this section, we set up the main model architecture using the `LightningModule` from PyTorch Lightning.\n",
    "- We start with defining our neural network  model in pure PyTorch, and then we use it in the `LightningModule` to get all the extra benefits that PyTorch Lightning provides."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9407f94a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "\n",
    "class PyTorchLeNet5(torch.nn.Module):\n",
    "\n",
    "    def __init__(self, num_classes, grayscale=False):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.grayscale = grayscale\n",
    "        self.num_classes = num_classes\n",
    "\n",
    "        if self.grayscale:\n",
    "            in_channels = 1\n",
    "        else:\n",
    "            in_channels = 3\n",
    "\n",
    "        self.features = torch.nn.Sequential(\n",
    "            \n",
    "            torch.nn.Conv2d(in_channels, 6*in_channels, kernel_size=5),\n",
    "            torch.nn.Tanh(),\n",
    "            torch.nn.MaxPool2d(kernel_size=2),\n",
    "            torch.nn.Conv2d(6*in_channels, 16*in_channels, kernel_size=5),\n",
    "            torch.nn.Tanh(),\n",
    "            torch.nn.MaxPool2d(kernel_size=2)\n",
    "        )\n",
    "\n",
    "        self.classifier = torch.nn.Sequential(\n",
    "            torch.nn.Linear(16*5*5*in_channels, 120*in_channels),\n",
    "            torch.nn.Tanh(),\n",
    "            torch.nn.Linear(120*in_channels, 84*in_channels),\n",
    "            torch.nn.Tanh(),\n",
    "            torch.nn.Linear(84*in_channels, num_classes),\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.features(x)\n",
    "        x = torch.flatten(x, start_dim=1)\n",
    "        logits = self.classifier(x)\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ea5f6167",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pytorch_lightning as pl\n",
    "import torchmetrics\n",
    "\n",
    "\n",
    "# LightningModule that receives a PyTorch model as input\n",
    "class LightningModel(pl.LightningModule):\n",
    "    def __init__(self, model, learning_rate):\n",
    "        super().__init__()\n",
    "\n",
    "        self.learning_rate = learning_rate\n",
    "        # The inherited PyTorch module\n",
    "        self.model = model\n",
    "\n",
    "        # Save settings and hyperparameters to the log directory\n",
    "        # but skip the model parameters\n",
    "        self.save_hyperparameters(ignore=['model'])\n",
    "\n",
    "        # Set up attributes for computing the accuracy\n",
    "        self.train_acc = torchmetrics.Accuracy()\n",
    "        self.valid_acc = torchmetrics.Accuracy()\n",
    "        self.test_acc = torchmetrics.Accuracy()\n",
    "        \n",
    "    # Defining the forward method is only necessary \n",
    "    # if you want to use a Trainer's .predict() method (optional)\n",
    "    def forward(self, x):\n",
    "        return self.model(x)\n",
    "        \n",
    "    # A common forward step to compute the loss and labels\n",
    "    # this is used for training, validation, and testing below\n",
    "    def _shared_step(self, batch):\n",
    "        features, true_labels = batch\n",
    "        logits = self(features)\n",
    "        loss = torch.nn.functional.cross_entropy(logits, true_labels)\n",
    "        predicted_labels = torch.argmax(logits, dim=1)\n",
    "\n",
    "        return loss, true_labels, predicted_labels\n",
    "\n",
    "    def training_step(self, batch, batch_idx):\n",
    "        loss, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.log(\"train_loss\", loss)\n",
    "        \n",
    "        # To account for Dropout behavior during evaluation\n",
    "        self.model.eval()\n",
    "        with torch.no_grad():\n",
    "            _, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.train_acc.update(predicted_labels, true_labels)\n",
    "        self.log(\"train_acc\", self.train_acc, on_epoch=True, on_step=False)\n",
    "        self.model.train()\n",
    "        return loss  # this is passed to the optimzer for training\n",
    "\n",
    "    def validation_step(self, batch, batch_idx):\n",
    "        loss, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.log(\"valid_loss\", loss)\n",
    "        self.valid_acc(predicted_labels, true_labels)\n",
    "        self.log(\"valid_acc\", self.valid_acc,\n",
    "                 on_epoch=True, on_step=False, prog_bar=True)\n",
    "\n",
    "    def test_step(self, batch, batch_idx):\n",
    "        loss, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.test_acc(predicted_labels, true_labels)\n",
    "        self.log(\"test_acc\", self.test_acc, on_epoch=True, on_step=False)\n",
    "\n",
    "    def configure_optimizers(self):\n",
    "        optimizer = torch.optim.Adam(self.parameters(), lr=self.learning_rate)\n",
    "        return optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19d96f93",
   "metadata": {},
   "source": [
    "## Setting up the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c86ee414",
   "metadata": {},
   "source": [
    "- In this section, we are going to set up our dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53623b3b",
   "metadata": {},
   "source": [
    "### Inspecting the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "85c2233b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "from torchvision import datasets\n",
    "from torchvision import transforms\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "\n",
    "train_dataset = datasets.CIFAR10(root='./data', \n",
    "                                 train=True, \n",
    "                                 transform=transforms.ToTensor(),\n",
    "                                 download=True)\n",
    "\n",
    "train_loader = DataLoader(dataset=train_dataset, \n",
    "                          batch_size=BATCH_SIZE, \n",
    "                          num_workers=NUM_WORKERS,\n",
    "                          drop_last=True,\n",
    "                          shuffle=True)\n",
    "\n",
    "test_dataset = datasets.CIFAR10(root='./data', \n",
    "                                train=False,\n",
    "                                transform=transforms.ToTensor())\n",
    "\n",
    "test_loader = DataLoader(dataset=test_dataset, \n",
    "                         batch_size=BATCH_SIZE,\n",
    "                         num_workers=NUM_WORKERS,\n",
    "                         drop_last=False,\n",
    "                         shuffle=False)\n",
    "\n",
    "# Checking the dataset\n",
    "all_train_labels = []\n",
    "all_test_labels = []\n",
    "\n",
    "for images, labels in train_loader:  \n",
    "    all_train_labels.append(labels)\n",
    "all_train_labels = torch.cat(all_train_labels)\n",
    "    \n",
    "for images, labels in test_loader:  \n",
    "    all_test_labels.append(labels)\n",
    "all_test_labels = torch.cat(all_test_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "48007518",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training labels: tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])\n",
      "Training label distribution: tensor([4990, 4991, 4994, 4994, 4991, 4991, 4995, 4993, 4991, 4990])\n",
      "\n",
      "Test labels: tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])\n",
      "Test label distribution: tensor([1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000])\n"
     ]
    }
   ],
   "source": [
    "print('Training labels:', torch.unique(all_train_labels))\n",
    "print('Training label distribution:', torch.bincount(all_train_labels))\n",
    "\n",
    "print('\\nTest labels:', torch.unique(all_test_labels))\n",
    "print('Test label distribution:', torch.bincount(all_test_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cc21f44",
   "metadata": {},
   "source": [
    "### Performance baseline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf7a9b53",
   "metadata": {},
   "source": [
    "- Especially for imbalanced datasets, it's quite useful to compute a performance baseline.\n",
    "- In classification contexts, a useful baseline is to compute the accuracy for a scenario where the model always predicts the majority class -- you want your model to be better than that!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0c1fed8d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline ACC: 10.00%\n"
     ]
    }
   ],
   "source": [
    "majority_prediction = torch.argmax(torch.bincount(all_test_labels))\n",
    "baseline_acc = torch.mean((all_test_labels == majority_prediction).float())\n",
    "print(f'Baseline ACC: {baseline_acc*100:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0437c28f",
   "metadata": {},
   "source": [
    "### Setting up a `DataModule`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67b34a24",
   "metadata": {},
   "source": [
    "- There are three main ways we can prepare the dataset for Lightning. We can\n",
    "  1. make the dataset part of the model;\n",
    "  2. set up the data loaders as usual and feed them to the fit method of a Lightning Trainer -- the Trainer is introduced in the next subsection;\n",
    "  3. create a LightningDataModule.\n",
    "- Here, we are going to use approach 3, which is the most organized approach. The `LightningDataModule` consists of several self-explanatory methods as we can see below:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e953aac8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from torch.utils.data.dataset import random_split\n",
    "from torch.utils.data import DataLoader\n",
    "from torchvision import transforms\n",
    "\n",
    "\n",
    "class DataModule(pl.LightningDataModule):\n",
    "    def __init__(self, data_path='./'):\n",
    "        super().__init__()\n",
    "        self.data_path = data_path\n",
    "        \n",
    "    def prepare_data(self):\n",
    "        datasets.CIFAR10(root=self.data_path,\n",
    "                         download=True)\n",
    "        \n",
    "        self.train_transform = transforms.Compose(\n",
    "            [transforms.ToTensor()])\n",
    "\n",
    "        self.test_transform = transforms.Compose(\n",
    "            [transforms.ToTensor()])\n",
    "        \n",
    "        return\n",
    "\n",
    "    def setup(self, stage=None):\n",
    "        train = datasets.CIFAR10(root=self.data_path, \n",
    "                                 train=True, \n",
    "                                 transform=self.train_transform,\n",
    "                                 download=False)\n",
    "\n",
    "        self.test = datasets.CIFAR10(root=self.data_path, \n",
    "                                     train=False, \n",
    "                                     transform=self.test_transform,\n",
    "                                     download=False)\n",
    "\n",
    "        self.train, self.valid = random_split(train, lengths=[45000, 5000])\n",
    "\n",
    "    def train_dataloader(self):\n",
    "        train_loader = DataLoader(dataset=self.train, \n",
    "                                  batch_size=BATCH_SIZE, \n",
    "                                  drop_last=True,\n",
    "                                  shuffle=True,\n",
    "                                  num_workers=NUM_WORKERS)\n",
    "        return train_loader\n",
    "\n",
    "    def val_dataloader(self):\n",
    "        valid_loader = DataLoader(dataset=self.valid, \n",
    "                                  batch_size=BATCH_SIZE, \n",
    "                                  drop_last=False,\n",
    "                                  shuffle=False,\n",
    "                                  num_workers=NUM_WORKERS)\n",
    "        return valid_loader\n",
    "\n",
    "    def test_dataloader(self):\n",
    "        test_loader = DataLoader(dataset=self.test, \n",
    "                                 batch_size=BATCH_SIZE, \n",
    "                                 drop_last=False,\n",
    "                                 shuffle=False,\n",
    "                                 num_workers=NUM_WORKERS)\n",
    "        return test_loader"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "337c34dc",
   "metadata": {},
   "source": [
    "- Note that the `prepare_data` method is usually used for steps that only need to be executed once, for example, downloading the dataset; the `setup` method defines the the dataset loading -- if you run your code in a distributed setting, this will be called on each node / GPU. \n",
    "- Next, lets initialize the `DataModule`; we use a random seed for reproducibility (so that the data set is shuffled the same way when we re-execute this code):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1381fbdf",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(1) \n",
    "data_module = DataModule(data_path='./data')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a750225f",
   "metadata": {},
   "source": [
    "## Training the model using the PyTorch Lightning Trainer class"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27a1e845",
   "metadata": {},
   "source": [
    "- Next, we initialize our model.\n",
    "- Also, we define a call back so that we can obtain the model with the best validation set performance after training.\n",
    "- PyTorch Lightning offers [many advanced logging services](https://pytorch-lightning.readthedocs.io/en/latest/extensions/logging.html) like Weights & Biases. Here, we will keep things simple and use the `CSVLogger`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "432879ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pytorch_lightning.callbacks import ModelCheckpoint\n",
    "from pytorch_lightning.loggers import CSVLogger\n",
    "\n",
    "\n",
    "pytorch_model = PyTorchLeNet5(\n",
    "    num_classes=10, grayscale=False)\n",
    "\n",
    "lightning_model = LightningModel(\n",
    "    pytorch_model, learning_rate=LEARNING_RATE)\n",
    "\n",
    "callbacks = [ModelCheckpoint(\n",
    "    save_top_k=1, mode='max', monitor=\"valid_acc\")]  # save top 1 model \n",
    "logger = CSVLogger(save_dir=\"logs/\", name=\"my-model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f7231e9",
   "metadata": {},
   "source": [
    "- Now it's time to train our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "470fdfeb",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "\n",
      "  | Name      | Type          | Params\n",
      "--------------------------------------------\n",
      "0 | model     | PyTorchLeNet5 | 548 K \n",
      "1 | train_acc | Accuracy      | 0     \n",
      "2 | valid_acc | Accuracy      | 0     \n",
      "3 | test_acc  | Accuracy      | 0     \n",
      "--------------------------------------------\n",
      "548 K     Trainable params\n",
      "0         Non-trainable params\n",
      "548 K     Total params\n",
      "2.196     Total estimated model params size (MB)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation sanity check: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6c1f9947c53a47c4b52f17e74322d422",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Training: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training took 3.88 min in total.\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "\n",
    "trainer = pl.Trainer(\n",
    "    max_epochs=NUM_EPOCHS,\n",
    "    callbacks=callbacks,\n",
    "    progress_bar_refresh_rate=50,  # recommended for notebooks\n",
    "    accelerator=\"auto\",  # Uses GPUs or TPUs if available\n",
    "    devices=\"auto\",  # Uses all available GPUs/TPUs if applicable\n",
    "    logger=logger,\n",
    "    log_every_n_steps=100)\n",
    "\n",
    "start_time = time.time()\n",
    "trainer.fit(model=lightning_model, datamodule=data_module)\n",
    "\n",
    "runtime = (time.time() - start_time)/60\n",
    "print(f\"Training took {runtime:.2f} min in total.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5b470c8",
   "metadata": {},
   "source": [
    "## Evaluating the model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36ded203",
   "metadata": {},
   "source": [
    "- After training, let's plot our training ACC and validation ACC using pandas, which, in turn, uses matplotlib for plotting (you may want to consider a [more advanced logger](https://pytorch-lightning.readthedocs.io/en/latest/extensions/logging.html) that does that for you):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "4d43da0f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='Epoch', ylabel='ACC'>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA9YklEQVR4nO3dd3hUZfr/8fed3kMKhFAkofeWUBSFABZAFAsiLFVlEZVdG67uftdd19/u6q67thVBVMSGWJCioqhI6FKC9F4lRCG0kIT0PL8/zoAxTApJJmeS3K/rOtfMnPOczCeHYe6c9jxijEEppZQqzsPuAEoppdyTFgillFJOaYFQSinllBYIpZRSTmmBUEop5ZSX3QGqUmRkpImJianQupmZmQQGBlZtoCqk+SpH81WO5qscd86XlJR00hhT3+lCY0ytmeLi4kxFLVu2rMLrVgfNVzmar3I0X+W4cz5goynhO1UPMSmllHJKC4RSSimntEAopZRyqladpHYmLy+P5ORksrOzS20XGhrKrl27qinV5XOHfH5+fjRp0gRvb29bcyilqketLxDJyckEBwcTExODiJTYLj09neDg4GpMdnnszmeM4dSpUyQnJxMbG2tbDqVU9an1h5iys7OJiIgotTiosokIERERZe6JKaVqj1pfIAAtDlVEt6NSdUutP8SklFIulfID7F5sPb/4R5T86nmzw4dg+XprPhce5Jd1xBO8fMHTx/HoC14+xR6LLi+hXRXTAqGUUhV1aCXMGQF550ttFgtw2IU5AuvDY/ur/MdqgXCxs2fPMmfOHO6///7LWm/IkCHMmTOHevXqXdZ6EyZMYOjQoQwfPvyy1lNKXaaDy2HOnRDWDMZ/BkENrPkXBmG7OBibIXF5Igl9+wGXLgOgMB/yc6Agt9hjDuTnFnt00s6z6vceQAuEy509e5ZXX331kgJRUFCAp6dniestXrzY1dGUUhV1MBHmjISwGEdxKNKVkcivH8E6hORZ2tetL/i4X19NLisQIjILGAqcMMZ0dLL8MWB0kRztgPrGmNMichhIBwqAfGNMfFVk+ttnO9iZcs7psrK+sEvSvlEIf72pQ4nLn3jiCQ4cOEDXrl3x9vYmKCiI6OhoNm/ezM6dO7nllls4evQo2dnZPPjgg0yaNAmAmJgYNm7cSEZGBoMHD6ZXr15s2LCBxo0bs3DhQvz9/cvMtnTpUqZOnUp+fj49evRg+vTp+Pr68sQTT7Bo0SK8vLy4/vrr+c9//sPHH3/M3/72Nzw9PQkNDWXFihWXvS2UqhMOLIMPRkJ4c6s4BEbanchlXLkHMRt4BXjH2UJjzHPAcwAichPwsDHmdJEm/Y0xJ12Yr1o8++yzbN++nc2bN5OYmMiNN97I9u3bL95LMGvWLMLDw8nKyqJHjx7cfvvtRERE/Opn7Nu3jzfeeIPZs2czYsQI5s2bx5gxY0p93+zsbCZMmMDSpUtp3bo148aNY/r06YwbN4758+eze/duRISzZ88C8PTTT7NkyRIaN258cZ5SqpgD38EHoyC8BYxfVKuLA7iwQBhjVohITDmbjwI+cFWWC0r7S7+6bkTr2bPnr240e/nll5k/fz4AR48eZd++fZcUiNjYWDp37gxAXFwchw8fLvN99uzZQ2xsLK1btwZg/PjxTJs2jSlTpuDn58fEiRO58cYbGTp0KAB9+vRhwoQJjBgxgttuu60qflWlapf9S2HubyCiJYxbBIERZa9Tw9l+DkJEAoBBwJQisw3wtYgY4DVjzMxS1p8ETAKIiooiMTHxV8tDQ0NJT08vM0dBQUG52l2ujIwMCgsLSU9P5/z58/j6+l58n5UrV7JkyRK+/vprAgICGDJkCKdPnyY9PR1jDBkZGWRkZODt7X0xX35+PpmZmSVmzcvLIysri4yMjF/9TufPnyc/P5+srCyWLl1KYmIiH3/8MS+99BKff/45zz33HBs2bGDJkiV06dKFVatWXVKowNozKb6NL/yezua7C81XOXU9X9jpTXTa9k/OBzRhS4vHyduw7bLWd/ftVxLbCwRwE7C62OGlPsaYFBFpAHwjIruNMU4PijuKx0yA+Ph4k5CQ8Kvlu3btKteegav2IKKjo8nMzCQ4OJiAgAC8vLwuvk9eXh6RkZFERUWxe/duNmzYQEBAAMHBwYgIQUFBAHh4eODp6UlwcDC+vr7k5eWVmNXb2xt/f3/i4uI4evQox48fp2XLlsybN4+BAwciIhQWFjJ8+HAGDBhAy5YtCQ4O5sCBAwwYMIABAwbw9ddfc/bsWZwNvuTn50e3bt0umZ+YmEjxbe9ONF/l1Ol8+76Flc9CVFuCxi2iT0D4Zf8Id99+JXGHAjGSYoeXjDEpjscTIjIf6AnUyLOmERER9OnTh44dO+Lv709UVNTFZYMGDWLGjBl07tyZNm3a0Lt37yp7Xz8/P9566y3uuOOOiyepJ0+ezOnTpxk2bBjZ2dkYY3jhhRcAeOyxx9i3bx/GGAYOHEiXLl2qLItSNdber+HD0VC/LYxbCBUoDjWZrQVCREKBfsCYIvMCAQ9jTLrj+fXA0zZFrBJz5sxxOt/X15cvv/zS6bIL5xkiIyPZvn37xUNFU6dOLfW9Zs+effH5wIED+eGHH361PDo6mvXr11+y3qefflrqz1Wqztm7BD4cAw3awdgFda44gGsvc/0ASAAiRSQZ+CvgDWCMmeFodivwtTEms8iqUcB8R78/XsAcY8xXrsqplFKX2PMVfDQWGrSHsfPrZHEA117FNKocbWZjXQ5bdN5BQI9vlOGBBx5g9erVv5r34IMPctddd9mUSKlaYs+X8OFYaNjRKg7+YXYnso07nINQFTBt2jS7IyhV++xeDB+Ng4adHMWhnt2JbFUnuvtWSqky7f7CKg7RnbU4OGiBUEqpXZ85ikMXLQ5FaIFQStVt2+fBxxOgUTcY+yn4hdqdyG3oOQilVN1UkA/fPQ2rX4KmvWH0x+AXYncqt6J7EG7mwt3TKSkpJY7pkJCQwMaNG0v8GTExMZw8WeP7OVTKdTJPwnu3WcUh7i6r4z0tDpfQPQg31ahRIz755BO7YyhV+xxLgg/HQWYqDJsG3UrvGbkuq1sF4ssn4GfnnWz5F+SXMaBHCRp2gsHPlrj48ccfp1mzZhcHDHrqqacQEVasWMGZM2fIy8vj73//O8OGDfvVeocPH2bo0KFs376drKwsJkyYwL59+2jXrh1ZWVnljvf8888za9YsACZOnMhDDz1EZmYmI0aMIDk5mYKCAp588knuvPNOp+NEKFWrJL0Ni6dCUEO4Z4l13kGVqG4VCBuMHDmShx566GKB+Oijj/jqq694+OGHCQkJ4eTJk/Tu3Zubb74ZKToCVRHTp08nICCArVu3snXrVrp3716u905KSuKtt95i3bp1GGPo1asX/fr14+DBgzRq1IgvvvgCgLS0NE6fPu10nAilaoX8HFj8GGx6G5r3h+Gz6uzd0ZejbhWIUv7Sz3JRb67dunXjxIkTpKSkkJqaSlhYGNHR0Tz88MOsWLECDw8Pjh07xvHjx2nYsKHTn7FixQomTpwIQOfOnS+ODVGWVatWceuttxIYaA1leNttt7Fy5UoGDRrE1KlTefzxxxk6dCjXXHMN+fn5TseJUKrGS0u27oxO2QTXPAr9/w88Ln/0yLpIT1JXg+HDh/PJJ5/w4YcfMnLkSN5//31SU1NJSkpi8+bNREVFkZ2dXerPKGnvojTm4sDov9a6dWuSkpLo1KkTf/zjH3n66afx8vJi/fr13H777SxYsIBBgwZd9vsp5XYOLofX+sLJfXDn+zDwL1ocLoMWiGowcuRI5s6dyyeffMLw4cNJS0ujQYMGeHt7s2zZMo4cOVLq+n379uWjjz4CYPv27WzdurVc79u3b18WLFjA+fPnyczMZP78+VxzzTWkpKQQEBDAmDFjmDp1Kps2bSIjI4O0tDSGDBnCiy++yObNmyv7aytlH2OsK5TevQUC68OkZdBO94ovV906xGSTDh06kJ6eTuPGjYmOjmb06NHcdNNNxMfH07VrV9q2bVvq+vfddx9jxoyhc+fOdO3alZ49e5brfbt3786ECRMutp84cSLdunVjyZIlPPbYY3h4eODt7c306dNJT093Ok6EUjVOTjosfAB2LoT2t1hXKvkG2Z2qRtICUU22bfvl6qnIyEjWrl3rtF1GRgZg3cuwfft2APz9/Zk9e3a5z5EUHbP6kUce4ZFHHvnV8htuuIEbbrjhkvWcjROhVI2Sutcaw+HUfrj+73DlFKjA4Vll0QKhlKoddn0G8+8DL18YtwBi+9qdqMbTAlGD9erVi5ycnF/Ne/fdd+nUqZNNiZSyQWEBsQffgcR50DgORrwLoY3tTlUr1IkCYYyp0FVA7m7dunXV+n4lXRWllC2Mgf1LYdnfaZbyg9VlxuB/WXsQqkrU+quY/Pz8OHXqlH65VZIxhlOnTuHn52d3FFXXGQMHlsGb18P7t0PmKXa2ewRuelGLQxWr9XsQTZo0ITk5mdTU1FLbZWdnu/WXnzvk8/Pzo0mTJrZmUHXcoZWw7J/w4xoIaQxDX4CuYzixag3t7c5WC7msQIjILGAocMIY09HJ8gRgIXDIMetTY8zTjmWDgJcAT+ANY0zJt0CXwdvbm9jY2DLbJSYm0q2b+/bL4u75lHKpI2th2T/g8EqrH6Uh/4Hu43SPwcVcuQcxG3gFeKeUNiuNMb+6e0VEPIFpwHVAMrBBRBYZY3a6KqhSyk0dXW/tMRxcBoENYNCzEDcBvP3tTlYnuKxAGGNWiEhMBVbtCew3xhwEEJG5wDBAC4RSdcWxJFj2DOz/BgIirHsa4u8BnwC7k9Updp+DuFJEtgApwFRjzA6gMXC0SJtkoJcd4ZRS1eynLVZh2Psl+IfBtU9Bj9/qndA2EVde3ePYg/i8hHMQIUChMSZDRIYALxljWonIHcANxpiJjnZjgZ7GmN+V8B6TgEkAUVFRcXPnzq1Q1oyMjIujubkjzVc5mq9yXJ0vMOMwMYc/oP7J78nzCuRo01s41ngoBV7l22Oo69uvMvr3759kjIl3tsy2PQhjzLkizxeLyKsiEom1x9C0SNMmWHsYJf2cmcBMgPj4eJOQkFChPImJiVR03eqg+SpH81XOZeUzBvLOQ9aZckxn4fxpOLEDfEOg3xN4976P5v71aO6qfDZw93wlsa1AiEhD4LgxxohIT6x7Mk4BZ4FWIhILHANGAr+xK6dSqgzrX4ft8379xV+QW3J7Tx/r8NGFKawZtLsJet2rg/i4GVde5voBkABEikgy8FfAG8AYMwMYDtwnIvlAFjDSWMe78kVkCrAE6zLXWY5zE0opd5M02xrCM6oTRLb69Rd/SZN3gHagV0O48iqmUWUsfwXrMlhnyxYDi12Ry5mtyWfJLdA7rZW6LLsXw+cPQ8vrYNQH4OltdyJVxey+isl2Z8/nMvr1ddT3K6Rrz2waBLvv3dRKuY0f18End0F0VxjxthaHWqrW98VUlnoBPvx7eGeS0wu5ddoadv10ruyVlKrLUvfAnBFWVxejPwafQLsTKRep8wUCYHCnaP7Uy4/8wkKGT1/DtzuP2x1JKfd0LgXevc3q4mLspxAYaXci5UJaIBxiQj1ZNOVqmtcP4rfvbuT1FQe1B1iliso6C+/dDtlp1p5DWIzdiZSLaYEoIirEj4/uvZLBHRvyj8W7eGLeNnLzC+2OpZTtPApyYe5v4OQ+GPkeRHexO5KqBlogivH38eSVUd353YCWfLjxKGPfXMeZzFKu6VaqtissoN2u5+HIarh1BjRPsDuRqiZaIJzw8BAevb4NL9zZhR9+PMstr65m/4kMu2MpVf2MgS8fp/7JtXDDM9BpuN2JVDXSAlGKW7s14YNJvcnMyefWV1ezat9JuyMpVb1W/hc2vM6PTW+FK++3O42qZlogyhDXLIwFD/ShUag/499az7vfH7E7klLV44f34Lv/B53v5GDzcXanUTbQAlEOTcICmHf/VfRrXZ8nF2znqUU7yC/Qk9eqFtu7BBb9HloMgJtfAdGvirpI/9UBdi7CJ+dUqU2CfL14fVw891wdy+w1h7n77Y2cy86rpoBKVaPkjfDReGjYCUa8A14+didSNtECcf40LHyALlv+CpmlFwlPD+HJoe155rZOrNl/kttfXcOPp85XU1ClqsHJffD+HRDc0LrXwTfY7kTKRlogAsJh5Bz8so/De7dZNwGVYVTPK3jnnp6cSM/hlldXs/7Q6WoIqpSLnfvJukvaw9O6Szqogd2JlM20QADEXsOODo/D8e0w507ILXuv4KoWkSx4oA/1/L0Z/cb3fLntp2oIqpSLZKdZew7nT1l7DuGXM1yPqq20QDicjoiH29+Ao+vgw9GQn1PmOrGRgcy/vw8dG4fy8EebtaM/VTPl58Dc0ZC6C+58Fxp1szuRchNaIIrqcCvc/D848B18cjcU5Je5SmiAN6+NjSPEz5t7300i7byeuFY1SOYpqzgcXgnDXoWWA+1OpNyIFojiuo2Bwf+G3Z/DwgegsOzLWRsE+zF9TBw/pWXx4Ic/UFionfypGuBgIky/Cg4th6EvQJc77U6k3IwWCGd63QsDnoStc63hFMvRq2tcszD+clMHEvek8uK3e6shpFIVlJ8L3/wV3rkF/EJg4rcQf7fdqZQbqvMjypXomkchJx1Wvwi+QXDt38ocR3dMryvYcvQsL3+3n05N6nFd+6jqyapUeZ06APPugZQfIG4C3PBPHfBHlUj3IEoiAtc+BT0mwuqXYOV/yrGK8PdbOtKpcSiPfLiZg6nawZ9yE8bA5jkw4xo4fci6Ae6ml7Q4qFK5rECIyCwROSEi20tYPlpEtjqmNSLSpciywyKyTUQ2i8hGV2UskwgMfg66jILv/g7fTy9zFT9vT6aP6Y6Xp3Dvu0lk5pR9olspl8o6a+01LLjPukLpvtXQfpjdqVQN4Mo9iNnAoFKWHwL6GWM6A/8PmFlseX9jTFdjTLyL8pWPh4fVF027m+CrJ2DTu2Wu0iQsgP+N6s6B1Az+8MlWHZlO2efHddZew44FMODPMH4RhDaxO5WqIVxWIIwxK4ASbzE2xqwxxpxxvPwecN9PracX3P4mtLwWPvs9bP+0zFWubhXJHwa15YttP/H6yoPVEFKpIgoLIPFf8NZga0/47iXQ9zHrLmmlyklc+detiMQAnxtjOpbRbirQ1hgz0fH6EHAGMMBrxpjiexdF150ETAKIioqKmzt3boWyZmRkEBQUVGobj4IcOm99ipBze9je8U/WzXWlMMYwbXMOSccLeKyHH+0jKv6fszz57KT5Kqcq8/lmp9Ju1/PUS9vJ8Qb92Nt6MgVeAW6TzxU0X8X1798/qcQjNcYYl01ADLC9jDb9gV1ARJF5jRyPDYAtQN/yvF9cXJypqGXLlpWvYVaaMa/1M+bp+sYcXF5m8/TsPHPtfxNNt6e/Nslnzrs+n000X+VUWb7t8415pqkx/2hszOa5VfMzTR3afi7izvmAjaaE71Rbr2ISkc7AG8AwY8zFrlSNMSmOxxPAfKCnPQmd8AuBMZ9afdXMGQlHN5TaPMjXixlj48jLL+S+95LIziuopqCqTsnNhIVT4OPxENESJq/QG99UpdlWIETkCuBTYKwxZm+R+YEiEnzhOXA94PRKKNsEhMO4BVZvl+/fDj9vK7V5i/pB/HdEF7Ymp/Hkgu160lpVrWNJ8FpfawS4ax61zjdoZ3uqCrjyMtcPgLVAGxFJFpF7RGSyiEx2NPkLEAG8Wuxy1ihglYhsAdYDXxhjvnJVzgoLbmhdEeITZN2Renxnqc2v79CQ3w1oycdJycxZ/2P1ZFS1W9oxmD8ZXh9o9UA8/jMY+Bfw9LY7maolXHYntTFmVBnLJwITncw/CHS5dA03VO8KGLfIulJkZj+4Zipc/RB4+Tpt/tC1rdmanMZTi3bQLjqE7leEVW9eVTvkpFs3b655BUwB9Pm9tefgF2p3MlXL6J3UlRXZ0rrxqN3NkPhP65rzI2udNvX0EF4a2ZXoUH/uey+J1PSyuxRX6qLCAkiaDS93hxXPQdshMGUjXPe0FgflElogqkJQAxj+Joz+BPKy4K1B8NmD1h2sxdQL8GHGmDjSsvJ4YM4m8grK7i1WKfZ/CzOutj5X4bEwcSkMnwVhzexOpmoxLRBVqdV18MD3cOUU2PQOTOsJO+Zf0hts+0YhPHtbZ9YfOs0zi3fbFFbVCMd3WMOAvnc75J2HO962TkI3sbeDAVU3aIGoaj6BcMM/4LfLrBPZH0+AD0bC2aO/anZLt8bc1SeGWasPsXDzMXuyKveVfhwW/c7aazi20ep19YH10OGWMnsVVqqqaIFwlUZdYeJ31n/sQytgWi9Y+6p1HNnhT0Pa0TM2nMfnbWVnig5XqrCuRlr+b3i5G2z+AHpNht9vhisfKPHiB6VcRQuEK3l6Wf+x7/8eml0FS/4IbwyEn7YA4O3pwbTfdKeevw93zV5P8pnzNgdWtjGFVnfc/4uDZf+AlgPggXUw6BnrvhulbKAFojqENYPRH1snFdOSYWZ/+PpJyM2kfrAvs+/uQVZuAeNmred0Zq7daVV1Mgb2LyUu6VGrO+7ghnDXl3DnexDRwu50qo7TAlFdRKDj7dZx5G6jYc3L8Gpv2P8tbRuG8Mb4Hhw7k8VdszfoGBJ1QX4ubJlrnWN47za889Lhtjesq5OaXWV3OqUALRDVLyAcbv4fTPgCPH2tq1PmTaRnRA6v/KY725LPct/7evlrrZV1Fla9CC91hvn3Wuekhk1jXa/p0PkOa/wRpdyEfhrtEnM1TF4F/R63BnN5sSPX7fo/XutvWLE3lcc+3kJhofbZVGucOQJf/RFe6ADf/hUiW8PoeXD/Wug2BuOh3WMo9+OyrjZUOXj7Qf8/WUOarp8Jm97lutyP+b5+R/65tR/PBHjwp5s6I3pZY811bBOs+R/sXPjLYcYrp0B0Z7uTKVUmLRDuIDzWulql/59g8xyi1r3Gyz7TOJ70PhtOj6Ln8EftTqguR2Eh7Fti9ZV0ZBX4hlhXs/WaDKGN7U6nVLlpgXAnvsHQ616kx28p3PcNpxc+R8/D0yl4/k3aNLga2oRBdM3ox7BOysuGrXOtwnBqH4Q2te6D6TbWGkdEqRpGC4Q78vDAo80NtHjkOh5/Yx4dj33IncdXWn3+X3EV9LoX2g617rNQ9kv/GZLetg4Tnj9pFfHb34T2w7TrbVWj6TeMG/Px8uDJu27lN69H80LKCBb2OUzTfe9ao4aFNIGeE6H7eL2RqrqlHYMja6zDR0fWwEnHeFetboCrfmddgKDnjVQtoAXCzQX5evHWhB4MeX4pN27owif3TqJ12mpYNwO+fQoS/wWdR1hT43jrxLeqOsbA2SNwePUvReHMYWuZbwhc0Ru6joY2g6F+G1ujKlXVtEDUABFBvkyN9+O5HwoZ91YS8+4fQOPxN1o9fa57DbZ+CJvetu6raNIDYvpAsz7Wc58Au+PXLMbAqQNWIbhQFM4lW8v8w6xDfD0nWdu3YSfw8LQ3r1IupAWihqgf4MHbd8cz4rW1jH1zHZ9MvorwqA5w88tw/f+zvsgOr7KmFc+B+Rd4eEPjOKtgxFwNTXtZvc2qXxgDp/bDwUQ44igIGcetZYH1rULQ7EFrG9ZvpzeyqTpFC0QN0i46hDfH92DMm+u4a/YG5kzsRaCvlzWaWJvB1gSQnQY/rvvlr+BVL8LK/4KHFzTqZn3pXSgYdfHqmoJ8OPo97PnSmk4fsOYHN4LYvo6i0AciW+m5BFWnuaxAiMgsYChwwhjT0clyAV4ChgDngQnGmE2OZYMcyzyBN4wxz7oqZ03TMzacV0Z1Y/J7Sdz3/ibeHB+Pt2exv2r9QqH19dYE1hjGR9c5DpmshrXTYPWLIB4Q3dX667hBe2tkvKCGVodx/uG16q9lz/zz1uBNe76EvUsg+6y1hxXbF3rfBy0HQlisFgSlinDlHsRs4BXgnRKWDwZaOaZewHSgl4h4AtOA64BkYIOILDLG7HRh1hrl+g4N+eetnXji02089vEWnh/RFQ+PUr7YfIOh5bXWBJCbCUfXW8Xi8GrrPEZBsV5kPbwgKKpI0YiyHoMaWAXkwrzABuDl47pftjLO/gh7voI9i+lzaCWYfOs8QpvB0HoQtBhQN/eglConlxUIY8wKEYkppckw4B1jjAG+F5F6IhINxAD7jTEHAURkrqOtFogiRva8glOZuTy3ZA8RQb78+cZ25e+SwycQWvS3JrBu8EpPsUYxy/jZ8eiY0n+GtKPWqGaZJwEn/UP5h9MTX9gdCd6B1olx7wDwCSryPNCaLjy/ZHkQ+AY5lgVWbO+lsBB++uGXQ0fHt1vzI1qR3GQoV1x7LzTpqfePKFVOYkzZHcKJSCCQZYwpFJHWQFvgS2NMXhnrxQCfl3CI6XPgWWPMKsfrpcDjWAVikDFmomP+WKCXMWZKCe8xCZgEEBUVFTd37twyfx9nMjIyCAoKqtC61cFZPmMM7+/K5dsf82kR6kGEvxDqK4T6OB59hRCfXx49S9vLKAcpzMc7Lw3fnNP45J7FJ/cMPrmn8ck9A9lp+Eo+ngU5eBZk41FoPV6cCnMu670KPPzI9/KnwNOPAs9LH61lFyZfgjIOE3FqA765ZzB4kBballMRPTkZ2YOsgCY18t/XnWi+ynHnfP37908yxjgd5Ly8f0qtAK4RkTBgKbARuBMYXYlczr6tTCnznTLGzARmAsTHx5uEhIQKhUlMTKSi61aHkvL162d4/pu9rD90mtSMHHadySE959K6LQLhAT7UD/alfrAvkUHWY33HY5+WkdQPrviQlmVuv8JCyM+yDm/lZkLeeWt4zbzMX+blZkBOBuRm4JmbiWdO+iXzyT0O2Zm/vL7w0fAJss4jtBmCtLyOeoER1AMuDLlTU/993YXmqxx3z1eS8hYIMcacF5F7gP8ZY/4tIj9U8r2TgaZFXjcBUgCfEuYrJzw8hKk3/PoGrazcAk5m5HAiPYfU9BxOZliPqRce03M4dDKTE+k55OZb405EBPowbXR3ejePcFXQXw4zVZULRScnA/zr6ZjNSlWxchcIEbkSa4/hnstctySLgCmOcwy9gDRjzE8ikgq0EpFY4BgwEvhNJd+rTvH38aRpeABNw0u/Sc4YQ3pOPvtPZDD14y2MfmMdf76xHROuiqkZXYy7ougopS4q75nAh4A/AvONMTtEpDmwrLQVROQDYC3QRkSSReQeEZksIpMdTRYDB4H9wOvA/QDGmHxgCrAE2AV8ZIzZcXm/lioPESHEz5vuV4Sx8IE+9G/TgL99tpNHP9pCdl6B3fGUUjYr116AMWY5sBxARDyAk8aY35exzqgylhvggRKWLcYqIKqaBPt5M3NsHK8s288L3+5l74l0ZoyJo0mYdtWhVF1Vrj0IEZkjIiGOq5l2AntE5DHXRlPVzcND+P3AVrwxLp4jJ89z8yurWbP/pN2xlFI2Ke8hpvbGmHPALVh/2V8BjHVVKGWvge2iWDilDxGBPox5cx1vrDxIeS6HVkrVLuUtEN4i4o1VIBY67n/Qb4xarHn9IOY/0Ifr2zfk71/s4sG5m8nK1fMSStUl5S0QrwGHgUBghYg0A865KpRyD0G+Xkwf053HbmjDZ1tTuG36Go6ePm93LKVUNSlXgTDGvGyMaWyMGWIsR4D+Ls6m3ICI8ED/lsya0INjZ85z0yurWLkv1e5YSqlqUN6T1KEi8ryIbHRM/8Xam1B1RP82DVg05Wqigv0YP2s9M5Yf0PMSStVy5T3ENAtIB0Y4pnPAW64KpdxTTGQgn95/FYM7RfPsl7uZ8sEPnM/NtzuWUspFyns3dAtjzO1FXv9NRDa7II9yc4G+XrwyqhudGofy7692c+BEBq+NjbM7llLKBcq7B5ElIldfeCEifYAs10RS7k5EmNyvBbPv6slPadnc9L9VLDmcx46UNAoK9bCTUrVFefcgJgPviEio4/UZYLxrIqmaom/r+nw25Wp+98EmPtidxge7VxHs50V8szB6xIbTMyacTk1C8fXytDuqUqoCytvVxhagi4iEOF6fE5GHgK0uzKZqgCsiAlg45Wrmffkdng3bsO7QaTYcPs2yPXsA8PXyoGvTevSMDadHTDjdm4UR5KsD9ihVE1zW/1TH3dQXPAK8WKVpVI0V4e9BQrfG3NKtMQCnMnLYcPgMGw5bBePVxAMUFO7H00Po0CiEHjHhjimMiCDtplspd1SZP+VqQH/Qyi4RQb4M6tiQQR0bApCRk8+mI1bBWH/oNO99f4Q3Vx0CoEX9QH57TXNG9rzCzshKqWIqUyD0bKQqtyBfL/q2rk/f1vUByMkvYFtyGusPn+abncd54tNtZOcVMKFPrM1JlVIXlFogRCQd54VAAH+XJFJ1gq+XJ/Ex4cTHhPPba5ozZc4mnvpsJx4ewrgrY+yOp5SijMtcjTHBxpgQJ1OwMUbPNKoq4e3pwf9Gdee69lH8ZeEO3vv+iN2RlFKU/z4IpVzKx8uDab/pzrXtGvDnBduZs+5HuyMpVedpgVBuw8fLg2mjuzOgbQP+NH8bc9drkVDKTloglFvx9fLk1dHdSWhTnz/O38ZHG47aHUmpOsulBUJEBonIHhHZLyJPOFn+mIhsdkzbRaRARMIdyw6LyDbHso2uzKnci5+3JzPGxHFNq/o8/ulWPklKtjuSUnWSywqEiHgC04DBQHtglIi0L9rGGPOcMaarMaYr8EdguTHmdJEm/R3L412VU7knP29PZo6N4+qWkTz2yRY+3aRFQqnq5so9iJ7AfmPMQWNMLjAXGFZK+1HABy7Mo2oYq0jEc1WLCKZ+vIUFPxyzO5JSdYq4atAXERkODDLGTHS8Hgv0MsZMcdI2AEgGWl7YgxCRQ1idAhrgNWPMzBLeZxIwCSAqKipu7ty5FcqbkZFBUFBQhdatDnU5X06B4cWkbHafLuTezr70bnT5V1jX5e1XFTRf5bhzvv79+yeVeJTGGOOSCbgDeKPI67HA/0poeyfwWbF5jRyPDYAtQN+y3jMuLs5U1LJlyyq8bnWo6/kyc/LMiBlrTOwTn5tFm49d9vp1fftVluarHHfOB2w0JXynuvIQUzLQtMjrJkBKCW1HUuzwkjEmxfF4ApiPdchK1VEBPl7MmtCD+GbhPPThZr7Y+pPdkZSq9VxZIDYArUQkVkR8sIrAouKNHGNM9AMWFpkXKCLBF54D1wPbXZhV1QCBvl68dVcPujWtx+/n/sBX27VIKOVKLisQxph8YAqwBNgFfGSM2SEik0VkcpGmtwJfG2Myi8yLAlaJyBZgPfCFMeYrV2VVNUegrxez7+5J16b1mDLnB5bs+NnuSErVWi7tT8kYsxhYXGzejGKvZwOzi807CHRxZTZVcwX5ejH7rh6Mm7WeB97fxPQxcVzXPsruWErVOnontaqRgv28efvunnRoHMr97yexbM8JuyMpVetogVA1VoifN+/c3ZNWDYJ55MPNnMrIsTuSUrWKFghVo4X6e/PSyK5k5hTw1Gc77Y6jVK2iBULVeK2igpkyoCWfbUnhm53H7Y6jVK2hBULVCvcltKBtw2D+vGAbaVl5dsdRqlbQAqFqBW9PD54b3oXU9ByeWbzL7jhK1QpaIFSt0alJKL/t25y5G46yev9Ju+MoVeNpgVC1ysPXtiY2MpAnPt3K+dx8u+MoVaNpgVC1ip+3J8/e1omjp7P479d77Y6jVI2mBULVOr2aRzC2dzNmrT7Eph/P2B1HqRpLC4Sqlf4wqA3RIX48/slWcvIL7I6jVI2kBULVSsF+3vzjtk7sO5HBtO/22x1HqRpJC4Sqtfq3acBt3RrzauIBfjynexFKXS4tEKpWe3Joe+oFeDNrey75BYV2x1GqRtECoWq1sEAf/nZzRw6fK+SNVYfsjqNUjaIFQtV6Qzo1JC7Kkxe+2cvB1Ay74yhVY2iBULWeiDC2nQ++Xh48MW8bhYXG7khK1QhaIFSdUM/Pgz8Pbc/6w6d5f/2PdsdRqkbQAqHqjDvimnBNq0ieXbyLY2ez7I6jlNtzaYEQkUEiskdE9ovIE06WJ4hImohsdkx/Ke+6Sl0uEeGft3bCAH/6dBvG6KEmpUrjsgIhIp7ANGAw0B4YJSLtnTRdaYzp6pievsx1lbosTcMD+MMNbVi+N5X5PxyzO45Sbs2VexA9gf3GmIPGmFxgLjCsGtZVqlTjrowhrlkYT3++k9R0HcdaqZKIq3azRWQ4MMgYM9HxeizQyxgzpUibBGAekAykAFONMTvKs26RnzEJmAQQFRUVN3fu3ArlzcjIICgoqELrVgfNVznF86VkFPKX1Vl0i/Lkga5+Niaz1LTt5240X8X1798/yRgT72yZlwvfV5zMK16NNgHNjDEZIjIEWAC0Kue61kxjZgIzAeLj401CQkKFwiYmJlLRdauD5qscZ/nOBO7nuSV7yI5sy6CODe0J5lATt5870Xyu4cpDTMlA0yKvm2DtJVxkjDlnjMlwPF8MeItIZHnWVaqyJvVtTvvoEJ5cuJ208zqOtVLFuXIPYgPQSkRigWPASOA3RRuISEPguDHGiEhPrIJ1Cjhb1rpKVZa3pwf/Ht6ZYdNWc/2Ly2kWEUhUiB8NQ3yJCvGjQYgfDUP8iHK89vP2tDuyUtXKZQXCGJMvIlOAJYAnMMtxfmGyY/kMYDhwn4jkA1nASGOdFHG6rquyqrqrY+NQXh7Zja92/MzxtGy2Jp/l67RscvIv7dgv1N/7YrGIchSOho5CEtcsjMggXxt+A6Vcx5V7EBcOGy0uNm9GkeevAK+Ud12lXOHGztHc2Dn64mtjDOey8jmens3xc9n8nJbNifQcfk6zXh9Pz2Hf8ZOkZuRQ4Oi2IyLQh5nj4olrFmbXr6FUlXNpgVCqJhIRQgO8CQ3wpnVUcIntCgoNpzJzOJSayR/mbWXU69/z3zu6cFOXRtWYVinX0a42lKogTw+hQbAfvZpHMP/+PnRuHMrvPviBV77bp3dpq1pBC4RSVSA80If3f9uLW7o24j9f72XqxzoWtqr59BCTUlXE18uTF+7sSkxkIC9+u4+jZ87z2pg4wgJ97I6mVIXoHoRSVUhEeOja1rx4Z1c2/3iW26av4dDJTLtjKVUhWiCUcoFbujVmzm97kZaVx62vrmbdwVN2R1LqsmmBUMpF4mPCmX//VYQH+jDmzXXMS0q2O5JSl0ULhFIu1CwikPn39SG+WTiPfryF57/eo1c4qRpDC4RSLhYa4M3bd/dkRHwTXv5uP7+fu5nsPL3CSbk/vYpJqWrg4+XBv27vTGxkEP/6ajfHzpxn5rh47Z5DuTXdg1CqmogI9yW04NXR3dmRco5bX13NvuPpdsdSqkRaIJSqZkM6RfPhvVeSlVvIbdPXsGrfSbsjKeWUFgilbNC1aT0WPHAV0aF+jH9rPZ8dyCUtS8ekUO5FC4RSNmkSFsAn911F/zYNmLcvj97/XMr/zd+mh52U29CT1ErZKMTPmzfGx/P2oqVsz4nk46Rk3l/3I31aRjD+yhgGtovC08PZCLxKuZ4WCKXcQLMQT8YndOGPQ9oxd8OPvLv2CJPeTaJJmD/jrmzGnfFXEBrgbXdMVcfoISal3Eh4oA/3J7Rk5R/6M310dxrX8+efi3fT65lv+eOnW9n98zm7I6o6RPcglHJDXp4eDO4UzeBO0ez66RxvrznMp5uO8cH6o/RuHs6Eq2K4tl0UXp76N55yHf10KeXm2kWH8Oztnfn+jwN5YnBbjp7OYvJ7m+j3XCLTEw9wJjPX7oiqlnJpgRCRQSKyR0T2i8gTTpaPFpGtjmmNiHQpsuywiGwTkc0istGVOZWqCcICfZjcrwUr/tCf18bG0SwigH99tZvezyzlwbk/sGhLil4qq6qUyw4xiYgnMA24DkgGNojIImPMziLNDgH9jDFnRGQwMBPoVWR5f2OM3kWkVBGeHsINHRpyQ4eG7Pk5nXfWHubL7T+zcHMKXh5Cj5hwBrZrwMB2UcRGBtodV9VgrjwH0RPYb4w5CCAic4FhwMUCYYxZU6T990ATF+ZRqtZp0zCYf9zaiaeHdWTz0TMs3XWCpbtO8PcvdvH3L3bRPDKQge0aMKBtFPExYXjrOQt1GcRVXQ+LyHBgkDFmouP1WKCXMWZKCe2nAm2LtD8EnAEM8JoxZmYJ600CJgFERUXFzZ07t0J5MzIyCAoKqtC61UHzVU5dy5d6vpAtqQVsTi1g96kC8g0EeEGnSE+6NvCiU6QnQT7lv7+irm2/qubO+fr3759kjIl3tsyVexDOPn1Oq5GI9AfuAa4uMruPMSZFRBoA34jIbmPMikt+oFU4ZgLEx8ebhISECoVNTEykoutWB81XOXUx3x2Ox4ycfFbtO8nSXcdZtucE637OwdNDiGsWxsC21qGoFvUDESm5YNTF7VeV3D1fSVxZIJKBpkVeNwFSijcSkc7AG8BgY8zFcRmNMSmOxxMiMh/rkNUlBUIpVbogXy8GdWzIoI4NKSw0bEk+y3e7T/DtrhM88+VunvlyNzERAVzbLorr2kcR1yxML59VgGsLxAaglYjEAseAkcBvijYQkSuAT4Gxxpi9ReYHAh7GmHTH8+uBp12YVak6wcND6HZFGN2uCOPR69uQcjaLpbtPsHTXcd5Ze4Q3Vh2iXoA3A9o04Nr2UfRtXZ8gX71dqq5y2b+8MSZfRKYASwBPYJYxZoeITHYsnwH8BYgAXnXs3uY7joVFAfMd87yAOcaYr1yVVam6qlE9f8b2bsbY3s3IyMln5d5Uvtl1nO92n+DTH47h4+nBlS0iaOaVR5u0LKJD/e2OrKqRS/80MMYsBhYXmzejyPOJwEQn6x0EuhSfr5RynSBfr4t3b+cXFJJ05Azf7jrONzuPs/xULu/s/I5OjUO5tl0U17ZvQPvokFLPW6iaT/cdlVKX8PL0oFfzCHo1j+BPQ9rxwRfLSAtqxre7jvPi0r288O1eGtfz59p21qGoXrER+HjpeYvaRguEUqpUIkKjIA9+k9CC+xJakJqew7LdJ/hm13E+3HiUt9ceoXE9f/56U3uu79DQ7riqCmmBUEpdlvrBvozo0ZQRPZqSnVfA8r2pPP/1Xia9m8S17Rrw15s60DQ8wO6YqgroPqFSqsL8vD25oUNDPv/91fxpSFvWHDjFdS8s59XE/eTmF9odT1WSFgilVKV5e3owqW8Lvn2kHwmtG/Dvr/Yw5OWVrD1wquyVldvSAqGUqjKN6vkzY2wcb03oQU5+AaNe/56HP9xManqO3dFUBWiBUEpVuf5tG/D1Q/343YCWfL41hQH/TeTdtYcpKHRN32/KNbRAKKVcwt/Hk0evb8NXD/WlU+NQnly4g9teXc225DS7o6ly0gKhlHKpFvWDeH9iL14a2ZWUtGxunraKvyzcroMb1QBaIJRSLiciDOvamKWP9mP8lTG89/0RBv53OQt+OIarhhxQlaf3QSilqk2InzdP3dyB27s34c8LtvHQh5v5cMNRHr6uNW2iggkN8LY7YpUzxpCTbzh6+jxnzudyKjOXM5m5nHZMZ87n4uvlSZ+WkfRuHk6wn/tsAy0QSqlq16lJKJ/e34cP1v/Iv7/azYjX1gIQHuhDbGTgxal5ZCCx9QOJiQjEz9vT5tSXyi8oZPPRs+z6Of2SL/3TRV7n5BfCt8suWd/LQwgL9CEjO5/Zaw7j5SF0u6Ie17SqzzWtIuncpB6eHvb1d6UFQillC08PYUzvZtzYKZqkI2c4dDKTgyczOZiawYq9qXySlPyr9o3r+f+qeMTWtwpI43rV28Ps8XPZLN+byvI9qazcl8q57PyLy4J9vQgP8iEswIeGIX60iw4hPNCHsz8fJb5TO8IDfQgL9CHC8Rji54WIkJNfwKYjZ1m5L5VV+0/ywrd7ef6bvYT4edGnZeTFglHdd6hrgVBK2Sos0Idr20ddMj8jJ5/DJzM5VGQ6eDKTBZuPkV7kS9nbU2jgDz1+/oEOjULp0CiEDo1Cq+xwVZ6jZ9vEPaks35vKrp/OAdAg2JcbOjQkoU0DujerR0Sgb4kdFiYmHiehR1OnywB8vTy5skUEV7aI4A/A6cxcVu8/ycp9qazcd5Ivt/8MQGxkIFe3jOSaVpFc2SLC5YejtEAopdxSkK8XHRuH0rFx6K/mG2M4nZlbZI8jk7U7D7P24CkWbP5l0MrG9fwvFov2jULo0CiE6FC/cnVRfuxsFsv3pLJ87wlW7z9FRk4+Xh5CfEwYjw9qS0Kb+rRtGOyy7s7DA324qUsjburSCGMMB1IzLxaLT5KSeff7I3h6CN2vqMfVLetzTetIujaph0cVH47SAqGUqlFEhIggXyKCfImPCQcg0f9nEhISOJmRw86Uc+xIOceOlDR2ppzjm13HuXChVFiA98W9jPaO4hEbGUh+YSEbDp1h+d4TJO5JZd+JDAAahfpxU5dGJLSpz1XV8Bd7Sb9vywZBtGwQxF19Yi8ejlq13yoYLy7dy9trD7Px/66t8vfWAqGUqjUig3zp27o+fVvXvzgvMyefXT9ZRWNnyjl2/JTGW6sPk1tgdSbo7zj5nZVXgI+nBz1jw7mzR1P6ta5PywZBbjcoUtHDUY/dgGNvKqPK9x5AC4RSqpYL9PUiPib84t4GQG5+IftPZLAjJY0dKdY5hQvH9QN8atbXYnigD+GB4WU3rICatSWUUqoK+Hh50N5xmOkOu8O4MZfeSS0ig0Rkj4jsF5EnnCwXEXnZsXyriHQv77pKKaVcy2UFQkQ8gWnAYKA9MEpE2hdrNhho5ZgmAdMvY12llFIu5Mo9iJ7AfmPMQWNMLjAXGFaszTDgHWP5HqgnItHlXFcppZQLufIcRGPgaJHXyUCvcrRpXM51ARCRSVh7H0RFRZGYmFihsBkZGRVetzpovsrRfJWj+SrH3fOVxJUFwtk1V8W7bSypTXnWtWYaMxOYCRAfH28SEhIuI+IvEhMTqei61UHzVY7mqxzNVznunq8kriwQyUDRe8ubACnlbONTjnWVUkq5kCvPQWwAWolIrIj4ACOBRcXaLALGOa5m6g2kGWN+Kue6SimlXMhlexDGmHwRmQIsATyBWcaYHSIy2bF8BrAYGALsB84Dd5W2rquyKqWUupTUptGcRCQVOFLB1SOBk1UYp6ppvsrRfJWj+SrHnfM1M8bUd7agVhWIyhCRjcaYeLtzlETzVY7mqxzNVznunq8kOia1Ukopp7RAKKWUckoLxC9m2h2gDJqvcjRf5Wi+ynH3fE7pOQillFJO6R6EUkopp7RAKKWUcqpOFYjKjE9RTfmaisgyEdklIjtE5EEnbRJEJE1ENjumv1RzxsMiss3x3hudLLdtG4pImyLbZbOInBORh4q1qdbtJyKzROSEiGwvMi9cRL4RkX2Ox7AS1nX5mCgl5HtORHY7/v3mi0i9EtYt9bPgwnxPicixIv+GQ0pY167t92GRbIdFZHMJ67p8+1WaMaZOTFh3ZB8AmmP19bQFaF+szRDgS6zOAnsD66o5YzTQ3fE8GNjrJGMC8LmN2/EwEFnKclu3YbF/75+xbgKybfsBfYHuwPYi8/4NPOF4/gTwrxLyl/p5dWG+6wEvx/N/OctXns+CC/M9BUwtx7+/Lduv2PL/An+xa/tVdqpLexCVGZ+iWhhjfjLGbHI8Twd2YXV9XpPYug2LGAgcMMZU9M76KmGMWQGcLjZ7GPC24/nbwC1OVq2WMVGc5TPGfG2MyXe8/B6rs0xblLD9ysO27XeBiAgwAvigqt+3utSlAlHS2BOX26ZaiEgM0A1Y52TxlSKyRUS+FJEO1ZsMA3wtIklijcVRnLtsw5GU/B/Tzu0HEGWsTilxPDZw0sZdtuPdWHuEzpT1WXClKY5DYLNKOETnDtvvGuC4MWZfCcvt3H7lUpcKRGXGp6hWIhIEzAMeMsacK7Z4E9Zhky7A/4AF1RyvjzGmO9ZwsA+ISN9iy23fhmL1AHwz8LGTxXZvv/Jyh+34f0A+8H4JTcr6LLjKdKAF0BX4CeswTnG2bz9gFKXvPdi1/cqtLhWIyoxPUW1ExBurOLxvjPm0+HJjzDljTIbj+WLAW0QiqyufMSbF8XgCmI+1K1+U7dsQ6z/cJmPM8eIL7N5+DscvHHZzPJ5w0sbW7Sgi44GhwGjjOGBeXDk+Cy5hjDlujCkwxhQCr5fwvnZvPy/gNuDDktrYtf0uR10qEJUZn6JaOI5ZvgnsMsY8X0Kbho52iEhPrH/DU9WUL1BEgi88xzqZub1YM1u3oUOJf7nZuf2KWASMdzwfDyx00sa2MVFEZBDwOHCzMeZ8CW3K81lwVb6i57RuLeF97R5T5lpgtzEm2dlCO7ffZbH7LHl1TlhX2OzFurrh/xzzJgOTHc8FmOZYvg2Ir+Z8V2PtBm8FNjumIcUyTgF2YF2V8T1wVTXma+543y2ODO64DQOwvvBDi8yzbfthFaqfgDysv2rvASKApcA+x2O4o20jYHFpn9dqyrcf6/j9hc/gjOL5SvosVFO+dx2fra1YX/rR7rT9HPNnX/jMFWlb7duvspN2taGUUsqpunSISSml1GXQAqGUUsopLRBKKaWc0gKhlFLKKS0QSimlnNICodRlEJEC+XWPsVXWS6iIxBTtFVQpu3nZHUCpGibLGNPV7hBKVQfdg1CqCjj69v+XiKx3TC0d85uJyFJHx3JLReQKx/wox1gLWxzTVY4f5Skir4s1HsjXIuJv2y+l6jwtEEpdHv9ih5juLLLsnDGmJ/AK8KJj3itY3Z93xur07mXH/JeB5cbqNLA71t20AK2AacaYDsBZ4HaX/jZKlULvpFbqMohIhjEmyMn8w8AAY8xBR4eLPxtjIkTkJFZXEHmO+T8ZYyJFJBVoYozJKfIzYoBvjDGtHK8fB7yNMX+vhl9NqUvoHoRSVceU8LykNs7kFHlegJ4nVDbSAqFU1bmzyONax/M1WD2JAowGVjmeLwXuAxARTxEJqa6QSpWX/nWi1OXxLzYI/VfGmAuXuvqKyDqsP7xGOeb9HpglIo8BqcBdjvkPAjNF5B6sPYX7sHoFVcpt6DkIpaqA4xxEvDHmpN1ZlKoqeohJKaWUU7oHoZRSyindg1BKKeWUFgillFJOaYFQSinllBYIpZRSTmmBUEop5dT/BxHYv0DyNziDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA4fUlEQVR4nO3deXgUVfbw8e/NvhJIAiEkEPadsIRNQQEZFVQUARUXBh2FAcVtfjrqjK+z6YyjjuMO4zbKGlRAUFEYFEQRMIQlhLCFQEgIJIGQjezd9/2jGowxCUnoSne6z+d5+klX1a2q09WdOlX3Vt1SWmuEEEK4Lw9HByCEEMKxJBEIIYSbk0QghBBuThKBEEK4OUkEQgjh5rwcHUBjhYeH686dOzdp3nPnzhEYGGjfgOzAWeMC541N4mociatxXDGuxMTE01rrtrVO1Fq3qFdcXJxuqo0bNzZ5XjM5a1xaO29sElfjSFyN44pxATt0HftV06qGlFLvK6VylFLJdUxXSqnXlFKpSqkkpdQQs2IRQghRNzPbCD4AJtQzfSLQw/aaDcw3MRYhhBB1MC0RaK03A3n1FLkJWGg7a9kGtFZKRZoVjxBCiNopbWIXE0qpzsDnWuv+tUz7HHhea/29bfhr4Amt9Y5ays7GOGsgIiIiLj4+vuZ0AgMD8fT0rDcerTVKqSZ+GvM4Q1wWi4Vz585R8/dQXFxMUFCQg6Kqm8TVOBJX47hiXOPGjUvUWg+tdWJdjQf2eAGdgeQ6pn0BjK42/DUQd7Fl1tZYnJaWpnNzc7XVaq23saSwsLD+1hQHcXRcVqtV5+bm6rS0tF9Mc8VGMzNJXI0jcTVOi2ssboBMoGO14WggqykLKisrIywszOFH1S2VUoqwsDDKysocHYoQwgEcmQjWAL+2XT00EijQWp9s6sIkCVwa2X5CuC/TbihTSi0DxgLhSqlM4E+AN4DWegGwFrgOSAVKgHvMikUI4b6qLFZyiso5WVDGqYIyThaU4umhiItpQ9/IVnh5SgcLpiUCrfXtF5mugQfMWr8QwvVVWqxkF57fwRs7+Z92+MbfnKIyrHVcExPo48mQmDYM6xzKsM6hDOrYGn+f+i86cZTi8irKLeZc3NPiuphwRvn5+SxdupT777+/UfNdd911LF269KJXOwkhoLTCws7jZ9medobtR/NIO32O08Xl1LzwMcDHk8gQPyJD/LmiRziRIX60D/E3xrX2I7KVP6WVFhKO5ZFwLI8fj+bx7w2H0Bq8PRX9o0Jo71lBVbtshnZuQ+sAH8d8YJsqi5XlOzL49/8OMyrCyrXj7b8OSQR2kJ+fz1tvvfWLRGCxWOrdya9duxaAoqIiU+MToiUqLq8iMf2nHX9SZj6VFo2Hgn4dQhjXqy2Rth18e9uOP7K1H8G+Xhdt8wrBm0kDOzBpYAcACkoqSTyex49Hz5JwLI/1xyr58qhxJXuviGCGdfnprKFDa3/TPzsYV3R+cyCHf3x5gNScYoZ1bsPAtqWmrMvlEsFfPttHSlZhrdMutmOuS98OrfjTpH51Tn/yySc5cuQIgwYNwtvbm6CgICIjI9m9ezcpKSlMnjyZjIwMysrKePjhh5k9ezYAnTt3ZseOHZw6dYpbbrmF0aNH88MPPxAVFcXq1avx96/9B/fOO+/w9ttvU1FRQffu3Vm0aBEBAQFkZ2czZ84c0tLSAJg/fz6XX345Cxcu5KWXXkIpRWxsLIsWLWr0NhDCbAWllew4lsf2o8Yr+UQBFqvGy0MxIDqE34zuwsguYcR1bkMrP2+7rjskwJurekdwVe8IANZ/vZGQLrHGGcOxs3y6K4vF244DEBMWwK1DO3LbsI6EB/naNY7z9mYW8NzaFLal5dE1PJD/zIjjmr4RfPvtt6asz+USgSM8//zzJCcns3v3bjZt2sT1119PcnIyXbp0AeD9998nNDSU0tJShg0bxtSpUwkLC/vZMg4fPsyyZct45513uPXWW1mxYgV33XVXreubMmUKs2bNAuDpp5/mvffe48EHH+Shhx5izJgxrFq1CovFQnFxMfv27eO5555jy5YthIeHk5dX383eQjSf/JIKErOr2PxZCtuPniHlZOGF6plBHVszd0w3RnQNZUinNgT6Nu+uysdTMaJrGCO6Gv+nVRYrB04V8ePRPP6Xks2L6w7yyoZDTOwfyYzLYhga08YuV95lni3hpXUH+XR3FqGBPvz1pn7cPrwT3iY3aLtcIqjvyL2oqIjg4GDTYxg+fPiFJADw2muvsWrVKgAyMjI4fPjwLxJBly5dGDRoEABxcXEcO3aszuUnJyfz9NNPk5+fT3FxMddeey0A33zzDQsXLgTA09OTkJAQFi5cyLRp0wgPDwcgNDTUXh9TiEbTWrP9aB6LtqWzLvkUVVaNr1c6gzu15qGrelzY8ft5O1e7mZenB/2jQugfZZyZpOYUsXjbcVYkZrJmTxa92wdz18gYJg+OIqgJSaugtJK3NqXy3y3HUMD9Y7sxZ2w3u5/51MXlEoEzqN5f+KZNm9iwYQNbt24lICCAsWPH1nrjlq/vT6eYnp6elJbWXRd499138+mnnzJw4EA++OADNm3aVGdZ7QTdVwhRXF7Fqp2ZLNqWzqHsYlr5efHryzoTUXmSu28ci6+Xc+34L6Z7u2D+fGM/fj+hF6t3Z7FoazpPf5rM818eYMqQKO4aGUPPiIsfdFZUWVmyPZ3Xvj5MfmklNw+O4rFrejVbO8R5kgjsIDg4uM4G34KCAtq0aUNAQAAHDhxg27Ztl7y+oqIiIiMjqaysZMmSJURFRQEwfvx45s+fzyOPPHKh76Dx48dz88038+ijjxIWFkZeXp6cFYhmcyi7iEVb01m5M5NzFRb6R7XihamxTBrYAX8fTzZtymlxSaC6AB8vbh/eienDOrLzeD6Lt6UT/2MGC7emM6JLKDMui+Gavu3x8fp51Y7Wmq+ST/HPrw5w7EwJo7qH8dTEPvSPCnHI55BEYAdhYWGMGjWK/v374+/vT0RExIVpEyZMYMGCBcTGxtKrVy9Gjhx5yev729/+xogRI4iJiWHAgAEXktCrr77K7Nmzee+99/D09GT+/Plcdtll/PGPf2TMmDF4enoyePBgPvjgg0uOQYi6VFqsrNt3ikVb09l+NA8fLw9uiI1kxsgYBnVs7ZJnqEoZN6jFxbTh6ev78HFiJou3pTNv6S7aBvty+7CO3D6iE5Eh/iSmn+Xva/eTmH6WnhFB/PeeYYzt2dah20USgZ0sXbq01vG+vr58+eWXtU473w7g6+tLcvJPz+957LHH6l3X3LlzmTt37i/GR0REsHr16l+MnzlzJjNnzqx3mUJcqlMFZSz98TjLfjxOblE50W38eXJib24d2pHQQMdei9+cwoJ8mTOmG7Ou6Mq3h3JYtDWd1zem8uamI/Tr0IqkzALaBvvy/JQBTIuLdoo7myURCCGaTGvN1iNnWLQtnfUp2Vi1ZmzPtsy4LIYxPdvh6eF6R/8N5emhLlySmpFXwpLtx9l4IIdHftWDWVd0bfYroerjPJGIX3jggQfYsmXLz8Y9/PDD3HOPdMskHEtrzdq9p/j3hkOk5hTTOsCb+0Z34Y4RnYgJc76Hvjtax9AAnpzYmycn9nZ0KLWSRODE3nzzTUeHIMQv7M0s4K+f7yPh2Fl6RQTz0i0DuSE20uku+RQNJ4lACNEg2YVlvPDVQVbszCQs0Ie/3zyA24Z1dOvqH1chiUAIUa/SCgvvfJfG/E1HsFg1vx3TlQfGdW+2m52E+SQRCCFqZbVq1uzJ4p9fHeBkQRkT+7fnqYl96BQW4OjQhJ1JIhBC/EJi+ln+9nkKuzPy6dehFf++bRAju4ZdfEbRIjn+AlY3FBQUBEBWVhbTpk2rtczYsWPZsWNHc4YlBCfyS3lo2S6mzv+BE/mlvDgtls/mjZYk4OLkjMCBOnTowCeffCLPIxAOd668igXfHuHtzUYX5g9e1Z05Y7o51bXuwjyu9y1/+SSc2lvrJH9LFXg24SO3HwATn69z8hNPPEFMTMyFB9P8+c9/RinF5s2bOXv2LJWVlTz77LPcdNNNP5vv2LFj3HDDDWzdupXS0lLuueceUlJS6NOnT72dzoFxd3FCQgKlpaVMmzaNv/zlLwAkJCTw8MMPc+7cOXx9ffn6668JCAjgiSeeYN26dSilmDVrFg8++GDjt4NwOVarZsXOTF5cd5CconJuHNiBJyb2JqqZOz0TjuV6icABpk+fziOPPHIhEXz00Ud89dVXPProo7Rq1YrTp08zcuRIbrzxxjr7E5k/fz4BAQEkJSWRlJTEkCFD6l3nc889R2hoKBaLhfHjx5OUlETv3r257bbbWL58OcOGDaOwsBB/f3/efvttjh49yq5du/Dy8pJnEggAkk8U8MdPk9mTkc+gjq2Zf1cccTFtHB2WcADXSwT1HLmXmvQ8gsGDB5OTk0NWVha5ubm0adOGyMhIHn30UTZv3oyHhwcnTpwgOzub9u3b17qMzZs389BDDwEQGxtLbGxsvev86KOPePvtt6mqquLkyZOkpKSglCIyMpJhw4YB0KpVKwA2bNjAnDlz8PIyvm7pfdS9FZVV8vL/DvHhD8cIDfTh5VsHMnlQFB5yP4Dbcr1E4CDTpk3jk08+4dSpU0yfPp0lS5aQm5tLYmIi3t7edO7cudbnEFTX0N4Hjx49yksvvURCQgJt2rTh7rvvpqysrM5nD8gzCQQYv4Mv9p7kb5+nkFNUzp0jOvH4Nb0JCZD7AdydXDVkJ9OnTyc+Pp5PPvmEadOmUVBQQLt27fD29mbjxo2kp6fXO/+VV17JkiVLAOMJZElJSXWWLSwsJDAwkJCQELKzsy/0btq7d2+ysrJISEgAjOcWVFVVcc0117BgwQKqqqoApGrIDWWfszLzvwnMW7qL8CBfVt0/imcnD5AkIAA5I7Cbfv36UVRURFRUFJGRkdx5551MmjSJoUOHMmjQIHr3rr+zqblz53LPPfcQGxvLoEGDGD58eJ1lBw4cyODBg+nXrx9du3Zl1KhRAPj4+LB8+XIefPBBSktL8ff3Z8OGDdx3330cOnSI2NhYvL29mTVrFvPmzbPr5xfOqbzKwoJNaby+pRQ/70r+NKkvM0bGOEXXx8J5SCKwo717f7paKTw8nK1bt9Zarri4GIDOnTuTnJxMUVER/v7+xMfHN3hddT1cZtiwYbU+Be3ll1/m5ZdfbvDyRcu3JfU0/+/TZNJOn2N4e09e/80YIlr5OTos4YQkEQjhYnKKynj28/2s2ZNFTFgAC38zHGvWPkkCok6SCJzciBEjKC8v/9m4RYsWMWDAAAdFJJyVxapZsj2dF786SHmVlYfH92Du2G74eXuyKcvR0QlnZmoiUEpNAF4FPIF3tdbP15jeBngf6AaUAb/RWif/YkEN4KpXxmzfvr1Z1qO1bpb1CHMkZebz9KfJJGUWMLp7OH+9qR9d2wY5OizRQpiWCJRSnsCbwNVAJpCglFqjtU6pVuwPwG6t9c1Kqd628uMbuy4/Pz/OnDlDWFiYSyYDs2mtOXPmDH5+UnXQ0lismhfWHeDtzWmEB/ny2u2DmRQbKf8HolHMPCMYDqRqrdMAlFLxwE1A9UTQF/gHgNb6gFKqs1IqQmud3ZgVRUdHk5mZSW5ubr3lysrKnHJn5wxx+fn5ER0d7dAYROMUllXy4NJdfHsol9uHd+Kp63rLMwJEkyizqgSUUtOACVrr+2zDM4ARWut51cr8HfDTWv9OKTUc+MFWJrHGsmYDswEiIiLiGnN1TXXFxcUXev50Js4aFzhvbO4eV06JlVcSy8gu0czo68PYjvUnAHffXo3linGNGzcuUWs9tNaJWmtTXsAtGO0C54dnAK/XKNMK+C+wG1gEJAAD61tuXFycbqqNGzc2eV4zOWtcWjtvbO4c15bUXD3wL+v0wL+s0z+knm7QPO68vZrCFeMCdug69qtmVg1lAh2rDUcDP7t2QWtdCNwDoIxKzaO2lxCiFku2p/On1fvoEh7IuzOHEhMW6OiQhAswMxEkAD2UUl2AE8B04I7qBZRSrYESrXUFcB+w2ZYchBDVVFmsPPvFfj744RjjerXltdsHEyztAcJOTEsEWusqpdQ8YB3G5aPva633KaXm2KYvAPoAC5VSFoxG5HvNikeIlqqgpJJ5y3by3eHTzLqiC09O7IOn9BQq7MjU+wi01muBtTXGLaj2fivQw8wYhGjJ0nKLue/DHWScLeGFqbHcOqzjxWcSopHkzmIhnNT3h09z/5JEvDw9WHLfSIZ3kedICHNIIhDCyWitWbQtnb98lkL3tkG8O3MoHUMDHB2WcGGSCIRwIpUWK39es48l24/zqz7teGX6YILkAfLCZPILE8JJnD1Xwf1LdrI17QxzxnTj8Wt7SaOwaBaSCIRwAqk5Rdz74Q5O5pfxr1sGMjVOuvsQzUcSgRAOtvFADg8t24WvtwfLZo8gLkYahUXzkkQghINorVnwbRovrDtA38hWvP3roUS19nd0WMINSSIQwgHKKi08sSKJ1buzuD42kpemDcTfx9PRYQk3JYlAiGZ2sqCU2QsTSc4q4PFre3H/2G7y/ADhUJIIhGhGiel5/HbRTkorqnhnxlB+1TfC0SEJIYlAiObyUUIGT3+aTGRrP5bOGkHPiGBHhyQEIIlACNNV7zl0dPdw3rhjMK0DfBwdlhAXSCIQwkRnz1Uwb9lOtqSe4TejuvCH63rj5enh6LCE+BlJBEKY5FB2Efd9uINTBWW8MC2WW4dKz6HCOUkiEMIE6/ed4tHluwnw9WLZ7JHExbRxdEhC1EkSgRB2pLXmjW8O89L6Q8RGh/CfGXFEhshNYsK5SSIQwk5KKqp4a085CacOMXlQB56fGouft9wkJpyfJAIh7CArv5T7PtzB/lMWnprYm9lXdpWbxESLIYlAiEu0OyOfWQt3UFZh4ZE4X347ppujQxKiUeQ6NiEuwedJWdz2n634eXuw8v7LGdhWjq1EyyOJQIgm0Frz2teHmbd0FwOiQvj0/lH0kDuFRQslhy9CNFL1nkOnDI7iH1MH4OsljcKi5ZJEIEQj5BaV89tFO9h5PF96DhUuQxKBEA104FQh936wgzPnynnrziFcNyDS0SEJYReSCIRogG8OZPPg0l0E+nrx0W8vIza6taNDEsJuJBEIUQ+tNe9vOcZzX6TQJ7IV784cKncKC5cjiUCIOlRarPxpzT6Wbj/ONX0jeGX6IAJ85F9GuB75VQtRi4KSSu5fmsiW1DPMHduNx6/phYeHNAoL12TqfQRKqQlKqYNKqVSl1JO1TA9RSn2mlNqjlNqnlLrHzHiEaIhjp89x8/wt/Hg0jxenxfLEhN6SBIRLM+2MQCnlCbwJXA1kAglKqTVa65RqxR4AUrTWk5RSbYGDSqklWusKs+ISoj7b0s4wZ3EiAIvvHcGIrmEOjkgI85l5RjAcSNVap9l27PHATTXKaCBYGRdiBwF5QJWJMQlRK601i7elM+O97YQF+rD6gVGSBITbUFprcxas1DRggtb6PtvwDGCE1npetTLBwBqgNxAM3Ka1/qKWZc0GZgNERETExcfHNymm4uJigoKCmjSvmZw1LnDe2OwZV7lFs3BfBVuyqhgQ7smcgb4EejetKsgdtpc9SVyNcylxjRs3LlFrPbTWiVprU17ALcC71YZnAK/XKDMN+DeggO7AUaBVfcuNi4vTTbVx48Ymz2smZ41La+eNzV5xHTtdrCe8sll3fvJz/fL6g9pisTpFXPYmcTWOK8YF7NB17FfNvGooE6j+kNZoIKtGmXuA521BpiqljmKcHfxoYlxCAPD1/mweWb4bD6V4f+YwxvVu5+iQhHAIM9sIEoAeSqkuSikfYDpGNVB1x4HxAEqpCKAXkGZiTEJgsWr+tf4g9364g06hAXz+4GhJAsKtmXZGoLWuUkrNA9YBnsD7Wut9Sqk5tukLgL8BHyil9mJUDz2htT5tVkxCnD1XwUPxu/ju8GluiYvmb5P7y+Mkhdsz9YYyrfVaYG2NcQuqvc8CrjEzBiHOS8rMZ+7ineQWlfOPKQOYPqyj9BwqBHJnsXADWmviEzL40+p9tA325ZO50mmcENVJIhAurazSwjOrk/loRyZX9Ajn1emDCQ30cXRYQjgVSQTCZWXklTBncSL7sgp58KruPPKrnnhKVxFC/IIkAuGSNh7M4ZH43Vi15r2ZQxnfJ8LRIQnhtCQRCJditWpe/fowr31zmN7tW7HgriHEhAU6OiwhnJokAuEyisureCR+Fxv25zB1SDTPTu6Pv49cGirExUgiEC7hRH4p936QwOGcYv56Uz9mjIyRS0OFaCBJBKLF23X8LLMWJlJeaeG/dw/jyp5tHR2SEC2KJALRoq3Zk8VjH++hfSs/ls0aQY+IYEeHJESLI4lAtEhaa17ZcIhXNhxmeOdQFsyIk/sDhGgiSQSixSmrtPCfpHK2nTzM1CHR/H1Kf3y9pFFYiKaSRCBalJyiMmYvTGTPSQtPTOjNnDFdpVFYiEskiUC0GPtPFnLvBwmcLalk3mBf5o7t5uiQhHAJZj6PQAi72ZCSzdT5P2DV8PGcy4iLkGMYIexFEoFwalpr3v0ujVmLdtCtbRCr542if1SIo8MSwqXIYZVwWhVVVp5ZnUx8QgbXDWjPv24ZJHcKC2ECSQTCKeWXVDB38U62pp1h3rju/O7qnnhIz6FCmEISgXA6abnF3PvhDk6cLeXlWwcyZUi0o0MSwqXVmQiUUi8AadUfLWkb/yjQXmv9hNnBCfezbt8pHvtoD95eHiydNYKhnUMdHZIQLq++M4IbgP61jH8VSAIkEQi7qbJYeXH9Qf7zbRqx0SG8eccQOoYGODosIdxCfYlAa62ttYy0KrmDR9hRTlEZDy7dxfajedw5ohPPTOordwoL0YzqSwQlSqkeWuvD1UcqpXoApeaGJdzFj0fzeGDpTorKKqU9QAgHqS8RPAN8qZR6Fki0jRsKPAU8YnJcwsVprXnnuzT++dVBOoUGsOje4fRu38rRYQnhlupMBFrrL5VSk4HHgQdto5OBqVrrvc0Qm3BRhWWV/P7jJL7ad4oJ/drz4i2xBPt5OzosIdxWfVcN+QHZWuuZNca3U0r5aa3LTI9OuJz9JwuZuziRjLOlPH19H+4d3UU6jRPCwerrYuI14Ipaxl8N/NuccIQr+yQxk5vf2kJJhYX42SO57wrpOVQIZ1BfG8ForfXsmiO11kuUUn8wMSbhYsoqLfzlsxSW/Xicy7qG8drtg2kb7OvosIQQNvUlgvoO1RrUWZ1SagLGfQeewLta6+drTH8cuLNaLH2AtlrrvIYsXzi/jLwS5i5JJPlEIXPHduP/ru6Jl6f0dSiEM6kvEeQopYZrrX+sPlIpNRzIvdiClVKewJsYVUmZQIJSao3WOuV8Ga31i8CLtvKTgEclCbiObw5k8+jyPVi15p1fD+XqvhGODkkIUYv6EsHjwEdKqQ/4+eWjvwamN2DZw4FUrXUagFIqHrgJSKmj/O3AsgYsVzi5KouVVzYc5o2NqfSNbMX8u4YQExbo6LCEEHWo8xzddiYwAqOK6G7g/NVDMzGSwcVEARnVhjNt435BKRUATABWNGC5womdKijjjne388bGVG4dGs3K+y+XJCCEk1Na64sXUmowxhH7rcBRYIXW+o2LzHMLcK3W+j7b8AxguNb6wVrK3gbcpbWeVMeyZgOzASIiIuLi4+MvGnNtiouLCQoKatK8ZnLWuKBxse3OqeLdveVUWuHXfX0YFWXevQHOus0krsaRuBrnUuIaN25cotZ6aK0Ttda1voCeGHcX7we+x7ipLL2u8rXMfxmwrtrwU8BTdZRdBdzRkOXGxcXpptq4cWOT5zWTs8aldcNiK6+06Gc/36djnvhcT3hls07NKXKKuBxB4mociatxLiUuYIeuY79aXxvBAeA7YJLWOhUudEHdUAlAD6VUF+AERrvCHTULKaVCgDHAXY1YtnASGXklzFu2iz0Z+cwYGcMfr++Dn7d0GCdES1JfIpiKsfPeqJT6Coin/ktKf0ZrXaWUmgesw7h89H2t9T6l1Bzb9PPPObgZWK+1PteUDyAc58u9J/n9iiQA5t85hIkDIh0ckRCiKerra2gVsEopFQhMBh4FIpRS84FVWuv1F1u41notsLbGuAU1hj8APmhs4MJxyiotPPtFCou3HWdgx9a8cftgeXaAEC3YRR9VaTtSXwIsUUqFArcATwIXTQTC9RzJLWbe0l3sP1nIrCu68Pi1vfHxkhvEhGjJGvXMYm3c7PUf20u4mZU7M3n602R8vTx4/+6hXNVbbhATwhXIw+vFRZVUVPHM6n18kpjJ8M6hvHr7ICJD/B0dlhDCTiQRiHplFFn52+vfk3b6HA9d1Z2HxveQvoKEcDGSCESttNYs+zGDv24tpXWgL4vvHcGo7uHNs/LTh8Fqgba9oDm7qa4ogYJMCOsOHpLshPuQRCB+Ib+kgqdW7uXL5FP0C/PggzlXmN9tdGUppKyGHf+FjG3GuOAO0O0q6DYOuo6DwDD7rtNqhZO7IW0jHNkIGdvBUmGst88N0OdG6HQZeMq/iXBt8gsXP7P1yBkeXb6b08XlPDmxNz2tx81NArmHIPED2L0EyvIhtBtc8yz4hUDq13Dgc9i9GFDQYZAtMVyFslY2bX1n03/a8R/9FkrPGuMj+sPw2RDWzVjvzoXw49sQEAa9rzeSQpcx4OVjn8/tKsqLwSewec/chN1JIhAAVFqsvPy/Qyz49gidwwJZdf8oBkSHsGlTxsVnbqyqctj/mZEAjn0HHt7GEXjcPdDlyp92KkN+bVQRZe2GI9/Aka/h+1fgu38xytMPTo27kBgI61b7zqg0H459/9POP++IMT44EnpONM42uoyB4GpXQA39DVScg8P/M+JMXmUkBt9W0HMC9L0Ruo0HHzveO1FZCp4+4NEC7sq2VBrbJeFdSN9ibJew7hDeA8J6GH/DexhJ3dvP0dGKBpBEIDh2+hwPx+9iT2YBtw3tyDOT+hLoa8JPIy/N2PnvWgIlp6F1DIz/Ewy+C4La1T6PhydExxmvMY9DWSEc+47szYuIyj0Ah740yoV0gu62pBAQbhztH9kIJxJBW8A7EDqPhuGzjGqmi7U/+ARCv8nGq7LMWF7KGjj4Bez9CLwDoPuvjDOFnteCX6u6l2WpguJTRvtD9VfhCSjIgIITUJoHnr4Q2hXCuxs71uqvgDDHH3UXnTK+vx3/NT5P6xi48nEoKzDadY5tgaTl1WZQ0LqTLTH0/ClZhPeEoAjHfx5xgSQCN6a15uPETP68Zh/enh4/7ybCaoH8dHzKzxg7wqYe2Vkq4eCXsON946hceUKviTD0Huh6VeMbZf1aQe/rOXwqkKixY43kcmSjccaQvNLYUQEoD+gwBK74nbHjjx7W9Godbz9jZ9/zWrC8CunfG0fE+z+H/WuMI/muY6HnBKIyD8D6r207+UxjJ1+UBdpa43OEQEhHaBUF0cOhVQdjh3rmCOQehINfQfXqL7/WNZJDN9tRd1cjaZlFa0j/ARLeMT6ztQq6Xw3DXzMSYc0zmIpzxmc4fQjOpBp/Tx82llFZ8lM5n2AI707/ci84uxwCQsG/NfiHgn8b23Cbn4ZdofpJayjOAd9g+55N2oEkAjdVUFLJHz7dyxdJJxnZNZR/T+5BZHEybFpoNNZmJEBFEZcDbAW8/IydkX9r29821d7XMs7L16jf37kQirONHd7YP8CQGcZOz15CuxqvYfcaSSdzh1HvH3OZEY+9eXoZO/2uY2Hii5CZYCSD/Wvg8Hp6ABz1gZBo4zN3uRJComzD0cbfkChjZ1AfSxUUHDd2qmdSjZ3pmVSjmiupRjfsraKgbW+jDaXDYOPVKurSdpzlxcbRfcJ7kLPPSFwj5hjVZmHd6p7PJxAiY41XdVarkRBPH7Z9FuOvb+ExOHrS+M4q6+luzNPn54khIBQCw40qvuD2P/0Nam+Md2QVW1W5cYByPgleSIipUF4Aty+HXhMcF18tJBG4oe1pZ3gu/ms6l+xlTbdsBlgPoOYnG1UoKGjXF2JvgQ6DObR/Hz07tTPq2kvPGg26pflQmAnZ+4xxFUV1rElBj2uMo//uV5t/9Y2nt5EAmouHB3QaYbyueRby0tiSuJdRv7rx0i8/9fT6Kcn1uPrn0ypKjB3NmfM7mVRjZ/39K7bvEKN67HxS6DAYn/JS44j0Ysnh9GGj7n/3UigvhPaxcOPr0H/apR3FenjYkmC00S5jk7hpE2PHjjUGKsuM31dJnvG7KrX9/cXwWePzH99mVDHWpDyNqqfqCeLCyzbsH2ocrHh6G0mmse0z54/ubQmtenIjP/3nZ4CtooyzuNhbjbO4dr2btAnNJInAHVgtkJOCJX0rhxI2EJWbyBp12vj2s/0heiiMftS4VDJ6qHFUb5NVuImeV4ytf/mWKqNa43ySKD1r7ESihxp1xO5AKQjrRqVPhvn3IPgEQPv+xqu6ylIjOWftMhrYs3YZDezaapzZ7X3KSAyRg35KEsERxvd36Cuj+idtk9F432+ycRVV9LDmq5Lx9gNv2w67oaoq4FyO0X5RdLLG31Nw9hgc32okkYtRHheSwuVWBYmBP08Unt5G8rBUwJk04+j+PC8/o6G8wyBjh3++0TysO/g63wNuapJE4KqKsiHlUzi0zqi+KC/EEwjVrTkVMoi2w67Gt+tlxhGf5yU+SczTy7jG397X+YvG8bYl9ehqD6GqKIHsZA5/+xE9AouN5HBoHWB7MmFwJKCMaptWUXDV0zBkZt2N987Gy+enM436VJYZVZTnE0XpWaO9w1JhVOVYKo33lgqwVJKbcYyoiLBq46tN9w6AAdOMRu/w7sbfVtEt+iZESQSu5NwZ2L/aaDQ99j2g0eE9Odp+Av852o49Hr2ZN+UqbhhY66OjhSvyCYCOwzkRXUKP81Uw5cVwaq+RFE7uNoYHvWBcTuuqN895+0GbGOPVAIc3bTIuRnATLvqtu5HSfDjwBSSvME7rtcU4HR3ze4p73MgfvqtkzZ4shncO5b3pg4hqLZ3FuT3fIKMtpTnbU4RTk0TQEpUXG5dk7lsJqRuM09XWnWDUQ9BvCrQfwMHsYuYsTuR4Xgn/d3VP7h/XHU+PFn75nRDCFJIIWorKUji83jjyP7QeqkqNPnGGzYL+UyFqyIVGvc/2ZPH7T5II9PVi2ayRDO8S6uDghRDOTBKBszv2PSR+CAfXQkUxBLY17sTtPwU6jvxZA1WVxcrzXx7g3e+PEhfThrfuHEJEK7nFXwhRP0kEzip9K2x8zuiLx7+NsePvPxViRtfaoJdbVM68pTvZfjSPmZfF8Mfr+8ojJIUQDSKJwNlk7jASwJFvILAdTHje6Iytni4edh4/y/2Ld5JfWsG/bxvIzYMvcimdEEJUI4nAWWTtho1/h8PrjA7Grv4bDLuv3rs5tdYs2X6cv3y2j/YhfqycO4q+Herp/EwIIWohicDRTiXTL/nvsGm70UfP+GeMOzov0hdNWaWFpz9N5pPETMb2assrtw2idYD0lS+EaDxJBI6ScwA2/QNSPqWNZ4DRIdvIOUbnXheRkVfC3CWJJJ8o5KHxPXhkfA885NJQIUQTSSJobqdT4dt/wt6PjZ4ar3ycbZZBjB57Q4Nm33wol4fid2Gxat6bOZTxfSIuPpMQQtRDEkFzyTsKm1+EPcuMDqpGPQyXPwSBYVRt2nTR2a1Wzfxvj/DS+oP0bBfMf2bE0TncxH7ohRBuQxKB2arKYf3/gx3vgYcXjJgLox9pVKdehWWV/N9He/hfSjY3DuzA81MHEOAjX50Qwj5kb2KmolOw/C6j98+h9xqP9WsV2ahFpOYUMXuh0VXEMzf05Z5RnVEt/UlNQginYmoiUEpNAF4FPIF3tdbP11JmLPAK4A2c1lqPMTOmZpOZCMvvNJ6xe+tC6HtToxdxsqCUO97ZjlXDUukqQghhEtMSgVLKE3gTuBrIBBKUUmu01inVyrQG3gImaK2PK6VaSCfoF7F7GXz2sPHQj3vX//IBIg1QUlHFfR/uoKTCwoq5l9Or/UUebSiEEE1kZh8Ew4FUrXWa1roCiAdqHhbfAazUWh8H0FrnmBiP+SxV8NUf4NM50HE4zNrUpCRgtWp+t3wP+08W8vrtgyUJCCFMpbTW5ixYqWkYR/r32YZnACO01vOqlXkFo0qoHxAMvKq1XljLsmYDswEiIiLi4uPjaxZpkOLiYoKCzHlsnFdlEX1TXiL07G4yo27gSLd70B4NO+GqGdcnhyr4PK2S23v7cG3nS3x62CUyc5tdComrcSSuxnHFuMaNG5eotR5a60SttSkv4BaMdoHzwzOA12uUeQPYBgQC4cBhoGd9y42Li9NNtXHjxibPW6/sFK1fGaj1X8O1TlzY6Nmrx/XJjgwd88Tn+skVSdpqtdovxiYybZtdIomrcSSuxnHFuIAduo79qpmNxZlAx2rD0UBWLWVOa63PAeeUUpuBgcAhE+OyrwNfwMrZxs1hd39hVAk1UcKxPJ5auZfLu4Xx15v6ydVBQohmYWYbQQLQQynVRSnlA0wH1tQosxq4QinlpZQKAEYA+02MyX6sVvj2BYi/w3h49exNl5QEMvJK+O2iRKLa+PPWnUPw9pQupIUQzcO0MwKtdZVSah6wDuPy0fe11vuUUnNs0xdorfcrpb4CkgArRlVSslkx2U15MXw6F/avgdjbYNKr4N30ZwGXVmnu/TDhQrcR0nmcEKI5mXofgdZ6LbC2xrgFNYZfBF40Mw67OnsM4u+EnBS45jm47IELj4hsiiqLlfm7y0nLs7LwN8Pp2tb5GqiEEK5N7ixujKOb4aOZoC1w5yfQffwlL/K5tftJOm3h7zcP4PLu4XYIUgghGkcqohvqx3dg4WTjmcGzNtolCSzZns5/txzjmhgv7hjR6dJjFEKIJpAzgoZI+gjWPgY9J8KUt8Hv0p8CtiX1NM+s3se4Xm2Z3vmcHYIUQoimkTOCi8k9BJ89Ap0uh9sW2yUJHMktZu7iRLq3DeK12wfjIZeJCiEcSBJBfSpK4OOZxhVB094Dz0s/gcovqeC+D3fg7enBuzOHEuzn2DuHhRBCqobq8+XjkLMf7loBrTpc8uIqLVbmLt7JibOlLJs9go6hdT+YXgghmoucEdRl9zLYtRiu+D+7NAxrrXlmdTJb087wwrRY4mKkS2khhHOQRFCbnAPwxe8gZjSMfcoui3zv+6Ms+zGDeeO6M3lwlF2WKYQQ9iCJoKaKc0a7gE+g3doFvjmQzXNr93PdgPb87uqedghSCCHsR9oIavriMcg9CDNWQXD7S15cRZWVxz9Ool+HVvzrlkF4eMgVQkII5yJnBNXtWgx7lsKY30O3cXZZ5MaDOZw5V8H/XdMLfx9PuyxTCCHsSRLBedkpxtlAlythzBN2W+zKnZmEB/lyhXQfIYRwUpIIwOhN9OOZ4BsMU94FD/scuZ89V8E3B3KYPKgDXtKttBDCSUkbgdbGFUJnUuHXq40HztvJ50lZVFo0U4ZE222ZQghhb3KYunMhJC03LhPtcqVdF71i5wl6tw+mb4dL75ZCCCHM4t6J4FQyfPl76DrWuHHMjo7kFrM7I5+pcjYghHBy7psIyouMdgG/1nZtFzhv1c4TeCi4afCld00hhBBmcs82Aq2NHkXz0mDmZxDU1q6Lt1o1q3ad4MqebWkX7GfXZQshhL255xlB4n8h+RMY90foPNrui99+NI8T+aXSSCyEaBHcLxGc3ANfPgndxsPo35myipU7Mwn29eKavva7AkkIIcziVonAs6oEPr4bAsKMJ4152P/jl1ZYWLv3JNcNiMTPW+4kFkI4P/dpI9CaXgffgLPpcPcXEGjOnb7r9p3iXIWFKUOkh1EhRMvgPmcEe5bRLncLjP9/EHOZaatZsTOT6Db+DOsszxsQQrQM7pMI+kziSNe74fKHTVvFqYIytqSeZsrgKOllVAjRYrhPIvANJqPTzaa0C5y3evcJrBpulquFhBAtiPskApNprVmxM5MhnVrTJTzQ0eEIIUSDSSKwk31ZhRzKLpZ7B4QQLY4kAjtZufMEPp4e3BAb6ehQhBCiUUxNBEqpCUqpg0qpVKXUk7VMH6uUKlBK7ba9njEzHrNUWqys2XOC8X3a0TrAx9HhCCFEo5h2H4FSyhN4E7gayAQSlFJrtNYpNYp+p7W+waw4msN3h3M5XVwh1UJCiBbJzDOC4UCq1jpNa10BxAM3mbg+h1mx8wShgT6M6WnfzuuEEKI5KK21OQtWahowQWt9n214BjBCaz2vWpmxwAqMM4Ys4DGt9b5aljUbmA0QERERFx8f36SYiouLCQoKatK8dTlXqXl4YwnjOnpxZx9fp4nLXpw1NomrcSSuxnHFuMaNG5eotR5a60SttSkv4Bbg3WrDM4DXa5RpBQTZ3l8HHL7YcuPi4nRTbdy4scnz1mXJtnQd88TnOikjv8nLMCMue3HW2CSuxpG4GscV4wJ26Dr2q2ZWDWUCHasNR2Mc9VdPQoVa62Lb+7WAt1LKnE6ATLJyZyY92gXRP0oeRymEaJnMTAQJQA+lVBellA8wHVhTvYBSqr1SStneD7fFc8bEmOwq/cw5dqSfZcqQaGwfQwghWhzTrhrSWlcppeYB6wBP4H2t9T6l1Bzb9AXANGCuUqoKKAWm205hWoSVO0+gFEyWx1EKIVowU7uhtlX3rK0xbkG1928Ab5gZg1m01qzclcmobuFEhvg7OhwhhGgyubO4iXaknyUjr1SeOyCEaPEkETTRyp2ZBPh4cm2/9o4ORQghLokkgiYoq7TwedJJJvRvT6Cv+zzkTQjhmiQRNMGG/dkUlVUxVbqUEEK4AEkETbBy5wkiQ/wY2TXM0aEIIcQlk0TQSLlF5Xx7KJfJg6PwlMdRCiFcgCSCRlq9+wQWq2bKYLlaSAjhGiQRNNLKnSeIjQ6hR0Swo0MRQgi7kETQCPtPFpJyslDOBoQQLkUSQSOs2nUCLw/FpIHSpYQQwnVIImigKouVVbtOMK53O8KCmvbcASGEcEaSCBpoy5Ez5BaVM1W6lBBCuBhJBA20cmcmIf7ejOvdztGhCCGEXUkiaICiskrW7TvFpIGR+Hp5OjocIYSwK0kEDVBYVsX43hHSpYQQwiVJj2kNENXanzfvHOLoMIQQwhRyRiCEEG5OEoEQQrg5SQRCCOHmJBEIIYSbk0QghBBuThKBEEK4OUkEQgjh5iQRCCGEm1Naa0fH0ChKqVwgvYmzhwOn7RiOvThrXOC8sUlcjSNxNY4rxhWjtW5b24QWlwguhVJqh9Z6qKPjqMlZ4wLnjU3iahyJq3HcLS6pGhJCCDcniUAIIdycuyWCtx0dQB2cNS5w3tgkrsaRuBrHreJyqzYCIYQQv+RuZwRCCCFqkEQghBBuziUTgVJqglLqoFIqVSn1ZC3TlVLqNdv0JKWU6U+dUUp1VEptVErtV0rtU0o9XEuZsUqpAqXUbtvrGbPjsq33mFJqr22dO2qZ7ojt1avadtitlCpUSj1So0yzbS+l1PtKqRylVHK1caFKqf8ppQ7b/rapY956f48mxPWiUuqA7btapZRqXce89X7vJsT1Z6XUiWrf13V1zNvc22t5tZiOKaV21zGvKdurrn1Ds/6+tNYu9QI8gSNAV8AH2AP0rVHmOuBLQAEjge3NEFckMMT2Phg4VEtcY4HPHbDNjgHh9Uxv9u1Vy3d6CuOGGIdsL+BKYAiQXG3cC8CTtvdPAv9syu/RhLiuAbxs7/9ZW1wN+d5NiOvPwGMN+K6bdXvVmP4v4Jnm3F517Rua8/flimcEw4FUrXWa1roCiAduqlHmJmChNmwDWiulIs0MSmt9Umu90/a+CNgPRJm5Tjtq9u1Vw3jgiNa6qXeUXzKt9WYgr8bom4APbe8/BCbXMmtDfo92jUtrvV5rXWUb3AY0+8O269heDdHs2+s8pZQCbgWW2Wt9DYyprn1Ds/2+XDERRAEZ1YYz+eUOtyFlTKOU6gwMBrbXMvkypdQepdSXSql+zRSSBtYrpRKVUrNrme7Q7QVMp+5/Tkdsr/MitNYnwfhnBtrVUsbR2+43GGdztbnY926GebYqq/frqOpw5Pa6AsjWWh+uY7rp26vGvqHZfl+umAhULeNqXiPbkDKmUEoFASuAR7TWhTUm78So/hgIvA582hwxAaO01kOAicADSqkra0x35PbyAW4EPq5lsqO2V2M4ctv9EagCltRR5GLfu73NB7oBg4CTGNUwNTlsewG3U//ZgKnb6yL7hjpnq2Vco7eXKyaCTKBjteFoIKsJZexOKeWN8UUv0VqvrDlda12otS62vV8LeCulws2OS2udZfubA6zCON2sziHby2YisFNrnV1zgqO2VzXZ56vIbH9zainjqN/aTOAG4E5tq0yuqQHfu11prbO11hattRV4p471OWp7eQFTgOV1lTFze9Wxb2i235crJoIEoIdSqovtaHI6sKZGmTXAr21Xw4wECs6fgpnFVv/4HrBfa/1yHWXa28qhlBqO8f2cMTmuQKVU8Pn3GA2NyTWKNfv2qqbOozRHbK8a1gAzbe9nAqtrKdOQ36NdKaUmAE8AN2qtS+oo05Dv3d5xVW9XurmO9TX79rL5FXBAa51Z20Qzt1c9+4bm+33ZuwXcGV4YV7kcwmhN/6Nt3Bxgju29At60Td8LDG2GmEZjnLIlAbttr+tqxDUP2IfR8r8NuLwZ4upqW98e27qdYnvZ1huAsWMPqTbOIdsLIxmdBCoxjsLuBcKAr4HDtr+htrIdgLX1/R5NjisVo974/O9sQc246vreTY5rke33k4Sxs4p0hu1lG//B+d9VtbLNsr3q2Tc02+9LupgQQgg354pVQ0IIIRpBEoEQQrg5SQRCCOHmJBEIIYSbk0QghBBuThKBEDUopSzq5z2f2q0HTKVU5+o9XwrhDLwcHYAQTqhUaz3I0UEI0VzkjECIBrL1R/9PpdSPtld32/gYpdTXts7UvlZKdbKNj1DG8wD22F6X2xblqZR6x9b3/HqllL/DPpQQSCIQojb+NaqGbqs2rVBrPRx4A3jFNu4NjG66YzE6eHvNNv414FttdIo3BOOOVIAewJta635APjDV1E8jxEXIncVC1KCUKtZaB9Uy/hhwldY6zdZJ2CmtdZhS6jRGdwmVtvEntdbhSqlcIFprXV5tGZ2B/2mte9iGnwC8tdbPNsNHE6JWckYgROPoOt7XVaY25dXeW5C2OuFgkgiEaJzbqv3danv/A0avjwB3At/b3n8NzAVQSnkqpVo1V5BCNIYciQjxS/7q5w8w/0prff4SUl+l1HaMg6jbbeMeAt5XSj0O5AL32MY/DLytlLoX48h/LkbPl0I4FWkjEKKBbG0EQ7XWpx0dixD2JFVDQgjh5uSMQAgh3JycEQghhJuTRCCEEG5OEoEQQrg5SQRCCOHmJBEIIYSb+/8lXwijN3DyeAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "\n",
    "metrics = pd.read_csv(f\"{trainer.logger.log_dir}/metrics.csv\")\n",
    "\n",
    "aggreg_metrics = []\n",
    "agg_col = \"epoch\"\n",
    "for i, dfg in metrics.groupby(agg_col):\n",
    "    agg = dict(dfg.mean())\n",
    "    agg[agg_col] = i\n",
    "    aggreg_metrics.append(agg)\n",
    "\n",
    "df_metrics = pd.DataFrame(aggreg_metrics)\n",
    "df_metrics[[\"train_loss\", \"valid_loss\"]].plot(\n",
    "    grid=True, legend=True, xlabel='Epoch', ylabel='Loss')\n",
    "df_metrics[[\"train_acc\", \"valid_acc\"]].plot(\n",
    "    grid=True, legend=True, xlabel='Epoch', ylabel='ACC')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00811a53",
   "metadata": {},
   "source": [
    "- The `trainer` automatically saves the model with the best validation accuracy automatically for us, we which we can load from the checkpoint via the `ckpt_path='best'` argument; below we use the `trainer` instance to evaluate the best model on the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "2f972950",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Restoring states from the checkpoint path at logs/my-model/version_29/checkpoints/epoch=7-step=2807.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "Loaded model weights from checkpoint at logs/my-model/version_29/checkpoints/epoch=7-step=2807.ckpt\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4070de9968ea4d72b73fbcaec6a3a1c8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Testing: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\">        Test metric        </span>┃<span style=\"font-weight: bold\">       DataLoader 0        </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_acc          </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.6643999814987183     </span>│\n",
       "└───────────────────────────┴───────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1m       Test metric       \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      DataLoader 0       \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_acc         \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.6643999814987183    \u001b[0m\u001b[35m \u001b[0m│\n",
       "└───────────────────────────┴───────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[{'test_acc': 0.6643999814987183}]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.test(model=lightning_model, datamodule=data_module, ckpt_path='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77f0f212",
   "metadata": {},
   "source": [
    "## Predicting labels of new data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f11137f",
   "metadata": {},
   "source": [
    "- You can use the `trainer.predict` method on a new `DataLoader` or `DataModule` to apply the model to new data.\n",
    "- Alternatively, you can also manually load the best model from a checkpoint as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "473d9139",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logs/my-model/version_29/checkpoints/epoch=7-step=2807.ckpt\n"
     ]
    }
   ],
   "source": [
    "path = trainer.checkpoint_callback.best_model_path\n",
    "print(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "7ca9ba4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "lightning_model = LightningModel.load_from_checkpoint(\n",
    "    path, model=pytorch_model)\n",
    "lightning_model.eval();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c1db4af",
   "metadata": {},
   "source": [
    "- Note that our PyTorch model, which is passed to the Lightning model requires input arguments. However, this is automatically being taken care of since we used `self.save_hyperparameters()` in our PyTorch model's `__init__` method.\n",
    "- Now, below is an example applying the model manually. Here, pretend that the `test_dataloader` is a new data loader."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "e1c35e59",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([3, 1, 0, 0, 4])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_dataloader = data_module.test_dataloader()\n",
    "\n",
    "all_true_labels = []\n",
    "all_predicted_labels = []\n",
    "for batch in test_dataloader:\n",
    "    features, labels = batch\n",
    "    \n",
    "    with torch.no_grad():  # since we don't need to backprop\n",
    "        logits = lightning_model(features)\n",
    "    \n",
    "    predicted_labels = torch.argmax(logits, dim=1)\n",
    "    all_predicted_labels.append(predicted_labels)\n",
    "    all_true_labels.append(labels)\n",
    "    \n",
    "all_predicted_labels = torch.cat(all_predicted_labels)\n",
    "all_true_labels = torch.cat(all_true_labels)\n",
    "all_predicted_labels[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "715bf727",
   "metadata": {},
   "source": [
    "Just as an internal check, if the model was loaded correctly, the test accuracy below should be identical to the test accuracy we saw earlier in the previous section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "03d5a220",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 0.6644 (66.44%)\n"
     ]
    }
   ],
   "source": [
    "test_acc = torch.mean((all_predicted_labels == all_true_labels).float())\n",
    "print(f'Test accuracy: {test_acc:.4f} ({test_acc*100:.2f}%)')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97a69f77",
   "metadata": {},
   "source": [
    "## Single-image usage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "f15ef928",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "e1029db9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a293e6a",
   "metadata": {},
   "source": [
    "- Assume we have a single image as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "36032a1a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD5CAYAAADhukOtAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAaT0lEQVR4nO2dbYykVZXH/6fe+r2nZ6Znppt5BWQFFhHYFo0YV0AMa0yQDxpN1vCBOGYjyZq4H1g2WdnsF3ezavywIQ7CihsXJassZENUlrhLVJi1wZEZBGEYB2aYpnvep9+7qp6zH+phM+A9p7qf6nqq9f5/Saer7ql7n/PcqlNP1f3XOVdUFYSQP3wKnXaAEJIPDHZCIoHBTkgkMNgJiQQGOyGRwGAnJBJKrXQWkZsBfB1AEcA3VfXL3uN7+tfpuo0jrRySREpBHKPYRsuiyCY5O4dCxiHtYzm2xDjWmZNvYG76bLBr5mAXkSKAfwZwE4CjAH4hIo+q6q+tPus2juDP//qeoC2L3p/jvJMgq/vB0Avoni77WMWCbRPDliRVx5O6aSmViqYtsSIQQNGxiWEqFO3zml9Kgu33/v1fmH1aebauBXBQVQ+p6hKA7wK4pYXxCCFtpJVg3wrgyHn3j6ZthJA1SCvBHvrQ9TsfSERkt4iMi8j43MyZFg5HCGmFVoL9KIDt593fBuDY2x+kqntUdUxVx3r7h1o4HCGkFVoJ9l8AuERELhSRCoBPAXh0ddwihKw2mVfjVbUmIncA+BEa0tv9qvq82weKJAmvIiLDanzi9BFXIyFrkoL9fBaSJdOmdeM1BUAK4dXz7or90jdfowDEWVWvL9k+NsSrMOVi2CZeTNQMNcHp05LOrqqPAXislTEIIfnAX9AREgkMdkIigcFOSCQw2AmJBAY7IZHQ0mr8ShF4WUgZxnPkNXe8VS6ymVXma0exz9WWHD0fxcrgaFiN8ewe9bptnJlbtDs6gxaM7BqdsaUw7xroJbt4mVl1sZNr1Ei88TPswsa6NxfOcISQPyAY7IREAoOdkEhgsBMSCQx2QiIh19V4VXs1000+WLEhOytfX/Z993Ddz7iqntUX2w3bjySxV5its8taSixBZcXHAuyST54jqvYcqlM7K+u5WfXwPOVCzKPZXvDKTkgkMNgJiQQGOyGRwGAnJBIY7IREAoOdkEjIVXoDgHrdkGsSL6nFknFqTh8bdd7jjPwCd1RxjrZmdq3xZBxPhso2JMxEGLdPVrKMavfxtobK9fl0BlQrsYaJMIQQBjshkcBgJyQSGOyERAKDnZBIYLATEgktSW8ichjANBq719dUdcztoAlQN2qJqV0TTIz3JIWTndTMD/NgngRojJpRumoHpiuejLPqB8v/vPNizZyX4Yjn32ro7Ner6olVGIcQ0kb4MZ6QSGg12BXAj0XkGRHZvRoOEULaQ6sf469T1WMishnA4yLyoqo+ef4D0jeB3QAwsH5Ti4cjhGSlpSu7qh5L/08BeBjAtYHH7FHVMVUd6+kbbOVwhJAWyBzsItInIgNv3gbwEQAHVssxQsjq0srH+C0AHk4LEpYA/Juq/tDrUBBBbyV8SBX7fWd+ISzXJWq7X3C2JqqUshVR1ELYx6qzbZFmLBzpZdJ5mpeZ9ORu45Qtl0udFEE/e3DluB7muJ1XO7bsyovMwa6qhwC8exV9IYS0EUpvhEQCg52QSGCwExIJDHZCIoHBTkgk5FpwslqtYeLoyaCta6Df7FfpDbspTvKao66hz5D/AKCeVE1bYsiDtbpX+NJ+P80qr2WhLXKS42IbtuGzj+Xu25afH9nJ4uTKZ5hXdkIigcFOSCQw2AmJBAY7IZHAYCckEnJdjT97dgY//NHPg7aBkc1mv6v/5LJg+5ahdWafcslequ+t2O9xSWLXwpuvhVdN1UmesVbwgWbbRq3yMrIzXNF5yy84i75VZ6k7cZSSbGRd+reMXoKP97xk0xm8MU1Lxi27LHhlJyQSGOyERAKDnZBIYLATEgkMdkIigcFOSCTkKr0VpIieSjjhpbph2Oy378DrwfarL7Xfq3ZssXWLEiqmreDoHTMLC8H2uWk7eaYq9rE8zajibG3laTJVo/hbwdny6optto/DA2XTtvfQvGk7u2RImE5tQE+eKqgjYbqZMEYf5zLn1+TLRpZkI18CDFOv2zIwr+yERAKDnZBIYLATEgkMdkIigcFOSCQw2AmJhKbSm4jcD+BjAKZU9Yq0bQOA7wHYBeAwgE+q6ulmY5W7SrhgZ1him90+avY7/rN9wfaDh5bMPts2v8O0dastNVUK9pgna2Hb0qItvc1VbVvdkVYqXrac8xZtSX2enFRbmjNt5Zr9Elmct+W8hXnj3BJbrisXbdmo4tiKXtqegaeuJYldU7DmSVsZJEDP6I1nvnQc35czS98CcPPb2u4E8ISqXgLgifQ+IWQN0zTY0/3WT72t+RYAD6S3HwDw8dV1ixCy2mT9zr5FVScAIP1vV54ghKwJ2r5AJyK7RWRcRMari7PtPhwhxCBrsE+KyCgApP+nrAeq6h5VHVPVsXJXX8bDEUJaJWuwPwrgtvT2bQAeWR13CCHtYjnS24MAPgRgWESOAvgSgC8DeEhEbgfwGoBPLOdg5bJgy9busHGn/bV/4OBgsP3FQxNmnwlHytt1kWnC/PykaUuMTK6l+UWzz0ZnW6u+ii1dzUzbcthi1e5XM5QXLdpy49ZN201bX2L7UZuz56o6F5YcNw/aL7mNg7aPvRUni7GYoZijk1WYOAVE604lzaSercpmwavquUK6Sl62ZxNU9dOG6casDhFC8oe/oCMkEhjshEQCg52QSGCwExIJDHZCIiHXgpP9PV247spdQdvBbkOSA/CuD18TbN//tYfNPs8+86JpG9tlZ8RtHrLlk6KG95ZbcuSYy3fY0ttW51j7Xzxr2iamzti2k29PY2gwddb+9eLiJbZ0ODdzzrTJ/LRp2zm8Mdi+bXOX2acidoagGvvsNcUsOOnIdU6GnZe9lniynCPniZHGWCrZ4WklxBUdFY9XdkIigcFOSCQw2AmJBAY7IZHAYCckEhjshERCrtJbuQCMlMPyxK/P2RLPdTdfHWyvLcyYfR778S9N295nD5u2W2+wM8CSmZPB9st3DJl9dvTZktfJqVdN22tHjpi2har9tB078ptge6k3nDkIAL/cv8+0FcSWRC/a9kemrb+/Jzyel4fmKl7edcnWm8TIKFMn602d8TwTHDmvULD3zDP1vIKxXx6A7p5e4zj2PPHKTkgkMNgJiQQGOyGRwGAnJBIY7IREQq6r8dPnTuN/nviPoO2AXmD2e//YO4Pt733vFWafAy/Z9en+66lx01Yv2skY1enwSvemPrt22tTwJtP20sHweAAwldgrsYnaK+QnToYTaIZL4dVxAFhCeGUXAHbutJOGeo0VYQBIauF5rItzXs61pw5n2yW1l8iLRpKJOKvWiZPs4m2jpc52Xl6dOWvIWs0+51rdUhns4/DKTkgkMNgJiQQGOyGRwGAnJBIY7IREAoOdkEhYzvZP9wP4GIApVb0ibbsbwGcBHE8fdpeqPtZsrHK5jG2jW4K2sydsmeGnj/wk2F49YW9NdOLIcdO2fcvFpu3Y67Z8smUg3E8Kb5h9zi3ZddrWbdhm2haXjH2cAMzN23O1Y9uFwfZSl52IMTJi+zGyJfx8AYCIPVdVS3qzlU2UKraEWSvY52yLeUBSDdfXE+c6p+rZ7HOem3O2ylqy6/xZFIv2mS3MhpPAvFp3y7myfwvAzYH2r6nqVelf00AnhHSWpsGuqk8CCJcsJYT83tDKd/Y7ROQ5EblfRNavmkeEkLaQNdjvAXAxgKsATAD4ivVAEdktIuMiMu59pyGEtJdMwa6qk6paV9UEwL0ArnUeu0dVx1R1rLfX/i01IaS9ZAp2ERk97+6tAA6sjjuEkHaxHOntQQAfAjAsIkcBfAnAh0TkKjSKZx0G8LnlHGxgcBAfuOnGoG3HkdNmv2OnwjLDuovDWwwBwC0fu8y0bd+2w7R1OdlhPZVwRlEFC2afilNzTZxsrRmvIJtTz0zqRm21gl1zrdvZesvamggAnMQxND70/S6Jka0FAKfP2jLl6VlHs1Nbptw0PBBs73FkPkdRRLVq+3H2rL1l1+SkLc8eMeoNnj5tx4Qle3rSYNNgV9VPB5rva9aPELK24C/oCIkEBjshkcBgJyQSGOyERAKDnZBIyLXg5PFTJ/CNB/8laKvN2FLT8cnwtkvV2pLZ58brrzdtN133RdPW199n2qzCgAVnGpcW7PNKnMqGw2V7TK9YIgzpxZLCGjbbDy/zCs4WSta5FZyCk8eNYpkAMHnMlqG6nYy+CzaFC5n2dneZfaD266qnx5Zmt22zswevfNe7TNszzzwbbH/66b1mn6HhoWB7uWxLiryyExIJDHZCIoHBTkgkMNgJiQQGOyGRwGAnJBJyld6KIhiqhGWS4no71329IXfUnQyqotqn9vRTYakDAEZHRk3b4GA4g2pmftbsM7RhnWnr6w2PBwAldbLePBnNaPcKLC4u2ll7ExPHTFu3I1/19feH+3TZ0tWO7fZ+fyMXbDZtpYIt55WMl0jizCEcedCTKU+fseXBroo9V8PDw8H2G264wexzzHheyiX7dc8rOyGRwGAnJBIY7IREAoOdkEhgsBMSCbmuxvf39OD9V747aEucemxLxhY+VbXrgc3Ozpu2n/3sEdPWXRk0bevWjQTbe/qHzD4XXvQO07Z5s11Dr9epC+clp1im/n77vJ5+yk64+OZ931jxsQBgaCisQoyM2ivu7xl7j2kbGQnPPQCsGxwybYMDxnl7l7mCU4TOYXrarqGnziZVVtLQ9Pw5s8/xE+HV+Jqx7RbAKzsh0cBgJyQSGOyERAKDnZBIYLATEgkMdkIiYTnbP20H8G0AI2gUHdujql8XkQ0AvgdgFxpbQH1SVe1MAADVxVlMvvxU0ObkF5hJC+rs01Mu23XJ1lnF5ADU5o+btlPzYblDKkNmn9ePvmjaikV7+r0tmfwNMsNzMjsbli8B4MirdrLLmdNTps1KxgAAMZJJrrzSrsXm1WmrOUlPG4a3OH6E2/cZdd8A4PSJcM1DAFBnO6963d6Gqlp1ZDlj+6qlJbsWnrXlVVJvTXqrAfiiql4G4H0APi8ilwO4E8ATqnoJgCfS+4SQNUrTYFfVCVV9Nr09DeAFAFsB3ALggfRhDwD4eJt8JISsAiv6zi4iuwBcDWAvgC2qOgE03hAA2AnHhJCOs+xgF5F+AN8H8AVVtX/H97v9dovIuIiMz87Z3xsJIe1lWcEuImU0Av07qvqDtHlSREZT+yiA4EqOqu5R1TFVHevrdQrzE0LaStNgFxFBYz/2F1T1q+eZHgVwW3r7NgB2dgkhpOMsJ+vtOgCfAbBfRPalbXcB+DKAh0TkdgCvAfhEs4E0qaE2fypoSxKnJhjC+kmxaG91kyzZUo01HgAU7CFRLoQlr1lni6f1W4ZMmzq10xbm7Lp2p06eMG1zc2GJ58Rxu8+rjvTW1W1fD3btsLc7mp0OZx2emrL9+PX+/abNk9fKFftlvLAQrq93+LfPmX1ee8W2ea+d/j5767CiIy13GVs2FYv23Fs7h6kj/zUNdlX9KewzvLFZf0LI2oC/oCMkEhjshEQCg52QSGCwExIJDHZCIiHXgpNLSQFHZsLyVbHkbOFjZIeVYetkRUfW8go2YtHONNIk/N44t2jLMaq2hCbOOS86GU/1ui31iRhz5ZzzwZdeMm3ObkLYOLTBtF32zkuD7V7W2Cu/ed60zcycNW2HXjpg2gpG2tupyTfMPoM9dnHO4yfsrMiS2tfODeucrb6MApcF51KcpSYmr+yERAKDnZBIYLATEgkMdkIigcFOSCQw2AmJhFylt1q9gKnpsPRWcHSG7q6wxFZ2dKGSI2t50luhaGcuFQphGaen35YA6xrOugKAgiPVVCq2nCdi91MNz8nsjF2A848vvcy01et2wZGBXvu8+3vDPg4O2hJUd0+P7Udiz+ORl+1suR6jOGevc6xSt+3jUN3WvPr77ddO/4Btq1TC89hltAMwK2l6hVZ5ZSckEhjshEQCg52QSGCwExIJDHZCIiHX1fh6XXHmXHhV1Vshn5kLb2lTLjmr2cYKPgB0Vewqt7399rZLZSP7YL5urxSXnTpiIxvsumqjo3YZ/npir5BX58JVvrdvtOfj8l3Dpq2x41eYYslOyKkn4Vpo4tRwE2dbrkLG65LAmCuZM/skTh23+kK4hiIAVAY2mbba3Ixpq86Gz3veUH8AoGAketVrdgIVr+yERAKDnZBIYLATEgkMdkIigcFOSCQw2AmJhKbSm4hsB/BtACNo6DB7VPXrInI3gM8CeLMo112q+pg3VrVWw/ETp4M2LxHGSkBxysyh5Eh5JS+BxhmzXA776I9n2yYnbfnn0GG7RhoKtrxi1TPrcvwQb+4dOaxcCEuiAFAwJDvruWyGOBKgh/W6KjmSqDgJSlJYZ9rOnrV9LBRXvoOxJ0Vaplrd9mE5OnsNwBdV9VkRGQDwjIg8ntq+pqr/tIwxCCEdZjl7vU0AmEhvT4vICwC2ttsxQsjqsqLv7CKyC8DVAPamTXeIyHMicr+IrF9t5wghq8eyg11E+gF8H8AXVPUcgHsAXAzgKjSu/F8x+u0WkXERGa/X7O94hJD2sqxgF5EyGoH+HVX9AQCo6qSq1lU1AXAvgGtDfVV1j6qOqepYsWRX0SCEtJemwS6NJcH7ALygql89r330vIfdCsDeloMQ0nGWsxp/HYDPANgvIvvStrsAfFpErgKgAA4D+FyzgQSAJGFpqO5kGlWTcHZV40NFGNUM++MAUCcry8Tp4mV5QSecQe1zk4Kz/VMh/JRKwa655mUcevJPV9GReYzLSME9lmlC2asp6Eif1rmVivbBPPnVlSkdm1cv0fLRymwDgJIhAy9UW5DeVPWnCL+cXU2dELK24C/oCIkEBjshkcBgJyQSGOyERAKDnZBIyLXg5ODgAD5y058GbfW6LSctLoWzvGpV+xd5tZo9Xq3myHxVr1/YlhjSIADUnSykuiFDNvp5Y9rnXTfOre7MVdWZj8Txo6a2NLRoDFlftMdLEkdKxbxtc2TWxLJ50qzjh+tjRrnXwpM9LUV3ZtbZbqxFfwghvycw2AmJBAY7IZHAYCckEhjshEQCg52QSMhVeuvuruDSS3euuJ+V3KZqSxOKbLKWOpKdJcm44zlyjDpZUklin5sn/1gyYK1uF6msOkVFEk86tBU7U3L0ZM+6I2HWvP3XnH6WdOh0QVKzn7Na1ZFts86j4YzXp2qc1xvHfmv24ZWdkEhgsBMSCQx2QiKBwU5IJDDYCYkEBjshkZCr9KZJgurc7Mr7Wdk/7l5YjiznZJuJI9mJ8dYo4mRdOcdS8Qo2eoUvnXOT8FNqtTeMGbLGYBcPBdwanJn8cFTWJlKkseec2Bl7ns3DmytPJrb6qXdeRp/xZ35u9uGVnZBIYLATEgkMdkIigcFOSCQw2AmJhKar8SLSDeBJAF3p4/9dVb8kIhsAfA/ALjS2f/qkqp72xioUiujvHwzavIQRqwadt1rpJcJ4K/Ve2a/EzMhx3jOdlXrxVmgznptm2CrLW0X28FatrTn2juTqD95z5tTCKxoSSpb6bk1MKHpjGttypc4YzStXmwpFZ3sq24P/ZxHADar6bjS2Z75ZRN4H4E4AT6jqJQCeSO8TQtYoTYNdG8ykd8vpnwK4BcADafsDAD7eDgcJIavDcvdnL6Y7uE4BeFxV9wLYotrYhjT9v7ltXhJCWmZZwa6qdVW9CsA2ANeKyBXLPYCI7BaRcREZn56ZzugmIaRVVrQar6pnAPw3gJsBTIrIKACk/6eMPntUdUxVxwb6B1rzlhCSmabBLiKbRGQovd0D4MMAXgTwKIDb0ofdBuCRNvlICFkFlpMIMwrgAREpovHm8JCq/qeIPAXgIRG5HcBrAD7RbCDVBIu1RcPqJQoYBcO8BA5HuvJqxrkYx3OTRdwBHaunUblSWdjm5dUU3KSbbFgjerJhlvEaRmdMw+SN58mU7tS7uq1du84a0pfejOu042DTYFfV5wBcHWg/CeDGZv0JIWsD/oKOkEhgsBMSCQx2QiKBwU5IJDDYCYkE8bLNVv1gIscBvJreHQZwIreD29CPt0I/3srvmx87VXVTyJBrsL/lwCLjqjrWkYPTD/oRoR/8GE9IJDDYCYmETgb7ng4e+3zox1uhH2/lD8aPjn1nJ4TkCz/GExIJHQl2EblZRH4jIgdFpGO160TksIjsF5F9IjKe43HvF5EpETlwXtsGEXlcRF5O/6/vkB93i8jr6ZzsE5GP5uDHdhH5iYi8ICLPi8hfpu25zonjR65zIiLdIvK/IvKr1I+/S9tbmw9VzfUPQBHAKwAuAlAB8CsAl+ftR+rLYQDDHTjuBwFcA+DAeW3/CODO9PadAP6hQ37cDeCvcp6PUQDXpLcHALwE4PK858TxI9c5QSMDtz+9XQawF8D7Wp2PTlzZrwVwUFUPqeoSgO+iUbwyGlT1SQCn3tacewFPw4/cUdUJVX02vT0N4AUAW5HznDh+5Io2WPUir50I9q0Ajpx3/yg6MKEpCuDHIvKMiOzukA9vspYKeN4hIs+lH/Pb/nXifERkFxr1Ezpa1PRtfgA5z0k7irx2IthD5Tc6JQlcp6rXAPgzAJ8XkQ92yI+1xD0ALkZjj4AJAF/J68Ai0g/g+wC+oKrn8jruMvzIfU60hSKvFp0I9qMAtp93fxuAYx3wA6p6LP0/BeBhNL5idIplFfBsN6o6mb7QEgD3Iqc5EZEyGgH2HVX9Qdqc+5yE/OjUnKTHPoMVFnm16ESw/wLAJSJyoYhUAHwKjeKVuSIifSIy8OZtAB8BcMDv1VbWRAHPN19MKbcihzmRRrG1+wC8oKpfPc+U65xYfuQ9J20r8prXCuPbVhs/isZK5ysA/qZDPlyEhhLwKwDP5+kHgAfR+DhYReOTzu0ANqKxjdbL6f8NHfLjXwHsB/Bc+uIazcGPD6DxVe45APvSv4/mPSeOH7nOCYArAfwyPd4BAH+btrc0H/wFHSGRwF/QERIJDHZCIoHBTkgkMNgJiQQGOyGRwGAnJBIY7IREAoOdkEj4P0w6IIvRV/IIAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from PIL import Image\n",
    "\n",
    "\n",
    "image = Image.open('data/cifar10_pngs/90_airplane.png')\n",
    "plt.imshow(image, cmap='Greys')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df4829b7",
   "metadata": {},
   "source": [
    "- Note that we have to use the same image transformation that we used earlier in the `DataModule`. \n",
    "- While we didn't apply any image augmentation, we could use the `to_tensor` function from the torchvision library; however, as a general template that provides flexibility for more complex transformation chains, let's use the `Compose` class for this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "d990431b",
   "metadata": {},
   "outputs": [],
   "source": [
    "transform = transforms.Compose([transforms.ToTensor()])\n",
    "\n",
    "image_chw = resize_transform(image)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "811de2fd",
   "metadata": {},
   "source": [
    "- Note that `ToTensor` returns the image in the CHW format. CHW refers to the dimensions and stands for channel, height, and width."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "14785180",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([3, 32, 32])\n"
     ]
    }
   ],
   "source": [
    "print(image_chw.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d17591e9",
   "metadata": {},
   "source": [
    "- However, the PyTorch / PyTorch Lightning model expectes images in NCHW format, where N stands for the number of images (e.g., in a batch).\n",
    "- We can add the additional channel dimension via `unsqueeze` as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "b4422205",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 3, 32, 32])\n"
     ]
    }
   ],
   "source": [
    "image_nchw = image_chw.unsqueeze(0)\n",
    "print(image_nchw.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdd09c4c",
   "metadata": {},
   "source": [
    "- Now that we have the image in the right format, we can feed it to our classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "6e76be24",
   "metadata": {},
   "outputs": [],
   "source": [
    "with torch.no_grad():  # since we don't need to backprop\n",
    "    logits = lightning_model(image_nchw)\n",
    "    probas = torch.softmax(logits, axis=1)\n",
    "    predicted_label = torch.argmax(probas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "88b1576d",
   "metadata": {},
   "outputs": [],
   "source": [
    "int_to_str = {\n",
    "    0: 'airplane',\n",
    "    1: 'automobile',\n",
    "    2: 'bird',\n",
    "    3: 'cat',\n",
    "    4: 'deer',\n",
    "    5: 'dog',\n",
    "    6: 'frog',\n",
    "    7: 'horse',\n",
    "    8: 'ship',\n",
    "    9: 'truck'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "ceabf906",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted label: airplane\n",
      "Class-membership probability 98.94%\n"
     ]
    }
   ],
   "source": [
    "print(f'Predicted label: {int_to_str[predicted_label.item()]}')\n",
    "print(f'Class-membership probability {probas[0][predicted_label]*100:.2f}%')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}