pytorch-lightning_ipynb/ordinal/beckham2016-light_cement.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b1739f6e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch            : 1.10.1\n",
      "pytorch_lightning: 1.6.0.dev0\n",
      "torchmetrics     : 0.6.2\n",
      "matplotlib       : 3.3.4\n",
      "coral_pytorch    : 1.2.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -p torch,pytorch_lightning,torchmetrics,matplotlib,coral_pytorch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "00079242",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext pycodestyle_magic\n",
    "%flake8_on --ignore W291,W293,E703"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d198b6a3",
   "metadata": {},
   "source": [
    "<a href=\"https://pytorch.org\"><img src=\"https://raw.githubusercontent.com/pytorch/pytorch/master/docs/source/_static/img/pytorch-logo-dark.svg\" width=\"90\"/></a> &nbsp; &nbsp;&nbsp;&nbsp;<a href=\"https://www.pytorchlightning.ai\"><img src=\"https://raw.githubusercontent.com/PyTorchLightning/pytorch-lightning/master/docs/source/_static/images/logo.svg\" width=\"150\"/></a>\n",
    "\n",
    "# Squared-error reformulation for ordinal regression and deep learning -- cement strength dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0ffe6ee",
   "metadata": {},
   "source": [
    "Implementation of a method for ordinal regression by Beckham and Pal 2016.\n",
    "\n",
    "**Paper reference:**\n",
    "\n",
    "- Beckham, Christopher, and Christopher Pal. \"[A simple squared-error reformulation for ordinal classification](https://arxiv.org/abs/1612.00775).\" arXiv preprint arXiv:1612.00775 (2016)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "654568ec",
   "metadata": {},
   "source": [
    "**Note:**\n",
    "    \n",
    "To keep the notation lean and minimal, this notebook only contains \"Squared-error reformulation\"-specific comments. For more comments on the PyTorch Lightning use, please see the cross-entropy baseline notebook [baseline-light_cement.ipynb](./baseline-light_cement.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13449160",
   "metadata": {},
   "source": [
    "## General settings and hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1acc2499",
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 128\n",
    "NUM_EPOCHS = 200\n",
    "LEARNING_RATE = 0.005\n",
    "NUM_WORKERS = 0\n",
    "\n",
    "DATA_BASEPATH = \".\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b7e4b44",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Converting a regular classifier into a *Reformulated Squared Error* ordinal regression model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4af15d92",
   "metadata": {},
   "source": [
    "Changing a classifier to a Reformulated Squared Error model for ordinal regression is actually really simple and only requires a few changes:\n",
    "\n",
    "**1)**\n",
    "\n",
    "We add an additional parameter layer `a`:\n",
    "\n",
    "```python\n",
    "self.a = torch.nn.Parameter(torch.zeros(\n",
    "    num_classes).float().normal_(0.0, 0.1).view(-1, 1))\n",
    "```\n",
    "\n",
    "\n",
    "**2)**\n",
    "\n",
    "We convert the logits (unnormalized outputs of the neural network) to \"predictions\" using the following function:\n",
    "\n",
    "\n",
    "```python\n",
    "def beckham_logits_to_predictions(logits, model, num_classes):\n",
    "    probas = torch.softmax(logits, dim=1)\n",
    "    predictions = ((num_classes-1)\n",
    "                   * torch.sigmoid(probas.mm(model.a).view(-1)))\n",
    "    return predictions\n",
    "```\n",
    "\n",
    "**3)** \n",
    "\n",
    "We swap the cross entropy loss from PyTorch,\n",
    "\n",
    "```python\n",
    "torch.nn.functional.cross_entropy(logits, true_labels)\n",
    "```\n",
    "\n",
    "with the squared error loss):\n",
    "\n",
    "```python\n",
    "def squared_error(targets, predictions):\n",
    "    return torch.mean((targets.float() - predictions)**2)\n",
    "```\n",
    "\n",
    "**4)**\n",
    "\n",
    "In a regular classifier, we usually obtain the predicted class labels as follows:\n",
    "\n",
    "```python\n",
    "predicted_labels = torch.argmax(logits, dim=1)\n",
    "```\n",
    "\n",
    "In this method, we replace this with the following code to convert the predicted probabilities into the predicted labels:\n",
    "\n",
    "```python\n",
    "def beckham_logits_to_labels(logits, model, num_classes):\n",
    "    predictions = beckham_logits_to_predictions(logits, model, num_classes)\n",
    "    return torch.round(predictions).float()\n",
    "\n",
    "predicted_labels = beckham_logits_to_labels(logits, model, model.num_classes)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "debfdfd1",
   "metadata": {},
   "source": [
    "## Implementing a `MultiLayerPerceptron` using PyTorch Lightning's `LightningModule`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1922e731",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "\n",
    "class MultiLayerPerceptron(torch.nn.Module):\n",
    "    def __init__(self, input_size, hidden_units, num_classes):\n",
    "        super().__init__()\n",
    "\n",
    "        self.num_classes = num_classes\n",
    "        \n",
    "        all_layers = []\n",
    "        for hidden_unit in hidden_units:\n",
    "            layer = torch.nn.Linear(input_size, hidden_unit)\n",
    "            all_layers.append(layer)\n",
    "            all_layers.append(torch.nn.ReLU())\n",
    "            input_size = hidden_unit\n",
    "\n",
    "        output_layer = torch.nn.Linear(hidden_units[-1], num_classes)\n",
    "        \n",
    "        all_layers.append(output_layer)\n",
    "        self.model = torch.nn.Sequential(*all_layers)\n",
    "        \n",
    "        # -----------------------------------------------------\n",
    "        # Beckham 2016-specific parameter layer\n",
    "        self.a = torch.nn.Parameter(torch.zeros(\n",
    "            num_classes).float().normal_(0.0, 0.1).view(-1, 1))\n",
    "        # -----------------------------------------------------        \n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = self.model(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce770cc2",
   "metadata": {},
   "source": [
    "Now, let's define the following Squared Error Reformulation-specific functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "05434da4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def squared_error(targets, predictions):\n",
    "    return torch.mean((targets.float() - predictions)**2)\n",
    "\n",
    "\n",
    "def beckham_logits_to_predictions(logits, model, num_classes):\n",
    "    probas = torch.softmax(logits, dim=1)\n",
    "    predictions = ((num_classes-1)\n",
    "                   * torch.sigmoid(probas.mm(model.a).view(-1)))\n",
    "    return predictions\n",
    "\n",
    "\n",
    "def beckham_logits_to_labels(logits, model, num_classes):\n",
    "    predictions = beckham_logits_to_predictions(logits, model, num_classes)\n",
    "    return torch.round(predictions).float()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6562578a",
   "metadata": {},
   "source": [
    "And then, we will use them below in the `LightningModule`. They are only used in the `_shared_step` method as indicated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1381f777",
   "metadata": {},
   "outputs": [],
   "source": [
    "from coral_pytorch.losses import corn_loss\n",
    "from coral_pytorch.dataset import corn_label_from_logits\n",
    "\n",
    "import pytorch_lightning as pl\n",
    "import torchmetrics\n",
    "\n",
    "\n",
    "class LightningMLP(pl.LightningModule):\n",
    "    def __init__(self, model, learning_rate):\n",
    "        super().__init__()\n",
    "\n",
    "        self.learning_rate = learning_rate\n",
    "        self.model = model\n",
    "\n",
    "        self.save_hyperparameters(ignore=['model'])\n",
    "\n",
    "        self.train_mae = torchmetrics.MeanAbsoluteError()\n",
    "        self.valid_mae = torchmetrics.MeanAbsoluteError()\n",
    "        self.test_mae = torchmetrics.MeanAbsoluteError()\n",
    "        \n",
    "    def forward(self, x):\n",
    "        return self.model(x)\n",
    "\n",
    "    def _shared_step(self, batch):\n",
    "        features, true_labels = batch\n",
    "        logits = self(features)\n",
    "\n",
    "        # Beckham 2016-specific functions-------------------------------### \n",
    "        predictions = beckham_logits_to_predictions(\n",
    "            logits, self.model, self.model.num_classes)\n",
    "        \n",
    "        loss = squared_error(predictions, true_labels)\n",
    "        \n",
    "        predicted_labels = beckham_logits_to_labels(\n",
    "            logits, self.model, self.model.num_classes)\n",
    "        # ---------------------------------------------------------------###   \n",
    "        \n",
    "        return loss, true_labels, predicted_labels\n",
    "\n",
    "    def training_step(self, batch, batch_idx):\n",
    "        loss, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.log(\"train_loss\", loss)\n",
    "        self.train_mae(predicted_labels, true_labels)\n",
    "        self.log(\"train_mae\", self.train_mae, on_epoch=True, on_step=False)\n",
    "        return loss\n",
    "\n",
    "    def validation_step(self, batch, batch_idx):\n",
    "        loss, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.log(\"valid_loss\", loss)\n",
    "        self.valid_mae(predicted_labels, true_labels)\n",
    "        self.log(\"valid_mae\", self.valid_mae,\n",
    "                 on_epoch=True, on_step=False, prog_bar=True)\n",
    "\n",
    "    def test_step(self, batch, batch_idx):\n",
    "        loss, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.test_mae(predicted_labels, true_labels)\n",
    "        self.log(\"test_mae\", self.test_mae, on_epoch=True, on_step=False)\n",
    "\n",
    "    def configure_optimizers(self):\n",
    "        optimizer = torch.optim.Adam(self.parameters(), lr=self.learning_rate)\n",
    "        return optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ae78a8e",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Note: There Are No Changes Compared To The Baseline Below\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39ac693d",
   "metadata": {},
   "source": [
    "## Setting up the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b070e95b",
   "metadata": {},
   "source": [
    "### Inspecting the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a26a6486",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>response</th>\n",
       "      <th>V1</th>\n",
       "      <th>V2</th>\n",
       "      <th>V3</th>\n",
       "      <th>V4</th>\n",
       "      <th>V5</th>\n",
       "      <th>V6</th>\n",
       "      <th>V7</th>\n",
       "      <th>V8</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4</td>\n",
       "      <td>540.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>162.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1040.0</td>\n",
       "      <td>676.0</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>540.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>162.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1055.0</td>\n",
       "      <td>676.0</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>332.5</td>\n",
       "      <td>142.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>228.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>932.0</td>\n",
       "      <td>594.0</td>\n",
       "      <td>270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>332.5</td>\n",
       "      <td>142.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>228.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>932.0</td>\n",
       "      <td>594.0</td>\n",
       "      <td>365</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>198.6</td>\n",
       "      <td>132.4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>192.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>978.4</td>\n",
       "      <td>825.5</td>\n",
       "      <td>360</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   response     V1     V2   V3     V4   V5      V6     V7   V8\n",
       "0         4  540.0    0.0  0.0  162.0  2.5  1040.0  676.0   28\n",
       "1         4  540.0    0.0  0.0  162.0  2.5  1055.0  676.0   28\n",
       "2         2  332.5  142.5  0.0  228.0  0.0   932.0  594.0  270\n",
       "3         2  332.5  142.5  0.0  228.0  0.0   932.0  594.0  365\n",
       "4         2  198.6  132.4  0.0  192.0  0.0   978.4  825.5  360"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "data_df = pd.read_csv(\"https://raw.githubusercontent.com/gagolews/\"\n",
    "                      \"ordinal_regression_data/master/cement_strength.csv\")\n",
    "data_df[\"response\"] = data_df[\"response\"]-1\n",
    "\n",
    "data_labels = data_df[\"response\"]\n",
    "data_features = data_df.loc[:, [\n",
    "    \"V1\", \"V2\", \"V3\", \"V4\", \"V5\", \"V6\", \"V7\", \"V8\"]]\n",
    "\n",
    "data_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b7044584",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of features: 8\n",
      "Number of examples: 998\n",
      "Labels: [0 1 2 3 4]\n",
      "Label distribution: [196 310 244 152  96]\n"
     ]
    }
   ],
   "source": [
    "print('Number of features:', data_features.shape[1])\n",
    "print('Number of examples:', data_features.shape[0])\n",
    "print('Labels:', np.unique(data_labels.values))\n",
    "print('Label distribution:', np.bincount(data_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a60df81f",
   "metadata": {},
   "source": [
    "### Performance baseline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "952a6c2d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline MAE: 1.03\n"
     ]
    }
   ],
   "source": [
    "avg_prediction = np.median(data_labels.values)  # median minimizes MAE\n",
    "baseline_mae = np.mean(np.abs(data_labels.values - avg_prediction))\n",
    "print(f'Baseline MAE: {baseline_mae:.2f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14f82826",
   "metadata": {},
   "source": [
    "### Creating a `Dataset` class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "aac85ec8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import Dataset\n",
    "\n",
    "\n",
    "class MyDataset(Dataset):\n",
    "\n",
    "    def __init__(self, feature_array, label_array, dtype=np.float32):\n",
    "        self.features = feature_array.astype(dtype)\n",
    "        self.labels = label_array\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        inputs = self.features[index]\n",
    "        label = self.labels[index]\n",
    "        return inputs, label\n",
    "\n",
    "    def __len__(self):\n",
    "        return self.features.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "998d6e7a",
   "metadata": {},
   "source": [
    "### Setting up a `DataModule`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "64efaa4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "\n",
    "class DataModule(pl.LightningDataModule):\n",
    "    def __init__(self, data_path='./'):\n",
    "        super().__init__()\n",
    "        self.data_path = data_path\n",
    "        \n",
    "    def prepare_data(self):\n",
    "        data_df = pd.read_csv(\n",
    "            'https://raw.githubusercontent.com/gagolews/'\n",
    "            'ordinal_regression_data/master/cement_strength.csv')\n",
    "        data_df.to_csv(\n",
    "            os.path.join(self.data_path, 'cement_strength.csv'), index=None)\n",
    "        return\n",
    "\n",
    "    def setup(self, stage=None):\n",
    "        data_df = pd.read_csv(\n",
    "            os.path.join(self.data_path, 'cement_strength.csv'))\n",
    "        data_df[\"response\"] = data_df[\"response\"]-1  # labels should start at 0\n",
    "        self.data_labels = data_df[\"response\"]\n",
    "        self.data_features = data_df.loc[:, [\n",
    "            \"V1\", \"V2\", \"V3\", \"V4\", \"V5\", \"V6\", \"V7\", \"V8\"]]\n",
    "        \n",
    "        # Split into\n",
    "        # 70% train, 10% validation, 20% testing\n",
    "        \n",
    "        X_temp, X_test, y_temp, y_test = train_test_split(\n",
    "            self.data_features.values,\n",
    "            self.data_labels.values,\n",
    "            test_size=0.2,\n",
    "            random_state=1,\n",
    "            stratify=self.data_labels.values)\n",
    "\n",
    "        X_train, X_valid, y_train, y_valid = train_test_split(\n",
    "            X_temp,\n",
    "            y_temp,\n",
    "            test_size=0.1,\n",
    "            random_state=1,\n",
    "            stratify=y_temp)\n",
    "        \n",
    "        # Standardize features\n",
    "        sc = StandardScaler()\n",
    "        X_train_std = sc.fit_transform(X_train)\n",
    "        X_valid_std = sc.transform(X_valid)\n",
    "        X_test_std = sc.transform(X_test)\n",
    "\n",
    "        self.train = MyDataset(X_train_std, y_train)\n",
    "        self.valid = MyDataset(X_valid_std, y_valid)\n",
    "        self.test = MyDataset(X_test_std, y_test)\n",
    "\n",
    "    def train_dataloader(self):\n",
    "        return DataLoader(self.train, batch_size=BATCH_SIZE,\n",
    "                          num_workers=NUM_WORKERS,\n",
    "                          drop_last=True)\n",
    "\n",
    "    def val_dataloader(self):\n",
    "        return DataLoader(self.valid, batch_size=BATCH_SIZE,\n",
    "                          num_workers=NUM_WORKERS)\n",
    "\n",
    "    def test_dataloader(self):\n",
    "        return DataLoader(self.test, batch_size=BATCH_SIZE,\n",
    "                          num_workers=NUM_WORKERS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "d909e6cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(1) \n",
    "data_module = DataModule(data_path=DATA_BASEPATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a448bc5",
   "metadata": {},
   "source": [
    "## Training the model using the PyTorch Lightning Trainer class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "8ee17411",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pytorch_lightning.callbacks import ModelCheckpoint\n",
    "from pytorch_lightning.loggers import CSVLogger\n",
    "\n",
    "\n",
    "pytorch_model = MultiLayerPerceptron(\n",
    "    input_size=data_features.shape[1],\n",
    "    hidden_units=(40, 20),\n",
    "    num_classes=np.bincount(data_labels).shape[0])\n",
    "\n",
    "lightning_model = LightningMLP(\n",
    "    model=pytorch_model,\n",
    "    learning_rate=LEARNING_RATE)\n",
    "\n",
    "\n",
    "callbacks = [ModelCheckpoint(\n",
    "    save_top_k=1, mode=\"min\", monitor=\"valid_mae\")]  # save top 1 model \n",
    "logger = CSVLogger(save_dir=\"logs/\", name=\"mlp-beckham-cement\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "9e21b802",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jovyan/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/callback_connector.py:90: LightningDeprecationWarning: Setting `Trainer(progress_bar_refresh_rate=50)` is deprecated in v1.5 and will be removed in v1.7. Please pass `pytorch_lightning.callbacks.progress.TQDMProgressBar` with `refresh_rate` directly to the Trainer's `callbacks` argument instead. Or, to disable the progress bar pass `enable_progress_bar = False` to the Trainer.\n",
      "  rank_zero_deprecation(\n",
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "\n",
      "  | Name      | Type                 | Params\n",
      "---------------------------------------------------\n",
      "0 | model     | MultiLayerPerceptron | 1.3 K \n",
      "1 | train_mae | MeanAbsoluteError    | 0     \n",
      "2 | valid_mae | MeanAbsoluteError    | 0     \n",
      "3 | test_mae  | MeanAbsoluteError    | 0     \n",
      "---------------------------------------------------\n",
      "1.3 K     Trainable params\n",
      "0         Non-trainable params\n",
      "1.3 K     Total params\n",
      "0.005     Total estimated model params size (MB)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation sanity check: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jovyan/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:258: PossibleUserWarning: The dataloader, val_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 4 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.\n",
      "  rank_zero_warn(\n",
      "/home/jovyan/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:258: PossibleUserWarning: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 4 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.\n",
      "  rank_zero_warn(\n",
      "/home/jovyan/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1850: PossibleUserWarning: The number of training samples (5) is smaller than the logging interval Trainer(log_every_n_steps=10). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.\n",
      "  rank_zero_warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1dab96b858854c048cacb51ec942ad8d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Training: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training took 0.30 min in total.\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "\n",
    "trainer = pl.Trainer(\n",
    "    max_epochs=NUM_EPOCHS,\n",
    "    callbacks=callbacks,\n",
    "    progress_bar_refresh_rate=50,  # recommended for notebooks\n",
    "    accelerator=\"auto\",  # Uses GPUs or TPUs if available\n",
    "    devices=\"auto\",  # Uses all available GPUs/TPUs if applicable\n",
    "    logger=logger,\n",
    "    deterministic=True,\n",
    "    log_every_n_steps=10)\n",
    "\n",
    "start_time = time.time()\n",
    "trainer.fit(model=lightning_model, datamodule=data_module)\n",
    "\n",
    "runtime = (time.time() - start_time)/60\n",
    "print(f\"Training took {runtime:.2f} min in total.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73a8e075",
   "metadata": {},
   "source": [
    "## Evaluating the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d23d4a7d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='Epoch', ylabel='MAE'>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA0VklEQVR4nO3dd3hUZfr/8fdNCDUBNMFIkyBVOoS2ItIsgCiugsJaWVkWy6prxe/qWtb9re66dpG1sNijIij2wgZRBBGQEpogNYJUgQQIkOT+/fFMJISZkBlycibM/bquuTJz2nxyCHPPeZ5zniOqijHGmNhVye8Axhhj/GWFwBhjYpwVAmOMiXFWCIwxJsZZITDGmBhX2e8A4UpOTtbU1NSI1t2zZw81a9Ys20BlJFqzWa7wRGsuiN5slis8keaaN2/eNlWtG3SmqlaoR1pamkYqIyMj4nW9Fq3ZLFd4ojWXavRms1zhiTQXMFdDfK561jQkIhNEZIuIZIaYX1tE3heRhSKyRERGepXFGGNMaF72EUwEBpQw/3pgqap2APoA/xaRKh7mMcYYE4RnhUBVZwA7SloESBQRARICy+Z5lccYY0xwoh4OMSEiqcAHqto2yLxEYCrQCkgELlXVD0NsZzQwGiAlJSUtPT09ojw5OTkkJCREtK7XojWb5QpPtOaC6M2Wk5NDYmIiNWvWJC4uzu84v1JV3PfU6HK0XPn5+ezZs4fin+19+/adp6pdQm7UqweQCmSGmDcUeAwQoBmwBqh1tG1aZ3H5slzhidZcqtGbLSMjQ1evXq1bt27VgoICv+P8avfu3X5HCKqkXAUFBbp161ZdvXr1EfPwo7O4FEYCkwMZVwUKQSsf8xhjfJKbm0tSUlJUfgOvSESEpKQkcnNzw1rPz0KwHugPICIpQEtgtY95jDE+siJQNiLZj55dUCYib+DOBkoWkSzgXiAeQFXHA38DJorIYlzz0J2qus2rPMYYY4LzrBCo6oijzN8InOPV+xtjjCkdG2vIGBPzdu7cybhx48Jeb9CgQezcuTPs9a6++momTZoU9npesUJgjIl5oQpBfn5+iet99NFH1KlTx6NU5afCDTpnjDm+3f/+EpZu3F2m22xdvxb3nt8m5PyxY8fy448/0rFjR+Lj40lISCA5OZklS5awdOlSLrzwQjZs2EBubi433XQTo0ePBiA1NZW5c+eSk5PDwIEDOeOMM/jmm29o0KAB7733HtWrVz9qtmnTpnHbbbeRl5dH165defbZZ6latSpjx45l6tSpVK5cmXPOOYdHHnmEt99+m3vvvZf4+Hhq167NjBkzymT/WCEwxsS8hx56iMzMTBYsWMD06dM577zzmD17Nu3atQNgwoQJnHjiiezbt4+uXbty8cUXk5SUdNg2Vq5cyRtvvMHzzz/PJZdcwjvvvMPll19e4vvm5uZy9dVXM23aNFq0aMGVV17Js88+y5VXXsmUKVNYvnw5IvJr89MDDzzAlClTaNmyZURNUqFYITDGRJWSvrmXl27dulF0uPsnn3ySKVOmALBhwwZWrlx5RCFo0qQJHTt2BCAtLY21a9ce9X1WrFhBkyZNaNGiBQBXXXUVzzzzDDfccAPVqlVj1KhRnHfeeQwePBiAnj17cu211zJixAguuuiiY/9FA6yPwBhjiik63v/06dP54osvmDVrFgsXLqRTp05BL9iqWrXqr8/j4uLIyzv60GkaYoifypUrM2fOHC6++GLeffddBgxw43eOHz+eu+++mw0bNtCxY0e2b98e7q8W/P3KZCvGGFOBJSYmkp2dHXTerl27OOGEE6hRowbLly9n9uzZZfa+rVq1Yu3ataxatYpmzZrxyiuv0Lt3b3Jycti7dy+DBg2iR48eNGvWDIAff/yRrl270q9fP95//302bNhwxJFJJKwQGGNiXlJSEj179qRt27ZUr16dlJSUX+cNGDCA8ePH0759e1q2bEmPHj3K7H2rVavGf//7X4YNG/ZrZ/GYMWPYsWMHQ4YMITc3F1XlscceA+D2229nxYoViAj9+/enQ4cOZZLDCoExxgCvv/76Ya8LjxCqVq3Kxx9/HHSdwn6A5ORkMjMP3YPrtttuK/G9Jk6c+Ovz/v378/333x82v169esyZM+eI9SZPnkx2djaJiYklbj9c1kdgjDExzo4IjDHGI9dffz0zZ848bNpNN93EyJHRdWdeKwTGGOORZ555xu8IpWJNQ8YYE+OsEBhjTIyzQmCMMTHOCoExxsQ4zwqBiEwQkS0iklnCMn1EZIGILBGRL73KYowxZSkhIQGAjRs3MnTo0KDL9OnTh7lz54bcRmpqKtu2RcdNGb08IpgIDAg1U0TqAOOAC1S1DTDMwyzGGFPm6tevH1U3mImUl7eqnCEiqSUs8jtgsqquDyy/xassxpgK5OOx8PPist3mye1g4EMhZ9955500btyY6667DoD77ruPAwcO8O233/LLL79w8OBBHnzwQYYMGXLYemvXrmXw4MFkZmayb98+Ro4cydKlSznttNPYt29fqeM9+uijTJgwAYBRo0Zx8803s2fPHi655BKysrLIz8/nnnvu4dJLL+Xee+/lk08+Oew+BcfKz+sIWgDxIjIdSASeUNWXgy0oIqOB0QApKSlMnz49ojfMycmJeF2vRWs2yxWeaM0F0ZstJyeH2rVrHxrS4eABKuUffeTOcBQcPMD+EIPKAZx//vmMHTuWK664AoD09HTefvttrr/+emrVqsX27dvp168fffv2RUQANwRFTk4OBQUFZGdn8/TTTxMfH8/MmTPJzMykV69e7NmzJ+RgdqpKTk4OS5cu5cUXX2TatGmoKv369aNLly6sXbuWunXrkp6eDrjB79atW8f777/PvHnzfr1PQbDt5+bmhvdvraqePYBUIDPEvKeB2UBNIBlYCbQ42jbT0tI0IgdzdfFb/0+1oCCy9T2WkZHhd4SgLFd4ojWXavRmy8jI0KVLl/odQ1u1aqU//fSTLliwQE8//XTdvn27Xn/99dquXTvt0KGDVqtWTTdt2qSqqjVr1lRV1TVr1mibNm1UVXXIkCE6bdq0X7fXqVMn/e6770K+X+PGjXXr1q36+OOP6z333PPr9LvvvlufeOIJXbFihaampuodd9yhM2bMUFXVgwcPatu2bfX3v/+9vvPOO7p///6g2w62P4G5GuJz1c+zhrKAT1R1j6puA2YAZTOUXjCL3qTtkodg3cyjL2uMiTlDhw5l0qRJvPnmmwwfPpy33nqLrVu3Mm/ePBYsWEBKSkrQ+xAUVXi0EA4NcU+CFi1aMG/ePNq1a8ddd93FAw88QOXKlcnIyDjiPgXHys9C8B7QS0Qqi0gNoDuwzLN3azeMA/G14JunPHsLY0zFNXz4cNLT05k0aRJDhw5l165dnHTSScTHx5ORkcG6detKXP/MM8/ktddeAyAzM5NFixaV6n3PPPNM3n33Xfbu3cuePXuYMmUKvXr1YuPGjdSoUYPLL7+c2267jfnz55OTk8Pu3bsZNGgQjz/+OAsWLDjWXxvwsI9ARN4A+gDJIpIF3AvEA6jqeFVdJiKfAIuAAuAFVQ15qukxi6/OTw3Oo8kPb8DWH6BuC8/eyhhT8bRp04bs7GwaNGhAvXr1uPTSSxkxYgRdunShY8eOtGrVqsT1r732WkaOHEn79u3p2LEj3bp1K9X7du7cmauvvvrX5UeNGkWnTp349NNPuf3226lUqRLx8fE8++yzZGdnM2zYMA4ePHjYfQqOlZdnDY0oxTL/Av7lVYbiNtYfSJOsKfDtszC4bHagMeb4sXjxobOVkpKSmDVrVtDlcnJyAHctQOF9CKpXr/5rx25pFL2n8S233MItt9xy2Pxzzz2Xc88994j1pk+fbvcjOBYHq9SGloNg+YdQUOB3HGOMiQqxNwx1i3NhyWT4eRHU7+h3GmPMca579+7s37//sGmvvPIK7dq18ynRkWKvEDTtDwis/NwKgTFRRFUjOusm2n377bfl+n6hzkIqSUw1DQGQUBcadIaVn/mdxBgTUK1aNbZv3x7Rh5g5RFXZvn071apVC2u92DsiAGh+Dkx/CPZsh5pJfqcxJuY1bNiQrKwstm7d6neUX+Xm5ob9gVoejparWrVqNGzYMKxtxmYhaNoPpv8D1n8Dp53vdxpjYl58fDxNmjTxO8Zhpk+fTqdOnfyOcQQvcsVe0xBAvQ4QVwU2zPE7iTHG+C42C0Hlqq4YZH3ndxJjjPFdbBYCgIbdYOP3kH/Q7yTGGOOrGC4EXSAvt+zHPTfGmAomdgtBo8A4INY8ZIyJcbFbCGo1gMR6VgiMMTEvdguBCDTsamcOGWNiXuwWAnCFYOc6yLHbJRtjYldsFwLrJzDGGO8KgYhMEJEtIlLizWZEpKuI5IvIUK+yhFSvA1SKt+YhY0xM8/KIYCJQ4g01RSQOeBj41MMcocVXh5PbQdZcX97eGGOigWeFQFVnADuOstifgHcA/xrpG3WDjfMhP8+3CMYY4yfxcthXEUkFPlDVtkHmNQBeB/oBLwaWmxRiO6OB0QApKSlp4dwOrqicnBwSEhIOm3bS5hm0XvZv5qY9Sk5i04i2WxaCZYsGlis80ZoLojeb5QpPpLn69u07T1W7BJ2pqp49gFQgM8S8t4EegecTgaGl2WZaWppGKiMj48iJOzeo3ltLdda4iLdbFoJmiwKWKzzRmks1erNZrvBEmguYqyE+V/08a6gLkC4ia4GhwDgRubDcU9RuCCekwtqvy/2tjTEmGvh2PwJV/XXwcRGZiGsaeteXMKlnwLIP3A3tK8X2GbXGmNjj5emjbwCzgJYikiUi14jIGBEZ49V7Riy1F+TuhC1L/E5ijDHlzrMjAlUdEcayV3uVo1Qa93Q/13zlTic1xpgYYu0gAHUaWT+BMSZmWSEolNoL1s10/QTGGBNDrBAUKuwn2FziiBjGGHPcsUJQKDXQT2DNQ8aYGGOFoFDthnBCEysExpiYY4WgqNQzYN3XUJDvdxJjjCk3VgiKSu0Fubusn8AYE1OsEBSVeob7ac1DxpgYYoWgqNoN4MRTrRAYY2KKFYLiUs8IXE9g/QTGmNhghaA46ycwxsQYKwTFFfYTrPnK3xzGGFNOrBAUV6s+nNjU+gmMMTHDCkEwqWfAum+sn8AYExOsEAST2gv274KfF/udxBhjPGeFIBgbd8gYE0O8vEPZBBHZIiJBT78RkctEZFHg8Y2IdPAqS9isn8AYE0O8PCKYCAwoYf4aoLeqtgf+BjznYZbwWT+BMSZGeFYIVHUGsKOE+d+o6i+Bl7OBhl5liUiTMwP9BIv8TmKMMZ4SVfVu4yKpwAeq2vYoy90GtFLVUSHmjwZGA6SkpKSlp6dHlCcnJ4eEhIRSLVtl/3ZOn/V7VjUdSVajCyN6v3CEk608Wa7wRGsuiN5slis8kebq27fvPFXtEnSmqnr2AFKBzKMs0xdYBiSVZptpaWkaqYyMjPBWeLKz6muXRPx+4Qg7WzmxXOGJ1lyq0ZvNcoUn0lzAXA3xuerrWUMi0h54ARiiqtv9zBKU9RMYY2KAb4VARE4BJgNXqOoPfuUoUWov2L/b+gmMMce1yl5tWETeAPoAySKSBdwLxAOo6njgr0ASME5EAPI0VPuVX4qOO1S/k79ZjDHGI54VAlUdcZT5o4CgncNRI/FkSGrurifoeaPfaYwxxhN2ZfHRFPYT5Of5ncQYYzxhheBompwJB7Jh0wK/kxhjjCesEBxNkzPdz9XTfY1hjDFesUJwNDWTIaUtrPnS7yTGGOMJKwSl0aQ3rP8WDu7zO4kxxpQ5KwSlcWpvyN8PG771O4kxxpQ5KwSlccpvQOJgzQy/kxhjTJmzQlAa1WpBgzRYbf0ExpjjjxWC0jq1N2ycD7m7/E5ijDFlygpBaTXpDVoAa2f6ncQYY8qUFYLSatgVKlezfgJjzHHHCkFpxVeDU3rY9QTGmOOOFYJwnNoXtiyFXVl+JzHGmDJjhSAcLQe6nys+9jeHMcaUISsE4UhuASc2hRUf+Z3EGGPKjGeFQEQmiMgWEckMMV9E5EkRWSUii0Sks1dZyoyIOypY85WdRmqMOW6UqhCISE0RqRR43kJELhCR+KOsNhEYUML8gUDzwGM08Gxpsviu1XlQcBBWfeF3EmOMKROlPSKYAVQTkQbANGAk7oM+JFWdAewoYZEhwMvqzAbqiEi9UubxT6PuULMuLJnidxJjjCkTpS0Eoqp7gYuAp1T1t0DrY3zvBsCGIq+zAtOiW6U4aDsUfvgU9v3idxpjjDlmpb1nsYjIb4DLgGvCXDfkNoNM0xBvPhrXfERKSgrTp0+P6A1zcnIiXreohLxmdMk/wIop/2RT/XOPeXtQdtnKmuUKT7TmgujNZrnC40kuVT3qA+gNTAXuDLw+FXiyFOulApkh5v0HGFHk9Qqg3tG2mZaWppHKyMiIeN3DFBSoPtVV9cVzy2Z7WobZypjlCk+05lKN3myWKzyR5gLmaojP1VI1Danql6p6gao+HOg03qaqNx5jDZoKXBk4e6gHsEtVNx3jNsuHCHQYDutnwdYf/E5jjDHHpLRnDb0uIrVEpCawFFghIrcfZZ03gFlASxHJEpFrRGSMiIwJLPIRsBpYBTwPXBfxb+GHTldAXBWY85zfSYwx5piUtp2/taruFpHLcB/gdwLzgH+FWkFVR5S0wcChyvWlDRp1Euq6TuMFr0P/e6Babb8TGWNMREp71lB84LqBC4H3VPUgITp2Y0r30XBwD8x/xe8kxhgTsdIWgv8Aa4GawAwRaQzs9ipUhVG/E6T2gplPwIG9fqcxxpiIlLaz+ElVbaCqgwId0OuAvh5nqxj63Q17tsB3z/udxBhjIlLazuLaIvKoiMwNPP6NOzowp/SAZmfD149Brh0kGWMqntI2DU0AsoFLAo/dwH+9ClXh9PuLu8p4dsUYLskYY4oqbSFoqqr3qurqwON+3EVlBlxfQavBMOtp2FvS8ErGGBN9SlsI9onIGYUvRKQnsM+bSBVU37/A/myY+bjfSYwxJiylvY5gDPCyiBSeLP8LcJU3kSqolNbQ/lKYPR66/gHqNPI7kTHGlEppzxpaqKodgPZAe1XtBPTzNFlF1O9uN/zEtAf8TmKMMaUW1h3KVHW3qhaeGnOLB3kqtjqNoMd1sPgtWP+t32mMMaZUjuVWlcGGkTa9boXajWDqnyBvv99pjDHmqI6lENgQE8FUTYDBj8G2Fe7aAmOMiXIlFgIRyRaR3UEe2UD9cspY8TQ/G9pdAjMegS3L/U5jjDElKrEQqGqiqtYK8khU1WO9Q9nxbcA/oGqiayIqyPc7jTHGhHQsTUOmJDWTYcBDkDUHZo/zO40xxoRkhcBL7S9xVxxPewA2L/U7jTHGBOVpIRCRASKyQkRWicjYIPNri8j7IrJQRJaIyEgv85Q7ETj/CXfTmkkj3ZXHxhgTZTwrBCISBzwDDARaAyNEpHWxxa4HlgYuVusD/FtEqniVyRc1k+Gi52HbDzBlDBQU+J3IGGMO4+URQTdgVWCQugNAOjCk2DIKJIqIAAnADiDPw0z+aNoXznkQln8AM0Le3dMYY3wh7tbBHmxYZCgwQFVHBV5fAXRX1RuKLJMITAVaAYnApar6YZBtjQZGA6SkpKSlp6dHlCknJ4eEhISI1j1mqrRa/gQnb84gs81dbKvbI3qylcByhSdac0H0ZrNc4Yk0V9++feepapegM1XVkwcwDHihyOsrgKeKLTMUeAx3lXIzYA1Qq6TtpqWlaaQyMjIiXrdMHNin+p8+qn87SXXlF4fN8j1bCJYrPNGaSzV6s1mu8ESaC5irIT5XvWwaygKKDsHZENhYbJmRwORAzlWBQtDKw0z+iq8Gl02CpObwxghY9r7fiYwxxtNC8B3QXESaBDqAh+OagYpaD/QHEJEUoCWw2sNM/quZBFdNhZPbwptXwFeP2gVnxhhfeVYIVDUPuAH4FFgGvKWqS0RkjIiMCSz2N+B0EVkMTAPuVNVtXmWKGjVOhKs+gNZDYNr98FwfauxZ73cqY0yM8nSYCFX9CPio2LTxRZ5vBM7xMkPUqlIDhk2EJVPg4zvpPP92aJoMrQb5ncwYE2PsymI/iUDbi+CPX7K3RiN48zL4/lW/UxljYowVgmhQqz4LOv4dTu0D710PX9wHB+2W0MaY8mGFIEoUxFWFEenQ6XJ3H4Pn+0Hu7qOvaIwxx8gKQTSpXBWGPAO/ewu2roCPbvc7kTEmBlghiEYtzoXed8Ci9MDppTY+kTHGO1YIolWv2wJDWN8PL50Pe3f4ncgYc5yyQhCt4irDpa+6pqKs72DiebC7+IXZxhhz7KwQRDMR13l82dvwyzp4uit89W87o8gYU6asEFQEp/aGMV+500unPeAKwryX4MBev5MZY44DVggqiqSmMPw1NzRFjSR4/0Z4vB1s/cHvZMaYCs4KQUXTpBeMng5XfwSaD1P+CPtzYNNC8OjeEsaY45sVgopIBFJ7wuDHYON8+GcT+M+ZkPmO38mMMRWQp4POGY+1+S38vBh2b3JHBF/cBy0HwZ4tcEKq3+mMMRWEFYKKrv9f3c81X8FLg12/wd5tMOgR6PYHf7MZYyoEaxo6XjTpBZ2ugDqnQMOu8NndbpgKY4w5CjsiOJ4Medr9zN4M43rAuN9AQgr0uxs6XeZvNmNM1PL0iEBEBojIChFZJSJjQyzTR0QWiMgSEfnSyzwxIzEFrnwPet4EdRrBe9fB21fDrHGQu8vvdMaYKOPZEYGIxAHPAGfjbmT/nYhMVdWlRZapA4wDBqjqehE5yas8Madee/fIz4Mv7oUFr7m7oa38DC5/ByrF+Z3QGBMlvDwi6AasUtXVqnoASAeGFFvmd8BkVV0PoKpbPMwTm+Iqw7l/hzvXwvlPwOoM+OBm+OFTVySMMTFP1KOLkERkKO6b/qjA6yuA7qp6Q5FlHgfigTZAIvCEqr4cZFujgdEAKSkpaenp6RFlysnJISEhIaJ1vVYu2VRpvnI8DTZ+AsBP9QeyssUY/3NFwHKFL1qzWa7wRJqrb9++81S1S9CZqurJAxgGvFDk9RXAU8WWeRqYDdQEkoGVQIuStpuWlqaRysjIiHhdr5Vrtt2bVN+7QfW+E1Q3Lytx0WjdZ5YrfNGazXKFJ9JcwFwN8bnqZdNQFtCoyOuGQPFxlLOAT1R1j6puA2YAHTzMZAAST4b+90GVmjB5FEz9E6yf7eYd2AsF+b7GM8aULy8LwXdAcxFpIiJVgOHA1GLLvAf0EpHKIlID6A4s8zCTKVQzyV2M9ss6yJwML1/oRjZ9pDl8eKvf6Ywx5cizQqCqecANwKe4D/e3VHWJiIwRkTGBZZYBnwCLgDm4pqRMrzKZYrr9Ae7aADcugBMau3sdxFWB+S/BtlV+pzPGlBNPLyhT1Y+Aj4pNG1/s9b+Af3mZwxxFQl03mun6We6q5Cc7woe3QM26NPslF9o3hhOb+J3SGOMRG2LCODWT4LTB7mK07mNgzZfw4zTqb/zE3Qhn7dd+JzTGeMQKgTlSn7vcDXBu/YHZPZ5zzUaTroEcu8zDmOORFQJzpMpV3CB2latwoGoSDHsJcnfC65dYMTDmOGSFwBzdyW1dMdi6Al7o7wa1M8YcN6wQmNJpOcA1F+3eBBkP+p3GGFOGrBCY0muYBl1Hwfevwha73MOY44UVAhOe3ndAlUSYeiPsz/E7jTGmDFghMOGpcSKc/zj8NBdeGwoH9vidyBhzjKwQmPC1vQgufsFdgDZ7nJu2e5ONUWRMBWW3qjSRaXsxLHobvnkaataF9292g9mdcQt0H+13OmNMGOyIwESuz53u+oL3b4IGnaFOY/jkTjeQnTGmwrBCYCJXv5M7MkhuCb97G4a+CAh897zfyYwxYbCmIXNsLgp86BfeA/m082H+y26Yiio1/ctljCk1OyIwx6ZS3KEiANDjWsjdBYvf9i+TMSYsVghM2WrUHZKaw+JJficxxpSSp4VARAaIyAoRWSUiY0tYrquI5AdueG8qMhFoN8wNW73rJzdtfzZsXupvLmNMSJ4VAhGJA54BBgKtgREi0jrEcg/j7mRmjgfthgIKSybD9h/huT7w7OllepRQ+WAOPNQYln9YZts0JlZ5eUTQDVilqqtV9QCQDgwJstyfgHcAG9/4eJHU1J1RlPH/4JnusHeHO7108mhY9QXkH4Q3RsB3L0T8FrV2r3Cnrv6YUXa5jYlRXp411ADYUOR1Fu7m9L8SkQbAb4F+QNdQGxKR0cBogJSUFKZPnx5RoJycnIjX9Vq0Zos014lJg6m/vzJ7azRiY/0BHIxPpNPOscS/NYqfTz6Lxus/In/lF8zZXptKBQfJq1yDg1XqlHr79ba7W1vvWvEV39cMP59XovXfEaI3m+UKjye5VNWTBzAMdzP6wtdXAE8VW+ZtoEfg+URg6NG2m5aWppHKyMiIeF2vRWu2Ms21cYHqfSeo3ltL9YVzVB+oq/pMD9X7T1T9e33Vr59QPbBXdetK1ekPq/68RDXvgOqW5aoFBaq52aof3qb6y3rd9kQ/t50H66nm56n+8Jnqnu1llzVC0frvqBq92SxXeCLNBczVEJ+rXh4RZAGNirxuCGwstkwXIF1EAJKBQSKSp6rvepjL+KVeB3d9wayn4LfPwvevwVePQJuL3OB1n98D3zwJubshf79rWqqSAAey4ZwHYc9WmPMcAInZKwPzcmDlZ/DGcNdJfXGx5qbszVCtNsRXK/vfJ+Mf0Kw/NOpW9ts2phx5WQi+A5qLSBPgJ2A48LuiC6hqk8LnIjIR+MCKwHGu9+3Q80aoXBX6/h+0+a27AxrAmhkwaxxUP8Ets3gS7NsBO9bAF/e7ZSrFw/xXqJK3D9JGwrz/whf3uXmLJ8GZt0NyC8jeBHMnwNePQduhcOGz8NL50LALnH1/5Pl/WQuJ9V3/xJcPuVFYL3/n8GX27oD46u7xw6fuwrrUMyJ/z9LI2w/T/+Eu5rvmc9dPY0wpeVYIVDVPRG7AnQ0UB0xQ1SUiMiYwf7xX722iXOWq7meluENFAKDJme5RqP897ufeHTD+DHf/g8GPwjvXuOkdf+dukrN1uTva2LYK3h7pLmjbneWWOaGJu7itXntY9zVsWuCKxaaFEBdfum/zO9dDrQZuu093gz5j4aTACXCrp7t8NU4EIP7AThjXA5qfDUOegQ9vhRpJ8Mcv4edM0HyXNRIFBfDKEGh2tiuUhfbthFcvgp/muddrZgQtBHW3fA3/mwn9/hLZ+5vjlqdDTKjqR8BHxaYFLQCqerWXWUwFVuNE9y33QI67WO1/f6NgZxaV6nWEk06DnxdBl2tg90aY8S9odpb7oGzU3Y2M+kQH+PT/oFod903+q0dg9njX/NTn/6D5WZDUDKomumsfsuZA3gFodR5sWQoTzoWB/3Tbyt8PKz+Hg/tctoI8WPERdLocVGm54mnI2QzrZrkCsWuDy3Vgjztrat8OuHmx+7DeuwPaDyv9flj1uVvvp++h8xXuyGl/NrzyW/h5MVzyMrz3J7c/iiso4NTVL8PyHdDrVm+aykyFZWMNmYqhdoNDz8/9B2vmfELTylXcaapbl0PrC9wHfc+boEqNw9dtfwkseA0GPAQzn3DNRVVrufb9jAfdo2otaDkIlr4LebluvdRe7sNaCyBzMpzUyk3P+s5NS2kL+3fD/FdgVxYs/4Dk7Ytd09S2H9xFdeCOAlZ8DFuWuNcLXocv7nXFofHph/9uRe3PcYXkpNPc69nPuv6O3F0w53l3t7iF6bBxPlz6Gpw2GL59DjYFKQTrZlI9d7N7vjnTNZEZE2CFwFQ8rQax4ecaNAXo+xfoFPh2DEcWAYB+d8OJp7rO5Nyd8MlYOOdv0Pkq2Pg97P4JFr0Ji9Jdn0XPm9w37Kk3AgqNerib8Oz40TXz7N0OG2ZD5ytd8fnmSdjwLTTqxspmo2jefYC7e9v8lw9lmPGvQL5E+PAWdyQhcTDzcah5EuxcB+c/4XKs/AwGPw7pl7n3vXG+OwJZnQH97nGFaPY4+M317jqKOqe4oxdwTWBz/+tuEqQFrvkLYMHr5FeqQlzBAdeEZIXAFGGFwFRsiSnuUZJa9eHM29zzrqMguTk07e+Gw2jQ2T1OOx/y8yAu8F+ifieoXB12rHYfsuN7uiaf/n+F6Q9B/gFokAatL3TrN+4JCSfx0/TpNK/fzm1j1RdQq6ErTluXQ41kOP0G17nd/lL3IR04CwpwRwjLproisXYm7AlcYzn7WddJHV/TdZCf8huYOMj1faz9yt0xzp15Bye3h7x97kjh4ztdwWt+Dix9l80pfaifvQh+mg97trujjfodD73/tpVQu5FrNtqV5Ypc1YQj92d+Hqz4EFqdD5WCXJP6yzp3hpcVmwrDCoGJLXHxrg8h6Lxi/x0K2+9V3RHFjtXQYqD7Fr72K1cIqtdxRxFF1Uxy39J3rncdwzVOcE1Fp/aBLr93H7K9bnMf2Ku/hC4jYftqWPCqu7nPGX92Rw3thrmjhu9ecMXhnAfdtmuc7vo0pj3gmqaa9jv03oUd0R/f4fpUPrzV9W1IHD81GEz97Hh3RDB5lOvHuONHd1bTulkw8TxIuxrO/X8wvpf7vQY/euR+Wvqu67Af/ga0GnTk/Kl/csXmjtVQucqR81UPFS4TFawQGHM0Iq75afEk117f5kL3rbfuaaHXqdcxUAjaQ2I9d3ZT076ujf+8fx9a7s/uCmkO7HUFo8MISGnj+i8S67sjiUXprj+i+5hDeTpfCZ//FaTS4WdaJbeAytVcEeg91nVkZ2+CkR+yZ8UvUKuT+za/faVbfvV01/Q16feuLyPzHXfEsW8H/Dgt+O/2w6eH1i1eCHZluQ5t1J2lVbRIgRt76tWL4PQboes1ofefKVc2DLUxpdHrFrjuG/ch3HUU/HnxkUcQRdXv5H6e3N41O3X9g/sZSpUa7ht/Shv3us4pbvsnt4ULx8OwiYfa+8EVjEqVA0clJxyaHlfZFY2add2ZU6O+gBsXHDpSaJDmftZqAFVru07sL/4Ke7e5/ofcnfDpXW6ZX9bCzsAoMTlbYeGb7myqVV+4aWu+PPL3WPQWoBBXxW27qN2b4OUL3XYXvBZ6X5hyZ0cExnjhtPNd527j013z0XmPRL6tjiOOnJZwEpz3KJzQ+Mh5FzzlmpIK7xBXeN0GQIMurgic/YA7Wlg61R09dP8j9LzZ9Ufs2eqasVZPd2c+1UiCd691xWL5B+5ooV4Hdy1G9uZDfTQFBa5v4pTfuP6FFZ+4024Lm4E+vt11tLe+EJa+54pLQl3XVLTqC9eclNTUzS96syPjOTsiMMYLyc3hqvddEfBK2lXuA7u4lNauSSqYarXglqVuqPAWA2H/LlcozvizO5poe5Fb7qz7ofqJ7synNy93zVun9nWd2VIJzrrPLbdmBmxdAV8/DuO6w7YV0PEyaDkAdq2HzYFTZrPmwrL33RlZPW8C9Nemp0YbpsCrF0PG310T1X/OhH2/HMq8dmboUWYLClwfSvbmMHeeKcoKgTGxqvlZ7kykHte5IwyA3nfC0AnubKLUnrD+G3fm0BVT4KLnXDNUox7QpLfr73j/Jnimm7suomqiu4d1p8tdkYmrAu+OcUXgk7vcWVO/uc71n9Ss6y7M25/DKesnuyIzdoO7GntzprtuA9xV0+kj3Om467458ndYNtV1iBeenltcQT5k/+zBzju+WCEwJlZVPwFuWuiuxShUMxnaXuyeFx5tDH7cNeEknASjprmCUCnOnQGV0to1//x5Cfzhf+7iPRHXXDT8Ddc5/EJ/d+3D2fe7YlGpkhsmY+Xn8OGtxOdluwzVarmjieQWrtMaYNYz7gK6hBR460q3vUIF+W5gQnB9Ewf2Hv775R90RzNPdHBXd5uQrBAYE8sS6ga/FgCg05Uwerq7artQUlOoExhU+Kz7XGd09z9C7YZHrt/8LBj5sTsd9c+Z7kihUPfRrg9jUTo7TugIjQK3IxFxgwSu+wbWz3YXzrW+EC6f7D74X+h/6IrtxZNcU1S30a6Ja+m7h7afdwCmjHH9IHm5ri+k0I8ZsOYrNzzHx2Nh3kuH5u1Y7W6mlDU3+D758p/wVJrbfjBblrlrNCoYKwTGmOAKh/A4FvU7uiugixeK+p1ccRj5MctOu+XweYW3Op0wwPVH9LvbDe/xh2muSSn9d+703f/9zZ2VNeBhd13F7HGur2BXljtFNXOSK1YpbQ8Via0/wGvD4KXB8Ghr+PZZmHa/O3oANwTJ1uXw/s1IQf6h6eCu7p49DravguXvH/m7Zm+G5/vBh38Ovi9+WeeOanK2hr8fPWaFwBjjj0px0Ph0Dlapffj0pKZunKfk5q65Kbm5m37iqTD8dfeB/EJ/d2X0gH+4I5reY9238Sfaw2Nt3JAfv/2P6wRvc6EbqmPXT+5CvSo1XIGo18ENwLd3u7uwL3szLHjDjSy7eTGdvr8DHkyBJVPc+2e+4zqxqyTAnCC3Wf36MTi4F5Z/5MaoKkoVPrjZnS218PVD0/MOHGrSWjPDvX9R2Ztdh7jHrBAYY6LP5ZPhum8PFYFCyc3dh/uerXDaBYfu89B+mFu+4+/cEcT130KH4W5e68CV3/85010RftZ9bhtXf+A6x6vWdkcPXz/qhg659FVoMYAaeze6I5D//d0NqzHneajbyg1Dvv4b9+3+nVGuKWjnBnf/i0Y9oODgoT6O7M2uCWv6P+DH/7lhSwrn5R1w98h4vJ27D8erQ+G96w5du7FxATzeFiaNdPeb+PDWQ81iZcyuIzDGRJ9gQ1MUOuMWiK/hLqorKrkZDH7syOWTm7kBBnM2uw7wzlcXeZ+q7pqPha+7Qfo6X+mOSC59jZlfZtA7ZQ+8fZUbinzTAjcwYOsh8NW/3Wmt+7Pd9Q/7d7v+jYuec4MFfjveDR64apq7YhvcsOitBrs78W1b5ZqlNsx213V8epc74vllLcx90RWoyaNd09jSd12fxe4styydj2XPBmWFwBhTscRXgzNuDm+dC54MPa/TZbDwDddM1PduNy2uMlop3h11nNTa3Ymu122uoIjArSvc6bHrZsIbI1zxuOApd4Ff5yvcWE8H97lrJloPccN+JDV1RzKf/9UdCWRvhNP/5G6UNOd5V9g+vsN1Xm9a5DrCL5/sjiiWTIaLX3T9J2V943o8LgQiMgB4AneHshdU9aFi8y8D7gy8zAGuVdWFXmYyxpjDND4d/u8nd2vR4ipVcsN7bF91aKhvOHS1duoZcOty1+RTePZV11Fumye1PvIK6Vr13ThSWd/BOX+HHte6ZQpHx+022l29vX6WO7pp1t+N1zTwYXd6rUc8KwQiEgc8A5yNu5H9dyIyVVWXFllsDdBbVX8RkYHAc0B3rzIZY0xQwYpAobot3SOUwqE8ClWKg5PbhV7+0ldd53Gwe2c0ORMuegEaprmmInBHIB4WAfD2iKAbsEpVVwOISDowBPi1EKhq0UsFZwNBTkY2xpjjSElFRyS825eWEVFVbzYsMhQYoKqjAq+vALqr6g0hlr8NaFW4fLF5o4HRACkpKWnp6ekRZcrJySEhIciNNqJAtGazXOGJ1lwQvdksV3gizdW3b995qhr8bkGq6skDGIbrFyh8fQXwVIhl+wLLgKSjbTctLU0jlZGREfG6XovWbJYrPNGaSzV6s1mu8ESaC5irIT5XvWwaygIaFXndEDhiwA8RaQ+8AAxU1Yp3bbYxxlRwXl5Q9h3QXESaiEgVYDgwtegCInIKMBm4QlV/8DCLMcaYEDw7IlDVPBG5AfgUd/roBFVdIiJjAvPHA38FkoBx4m5ekaeh2rCMMcZ4wtPrCFT1I+CjYtPGF3k+Cjiic9gYY0z5sbGGjDEmxlkhMMaYGGeFwBhjYpwVAmOMiXFWCIwxJsZZITDGmBhnhcAYY2KcFQJjjIlxVgiMMSbGWSEwxpgY59n9CLwiIluBdRGungxsK8M4ZSlas1mu8ERrLojebJYrPJHmaqyqdYPNqHCF4FiIyNxoHdQuWrNZrvBEay6I3myWKzxe5LKmIWOMiXFWCIwxJsbFWiF4zu8AJYjWbJYrPNGaC6I3m+UKT5nniqk+AmOMMUeKtSMCY4wxxVghMMaYGBczhUBEBojIChFZJSJjfczRSEQyRGSZiCwRkZsC0+8TkZ9EZEHgMciHbGtFZHHg/ecGpp0oIp+LyMrAzxN8yNWyyH5ZICK7ReRmP/aZiEwQkS0ikllkWsh9JCJ3Bf7mVojIueWc618islxEFonIFBGpE5ieKiL7iuy38SE37E2ukP9u5bW/Ssj2ZpFca0VkQWB6ueyzEj4fvP0bU9Xj/gHEAT8CpwJVgIVAa5+y1AM6B54nAj8ArYH7gNt83k9rgeRi0/4JjA08Hws8HAX/lj8Djf3YZ8CZQGcg82j7KPDvuhCoCjQJ/A3GlWOuc4DKgecPF8mVWnQ5H/ZX0H+38txfobIVm/9v4K/luc9K+Hzw9G8sVo4IugGrVHW1qh4A0oEhfgRR1U2qOj/wPBtYBjTwI0spDQFeCjx/CbjQvygA9Ad+VNVIry4/Jqo6A9hRbHKofTQESFfV/aq6BliF+1ssl1yq+pmq5gVezgYaevHe4eYqQbntr6NlExEBLgHe8Or9Q2QK9fng6d9YrBSCBsCGIq+ziIIPXxFJBToB3wYm3RA4jJ/gRxMMoMBnIjJPREYHpqWo6iZwf6TAST7kKmo4h//n9HufQeh9FE1/d78HPi7yuomIfC8iX4pILx/yBPt3i6b91QvYrKori0wr131W7PPB07+xWCkEEmSar+fNikgC8A5ws6ruBp4FmgIdgU24w9Ly1lNVOwMDgetF5EwfMoQkIlWAC4C3A5OiYZ+VJCr+7kTkL0Ae8Fpg0ibgFFXtBNwCvC4itcoxUqh/t6jYXwEjOPwLR7nusyCfDyEXDTIt7H0WK4UgC2hU5HVDYKNPWRCReNw/8muqOhlAVTerar6qFgDP4+EhcSiqujHwcwswJZBhs4jUC+SuB2wp71xFDATmq+pmiI59FhBqH/n+dyciVwGDgcs00KgcaEbYHng+D9eu3KK8MpXw7+b7/gIQkcrARcCbhdPKc58F+3zA47+xWCkE3wHNRaRJ4FvlcGCqH0ECbY8vAstU9dEi0+sVWey3QGbxdT3OVVNEEguf4zoaM3H76arAYlcB75VnrmIO+5bm9z4rItQ+mgoMF5GqItIEaA7MKa9QIjIAuBO4QFX3FpleV0TiAs9PDeRaXY65Qv27+bq/ijgLWK6qWYUTymufhfp8wOu/Ma97waPlAQzC9cD/CPzFxxxn4A7dFgELAo9BwCvA4sD0qUC9cs51Ku7sg4XAksJ9BCQB04CVgZ8n+rTfagDbgdpFppX7PsMVok3AQdy3sWtK2kfAXwJ/cyuAgeWcaxWu/bjw72x8YNmLA//GC4H5wPnlnCvkv1t57a9Q2QLTJwJjii1bLvushM8HT//GbIgJY4yJcbHSNGSMMSYEKwTGGBPjrBAYY0yMs0JgjDExzgqBMcbEOCsExhQjIvly+GinZTZabWAUS7+udzAmqMp+BzAmCu1T1Y5+hzCmvNgRgTGlFBif/mERmRN4NAtMbywi0wKDqE0TkVMC01PE3QdgYeBxemBTcSLyfGC8+c9EpLpvv5QxWCEwJpjqxZqGLi0yb7eqdgOeBh4PTHsaeFlV2+MGdnsyMP1J4EtV7YAb935JYHpz4BlVbQPsxF21aoxv7MpiY4oRkRxVTQgyfS3QT1VXBwYG+1lVk0RkG26YhIOB6ZtUNVlEtgINVXV/kW2kAp+ravPA6zuBeFV9sBx+NWOCsiMCY8KjIZ6HWiaY/UWe52N9dcZnVgiMCc+lRX7OCjz/BjeiLcBlwNeB59OAawFEJK6cx/w3ptTsm4gxR6ougZuWB3yiqoWnkFYVkW9xX6JGBKbdCEwQkduBrcDIwPSbgOdE5BrcN/9rcaNdGhNVrI/AmFIK9BF0UdVtfmcxpixZ05AxxsQ4OyIwxpgYZ0cExhgT46wQGGNMjLNCYIwxMc4KgTHGxDgrBMYYE+P+P/NelQL0sWivAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABVIklEQVR4nO2dd3hcxfW/39lV712Wmyz3jjvuCNN77yGEEkILLSSQb0ghPxICpNAhQIDQYnrvNjbuNja4y73K6rLV++78/ph7tbvSypZsryRL530ePdq99ezVaj5zzpk5o7TWCIIgCN0XR0cbIAiCIHQsIgSCIAjdHBECQRCEbo4IgSAIQjdHhEAQBKGbE9TRBrSVpKQk3a9fv8M6t7KyksjIyKNr0FGgs9oFndc2sattiF1toyvatWrVqiKtdbLfnVrrY+pn/Pjx+nCZN2/eYZ8bSDqrXVp3XtvErrYhdrWNrmgXsFK30K5KaEgQBKGbI0IgCILQzREhEARB6OYcc8liQRC6BvX19URFRZGVldXRpjQjNjb2mLUrLCyM3r17Exwc3OrrihAIgtAhZGdnk5qaSu/evVFKdbQ5PpSXlxMdHd3RZjTjUHZprSkuLiY7O5uMjIxWX1dCQ4IgdAg1NTXExsZ2OhE4llFKkZiYSE1NTZvOEyEQBKHDEBE4+hzOM+0+oaH8jfTb+Qa4Fx/e+c4QmHg9RCQcXbsEQRA6mO4jBEWbSd/9Duw+nJOtNRuie8C4q4+mVYIgCB1O9xGCERfwXWE8mZmZbT+3PB/+MRhcdUfdLEEQOoaSkhLefPNNbrnlljadd+aZZ/Lmm28SFxcXGMM6AMkRtAaHpZfuho61QxCEo0ZJSQnPPPNMs+0ul+ug533++eddSgSgO3kER4LDaX6LEAhCQHjgkw1szCk7qtcc3jOGP54zosX99913H9u3b2fMmDEEBwcTFRVFWloaq1evZvny5Zx//vns3buXmpoa7rjjDm688UYA+vXrx8qVK6moqOCMM85g+vTpLFmyhF69evHRRx8RHh7u936ZmZmMHTuWVatWUVhYyKuvvspDDz3EunXruOyyy3jwwQcBWrzv119/zf33309DQwMDBgzg5ZdfJioq6qg8K/EIWoPTmpghQiAIXYa//e1vDBgwgNWrV/Poo4+yYsUK/vKXv7Bx40YAXnrpJVatWsXKlSt54oknKC4ubnaNrVu3cuutt7Jhwwbi4uJ47733DnrPkJAQFixYwE033cR5553H008/zfr163nllVcar+/vvkVFRTz44IN8/PHH/PDDD0yYMIF//vOfR+1ZiEfQGuzQkKu+Y+0QhC7KwXru7cWkSZN8JmE98cQTfPDBBwDs3buXrVu3kpiY6HNORkYGY8aMAWD8+PHs2rXroPc499xzARg1ahQjRowgLS0NgP79+7N3714SExP93reoqIiNGzdy6qmn4nA4qKurY8qUKUfjYwPdUAhufHUl2wor2nSOQ7uYA7y8aBubC9fyt4tGB8Y4QRA6DO86/wsXLmTOnDksXbqUiIgIMjMz/U7SCg0NbXztdDqprq4+6D3s4x0Oh8+5DoeDhoYG5s+f7/e+WmtOOeUUnn/++YDMeO52QtA/OYqQoLZHxNxbFE7t4ssNeSIEgtAFiI6Opry83O++srIy4uPjiYiIYNOmTSxbtqxdbCotLfV738mTJ3Prrbc25jSqqqrIzs5m8ODBR+W+3U4I7jtj6OGd+OcgBiaFUZvtProGCYLQISQmJjJt2jRGjhxJeHg4qampjftOPvlk/vvf/zJ69GiGDBnC5MmT28Wm008/neeee67ZfZOTk3nllVe47rrraGgwucoHH3xQhKDdcQYTrNzUNrjQWsvUeEHoArz55pt+t4eGhvLFF1/43WfnAZKSkli/fn3j9nvuueeg95o/f37j68zMTJ85Td77WrrvrFmz+O677wISGpJRQ63FEUSIcuHW0ODWHW2NIAjCUUM8gtbicBKkTFiotsFNsFM0VBCE5tx6660sXuxb0+yOO+7g2muv7SCLDk3AhEAp9RJwNlCgtR7pZ78CHgfOBKqAn2mtfwiUPUeMI5hgzIzD2noXUaGioYIgNOfpp5/uaBPaTCC7ta8Apx9k/xnAIOvnRuDZANpy5DiCCLKEoKZBEsaCIHQdAiYEWusFwP6DHHIe8Ko2LAPilFJpgbLniHEGeUJD9QevRSIIgnAs0ZHxjV7AXq/32da23KYHKqVuxHgNpKam+mTY20JFRcVhnzuptp5Kt5kCvnjZCvZEHz0NPRK7Ak1ntU3sahud0a7Y2FhcLleLY/k7kmPdrpqamjb9vTtSCPyNv/Q7HEdr/TzwPMCECRP0YZWSxgzROtxzWR9NYng0FMCoMeMY0yfu8K5ztO0KMJ3VNrGrbXRGu7KysnA6ncfk2sAdRWvtCgsLY+zYsa2+bkcOfckG+ni97w3kdJAth8YRjNMrWSwIQvfCrvSZk5PDxRdf7PeYzMxMVq5c2Z5mHRU6Ugg+Bn6qDJOBUq11s7BQp8Hh9AiBJIsFodvSs2dP3n333Y4246gSyOGj/wMygSSlVDbwRyAYQGv9HPA5ZujoNszw0c47yBbAGYxTW6OGxCMQhKPLF/dB3rqje80eo+CMv7W4+9577yU9Pb1xhbI//elPKKVYsGABxcXFuFwuHnzwQc477zyf83bt2sXZZ5/N+vXrqa6u5tprr2Xjxo0MGzbskEXnoqKiuPXWW5kzZw7x8fH89a9/5Te/+Q179uzhscce49xzz2XXrl1cffXVVFZWAvDUU08xdepUAB5//HE++ugjamtrueCCC3jggQeO5Ak1EjAh0FpfcYj9Grg1UPc/6jiCGoVAPAJBOPa5/PLLufPOOxuF4O233+bLL7/krrvuQilFbW0tkydP5txzz22xpMyzzz5LREQEa9euZe3atYwbN+6g96ysrCQzM5OHH36YCy64gPvvv59vvvmGjRs3cs0113DuueeSkpLCN998Q1hYGFu3buWKK65g5cqVfP3112zfvp0VK1agtebcc89lwYIFzJw584ifhcyKai2OIBxWsScRAkE4yhyk5x4oxo4dS0FBATk5ORQWFhIfH09aWhp33XUX8+fPJygoiH379pGfn0+PHj38XmPBggXcfvvtAIwePZrRow9emTgkJITTTzfTq0aNGkVoaCjBwcGMGjWqsYZRfX09t912G6tXr8bpdLJlyxbArFD27bffNiaBKyoq2Lp1qwhBu+IIwqGN21fbIKEhQegKXHzxxbz77rvk5eVx+eWX88Ybb1BYWMiCBQtISEigX79+ftch8KYtBSiDg4Mbj/dek8BejwDgX//6F6mpqaxZswa3201YWBgAWmvuvvtu7rjjjsP5qAdFCua0FkcQDm15BPXiEQhCV+Dyyy9n9uzZvPvuu1x88cWUlpaSkpJCcHAw8+bNY/fu3Qc9f+bMmbzxxhsArF+/nrVr1x6xTaWlpaSlpeFwOHjttddwuUzH87TTTuO1116josIsrLVv3z4KCgqO+H4gHkHrcQbjkByBIHQpRowYQXl5Ob169SItLY2rrrqKc845hxNOOIFx48YxdOjB1y+5+eabufbaaxk9ejRjxoxh0qRJR2zTLbfcwkUXXcQ777zDiSee2Lhy2qmnnsqPP/7YuERlVFQUr7/+OikpKUd8TxGC1uIIQlkegYwaEoSuw7p1ntFKSUlJLF261O/ELbsn3q9fv8Z1CMLDw5k9e3ar72VfA8woJX/7Bg0a5ONZPPTQQ42vb7nlFu69995W36+1SGiotTicKFcDIUEO8QgEQehSiEfQWhzB4G4gNMghyWJBEA7K8ccfT21trc+21157jVGjRnWQRQdHhKC1OIIsIXCKRyAIRwkznajrsXz58g679+E8UwkNtRZnkMcjkFFDgnDEhIWFUVpa2mXFoCPQWlNcXNw45LS1iEfQWmyPIFhCQ4JwNOjduzdr1qzxSaB2FmpqatrcmLYHrbErLCyM3r17t+m6IgStxREErnrCwpzUiEcgCEdMcHAwFRUVTJgwoaNNacb8+fPbVMa5vQiUXRIaai2OYHC7xCMQBKHLIULQWhxOr1FD4hEIgtB1ECFoLY4gcNfLqCFBELocIgStxek1j0BmFguC0IUQIWgtjiDQbsKCFHXiEQiC0IUQIWgtDjPAKtyppdaQIAhdChGC1mIJQUSwlhyBIAhdChGC1mJ7BA4RAkEQuhYiBK3FGQxAmFPLPAJBELoUIgStxeEETI6g3qVxuaU+iiAIXQMRgtbisD0CExaSkUOCIHQVRAhai5UjsIVARg4JgtBVECFoLbYQOExISBLGgiB0FUQIWovT1yOQhLEgCF0FEYLWYnkEoeIRCILQxRAhaC1WsjjUYXkEsiaBIAhdBBGC1mJ5BCEOCQ0JgtC1ECFoLdY8Ajs0JKuUCYLQVRAhaC3WzOIQZTwB8QgEQegqiBC0lmahIfEIBEHoGogQtBY7WawkRyAIQtdChKC1WDmCYIcVGpIcgSAIXQQRgtZihYaClZ0sFo9AEISugQhBa2mSLK4Wj0AQhC6CCEFrafQIjABUi0cgCEIXQYSgtVhC4NAuwoOdVNc1dLBBgiAIRwcRgtZiCQGuesJDnOIRCILQZQioECilTldKbVZKbVNK3ednf6xS6hOl1Bql1Aal1LWBtOeIsIXA3UB4sJOqOhECQRC6BgETAqWUE3gaOAMYDlyhlBre5LBbgY1a6+OATOAfSqmQQNl0RFjJYtwNxiMQIRAEoYsQSI9gErBNa71Da10HzAbOa3KMBqKVUgqIAvYDnTP4bs0jwN1AhISGBEHoQiitA7MIu1LqYuB0rfUN1vurgeO11rd5HRMNfAwMBaKBy7TWn/m51o3AjQCpqanjZ8+efVg2VVRUEBUVdVjnOhuqmLHoCrYNuJZbc04D4LfHhx/WtY6mXYGms9omdrUNsattdEW7TjzxxFVa6wn+9gUdkVUHR/nZ1lR1TgNWA7OAAcA3SqmFWusyn5O0fh54HmDChAk6MzPzsAyaP38+h3su9TWwCAb260taQyL7K+vIzJx+eNc6mnYFmM5qm9jVNsSuttHd7ApkaCgb6OP1vjeQ0+SYa4H3tWEbsBPjHXQ+GpPFLiJCJFksCELXIZBC8D0wSCmVYSWAL8eEgbzZA5wEoJRKBYYAOwJo0+HTmCOoJyxYksWCIHQdAhYa0lo3KKVuA74CnMBLWusNSqmbrP3PAf8PeEUptQ4TSrpXa10UKJuOCKWMVyDJYkEQuhiBzBGgtf4c+LzJtue8XucApwbShqOKJQTh4hEIgtCFkJnFbcERDK4GwkOCqK534XYHZsSVIAhCeyJC0BYczkaPAGSVMkEQugYiBG3BGdyYIwCoksJzgiB0AUQI2oIjCNz1jR6BJIwFQegKiBC0BUcQuF2EWx6BJIwFQegKiBC0BUeQKUMtHoEgCF0IEYK24DWPAJDZxYIgdAlECNqClSwOCxGPQBCEroMIQVuwho9GSI5AEIQuhAhBW7BDQ8FmQrYIgSAIXQERgrbgsEND5rFVSWhIEIQugAhBW7BGDUWE2B6BTCgTBOHYR4SgLTiteQT28NE6KTEhCMKxjwhBW7BmFjsdipAgB1X14hEIgnDsI0LQFqxkMUB4sJMaSRYLgtAFECFoC1ayGJDlKgVB6DKIELQFhxNcHo9AJpQJgtAVCOgKZV0OZzC4amHOn/i/urWUFGYA4zraKkEQhCNChKAtOIJg/w5Y9C8mqhhiD8yF3RdB+tSOtkwQBOGwkdBQW3BYupk6il/1fI39jnj49i+gZclKQRCOXUQI2oItBLN+hyM0itmhl8DuRfC/K+Dta+Cdn8HeFR1qoiAIQluR0FBbGDALtAsGn07Ej6t5j1O4ZcB2Ey4CKMuBoq3wi4XgEI0VBOHYQISgLYw43/wA4SFBlDU44Or3PfvXvgPv3wAbP4SRF3aEhYIgCG3moEKglIrRWpe1sK+v1npPYMzq/ESHBVFaVc++kmp6xYWbjSMvhIV/hy/vgzWzW32tUcXFsO8Z8yYkAs76J0QkBMBqQRCE5hwqfjHffqGUmttk34dH25hjicsm9iE0yMEN/11JlV18zuGEMx6G2N5Qkd/qn5C6A+Z1yW7Y8AHsWdqxH04QhG7FoUJDyut10y6qohszIDmKJ64cy/WvfM+v3l7D01eOw+FQ0D/T/LSBVfPnk5mZCeV58I8hUJ4bCJMFQRD8ciiPQLfw2t/7bseJQ1L47RnD+GJ9Ho/N3XrkF4xMBuUwgiAIgtBOHMojSFFK3Y3p/duvsd4nB9SyY4QbZmSwKa+cJ+ZuZe/+Kh44bwQxYcGHdzGHE6JSoUw8AkEQ2o9DCcELQLSf1wAvBsSiYwylFI9cPJq+CRE8PncLc7PyyRySwtaCCqYNSOSe04YQZq1f0Cqi0yQ0JAhCu3JQIdBaP9DSPqXUxKNvzrGJ06G44+RBnDQshSe/3cqS7UWkJ0by4qKdLNlezJNXjmVAchQut8bpOERqJToNDuxqF7sFQRCgjfMIlFLDgcuBK4BSYEIgjDpWGdkrln9f7Xkkc7PyueedNZz9xCJ6xIaxZ38VI3rGeIabAicPSyXR+yLRPWDPkvYzWhCEbs8hhUAplY5p+K8AGoB0YILWeldgTTv2OWlYKl/cMZP/99lGautdnDwshbXZpWwvrACgstbFF+vzuGhQMMmDS0mJDiM5Og2qD0B9DQSHdfAnEAShO3CoCWVLgFhgNnCx1nqrUmqniEDr6REbxtNX+i9VXe9yc9dbq3lvbS7vbV0EwC1xpfwGWLd5C6NGjm5HSwVB6K4cyiMoBHoDqZhRQluRYaNHjWCng8cvH8vQkAMMHDqC3cWVVG7cDTXw8pdL+KcIgSAI7cChksXnKaVigYuAB5RSA4E4pdQkrbWU2TwKOB2KkUlOMkf2MBuGngDPQkPJPuoa3IQESfE6QRACyyFbGa11qdb6Ja31KcBk4I/AY0qpvQG3rjsSnQZAkj7QmEsQBEEIJG3qbmqt87XWT2itpwLTA2RT9yY8HrczlBR1gE15fuv9CYIgHFUOlSz++BDnn3sUbREAlEJF96Bn/QE25JbD2I42SBCErs6hksVTgL3A/4DldPNCc+2Fik6jX0Up7+aVd7QpgiB0Aw4VGuoB/B8wEngcOAUo0lp/p7X+7lAXV0qdrpTarJTappS6r4VjMpVSq5VSG5RSh7xmtyChP+nksilXQkOCIASegwqB1tqltf5Sa30NJlG8DZivlPrloS6slHICTwNnAMOBK6yZyd7HxAHPAOdqrUcAlxzWp+hqpAwjtqGYuvIi9lfWdbQ1giB0cQ6ZLFZKhSqlLgReB24FngDeP/hZAEwCtmmtd2it6zCT0s5rcsyVwPv2Smda64K2GN9lSTF6OVhlS8JYEISAo7RueX6YUuq/mLDQF8BsrfX6Vl9YqYuB07XWN1jvrwaO11rf5nXMY0AwMAJT2fRxrfWrfq51I3AjQGpq6vjZs1u/DKQ3FRUVREVFHda5gaSpXaE1RUxZdj33119LyLCzmNn7MMtaB8C2zoLY1TbErrbRFe068cQTV2mt/deH01q3+AO4gXLrp8zrpxwoO8S5lwAver2/GniyyTFPAcuASCAJM3N58MGuO378eH24zJs377DPDSTN7HK7tfuvvfWr91+o//XN5g6xyeaYeWadBLGrbYhdbeNI7AJW6hba1UPNLD6Saa3ZQB+v972BHD/HFGmtK4FKpdQC4DhgyxHc99hHKVTKcEZk5zC7pLqjrREEoYsTyPoF3wODlFIZSqkQTPnqpvMSPgJmKKWClFIRwPFAVgBtOnZIGcZA9pIrQiAIQoBp03oEbUFr3aCUug34CnACL2mtNyilbrL2P6e1zlJKfQmsxYShXtRtyEN0aVKGE6PLqTnQxIlyuzyvlQOUTO0QBOHICJgQAGitPwc+b7LtuSbvHwUeDaQdxyQpwwDoUb4erS9AKQVf3AfLn/Uc03si3DCngwwUBKGrIKUtOyt9JlEe1oPr+ZDSqjoo3g4rnoeBJ8OJv4OBp0D291Ajw0sFQTgyRAg6K0Gh7Bx+K2Mc2yn/8QOY/zdwhsB5z8AJv4GJ15vjCjd3rJ2CIBzziBB0YlyjL2eXO5U+c34B696GST+H6FSz0wodUbARljwFT00EtxtWvgSPjQZXfccZLgjCMUVAcwTCkdEzIYaf1t/FX0YXMiEjBcZe5dkZ2xeCI6Agy4SIirZAwQbY8CGU7IZ9P0Df4zvMdkEQjh3EI+jEJEWFsl2lMzf+Uph8E4RGe3Y6HJA8FPYuh5wfzbat35j3ADsXtL/BgiAck4gQdGKcDkWP2LCW5xKkDIecH0C7wBEEy56FhhrzepcIgSAIrUOEoJPTMzac7AMtCYGVJ3CGwsiLobLAzC047nLYsxzqa9rPUEEQjllECDo5E/rFs3L3AT5Z07Q6Bx4h6DMJBp9qXqeNgaHngKsWNn8GVfvNdq2hto0L3dRVHbbdgiAcO4gQdHLuPHkwE/vFc887a3h92W5qG7xmFqeOABT0PwH6zTTeQP9MSJ8KjmB49zr410goyzFzEP4+GPbvaN2N89bD3/oQXbY1EB9LEIROhAhBJyckyMGzPxnPiJ4x3P/hek54ZD4vL95pBCG6B3vOe5/aiTdBVDJc9zXMuBvCYuCaj+HsfxnPYM4D8N3DUF8F3z3SuhvvXQ7uBhKLVwT2AwqC0OGIEBwDJEWF8t7NU3n9+uPpmxjBA59s5MJnlvCbd9cw861qfvbaBiprG8iPHYUr2KpVnj4VJlwH434Ka2dDVTEMmAVr34LCgxR3tUNJhZsAiD+wLsCfTjhq1JR2tAXQUAv1BymUWFNqwpQWyu3yhCzdruYz5WsrZE5MOyBCcIyglGL6oCTe/sUUnr96PDkl1by9MpuzRqWxfGcx4x/8huP/OpdHvtrke+KMe0wyefDpcOELEBQO8x/yf5OdC+HRAZC3zsxPAKLLt5h/RqFzc2AXPJwBezvYg/v4l/D2Nf73Ve2Hvw+BTZ82buqd/aFnMuSK5+GJMb6FFZ8/ARb+M6AmCzKh7Jjk1BE9GNs3nvyyGkb2imVuVj5frs8jr6yG/yzcySXj+zAwxfIMYnvBjfMgOg0iEsx8hIX/gBm/gh4jfS+85UvQbtjyFeRvgLh0HCW7Ye8yU+NI6LwUbzfDiIu3m8EDHUXhZqhtof5V0RZoqDbHDDsHgKiK3VCeC6V7YM8y47nWlkF4vBGE4u1moqQQUMQjOEZJjg5lZK9YAE4alsqjlxzHvy4bQ3iwkwc+2YDWmk/W5HDWEwupSRhqRABg6i8hNBbm/An2rfLt7duT0Na/D9X7Yfw1uFWQZ3ttuTknb72Pey90AiqLzO8WwkPKXQ/VB9rHDju82JQDuz3HWITUWccWZDV6oY2foaYU0FCeFxhbhUZECLoQSVGh3HnKYBZuLeKzdbn89fMsNuSU8dUGr3+k8HiYehts+wZemAXv/Mxsr9pvQkJB4Z4eWK/xlMUMgaxPTe9s9pXmnOem+bj3QiegstD8rinxu7vfrrfgxVMCa4PWxo6aUt/wjk2JLQSFjZtCay0hyFkNxdvMa1sIbEEpyw2MvUIjIgRdjJ9OSWdQShR3vbWa3NIaIkOcvLsq2/eg6XfBT96H4282grB7KexeDGg4/kbPcSnDye59NuzfDh/fbjyDKbeZmct2WQuhc9AoBP49gvDqfSaPEEhPrrbcjFJDQ3VJ8/0HmgtBo0eQ9YkJbYHn3GprX3mueKABRoSgixHsdPCnc0dQ79Icn5HA9dMzWLStiKzcMrYXWmEgZzAMPAlO+gNEpsDcB2DDB6aI3dTbzXyEiESITKYoaTL0GA2rX4eY3uacxIEeN96b/TvA1dC+HzjQVO3vHKNxwL8tB3aZ34cIDYXUlYK73gwh9kd5Xssz0bWG/Ts9r+0GvSleDbxPGGr/TnNeSZPQUG05QS5rhJF3HsD+DPY13PX+w01aez6/cESIEHRBpg1M4tmrxvGvy8Zw0fjeaA1nPL6Qk//5HUu3F3sODImAmb+GPUth/XvQbwZEJkGf46HnWLMMpnLArN+b40/4DQSFmhnNBRt9b1qQBU9OaHlE0rHKm5fBp3d3tBWGt34CH9zkeZ+7Bh4/zowUshthfz1xILjeO+7eBLcbnpkCy57xf9+lT8GT48zExB3zzD2Ltzc/rsrru2X35vfvNOeuf6+5R2DH/qN7+l7HDm95N/7lfmbW75gPj4/xb4vQJmTUUBfljFFpja//cPZwquoaeHdVNve8s4Yv75xBdFiw2TnxBugxClx15jfA5W/6XmzwqXDrCkgabN6nDDflrusqISTSbJv3V+PaL3sWjrcmuB3ruBogd7UnZNHRHNhtGtiGOggKgSJr1nf+hkOGhkLq7F52CcQ0aXir95uf0r3NT6wtN8M3tRtKs62Z6dr8Thzge6y3R2A34rmrrZFoX0LZPtOxqCoy4lNuxf4HzLI8zl7mmEaPwFsI8jzfz8bnscvYUrK7uS1CmxCPoBtw3fQMbps1iH9cehy5pdX8/SuvVc0cDkifYspU2COLIhI8r22ShxgPAawaR9qzOlruGsj6GEZdYoYHLn4s0B+JoPrywCcR9+8wAundwNlUFvkdzeJsqIQSq0GtLTeNZ2txNXgad39UHzChnX2rzHv7/iW7vUJDJX6uW09wgzVpy59Q2A2yva9kr2c02bLnPA1yZaHnPuV+nr1PaMhrNBCYAQfaBSkjjDBUH/DYP+BE87v3RCMUtlfj4xH4uZ8dOvIaheRD1X6oKPC/T/BBhKAbMT49gcsn9eXNFXvIPnAEBeWS7dXRrH/ylS9BSDSc9Q8YeRGseiXgyb1BW5+HNy4J6D0aw1/+GpoPb4b/nGp6516MXP83eP1C82b+3+Cl01t/vzX/g2cmQ2Vx830NtVBfaV7bw3ntxvHA7oN7BN4hG39CYTfIdgP8n1NhwaPm9Q+vQqo136SysHlYxxt/HoH9HWmwcgG9xnmOLbPCPRknQGiMmf8QFuvrEYTGmNf+RN9boPzx6Z2m3pZwSEQIuhm/nDUQpRRPzt12+BdJyDCzle2GMn8D9Bxj/ol7T4K6ioD3xCIr95j7B7LUtt2I1VeZMJg3OatNT/zH1zzbdnxHfMla06O2E5mle1ufQC/cBO4GM7mqKd69410LzW9bCAqyrIZW+c8ReDeU/oTCbpBrSo2t5Tnms2lt7pFxgnWdIs+1yvzE7CuLTIdAOT299YIsE0q06TXeY1N5Hg3OcBNG/OUqmHQjhMV5xKr6AESlmoEL/jyCqkN4BCV7/Z8nNEOEoJuRFhvOlZP68u4P2ewpPkyvwOE0oaKCLNNYFGR5SmLH9TW/S7xGluT8CBs/huyVrb+H2w1bvjbnlTSJXWtNWE2+NZN2q+lB21VVqw9AkSVy1SVmWGLWp/5LcBdtbTG5CvgmxL0bm8piz9oPC/5uxEhrmPcXs7+h2giH3ROvbjLipabM2JT1qW/DbI+AKc8z9XW8h+ja14jtawoC1ld7euVWXSji+kJduUd4KotNItXbdn+f175OTYlvuKXGGmkU28v0zCuLvEJDLXgEUSlmrkr1fvNc9m+HoWeZhLBymA6DfWx5LnUhVggyKsWMZvP2CKr2mxBldE9zv+LtvoJ4KI+gen/z2kWCX0QIuiE3Zw7AoeA/i1pZktofqSMgby2U7DEegC0E8enmtz1CZP9OePFkePtqE3Jo7ezWlf+BNy8x5713ve++6gNeww6z4PN74OUzTWP89f3wn5NNY/jtg2akzVtXwVf/53uNhjozOW7On1q2oSALgq1kuHdjWmh5ClN/aXrPmz41z2LvcsqiB1nHe4VRmjZUX/zG2PTWVTD/Yc92WzzLcsxw3uczYfu8xs8MmNIMrjqzJnVjb9cKwyUONL9ry8yzeOsn8Or5vrYfKkfgLx8QmWxGk/mEhlrIEUQmGSGo2m9KSmi38QiGnG7yA/YIoapiKM+jNrRJLio8zjc0FB4P0T2M2D2fCXP+6DnWFoUWcwQHWi53IfggQtANSY0J47wxvXh7ZTYHKusOfYI/0qeZf/yNH5r3tvvf6BHsMr+/e8RMQDvlz6YHX7Cp6ZWaU19tetp9jjejmrJX+vbsvMeO5683QxrLc03DvX2eaTRz15jhhf1mwNir4cc3fIcZ7ltlGokd81qwocZ4GelTzHvvxtwOGU260fRgdy5ojNvn9DzNOr7IvxAUbjEVYCdcB32nGhsbP5cVEirPM40oGC9Da0+j198K0xRsNHHzaM/oMJIsEaopge3fwp4lJsxkX0s5Di4E1SVejatXox+ZZMTgkEJQZI6LSDCNuP2cUobD6X+D676wBiEoyyPIoTY00fcaYbFeyeIDEJ5ghODATvP3KvIKaR7MI3A1QG2pWbq14TC/490IEYJuyo0z+1Nd7+K57w5zDHbGDPN7xYvmd/JQ8zsk0jQGJXusRm+2acxHWAnUgo2mod+zzPd6BZs8CcHv/wMVeWby2rBzjYDsWeo51u45O0Nh/Qee3vKqV8zwQzCNbfFWGHwazLofnCG+azHYCdcDu/xPkCreau6bMdO89xGCjabBiukF6dM9QpA0mMrIfuYY72GQdv2dte/Al/eaMh4n/g4GnWwmUlUUms9Qax1fnuuxKft72Pq1p9FLGW5qRe1ZZkJQfY732GV7BNUlxhtSTvN+10JTMyqqhyf8k+dVXtxu1N31nudXtd+zPTLZ/JTnmnMdQSYH1FALGz+CNW+ZBrqyyPIIEkwjXphlFkhKHGDmn4RGm7BiRKI5vzzPExqyCYvz9QgiEnyHu3qHHBvDWF5/mwO7zXfP2/NsaWW+7JXmMwgiBN2VwanRXDy+N/9esIOn5x1G4ji+n+n9l+4xDWJ4nGdfXLr5h1z1imkIpt0Jsb1NIrEgC5b/G146DXYvMcdXHzBho0/vMu+/f9E0sP2mm5EkzlBPww2eRjJjpiexGp4Aq15u/jpjpulRTvq5EQfbI9m5wBwHnuSrN1u+NL8HnGR+N/UIUoab4bQZM01vdcd8yJhJXYj1HAq9PJ/KQlPx9f0bTE996m2mwbSTsLsW+opRea5p8PpMNj3+1W94euoRCSYMZ3syfad4zrPH0u9ZCjk/wLQ7zPvs76kPjjFhlppSMy/gxZM9S5GW53lE44A1gxjt8SQik03jbedhkq3hwytfhrd/Ch/cCK9dYOYHNHoEB0xCPXmIif17E5ls7HfVUR2e6rsvLNaIVX2NSdKHx5tORlC4GZ5clmMab639Dx99/0b46LYmQuDHC6ooMM9g/XvN93VDRAi6MX+7cBTnjenJo19t5n8rTINaWm0WAdlWUMH5Ty+moMoNQFmNn8VB7N6ynR+wiU83DdnOBdB3shkVopQ1IznLEw751gp7LHnK/LPuWmRyCgd2wrCzzTHB4UYMdn7nuX7JHuqDoqCv1RtOHGgSkq46E4MedYl5HRYHqdYkpGl3Gm9l/kPGI8leAWOuhIgkX5EB06Ne8qRZw6HHSJMnsBsbrY1HYH9m2zNy1UG/GabBhSaJ5kITlkoaDLevhhPuM9vTxhhx3LnA9GLBlPEozzPCkDjQDN3cv9P0jp2hpgxIyjBPIrrHKLMtJMr0+MEjsKMuMUledwN1IbGesEvxNhMy2bvMJKUrCjzehF1KwvszWOVGcDd47glmuGtIFFzwvBFk7TbHhccbUdi73IQQmxKZZDyxiEQKUmb67guPM7ZVWMnoiAQYcQH8epuZeIY2czNqy4w9EYlmWG1dpRlgkLfOfPe8E/T+PILyPHOtlhLN3QwRgm5MkNPBPy8dw8zByfzho/Vc+Mxixvz5az5avY8/frye1XtL+HZPA6v3ljD2z98wb1OTIaF2j7apEMSlm5E++es8DaV9XP56E9aISoXdi0zDvPw5876uHJY8YV3bq4HIOMH8g694AXLXQsluasJSPXmJjJme4zNmeuLoGTPMhDmAyESYfLPJaXz9e9Nw9880x+9caBr4ukr48XX4/Nem53yilWC2E6X5G00ZhppSz72ThxkxAeg3A7cz1OP52FQWWrNfB5qht7ZNziDoN814BHbIo+/xRgQq8oygxqd7Qh0RCZageg3HjEkznllkkmnowTxfR5C5n/W3qQ+O9SRibe9j5wJrmK+GFCu05yMEWaZRdwabBt7GFoLc1aahH32pJ0RlJ4sbrB59RpOG3j4GYPpduIIifPfZn8G2IzzefObQKPO9AiMitoeUNMR6xkVGjOorTSPvPbrIO7+0fZ7JGdhC0cKiS0H1FR2/yE87IkLQzXE6FE9eMZb0xEhySmoY2iOGu99ew+JtxcSGB7Mkp4Gnvt2Ky615afFO35P7Z5pGr2mvLz7dU5bBFgswDVhNiYltn/43E1767mHTKF/yijnmh1dNw5rsJS6DTwWUGR30+oVQuIWasBToOc6EDIaebdkSZTyJ9GnGGxh6jq9dU24z1/7+BRNn7zvZFN8rzzGN4vy/wUe3wrq3zcS4tOPMeXai9L0brNFHCnpPMPscDuONpE8zYgOmobMT08GRJgdQssfTkHkz4CTTQ9/wgbEpeagRRDDHx6WbZ3ZgtyeU5S280WnGltSRnvBcVREkDjJlKKxj60LiPGEXW3R2LvQMA7VzPAe8/sbF2z0CYDfe4FvqIWOmaahP+qPJwyQN8ZqVrozQNSVluPlcE65vvi/M+gx2GMr+zOAZkebd40+2yp5UFnnCfg01nvPB4xEUbYXXzjez4O3QUZ1/IeiZ87kZidZNcghSa0ggNjyYz2+fgUOZ0NB5Ty8mJiyY22YN5JY3fmBOVgE9YsJYuLWInUWVRIY4SY4ORUWlwL07Te/TG7vBC4kyxets7F6nchg3f8gZJlQREmEaqZQRJnnq3ZMH0yDfuxOyV8EbFwFQ3Wec6Q3/dq8nBv2bnabxA7hna/PYdHgc3LnW9BBDo00vc+TFpk7S178zCc+RF8OpD5px7TaRyab3W55rlv6cepvpqdqc/S/fmdSRyZ4GNWWoibXXVXgaMm/GXGnEcN8q08B6jwKKT4fgMPM6d41HmGwhCIszobNznrSeqzJ/C3eD55hk2yOIMceX5RiRDk8weYQiq0yILQTluR7h0y4vIfDyCFKGm5yCdzK93zT47T7z/O11BdKO831ONjN/bUJ19t/KG1sI7OfnXeokOs3knA7s9oxOsz2CqiLfcFy+VzVTewipbVdZjqdGVguJ5JC6MqvqaXHz2kxdEPEIBABCghwEOR0kRoXy9V0zeeemKZw8LJXoYLPvpZ9NJMihuOzfS5n017n8a45VE8cZ7KlBZGM3eOlTfRtjO6SRdpxplIPDTWNuhwO8wztNCY83o2wGmsVVasJSPfe38W5YgkKa2wWmAYhJMyIApqGd8SsTenLVmXBQTJoZ3WITmeQZQTPkzOaNm8NpwjyA1hpt956doRCfYSZVgafx8iY0yqwPAUZAvYXA9gjAWr4xzmNPZLLnWIfD/ChFjTPabLOftXdoKCzW46mNvszE9Jf/2+c4ABL6e5LHEbaXYwmBI8g0zlGp5jnY5SfA8/ztxtvf3xHM38WfCICf0JCXEDicENfHeFf2rGK7EGJloW84zlsU7NCQd/XTxtCQfyFwuio9x3YDRAiEZkSEBBEZGkRIkIOrhoXywLkjGN4zhgvH9cKtYXL/BJ6Yu5UPf9zn/wKxfcxIouHn+W6PTDYNVNPtNsPOMaGmg62PfNLvISSacnvi1tFg7NWmZznxev9VLO1GMCTa0ytvgbc21zFnj9s6L8nXs/ATGtqUV8aj+6ehEwaYpHi0lfB1hprG1ls8vBvF/id6yjVY1Lvc5NcbDyInxLpX2miISKI8ur/vyK5Rl5i1KHJXmwR7gtfnjkj0hIKaegQRSaYh7z3e/B0dfpqQxIEmST3snOb7DoVt4+7FJgnuHZICK//kLzRkCYEtDIWbPKJiewTe6yHYQtJCaCjIro3UTYRAQkPCQZncM4jMSaYxevii0QDUudxc9cJy7nxrNf9bsYdBqVEMTo3m6snpPPXtNr7emM9z162iV2yY78WUgluWNr2Fh37T4P8OUa0z7Ti4bw/lCxYc/Li2EBQCtyzz70GApxHsN62x519R24Bba2LCPB6Jy61ZnNPASHcopwRh9dy9GjI/oaGn523nkzU5XHbPAvomRXmSnHF9TCMbHm8a1doy3zDJRS80u9b8zYUku8NJd8Br2yO4dwrm/N9s58D8+RDmVR8oaRDctcEkdIMjzDMIiTINY3iCafAr8r0EwJoIZr+/7PWWCwvG9IT79rT8PA+G3XhXHzDDX4NCfffH9YVNn3meU3RPk4cpzzNhrvHXmlBcQ43JQdVXe4TAZz0Ey/aWksUNtkfQwqzlLoZ4BEKrUUqhlCI0yMlr1x/P/WcNI7+shk/X5vKHjzbwk/8s5x/fbGF9TimXPreUn7+2imtfXkF1nQlH1DW4j44h/nqhR+OahxICr1DH7f/7kV+8usrnsJW79lNeB8U6xnOefW54gslLeFFV18CcjfkAbC6wGp7weJN0tb0HpTyvwxPYu7+KK55fxs6iJkXwgHdX7aXaGU29CuGlLMgva1KQz46/h8dDWIxp/MPjPGEauxGOiPfyCKzf9kQwb2E7WEN/kH17iqv4bksLPW3bhpBok0doSny6yQeUZptjnUHGpi1fmdBez7G+nzM0xhP+8V4z2V+yuCzHzPMAnC4/HoH3MN8uhgiBcFiEhzi5YUZ/5v/6RH78/SncOLM/i7cVc3xGAu/fPBWAzXnlzNtcyENfZPHrd9Yw8S9z2FdS3W42FlfUsi7b/0ItbaLHKBNGGWxKSlfXuVi0tYi12SVor17xVxvycSoo1lZj5i0EfryBuVkFVNcbkdySbzVWSpmJdN7Dbu1zIxJYtK2IpTuKufvt1TS4PMJ6oLKObzcV4Ooxlrr+J1PrUnzQNHRnN7L+Ri+BVwOa4D9JnD7FhK+OkMfmbOG6V773/10ICjV5h8x7m6+JAZ6cxObPPKGynmNNcjnImnNiJ3fDE4zg1ZT5LrFpz/QG3xzBkqfgzctBay+PwEsI3vqJmV/SBZHQkHDEKKX47RlDmTEoieP6xBETFsyie09EKcWfPt7AK0t2AWao6l8/y+Lpq8YF3Ka9+6u48sVl5JTU8N7NUxnaI5riyjp6xYW3/WIpQ+HXngVjlu0sps7lps4FuaU19IwLR2vNVxvyGJnkxFWVCC48SV3w2/h+ujaHlOhQghzKIwQAV3/Q+PLdVdlM1sn0BgiPZ/PecpSCH/eU8PzCHdySaSaCLdxWRL1LE3XmA0T2iWPkkwv5akMeN53gif03hMQQBOQ5UklxaxyOJr32Ro+gBSG47PU2Pjj/bMwtM8ORF+1kRpTvvroGN+fXPsTVQf24wt/JA08xeaaCjSYRD2bosavejEZzBpk8S8FG49mERpvQUPUBMyw3KMw07vZoLB+PIBtctWYt5QZr1rUdGmqos4ryec1Y7kKIRyAcFZRSzBiU3BgzV1Zo4N7ThzJraAq/P3s4t88axGfrcrnyhWVc+txSdhT6j88eLnYIqqK2gcufX0ZpVT1JUSHcOftHznxiIbP+Pp+80tavX2DPsm7Kwi2euPGW/HIqahv41dtr2FdSzcQeThJSewFQ6I7hu32Wx9DEI9icV863mwo4a3QaQ9Ni2JzXfPRKvcvNHz5az3+zrGuEJ7Alv5zRvWI5eVgqz87fTkWtme27dHsR0WFBjOplGvPThvfgxz0lPuGh11eXAPDh7mBOf3wBH63eh8vtFee3E7Xh8c2TxU2oqXdR29DyEp5lNfU+3pL3Z9peWIHToZi9Yg+V9b7HfLUhj4155Xy2toV1BBwOz0Q/22OwRyFZ+ZvGCqfhCZ7QkF2oMG2Maey9V5GzaVygZz9Oly0EhY3bzAc/StVMtTaTF+37r3nLt0R4yR7Y9PnRuVcrCKgQKKVOV0ptVkptU0rdd5DjJiqlXEqpiwNpj9D+hIc4eelnE7l+ega/OKE/I3rGkF9Ww9aCci57fhlb8/0P32sr2wrKGf/gN/z7u+28vGgn+0qqeelnE/nnpWPYVVxFRU0D9S43Ly8xwxL9NVLevLsqm/H/7xs25TX/x1+4tZCRvWKs+1Zw/wfr+HD1Pu48eRBTewaR0mcwW9y9+OMPEdzycQ6u5BHQz5NfqGtwc/fbq81cjRMHMig1ih2FldS7fHMoa7NLqKpzsah+CHnONNyJg9iSX8GQHtHcNmsg5TUNzLZKgyzdXszxGYk4rV7+aSPN6KNvrBzE7jIXjyw+QHZIf46bcTZawx2zV/P3r72WLbU9gvAEE2JJHmZqRDXB5dZc8MwSft4kR2KzKa+MsX/+houeXcIna3IoqvBMyjKfU3PD9Awq61y8uqHWJ8T12jITvlm1+0Cz59HI0LPNyLI+k/3vt0de2TmCmjJPfsCeCOiqNUNhXXWe6qT2EOGKApxua1ujEFieQEsF7NpK7mozeTHrUyjdZ+o1rXvHs3/JU/DONQFf6c8mYEKglHICTwNnAMOBK5RSw1s47mHgq0DZInQOwoKdfHb7DOb+KpO3f2GKpV3+/DKyco+8l/Wvb7ZSVefi719v5t8LdnDK8FQm9Etg2sAkPrx1Gl/fNZMzRqbx5rI93PXWaqY/PI+SKv/liUur63no8ywa3Jq3v/cdxZRTUs3WggrOO64XSVEhbMwp45uN+Vw2sQ93njwYh1IM7pvGqXWP8nlZBpWuID6d9o6ZA2Hx6Feb2JBTxkMXjiIxKpQhqdHUudx8vSGfP3+ykcJy03Au3W7qCZ196qlMrvwHn2yrpaiilsGp0YzpE8ekjAReWrSTPcVV7CquYuoAT0nnQSlRZCRF8vGaHGrqXbywtpbIiAgi71jOlNOu4Ks7Z3Lh2F68uHAH2y3PrCHEiNvjS4t5ZHMKK8783Ez2a8Ina3LIyi1jwZZClmwvIr+sptFmgM/W5qK1Jr+sll/+70cm/3UuS7YbL8oW1gvG9eLe04eyPM/Fr95ZAxjvasXO/YzpE0d1vYsNOS18L5SCn7wHJ/za/35bCCKsHEFtuSc/0MsrLBljPDfqKqzV2CyPwLsAoB0aaswpHCWPIN+a51B9wONteC8pemCnEan69smpBdIjmARs01rv0FrXAbMBfwPIfwm8B8gq092IQanRvHXjZIKdDi7791Kenb+dDTmlbMwpY2NOGTX1LYcdAGpduvHYuVn5fLYul59OSSc+IoSK2gbuPmVw47Fj+sQRFxHCjTP7U17bwEer97GvpJo3ljcfAVJV18CfP9nI/qo6RvSM4cPV+6hrcFNQbkIsLy/eiUPBKcNTGZgSxefrc6msczFraKrP/YIcilsyB5AcHcpXGzyreb23KpsXFu7kp1PSOXWEabAGp5rRRLfP/pGXFu/kjMcXsGLnfpZsL2Z4WgzXT88gKjSIx6xJfEN6mONvPmEAOaU1/OwVUxNn6kCPECiluHpyOit27ue0xxaQXaF55KLRxEeaEUIOh+K3Zw4jLMjJ/R+sZ3dxJe9tML3djzZX8/yCHVz676V8tNo34dzgcvPYnC0M7RFNj5gwfvv+OjIfnc9PX1rR6GV9tSGPSRkJfPfrTN6/ZSrRYUG8s9IIalZuOcFORf+kKG7OHMA5A4L5aHUO6/eV8sqSXYQ4HY3DlFfs9LN2c2uIaRoaKjUeQVic73wJe45Gbblp6F1Wx8BeSyMiyQiB1ocODdVVmsl57laOjLMXN6op8Vp/was+ki1G7bSwTiCTxb0A7zUGs4HjvQ9QSvUCLgBmARNbupBS6kbgRoDU1FTmz59/WAZVVFQc9rmBpLPaBYG37VdjFP/dqHn4y008/KVne48Ixe3jwvh6Vz21bs1ZGSH0jjb9li0HXDy7upoD33jKR0cEwfERhQwYpcipCCV/8w/kb256N7h2RAg9oxx8tL2ef8/bwiD3XkKcJpyy7YCLx3+oobweTu8XzNCEWh7LqeO0R75iZ5mb0/sFM3dPPVPSgti1/nsiGmqpqXcT5AB3zkbmF2RRUVFB1g/LeGRmGPGhuWyMczF3Yx4vfjCXL3fVszLfxbAEBzOjCxufa51LowCn0vx8dCgfbq/jmv8spd4NJ/UJYtnihYxOhCU5ZiRL0fZ1zN/nQAEXDAzmg22VRAdDTtYq8jZ5EsAZWnPegGA+2l7F1FSNytvI/LyNPs/jooEO/ruhmBMenc8Jjr5MjRnK7yclU+t28q9VNdzz9moWrtpAsFPhcmuW5rrYW+7m9rGhlNRqXt1YQ2KYIiu3jBc+/JaYEMWW/GquGhrCooVmrseIeM1X6/YxJ/kASzbW0iNCsWSR2TcjuY6vdir+8NZS1hW5mNIziNxNq+gRofj8+60MdjdZprQVhNTWcFxEX9btriYtbz99a8op27IYQlLZuHYbduHu3Jpg0oDvF88DPA1Q7sZlpAGlQUnEVhWxcO6XpBQsZQhQX7mfxX7+H1Lyv2N41j9Zle+gPObQkx1HbVpMIpC9bQMlRQ5GAvm7NpE1fz5ozYz9O3ECyxfOoTrCE54L1P9jIIXA30DipgGvx4B7tdYudZBxx1rr54HnASZMmKAzMzMPy6D58+dzuOcGks5qF7SPbZecCRtzytiz3yToymrq+fMnG/m/RdU4lAkpLcut5skrxhIZGsTDX68kIdTBY5cdR1iwKYUwMCWSgSnRB7sNAJnW7+HbirjqxeVkh/bjuukZVNQ2cP9jC4iLDueVy8YwPj2Bepeb17d8y56KOsb1jePLXSUEORR/vWo66YmR7Andxbd7NjB1YDKnnWyGVTZ9Xo5ehcx/aQUPLq8hOjSIX84ayI0z+xPtNREN4FdqKyN7xZI5JIWr9ldx3tOLqa6s49LMMWQOS0X3KGDJK98TGx7M+aed2JiMz8yEKd/vJThIMWts83j+iSfCz/eVkrf5B79/x0zgqvxy/rt0FycNm0ifIb+lj7Vv4uRaLvn3Ut7d6pmz0D85ksfOHMT5Y3uhtebC7FIGpkQx5aG5bKhNYEh8NLCZW86f3jhCqz4ln0WvriS0z0gKl6/l+AEJZGaObXxeF4xL4K2VpsH/w6XTGJgSRWbxWj5dm8uOoHQundiHqNA2NlWnXcRkgMV5sOcdYsu2wIy7mXLCOWCtiZQ29HjI+5aJo4eZnre1pHZamAlzxWaMhTWbmDFuKGzcBFsg2FVN5gknNJ8nsXgNZMH4IX1gSPPn3IwfbgGgd2IUvTN6wwZIjQkmNTPTVIT9zngnx48eZmZxWwTq/zGQQpANjd8pgN5ATpNjJgCzrS91EnCmUqpBa/1hAO0SOiHDe8YwvGdM4/vBqdE8PmcLN2cOZFBKFD9/dSX3vLOGYIeDIanR3Da8gTPH9jrs+00dkMiMQUk8+NlGqutd/LD7APtKqnnnF1MYn25GowQ7Hbx63SSUMvb84+vNxEeEkJ5oCpYNTDFjH2cN8T+yBmBK/0TOPa4nA1OiuGZqP2LDg/0ed9ssTy+yT0IEL/x0PM99t4MpVtx/2sAk4iKCGZwSTdNO06UT+3AwRvaKpWhryx2tQanRPHj+qGbbE6NC+eauE6isa2jcFh0a1Hh/pRRj+sQBcNG43ry6dBcfr4EJ6fE+w3RnDEoiPNjJXz/fRG5pDUPTYnzu85PJ6by1ci8nD0ttfKYXje/N4u1F/PnTjWzJL+dvVriozYTa99Jm2dKgUFPltbbUM6S3rtx36U07sWyHkSqLPMli7TKzse2idTZ2fqHKaybytjmeAou7FpvzBp1iQkH2SnA1pZ4V2ezQkHeOoguEhr4HBimlMoB9wOXAld4HaK0z7NdKqVeAT0UEBDBx9pev9Uxeeu7q8Zz31GJqG1y8cM0Etq5efkTXV0rx/NUT+PmrK3n0q80EORT3nDqECf18JzF5i9NvTh/qs+/4jEQeOHcEl0xo3hO3CQly8MQVY1vc3xLj0xN44aceW0KCHDx5xdi294yPEKdD+ZTRaIlrp/Vj+c79nDwsheumZfjsCwt2ctKwFD5dm8vJw1K4vIlwjeodyyMXjW4UPYCJ/RJY+JtZ3PrGD8zbXIDWmvveW0duWQ0zByVx/fSMZoLoF3s2tzPEd82E2lLfHIHdkMf0MqN4ABL7m9+Vhc3XN2gmBLmeY20++5XJUwyYZdbdqCw0QmCvXqccvjkCOw9R0oWEQGvdoJS6DTMayAm8pLXeoJS6ydr/XKDuLXQ9kqJC+fSX06l3u0mJDmProU85JOEhTl6+diKb88oZkBxFeIjz0Cd54XQorpna7yhY0jpmDGrZ8+ho0hMj+eKOGS3u/8sFo/j1aUMavammtOTVzBycxGfrcvl8XR5vrdxLbHgwC7YUMn1QEkN7xPg9xwd7SGzvSZ4RUJHJZr0CO6lcW2HKS0QkmUJ/dm89wUsIfJa+LAO8qsSCZ71te5TRgd1m7oI9NLUsx+Mt2JVRe4xq4hFY97DnPMDRm7dwCAI6j0Br/bnWerDWeoDW+i/Wtuf8iYDW+mda63cDaY9wbBMfGUJKdNihD2wDwU4HI3vFtlkEhLYRGx7coggcjOmW+P3x4/UEOVSjd7WzsHmtJb/YHoF3yY7IJDN5zhaJugrjEUSn+ZYYT2jBI6gth5UvwQc3wVe/A1dDc4/AXge7Ih/cLnP9mlKzTnTBJlPgL3WkrxDUlpprlew2M6Dte236zMw5COAiOTKzWBCETkuvuHAGJEdSVFFH5pBkxqebhnqHn6J7fkkeCunTzYJDNkPPMusxhFj1LWorTEMe3aNxtrJbBRmhiEi0yl4f8MyyrimFOQ+Yhe+XPgU5P3pCS7YQ2Otga5fxPuoteyvyoHirKdUdHm/CQjUlHtvs1ejs9SFqyyD7ezPz2NnCGg5HARECQRA6NXZI7OLxvYkKDSIlOpRdrRWC8Di49jNIGujZNuZKOONhqyxFqCdZHJPWWMiuIcjyXlKGmx589X5T1hpMY19TYtZqBrN2gqvWs09rIwShlsex7wfPvcvzTEMf388ITX2lb6nrqv1GeOIzTHnt2nJzzcjkwyvr3UpECARB6NT8ZHI6P5nct3HSXkZSpN8y3IdFaJTplVcUmNBQhC0EVj4hZZhZ8KZqv2eUUdEW87vPZFPx1CpdTWiMadSLtxlhGX6u2Z7jJQSl+6B0r6k9ZVd7LdljGn0wjX5ptrU/xngflUXNF+g5yogQCILQqRmYEsWD548iJMg0VxlJkewqPkpCEBIFe5cD2grXGCFwOb2EoK7crF9sewSF1kzF2F6mMu0ea7GlHqNMQ77PqsE04nzzO+dHz/1yfjQzmOPSPTmKygJIyPDsdzeY/IS9IJHtEQQQEQJBEI4pMpJMzqBpddjS6nr2V9b5VlQ9FKHRnuGc/aY3JosbPYJkr7Wc46yRTbZHEJ1mQkd2aYoeo0wjvmcpOIIhfRqgIHet2e8Ihr3WbLa4dN+lQ22R2b3E/E4ZYZXQtkNDgfUIZD0CQRCOKfolmTDKrqJKhqZF88TcrXyyJrdxZvrZo9N46spWrnlhJ4wTB5nhpBFGCDwegdfckYgks3La/h3mfXQPk4y2sRfN2bnQrJ0cHG7WrK7IN/mCyETINQX2zEprXrWUbI9gjyUEyUM8i+pUFolHIAiC4E1/SwgWby/iwmeW8PS87QxMieK+M4YyIT2e73ft9zl+7/6qlsuONw4vtcqEhzfJEYTHe9Y3iLCWG3U3mNxAWJzxCOzzYq2Z7vu3ewTEroQa3cN4EG5rlnZsH09oyH7vCDKjk+LSTe4iNNqISH2V5AgEQRC86ZsYgVLwyJeb2bO/ihd/OoGXfjaRm04YwKkjUskvq20sMf7DngPMeGQeX23I93+xUMsjsOcZNE0Wg2cop730JZiGXSnPvpievr12e7stIjFpXqKQZlZIs5PFYATHnsNgi0tojEkcg3gEgiAI3oQGOUlPiCAuIpj//XwyJw/3lAAfZJX03pJv1lh4+3tTzO7TtU3LnFnYoaF+lhA0TRaDp1GPSPDULoq2ZhbH9DRhn+geTYTAasy9G3/7HHv0kbdHEBbrWYPZvl9YLI11OgMsBJIjEAThmOPpq8YRHRpM30TfhXOGWEKwOb+cUb1i+XRtLkrB/M2F1Da4CA1qMoN85IVWI26FXkKjYcptFNX2o3Fx0eMuh4YakyOwQ0l2A6+UWSAnrq+ZfGbT6BGkeY6PsO5hL1saHG4mibnqjHdgL73Z6BF4VdOV0JAgCIIvI3rGNhMBgLTYMKJDg9iSV85XG/KoqG3gxpn9qahtYMk2PwvdDJgFs+73vFcKTvuL75oCPUbBWf8w6yXboSG7ThHA1F/C8PPAGWzCO0HhENfPOs4Wgp4e8bA9AqU8XkFYrFdoyMovhHrVUpLQkCAIQutQSjG4RzSb88p5Y/lueseHc9fJg4kKDfJZKe6waeoRNCUy2Yz4cVhNq7dHYL+OT/cc7yMECaCcZgST973A400ECAkNCYLQpRicGs3bK/ficmv+fN4IwoKdnDoilfd/2MelE/swrm/8oS/SEk1zBE2Z+kvfBrzvZBj/M5OMDgqD8dfCoFM9++2EcXgcjLkCEgeYRDJ4vI+QKL9rRx9NxCMQBKFLMSQ1Cpdb0ysunMus8ta/P2s4aXFh3PjqKgrKahqPfWHBDm7478qWh5c2xe7BtyQE434KIy7wvA+NhnMeN2Gf4HA45zEzt8D7es4QIxL9psOMu73OtYQgwPkBECEQBKGLMbKXaaxvP2lgY3I4PjKE56+eQFFFLe+sMkMyV+3ez0NfZDEnK58l2/3kD/xxqNBQWwmLNT/+Cso1CkHg16GQ0JAgCF2K8enxfHzbNEb1ivXZPqRHNKN7xzI3K58bZmTwq7fXkBYbTlVdA68v2820ga3oeQ89y5R8iM849LGtYeL1nslsTQlrPyEQj0AQhC6FUorRveP8LmV54pAUftxbwkuLdrGruIq/XDCSSyf04euN+Xy3pZDsA6ZMRWGVmxcX7qDe5fa9QFxfOOkPnmTwkdJvOky41v8+2/uQ0JAgCMLRY9bQFLSGf3y9mRE9YzhhcDJXHt8XrTXXvLSCMx5fSL3Lzec763nwsyx++eaP1DW4D33hQGCHhgI8YghECARB6EaM6hVLUlQoDW7NjTP7o5QiPTGSz26fwV0nD6a8poGNOWVsL3GTEBnClxvy+Pd32zvG2JAIOO0hs5BOgBEhEASh2+BwKM4a1YP+yZGcNcoz8mdYWkzjCKNF24rIrnBz5aS+jO4dy+LtZgWxV5fu4sc9B/xeN2BMuQWSBh36uCNEksWCIHQr/nDOCBrcboKcvv3gHrFh9IoL541lu3FrGNs3joraBmZ/v4f8shr++PEGTh2eyr+vnkBFbQOhQQ6CnV2jL901PoUgCEIrcTpU85pDFuPT48kpNfMMxvSJY0K/eGrq3Twzbxtaw6rdB9Bac86Ti7hj9o9+r3EsIkIgCIJgMT7dzDpODlckRoU2vn9zxR4Aiirq+GpDHjuLKvl8XR4LthR2mK1HExECQRAEC7vhHxBnmsa02HB6xYVT79KM6RMHwN+/NktV9owN44FPNlDvcrMhp5RT/vld4/DTYw0RAkEQBIuhPaKZlJHAxB6e9Ok4Sxx+MbM/0aFBbCuoYHTvWP7vrGFsL6xk+Y79fLwmh60FFby4cGdHmX5EiBAIgiBYBDkdvP2LKYxP9QjBqcNT6RkbxozByYy1ROHEISnMGppCSJCDbzcVsHCLGVn01vd7OVBZ1yG2HwkyakgQBOEgnHNcT845zqw/MCE9ngVbCpk1NIWIkCCm9E/k07U5FJTXct6Ynny0Oodzn15EYXktLrfm4vF9eOjCUR38CQ6NeASCIAit5OrJ6Txy0WhG9zZ1jGYNTaGgvBaAG6b35yeT+5IYGcoVk/oyvGcsn6/Lxe3WbMgp5e63VnPWEwvJyi3ryI/gFxECQRCEVhIfGcKlE/s01jGaNdSUlE6IDGFEzxgePH8UH946jT+eM4KrJvWltLqeHUUV/OrtNXyzMZ8dhZU8MXcrWmuW7SimoWktow5ChEAQBOEw6ZMQwZg+cZw2ogcOh2+Ru/H9TD7hkzW5bMor57ZZA7l2Wj++3JDH7z5cz+XPL+OVJbsaj//r51mc8+Si1q+NcBSRHIEgCMIR8M5NU3D4qXTaPymS+Ihg/rPIjCSaNTSF2PBgXly4kzeX70Ep+GRNDjfM6M/CrYU8v2AHAJvzyxnaI6bZ9QKJeASCIAhHQLDTgdPRXAiUUoxPj6eitoHe8eEMTIkiJSaMX5zQn8whydx50mDWZJeyem8Jv3l3LX0TzHKUc7MKAHC7288zEI9AEAQhQIxPT2BOVgEnDU1pzCv86tQhAGQfqOJfc7Zw5QvLcLk17940ld9+sJZ5mwqoqG3g49U5vH3TFHYXV/La0t243Jp0Zz2ZAbBThEAQBCFATB+YxCMKTh/ZfI3j3vERjOsbxw97Snj88jGM6h3LrKGpPPntVlbuNlVOr3phGTklNcSEB5MUFUJiXGDslNCQIAhCgBjVO5bvf3cyUwYk+t3/1wtH8exV4zhvTC/As3BO/6RInvvJOPbsr2JoWjRz7p7Jl3fO5LR+wQGxUzwCQRCEAJIUFdrivqE9YnwSw6N7xXLTCQM4b0xPhqXF8PVdM+kVF0F4iP9qqUcLEQJBEIROgsOhuO+MoY3vB6ZEt8992+UugiAIQqcloEKglDpdKbVZKbVNKXWfn/1XKaXWWj9LlFLHBdIeQRAEoTkBEwKllBN4GjgDGA5coZQa3uSwncAJWuvRwP8Dng+UPYIgCIJ/AukRTAK2aa13aK3rgNnAed4HaK2XaK3t1aCXAb0DaI8gCILgh0AKQS9gr9f7bGtbS1wPfBFAewRBEAQ/qEAVOFJKXQKcprW+wXp/NTBJa/1LP8eeCDwDTNdaF/vZfyNwI0Bqaur42bNnH5ZNFRUVREVFHda5gaSz2gWd1zaxq22IXW2jK9p14oknrtJaT/C7U2sdkB9gCvCV1/vfAr/1c9xoYDswuDXXHT9+vD5c5s2bd9jnBpLOapfWndc2sattiF1toyvaBazULbSrgQwNfQ8MUkplKKVCgMuBj70PUEr1Bd4HrtZabwmgLYIgCEILBCw0BKCUOhN4DHACL2mt/6KUuglAa/2cUupF4CJgt3VKg27JdfFcs9Dr+LaSBBQd5rmBpLPaBZ3XNrGrbYhdbaMr2pWutU72tyOgQtDZUEqtPJTQdASd1S7ovLaJXW1D7Gob3c0umVksCILQzREhEARB6OZ0NyHorDOXO6td0HltE7vahtjVNrqVXd0qRyAIgiA0p7t5BIIgCEITRAgEQRC6Od1GCA5VErsd7eijlJqnlMpSSm1QSt1hbf+TUmqfUmq19XNmB9i2Sym1zrr/SmtbglLqG6XUVut3fDvbNMTrmaxWSpUppe7siOellHpJKVWglFrvta3F56OU+q31fduslDqtne16VCm1ySrx/oFSKs7a3k8pVe313J5rZ7ta/Lt18PN6y8umXUqp1db29nxeLbUNgf+OtTTluCv9YCa0bQf6AyHAGmB4B9mSBoyzXkcDWzBluv8E3NPBz2kXkNRk2yPAfdbr+4CHO/jvmAekd8TzAmYC44D1h3o+1t90DRAKZFjfP2c72nUqEGS9ftjLrn7ex3XA8/L7d+vo59Vk/z+AP3TA82qpbQj4d6y7eASHLIndXmitc7XWP1ivy4EsDl6VtaM5D/iv9fq/wPkdZwonAdu11oc7s/yI0FovAPY32dzS8zkPmK21rtVa7wS2Yb6H7WKX1vprrXWD9bZDSry38LxaokOfl41SSgGXAv8LxL0PxkHahoB/x7qLELS1JHa7oJTqB4wFllubbrNc+ZfaOwRjoYGvlVKrrIqvAKla61wwX1QgpQPssrkc33/Qjn5e0PLz6UzfuevwLfGeoZT6USn1nVJqRgfY4+/v1lme1wwgX2u91Wtbuz+vJm1DwL9j3UUIlJ9tHTpuVikVBbwH3Km1LgOeBQYAY4BcjHva3kzTWo/DrCp3q1JqZgfY4BdlCheeC7xjbeoMz+tgdIrvnFLqd0AD8Ia1KRfoq7UeC9wNvKmUimlHk1r6u3WK5wVcgW9no92fl5+2ocVD/Ww7rGfWXYQgG+jj9b43kNNBtqCUCsb8od/QWr8PoLXO11q7tNZu4AUC5BYfDK11jvW7APjAsiFfKZVm2Z0GFLS3XRZnAD9orfMtGzv8eVm09Hw6/DunlLoGOBu4SltBZSuMUGy9XoVVAr69bDrI360zPK8g4ELgLXtbez8vf20D7fAd6y5CcMiS2O2FFYP8D5Cltf6n1/Y0r8MuANY3PTfAdkUqpaLt15hk43rMc7rGOuwa4KP2tMsLn55aRz8vL1p6Ph8DlyulQpVSGcAgYEV7GaWUOh24FzhXa13ltT1ZmfXEUUr1t+za0Y52tfR369DnZXEysElrnW1vaM/n1VLbQHt8x9ojG94ZfoAzMVn47cDvOtCO6Rj3bS2w2vo5E3gNWGdt/xhIa2e7+mNGIKwBNtjPCEgE5gJbrd8JHfDMIoBiINZrW7s/L4wQ5QL1mN7Y9Qd7PsDvrO/bZuCMdrZrGyZ+bH/HnrOOvcj6+64BfgDOaWe7Wvy7deTzsra/AtzU5Nj2fF4ttQ0B/45JiQlBEIRuTncJDQmCIAgtIEIgCILQzREhEARB6OaIEAiCIHRzRAgEQRC6OSIEgtAEpZRL+VY8PWrVaq1qlh0150EQ/BLU0QYIQiekWms9pqONEIT2QjwCQWglVp36h5VSK6yfgdb2dKXUXKuQ2lylVF9re6oyawGssX6mWpdyKqVesGrOf62UCu+wDyUIiBAIgj/Cm4SGLvPaV6a1ngQ8BTxmbXsKeFVrPRpT3O0Ja/sTwHda6+Mw9e83WNsHAU9rrUcAJZjZq4LQYcjMYkFoglKqQmsd5Wf7LmCW1nqHVRwsT2udqJQqwpRKqLe252qtk5RShUBvrXWt1zX6Ad9orQdZ7+8FgrXWD7bDRxMEv4hHIAhtQ7fwuqVj/FHr9dqF5OqEDkaEQBDaxmVev5dar5dgKtoCXAUssl7PBW4GUEo527nuvyC0GumJCEJzwpW1eLnFl1prewhpqFJqOaYTdYW17XbgJaXUr4FC4Fpr+x3A80qp6zE9/5sxVS8FoVMhOQJBaCVWjmCC1rqoo20RhKOJhIYEQRC6OeIRCIIgdHPEIxAEQejmiBAIgiB0c0QIBEEQujkiBIIgCN0cEQJBEIRuzv8HYwspvwudRDwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "metrics = pd.read_csv(f\"{trainer.logger.log_dir}/metrics.csv\")\n",
    "\n",
    "aggreg_metrics = []\n",
    "agg_col = \"epoch\"\n",
    "for i, dfg in metrics.groupby(agg_col):\n",
    "    agg = dict(dfg.mean())\n",
    "    agg[agg_col] = i\n",
    "    aggreg_metrics.append(agg)\n",
    "\n",
    "df_metrics = pd.DataFrame(aggreg_metrics)\n",
    "df_metrics[[\"train_loss\", \"valid_loss\"]].plot(\n",
    "    grid=True, legend=True, xlabel='Epoch', ylabel='Loss')\n",
    "df_metrics[[\"train_mae\", \"valid_mae\"]].plot(\n",
    "    grid=True, legend=True, xlabel='Epoch', ylabel='MAE')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "911969d1",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Restoring states from the checkpoint path at logs/mlp-beckham-cement/version_2/checkpoints/epoch=198-step=994.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "Loaded model weights from checkpoint at logs/mlp-beckham-cement/version_2/checkpoints/epoch=198-step=994.ckpt\n",
      "/home/jovyan/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:258: PossibleUserWarning: The dataloader, test_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 4 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.\n",
      "  rank_zero_warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7f5a8ca959f641c59a458ecc2cf3a747",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Testing: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\">        Test metric        </span>┃<span style=\"font-weight: bold\">       DataLoader 0        </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_mae          </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.3400000035762787     </span>│\n",
       "└───────────────────────────┴───────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1m       Test metric       \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      DataLoader 0       \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_mae         \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.3400000035762787    \u001b[0m\u001b[35m \u001b[0m│\n",
       "└───────────────────────────┴───────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[{'test_mae': 0.3400000035762787}]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.test(model=lightning_model, datamodule=data_module, ckpt_path='best')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}