training_TabNet.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the TabNet model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we are going to train the TabNet neural network model, in which the following architectures and tools have been implemented:\n",
    "\n",
    "- Ghost Batch Normalization\n",
    "- Sparsemax activation function\n",
    "- Gated Linear Unit blocks\n",
    "- Shared and dependant GLU fully connected layers across all decision steps\n",
    "- Attentive Transformer\n",
    "- Feature Transformer\n",
    "\n",
    "The architecture of the TabNet model is described in the following paper: https://arxiv.org/pdf/1908.07442.pdf.\n",
    "\n",
    "The implementation of all building blocks of the model can be found in the file: <i>TabNetBlocks.py</i> in this repository.\n",
    "\n",
    "Pytorch Datasets are defined in the following file: <i>PytorchDatasets.py</i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with importing all indispensable libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.utils.data.dataset import Dataset\n",
    "from torch.utils.data import DataLoader\n",
    "from TabNetBlocks import TabNet\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from tqdm import tqdm\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import pickle\n",
    "import h5py\n",
    "import mariadb\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from getpass import getpass\n",
    "from PytorchDatasets import DatasetFromHDF5, PandasDataset, MariaDB_chunk_indices, MariaDB_dataset\n",
    "from config import mariadb_hostname, mariadb_database_name, mariadb_port, mariadb_table_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also have to load the normalization parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(r\"norm_params.pickle\", \"rb\") as output_file:\n",
    "    norm_params = pickle.load(output_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will connect to the MariaDB server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Login to MariaDB:\n",
      "User: root\n",
      "Password: ········\n"
     ]
    }
   ],
   "source": [
    "print('Login to MariaDB:')\n",
    "mariadb_user = input('User: ')\n",
    "mariadb_password = getpass('Password: ')\n",
    "\n",
    "# Connect to MariaDB server\n",
    "try:\n",
    "    cnx = mariadb.connect(host=mariadb_hostname, port=mariadb_port, user=mariadb_user, password=mariadb_password)\n",
    "except mariadb.Error as msg:\n",
    "    logging.error(\"Error connecting to MariaDB Platform: {}\".format(msg))\n",
    "    sys.exit(1)\n",
    "\n",
    "# Instantiate cursor object\n",
    "cursor = cnx.cursor()\n",
    "\n",
    "# Use given database\n",
    "cursor.execute(\"USE {};\".format(mariadb_database_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the Pytorch MariaDB Dataset class to load the data in a chunked manner. At first, we will generate the chunks indices that will be provided into the loader. To do this we will utilize the <i>MariaDB_chunk_indices</i> class from the <i>PytorchDatasets.py file</i>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parameters: cursor, table_name, chunk_size\n",
    "chunk_indices = MariaDB_chunk_indices(cursor, mariadb_table_name, 1024 * 50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Subsequently, we will continuously read rows specified by the indices parameter from the MariaDB database.\n",
    "\n",
    "To load only particular input features, provide them under the <i>feat_cols</i> variable (list of input column names)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input feat:  tensor([[8.4529e-02, 1.6240e-04, 4.1961e-01, 1.6239e-04, 4.1176e-01, 1.6282e-04,\n",
      "         6.6667e-01, 1.6284e-04, 2.4314e-01, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n",
      "         0.0000e+00, 7.8333e-01, 5.1944e-01, 6.4251e-02, 7.8333e-01, 5.1944e-01,\n",
      "         1.1065e-02, 7.8333e-01, 5.1944e-01, 1.0792e-02, 7.8333e-01, 5.1667e-01,\n",
      "         7.6991e-03, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n",
      "         0.0000e+00, 4.6947e-01, 4.6207e-01, 5.0287e-01, 4.9535e-01, 5.3608e-01,\n",
      "         5.2707e-01, 7.8438e-01, 5.1848e-01, 7.8438e-01, 5.1848e-01],\n",
      "        [8.1506e-01, 3.3662e-03, 7.2941e-01, 3.3661e-03, 7.8431e-02, 3.3665e-03,\n",
      "         7.4510e-02, 3.3666e-03, 3.9216e-02, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n",
      "         0.0000e+00, 7.9444e-01, 5.1944e-01, 7.6223e-03, 7.9444e-01, 5.1667e-01,\n",
      "         9.8353e-04, 7.9444e-01, 5.1667e-01, 1.2291e-03, 7.9444e-01, 5.1389e-01,\n",
      "         1.4749e-03, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n",
      "         0.0000e+00, 4.3029e-01, 4.2169e-01, 5.0170e-01, 4.6945e-01, 5.4985e-01,\n",
      "         5.4259e-01, 7.9350e-01, 5.1687e-01, 7.9350e-01, 5.1687e-01]],\n",
      "       dtype=torch.float64)\n",
      "target:  tensor([[0.7838, 0.5181, 0.0832],\n",
      "        [0.7945, 0.5161, 0.8042]], dtype=torch.float64)\n"
     ]
    }
   ],
   "source": [
    "feat_cols = None\n",
    "\n",
    "for chunk_ind in chunk_indices:\n",
    "    \n",
    "    mariadb_dataset = MariaDB_dataset(chunk_ind, cursor, mariadb_table_name, norm_params, feat_cols=feat_cols)\n",
    "    \n",
    "    batch_loader = DataLoader(mariadb_dataset, batch_size=2, drop_last=True)\n",
    "    \n",
    "    for input_feat, target in batch_loader:\n",
    "        print('Input feat: ', input_feat)\n",
    "        print('target: ', target)\n",
    "        \n",
    "        break\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now is the time to create the evaluation and test datasets. To limit the size of these sets we will load 3 million rows from the CSV, and then take a random sample from it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_test_df = pd.read_csv('round2_competition_data/eval_test/eval_test.csv', nrows=3e+6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Take a random sample of eval_test dataframe\n",
    "eval_test_df = eval_test_df.sample(n=800000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we will split the dataset into the evaluation and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split dataframe to evaluation and test sets\n",
    "eval_df, test_df = train_test_split(eval_test_df, test_size=0.4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# feat_cols = norm_params['input_features'][:30]\n",
    "feat_cols = None\n",
    "eval_set = PandasDataset(eval_test_df, norm_params, feat_cols=feat_cols)\n",
    "# test_set = PandasDataset(test_df, norm_params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_loader = DataLoader(eval_set, batch_size=2, shuffle=False, drop_last=True)\n",
    "# test_loader = DataLoader(test_set, batch_size=2, shuffle=False, drop_last=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at how the data in the evaluation set present."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input_feat: tensor([[ 4.0113e-01,  2.1952e-01,  7.2157e-01,  2.3402e-01,  7.2549e-01,\n",
      "          2.1952e-01,  2.7843e-01,  2.1952e-01,  7.5294e-01,  2.2275e-01,\n",
      "          3.4902e-01,  2.4431e-01,  5.2549e-01,  7.9468e-01,  4.9381e-01,\n",
      "          4.8930e-02,  7.9572e-01,  4.9750e-01,  1.5491e-02,  7.9723e-01,\n",
      "          4.9359e-01,  2.0895e-02,  7.9762e-01,  4.9150e-01,  4.9164e-03,\n",
      "          7.9690e-01,  4.9177e-01,  1.1259e-02,  7.9762e-01,  4.9150e-01,\n",
      "          3.6873e-03,  2.0403e+04,  5.7852e-01, -2.0292e+04,  1.0799e+00,\n",
      "         -2.0145e+04,  9.9250e-01,  7.9663e-01,  4.9328e-01,  7.9664e-01,\n",
      "          4.9323e-01],\n",
      "        [ 5.7070e-01,  2.4573e-02,  2.5098e-01,  2.7955e-02,  2.1961e-01,\n",
      "          0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,\n",
      "          0.0000e+00,  0.0000e+00,  0.0000e+00,  7.2309e-01,  5.2393e-01,\n",
      "          2.2612e-02,  7.3253e-01,  5.3507e-01,  3.3159e-02,  0.0000e+00,\n",
      "          0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,\n",
      "          0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,\n",
      "          0.0000e+00,  4.7608e+03,  4.9198e-01,  5.0171e-01,  5.0133e-01,\n",
      "          5.1117e-01,  5.0236e-01,  7.2781e-01,  5.2950e-01,  7.2751e-01,\n",
      "          5.2914e-01]], dtype=torch.float64)\n",
      "target: tensor([[0.8008, 0.4950, 0.4003],\n",
      "        [0.7338, 0.5299, 0.5561]], dtype=torch.float64)\n"
     ]
    }
   ],
   "source": [
    "for input_feat, target in eval_loader:\n",
    "    print('input_feat:', input_feat)\n",
    "    print('target:', target)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we will define a function that inverses the normalization and converts WSG84 coordinates to cartesian ones. This function will be used as a helper function while calculating the distance error in kilometres."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Target's normalization parameters\n",
    "y_min = np.array([norm_params[col]['min'] for col in norm_params['target']])\n",
    "y_max = np.array([norm_params[col]['max'] for col in norm_params['target']])\n",
    "\n",
    "def lla_to_ecef(df):\n",
    "    \"\"\"Reverses the normalization and converts WSG84 coordinates to cartesian ones.\n",
    "    \n",
    "    \"\"\"\n",
    "    \n",
    "    # Inverse the normalization\n",
    "    df = df * (y_max - y_min) + y_min\n",
    "    \n",
    "    latitude = np.radians(df[:, 0])\n",
    "    longitude = np.radians(df[:, 1])\n",
    "    altitude = df[:, 2]\n",
    "\n",
    "    # WSG84 ellipsoid constants\n",
    "    a = 6378137\n",
    "    e = 8.1819190842622e-2\n",
    "\n",
    "    # Prime vertical radius of curvature\n",
    "    N = a / np.sqrt(1 - e**2 * np.sin(latitude)**2)\n",
    "    \n",
    "    x = (N + altitude) * np.cos(latitude) * np.cos(longitude)\n",
    "    y = (N + altitude) * np.cos(latitude) * np.sin(longitude)\n",
    "    z = ((1 - e**2) * N + altitude) * np.sin(latitude)\n",
    "\n",
    "    # df = torch.cat([x.unsqueeze(1), y.unsqueeze(1), z.unsqueeze(1)], dim=1)\n",
    "\n",
    "    df = np.hstack([np.expand_dims(x, axis=1), np.expand_dims(y, axis=1), np.expand_dims(z, axis=1)])\n",
    "    \n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Subsequently, we will define functions that will perform a single training or evaluation epoch. Those functions will take as an argument the Pytorch TabNet model and training/evaluation iterator, and return the metrics such as average loss, mean distance error or explanatory mask."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_model(model, train_iterator):\n",
    "    \"\"\"Perform single training epoch.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    model: TabNet class object\n",
    "    train_iterator: torch.utils.data.DataLoader\n",
    "        Pytorch DataLoader object that loads data from MariaDB_dataset/PandasDataset/DatasetFromHDF5\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    train_losses: list\n",
    "        List of the average training batch losses\n",
    "    avg_loss: float\n",
    "        Average loss on the entire training set (1 epoch)\n",
    "    mean_dist_error: float\n",
    "        Mean distance error in kilometres\n",
    "    mask_list: list\n",
    "        List of masks.\n",
    "    mask_explain_list: list\n",
    "        List of explanatory masks\n",
    "    \n",
    "    \"\"\"\n",
    "    \n",
    "    model.train()\n",
    "    model.double()        \n",
    "\n",
    "    train_losses = []\n",
    "    losses = []\n",
    "    dist_errors_list = []\n",
    "    pred_total = torch.DoubleTensor()\n",
    "    target_total = torch.DoubleTensor()\n",
    "\n",
    "    for i, batch in enumerate(train_iterator, 1):\n",
    "\n",
    "        input_feat, target = batch\n",
    "        \n",
    "        mask_list = []\n",
    "        mask_explain_list = []\n",
    "\n",
    "        input_feat.to(model.device)\n",
    "        target.to(model.device)\n",
    "\n",
    "        model.optimizer.zero_grad()\n",
    "\n",
    "        pred, sparse_loss, masks, mask_explain = model.forward(input_feat)\n",
    "        loss = model.loss_fn(pred, target)\n",
    "        loss -= 1e-3 * sparse_loss.squeeze(-1)\n",
    "        loss.backward()\n",
    "        losses.append(loss.data.cpu().numpy())\n",
    "        \n",
    "        model.optimizer.step()\n",
    "     \n",
    "        pred_total = torch.cat([pred_total, pred], dim=0)\n",
    "        target_total = torch.cat([target_total, target], dim=0)\n",
    "        \n",
    "        print_every = len(train_iterator) // 2\n",
    "        \n",
    "        if i % print_every == 0:\n",
    "            avg_train_loss = np.mean(losses)\n",
    "            train_losses.append(avg_train_loss)\n",
    "\n",
    "            pred_ecef = lla_to_ecef(pred_total.detach().numpy())\n",
    "            target_ecef = lla_to_ecef(target_total.detach().numpy())\n",
    "\n",
    "            # Calculate the average prediciton - target distance error in kilometers\n",
    "            dist_error = np.sqrt((pred_ecef[:, 0] - target_ecef[:, 0])**2 + (pred_ecef[:, 1] - target_ecef[:, 1])**2 + \\\n",
    "                    (pred_ecef[:, 2] - target_ecef[:, 2])**2) / 1000\n",
    "\n",
    "            dist_error = np.mean(np.abs(dist_error))\n",
    "            dist_errors_list.append(dist_error)\n",
    "\n",
    "            print('Average training loss: {:.4f}. Mean distance error (km): {:.3f}'\\\n",
    "                  .format(avg_train_loss, dist_error))\n",
    "\n",
    "            losses = []\n",
    "\n",
    "            pred_total = torch.DoubleTensor()\n",
    "            target_total = torch.DoubleTensor()\n",
    "            \n",
    "            mask_list.append(masks)\n",
    "            mask_explain_list.append(mask_explain)\n",
    "\n",
    "    avg_loss = np.mean(train_losses)\n",
    "\n",
    "    mean_dist_error = np.mean(dist_errors_list)\n",
    "\n",
    "    return train_losses, avg_loss, mean_dist_error, mask_list, mask_explain_list\n",
    "\n",
    "\n",
    "def evaluate_model(model, eval_iterator):\n",
    "    \"\"\"Perform single evaluation epoch.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    model: TabNet class object\n",
    "    eval_iterator: torch.utils.data.DataLoader\n",
    "        Pytorch DataLoader object that loads data from MariaDB_dataset/PandasDataset/DatasetFromHDF5\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    eval_losses: list\n",
    "        List of the average evaluation batch losses\n",
    "    avg_loss: float\n",
    "        Average loss on the entire training set (1 epoch)\n",
    "    mean_dist_error: float\n",
    "        Mean distance error in kilometres\n",
    "    \n",
    "    \"\"\"\n",
    "    \n",
    "    model.eval()\n",
    "    model.double() \n",
    "\n",
    "    eval_losses = []\n",
    "    losses = []\n",
    "    dist_errors_list = []\n",
    "    pred_total = torch.DoubleTensor()\n",
    "    target_total = torch.DoubleTensor()\n",
    "\n",
    "    with torch.no_grad():\n",
    "        for i, batch in tqdm(enumerate(eval_iterator, 1), total=len(eval_iterator), desc='Evaluation'):\n",
    "\n",
    "            input_feat, target = batch\n",
    "\n",
    "            input_feat.to(model.device)\n",
    "            target.to(model.device)\n",
    "            \n",
    "            pred, sparse_loss, _, _ = model.forward(input_feat)\n",
    "            loss = model.loss_fn(pred, target)\n",
    "            loss -= 1e-3 * sparse_loss.squeeze(-1)\n",
    "            losses.append(loss.data.cpu().numpy())\n",
    "            \n",
    "            pred_total = torch.cat([pred_total, pred], dim=0)\n",
    "            target_total = torch.cat([target_total, target], dim=0)     \n",
    "            \n",
    "            print_every = len(eval_iterator) // 2\n",
    "\n",
    "            if i % print_every == 0:\n",
    "                avg_eval_loss = np.mean(losses)\n",
    "                eval_losses.append(avg_eval_loss)\n",
    "\n",
    "                pred_ecef = lla_to_ecef(pred_total.detach().numpy())\n",
    "                target_ecef = lla_to_ecef(target_total.detach().numpy())\n",
    "\n",
    "                # Calculate the average prediciton - target distance error in kilometers\n",
    "                dist_error = np.sqrt((pred_ecef[:, 0] - target_ecef[:, 0])**2 + (pred_ecef[:, 1] - target_ecef[:, 1])**2 + \\\n",
    "                        (pred_ecef[:, 2] - target_ecef[:, 2])**2) / 1000\n",
    "\n",
    "                dist_error = np.mean(np.abs(dist_error))\n",
    "                dist_errors_list.append(dist_error)\n",
    "\n",
    "                print('Iteration: {}. Average evaluation loss: {:.4f}. Mean distance error (km): {:.3f}'\\\n",
    "                      .format(i, avg_eval_loss, dist_error))\n",
    "\n",
    "                losses = []\n",
    "\n",
    "                pred_total = torch.DoubleTensor()\n",
    "                target_total = torch.DoubleTensor()\n",
    "\n",
    "    avg_loss = np.mean(eval_losses)\n",
    "\n",
    "    mean_dist_error = np.mean(dist_errors_list)\n",
    "\n",
    "    return eval_losses, avg_loss, mean_dist_error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we will set the parameters of the model. Note that I decided to train a not too complex model, but you can still play with those parameters to fine-tune a more sophisticated neural network. Then we will instantiate the model, add loss function, optimizer, and device to it and begin the training. We will also define a scheduler object that will be responsible for adjusting the learning rate based on the number of epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|                                                                                           | 0/19 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Start epoch [1/8]\n",
      "Current learning rate: 0.00700\n",
      "Average training loss: 0.0376. Mean distance error (km): 2911.726\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  5%|████▏                                                                           | 1/19 [05:01<1:30:20, 301.14s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0174. Mean distance error (km): 1073.341\n",
      "Average training loss: 0.0163. Mean distance error (km): 862.944\n",
      "Average training loss: 0.0129. Mean distance error (km): 884.390\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 11%|████████▍                                                                       | 2/19 [09:55<1:24:45, 299.17s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0063. Mean distance error (km): 771.523\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 16%|████████████▋                                                                   | 3/19 [14:55<1:19:47, 299.23s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0056. Mean distance error (km): 783.126\n",
      "Average training loss: 0.0036. Mean distance error (km): 685.874\n",
      "Average training loss: 0.0024. Mean distance error (km): 555.967\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 21%|████████████████▊                                                               | 4/19 [19:50<1:14:29, 297.98s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0035. Mean distance error (km): 718.883\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 26%|█████████████████████                                                           | 5/19 [24:50<1:09:40, 298.64s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0027. Mean distance error (km): 560.405\n",
      "Average training loss: 0.0027. Mean distance error (km): 527.124\n",
      "Average training loss: 0.0024. Mean distance error (km): 503.026\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 32%|█████████████████████████▎                                                      | 6/19 [29:38<1:03:59, 295.36s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0025. Mean distance error (km): 482.787\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 37%|██████████████████████████████▏                                                   | 7/19 [34:29<58:51, 294.29s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0026. Mean distance error (km): 602.369\n",
      "Average training loss: 0.0016. Mean distance error (km): 542.421\n",
      "Average training loss: 0.0016. Mean distance error (km): 508.655\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 42%|██████████████████████████████████▌                                               | 8/19 [39:07<53:03, 289.43s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0015. Mean distance error (km): 498.913\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 47%|██████████████████████████████████████▊                                           | 9/19 [43:37<47:13, 283.37s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0013. Mean distance error (km): 493.110\n",
      "Average training loss: 0.0011. Mean distance error (km): 486.107\n",
      "Average training loss: 0.0008. Mean distance error (km): 486.586\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 53%|██████████████████████████████████████████▋                                      | 10/19 [48:05<41:51, 279.02s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0008. Mean distance error (km): 480.760\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 58%|██████████████████████████████████████████████▉                                  | 11/19 [53:10<38:12, 286.57s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0010. Mean distance error (km): 486.434\n",
      "Average training loss: 0.0011. Mean distance error (km): 465.807\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 63%|███████████████████████████████████████████████████▏                             | 12/19 [58:02<33:38, 288.40s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0010. Mean distance error (km): 456.934\n",
      "Average training loss: 0.0011. Mean distance error (km): 508.564\n",
      "Average training loss: 0.0010. Mean distance error (km): 530.967\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 68%|██████████████████████████████████████████████████████                         | 13/19 [1:03:17<29:38, 296.42s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0009. Mean distance error (km): 497.309\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 74%|██████████████████████████████████████████████████████████▏                    | 14/19 [1:08:29<25:04, 300.91s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0008. Mean distance error (km): 484.887\n",
      "Average training loss: 0.0008. Mean distance error (km): 491.257\n",
      "Average training loss: 0.0007. Mean distance error (km): 476.943\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 79%|██████████████████████████████████████████████████████████████▎                | 15/19 [1:13:22<19:54, 298.72s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0007. Mean distance error (km): 472.377\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 84%|██████████████████████████████████████████████████████████████████▌            | 16/19 [1:18:24<14:58, 299.65s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0010. Mean distance error (km): 492.963\n",
      "Average training loss: 0.0008. Mean distance error (km): 506.223\n",
      "Average training loss: 0.0008. Mean distance error (km): 505.855\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 89%|██████████████████████████████████████████████████████████████████████▋        | 17/19 [1:23:29<10:02, 301.03s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0007. Mean distance error (km): 496.310\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 95%|██████████████████████████████████████████████████████████████████████████▊    | 18/19 [1:28:46<05:05, 305.98s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0006. Mean distance error (km): 501.225\n",
      "Average training loss: 0.0008. Mean distance error (km): 496.275\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████| 19/19 [1:31:13<00:00, 288.09s/it]\n",
      "Evaluation:   0%|                                                                              | 0/130 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0007. Mean distance error (km): 482.878\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluation:  50%|██████████████████████████████████▌                                  | 65/130 [01:20<01:14,  1.14s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 65. Average evaluation loss: 0.0182. Mean distance error (km): 723.815\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluation: 100%|████████████████████████████████████████████████████████████████████| 130/130 [02:33<00:00,  1.18s/it]\n",
      "  0%|                                                                                           | 0/19 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 130. Average evaluation loss: 0.0183. Mean distance error (km): 726.390\n",
      "\n",
      "Epoch [1/8]: Train distance error (km): 489.576. Train loss: 0.0008. Evaluation distance error (km): 725.102. Evaluation loss: 0.0182\n",
      "\n",
      "Start epoch [2/8]\n",
      "Current learning rate: 0.00490\n",
      "Average training loss: 0.0007. Mean distance error (km): 482.008\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  5%|████▏                                                                           | 1/19 [05:02<1:30:49, 302.76s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0007. Mean distance error (km): 478.413\n",
      "Average training loss: 0.0007. Mean distance error (km): 475.081\n",
      "Average training loss: 0.0005. Mean distance error (km): 473.678\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 11%|████████▍                                                                       | 2/19 [09:59<1:25:18, 301.09s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0006. Mean distance error (km): 478.480\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 16%|████████████▋                                                                   | 3/19 [14:59<1:20:10, 300.67s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0007. Mean distance error (km): 469.121\n",
      "Average training loss: 0.0007. Mean distance error (km): 470.407\n",
      "Average training loss: 0.0007. Mean distance error (km): 478.342\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 21%|████████████████▊                                                               | 4/19 [19:58<1:15:02, 300.18s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0006. Mean distance error (km): 465.003\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 26%|█████████████████████                                                           | 5/19 [24:43<1:08:58, 295.63s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0005. Mean distance error (km): 472.215\n",
      "Average training loss: 0.0006. Mean distance error (km): 467.212\n",
      "Average training loss: 0.0005. Mean distance error (km): 466.108\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 32%|█████████████████████████▎                                                      | 6/19 [29:29<1:03:23, 292.57s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0006. Mean distance error (km): 464.738\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 37%|██████████████████████████████▏                                                   | 7/19 [34:43<59:48, 299.00s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0005. Mean distance error (km): 467.135\n",
      "Average training loss: 0.0005. Mean distance error (km): 460.887\n",
      "Average training loss: 0.0005. Mean distance error (km): 460.367\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 42%|██████████████████████████████████▌                                               | 8/19 [40:02<55:55, 305.08s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0005. Mean distance error (km): 461.228\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 47%|██████████████████████████████████████▊                                           | 9/19 [45:14<51:11, 307.15s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0005. Mean distance error (km): 459.622\n",
      "Average training loss: 0.0005. Mean distance error (km): 463.985\n",
      "Average training loss: 0.0005. Mean distance error (km): 465.190\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 53%|██████████████████████████████████████████▋                                      | 10/19 [50:49<47:19, 315.45s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0005. Mean distance error (km): 460.087\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 58%|██████████████████████████████████████████████▉                                  | 11/19 [56:26<42:55, 321.89s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0004. Mean distance error (km): 458.620\n",
      "Average training loss: 0.0004. Mean distance error (km): 460.138\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 63%|█████████████████████████████████████████████████▉                             | 12/19 [1:01:28<36:51, 315.96s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0004. Mean distance error (km): 458.175\n",
      "Average training loss: 0.0004. Mean distance error (km): 456.381\n",
      "Average training loss: 0.0004. Mean distance error (km): 451.279\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 68%|██████████████████████████████████████████████████████                         | 13/19 [1:06:26<31:03, 310.63s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0007. Mean distance error (km): 464.890\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 74%|██████████████████████████████████████████████████████████▏                    | 14/19 [1:11:35<25:50, 310.11s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0006. Mean distance error (km): 451.709\n",
      "Average training loss: 0.0007. Mean distance error (km): 455.052\n",
      "Average training loss: 0.0006. Mean distance error (km): 460.033\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 79%|██████████████████████████████████████████████████████████████▎                | 15/19 [1:16:44<20:39, 309.97s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0006. Mean distance error (km): 447.939\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 84%|██████████████████████████████████████████████████████████████████▌            | 16/19 [1:21:58<15:33, 311.16s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0005. Mean distance error (km): 452.357\n",
      "Average training loss: 0.0005. Mean distance error (km): 445.118\n",
      "Average training loss: 0.0005. Mean distance error (km): 443.197\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 89%|██████████████████████████████████████████████████████████████████████▋        | 17/19 [1:27:11<10:23, 311.64s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0005. Mean distance error (km): 442.439\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 95%|██████████████████████████████████████████████████████████████████████████▊    | 18/19 [1:32:23<05:11, 311.67s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0005. Mean distance error (km): 447.608\n",
      "Average training loss: 0.0004. Mean distance error (km): 450.952\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████| 19/19 [1:34:47<00:00, 299.36s/it]\n",
      "Evaluation:   0%|                                                                              | 0/130 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0004. Mean distance error (km): 449.193\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluation:  50%|██████████████████████████████████▌                                  | 65/130 [01:12<01:15,  1.16s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 65. Average evaluation loss: 0.0061. Mean distance error (km): 738.533\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluation: 100%|████████████████████████████████████████████████████████████████████| 130/130 [02:26<00:00,  1.13s/it]\n",
      "  0%|                                                                                           | 0/19 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 130. Average evaluation loss: 0.0061. Mean distance error (km): 737.688\n",
      "\n",
      "Epoch [2/8]: Train distance error (km): 450.072. Train loss: 0.0004. Evaluation distance error (km): 738.111. Evaluation loss: 0.0061\n",
      "\n",
      "Start epoch [3/8]\n",
      "Current learning rate: 0.00343\n",
      "Average training loss: 0.0004. Mean distance error (km): 445.400\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  5%|████▏                                                                           | 1/19 [05:01<1:30:26, 301.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0004. Mean distance error (km): 454.910\n",
      "Average training loss: 0.0004. Mean distance error (km): 454.661\n",
      "Average training loss: 0.0003. Mean distance error (km): 448.194\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 11%|████████▍                                                                       | 2/19 [10:43<1:28:53, 313.74s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0003. Mean distance error (km): 447.482\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 16%|████████████▋                                                                   | 3/19 [16:31<1:26:22, 323.92s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0003. Mean distance error (km): 440.622\n",
      "Average training loss: 0.0003. Mean distance error (km): 435.640\n",
      "Average training loss: 0.0003. Mean distance error (km): 435.459\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 21%|████████████████▊                                                               | 4/19 [21:53<1:20:50, 323.36s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0003. Mean distance error (km): 437.390\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 26%|█████████████████████                                                           | 5/19 [27:05<1:14:38, 319.92s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0003. Mean distance error (km): 432.056\n",
      "Average training loss: 0.0003. Mean distance error (km): 433.175\n",
      "Average training loss: 0.0003. Mean distance error (km): 429.800\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 32%|█████████████████████████▎                                                      | 6/19 [32:55<1:11:14, 328.84s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0003. Mean distance error (km): 425.984\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 37%|█████████████████████████████▍                                                  | 7/19 [38:14<1:05:12, 326.02s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0003. Mean distance error (km): 420.961\n",
      "Average training loss: 0.0003. Mean distance error (km): 420.993\n",
      "Average training loss: 0.0003. Mean distance error (km): 419.188\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 42%|██████████████████████████████████▌                                               | 8/19 [43:27<59:02, 322.06s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0003. Mean distance error (km): 426.652\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 47%|██████████████████████████████████████▊                                           | 9/19 [48:57<54:05, 324.57s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0003. Mean distance error (km): 416.627\n",
      "Average training loss: 0.0003. Mean distance error (km): 421.161\n",
      "Average training loss: 0.0003. Mean distance error (km): 414.333\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 53%|██████████████████████████████████████████▋                                      | 10/19 [54:12<48:14, 321.60s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0003. Mean distance error (km): 417.749\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 58%|██████████████████████████████████████████████▉                                  | 11/19 [59:25<42:31, 318.99s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0003. Mean distance error (km): 411.957\n",
      "Average training loss: 0.0003. Mean distance error (km): 411.826\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 63%|█████████████████████████████████████████████████▉                             | 12/19 [1:04:28<36:38, 314.12s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0003. Mean distance error (km): 412.339\n",
      "Average training loss: 0.0003. Mean distance error (km): 408.730\n",
      "Average training loss: 0.0003. Mean distance error (km): 408.301\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 68%|██████████████████████████████████████████████████████                         | 13/19 [1:09:28<31:00, 310.05s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0003. Mean distance error (km): 409.824\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 74%|██████████████████████████████████████████████████████████▏                    | 14/19 [1:14:28<25:34, 306.93s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0003. Mean distance error (km): 407.228\n",
      "Average training loss: 0.0003. Mean distance error (km): 407.507\n",
      "Average training loss: 0.0003. Mean distance error (km): 405.422\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 79%|██████████████████████████████████████████████████████████████▎                | 15/19 [1:19:40<20:34, 308.57s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0003. Mean distance error (km): 412.966\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 84%|██████████████████████████████████████████████████████████████████▌            | 16/19 [1:24:44<15:21, 307.14s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0003. Mean distance error (km): 406.693\n",
      "Average training loss: 0.0003. Mean distance error (km): 409.699\n",
      "Average training loss: 0.0003. Mean distance error (km): 402.010\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 89%|██████████████████████████████████████████████████████████████████████▋        | 17/19 [1:29:52<10:14, 307.34s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0003. Mean distance error (km): 403.465\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 95%|██████████████████████████████████████████████████████████████████████████▊    | 18/19 [1:34:58<05:07, 307.07s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0003. Mean distance error (km): 401.325\n",
      "Average training loss: 0.0003. Mean distance error (km): 396.461\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████| 19/19 [1:37:20<00:00, 307.37s/it]\n",
      "Evaluation:   0%|                                                                              | 0/130 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0003. Mean distance error (km): 399.310\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluation:  50%|██████████████████████████████████▌                                  | 65/130 [01:14<01:16,  1.18s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 65. Average evaluation loss: 0.0017. Mean distance error (km): 725.490\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluation: 100%|████████████████████████████████████████████████████████████████████| 130/130 [02:28<00:00,  1.14s/it]\n",
      "  0%|                                                                                           | 0/19 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 130. Average evaluation loss: 0.0017. Mean distance error (km): 727.854\n",
      "\n",
      "Epoch [3/8]: Train distance error (km): 397.886. Train loss: 0.0003. Evaluation distance error (km): 726.672. Evaluation loss: 0.0017\n",
      "\n",
      "Start epoch [4/8]\n",
      "Current learning rate: 0.00240\n",
      "Average training loss: 0.0002. Mean distance error (km): 396.306\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  5%|████▏                                                                           | 1/19 [05:01<1:30:22, 301.27s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 394.759\n",
      "Average training loss: 0.0003. Mean distance error (km): 394.784\n",
      "Average training loss: 0.0002. Mean distance error (km): 392.998\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 11%|████████▍                                                                       | 2/19 [10:17<1:26:37, 305.71s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0003. Mean distance error (km): 395.387\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 16%|████████████▋                                                                   | 3/19 [15:15<1:20:53, 303.37s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 393.788\n",
      "Average training loss: 0.0003. Mean distance error (km): 392.090\n",
      "Average training loss: 0.0002. Mean distance error (km): 391.349\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 21%|████████████████▊                                                               | 4/19 [20:02<1:14:36, 298.45s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 391.356\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 26%|█████████████████████                                                           | 5/19 [24:50<1:08:54, 295.32s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 391.105\n",
      "Average training loss: 0.0002. Mean distance error (km): 387.065\n",
      "Average training loss: 0.0002. Mean distance error (km): 388.017\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 32%|█████████████████████████▎                                                      | 6/19 [29:35<1:03:21, 292.40s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 391.787\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 37%|██████████████████████████████▏                                                   | 7/19 [34:25<58:18, 291.51s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 386.867\n",
      "Average training loss: 0.0002. Mean distance error (km): 388.706\n",
      "Average training loss: 0.0002. Mean distance error (km): 387.457\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 42%|██████████████████████████████████▌                                               | 8/19 [39:11<53:09, 289.98s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 385.153\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 47%|██████████████████████████████████████▊                                           | 9/19 [43:56<48:04, 288.44s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 385.876\n",
      "Average training loss: 0.0002. Mean distance error (km): 385.599\n",
      "Average training loss: 0.0003. Mean distance error (km): 388.350\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 53%|██████████████████████████████████████████▋                                      | 10/19 [48:41<43:06, 287.42s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 386.991\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 58%|██████████████████████████████████████████████▉                                  | 11/19 [53:26<38:13, 286.66s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 384.654\n",
      "Average training loss: 0.0002. Mean distance error (km): 384.748\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 63%|███████████████████████████████████████████████████▏                             | 12/19 [58:15<33:31, 287.42s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 386.874\n",
      "Average training loss: 0.0002. Mean distance error (km): 384.640\n",
      "Average training loss: 0.0003. Mean distance error (km): 386.349\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 68%|██████████████████████████████████████████████████████                         | 13/19 [1:03:02<28:44, 287.36s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 384.821\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 74%|██████████████████████████████████████████████████████████▏                    | 14/19 [1:07:49<23:56, 287.22s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 385.151\n",
      "Average training loss: 0.0002. Mean distance error (km): 387.294\n",
      "Average training loss: 0.0002. Mean distance error (km): 383.614\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 79%|██████████████████████████████████████████████████████████████▎                | 15/19 [1:12:52<19:27, 291.85s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 384.915\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 84%|██████████████████████████████████████████████████████████████████▌            | 16/19 [1:18:02<14:52, 297.45s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 381.610\n",
      "Average training loss: 0.0002. Mean distance error (km): 381.600\n",
      "Average training loss: 0.0002. Mean distance error (km): 381.342\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 89%|██████████████████████████████████████████████████████████████████████▋        | 17/19 [1:23:17<10:05, 302.58s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 381.249\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 95%|██████████████████████████████████████████████████████████████████████████▊    | 18/19 [1:28:41<05:08, 308.88s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 380.077\n",
      "Average training loss: 0.0002. Mean distance error (km): 380.114\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████| 19/19 [1:31:08<00:00, 287.83s/it]\n",
      "Evaluation:   0%|                                                                              | 0/130 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 380.157\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluation:  50%|██████████████████████████████████▌                                  | 65/130 [01:11<01:13,  1.14s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 65. Average evaluation loss: 0.0017. Mean distance error (km): 719.824\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluation: 100%|████████████████████████████████████████████████████████████████████| 130/130 [02:25<00:00,  1.12s/it]\n",
      "  0%|                                                                                           | 0/19 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 130. Average evaluation loss: 0.0016. Mean distance error (km): 720.913\n",
      "\n",
      "Epoch [4/8]: Train distance error (km): 380.135. Train loss: 0.0002. Evaluation distance error (km): 720.369. Evaluation loss: 0.0016\n",
      "\n",
      "Start epoch [5/8]\n",
      "Current learning rate: 0.00168\n",
      "Average training loss: 0.0002. Mean distance error (km): 377.869\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  5%|████▏                                                                           | 1/19 [05:15<1:34:46, 315.91s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 377.854\n",
      "Average training loss: 0.0002. Mean distance error (km): 378.337\n",
      "Average training loss: 0.0002. Mean distance error (km): 376.622\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 11%|████████▍                                                                       | 2/19 [10:40<1:30:14, 318.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 377.527\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 16%|████████████▋                                                                   | 3/19 [16:18<1:26:28, 324.29s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 375.485\n",
      "Average training loss: 0.0002. Mean distance error (km): 378.764\n",
      "Average training loss: 0.0002. Mean distance error (km): 376.911\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 21%|████████████████▊                                                               | 4/19 [21:41<1:21:01, 324.12s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 376.021\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 26%|█████████████████████                                                           | 5/19 [27:43<1:18:15, 335.40s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 375.472\n",
      "Average training loss: 0.0002. Mean distance error (km): 373.145\n",
      "Average training loss: 0.0002. Mean distance error (km): 376.033\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 32%|█████████████████████████▎                                                      | 6/19 [33:01<1:11:30, 330.03s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 374.644\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 37%|█████████████████████████████▍                                                  | 7/19 [38:05<1:04:29, 322.44s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 376.727\n",
      "Average training loss: 0.0002. Mean distance error (km): 372.139\n",
      "Average training loss: 0.0002. Mean distance error (km): 373.771\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 42%|██████████████████████████████████▌                                               | 8/19 [43:06<57:54, 315.87s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 371.023\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 47%|██████████████████████████████████████▊                                           | 9/19 [48:11<52:06, 312.62s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 373.317\n",
      "Average training loss: 0.0002. Mean distance error (km): 370.120\n",
      "Average training loss: 0.0002. Mean distance error (km): 370.596\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 53%|██████████████████████████████████████████▋                                      | 10/19 [53:13<46:25, 309.52s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 369.158\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 58%|██████████████████████████████████████████████▉                                  | 11/19 [58:18<41:04, 308.08s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 371.292\n",
      "Average training loss: 0.0002. Mean distance error (km): 369.804\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 63%|█████████████████████████████████████████████████▉                             | 12/19 [1:03:22<35:48, 306.99s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 369.238\n",
      "Average training loss: 0.0002. Mean distance error (km): 369.001\n",
      "Average training loss: 0.0002. Mean distance error (km): 369.126\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 68%|██████████████████████████████████████████████████████                         | 13/19 [1:08:27<30:37, 306.17s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 370.610\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 74%|██████████████████████████████████████████████████████████▏                    | 14/19 [1:13:32<25:29, 305.94s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 367.977\n",
      "Average training loss: 0.0002. Mean distance error (km): 368.366\n",
      "Average training loss: 0.0002. Mean distance error (km): 368.241\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 79%|██████████████████████████████████████████████████████████████▎                | 15/19 [1:18:44<20:30, 307.70s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 369.546\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 84%|██████████████████████████████████████████████████████████████████▌            | 16/19 [1:24:07<15:37, 312.36s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 368.142\n",
      "Average training loss: 0.0002. Mean distance error (km): 368.342\n",
      "Average training loss: 0.0002. Mean distance error (km): 368.730\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 89%|██████████████████████████████████████████████████████████████████████▋        | 17/19 [1:29:39<10:36, 318.12s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 367.228\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 95%|██████████████████████████████████████████████████████████████████████████▊    | 18/19 [1:35:08<05:21, 321.57s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 370.405\n",
      "Average training loss: 0.0002. Mean distance error (km): 369.468\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████| 19/19 [1:37:45<00:00, 308.69s/it]\n",
      "Evaluation:   0%|                                                                              | 0/130 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 371.754\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluation:  50%|██████████████████████████████████▌                                  | 65/130 [01:13<01:11,  1.10s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 65. Average evaluation loss: 0.0020. Mean distance error (km): 734.003\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluation: 100%|████████████████████████████████████████████████████████████████████| 130/130 [02:22<00:00,  1.10s/it]\n",
      "  0%|                                                                                           | 0/19 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 130. Average evaluation loss: 0.0020. Mean distance error (km): 737.820\n",
      "\n",
      "Epoch [5/8]: Train distance error (km): 370.611. Train loss: 0.0002. Evaluation distance error (km): 735.911. Evaluation loss: 0.0020\n",
      "\n",
      "Start epoch [6/8]\n",
      "Current learning rate: 0.00118\n",
      "Average training loss: 0.0002. Mean distance error (km): 370.495\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  5%|████▏                                                                           | 1/19 [04:56<1:29:03, 296.85s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 370.698\n",
      "Average training loss: 0.0002. Mean distance error (km): 374.244\n",
      "Average training loss: 0.0002. Mean distance error (km): 370.826\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 11%|████████▍                                                                       | 2/19 [09:48<1:23:39, 295.26s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 369.047\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 16%|████████████▋                                                                   | 3/19 [14:43<1:18:42, 295.18s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 366.832\n",
      "Average training loss: 0.0002. Mean distance error (km): 367.196\n",
      "Average training loss: 0.0002. Mean distance error (km): 364.290\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 21%|████████████████▊                                                               | 4/19 [19:37<1:13:42, 294.85s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 365.907\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 26%|█████████████████████                                                           | 5/19 [24:32<1:08:49, 294.99s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 364.040\n",
      "Average training loss: 0.0002. Mean distance error (km): 362.168\n",
      "Average training loss: 0.0002. Mean distance error (km): 361.026\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 32%|█████████████████████████▎                                                      | 6/19 [29:25<1:03:47, 294.40s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 361.941\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 37%|██████████████████████████████▏                                                   | 7/19 [34:21<58:57, 294.76s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 361.842\n",
      "Average training loss: 0.0002. Mean distance error (km): 361.064\n",
      "Average training loss: 0.0002. Mean distance error (km): 360.200\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 42%|██████████████████████████████████▌                                               | 8/19 [39:14<53:56, 294.20s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 361.331\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 47%|██████████████████████████████████████▊                                           | 9/19 [44:09<49:04, 294.45s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 360.797\n",
      "Average training loss: 0.0002. Mean distance error (km): 361.442\n",
      "Average training loss: 0.0002. Mean distance error (km): 361.971\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 53%|██████████████████████████████████████████▋                                      | 10/19 [49:01<44:04, 293.87s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 363.197\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 58%|██████████████████████████████████████████████▉                                  | 11/19 [53:56<39:14, 294.25s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 361.146\n",
      "Average training loss: 0.0002. Mean distance error (km): 360.879\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 63%|███████████████████████████████████████████████████▏                             | 12/19 [58:58<34:35, 296.56s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 363.498\n",
      "Average training loss: 0.0002. Mean distance error (km): 362.157\n",
      "Average training loss: 0.0002. Mean distance error (km): 363.665\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 68%|██████████████████████████████████████████████████████                         | 13/19 [1:03:56<29:40, 296.72s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 364.540\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 74%|██████████████████████████████████████████████████████████▏                    | 14/19 [1:08:58<24:52, 298.59s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 362.295\n",
      "Average training loss: 0.0002. Mean distance error (km): 366.538\n",
      "Average training loss: 0.0002. Mean distance error (km): 365.898\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 79%|██████████████████████████████████████████████████████████████▎                | 15/19 [1:14:00<19:58, 299.50s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0003. Mean distance error (km): 369.315\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 84%|██████████████████████████████████████████████████████████████████▌            | 16/19 [1:19:04<15:02, 300.74s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0003. Mean distance error (km): 367.862\n",
      "Average training loss: 0.0003. Mean distance error (km): 365.320\n",
      "Average training loss: 0.0002. Mean distance error (km): 365.524\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 89%|██████████████████████████████████████████████████████████████████████▋        | 17/19 [1:24:03<10:00, 300.21s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 362.939\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 95%|██████████████████████████████████████████████████████████████████████████▊    | 18/19 [1:29:07<05:01, 301.55s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 360.055\n",
      "Average training loss: 0.0002. Mean distance error (km): 358.019\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████| 19/19 [1:31:28<00:00, 288.87s/it]\n",
      "Evaluation:   0%|                                                                              | 0/130 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 358.130\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluation:  50%|██████████████████████████████████▌                                  | 65/130 [01:11<01:11,  1.11s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 65. Average evaluation loss: 0.0017. Mean distance error (km): 751.615\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluation: 100%|████████████████████████████████████████████████████████████████████| 130/130 [02:20<00:00,  1.08s/it]\n",
      "  0%|                                                                                           | 0/19 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 130. Average evaluation loss: 0.0018. Mean distance error (km): 752.035\n",
      "\n",
      "Epoch [6/8]: Train distance error (km): 358.074. Train loss: 0.0002. Evaluation distance error (km): 751.825. Evaluation loss: 0.0017\n",
      "\n",
      "Start epoch [7/8]\n",
      "Current learning rate: 0.00082\n",
      "Average training loss: 0.0002. Mean distance error (km): 359.351\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  5%|████▏                                                                           | 1/19 [05:04<1:31:24, 304.70s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 358.577\n",
      "Average training loss: 0.0002. Mean distance error (km): 358.619\n",
      "Average training loss: 0.0002. Mean distance error (km): 358.875\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 11%|████████▍                                                                       | 2/19 [10:01<1:25:38, 302.27s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 357.425\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 16%|████████████▋                                                                   | 3/19 [15:02<1:20:32, 302.02s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 359.712\n",
      "Average training loss: 0.0002. Mean distance error (km): 356.606\n",
      "Average training loss: 0.0002. Mean distance error (km): 357.982\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 21%|████████████████▊                                                               | 4/19 [20:01<1:15:17, 301.14s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 357.489\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 26%|█████████████████████                                                           | 5/19 [25:04<1:10:20, 301.47s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 357.003\n",
      "Average training loss: 0.0002. Mean distance error (km): 355.985\n",
      "Average training loss: 0.0002. Mean distance error (km): 355.449\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 32%|█████████████████████████▎                                                      | 6/19 [30:00<1:04:58, 299.87s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 357.265\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 37%|█████████████████████████████▍                                                  | 7/19 [35:01<1:00:05, 300.44s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 356.900\n",
      "Average training loss: 0.0002. Mean distance error (km): 356.917\n",
      "Average training loss: 0.0002. Mean distance error (km): 355.357\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 42%|██████████████████████████████████▌                                               | 8/19 [39:58<54:53, 299.36s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 355.449\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 47%|██████████████████████████████████████▊                                           | 9/19 [45:01<50:04, 300.48s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 355.123\n",
      "Average training loss: 0.0002. Mean distance error (km): 355.538\n",
      "Average training loss: 0.0002. Mean distance error (km): 356.113\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 53%|██████████████████████████████████████████▋                                      | 10/19 [50:02<45:03, 300.44s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 354.152\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 58%|██████████████████████████████████████████████▉                                  | 11/19 [55:02<40:03, 300.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 353.395\n",
      "Average training loss: 0.0002. Mean distance error (km): 353.453\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 63%|█████████████████████████████████████████████████▉                             | 12/19 [1:00:05<35:07, 301.02s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 356.844\n",
      "Average training loss: 0.0002. Mean distance error (km): 379.112\n",
      "Average training loss: 0.0002. Mean distance error (km): 369.973\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 68%|██████████████████████████████████████████████████████                         | 13/19 [1:05:10<30:13, 302.21s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 368.528\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 74%|██████████████████████████████████████████████████████████▏                    | 14/19 [1:10:15<25:15, 303.08s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 367.393\n",
      "Average training loss: 0.0002. Mean distance error (km): 376.139\n",
      "Average training loss: 0.0002. Mean distance error (km): 378.372\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 79%|██████████████████████████████████████████████████████████████▎                | 15/19 [1:15:16<20:09, 302.45s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 375.884\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 84%|██████████████████████████████████████████████████████████████████▌            | 16/19 [1:20:21<15:09, 303.26s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 375.696\n",
      "Average training loss: 0.0002. Mean distance error (km): 373.875\n",
      "Average training loss: 0.0002. Mean distance error (km): 372.102\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 89%|██████████████████████████████████████████████████████████████████████▋        | 17/19 [1:25:21<10:04, 302.42s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 371.478\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 95%|██████████████████████████████████████████████████████████████████████████▊    | 18/19 [1:30:21<05:01, 301.55s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 371.065\n",
      "Average training loss: 0.0002. Mean distance error (km): 370.829\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████| 19/19 [1:32:43<00:00, 292.80s/it]\n",
      "Evaluation:   0%|                                                                              | 0/130 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 370.469\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluation:  50%|██████████████████████████████████▌                                  | 65/130 [01:11<01:14,  1.15s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 65. Average evaluation loss: 0.0052. Mean distance error (km): 762.178\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluation: 100%|████████████████████████████████████████████████████████████████████| 130/130 [02:22<00:00,  1.09s/it]\n",
      "  0%|                                                                                           | 0/19 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 130. Average evaluation loss: 0.0053. Mean distance error (km): 765.883\n",
      "\n",
      "Epoch [7/8]: Train distance error (km): 370.649. Train loss: 0.0002. Evaluation distance error (km): 764.030. Evaluation loss: 0.0052\n",
      "\n",
      "Start epoch [8/8]\n",
      "Current learning rate: 0.00058\n",
      "Average training loss: 0.0002. Mean distance error (km): 368.687\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  5%|████▏                                                                           | 1/19 [05:05<1:31:34, 305.27s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 366.801\n",
      "Average training loss: 0.0002. Mean distance error (km): 366.510\n",
      "Average training loss: 0.0002. Mean distance error (km): 370.916\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 11%|████████▍                                                                       | 2/19 [10:04<1:25:57, 303.40s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 375.523\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 16%|████████████▋                                                                   | 3/19 [15:08<1:20:59, 303.73s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 373.347\n",
      "Average training loss: 0.0002. Mean distance error (km): 371.628\n",
      "Average training loss: 0.0002. Mean distance error (km): 368.355\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 21%|████████████████▊                                                               | 4/19 [20:10<1:15:48, 303.25s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 366.258\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 26%|█████████████████████                                                           | 5/19 [25:17<1:10:59, 304.27s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 365.037\n",
      "Average training loss: 0.0002. Mean distance error (km): 364.162\n",
      "Average training loss: 0.0002. Mean distance error (km): 365.888\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 32%|█████████████████████████▎                                                      | 6/19 [30:20<1:05:50, 303.87s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 364.611\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 37%|█████████████████████████████▍                                                  | 7/19 [35:25<1:00:51, 304.26s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 364.545\n",
      "Average training loss: 0.0002. Mean distance error (km): 362.658\n",
      "Average training loss: 0.0002. Mean distance error (km): 361.987\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 42%|██████████████████████████████████▌                                               | 8/19 [40:29<55:44, 304.07s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 362.872\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 47%|██████████████████████████████████████▊                                           | 9/19 [45:35<50:48, 304.84s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 362.443\n",
      "Average training loss: 0.0002. Mean distance error (km): 361.122\n",
      "Average training loss: 0.0002. Mean distance error (km): 360.746\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 53%|██████████████████████████████████████████▋                                      | 10/19 [50:37<45:34, 303.85s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 358.749\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 58%|██████████████████████████████████████████████▉                                  | 11/19 [55:40<40:28, 303.57s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 358.589\n",
      "Average training loss: 0.0002. Mean distance error (km): 357.586\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 63%|█████████████████████████████████████████████████▉                             | 12/19 [1:00:45<35:28, 304.09s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 357.746\n",
      "Average training loss: 0.0002. Mean distance error (km): 358.602\n",
      "Average training loss: 0.0002. Mean distance error (km): 357.878\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 68%|██████████████████████████████████████████████████████                         | 13/19 [1:05:47<30:19, 303.32s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 360.105\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 74%|██████████████████████████████████████████████████████████▏                    | 14/19 [1:10:48<25:13, 302.66s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 363.458\n",
      "Average training loss: 0.0002. Mean distance error (km): 361.960\n",
      "Average training loss: 0.0002. Mean distance error (km): 359.991\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 79%|██████████████████████████████████████████████████████████████▎                | 15/19 [1:15:50<20:10, 302.54s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 360.567\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 84%|██████████████████████████████████████████████████████████████████▌            | 16/19 [1:20:55<15:09, 303.21s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 358.908\n",
      "Average training loss: 0.0002. Mean distance error (km): 358.325\n",
      "Average training loss: 0.0002. Mean distance error (km): 358.308\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 89%|██████████████████████████████████████████████████████████████████████▋        | 17/19 [1:25:57<10:06, 303.02s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 355.598\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 95%|██████████████████████████████████████████████████████████████████████████▊    | 18/19 [1:31:05<05:04, 304.35s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 356.784\n",
      "Average training loss: 0.0002. Mean distance error (km): 357.429\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████| 19/19 [1:33:27<00:00, 295.14s/it]\n",
      "Evaluation:   0%|                                                                              | 0/130 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training loss: 0.0002. Mean distance error (km): 356.645\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluation:  50%|██████████████████████████████████▌                                  | 65/130 [01:11<01:14,  1.14s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 65. Average evaluation loss: 0.0034. Mean distance error (km): 706.518\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluation: 100%|████████████████████████████████████████████████████████████████████| 130/130 [02:23<00:00,  1.10s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 130. Average evaluation loss: 0.0035. Mean distance error (km): 710.760\n",
      "\n",
      "Epoch [8/8]: Train distance error (km): 357.037. Train loss: 0.0002. Evaluation distance error (km): 708.639. Evaluation loss: 0.0034\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# Initialize parameters\n",
    "input_dim = eval_set.n_features\n",
    "output_dim = 3\n",
    "n_d = 32\n",
    "n_a = 32\n",
    "gamma = 1.2\n",
    "epsilon = 1e-10\n",
    "n_shared = 3\n",
    "n_independent = 3\n",
    "n_steps = 3\n",
    "n_chunks = 64\n",
    "chunk_size = 0\n",
    "track_running_stats = False\n",
    "momentum = 0.02\n",
    "learning_rate = 0.007\n",
    "epochs = 8\n",
    "batch_size = 6144\n",
    "feat_cols = None\n",
    "ghost_batch_norm = True\n",
    "\n",
    "train_iterator = MariaDB_chunk_indices(cursor, mariadb_table_name, 1024 * 1000)\n",
    "\n",
    "eval_loader = DataLoader(eval_set, batch_size=batch_size, shuffle=False, drop_last=True)\n",
    "# test_loader = DataLoader(test_set, batch_size=batch_size, shuffle=False)\n",
    "\n",
    "# Check whether system supports CUDA\n",
    "CUDA = torch.cuda.is_available()\n",
    "\n",
    "model = TabNet(input_dim, output_dim, n_d, n_a, gamma=gamma, epsilon=epsilon, n_shared=n_shared, n_independent=n_independent,\n",
    "               n_steps=n_steps, n_chunks=n_chunks, chunk_size=chunk_size, track_running_stats=track_running_stats,\n",
    "               momentum=momentum, ghost_batch_norm=ghost_batch_norm)\n",
    "\n",
    "# Move the model to GPU if possible\n",
    "# if CUDA:\n",
    "#     model.cuda()\n",
    "\n",
    "model.add_loss_fn(nn.MSELoss())\n",
    "\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n",
    "model.add_optimizer(optimizer)\n",
    "\n",
    "scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=1, gamma=0.7)\n",
    "\n",
    "# device = torch.device('cuda' if CUDA else 'cpu')\n",
    "device = torch.device('cpu')\n",
    "\n",
    "model.add_device(device)\n",
    "\n",
    "train_losses_list, train_avg_loss_list, train_dist_list = [], [], []\n",
    "eval_avg_loss_list, eval_dist_list = [], []\n",
    "train_mask_list, train_mask_explain_list = [], []\n",
    "\n",
    "for epoch in range(epochs):\n",
    "\n",
    "    print('\\nStart epoch [{}/{}]'.format(epoch+1, epochs))\n",
    "    print('Current learning rate: {:.5f}'.format(scheduler.get_last_lr()[0]))\n",
    "    \n",
    "    for indices in tqdm(train_iterator):\n",
    "        \n",
    "        train_loader = DataLoader(MariaDB_dataset(indices, cursor, mariadb_table_name, norm_params, feat_cols=feat_cols),\n",
    "                                  batch_size=batch_size, shuffle=True, drop_last=True)\n",
    "        \n",
    "        train_losses, train_avg_loss, train_dist, mask_list, mask_explain_list = train_model(model, train_loader)\n",
    "\n",
    "    train_losses_list.append(train_losses)\n",
    "    train_avg_loss_list.append(train_avg_loss)\n",
    "    train_dist_list.append(train_dist)\n",
    "    train_mask_list.append(mask_list)\n",
    "    train_mask_explain_list.append(mask_explain_list)\n",
    "\n",
    "    _, eval_avg_loss, eval_dist = evaluate_model(model, eval_loader)\n",
    "\n",
    "    eval_avg_loss_list.append(eval_avg_loss)\n",
    "    eval_dist_list.append(eval_dist)\n",
    "\n",
    "    print('\\nEpoch [{}/{}]: Train distance error (km): {:.3f}. Train loss: {:.4f}. Evaluation distance error (km): {:.3f}. Evaluation loss: {:.4f}'\\\n",
    "          .format(epoch+1, epochs, train_dist, train_avg_loss, eval_dist, eval_avg_loss))\n",
    "    \n",
    "    scheduler.step()\n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the training is completed, we can plot the training and the validation learning curves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAncAAAGGCAYAAAAZ7KdDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAA/IklEQVR4nO3deXRU9f3/8eebhE1AcAkqiwUsisgSbEQqirhUQVHUqoBaRa1K1dqv1hZpTxW/319b29raaqkWV6wLUpeKFYtii6jVSlBEEUQ2JUIRQRAECUnevz/uTRyGSTITkrmzvB7nzMnMXd93woEXn7u8zd0RERERkdzQLOoCRERERKTxKNyJiIiI5BCFOxEREZEconAnIiIikkMU7kRERERyiMKdiIiISA5RuBORlJiZm9nXo66jKZjZUDMrS3LZiWb20O5up7GZ2V1m9rMo9i0imUHhTiRPmdmWmFeVmW2L+Xx+A7c5Ngx/P4qbXmZmQ5NYv1u4fmFD9i/g7uPc/f+irkNEoqNwJ5Kn3L1t9Qv4CDgtZtrDu7HpDcB4M9uzcSqVarkQenPhGEQyncKdiOzEzAaa2WtmttHM1pjZH82sRdxip5jZcjP71Mx+Y2axf5csAl4Drq1l+83M7AYzW2Zm681smpntHc6eE/7cGI4gfjPB+hPN7K9m9pCZbTazd8zsYDObYGafmNkqMzspZvlOZjbdzDaY2VIzuyxmXmsze8DMPjOz94Aj4vbVycyeMLN1ZrbCzK5J+otMcjv1fd/hSOZVZvYB8EH1KV8z+2F4vGvM7OKY5R8ws/8Xvq9v2X3M7Bkz+9zM5prZ/zOzV+o4jqPN7N9hravMbGw4fbaZfTdmubGx20lwDHeZ2a1x237azK5L8vsqDWtea2a/S/HXIZLzFO5EJF4lQTDbF/gmcAJwZdwyZwIlwOHASOCSuPk/A66NCW2xrgHOAI4FOgGfAZPCeUPCnx3CEcTXaqnxNOAvwF7AW8BMgr/POgP/C/w5ZtlHgbJwX2cDvzCzE8J5NwEHha+TgYuqVwoD6zPA2+F2TwD+x8xOrqWmhJLYTjLf9xnAkUDv8PP+QPtwe5cCk8xsr1pKqGvZScAX4TIXxR5/guM4EHgOuAMoAoqB+XUcerzYY3gEGGVmFm57L+AkYGoS39cfgD+4+54Ev7dpKdQgkhcU7kRkJ+4+z91fd/cKd19JEJSOjVvsV+6+wd0/An4PjInbxnzgeWB8gl1cAfzU3cvcfTswETg7xdN1L7v7THevAP5KEDZucfcdwFSgm5l1MLOuwNHAeHf/MqzrHuA74XbOBX4eHssq4PaYfRwBFLn7/7p7ubsvB+4GRqdQZ73bSfL7/mVY47bw8w7gf919h7vPALYAh9Sy/4TLmlkB8G3gJnff6u7vAVPqOI7zgVnu/mi4rfXh95ms2GN4GXDgmHDe2cBr7r6a+r/3HcDXzWxfd9/i7q+nUINIXtC1DyKyEzM7GPgdwcjcHgR/T8yLW2xVzPsPCUbF4t0IvGFmt8VN/xrwlJlVxUyrBPZLocy1Me+3AZ+6e2XMZ4C2YV0b3H1zXL0l4ftO7HossXV2MrONMdMKCIJJKurcTgO+b4D1YbCttpXgeBOpbdmicF+x247fT6yuwLI65tenZtvu7mY2leA/BXOA84DqO4/r+94vJRidXWxmK4Cb3f3vu1GXSM7RyJ2IxLsTWAz0DE99/QSwuGW6xrw/EFgdvxF3Xww8Ga4faxUw3N07xLxaufvHBKM5jWk1sLeZtYur9+Pw/Rp2PZbYOlfE1dnO3U9JsYb6tpPM993Y3wvAOqAC6BIzrWsty0JwHAfVMu8LgmBabf8Ey8Qfw6MEI7ZfIzhd+0TMfmr9vtz9A3cfA3QEfgU8bmZt6qhbJO8o3IlIvHbA58AWM+sFfC/BMj8ys73C054/AB6rZVs3AxcDHWKm3QX8PPxHHTMrMrOR4bx1QBXQY7ePAghPtf4b+KWZtTKzfgQjP9V3A08DJoTH0gX4fszqbwCfm9n48MaLAjPrY2Y73XSRhPq2k8z33ejCkc4ngYlmtke47wvrWOVh4EQzO9fMCsObMYrDefOBs8LtfJ3gO65v/28R/L7vAWa6+8ZwVp3fl5ldYGZF7l4FVK9TiYjUULgTkXjXE5wm20xwrVOi4PY0wanD+cCzwL2JNuTuKwhufIgdWfkDMB143sw2A68TjNzg7luBnwOvhndkDmqE4xkDdCMYxXuK4BqzF8J5NxOcil1BcI3gX2JqryS4caM4nP8pQRBpn8rOk9hOMt93U7k6rOO/BMf+KLA90YLh9ZWnAD8keNzNfKB/OPs2oJzgdPkUvgrP9XkUOJHgBovq/dT3fQ0DFprZFoI/S6Pd/csk9yeSF8y9KUb7RUQk25jZr4D93b3Wu2ZFJPNp5E5EJE+ZWS8z62eBgQSnU5+Kui4R2T26W1ZEJH+1Izg12gn4BPgtwSl3EcliOi0rIiIikkN0WlZEREQkhyjciYiIiOQQXXMX2nfffb1bt25RlyEiIiJSr3nz5n3q7kWJ5inchbp160ZpaWnUZYiIiIjUy8w+rG2eTsuKiIiI5BCFOxEREZEconAnIiIikkN0zZ2IiEge2LFjB2VlZXz5pVrxZpNWrVrRpUsXmjdvnvQ6CnciIiJ5oKysjHbt2tGtWzfMLOpyJAnuzvr16ykrK6N79+5Jr6fTsiIiInngyy+/ZJ999lGwyyJmxj777JPyaKvCnYiISJ5QsMs+DfmdKdyJiIhIk1u/fj3FxcUUFxez//7707lz55rP5eXlda5bWlrKNddcU+8+jjrqqEapdfbs2YwYMaJRthUFXXMnIiIiTW6fffZh/vz5AEycOJG2bdty/fXX18yvqKigsDBxLCkpKaGkpKTeffz73/9ulFqznUbuREREJBJjx47luuuu47jjjmP8+PG88cYbHHXUUQwYMICjjjqK999/H9h5JG3ixIlccsklDB06lB49enD77bfXbK9t27Y1yw8dOpSzzz6bXr16cf755+PuAMyYMYNevXpx9NFHc80119Q7QrdhwwbOOOMM+vXrx6BBg1iwYAEAL730Us3I44ABA9i8eTNr1qxhyJAhFBcX06dPH15++eVG/86SoZE7ERGRPHPzMwt5b/XnjbrN3p325KbTDkt5vSVLljBr1iwKCgr4/PPPmTNnDoWFhcyaNYuf/OQnPPHEE7uss3jxYv71r3+xefNmDjnkEL73ve/t8qiQt956i4ULF9KpUycGDx7Mq6++SklJCVdccQVz5syhe/fujBkzpt76brrpJgYMGMDf/vY3/vnPf3LhhRcyf/58br31ViZNmsTgwYPZsmULrVq1YvLkyZx88sn89Kc/pbKykq1bt6b8fTQGhbt0qaqEFS/BvodA+85RVyMiIpIRzjnnHAoKCgDYtGkTF110ER988AFmxo4dOxKuc+qpp9KyZUtatmxJx44dWbt2LV26dNlpmYEDB9ZMKy4uZuXKlbRt25YePXrUPFZkzJgxTJ48uc76XnnllZqAefzxx7N+/Xo2bdrE4MGDue666zj//PM566yz6NKlC0cccQSXXHIJO3bs4IwzzqC4uHh3vpoGU7hLl83/hb+cBceOh+MmRF2NiIjksYaMsDWVNm3a1Lz/2c9+xnHHHcdTTz3FypUrGTp0aMJ1WrZsWfO+oKCAioqKpJapPjWbikTrmBk33HADp556KjNmzGDQoEHMmjWLIUOGMGfOHJ599lm+853v8KMf/YgLL7ww5X3uLl1zly7tO0P3Y2DBY9CAP1wiIiK5btOmTXTuHJzdeuCBBxp9+7169WL58uWsXLkSgMcee6zedYYMGcLDDz8MBNfy7bvvvuy5554sW7aMvn37Mn78eEpKSli8eDEffvghHTt25LLLLuPSSy/lzTffbPRjSIbCXTr1Gw2frYBVb0RdiYiISMb58Y9/zIQJExg8eDCVlZWNvv3WrVvzpz/9iWHDhnH00Uez33770b59+zrXmThxIqWlpfTr148bbriBKVOmAPD73/+ePn360L9/f1q3bs3w4cOZPXt2zQ0WTzzxBD/4wQ8a/RiSYQ0ZosxFJSUlXlpa2rQ72b4ZftMTisfAiNuadl8iIiIxFi1axKGHHhp1GZHbsmULbdu2xd256qqr6NmzJ9dee23UZdUp0e/OzOa5e8Lnw2jkLp1atoNep8K7T0LF9qirERERyTt33303xcXFHHbYYWzatIkrrrgi6pIanW6oSLf+Y+Ddx+GD5+HQ06KuRkREJK9ce+21GT9St7s0cpduPYZCm47w9tSoKxEREZEcpHCXbgWF0PccWDITtm6IuhoRERHJMQp3Ueg/Cqp2wMKnoq5EREREckzawp2ZDTOz981sqZndkGC+mdnt4fwFZnZ4zLz7zOwTM3s3bp3HzGx++FppZvPD6d3MbFvMvLua/ABTsX8/KDo0eOadiIiISCNKS7gzswJgEjAc6A2MMbPecYsNB3qGr8uBO2PmPQAMi9+uu49y92J3LwaeAJ6Mmb2sep67j2usY2kUZsHo3ar/wIblUVcjIiKSFgUFBRQXF9e8brnllgZtZ+jQoTT08WWzZ8/m3//+d83nu+66iwcffLBB24q1cuVK+vTps9vbaQzpult2ILDU3ZcDmNlUYCTwXswyI4EHPXjw3utm1sHMDnD3Ne4+x8y61bZxMzPgXOD4JjuCxtb3XJh1M7z9mNqRiYhIXmjdujXz58+PtIbZs2fTtm1bjjrqKADGjcus8Z/GkK7Tsp2BVTGfy8JpqS5Tm2OAte7+Qcy07mb2lpm9ZGbHJFrJzC43s1IzK123bl2Su2okakcmIiLCc889x7nnnlvzefbs2Zx2WvCosO9973uUlJRw2GGHcdNNNyVcv23btjXvH3/8ccaOHQvAM888w5FHHsmAAQM48cQTWbt2LStXruSuu+7itttuo7i4mJdffpmJEydy6623AjB//nwGDRpEv379OPPMM/nss8+AYKRw/PjxDBw4kIMPPpiXX365zmP68ssvufjii+nbty8DBgzgX//6FwALFy5k4MCBFBcX069fPz744AO++OILTj31VPr370+fPn2SaolWn3SN3FmCafGJJpllajMGeDTm8xrgQHdfb2bfAP5mZoe5++c7bdx9MjAZgg4VSe6r8fQbDU9fGbQjO/DItO9eRETy1HM3wH/fadxt7t8Xhtd9mnXbtm0UFxfXfJ4wYQLf/va3ueKKK/jiiy9o06YNjz32GKNGjQLg5z//OXvvvTeVlZWccMIJLFiwgH79+iVVztFHH83rr7+OmXHPPffw61//mt/+9reMGzeOtm3bcv311wPw4osv1qxz4YUXcscdd3Dsscdy4403cvPNN/P73/8egIqKCt544w1mzJjBzTffzKxZs2rd96RJkwB45513WLx4MSeddBJLlizhrrvu4gc/+AHnn38+5eXlVFZWMmPGDDp16sSzzz4LBP11d1e6Ru7KgK4xn7sAqxuwzC7MrBA4C6iJuu6+3d3Xh+/nAcuAgxtUeVPqfToUtoYFeuadiIjkvurTstWvUaNGUVhYyLBhw3jmmWeoqKjg2WefZeTIkQBMmzaNww8/nAEDBrBw4ULee++9evbwlbKyMk4++WT69u3Lb37zGxYuXFjn8ps2bWLjxo0ce+yxAFx00UXMmTOnZv5ZZ50FwDe+8Q1WrlxZ57ZeeeUVvvOd7wDQq1cvvva1r7FkyRK++c1v8otf/IJf/epXfPjhh7Ru3Zq+ffsya9Ysxo8fz8svv1xvr9tkpGvkbi7Q08y6Ax8Do4Hz4paZDlwdXo93JLDJ3dckse0TgcXuXlY9wcyKgA3uXmlmPQhu0si8Oxdi25ENuwUKW0ZdkYiI5IN6RtjSbdSoUUyaNIm9996bI444gnbt2rFixQpuvfVW5s6dy1577cXYsWP58ssvd1k3uOw+EDv/+9//Ptdddx2nn346s2fPZuLEibtVY8uWwb/RBQUFVFRU1Lms13K51XnnnceRRx7Js88+y8knn8w999zD8ccfz7x585gxYwYTJkzgpJNO4sYbb9ytWtMycufuFcDVwExgETDN3Rea2Tgzq76ScQZBAFsK3A1cWb2+mT0KvAYcYmZlZnZpzOZHs/MpWYAhwAIzext4HBjn7pn5xOD+o+HLjUE7MhERkTw0dOhQ3nzzTe6+++6aU7Kff/45bdq0oX379qxdu5bnnnsu4br77bcfixYtoqqqiqee+ur5sZs2baJz5+DS/SlTptRMb9euHZs3b95lO+3bt2evvfaquZ7uL3/5S80oXqqGDBnCww8/DMCSJUv46KOPOOSQQ1i+fDk9evTgmmuu4fTTT2fBggWsXr2aPfbYgwsuuIDrr7+eN998s0H7jJW23rLuPoMgwMVOuyvmvQNX1bLumDq2OzbBtCcIHo2S+Xoc91U7MvWaFRGRHBZ/zd2wYcO45ZZbKCgoYMSIETzwwAM1Qax///4MGDCAww47jB49ejB48OCE27zlllsYMWIEXbt2pU+fPmzZsgWAiRMncs4559C5c2cGDRrEihUrADjttNM4++yzefrpp7njjjt22taUKVMYN24cW7dupUePHtx///0NOs4rr7yScePG0bdvXwoLC3nggQdo2bIljz32GA899BDNmzdn//3358Ybb2Tu3Ln86Ec/olmzZjRv3pw777yz/h3Uw2obOsw3JSUl3tBn5uy2f/wE3pgM1y+BPfaOpgYREclpixYt4tBDD426DGmARL87M5vn7iWJllf7sUygdmQiIiLSSBTuMoHakYmIiEgjUbjLBGpHJiIiIo1E4S5T9D0XMFgwLepKREQkR+k6++zTkN+Zwl2mqG5H9vZUtSMTEZFG16pVK9avX6+Al0XcnfXr19OqVauU1kvbo1AkCWpHJiIiTaRLly6UlZWR9l7qsltatWpFly5dUlpH4S6T9D4dnv1h0I5M4U5ERBpR8+bN6d69e9RlSBrotGwmiW1HVrE96mpEREQkCyncZRq1IxMREZHdoHCXaWLbkYmIiIikSOEu0xQUQt+zYclM2Loh6mpEREQkyyjcZaJ+akcmIiIiDaNwl4kO6K92ZCIiItIgCneZSO3IREREpIEU7jKV2pGJiIhIAyjcZSq1IxMREZEGULjLZP1Gw2croGxu1JWIiIhIllC4y2S9T4fC1nrmnYiIiCRN4S6T1bQje0LtyERERCQpCneZTu3IREREJAUKd5lO7chEREQkBQp3mU7tyERERCQFCnfZQO3IREREJEkKd9nggP5Q1EvtyERERKReCnfZwCwYvVM7MhEREamHwl226Kd2ZCIiIlI/hbts0b6L2pGJiIhIvRTusonakYmIiEg9FO6yidqRiYiISD0U7rJJdTuyhU9CRXnU1YiIiEgGUrjLNv1Hw7bP1I5MREREEkpbuDOzYWb2vpktNbMbEsw3M7s9nL/AzA6PmXefmX1iZu/GrTPRzD42s/nh65SYeRPCbb1vZic37dGlUU07skejrkREREQyUFrCnZkVAJOA4UBvYIyZ9Y5bbDjQM3xdDtwZM+8BYFgtm7/N3YvD14xwf72B0cBh4Xp/CmvIfmpHJiIiInVI18jdQGCpuy9393JgKjAybpmRwIMeeB3oYGYHALj7HCCVJDMSmOru2919BbA0rCE3qB2ZiIiI1CJd4a4zsCrmc1k4LdVlErk6PI17n5ntlcq2zOxyMys1s9J169YlsasMoXZkIiIiUot0hTtLMC3+SbzJLBPvTuAgoBhYA/w2lW25+2R3L3H3kqKionp2lUHUjkxERERqka5wVwZ0jfncBVjdgGV24u5r3b3S3auAu/nq1GvK28o6akcmIiIiCaQr3M0FeppZdzNrQXCzw/S4ZaYDF4Z3zQ4CNrn7mro2Wn1NXuhMoPpu2unAaDNraWbdCW7SeKMxDiRjtO8C3Y5WOzIRERHZSVrCnbtXAFcDM4FFwDR3X2hm48xsXLjYDGA5wc0PdwNXVq9vZo8CrwGHmFmZmV0azvq1mb1jZguA44Brw/0tBKYB7wH/AK5y98qmPs606z9G7chERERkJ+Ya9QGgpKTES0tLoy4jNds3w296QvF5MOJ3UVcjIiIiaWJm89y9JNE8dajIZmpHJiIiInEU7rKd2pGJiIhIDIW7bFfdjmzB1KgrERERkQygcJft1I5MREREYijc5YJ+o6CyXO3IREREROEuJ6gdmYiIiIQU7nKB2pGJiIhISOEuV6gdmYiIiKBwlzvUjkxERERQuMst/UerHZmIiEieU7jLJYeeDoWtgtE7ERERyUsKd7mk1Z7Qa4TakYmIiOQxhbtco3ZkIiIieU3hLteoHZmIiEheU7jLNbHtyLZ9FnU1IiIikmYKd7lI7chERETylsJdLqpuR/a22pGJiIjkG4W7XFTTjux1tSMTERHJMwp3uUrtyERERPKSwl2uUjsyERGRvKRwl8vUjkxERCTvKNzlMrUjExERyTsKd7ms1Z7Q61S1IxMREckjCne5rp/akYmIiOQThbtcd9Dx0KZI7chERETyhMJdrisohL7nqB2ZiIhInlC4ywdqRyYiIpI3FO7ygdqRiYiI5A2Fu3ywUzuyFVFXIyIiIk1I4S5fqB2ZiIhIXlC4yxc17cgeVTsyERGRHKZwl0/UjkxERCTnpS3cmdkwM3vfzJaa2Q0J5puZ3R7OX2Bmh8fMu8/MPjGzd+PW+Y2ZLQ6Xf8rMOoTTu5nZNjObH77uavIDzAZqRyYiIpLz0hLuzKwAmAQMB3oDY8ysd9xiw4Ge4ety4M6YeQ8AwxJs+gWgj7v3A5YAE2LmLXP34vA1rlEOJNupHZmIiEjOS9fI3UBgqbsvd/dyYCowMm6ZkcCDHngd6GBmBwC4+xxgQ/xG3f15d68IP74OdGmyI8gVakcmIiKS09IV7joDq2I+l4XTUl2mLpcAz8V87m5mb5nZS2Z2TKIVzOxyMys1s9J169alsKsspnZkIiIiOS1d4c4STIu/ZTOZZRJv3OynQAXwcDhpDXCguw8ArgMeMbM9d9m4+2R3L3H3kqKiomR2lf0KCqHP2WpHJiIikqPSFe7KgK4xn7sAqxuwzC7M7CJgBHC+e/CMD3ff7u7rw/fzgGXAwQ2uPtf0H612ZCIiIjkqXeFuLtDTzLqbWQtgNDA9bpnpwIXhXbODgE3uvqaujZrZMGA8cLq7b42ZXhTexIGZ9SC4SWN54x1OllM7MhERkZyVlnAX3vRwNTATWARMc/eFZjbOzKrvZJ1BEMCWAncDV1avb2aPAq8Bh5hZmZldGs76I9AOeCHukSdDgAVm9jbwODDO3Xe5ISNvqR2ZiIhIzjJXtwIASkpKvLS0NOoy0mdTGdzWB4ZOgKHjo65GREREUmBm89y9JNE8dajIV9XtyBZMVTsyERGRHKJwl8/6j4YNy6Esj0YsRUREcpzCXT6raUf2aNSViIiISCNRuMtnakcmIiKScxTu8p3akYmIiOQUhbt8p3ZkIiIiOUXhLt+pHZmIiEhOUbgT6D9K7chERERyhMKdwAHFsO8hakcmIiKSAxTuJGhH1n+02pGJiIjkAIU7CfQ7FzBYMC3qSkRERGQ3KNxJQO3IREREcoLCnXxF7chERESynsKdfKW6HZmeeSciIpK1FO7kK9XtyN59Qu3IREREspTCnexM7chERESymsKd7EztyERERLKawp3sTO3IREREsprCnexK7chERESylsKd7ErtyERERLKWwp3syiwYvVM7MhERkayjcCeJ9T03+Kl2ZCIiIllF4U4S69AVuh2jdmQiIiJZRuFOaqd2ZCIiIllH4U5qp3ZkIiIiWUfhTmqndmQiIiJZR+FO6lbdjmzpC1FXIiIiIklQuJO6Vbcje1unZkVERLKBwp3UraYd2T/UjkxERCQLKNxJ/dSOTEREJGso3En91I5MREQkayjcSf3UjkxERCRrpC3cmdkwM3vfzJaa2Q0J5puZ3R7OX2Bmh8fMu8/MPjGzd+PW2dvMXjCzD8Kfe8XMmxBu630zO7lpjy4PqB2ZiIhIVkhLuDOzAmASMBzoDYwxs95xiw0Heoavy4E7Y+Y9AAxLsOkbgBfdvSfwYviZcNujgcPC9f4U1iANpXZkIiIiWSFdI3cDgaXuvtzdy4GpwMi4ZUYCD3rgdaCDmR0A4O5zgA0JtjsSmBK+nwKcETN9qrtvd/cVwNKwBtkd/UapHZmIiEiGS1e46wysivlcFk5LdZl4+7n7GoDwZ8fd2JbUp/dItSMTERHJcOkKd5ZgWvy5vWSWacz9YWaXm1mpmZWuW7eugbvKI2pHJiIikvHSFe7KgK4xn7sAqxuwTLy11aduw5+fpLItd5/s7iXuXlJUVFTvQQhqRyYiIpLh0hXu5gI9zay7mbUguNlhetwy04ELw7tmBwGbqk+51mE6cFH4/iLg6Zjpo82spZl1J7hJ443GOJC8p3ZkIiIiGS3pcGdmx4VBCTM7wMymhI8o2b++dd29ArgamAksAqa5+0IzG2dm48LFZgDLCW5+uBu4MmbfjwKvAYeYWZmZXRrOugX4lpl9AHwr/Iy7LwSmAe8B/wCucvfKZI9V6qB2ZCIiIhnNPMnHWpjZIuBkd//IzB4JJ28Ditz99KYqMF1KSkq8tFR3gSZl9VsweSiM+D2UXBx1NSIiInnHzOa5e0mieYUpbKdzGOwKgZOBrwHl1H9dnOSamnZkUxXuREREMkwq19x9bmb7AccC77n7lnB688YvSzKa2pGJiIhkrFTC3R0EN0Y8TNBtAmAwsLixi5IsoHZkIiIiGSnpcOfuvwJOBAa7e/Wtkh8D322KwiTDqR2ZiIhIRkrpUSjuvsTdl0Fw9yywv7u/0ySVSeZTOzIREZGMk8qjUF4ys8Hh+/EE/WEfNbOfNFVxkuHUjkxERCTjpDJy1wd4PXx/GTAUGASMq20FyXGt9oRDTlE7MhERkQySSrhrBriZHUTwfLxF7r4K2KtpSpOs0H+M2pGJiIhkkFTC3SvAH4FbgacAwqD3aRPUJdlC7chEREQySirhbiywEVgATAyn9QL+0KgVSXZROzIREZGMksqjUNa7+0/c/abqBxi7+7Pu/vsmq06yQ/9RUFkOC/8WdSUiIiJ5L5W7ZZub2c1mttzMvgx/3mxmLZqyQMkC1e3IFjwWdSUiIiJ5L5XTsr8meIjxOKB/+PN44FdNUJdkk+p2ZB+9pnZkIiIiEUsl3J0DnO7uz7v7++7+PHAmcG7TlCZZRe3IREREMkIq4c5SnC75RO3IREREMkIq4e6vwDNmdrKZHWpmw4C/ARqqkYDakYmIiEQulXD3Y2AWMAmYB9wB/AtQawIJqB2ZiIhI5FJ5FEq5u9/o7l939z3cvSfwc+CHTVeeZBW1IxMREYlcKiN3iTi65k5i9R+tdmQiIiIR2t1wB0HAEwkcdDzssa/akYmIiESksL4FzOz4OmbrAcays4Lm0PccKL03GMFrvVfUFYmIiOSVesMdcG898z9qjEIkh/QfBf+5M2hHVnJx1NWIiIjklXrDnbt3T0chkkNi25Ep3ImIiKRVY1xzJ7Kz2HZkn62MuhoREZG8onAnTUPtyERERCKhcCdNo7od2dtqRyYiIpJOCnfSdPqNgg3L1I5MREQkjRTupOmoHZmIiEjaKdxJ01E7MhERkbRTuJOmpXZkIiIiaaVwJ01L7chERETSSuFOmlZBc+h7Niz5RzCCJyIiIk0qbeHOzIaZ2ftmttTMbkgw38zs9nD+AjM7vL51zewxM5sfvlaa2fxwejcz2xYz7660HKQk1m8UVJYH7chERESkSSXTW3a3mVkBMAn4FlAGzDWz6e7+Xsxiw4Ge4etI4E7gyLrWdfdRMfv4LbApZnvL3L24CQ9LktVpgNqRiYiIpEm6Ru4GAkvdfbm7lwNTgZFxy4wEHvTA60AHMzsgmXXNzIBzgUeb+kCkAdSOTEREJG3SFe46A6tiPpeF05JZJpl1jwHWuvsHMdO6m9lbZvaSmR2zO8VLI1A7MhERkbRIV7izBNPie1LVtkwy645h51G7NcCB7j4AuA54xMz23KUos8vNrNTMStetW1dr8dII1I5MREQkLdIV7sqArjGfuwCrk1ymznXNrBA4C3isepq7b3f39eH7ecAy4OD4otx9sruXuHtJUVFRAw5LUlLdjuzjeVFXIiIikrPSFe7mAj3NrLuZtQBGA9PjlpkOXBjeNTsI2OTua5JY90RgsbuXVU8ws6LwRgzMrAfBTRrLm+rgJEnV7cj0zDsREZEmk5Zw5+4VwNXATGARMM3dF5rZODMbFy42gyCALQXuBq6sa92YzY9m1xsphgALzOxt4HFgnLtvaJKDk+TVtCN7XO3IREREmoi5rn8CoKSkxEtLS6MuI/ctmQmPnAujH4Fep0ZdjYiISFYys3nuXpJonjpUSHqpHZmIiEiTUriT9FI7MhERkSalcCfpp3ZkIiIiTUbhTtKv0wDY9+CgHZmIiIg0KoU7ST+zYPRO7chEREQancKdRKPfqOCn2pGJiIg0KoU7iYbakYmIiDQJhTuJjtqRiYiINDqFO4mO2pGJiIg0OoU7iU5NO7In1I5MRESkkSjcSbT6j4ZtG2DprKgrERERyQkKdxKt6nZkC3RqVkREpDEo3Em0qtuRvf+c2pGJiIg0AoU7iZ7akYmIiDQahTuJntqRiYiINBqFO4me2pGJiIg0GoU7yQz9zg1+qh2ZiIjIblG4k8zQ4cCgHdm8B2Dz2qirERERyVoKd5I5TrgRtm2EKSMU8ERERBpI4U4yR9eBcP5fYVOZAp6IiEgDKdxJZuk2GM5/XAFPRESkgRTuJPMo4ImIiDSYwp1kJgU8ERGRBlG4k8ylgCciIpIyhTvJbAp4IiIiKVG4k8yngCciIpI0hTvJDgp4IiIiSVG4k+yhgCciIlIvhTvJLgp4IiIidVK4k+yjgCciIlIrhTvJTgp4IiIiCSncSfZSwBMRSb8d26KuQOqRtnBnZsPM7H0zW2pmNySYb2Z2ezh/gZkdXt+6ZjbRzD42s/nh65SYeRPC5d83s5Ob/gglEgp4IiLpUbkDnrsBfr4/TD4O3noIyrdGXZUkkJZwZ2YFwCRgONAbGGNmveMWGw70DF+XA3cmue5t7l4cvmaE6/QGRgOHAcOAP4XbkVykgCci0rS2rIMHz4D/3AmHnQXlX8DTV8HvegWBb92SqCuUGOkauRsILHX35e5eDkwFRsYtMxJ40AOvAx3M7IAk1403Epjq7tvdfQWwNNyO5CoFPBGRpvHxPJh8LHxcCmdOhnPuh6v+A2NnwNdPhLn3wKQj4IERsPApqCiPuuK8l65w1xlYFfO5LJyWzDL1rXt1eBr3PjPbK4X9Sa5RwBMRaVxvPQT3DQcrgEtmQv9RwXSz4O/cs++D696DE26CjR/CX8fCbYfBi/8HG1fVuWlpOukKd5Zgmie5TF3r3gkcBBQDa4DfprA/zOxyMys1s9J169YlWEWyzk4B7zQFPBGRhqgoh2d/GJx6PXAQXD4bOhUnXrZtRzjmOrhmPpz3V+h8OLz8W/hDP3hkFCx5Hqoq01i8pCvclQFdYz53AVYnuUyt67r7WnevdPcq4G6+OvWazP5w98nuXuLuJUVFRSkflGSomoC3SgFPRCRVm/8b/N059x446hq44Elos0/96zUrgINPgvMeg/9ZAEdfBx+/CY+cA7cXw8u/C67dkyaXrnA3F+hpZt3NrAXBzQ7T45aZDlwY3jU7CNjk7mvqWje8Jq/amcC7MdsabWYtzaw7wU0abzTVwUkGUsATEUndqjfgz8fCfxfAt++Fk/4PCgpT306HA+GEn8G1C+Hs+6HD1+DFm+F3h8Ljl8DKV8F3OaEmjaQBv7HUuXuFmV0NzAQKgPvcfaGZjQvn3wXMAE4huPlhK3BxXeuGm/61mRUTnHJdCVwRrrPQzKYB7wEVwFXurjHhfFMd8B4+Owh4Fz0D7faLuioRkcxUej/M+BG07wwXPAH799n9bRa2gD5nBa91S6D0Pnj7EXj3CSjqBSWXQP/R0Kr97u9LapgrOQNQUlLipaWlUZchTWHlq0HAa99VAU9EJF7FdphxPbz5YHD361l3wx57N93+yrfCwidh7r2w+k1ovgf0PRtKLq39uj7ZhZnNc/eShPMU7gIKdzlOAU9EZFebPoZpFwaPOTnmh3DcT4Nr59Jl9VtByHvncajYBp0OhyMuDZ6l12KP9NWRhRTukqBwlwcU8EREvrLyVfjrRUE7sTPuhN6nR1fLto2w4LEg6H36fnCatv95wWnbooOjqyuDKdwlQeEuT6x8BR4+RwFPRPKXO7xxN8ycENzoMPoR6Ngr6qoC7vDhv6H0XnhvOlTtgG7HBKN5vUZAQfOoK8wYCndJULjLIwp4IpKvdmyDv18Lbz8KBw+Hs/6cuTczbPkE3voLlD4Amz6CtvvBgO/AN8ZCh671rZ3zFO6SoHCXZxTwRCTfbPwIHrsA1rwNQyfAkB9Ds3Q9EW03VFXC0heD0bwlM4PuGD1PDk7Zfv2E9F4jmEEU7pKgcJeHFPBEJF8sfwkevxgqd8BZk+GQ4VFX1DAbP4J5U4I7e7/4JHie3jcuDkb02uZXMwKFuyQo3OUpBTwRyWXu8NokeOFnsE9PGP0w7Nsz6qp2X0U5LP578Ny8lS9Ds+bBDSEll8LXjgpG93Kcwl0SFO7ymAKeiOSi8q0w/fvw7uNw6GnBHbEt20VdVeOrfjjy/Edg+yYoOjR8OPKozL2esBEo3CVB4S7PKeCJSC7ZsCK4vm7twqAN2NHX5f5oVvnWoPNF6X158XBkhbskKNyJAp6I5ISlLwb9W3H49n3Q88SoK0q/PHg4ssJdEhTuBFDAE5Hs5Q6v3AYv/i907A2jH4K9e0RdVbQSPRy5+PzgtG2WX3uocJcEhTupoYAnItlm+2Z4+ip47+lgdGrkH6FFm6iryhzu8OGrQchb9ExOPBxZ4S4JCneyk9iAN/bv0LZj1BWJiCS2fhlMPQ8+XQIn3gxHfT/3r6/bHTnycGSFuyQo3MkuFPBEJNMtmQlPXBY8yPfs++Cg46KuKHvU9nDkIy6Fg47P+IcjK9wlQeFOElLAE5FMVFUFc34Ds38J+/eFUQ/BXl+LuqrstfEjmPdA+HDkdVnxcGSFuyQo3EmtFPBEJJN8uQmeGgfvz4B+o+C0P0Dz1lFXlRsSPhx5ZHADRoY9HFnhLgkKd1InBTwRyQTrlgTX121YDif/Ao68IqMCR05Z9z6U3p+xD0dWuEuCwp3USwFPRKK06O/BiF1hSzh3CnQ7OuqK8kOGPhxZ4S4JCneSFAU8EUm3qsrg2ro5vwkexjvqL9C+S9RV5af4hyN3/kYwmhfBw5EV7pKgcCdJU8ATkXTZ9llwN+zSF2DABXDKb6F5q6irkgx4OLLCXRIU7iQlCngi0tTWvhdcX7epDIb/KggOur4us9T2cOSBl0Pv05t013WFu2ZNumeRXNXtaDj/r7BpFTwwIngopohIY1n4FNxzIuzYGvwH8ohLFewykVnw78E598N178EJN8JnH8I706ItSyN3AY3cSYNoBE9EGlNVJbx4M7z6B+gyEM59EPY8IOqqJBVVlcHjavbYu0l3o5E7kaaiETwRaSxbN8BD3w6CXcklMPZZBbts1KygyYNdvSVEuneRXKCAJyK7a80CmHxscP3W6XfAiNugsEXUVUmWUrgTaQwKeCLSUAumwb0nQWUFXPwcHH5h1BVJllO4E2ks3Y6G86Yp4IlIcior4B8T4MnLoNMAuOIl6JLwEiqRlCjciTSm7sco4IlI/basg7+cAa//CY4cBxdN1w1Z0mgU7kQamwKeiNTl4zdh8lAomwtn/jl4hl1B86irkhyicCfSFBTwRCSRtx6G+4YFz0e7ZCb0Hx11RZKDFO5EmooCnohUqyiHZ6+Hp6+EA4+Ey2dH2nRecpvCnUhTig14U05TwBPJR5vXwoOnw9y74ZtXwwVPQZt9o65KcpjCnUhTqw54Gz9SwBPJN6vmBs+vWz0fvn0vnPxzKCiMuirJcWkLd2Y2zMzeN7OlZnZDgvlmZreH8xeY2eH1rWtmvzGzxeHyT5lZh3B6NzPbZmbzw9ddaTlIkdoo4Inkn9L74f7hUNgSvjsL+p4ddUWSJ9IS7sysAJgEDAd6A2PMrHfcYsOBnuHrcuDOJNZ9Aejj7v2AJcCEmO0tc/fi8DWuaY5MJAUKeCL5oWI7TL8G/v4/0H0IXPYv2L9P1FVJHknXyN1AYKm7L3f3cmAqMDJumZHAgx54HehgZgfUta67P+/uFeH6rwNd0nEwIg2mgCeS2z5fDfefAm9OgaOvCzrXRNxnVPJPusJdZ2BVzOeycFoyyySzLsAlwHMxn7ub2Vtm9pKZHZOoKDO73MxKzax03bp1yR2JyO5SwBPJTR++Bn8+Fj5ZBOc+CCfeFDSRF0mzdIU7SzDNk1ym3nXN7KdABfBwOGkNcKC7DwCuAx4xsz132Yj7ZHcvcfeSoqKieg5BpBEp4InkDnd4426YMgJatoPLXoTe8SenRNInXeGuDOga87kLsDrJZepc18wuAkYA57u7A7j7dndfH76fBywDDm6UIxFpLAp4Itlvxzb425Uw43r4+olw2T+h46FRVyV5Ll3hbi7Q08y6m1kLYDQwPW6Z6cCF4V2zg4BN7r6mrnXNbBgwHjjd3bdWb8jMisIbMTCzHgQ3aSxv2kMUaQAFPJHstXFV0G3i7Ufg2Btg9KPQukPUVYmkJ9yFNz1cDcwEFgHT3H2hmY0zs+o7WWcQBLClwN3AlXWtG67zR6Ad8ELcI0+GAAvM7G3gcWCcu29o6uMUaRAFPJHss2JO8Py69cuCUHfcBGimR8dKZrDwTGbeKykp8dLS0qjLkHy24mV45FzocCBc9Ay07Rh1RSISzx1e/xM8/zPY5yAY/Qjs2zPqqiQPmdk8dy9JNE//zRDJFBrBE8ls5Vvhyctg5k/gkOHw3RcV7CQjqQeKSCapDniPnBsEPI3gye5why83wRfrYMva4D8M2zZA8zbBtWGt94JW4c/WHYJOCpLYZyth6gWw9l04/mfBM+x0GlYylMKdSKZRwJO6JApsX6wLfm5ZG/M+nF65PfltN98jDHsddg1+rTvEfY6Z36p9bj/Pbdk/4fFLwKuChxL3/FbUFYnUSeFOJBMp4OWXWgNbfHirI7BZAbQpgrZF0HY/KOoVvG/TMfhc/X6PvWHHVtj2GWzbGPz8cmOC9xuD0ao184PpO7buus9YLdt/FQJ3CYZxn2Pft2gLluhxphnAHV79A7x4c/B9jnoouM5OJMMp3IlkKgW87FYd2LZ8Al98Uk9g+wQqy3fdRrKBrW1HaL13054mrNgeBL4vNyYIhp/tOm/Tx199rqqofbvNCpMPguk8jbx9Czx9Fbz3NzjsTDj9j9CybdPtT6QR6W7ZkO6WlYylu2gzR1KBbS1sWZdcYNslpKU5sKWDO5R/sWsQrCsYVs/7clPd2645jRwf/jrs3mnk9ctg6vnw6ftw4kQ46prMHV2UvFXX3bIauRPJdLEjeA+MgOIx0Kw5FLSAguqf1e8TTG9WGLdM/LItcvt6qfo0RWArOnTnkNa2Y24FtlSYBSNeLduyc7OhJFRVBr+bWoPhxp0/b1jx1fv6TiO3ap/4+sIWbeDNvwS/owuehIOOS/GARaKnkbuQRu4k4614GR67IPiHq7FZs3oCY9y0hMsmCJHNEqybKFymus1mBXWPpCQKbAnDWyojbHEhLZ8DWzaoPo1c34hhohHE/fvBOffDXt0iK1+kPhq5E8kF3Y+BHy8PgkhlOVRWxLzf8dX7qlqmJ1q+akfcMjtifiaYXr18+RdxyyXaRjnQVP95tNpHLCvK6wls+34VzIp67RzS2oTXtimwZb/CltBuv+CViuoBD52GlSymcCeSTZoVQLPW0Lx11JUkp6oyLvTt2DUAVu4IQ2OC6YmWry+8FjTfOaQpsEkqFOokByjciUjTybYwKiKSA/RfWBEREZEconAnIiIikkMU7kRERERyiMKdiIiISA5RuBMRERHJIQp3IiIiIjlE4U5EREQkhyjciYiIiOQQhTsRERGRHKJwJyIiIpJD1H4sTXZUVnH1I2+yb9uWwatdS4ratqSoXYuaaW1a6tchIiIiu0dpIk22fFnByk+3MnflZ3y2tRz3XZfZo0VBGPRaUNSupYKgiIiIpEwJIU32atOCmdcOAaCisooNX5TzyebtfLplO59uKefTLdtZV/N5Oys+/UJBUERERFKmf/UjUFjQjI57tqLjnq3qXTZREIwNgbFBcMMX5Qm3ERsE923b8qswqCAoIiKSc/QveYZLJQjuCIPgujqC4Mr1X1D6oYKgiIhIrtK/zjmkeUEz9tuzFftlRBDc+XSxgqCIiEh66F/cPNWYQXDd5vqDYOvmBWHYqzsIdmjdguaFRvOCZhQ2M8yssQ9dREQkpyncSb12JwgGP8tTCoI77zsIetWvFgVG88Ig+DUvaEaLwup5scuF88LPhQWxy+66XM2yNaGyGS0K4/cbs52YZZs3a6YwKiIiGUXhThrV7gbBTdt2UFHlVFRWUV7p7KisYkdFFTvCzxWVwfsdlU55zfvg85btFeHy4XpVX70vj1musirB7ceNpEUYGgvjwuiugXLXAFr9vnlhHQE03HaLOpaLDaCFzWpfVmFURCQ3KdxJZFIJgo2pqsqD4FfpNcFxR5XHhMhwXkwgjJ9XHTJ3CqBVu4bRHZVV4bL1h9HyyioqKjMnjBY2s11GRncZwYwNkUmMjBY2Cz/HjYzWtuzOI6oKoyIiyVC4k7zTrJnRslkBLQuBllFXU7/YMFpeURWOasYHwcYJo4lGRivC9zuF0aqqXYJp9bIVTRhGE52mL4wNn4XNaGaGMqCIROnwA/fiZyN6R7b/tIU7MxsG/AEoAO5x91vi5ls4/xRgKzDW3d+sa10z2xt4DOgGrATOdffPwnkTgEuBSuAad5/ZxIco0iSyOYwGIXLXwLlLMK2ooqIqDJ8VX41wltcSTCuqwvBZERNoqwNmE4ZLEZFktG5eEOn+0xLuzKwAmAR8CygD5prZdHd/L2ax4UDP8HUkcCdwZD3r3gC86O63mNkN4efxZtYbGA0cBnQCZpnZwe5emY7jFcln2RZGRURyTbM07WcgsNTdl7t7OTAVGBm3zEjgQQ+8DnQwswPqWXckMCV8PwU4I2b6VHff7u4rgKXhdkRERERyWrrCXWdgVcznsnBaMsvUte5+7r4GIPzZMYX9YWaXm1mpmZWuW7cupQMSERERyUTpCneJLm+OvzCmtmWSWbch+8PdJ7t7ibuXFBUV1bNJERERkcyXrnBXBnSN+dwFWJ3kMnWtuzY8dUv485MU9iciIiKSc9IV7uYCPc2su5m1ILjZYXrcMtOBCy0wCNgUnmqta93pwEXh+4uAp2OmjzazlmbWneAmjTea6uBEREREMkVa7pZ19wozuxqYSfA4k/vcfaGZjQvn3wXMIHgMylKCR6FcXNe64aZvAaaZ2aXAR8A54ToLzWwa8B5QAVylO2VFREQkH5i7ngkFUFJS4qWlpVGXISIiIlIvM5vn7iWJ5qXrtKyIiIiIpIHCnYiIiEgOUbgTERERySEKdyIiIiI5ROFOREREJIco3ImIiIjkED0KJWRm64AP07CrfYFP07CfbKXvp376juqm76d++o7qpu+nbvp+6peO7+hr7p6wd6rCXZqZWWltz6URfT/J0HdUN30/9dN3VDd9P3XT91O/qL8jnZYVERERySEKdyIiIiI5ROEu/SZHXUCG0/dTP31HddP3Uz99R3XT91M3fT/1i/Q70jV3IiIiIjlEI3ciIiIiOUThLk3MbJiZvW9mS83shqjryTRmdp+ZfWJm70ZdSyYys65m9i8zW2RmC83sB1HXlGnMrJWZvWFmb4ff0c1R15SJzKzAzN4ys79HXUsmMrOVZvaOmc03s9Ko68k0ZtbBzB43s8Xh30ffjLqmTGJmh4R/dqpfn5vZ/6S9Dp2WbXpmVgAsAb4FlAFzgTHu/l6khWUQMxsCbAEedPc+UdeTaczsAOAAd3/TzNoB84Az9GfoK2ZmQBt332JmzYFXgB+4++sRl5ZRzOw6oATY091HRF1PpjGzlUCJu+s5bgmY2RTgZXe/x8xaAHu4+8aIy8pI4b/9HwNHuns6nqNbQyN36TEQWOruy929HJgKjIy4pozi7nOADVHXkancfY27vxm+3wwsAjpHW1Vm8cCW8GPz8KX/vcYwsy7AqcA9Udci2cfM9gSGAPcCuHu5gl2dTgCWpTvYgcJdunQGVsV8LkP/MEsDmVk3YADwn4hLyTjhKcf5wCfAC+6u72hnvwd+DFRFXEcmc+B5M5tnZpdHXUyG6QGsA+4PT+3fY2Ztoi4qg40GHo1ixwp36WEJpmlEQVJmZm2BJ4D/cffPo64n07h7pbsXA12AgWamU/whMxsBfOLu86KuJcMNdvfDgeHAVeElIxIoBA4H7nT3AcAXgK4hTyA8ZX068Nco9q9wlx5lQNeYz12A1RHVIlkqvI7sCeBhd38y6noyWXiqaDYwLNpKMspg4PTwmrKpwPFm9lC0JWUed18d/vwEeIrgshoJlAFlMSPijxOEPdnVcOBNd18bxc4V7tJjLtDTzLqHaX40MD3imiSLhDcL3AsscvffRV1PJjKzIjPrEL5vDZwILI60qAzi7hPcvYu7dyP4O+if7n5BxGVlFDNrE96wRHi68SRAd/CH3P2/wCozOyScdAKgm7oSG0NEp2QhGGKVJubuFWZ2NTATKADuc/eFEZeVUczsUWAosK+ZlQE3ufu90VaVUQYD3wHeCa8pA/iJu8+IrqSMcwAwJbxDrRkwzd31uA9JxX7AU8H/pSgEHnH3f0RbUsb5PvBwOFCxHLg44noyjpntQfB0jCsiq0GPQhERERHJHTotKyIiIpJDFO5EREREcojCnYiIiEgOUbgTERERySEKdyIiIiI5ROFORCRDmJmb2dejrkNEspvCnYhILcxspZltM7MtMa8/Rl2XiEhd9BBjEZG6nebus6IuQkQkWRq5ExFJkZmNNbNXzewOM9tkZovN7ISY+Z3MbLqZbTCzpWZ2Wcy8AjP7iZktM7PNZjbPzGJ7T59oZh+Y2WdmNilsPScikjSN3ImINMyRBI3T9wXOAp40s+7uvoGgp+RCoBPQC3jBzJa7+4vAdQR9J08BlgD9gK0x2x0BHAHsCcwDngHUAktEkqb2YyIitTCzlQThrSJm8o+AHcAvgM4e/iVqZm8AdwCzgZVAB3ffHM77JXCAu481s/eBH7v70wn258Ax7v5K+Hka8Ka739IkBygiOUmnZUVE6naGu3eIed0dTv/Yd/7f8YcEI3WdgA3VwS5mXufwfVdgWR37+2/M+61A290rX0TyjcKdiEjDdI67Hu5AYHX42tvM2sXN+zh8vwo4KD0likg+UrgTEWmYjsA1ZtbczM4BDgVmuPsq4N/AL82slZn1Ay4FHg7Xuwf4PzPraYF+ZrZPJEcgIjlJN1SIiNTtGTOrjPn8AvA08B+gJ/ApsBY4293Xh8uMAe4iGMX7DLjJ3V8I5/0OaAk8T3A932LgzKY+CBHJH7qhQkQkRWY2Fviuux8ddS0iIvF0WlZEREQkhyjciYiIiOQQnZYVERERySEauRMRERHJIQp3IiIiIjlE4U5EREQkhyjciYiIiOQQhTsRERGRHKJwJyIiIpJD/j8hvg+457b3UgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot training and validation learning curves\n",
    "plt.figure(figsize=(10,6))\n",
    "plt.plot(train_avg_loss_list, label='Training loss')\n",
    "plt.plot(eval_avg_loss_list, label='Evaluation loss')\n",
    "plt.xlabel('Epoch', size=12)\n",
    "plt.ylabel('Loss', size=12)\n",
    "plt.title('TabNet model learning curves')\n",
    "plt.xticks(ticks=range(epochs))\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also visualize some random explanatory masks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "rand_mask_indices = np.random.randint(len(train_mask_explain_list[-1][0]), size=10)\n",
    "\n",
    "expl_mask = train_mask_explain_list[-1][0][rand_mask_indices]\n",
    "expl_mask = expl_mask.detach().numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAi0AAAG6CAYAAADEYnYUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAABFD0lEQVR4nO3dd7wkRb3+8c+ziSDZRdKSQZCriEi6ihFRQIJeQUEF5CpHryQjYPipGC7hKooJdkW4IFwRUGRBFCQIouScZV0RdgkCwgqCLMt+f39UD9vbpyedMzNn+szz3le/znT3t6tq5syeqamqrlJEYGZmZtbvJox1AczMzMxa4UqLmZmZVYIrLWZmZlYJrrSYmZlZJbjSYmZmZpXgSouZmZlVgistZl0k6X8lfX2sy1FVkkLSBmNdDjPrD660mAGS7pP0rKSnc9v3x7pcrXLlyMwGwaSxLoBZH9klIi4e60KMBUkTI+KFsS6HmVkjbmkxa0LS8ZLOzu0fLekSJW+WNEfS5yU9lrXYfKBOOitKOl/So5KeyB5Py53/naSvSfqDpKckXSRpau78WZIeljRP0hWS/i07PgR8ADg0ayE6Lzv+iizNJyXdIWnXXFr/mz2vCyT9E/iUpEckTcrFvEfSzXWey/9K+qGkX2d5/kHSqpK+kz23uyW9Jhd/uKQ/Z8/rTknvzp3bQNLl2fN6TNLP6uS5raQHJL2l/m/LzMYzV1rMmvs0sKmkD0l6A/BhYN9YtAbGqsBUYA1gX2CGpI1K0pkAnAysDawFPAsUu6DeD+wHvAyYAnwmd+7XwIbZuRuB0wEiYkb2+JiIWCYidpE0GTgPuCiLPwg4vVCu9wPfAJYFvgc8DmyfO/9B4CcNXpf3Al/MnvtzwFVZuaYCZwPH5mL/DLwBWB44AjhN0mrZua9l5VwRmJaVZTGS3gH8FHhPRFzWoExmNo650mK2yC+zVonatj9ARDxD+gA/FjgNOCgi5hSu/X8R8VxEXA78ivSBvpiIeDwifh4Rz0TEU6QKw5sKYSdHxJ8i4lngTGCz3PUnRcRTEfEc8BXg1ZKWr/NctgGWAY6KiPkRcSlwPrBXLubciPhDRCyMiH8Bp2TPE0krAe8A/q/B63VORNyQXXsO8K+IODXrZvoZ8GJLS0ScFREPZnn9DLgX2Co7/TypIrd6RPwrIq4s5LMHMAPYKSKubVAeMxvnXGkxW+RdEbFCbvtR7UT2YTkbEKkykfdERPwzt/9XYPVi4pKWljRd0l8l/QO4AlhB0sRc2MO5x8+QKh5ImijpqKyL5R/AfVnMVMqtDjwQEQsL5Vojt/9A4ZrTgF0kLUOqdP0+Ih6qkz7AI7nHz5bsL1PbkbSPpJtrFULglbmyH0p6Xa/NurH+s5DPJ4AzI+K2BmUxswHgSotZCyQdACwBPEj6kM1bUdJLcvtrZXFFnwY2AraOiOWAN9aSb6EI7wd2A95G6mJZp3Btcbn2B4E1JeX/j68FzM3tL3ZNRMwldfG8G9ibxl1DLZO0NvAj4EDgpRGxAnB7rewR8XBE7B8RqwMfBX5YuM15D+Bdkj7RifKYWXW50mLWhKSXA18ndZ3sTRrwulkh7AhJU7IxLzsDZ5UktSypBeLJrPvly20UY1nSuJHHgaWB/y6cfwRYL7d/DfDPrKyTJb0Z2AU4o0k+p5IqZa8idfl0wktIFaRHASTtR2ppIdvfIzcg+YksNn8n04PAdsDBkj7eoTKZWQW50mK2yHlafJ6Wc7K7aU4Djo6IWyLiXuDzwE8kLZFd9zDpw/ZB0oDYj0XE3SXpfwdYCngMuBr4TRtlO5XUvTMXuDO7Pu/HwCZZ98svI2I+sCuwY5bfD4F96pQr7xzS+JJzCl1eIxYRdwLfIrXiPEKqEP0hF7IlcI2kp4GZwCER8ZdCGveTKi6HSfpIJ8plZtWjRTdAmFm7shaM0yJiWpPQypD0Z+CjgzpnjZn1L7e0mNmLJL2H1D1z6ViXxcysyJUWMwPS5HbA8cABhbuOzGzASdpB0j2SZkk6vOS8JH03O3+rpM1z506S9DdJtxeuWUnSbyXdm/1csWk53D1kZmZm9WTTMvyJNPnkHOA6YK9svFotZifSJJY7AVsDx0XE1tm5NwJPA6dGRH4Q/jHA3yPiqKwitGJEHNaoLG5pMTMzs0a2AmZFxOxskP8ZpCkY8nYjVUoiIq4mzUG1GkBEXAH8vSTd3UiTWpL9fFezgrjSYmZmZo2sweKTUc5h8YkqW40pWqU2gWX282XNCtKLVZ7d/2RmZoOmlUkjO+b5x2aP6rN2ysrrfxQYyh2aka1rBuXPpZhfKzGj1otKC5OmNKtswYL5c/sm9ri1Ptg09pD7T+t4GfrpNRjk2LHO37H9kf9IY59/bHbT2MlT1+tqGcbra1u12CrJKigz6pyeA6yZ25/G8Fm/W4kpekTSahHxUNaV9Ldm5XT3kJmZWdUtfGF0W2PXARtKWlfSFGBP0kSQeTOBfbK7iLYB5jVZu6x2zb7Z432Bc5sVpCctLWZmZtZFXZylICIWSDoQuBCYCJwUEXdI+lh2/gTgAtKdQ7NIi73uV7te0k+BNwNTJc0BvhwRPwaOAs6U9GHgftI6Yw250mJmZlZ1C7s7tVJEXECqmOSPnZB7HMABda7dq87xx0nLc7TM3UNmZmZWCW5pMTMzq7hBmcTalRYzM7Oq63L3UL9wpcXMzKzq3NJiZmZmldD8tuVxwQNxzczMrBLc0mJmZlZ17h4yMzOzSvBAXDMzM6sC3/KckbQxsBtpiekgLYA0MyLu6nLZzMzMzF7UcCCupMOAM0hLTl9LWjRJwE8lHd794pmZmVlTCxeObquIZi0tHwb+LSKezx+UdCxwB2mxo2EkDQFDANOnT+9AMc3MzKwudw8BsBBYHfhr4fhq2blSETEDmFHb/fiBR4y4gGZmZtbEgMzT0qzS8gngEkn3Ag9kx9YCNgAO7GK5zMzMrFVuaYGI+I2klwNbkQbiCpgDXBcRg1GtMzMzs77Q9O6hSPdRXd2DspiZmdlIVGgw7Wh4nhYzM7Oqc/eQmZmZVYJbWszMzKwKBmWYqVd5NjMzs0pwS4uZmVnVeUyLmZmZVYLHtJiZmVklDEhLi8e0mJmZWSW4pcXMzKzqvPaQmZmZVcKAdA+50mJmZlZ1AzIQVxHR7Ty6noGZmVmfUS8z+9dVPx3VZ+2S/75XT8s7Uh6Ia2ZmZpXQk+6hjV+2ZdOYu/92HQCTpqzRNHbB/LnjMnas83dsf+Tv2P7IfzzHjnX+gxLbUwPSPeQxLWZmZlXnSouZmZlVwaAsmOhKi5mZWdUNSEuLB+KamZlZJbilxczMrOo8uZyZmZlVwoB0D7nSYmZmVnUD0tLiMS1mZmZWCW5pMTMzqzp3D5mZmVklDEj3kCstZmZmVeeWFjMzM6uEAam0jHggrqT9OlkQMzMzs0ZGc/fQER0rhZmZmY1cLBzdVhENu4ck3VrvFLBKg+uGgCGA6dOnj7hwZmZm1oIB6R5qNqZlFeAdwBOF4wL+WO+iiJgBzKjtHvvFH424gGZmZtZEhVpLRqNZpeV8YJmIuLl4QtLvulEgMzMza5NbWiAiPtzg3Ps7XxwzMzOzcr7l2czMrOrcPWRmZmaV4O4hMzMzq4QBqbR4lWczMzOrBLe0mJmZVV3EWJegJ1xpMTMzq7oB6R5ypcXMzKzqXGkxMzOzShiQW549ENfMzMwakrSDpHskzZJ0eMl5Sfpudv5WSZs3u1bSZpKulnSzpOslbdWsHK60mJmZVd3ChaPbGpA0EfgBsCOwCbCXpE0KYTsCG2bbEHB8C9ceAxwREZsBX8r2G3KlxczMrOoiRrc1thUwKyJmR8R84Axgt0LMbsCpkVwNrCBptSbXBrBc9nh54MFmBVF0/zapwbgPy8zMbBH1MrNnTz50VJ+1S+13TN3yStod2CEiPpLt7w1sHREH5mLOB46KiCuz/UuAw4B16l0r6RXAhaTXagLwuoj4a6NyuqXFzMys6kbZPSRpKBtXUtuGcqmXVWiKlaR6MY2u/S/gkxGxJvBJ4MfNnmZP7h5aefmNmsY8Ou8eACZNWaNp7IL5c8dl7Fjn79j+yN+x/ZF/Pvb5x2Y3jZ08db2ulmG8vrbjObZKImIGMKPO6TnAmrn9aQzvyqkXM6XBtfsCh2SPzwJObFZOt7SYmZlVXSwc3dbYdcCGktaVNAXYE5hZiJkJ7JPdRbQNMC8iHmpy7YPAm7LHbwXubVYQz9NiZmZWcbGwe8NHI2KBpANJ408mAidFxB2SPpadPwG4ANgJmAU8A+zX6Nos6f2B4yRNAv5FuuuoIVdazMzMqq7LM+JGxAWkikn+2Am5xwEc0Oq12fErgde2Uw53D5mZmVkluKXFzMys6gZkGn9XWszMzKqui2Na+okrLWZmZlXnVZ7NzMysEgak0uKBuGZmZlYJbmkxMzOruu6vI9gXXGkxMzOrugHpHnKlxczMrOoG5O6hpmNaJG0saTtJyxSO79C9YpmZmZktrmGlRdLBwLnAQcDtknbLnf7vbhbMzMzMWtTdBRP7RrPuof2B10bE05LWAc6WtE5EHAeo66UzMzOz5gake6hZpWViRDwNEBH3SXozqeKyNg0qLZKGyFZrnD59emdKamZmZqViQAbiNhvT8rCkzWo7WQVmZ2Aq8Kp6F0XEjIjYIiK2GBpqutK0mZmZjcbCGN1WEc0qLfsAD+cPRMSCiNgHeGPXSmVmZmZW0LB7KCLmNDj3h84Xx8zMzNpWocG0o+F5WszMzKquQl08o+FKi5mZWdUNyEBcV1rMzMyqbkBaWrzKs5mZmVWCW1rMzMyqzgNxzczMrBIGpHvIlRYzM7OK84y4ZmZmZn3ELS1mZmZV5+4hMzMzqwRXWszMzKwSBuTuIUV0vXY2GNU/MzOzRdTLzJ7+1K6j+qxd5tiZPS3vSHkgrpmZmVVCT7qHJk1Zo2nMgvlz+yb2G2t/oGnsF/56esfL0E+vwSDHjnX+ju2P/Eca+/xjs5vGTp66XlfLMF5f26rF9lJ4TIuZmZlVgistZmZmVgkDMrmcKy1mZmZVNyAtLR6Ia2ZmZpXglhYzM7OqG5CWFldazMzMKq4Hc671BVdazMzMqm5AWlo8psXMzMwqwS0tZmZmVTcgLS2utJiZmVWcZ8Q1MzOzanClxczMzCphMCbEbV5pkbQVEBFxnaRNgB2AuyPigq6XzszMzCzTsNIi6cvAjsAkSb8FtgZ+Bxwu6TUR8Y3uF9HMzMwa8ZiWZHdgM2AJ4GFgWkT8Q9L/ANcApZUWSUPAEMD06dM7VlgzMzMrMSCVlmbztCyIiBci4hngzxHxD4CIeJYGPWgRMSMitoiILYaGhjpYXDMzMxtm4Si3imjW0jJf0tJZpeW1tYOSlqdST9PMzGz8cvdQ8saIeA4gIvKVlMnAvl0rlZmZmVlBw0pLrcJScvwx4LGulMjMzMzaMyB9H56nxczMrOLcPWRmZmbVMCAtLV7l2czMzCrBLS1mZmYVFwPS0uJKi5mZWdW50mJmZmZV4JYWMzMzq4YBqbR4IK6ZmZlVgistZmZmFRcLR7c1I2kHSfdImiXp8JLzkvTd7PytkjZv5VpJB2Xn7pB0TLNyuHvIzMys4ro5pkXSROAHwPbAHOA6STMj4s5c2I7Ahtm2NXA8sHWjayW9BdgN2DQinpP0sqZliej6LHqDMU2fmZnZIuplZo+85U2j+qxd5bLL65ZX0r8DX4mId2T7nwOIiCNzMdOB30XET7P9e4A3A+vUu1bSmcCMiLi41XK6e8jMzKzqQqPaJA1Juj63DeVSXwN4ILc/JztGCzGNrn058AZJ10i6XNKWzZ5mT7qHVl3hFU1jHn7yLgAmTSm+DsMtmD93XMaOdf6O7Y/8Rxr7/GOzm8ZOnrpeV8vg30M1Ysc6/0GJrZKImAHMqHO6rBWm2LJTL6bRtZOAFYFtgC2BMyWtFw26gDymxczMrOK6PE/LHGDN3P404MEWY6Y0uHYO8IusknKtpIXAVODRegVx95CZmVnFxUKNamviOmBDSetKmgLsCcwsxMwE9snuItoGmBcRDzW59pfAWwEkvZxUwXmsUUHc0mJmZlZx3WxpiYgFkg4ELgQmAidFxB2SPpadPwG4ANgJmAU8A+zX6Nos6ZOAkyTdDswH9m3UNQSutJiZmVkTEXEBqWKSP3ZC7nEAB7R6bXZ8PvDBdsrhSouZmVnFRfT0Dusx40qLmZlZxXnBRDMzM6uEFgbTjguutJiZmVVc9ye37w++5dnMzMwqwS0tZmZmFefuITMzM6sEV1rMzMysEjympQ5Jp3ajIGZmZjYyXZ7Gv280bGmRVFxbQMBbJK0AEBG7dqlcZmZmZotp1j00DbgTOJFFS0xvAXyry+UyMzOzFg3KjLjNuoe2AG4AvkBasfF3wLMRcXlEXF7vIklDkq6XdP2MGTM6V1ozMzMbJhaObquKhi0tEbEQ+Laks7KfjzS7JrtuBlCrrcSXDv32qAtqZmZm5RYOSEtLS3cPRcQcYA9J7wT+0d0imZmZmQ3X1i3PEfEr4FddKouZmZmNwKCMafE8LWZmZhVXpduWR8OVFjMzs4oblMnlXGkxMzOruEFpafEqz2ZmZlYJbmkxMzOrON/ybGZmZpXgu4fMzMysEjwQ18zMzCphULqHPBDXzMzMKsEtLWZmZhXnMS1mZmZWCYMypkXR/Wc6IC+lmZnZi3ra9HH9tHeN6rN2izm/rERTjce0mJmZWSX0pHtotRU2aRrz0JN3AjBpyhpNYxfMn9vV2EPW2bNp7HH3ndHxMnT7eTnWv4cqxY51/uM5dqzzH5TYXvKYFjMzM6uEQbnl2ZUWMzOzihuUwaOutJiZmVXcoLS0eCCumZmZVYJbWszMzCrOA3HNzMysEhaOdQF6xJUWMzOziovezmU3ZjymxczMzCrBLS1mZmYVt3BA7nl2pcXMzKziFg5I95ArLWZmZhU3KGNaXGkxMzOrON89VELStsBWwO0RcVF3imRmZmY2XMO7hyRdm3u8P/B9YFngy5IO73LZzMzMrAWBRrVVRbOWlsm5x0PA9hHxqKRvAlcDR5VdJGkoi2f69OmdKKeZmZnV4e6hZIKkFUktMoqIRwEi4p+SFtS7KCJmADNqu18+9DudKKuZmZmVcKUlWR64ARAQklaNiIclLZMdMzMzszFWpS6e0WhYaYmIdeqcWgi8u+OlMTMzM6tjRLc8R8QzwF86XBYzMzMbgYWD0dDieVrMzMyqzjPimpmZWSUMyNJDXuXZzMzMqsEtLWZmZhXnW57NzMysEhbKY1rMzMysAgZlTIsrLWZmZhU3KN1DHohrZmZmleBKi5mZWcUt1Oi2ZiTtIOkeSbMkHV5yXpK+m52/VdLmbVz7GUkhaWqzcrjSYmZmVnEL0ai2RiRNBH4A7AhsAuwlaZNC2I7Ahtk2BBzfyrWS1gS2B+5v5Xm60mJmZlZxMcqtia2AWRExOyLmA2cAuxVidgNOjeRqYAVJq7Vw7beBQ1srBiii62OOB2VQs5mZWU1P70E+dY0Pjuqzdp+5p9Utr6TdgR0i4iPZ/t7A1hFxYC7mfOCoiLgy278EOAxYp961knYFtouIQyTdB2wREY81KqfvHjIzMxtwkoZI3To1MyJiRu10ySXFSlK9mNLjkpYGvgC8vZ1y9qTSsvLyGzWNeXTePQBMmrJG09gF8+eOy9ixzt+x/ZG/Y/sj/3zs84/Nbho7eep6XS3DeH1tx3NsL432luesgjKjzuk5wJq5/WnAgy3GTKlzfH1gXeAWpYnxpgE3StoqIh6uV06PaTEzM6u4Lo9puQ7YUNK6kqYAewIzCzEzgX2yu4i2AeZFxEP1ro2I2yLiZRGxTkSsQ6r0bN6owgLuHjIzM6u8Vm5bHqmIWCDpQOBCYCJwUkTcIelj2fkTgAuAnYBZwDPAfo2uHWlZXGkxMzOzhiLiAlLFJH/shNzjAA5o9dqSmHVaKYcrLWZmZhU3KNP4u9JiZmZWca60mJmZWSVET2eFGTuutJiZmVXcoLS0+JZnMzMzqwS3tJiZmVXcoLS0uNJiZmZWcYOyyJ8rLWZmZhXXzcnl+okrLWZmZhU3KN1DDQfiStpa0nLZ46UkHSHpPElHS1q+N0U0MzMza3730EmkNQQAjgOWB47Ojp3cxXKZmZlZixaOcquKZt1DEyJiQfZ4i4jYPHt8paSb610kaQgYApg+ffqoC2lmZmb1DcpA3GYtLbdL2i97fIukLQAkvRx4vt5FETEjIraIiC2GhoY6VFQzMzMrs1Cj26qiWaXlI8CbJP0Z2AS4StJs4EfZOTMzM7OeaNg9FBHzgA9JWhZYL4ufExGP9KJwZmZm1lyVxqWMRku3PEfEU8AtXS6LmZmZjcCgjGnxPC1mZmYVt3BAqi2utJiZmVXcoHQPeZVnMzMzqwS3tJiZmVXcYHQOudJiZmZWeYPSPeRKi5mZWcVVaYK40XClxczMrOIG5e4hD8Q1MzOzSnBLi5mZWcUNRjuLKy1mZmaVNygDcRXR9frZoFQAzczMano6NPawdfYa1Wft0ff9tBJDeT2mxczMzCqhJ91D01Z6ZdOYOX+/HYBJU9ZoGrtg/tyuxh66zl5NY4+576cdL0O3n5dj/XuoUuxY5z+eY8c6/0GJ7aVB6dLwmBYzM7OKG5QxLa60mJmZVdygzNPiSouZmVnFDUaVxQNxzczMrCLc0mJmZlZxHtNiZmZmlRAD0kHkSouZmVnFuaXFzMzMKmFQ7h7yQFwzMzOrBLe0mJmZVdxgtLO40mJmZlZ5g9I95EqLmZlZxQ3KQNyGY1okHSxpzV4VxszMzKyeZgNxvwZcI+n3kj4uaeVeFMrMzMxaF6P8VxXNKi2zgWmkystrgTsl/UbSvpKW7XrpzMzMrKmFo9yqotmYloiIhcBFwEWSJgM7AnsB3wRKW14kDQFDANOnT+9cac3MzGyYKrWWjEazSovyOxHxPDATmClpqXoXRcQMYEZt96uHf3dUhTQzM7P6qtRaMhrNuofeV+9ERDzb4bKYmZmZ1dWwpSUi/tSrgpiZmdnILAx3D5mZmVkFDEaVxZUWMzOzyhuUGXG9YKKZmZlVgltazMzMKs63PJuZmVklDMotz660mJmZVdygjGlxpcXMzKziBqV7yANxzczMrBJcaTEzM6u4bi+YKGkHSfdImiXp8JLzkvTd7PytkjZvdq2k/5F0dxZ/jqQVmpXDlRYzM7OKi4hRbY1Imgj8gLRg8ibAXpI2KYTtCGyYbUPA8S1c+1vglRGxKfAn4HPNnqcrLWZmZhW3kBjV1sRWwKyImB0R84EzgN0KMbsBp0ZyNbCCpNUaXRsRF0XEguz6q4FpzQqiZjWsDhiM0UFmZmaLqJeZ7bLWzqP6rD3v/vPrllfS7sAOEfGRbH9vYOuIODAXcz5wVERcme1fAhwGrNPs2uz4ecDPIuK0RuV0S4uZmdmAkzQk6frcNpQ/XXJJsZJUL6bptZK+ACwATm9Wzp7c8rzaCsWur+EeevJOACZNWaNp7IL5c8dl7Fjn79j+yH+ksc8/Nrtp7OSp63W1DP49VCN2rPMflNheGu0tzxExA5hR5/QcYM3c/jTgwRZjpjS6VtK+wM7AdtFC149bWszMzCquy2NargM2lLSupCnAnsDMQsxMYJ/sLqJtgHkR8VCjayXtQOpC2jUinmnleXpyOTMzs4rr5vjUiFgg6UDgQmAicFJE3CHpY9n5E4ALgJ2AWcAzwH6Nrs2S/j6wBPBbSQBXR8THGpXFlRYzMzNrKCIuIFVM8sdOyD0O4IBWr82Ob9BuOVxpMTMzqzgvmGhmZmaVMChrD7nSYmZmVnFe5dnMzMwqoQcTxfYF3/JsZmZmleCWFjMzs4pz95CZmZlVggfimpmZWSUsHJAxLa60mJmZVdxgVFmaVFpy6wQ8GBEXS3o/8DrgLmBGRDzfgzKamZmZNW1pOTmLWTpbiXEZ4BfAdsBWwL7dLZ6ZmZk144G4yasiYlNJk4C5wOoR8YKk04Bb6l0kaQgYApg+fXrHCmtmZmbDDUqlpdk8LROyLqJlgaWB5bPjSwCT610UETMiYouI2GJoaKgzJTUzM7NSETGqrSqatbT8GLibtJz0F4CzJM0GtgHO6HLZzMzMzF7UsNISEd+W9LPs8YOSTgXeBvwoIq7tRQHNzMyssUHpHmp6y3NEPJh7/CRwdjcLZGZmZu3x5HJmZmZWCVUalzIarrSYmZlV3KB0D3mVZzMzM6sEt7SYmZlVnLuHzMzMrBIGpXvIlRYzM7OK891DZmZmVgkLB6R7yANxzczMrBLc0mJmZlZx7h4yMzOzShiU7iFXWszMzCpuUFpa1IN7uwfjlTQzM1tEvcxs45dtOarP2rv/dl1PyztSbmkxMzOrOHcPddDU5V7eNOaxf/wJgElT1mgau2D+3HEZO9b5O7Y/8ndsf+Sfj33+sdlNYydPXa+rZRivr+14ju2lQekeckuLmZlZxbmlxczMzCphUFpaPLmcmZmZVYJbWszMzCouYuFYF6EnXGkxMzOrOK/ybGZmZpXQgznX+oIrLWZmZhU3KC0tHohrZmZmleCWFjMzs4pz95CZmZlVgieXMzMzs0oYlMnlmlZaJK0PvBtYE1gA3Av8NCLmdblsZmZmZi9qOBBX0sHACcCSwJbAUqTKy1WS3tztwpmZmVlzETGqrSqatbTsD2wWES9IOha4ICLeLGk6cC7wmq6X0MzMzBoalFueWxnTMgl4AVgCWBYgIu6XNLneBZKGgCGA6dOnd6CYZmZmVk+VWktGo1ml5UTgOklXA28EjgaQtDLw93oXRcQMYEZt9/Of+WYHimpmZmZlfPcQEBHHSboYeAVwbETcnR1/lFSJMTMzM+uJpt1DEXEHcEcPymJmZmYj4O4hMzMzqwQPxDUzM7NKcEuLmZmZVcKgDMT1Ks9mZmZWCW5pMTMzqzivPWRmZmaVMCjdQ660mJmZVdygDMT1mBYzMzOrBFdazMzMKi5G+a8ZSTtIukfSLEmHl5yXpO9m52+VtHmzayWtJOm3ku7Nfq7YrByutJiZmVVcRIxqa0TSROAHwI7AJsBekjYphO0IbJhtQ8DxLVx7OHBJRGwIXJLtN+RKi5mZWcV1s9ICbAXMiojZETEfOAPYrRCzG3BqJFcDK0harcm1uwGnZI9PAd7VrCDqweCdwRgdZGZmtoh6mdmkKWuM6rN2wfy5dcsraXdgh4j4SLa/N7B1RByYizkfOCoirsz2LwEOA9apd62kJyNihVwaT0REwy6iXrS0qGyT9NF650Ya2400HevfQ1Vjxzp/x/ZH/uM5dqzzbxLbUwvmz9VoNklDkq7PbUO55MueT7GSVC+mlWtbN9ompVE0RV3f6dhupOlY/x6qGjvW+Tu2P/Ifz7FjnX+7sVXdgH8HLsztfw74XCFmOrBXbv8eYLVG19ZisserAfc0K4vHtJiZmVkj1wEbSlpX0hRgT2BmIWYmsE92F9E2wLyIeKjJtTOBfbPH+wLnNiuIJ5czMzOzuiJigaQDgQuBicBJEXGHpI9l508ALgB2AmYBzwD7Nbo2S/oo4ExJHwbuB/ZoVpaxrLTM6EJsN9J0bPuxY52/Y/sjf8f2R/7jOXas8283trIi4gJSxSR/7ITc4wAOaPXa7PjjwHbtlKMXdw+ZmZmZjZrHtJiZmVkluNJiZmZmleBKi5mZmVVC31Vastultpb0H5LenT3u2kQ9kjbuQBqTS45NHWWaEyRNyB5PkbS5pJVGk2ZJHh/vZHpZmstkZV2hw+mupBYW07LBoNxibDZ4lGZbbXqszTSXaOWYja2eVlokvVzSJZJuz/Y3lfTF3Pm3A/cCXyHdOvVO4Ajg3uxcMb1Jkj4q6TdKq0reIunXkj5WVpGo46I2n8OM3OO3SJoDPCjpIknr1EtX0sSsrF+T9PrCuS8W9t8FPATMlbQb8Hvgm8CtknYpxC4t6VBJn5W0pKQPSZop6RhJy+TiPlXYPg18tbbf4nP/U8mxH+YebwvcCXwLuE3STiXxm+YeT5b0xay8/y1p6ULsWpLOkPQocA1wnaS/ZcfWKcS2/F6QtLGkw5RWJD0ue/yKVl6DXBq3tRHb8t0FkvYr7K+ZPd/fS/p8/rlI+mUhtuX3WHZsY0nb5d8n2fEdSuJ+LelXktaX9L+SnpR0bfF1a/X9mMVuJWnL7PEm2Xux7D2zeWF7LTBT0mtaqbyUvW8bxJ5a5/h/5h5Py/6OPSnpj5JeXoht+T1eks+22etQ9vdueUnvy85/Mnu8Qp10Wo5tUJbte5jucpLWLzm+aWF/SaUvb1Mlraj0ZWal7O/B6nXyWzn7vzND0km1rST0qhaP1T7LfqT0d//S2lYWa53V07uHJF0OfBaYHhGvyY7dHhGvzB7fBewYEfcVrlsXuCAiin8gfwo8SVpoaU52eBppkpqVIuJ9Wdx36xUJ2DciliukW69FQ8AtETEti7sO+FB2v/ruwJHA3hFxtaSbas8xiz0RWBq4FtgbuDwiPpWduzEi8st430RaEXMp4BZgy4i4R9LawM8jYotc7JnAA1nsRsBdwJnALsCqEbF3FvcU6ZazO1g0rfIngO8ARMQRhdfgKRZNtVyLX5p0/33UXrN82SVdBnw6Im6UtB5wZr6sJfHfAl4KnExaKOulEbFPLvaqrHxnR8QL2bGJpHv5PxER2+RiW30vHAbsRVq0Kx+3J3BGRByVS/M/KCfghIhYORfb0numGUn3R8Rauf3fAj8HrgY+DLwW2CUiHh/le+xg0u2JdwGbAYdExLl1Yq8A/gdYhjSvwmHAz4CdSb+H7XKxrb4fv0x6j08CfgtsDfwOeBtp9sxv5NJcmD3/53Iv1TbZsYiIt+ZiW3rfZrHFybEEvAW4lBS8ay42/749k7Qi7Y9IC74dWHgN2nmPXxsRW2WP9yf9Ts4B3g6cV3s/StoH+DLpy9Dc7PJpwPbAERFxai7NlmMbKXkvdivd95L+n/8NmEz6m3pddq74XjyE9Hdr9awMtd/xP4AfRcT3S/L7I+mL3w3AC7XjEfHz7PyqwBrAacD7c2kuR/p/Pqw1XtItwAklad7Qymtgo9DjqYCvy37elDt2c+7xvcCkkuumkFaJLB6vO+Uv8Kfc46dIS2XvW7I9VnLtC8Bs4C+5rbY/Pxd3S+G6fyNNS/xu4MbCuVtzjyeR7u3/BbBE/vUoeX1uL5wrpntz9lPAwyyqiKqQ51rA2cDRwNLZsdkNXr/vAacCq+SO/aUk7sbc4xvqPY86z+1mYHJZeWvvhwblu7ew3+p74U+1PEveY8U0nwf+l/SBU9yeGsl7pvZeqLPdBjxX9vvN7X+QVPFcf5TvsduAZbLH6wDXkyouw35vhd/ZrMK5kb4fbyNNNLU06QNnuez4UiXvg92By4GdGr0X23nf1spO+qB6M/Cm7OdD2eM3NXifF38njV6vm2n8Hs/HXgesnD1+CXBb/v0NrFDyHFYk9/4eQezMOtt5wD97lO7NLJrKfSvgbuA/yl7b3DUHlR2vE3tzk/P7ApeRPicuy20za+UoueaGVvP31tmt15PLPZY1Aab/wal14qHc+ZNIXQBnkL6tAaxJ+hb845L0npC0B6n1YWGW5gTSN/EncnHXkT78/1hMQNJXStKdDWwXEfeXxD+Q231e0qoR8TBApBaX7YDzSR8qeVNqDyJiATAk6Uukb3XLFGKRNCF7Tvlm6Yn5dPIiIiRdENn/qGw/cufvB3ZX6m76raRvl6WTiz9IqRn+p0rdEN+nfJGrjSXdSvqDvI6kFSPiiez3UNZFt7ykd5O6JpeIiOfLypu5Qan76RQWfz/sC9xUiG31vbCQ9C3tr4XrV8vO5d0KfDMibi8+CUlvKxxq9T0DsArwjkK5IL2GxffoZElLRsS/ACLiNEkPk2aXfEkhtp332MSIeDqLvU/Sm4Gzs9a84hiyibnHx9bLM6/Z+xFYEKn17BlJf46If2Rxz2YtK/m0zpb0G+BrSt1nn6bOgmttvG8BtgAOAb4AfDYibpb0bERcXhI7LWuxFbCypMm19y7D3+ftvMcnKI3VmkCq4D2axf5T0oJcnOo8j4UM/321E/sGUkX46ZI0tupRuhMjTfdORFwr6S3A+ZKm1cmPiPiepFcCmwBL5o6XtfacL2mnSBOclaV1CnCKpPdE1vrSgvOUxgSeQ64FMCL+3uL1NkK9rrQcQPr2t7GkuaRvoR+snYyII7M/NLuRFlkSqQn/AxFxZ0l6e5JaDn4oqfYBsAKplrxnLm534F9lBYqIdUsOf4f07WHYBxBwTO7x4aQPoIdz6c2R9CbgwMJ110vaISJ+k4v9qqQHgeMLsUOkD4N/RcS1ueNrkprni+kuExFPR0S+grM+6ZvDYiLi3KzL4QgWdY+Uiogbsg/nA0nfdJcsCSuOBan9kVoJ+FJJ/OVArdn9akmrRMQjWRPtY4XYfUhdIkeQmm9Fqrycx/BKbKvvhU8Al0i6l0UVobWADRj+O/sEqRWgzLsL+9+htfcMpErtMhFxczFQ0u8Kh04kdZ28+EEaERdnFbRiuu28xx6WtFmtDBHxtKSdSV8cXlWI/UHuPZYfw7QBcHFJGVp5P86XtHREPEPq8qrFLc/wyiNZBeuTkjYjVWKHVfRzsa28b8kqt9+WdFb28xHq/038bP45Zvk/kb1vi91M7bzHlyd1MQiI2pcgpfE/+YrAN4AbJV3E4u/b7YGvFdJsJ/Zq4Jmyipqke3qU7lOS1o+IPwNExENZJfqXpNbrYbLuxTeTKi0XkLoaryS1shUdAnxe0nxS62mWzeLDAiLi55LemeWZrwh9tSTNfbOf+fdFAOuVldc6aCyad0jfEJcdxfU/Lzn2UmBqnfjtR5puk/iOp9tqmq2kS9Y034mykloidio5PqavbRZbXG204XuB9K12G+A9pArtNqRve/m4FUeafwefV9fKQBqLsGqd86/vRhkK78dh76Xs+FTgVY3yJ32YL9dK/u28b0kD//+7l7+HBueWBtbNlyHb9iS1NH0me1xatnZi29m6kS7wamCDkuOTSV9Yy665Lft/fEu2vwppDNBoynECqdLzAGnszm3Aj0f7mnnr7NabTOBTjbYRpHdTm/E3ViXdVtPsVrr98BqM4HXoeOxY5+/Y/si/arHAVW2kWdlY4Nrs5w2kAbMC7mhw7a6kOzC/CexcJ+bWws9lgIvqxE4GDiaNEzyb1Ko3bKyct85vveoeWjb7uRGwJYuaU3cBrhhBevX6qOtpdZ6Xfki3nTlpupFuP7wG7aTbrdixzt+x/ZF/1WJLu8LGYez1Srda/4hUcXmadNfcMJKOIn3unJ4dOkTSthFxeCH02eznM5JWBx4HyoYPQOpunQzUukv3zo59pOVnYyPSk0pLZLfTZn2hm0fEU9n+V4CzelGECqVbpbL2S7rdiB3r/B3bH/lXLXas8+9JbETUJsY8IRukvVxE3Frnup2AzWLRAP1TSAP5i5WW87OK0P+Q7iwL0piyMltGxKtz+5dmt0Fbl/V6IO5awPzc/nzS7Zbt6tYMuU63WmXtRdpVyB9chn7IH/qjDOOWGkwmKGnziLixzukVgNqdPcuXBUREbTDxzyWdDywZEfPqpPdCfvCw0rxUL9SJtQ7qdaXlJ8C1ks4h1WLfTflobyRNATbO4u6JiHxl57A2872vQunel9/pRroVeA0WS7cF7bTWtZpuOx8+3cjfZeiP/KtWhrHunup27LcanA/grSXHjwRuUpr8UsAbgc+9mGj9SSSRRET8ouTUZ4HLJM3O0lwb2K/pM7BR6+mMuPBiTfkN2e4VEXFTScw7SSO5/0x6Q6wLfDQifl2Iq/tmA4a92cYy3XbT7Fa6VXtts2teTuovXiUiXqk0tfeuEfH1ktilSXc2rBUR+0vaENgoIs4vxP0kstlZy45JWimyORe6kb/L0B/5V7EMjUh6ZZTMKzSAsdtHxG9z+6uRxrUIuCayubWycyc3SCoid+t+IY8lSOM0BdwdEc+VxVln9Xoa/7XKjkdhQi5Jd5NGeM/K9tcHfhWF6ZTbfbONZboj+Y/RjXSr9tpm11xOg+UfCrE/Iw3M2yf7UFmKdNfBZoW44vTgE0kzkG7Si/xdhv7Iv4plKLn2toh4VW5/TdK4jDWAXwP/E9kEd5J+GRHvGmHsxsC3SfPoHAz8P9LSBH8iLYdyV7djWyXpRpoMio36XUn10tyXkrmvCmmWtcpYB/W6e+hXLBpMtRTp2/g9DJ9A6G+1D7/MbNK6FIuJiHab48Ys3RGk2a10q/baQlp24Fotvtj3gjqx60fE+yTtleX3rHIXSvoc8HlgKUm1ieNEGl9Vb2HDjuXvMvRH/lUrgxqvg7Vq4dhJLL5e1eWSdomIx0ndGCONncGiNaguJXX57kdag+r7wHY9iG3VSLuSGjmEtBZcozRdaem2GMP7rYHNSd9YisePJ81y+CHSzIPnk96A/0FuLQjSLdNr5/a/RHpTzSQ3MVM/pNtumt1Kt2qvbRbza3Jr7ZAmhPt1ndg/kirEtdj1yeZ0KMQd2cb7tOP5uwz9kX9VykB762DdXNhvtF5VO7E35R43W4OqK7Ft/J7amQOn1Qkyb2ojzX1HUm5vzbeej2kpKjaLZsda6kJQWvNmm4h4RmkK8mNJK/i+BtgjIt7RL+m2m2a30q3aa5tdsx7p29jrSOv1/AX4YBRWA89itwe+SJre+yLg9aRVY39XErsG6dvkiy2OETFs3qBu5e8y9Ef+VSiDpBtIH4Rl62A9EBFr5vbvAF4b2XpV2bG3kcacvSQiVhth7K0RsWn2+OOx+JIOi3WRdSu2VWWfK6ON7Uaa1r5ej2n5VG53Aqml5aVlH1QtpndLZPfKSzqJdMfK0dn+iN803Ui3SmXt13QlvQSYENk8Pw3iXkqaml/A1RFRXO8FpQmn9gTuZNGtihERuxZju5G/y9Af+VelDJLeAPw1yhfk3CIirs/tf5LU0nB5Ie41wDERsf0IYz8KnB7ZQpu54xsAB0bEJ7od2ypJv4iIhoP+c7E3RTY+qRNx7cZae3pdaflybncB6Za/n+dr+VncusBBpDlc8t86di3E3Ur6xvMM6RvPe2r/eSXdGYVBbGOZbrtpdivdKr22hUruMBFxbC62YSUqCoPulBZt2zQajPjvZv4uQ3/kX6UytELS5yLiyEGIlfQ6hv+9KZ1Co0m6rba0fD8iiouqjipNa1+vB+LeGRGLzWOgtFptcW6DX5JW8T2PkhVfc74D3Exaifeu3Iffa4CHSuLHMt120+xWuq2k2S/ptrP8Q23Q3ZLAFqRxMgI2Ba4Bti3EzyZNw93oQ6Kb+bsM/ZF/lcrQij1I85KM61hJPyGNu7mZXMsUdeb9aoXSLczvYXhF6KvZz5YqLLXkRloOa6zXLS1l41fKjl0TEVu3mOYawMtIq33WpmleFZhSbEod63TbSbNb6Y71a9Buutm5i0gtMrXlH5YFzoqIHUpizwC+ERG3ZfuvBD4TER/K9r9H+uO2Bml12UvIfVBExMHdzN9l8O9htGVopFtdGP0WK+kuYJPowAdYrStJaTmAeaTb1F+c3TYiGt2FVC/NlltlrD09aWmRtCNp/Yc1JH03d2o5ym8XPC7rSrqIxf8DF5uW1waejIi52f5bSPf3/5V0q1zfpDuCNLuVbtVeW2hv+YeNax8SWf63S9osd77W938Di74tN9PJ/F2G/si/ymVopJ0P8SrH3k661bteK/ViGnUlxaKxL9PKKp910utkq4y1oVfdQw+S/nPuSvrPWfMU8MmS+FeRVs18K4u6GoLh99WfSVoKYF72x+AsUvPhq0mrbxYnFxrLdNtNs1vpVu21hTaWfwDuknQicFoW+0HgxcmpIuKUOtc10rH8XYb+yL/iZWiknW6JKsdOBe6UdC2Lf0kaNni6ja6kP0p6Vb6i2cC5LGqV8Uy4PdTr7qFJEVFvIqZ83N2kQWnzm8Tlb5X7JrAwIg6VNIE0/8Cm/ZJuu2l2K92qvba56zanyfIPWdySwH+R1heBNN7g+Bg+2Ps2hn/Lm0eqXH890uRaXcvfZeiP/KtSBuDQiDhM0h5RGBdYSOfzwPLjNTYi/jt7/KaymCjcBZXFttSVJOlOYAPSDQLPkSpJUedv84huxbbR60mlRdKZEfHeOv8xKfkA/BlwUEQMm1G1EPfi9NVK0zZ/LiIuzPZv7ad0202zW+lW7bXNjq9VllaUjH9plaRjSN+6/i87tCfpj9Q8YNuI2KWb+bsM/ZF/VcpA6obYnLRuTsO7UrK/s+MydiQknQUcHBENu5KUuq6HiYi/lsTOAL7XYquMdVCvuocOyX7u3GL8KsDdkq6jcdPfpZLOJPVrrkiaAhqlxbHKvvGPZbrtptmtdKv22kLryz8g6S+UV4zXKxx6fUS8Prd/m6Q/RMTrJX2wB/m7DP2Rf1XK8BvgMeAlWjTdPyxqDVgud2w8x6YT0jbA94BXAFOAicA/y2JpsSupVjmR9DLSnV+NbAt8KPsdN2yVsc7qSaUlV8P9eEQclj8n6WjSWhN5S5P6il8MA44uSfoTwPuA1UjfiJ7Pjq8KfKEkfizTbTfNbqVbtdeWyC0GBy82z3+0LJZ0i2nNkqTbJFcqiVtG0tYRcU2W5laktU+gMDi8S/m7DP2Rf1XK8MWI+KykcyNitzpp1Izn2Jrvk1qjziK9zvsAG9aJ/UorCUralXS7+uqkddPWJo1BGlYhBXZssZzWadHDNQMoWQ8CuHWkcQ3ymQh8oArp1kuzW+n242vQ7HVo9bVpEHtlybEtgdtI/df3AbcCWwEvAd7b7fxdhv7IvyplYNH6RT9ptTzjMTZ3zfXZz1tzx/7Y6vV10rwFeCnZGkPAW4AZTa55GemOsrWAtUaTv7fWtl7d8vxfwMeB9ZRmRK1ZFvhDu3G5+OWAA0hzHMwEfgscCHyGNFL89H5Jt9U0u5VuP7wG7b4OuWs+ldudQOr/frQYl8VuXojdgkUTg70oIq4DXiVpedLYridzp8/sdv4uQ3/kX5UySPqSpH2B16lkxeeI+EVud8o4jq15RtIU4Gal8UAPkSp4w7TRlfR8RDwuaYKkCRFxWdYTUJZmO60y1kG9Goi7PGn8wpHA4blTT0XE39uNy8WfS1q07CrS8uUrkt6Uh0TEzf2UbqtpdivdfngN2n0dctd8Obdbd/mHLPayQuxfgG9FxD3Z+Q9GxGmqMy175KZj70b+LoN/DyMpg6RtgQ+QWl1mDg9LC5KO99jcNWsDj5D+dnwSWB74YUTMKom9npKupIj4fCHuYtKcUUeRWlz+BmwZEa8rSfMW0nQOF0fEa5Tmm9orIoaKsdZZvRqIO5E0bfsBxROSVqp9uEXEPNJo+b1aTHe9WHQnyomkwVxrRWERsz5Jt6U0u5Vun7wGLadb0OryDwAfjojZhdh1c7u1b2Ol33h7kL/L0B/5V6oMEXElcKWk6yPix4Mam7vmr5KWAlaLiCNaiJ8laWJEvACcLOmPJWG7Ac+Sxt19gFQR+mqdJFtulbHO6lVLS34UfXEyoYjy0fStpLvYEgDF/ZHqRrpVKmu/pVsWU++6OrE3RMRrO1XmXufvMvRH/mNZBklvjYhLy7pPYPEulPEcm7tmF+CbpKU/1lWaqPKrUT653BXA24ATgYdJXUkfimy1+ULs2qRWmIslLQ1MLPtC1U6rjHVWr+4eKn676JRXa9EtcgKWyvbr3io3hulWqax9ka7aWP5B0sak/uTlC3/8lqPk9kVJLweOB1aJiFdK2hTYNSK+3ov8XYb+yL9CZXgjaXqAXUhfAFX4mf9gH8+xNV8hDVT+HUBE3CxpnZI4SDN1TyCNnfsksCZpCv7FSNofGCLd4bU+adzdCaRu7KJ2WmWsg3q9yvOLJK1P6mfcK0Y4s2BETOxsqbqXbpXK2kfptrP8w0akeYBWIP3xy8fuX5L2j4DPAtOzct0q6f9Is4/2In+XoT/yr0oZnlIa93I7iz7QoWQemHEeW7MgIuZJxYb74droSjqAVBG6JrvuXqU5W8rS/GeuVeaUWqtM08LYqPW00qI0gdj7gPeTlmk/ktbHQtiAiYhbgFsknR5Nln+IiHOBcyX9e0Rc1ULyS0fEtYU/esU5ObqZv8vQH/lXpQy1OVs2It0efS7pw30X0vIADEhsze2S3g9MlLQhcDBQNk5lsa4koFFX0nMRMb/2O5A0iToVpzZbZayDenXL8/6kysk00i2EHwHObVLrtQGnbPkH4CZJw/54RG72SUmHRsQxwPslDasIR8TBhUOPZa19kV2/O4UVY7ucv8vQH/lXogy1v5WSLgI2r42zkPQVCoOAx3NszkGkySifIy19cCHwtTqxX6G1rqTLldZCWkrS9qSpHM6rk2bLrTLWWb1qafkB6RbX90fE9QBl//HNCg7Jfray/ENt9dzrW0z7AGAGsLGkuaTbUT/Qw/xdhv7Iv0plgDSJWX65i/mkdYnKjOfYTbJtUrbtRuq6K5tGv9WupMOBD5Mm+fsocAFp8G6ZlltlrLN6dffQVNIU1nuR1qg5kzR6e82uZ26VJ+noKFn+oXiszTSXAHYn/VFciXRLfkTEsMF03cjfZeiP/KtWBklfIM1ncg7pQ/LdwM8i4siSNMdz7D2kCSlvBxbWjkf54oY/Bi4hVUreQ+pKmhwRHyvGtkppQrsnSXO+HERqlbkzIuotyWId0pNKy2IZStPIBuCS1qs5JwqT/JjlqfzW0XorQp/H8G8880jfeqdHNgmYpN+Q/ujcSFpdF4CI+FYv8ncZ/HsYRRk2B96Q7V4RETcVY8Z7rKQrI2LbeukUYpcmdSW9PTt0IfC1iHiuELczqYtpbVLrTd07JSVNILXKvD2LuxA4MXr9gTqAel5pWSzzdJvfXh7bYmWUWyIA+HPu1LLAHyKiuAIvko4DVgZ+mh16H2luhqWA5SJi7yzu9mhy11o383cZ/Htotwy2iKTtSF98L2HxlZvL5nTZglRpWYdFQyKiWNGUNAv4D+A2Vz76WPRwoSNgMqlp7uxsO4jUTNfTcnirxkaa+2Ad0h/9tXPbSg2uuaLeMeCO3LEZwKvGKn+XoT/yr1IZvC32ep1GarE6BTg5206qE3sP6U6kdfO/u5K4y4AJLea/M3AT8HdSV95TwD/G+nUZhK2nLS1K07ZPzt5okCb9WRAR9eYuMHtRNjr/xcm5IuL+kpi7gHfUzklaC/hNRGwi6SbSN63Ifm4IzCZ9U6s1BZcN5OtY/pHWKblt0MuAfw+jLsMgk3RbZMuBtBDbUleSpC1J3UOXs3jrTdkaVG6VGSO9nlxuy1h86uRLlRaeMqtLaZ6FY2ltRdVPk9Yx+TPpD/+6wMclvYRUWT5njPOH1u5AGe9l8O9hhGUwAK6WtElE3NlC7JezL8zNupK+ATxNqoxOaZLmA8DtrrD0Xq9bWm4E9oiIP2f76wFnRwfWtLHxS22uqKp0J8bGpA+Ku6NkBd4q5e8y9Ef+/VIGe7EVa33SreENW6YknUb6HdzBojuNIgqrRyst2LhFi/m33CpjndXrlpbPAJdJmk16k60N7NfjMlj1tLui6mtZNOhuU0lExKkVzt9l6I/8+6UMBju0EfvqFruSLpb09oi4qIXYdlplrIN6VmmRNBF4NanfdiMWffN4ruGFZvCkpGVIU3qfLulvFKZZr5H0E9I3sJtZdOtoAKP5oBjr/F2G/si/X8ow8KJkPpYGWu1KOgA4VNJzwPM0uOWZNAD77SXHrct63T10WUS8pWcZ2riQjQH4F+mPSG1F1dMj4vGS2LuATTrZ1zzW+bsM/ZF/v5TB2tNOV1KTdP4tIu7IHh8FXNpiq4x1UK8rLd8g/Sf/GfDP2vGIuLFnhbBxTdJZwMER8VDT4HGYv8vQH/n3SxkMlFZjHqbN1prFJhaU9BTwElIlqFmrjHVQr8e0vC77mZ+aOkgD28wWk/1hKKtVN/oDMRW4U9K1LD5Arriia9/n7zL0R/79UgYbmXYrJw28uHhRRCzbMDDXKmOdNaYz4pp1mqQ3lR2PiMsHIX+XoT/y75cyWOeoZAmHTsRae8Zi7aF3kuY0yE/MNGxhMrORkrQKsGW2e21E/G2Q8ncZ+iP/fimDdUablZabIuI13S7TIJrQy8wknUBaf+MgUlPbHqTbns06QtJ7gWtJ7633AtdI2n1Q8ncZ+iP/fimDtU7STyTtL2njOiHz20jOXRhd0uuBuLdGxKa5n8sAv/CtY9YpSpN/bV/7RitpZdJEYK9ufOX4yN9l6I/8+6UM1jpJbwW2Ja0yvR7pVvUrIuK4EaTl7qEu6fVA3Gezn89IWh14nDS1tVmnTCg0wT9Ob1sUxzp/l6E/8u+XMliLIuJSSZeTuvPeAnyMNJRhWKUlm4PnCuD3EXF3SXLttMpYG3pdaTlf0grAMcAN2bETe1wGG99+I+lC0mq8kLojLxig/F2G/si/X8pgLZJ0Cek25quA35PWyqs3BulkUqvM95SWo7mZXKtMRGzT/RIPpl53Dy0F/Bep+S1Ib4zjw+txWAdJeg/wetK4qSsiou3F+aqcv8vQH/n3SxmsNZK+TVp24TngD6SWlKsi4tk68RNZvFXm2YioNx7GOqTXlZYzgaeA07JDewErRMR7e1YIMzOzOrKxlvuR1spbNSKWKIkptspc6TvDeqPX3UMbFQahXZYNVjMblRFO/jVu8ncZ+iP/fimDtU/SgaRegNcCfwVOIlVIytyaxb0SmEdak6puq4x1Tq8rLTdJ2iYirgaQtDWpGc5sVJrNUDne83cZ+iP/fimDjchSwLHADRFRughmTUR8EhZrlTkZWBUY1ipjndWT7iFJt5G+eUwmrfB8f7a/NnBnRLyy64UwMzPrgJJWmdqdRJeOacEGQK9aWnbuUT5mZmbd1nKrjHWW1x4yMzOzSvBER2ZmZlYJrrSYmZlZJbjSYmZmZpXgSovZGJF0n6RnJT2d21YfZXpv62QZzcz6iSstZmNrl4hYJrc9OFYFkdTreZvMzNriSotZH5G0vKQfS3pI0lxJX8/WOEHS+pIulfS4pMcknZ4tQFpbdXYt4LysxeZQSW+WNKeQ/outMZK+IulsSadJ+gfwoSb5byDpcknzsvx/1svXxszMlRaz/nIKsADYAHgN8HbgI9k5AUcCqwOvANYEvgIQEXuTJm2stdwc02J+uwFnAysApzfJ/2vARcCKwDTgeyN7imZmI+PmYLOx9UtJtcmprgLeSlpE9Fngn9nKs0PA9IiYBczKYh+VdCzw5VHmf1VE/BJA0nLAjvXyB54nzWK9ekTMAa4cZd5mZm1xpcVsbL0rIi4GkLQV8A7gIUm18xOAB7LzLwO+S5o+fNns3BOjzP+B3OO1SUttlOYPHEpqbblW0hPAtyLipFHmb2bWMldazPrHA8BzwNQ6U4MfSVqza9OIeFzSu4Dv584Xp7f+J7B0bScbm7JyISZ/TcP8I+JhYP8srW2BiyVdkbUAmZl1nce0mPWJiHiINGbkW5KWkzQhG3z7pixkWeBp4ElJawCfLSTxCLBebv9PwJKS3ilpMvBFGqxC2yx/SXtImpaFP0Gq8LwwqidtZtYGV1rM+ss+wBTgTlLF4GxgtezcEcDmwDzgV8AvCtceCXxR0pOSPhMR84CPAycCc0ktL3NorFH+WwLXSHoamAkcEhF/GeHzNDNrmxdMNDMzs0pwS4uZmZlVgistZmZmVgmutJiZmVkluNJiZmZmleBKi5mZmVWCKy1mZmZWCa60mJmZWSW40mJmZmaV4EqLmZmZVcL/B13Rfgss2x1eAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x432 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10,6))\n",
    "ax = sns.heatmap(expl_mask, linewidths=1, vmin=0, vmax=0.01)\n",
    "ax.set_xlabel('Features', size=12)\n",
    "ax.set_title('Explanatory mask', size=12); \n",
    "ax.xaxis.set_ticklabels(norm_params['input_features'])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Save the model's learned parameters for Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = 'tabnet_params.pt'\n",
    "torch.save(model.state_dict(), path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will save the user-defined TabNet parameters as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "tabnet_user_params = {\n",
    "    'input_dim': input_dim,\n",
    "    'output_dim': output_dim,\n",
    "    'n_d': n_d,\n",
    "    'n_a': n_a,\n",
    "    'gamma': gamma,\n",
    "    'epsilon': epsilon,\n",
    "    'n_shared': n_shared, \n",
    "    'n_independent': n_independent, \n",
    "    'n_steps': n_steps,\n",
    "    'n_chunks': n_chunks,\n",
    "    'chunk_size': chunk_size,\n",
    "    'track_running_stats': track_running_stats,\n",
    "    'momentum': momentum,\n",
    "    'learning_rate': learning_rate,\n",
    "    'feat_cols': feat_cols,\n",
    "    'ghost_batch_norm': ghost_batch_norm\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(r\"tabnet_user_params.pickle\", \"wb\") as output_file:\n",
    "    pickle.dump(tabnet_user_params, output_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[1]. <i>TabNet: Attentive Interpretable Tabular Learning</i>, Sercan O. Arık, Tomas Pfister, Google Cloud AI, https://arxiv.org/pdf/1908.07442.pdf"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "aircraft",
   "language": "python",
   "name": "aircraft"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}