3_biGRU.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build a biGRU neural network model\n",
    "\n",
    "In this notebook, we are going to build a bidirectional Gated Recurrent Unit neural network model, in which we will implement and use the following architectures and tools:\n",
    "- Bidirectional GRU\n",
    "- Multi-layer (stacked) GRU\n",
    "- Dropout\n",
    "- Spatial Dropout after embedding layer\n",
    "- Max-pooling\n",
    "- Average-pooling\n",
    "- pack_padded_sequence\n",
    "\n",
    "The model is going to be trained on the clean_review column from the training dataset, other features will be used in the next notebook. In the end, the model will be evaluated on the test set to determine the generalization error.\n",
    "\n",
    "We will perform the hyperparameter fine-tuning and visualize model's learning curves to compare the model's performance while working on different set of parameters. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Gated Recurrent Unit\n",
    "Gated Recurrent Unit is the simpler variation of the LSTM (Long-Short Term Memory). Both these architectures were proposed as the remedy for the RNN's vanishing gradient problems. GRU comprised of only two gates, while its ancestor - LSTM had 3 gates in its unit, thus the GRU unit that was firstly introduced by the Kyunghyun Cho in the following paper:\n",
    "https://arxiv.org/pdf/1406.1078v3.pdf, is much simpler to compute and implement.\n",
    "<br><br>\n",
    "The GRU consists of two gates:\n",
    "- Update  gate *z* that selects whether  the  hidden  state  is  to  be  updated  with a  new  hidden  state ̃*h*.\n",
    "- Reset  gate *r* that decides whether the previous hidden state is ignored.\n",
    "\n",
    "<div>\n",
    "<img src=\"assets/GRU_shema.png\" width=\"400\" />\n",
    "</div>\n",
    "\n",
    "<p style='text-align: center;'>GRU shema [By Jeblad - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=66225938]</p>\n",
    "\n",
    "For each element in the input sequence, each layer computes the following function:<br><br>\n",
    "$ r_t = sigma(W_{ir} x_t + b_{ir} + W_{hr} h_{(t-1)} + b_{hr})$<br>\n",
    "$ z_t = sigma(W_{iz} x_t + b_{iz} + W_{hz} h_{(t-1)} + b_{hz})$<br>\n",
    "$ `h_t = tanh(W_{in} x_t + b_{in} + r_t * (W_{hn} h_{(t-1)}+ b_{hn}))$<br>\n",
    "$ h_t = (1 - z_t) * `h_t + z_t * h_{(t-1)}$\n",
    "<br>\n",
    "\n",
    "#### Bidirectional RNN\n",
    "Bidirectional Recurrent Neural Network consists of two recurrent hidden layers, that have opposite direction, in other words, one layer reads the sequence from the beginning to the end, while the other one reads it in reverse order. This architecture provides the model with much more information, as it includes both past and future context.<br>\n",
    "<div>\n",
    "<img src=\"assets/RNN-bidirectional.png\" width=\"400\" />\n",
    "</div>\n",
    "\n",
    "<p style='text-align: center;'>Bidirectional RNN schema [http://colah.github.io/posts/2015-09-NN-Types-FP/]</p>\n",
    "<br>\n",
    "Another concept that is worth to mention is the <b>stacked RNN</b> that simply consists of two or more hidden layers stacked together.\n",
    "\n",
    "<br>\n",
    "\n",
    "#### Spatial dropout\n",
    "\n",
    "Spatial dropout is another method of using dropout regularization that is applied most commonly in Convolutional Neural Networks. In contrast to the traditional dropout, the spatial dropout zeroes out the entire 1D feature map from the embedding feature vector of each word. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model's architecture\n",
    "\n",
    "The graph of the model created using tensorboardX is depicted below.<br>\n",
    "\n",
    "<div>\n",
    "<img src=\"assets/BiGRU_model_graph.png\" width=\"700\"/>\n",
    "</div>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A lot is going on the above plot, so for the sake of simplicity, I have created an additional schema that shows the layout of the model.\n",
    "<br>\n",
    "<br>\n",
    "<div>\n",
    "<img src=\"assets/BiGRU_schema.png\" width=\"750\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the GRU layer we will concatenate both, the average pooling and max pooling of the hidden representation and the last hidden state of GRU in order to prevent our model from forgetting infromations. This architecture is described in the following paper: https://arxiv.org/pdf/1801.06146.pdf. There is also the possibility to get rid of the last hidden state from our model at all, this kind of architecture, that uses max-pooling or avg-pooling is depicted in the paper: https://arxiv.org/pdf/1705.02364.pdf."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building and training the model\n",
    "\n",
    "Let's start with importing all indispensable libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from batch_iterator import BatchIterator\n",
    "from early_stopping import EarlyStopping\n",
    "import pandas as pd\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch import device\n",
    "from tqdm import tqdm_notebook\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from tensorboardX import SummaryWriter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we are going to load the tarining and validation sets, but we will use only the clean_review column and label column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the dataset. Use clean_review and label columns\n",
    "train_dataset = pd.read_csv('dataset/datasets_feat_clean/train_feat_clean.csv', \n",
    "                            usecols=['clean_review', 'label'])\n",
    "\n",
    "# Change columns order\n",
    "train_dataset = train_dataset[['clean_review', 'label']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>clean_review</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>amaze good wonderful film early ninety franchi...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>wrong end see tell chick go crazy eat old woma...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>guess emperor clothe see list pbs night hopefu...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>earth well movie funny sweet good plot unique ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>doe eye high school student kathleen beller fi...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        clean_review  label\n",
       "0  amaze good wonderful film early ninety franchi...      1\n",
       "1  wrong end see tell chick go crazy eat old woma...      0\n",
       "2  guess emperor clothe see list pbs night hopefu...      0\n",
       "3  earth well movie funny sweet good plot unique ...      1\n",
       "4  doe eye high school student kathleen beller fi...      1"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Depict the first 5 rows of the training set\n",
    "train_dataset.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the dataset. Use clean_review and label columns\n",
    "val_dataset = pd.read_csv('dataset/datasets_feat_clean/val_feat_clean.csv',\n",
    "                          usecols=['clean_review', 'label'])\n",
    "\n",
    "# Change columns order\n",
    "val_dataset = val_dataset[['clean_review', 'label']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>clean_review</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>go movie twice week sum word normally use ligh...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>year big fan park work old boy time favorite.w...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>movie potential handle differently need differ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>movie difficult review give away plot suffice ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>plot worth discussion hint corruption murder p...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        clean_review  label\n",
       "0  go movie twice week sum word normally use ligh...      1\n",
       "1  year big fan park work old boy time favorite.w...      0\n",
       "2  movie potential handle differently need differ...      0\n",
       "3  movie difficult review give away plot suffice ...      1\n",
       "4  plot worth discussion hint corruption murder p...      0"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Depict the first 5 rows of the validation set\n",
    "val_dataset.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we will use the BatchIterator class defined in the previous notebook to create the vocabulary, trim sequences in terms of the rare word occurrence and the length, map words to their numerical representation (word2index), furthermore BatchIterator sorts dataset examples, generates batches, performs sequence padding and enables to use it instance to iterate through all batches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trimmed vocabulary using as minimum count threashold: count = 3.00\n",
      "35371/130416 tokens has been retained\n",
      "Trimmed input strings vocabulary\n",
      "Trimmed input sequences lengths to the length of: 198\n",
      "Mapped words to indices\n",
      "Batches created\n"
     ]
    }
   ],
   "source": [
    "train_iterator = BatchIterator(train_dataset, batch_size=256, vocab_created=False, vocab=None, target_col=None,\n",
    "                               word2index=None, sos_token='<SOS>', eos_token='<EOS>', unk_token='<UNK>',\n",
    "                               pad_token='<PAD>', min_word_count=3, max_vocab_size=None, max_seq_len=0.9,\n",
    "                               use_pretrained_vectors=False, glove_path='glove/', glove_name='glove.6B.100d.txt',\n",
    "                               weights_file_name='glove/weights.npy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trimmed vocabulary using as minimum count threashold: count = 3.00\n",
      "19357/59089 tokens has been retained\n",
      "Trimmed input strings vocabulary\n",
      "Trimmed input sequences lengths to the length of: 195\n",
      "Mapped words to indices\n",
      "Batches created\n"
     ]
    }
   ],
   "source": [
    "val_iterator = BatchIterator(val_dataset, batch_size=256, vocab_created=False, vocab=None, target_col=None,\n",
    "                             word2index=train_iterator.word2index, sos_token='<SOS>', eos_token='<EOS>',\n",
    "                             unk_token='<UNK>', pad_token='<PAD>', min_word_count=3, max_vocab_size=None,\n",
    "                             max_seq_len=0.9, use_pretrained_vectors=False, glove_path='glove/',\n",
    "                             glove_name='glove.6B.100d.txt', weights_file_name='glove/weights.npy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have to check out how batches that we created look like before we pass them into the model. For the record, the set of batches for input and output variables is returned as a dictionary, thus we will just look at the dictionary keys to find out how to extract particular variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['input_seq', 'target', 'x_lengths'])\n"
     ]
    }
   ],
   "source": [
    "for batches in train_iterator:\n",
    "    print(batches.keys())\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the output batch has the dimensions: (batch_size, seq_len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input_seq shape:  torch.Size([256, 199])\n",
      "target shape:  torch.Size([256])\n",
      "x_lengths shape:  torch.Size([256])\n"
     ]
    }
   ],
   "source": [
    "for batches in train_iterator:\n",
    "    # Unpack the dictionary of batches\n",
    "    input_seq, target, x_lengths = batches['input_seq'], batches['target'], batches['x_lengths']\n",
    "    print('input_seq shape: ', input_seq.size())\n",
    "    print('target shape: ', target.size())\n",
    "    print('x_lengths shape: ', x_lengths.size())\n",
    "    break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input_seq shape:  torch.Size([256, 55])\n",
      "target shape:  torch.Size([256])\n",
      "x_lengths shape:  torch.Size([256])\n"
     ]
    }
   ],
   "source": [
    "for batches in val_iterator:\n",
    "    # Unpack the dictionary of batches\n",
    "    input_seq, target, x_lengths = batches['input_seq'], batches['target'], batches['x_lengths']\n",
    "    print('input_seq shape: ', input_seq.size())\n",
    "    print('target shape: ', target.size())\n",
    "    print('x_lengths shape: ', x_lengths.size())\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next step is to build the biGRU model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BiGRU(nn.Module):\n",
    "    \"\"\"BiDirectional GRU neural network model.\n",
    "        \n",
    "    Parameters\n",
    "    ----------\n",
    "    hidden_size: int\n",
    "        Number of features in the hidden state.\n",
    "    vocab_size: int\n",
    "        The size of the vocabulary.\n",
    "    embedding_dim: int\n",
    "        The size of each embedding vector.\n",
    "    output_size: int\n",
    "        Number of classes.\n",
    "    n_layers: int, optional (default=1)\n",
    "        Number of stacked recurrent layers.\n",
    "    dropout: float, optional (default=0.2)\n",
    "        Probability of an element of the tensor to be zeroed.\n",
    "    spatial_dropout: boolean, optional (default=True)\n",
    "        Whether to use the spatial dropout.\n",
    "    bidirectional: boolean, optional (default=True)\n",
    "        Whether to use the bidirectional GRU.\n",
    "\n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__(self, hidden_size, vocab_size, embedding_dim, output_size, n_layers=1, dropout=0.2,\n",
    "                 spatial_dropout=True, bidirectional=True):\n",
    "        \n",
    "        # Inherit everything from the nn.Module\n",
    "        super(BiGRU, self).__init__()\n",
    "        \n",
    "        # Initialize attributes\n",
    "        self.hidden_size = hidden_size\n",
    "        self.vocab_size = vocab_size\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.output_size = output_size\n",
    "        self.n_layers = n_layers\n",
    "        self.dropout_p = dropout\n",
    "        self.spatial_dropout = spatial_dropout\n",
    "        self.bidirectional = bidirectional\n",
    "        self.n_directions = 2 if self.bidirectional else 1\n",
    "        \n",
    "        # Initialize layers\n",
    "        self.embedding = nn.Embedding(self.vocab_size, self.embedding_dim)\n",
    "        self.dropout = nn.Dropout(self.dropout_p)\n",
    "        if self.spatial_dropout:\n",
    "            self.spatial_dropout1d = nn.Dropout2d(self.dropout_p)\n",
    "        self.gru = nn.GRU(self.embedding_dim, self.hidden_size, num_layers=self.n_layers, \n",
    "                          dropout=(0 if n_layers == 1 else self.dropout_p), batch_first=True,\n",
    "                          bidirectional=self.bidirectional)\n",
    "        # Linear layer input size is equal to hidden_size * 3, becuase\n",
    "        # we will concatenate max_pooling ,avg_pooling and last hidden state\n",
    "        self.linear = nn.Linear(self.hidden_size * 3, self.output_size)\n",
    "\n",
    "        \n",
    "    def forward(self, input_seq, input_lengths, hidden=None):\n",
    "        \"\"\"Forward propagate through the neural network model.\n",
    "        \n",
    "        Parameters\n",
    "        ----------\n",
    "        input_seq: torch.Tensor\n",
    "            Batch of input sequences.\n",
    "        input_lengths: torch.LongTensor\n",
    "            Batch containing sequences lengths.\n",
    "        hidden: torch.FloatTensor, optional (default=None)\n",
    "            Tensor containing initial hidden state.\n",
    "            \n",
    "        Returns\n",
    "        -------\n",
    "        torch.Tensor\n",
    "            Logarithm of softmaxed input tensor.\n",
    "        \n",
    "        \"\"\"\n",
    "        # Extract batch_size\n",
    "        self.batch_size = input_seq.size(0)\n",
    "        \n",
    "        # Embeddings shapes\n",
    "        # Input: (batch_size,  seq_length)\n",
    "        # Output: (batch_size, seq_length, embedding_dim)\n",
    "        emb_out = self.embedding(input_seq)\n",
    "        \n",
    "        if self.spatial_dropout:\n",
    "            # Convert to (batch_size, embedding_dim, seq_length)\n",
    "            emb_out = emb_out.permute(0, 2, 1)\n",
    "            emb_out = self.spatial_dropout1d(emb_out)\n",
    "            # Convert back to (batch_size, seq_length, embedding_dim)\n",
    "            emb_out = emb_out.permute(0, 2, 1)\n",
    "        else:\n",
    "            emb_out = self.dropout(emb_out)\n",
    "        \n",
    "        # Pack padded batch of sequences for RNN module\n",
    "        packed_emb = nn.utils.rnn.pack_padded_sequence(emb_out, input_lengths, batch_first=True)\n",
    "                \n",
    "        # GRU input/output shapes, if batch_first=True\n",
    "        # Input: (batch_size, seq_len, embedding_dim)\n",
    "        # Output: (batch_size, seq_len, hidden_size*num_directions)\n",
    "        # Number of directions = 2 when used bidirectional, otherwise 1\n",
    "        # shape of hidden: (n_layers x num_directions, batch_size, hidden_size)\n",
    "        # Hidden state defaults to zero if not provided\n",
    "        gru_out, hidden = self.gru(packed_emb, hidden)\n",
    "        # gru_out: tensor containing the output features h_t from the last layer of the GRU\n",
    "        # gru_out comprises all the hidden states in the last layer (\"last\" depth-wise, not time-wise)\n",
    "        # For biGRu gru_out is the concatenation of a forward GRU representation and a backward GRU representation\n",
    "        # hidden (h_n) comprises the hidden states after the last timestep\n",
    "        \n",
    "        # Extract and sum last hidden state\n",
    "        # Input hidden shape: (n_layers x num_directions, batch_size, hidden_size)\n",
    "        # Separate hidden state layers\n",
    "        hidden = hidden.view(self.n_layers, self.n_directions, self.batch_size, self.hidden_size)\n",
    "        last_hidden = hidden[-1]\n",
    "        # last hidden shape (num_directions, batch_size, hidden_size)\n",
    "        # Sum the last hidden state of forward and backward layer\n",
    "        last_hidden = torch.sum(last_hidden, dim=0)\n",
    "        # Summed last hidden shape (batch_size, hidden_size)\n",
    "        \n",
    "        # Pad a packed batch\n",
    "        # gru_out output shape: (batch_size, seq_len, hidden_size*num_directions)\n",
    "        gru_out, lengths = nn.utils.rnn.pad_packed_sequence(gru_out, batch_first=True)\n",
    "              \n",
    "        # Sum the gru_out along the num_directions\n",
    "        if self.bidirectional:\n",
    "            gru_out = gru_out[:,:,:self.hidden_size] + gru_out[:,:,self.hidden_size:]\n",
    "        \n",
    "        # Select the maximum value over each dimension of the hidden representation (max pooling)\n",
    "        # Permute the input tensor to dimensions: (batch_size, hidden, seq_len)\n",
    "        # Output dimensions: (batch_size, hidden_size)\n",
    "        max_pool = F.adaptive_max_pool1d(gru_out.permute(0,2,1), (1,)).view(self.batch_size,-1)\n",
    "        \n",
    "        # Consider the average of the representations (mean pooling)\n",
    "        # Sum along the batch axis and divide by the corresponding lengths (FloatTensor)\n",
    "        # Output shape: (batch_size, hidden_size)\n",
    "        avg_pool = torch.sum(gru_out, dim=1) / lengths.view(-1,1).type(torch.FloatTensor) \n",
    "\n",
    "        # Concatenate max_pooling, avg_pooling and last hidden state tensors\n",
    "        concat_out = torch.cat([last_hidden, max_pool, avg_pool], dim=1)\n",
    "\n",
    "        #concat_out = self.dropout(concat_out)\n",
    "        out = self.linear(concat_out)\n",
    "        return F.log_softmax(out, dim=-1)\n",
    "    \n",
    "    \n",
    "    def add_loss_fn(self, loss_fn):\n",
    "        \"\"\"Add loss function to the model.\n",
    "        \n",
    "        \"\"\"\n",
    "        self.loss_fn = loss_fn\n",
    "        \n",
    "\n",
    "    def add_optimizer(self, optimizer):\n",
    "        \"\"\"Add optimizer to the model.\n",
    "        \n",
    "        \"\"\"\n",
    "        self.optimizer = optimizer\n",
    "        \n",
    "        \n",
    "    def add_device(self, device=torch.device('cpu')):\n",
    "        \"\"\"Specify the device.\n",
    "        \n",
    "        \"\"\"\n",
    "        self.device = device\n",
    "    \n",
    "    \n",
    "    def train_model(self, train_iterator):\n",
    "        \"\"\"Perform single training epoch.\n",
    "        \n",
    "        Parameters\n",
    "        ----------\n",
    "        train_iterator: BatchIterator\n",
    "            BatchIterator class object containing training batches.\n",
    "            \n",
    "        Returns\n",
    "        -------\n",
    "        train_losses: list\n",
    "            List of the training average batch losses.\n",
    "        avg_loss: float\n",
    "            Average loss on the entire training set.\n",
    "        accuracy: float\n",
    "            Models accuracy on the entire training set.\n",
    "            \n",
    "        \"\"\"\n",
    "        self.train()\n",
    "        \n",
    "        train_losses = []\n",
    "        losses = []\n",
    "        losses_list = []\n",
    "        num_seq = 0\n",
    "        batch_correct = 0\n",
    "            \n",
    "        for i, batches in tqdm_notebook(enumerate(train_iterator, 1), total=len(train_iterator), desc='Training'):\n",
    "            input_seq, target, x_lengths = batches['input_seq'], batches['target'], batches['x_lengths']\n",
    "            \n",
    "            input_seq.to(self.device)\n",
    "            target.to(self.device)\n",
    "            x_lengths.to(self.device)\n",
    "            \n",
    "            self.optimizer.zero_grad()\n",
    "\n",
    "            pred = self.forward(input_seq, x_lengths)\n",
    "            loss = self.loss_fn(pred, target)\n",
    "            loss.backward()\n",
    "            losses.append(loss.data.cpu().numpy())\n",
    "            self.optimizer.step()\n",
    "            \n",
    "            losses_list.append(loss.data.cpu().numpy())\n",
    "            \n",
    "            pred = torch.argmax(pred, 1)\n",
    "\n",
    "            if self.device.type == 'cpu':\n",
    "                batch_correct += (pred.cpu() == target.cpu()).sum().item()\n",
    "\n",
    "            else:\n",
    "                batch_correct += (pred == target).sum().item()\n",
    "\n",
    "            num_seq += len(input_seq)     \n",
    "    \n",
    "            if i % 100 == 0:\n",
    "                avg_train_loss = np.mean(losses)\n",
    "                train_losses.append(avg_train_loss)\n",
    "                \n",
    "                accuracy = batch_correct / num_seq\n",
    "                \n",
    "                print('Iteration: {}. Average training loss: {:.4f}. Accuracy: {:.3f}'\\\n",
    "                      .format(i, avg_train_loss, accuracy))\n",
    "                \n",
    "                losses = []\n",
    "                \n",
    "            avg_loss = np.mean(losses_list)\n",
    "            accuracy = batch_correct / num_seq\n",
    "                              \n",
    "        return train_losses, avg_loss, accuracy\n",
    "    \n",
    "    \n",
    "    def evaluate_model(self, eval_iterator, conf_mtx=False):\n",
    "        \"\"\"Perform the one evaluation epoch.\n",
    "        \n",
    "        Parameters\n",
    "        ----------\n",
    "        eval_iterator: BatchIterator\n",
    "            BatchIterator class object containing evaluation batches.\n",
    "        conf_mtx: boolean, optional (default=False)\n",
    "            Whether to print the confusion matrix at each epoch.\n",
    "            \n",
    "        Returns\n",
    "        -------\n",
    "        eval_losses: list\n",
    "            List of the evaluation average batch losses.\n",
    "        avg_loss: float\n",
    "            Average loss on the entire evaluation set.\n",
    "        accuracy: float\n",
    "            Models accuracy on the entire evaluation set.\n",
    "        conf_matrix: list\n",
    "            Confusion matrix.\n",
    "            \n",
    "        \"\"\"\n",
    "        self.eval()\n",
    "        \n",
    "        eval_losses = []\n",
    "        losses = []\n",
    "        losses_list = []\n",
    "        num_seq = 0\n",
    "        batch_correct = 0\n",
    "        pred_total = torch.LongTensor()\n",
    "        target_total = torch.LongTensor()\n",
    "        \n",
    "        with torch.no_grad():\n",
    "            for i, batches in tqdm_notebook(enumerate(eval_iterator, 1), total=len(eval_iterator), desc='Evaluation'):\n",
    "                input_seq, target, x_lengths = batches['input_seq'], batches['target'], batches['x_lengths']\n",
    "                \n",
    "                input_seq.to(self.device)\n",
    "                target.to(self.device)\n",
    "                x_lengths.to(self.device)\n",
    "\n",
    "                pred = self.forward(input_seq, x_lengths)\n",
    "                loss = self.loss_fn(pred, target)\n",
    "                losses.append(loss.data.cpu().numpy())\n",
    "                losses_list.append(loss.data.cpu().numpy())\n",
    "                \n",
    "                pred = torch.argmax(pred, 1)\n",
    "                                \n",
    "                if self.device.type == 'cpu':\n",
    "                    batch_correct += (pred.cpu() == target.cpu()).sum().item()\n",
    "                    \n",
    "                else:\n",
    "                    batch_correct += (pred == target).sum().item()\n",
    "                    \n",
    "                num_seq += len(input_seq)     \n",
    "                \n",
    "                pred_total = torch.cat([pred_total, pred], dim=0)\n",
    "                target_total = torch.cat([target_total, target], dim=0)\n",
    "                \n",
    "                if i % 100 == 0:\n",
    "                    avg_batch_eval_loss = np.mean(losses)\n",
    "                    eval_losses.append(avg_batch_eval_loss)\n",
    "                    \n",
    "                    accuracy = batch_correct / num_seq\n",
    "                    \n",
    "                    print('Iteration: {}. Average evaluation loss: {:.4f}. Accuracy: {:.2f}'\\\n",
    "                          .format(i, avg_batch_eval_loss, accuracy))\n",
    "\n",
    "                    losses = []\n",
    "                    \n",
    "            avg_loss_list = []\n",
    "                    \n",
    "            avg_loss = np.mean(losses_list)\n",
    "            accuracy = batch_correct / num_seq\n",
    "            \n",
    "            conf_matrix = confusion_matrix(target_total.view(-1), pred_total.view(-1))\n",
    "        \n",
    "        if conf_mtx:\n",
    "            print('\\tConfusion matrix: ', conf_matrix)\n",
    "            \n",
    "        return eval_losses, avg_loss, accuracy, conf_matrix\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will instantiate the model, add loss function, optimizer, and device to it and begin the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Start epoch [1/20]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dcd77bca73c940c39e955ff0ba53d001",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=136, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.7326. Accuracy: 0.516\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ccc30f8c1d224ecbbcf842aca30dfede",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=39, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Epoch [1/20]: Train accuracy: 0.524. Train loss: 0.7203. Evaluation accuracy: 0.595. Evaluation loss: 0.6646\n",
      "\n",
      "Start epoch [2/20]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f07a40184c6540bb9b1f8900c64c6004",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=136, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.6676. Accuracy: 0.591\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "252d300165044a25a8591f3d248c149c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=39, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Epoch [2/20]: Train accuracy: 0.606. Train loss: 0.6601. Evaluation accuracy: 0.725. Evaluation loss: 0.5868\n",
      "\n",
      "Start epoch [3/20]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6fe78369f680447fb5d373cfb9f30189",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=136, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.6024. Accuracy: 0.686\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7996055d84cd46d3a80ea06fc92cef7a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=39, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Epoch [3/20]: Train accuracy: 0.697. Train loss: 0.5909. Evaluation accuracy: 0.792. Evaluation loss: 0.4759\n",
      "\n",
      "Start epoch [4/20]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "644c6741cac44fa58a6c9e8e47455db0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=136, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.5169. Accuracy: 0.753\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ba2d0c3ba31142028877d063c29775af",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=39, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Epoch [4/20]: Train accuracy: 0.759. Train loss: 0.5079. Evaluation accuracy: 0.823. Evaluation loss: 0.4028\n",
      "\n",
      "Start epoch [5/20]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0669f839b27f4c89825be3e941c226d4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=136, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.4483. Accuracy: 0.792\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "21e7991ee6b74a4085f53c49bc469fda",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=39, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Epoch [5/20]: Train accuracy: 0.795. Train loss: 0.4417. Evaluation accuracy: 0.836. Evaluation loss: 0.3679\n",
      "\n",
      "Start epoch [6/20]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "75bc44e9cf934afe9b5b295ffe41b5ff",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=136, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.4062. Accuracy: 0.819\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f6e3d8d94abe472cb50a177c0959205f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=39, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Epoch [6/20]: Train accuracy: 0.822. Train loss: 0.4015. Evaluation accuracy: 0.846. Evaluation loss: 0.3507\n",
      "\n",
      "Start epoch [7/20]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e3594666987c43c6ac2e9c0630272981",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=136, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3746. Accuracy: 0.836\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "599259e9cc7046648ef5adefa05cec66",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=39, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Epoch [7/20]: Train accuracy: 0.837. Train loss: 0.3724. Evaluation accuracy: 0.850. Evaluation loss: 0.3415\n",
      "\n",
      "Start epoch [8/20]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "657a537fb16048abb39e57cd0f824426",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=136, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3542. Accuracy: 0.848\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5ed32ee860164d14b6340a6735d187d9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=39, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Epoch [8/20]: Train accuracy: 0.850. Train loss: 0.3517. Evaluation accuracy: 0.856. Evaluation loss: 0.3339\n",
      "\n",
      "Start epoch [9/20]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d1ce5f6b877b41c8b425e8577c77cf01",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=136, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3367. Accuracy: 0.857\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f8373a9fdf2342f3968a22c770c8a650",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=39, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Epoch [9/20]: Train accuracy: 0.858. Train loss: 0.3342. Evaluation accuracy: 0.862. Evaluation loss: 0.3298\n",
      "\n",
      "Start epoch [10/20]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ec221f5273264f10953475a7d951f539",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=136, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3215. Accuracy: 0.865\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "46b67f00979346dfa5ebb39c13fc4c18",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=39, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Epoch [10/20]: Train accuracy: 0.866. Train loss: 0.3189. Evaluation accuracy: 0.867. Evaluation loss: 0.3258\n",
      "\n",
      "Start epoch [11/20]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "585f1d200ef04de7bca37d5914f6793d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=136, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.3074. Accuracy: 0.871\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "82d82cd3a80f47d3809a979790f38bb5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=39, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Epoch [11/20]: Train accuracy: 0.872. Train loss: 0.3056. Evaluation accuracy: 0.869. Evaluation loss: 0.3243\n",
      "\n",
      "Start epoch [12/20]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f59db92a90d54735b8fc4baf44886c90",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=136, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.2999. Accuracy: 0.876\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f39ce5fef3e84045a7ecf9b0d463a87e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=39, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Epoch [12/20]: Train accuracy: 0.878. Train loss: 0.2975. Evaluation accuracy: 0.871. Evaluation loss: 0.3204\n",
      "\n",
      "Start epoch [13/20]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e9d851c4c15e48b0af81cb16ed57afb0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=136, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.2836. Accuracy: 0.885\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "34b4307980ec468a8881b448b56f31af",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=39, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Epoch [13/20]: Train accuracy: 0.886. Train loss: 0.2830. Evaluation accuracy: 0.871. Evaluation loss: 0.3191\n",
      "\n",
      "Start epoch [14/20]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4a04cf7fa485430b90885e6acacc1f40",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=136, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.2774. Accuracy: 0.888\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "77116f31a1664cfea785f1230280ca66",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=39, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Epoch [14/20]: Train accuracy: 0.889. Train loss: 0.2749. Evaluation accuracy: 0.873. Evaluation loss: 0.3178\n",
      "\n",
      "Start epoch [15/20]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a1d47279a42b4142b349f9f3ac09fc99",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=136, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.2652. Accuracy: 0.892\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b27b593e8895403d989adc7d3dd6c96a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=39, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Epoch [15/20]: Train accuracy: 0.894. Train loss: 0.2626. Evaluation accuracy: 0.873. Evaluation loss: 0.3185\n",
      "\n",
      "Start epoch [16/20]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "43f60d7423e14e22962ee88bb856588e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=136, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.2601. Accuracy: 0.894\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5ca9df0142c84ed1925bb9c2bd54ef0b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=39, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Epoch [16/20]: Train accuracy: 0.895. Train loss: 0.2582. Evaluation accuracy: 0.875. Evaluation loss: 0.3180\n",
      "\n",
      "Start epoch [17/20]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7a02084406fe47058445a012373849e0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=136, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.2493. Accuracy: 0.900\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b28de97a94d94f66af256069408e6de8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=39, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Epoch [17/20]: Train accuracy: 0.899. Train loss: 0.2485. Evaluation accuracy: 0.874. Evaluation loss: 0.3179\n",
      "\n",
      "Start epoch [18/20]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9b298615b19649bebf575ee5590c7fba",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=136, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.2424. Accuracy: 0.902\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0223d212a9f941039881b33a7c397eab",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=39, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Epoch [18/20]: Train accuracy: 0.903. Train loss: 0.2412. Evaluation accuracy: 0.875. Evaluation loss: 0.3181\n",
      "\n",
      "Start epoch [19/20]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8c569ef645c34a30bf9cf08256efe60c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=136, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.2381. Accuracy: 0.904\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "81bc142ca08b471d9de05924d3f5cc19",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=39, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Epoch [19/20]: Train accuracy: 0.906. Train loss: 0.2353. Evaluation accuracy: 0.877. Evaluation loss: 0.3180\n",
      "\n",
      "Start epoch [20/20]\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "537051da51a843a7b96e638ab9f59eb0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=136, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 100. Average training loss: 0.2330. Accuracy: 0.907\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3cea9268ce1f45758f7c5d0335acff68",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=39, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Epoch [20/20]: Train accuracy: 0.908. Train loss: 0.2314. Evaluation accuracy: 0.878. Evaluation loss: 0.3177\n"
     ]
    }
   ],
   "source": [
    "# Initialize parameters\n",
    "hidden_size = 8\n",
    "vocab_size = len(train_iterator.word2index)\n",
    "embedding_dim = 200\n",
    "output_size = 2\n",
    "n_layers = 1\n",
    "dropout = 0.5\n",
    "learning_rate = 0.001\n",
    "epochs = 20\n",
    "spatial_dropout = True\n",
    "\n",
    "# Check whether system supports CUDA\n",
    "CUDA = torch.cuda.is_available()\n",
    "\n",
    "model = BiGRU(hidden_size, vocab_size, embedding_dim, output_size, n_layers, dropout,\n",
    "              spatial_dropout, bidirectional=True)\n",
    "\n",
    "# Move the model to GPU if possible\n",
    "if CUDA:\n",
    "    model.cuda()\n",
    "\n",
    "model.add_loss_fn(nn.NLLLoss())\n",
    "\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n",
    "model.add_optimizer(optimizer)\n",
    "\n",
    "device = torch.device('cuda' if CUDA else 'cpu')\n",
    "\n",
    "model.add_device(device)\n",
    "\n",
    "# Instantiate the EarlyStopping\n",
    "early_stop = EarlyStopping(wait_epochs=1)\n",
    "\n",
    "train_losses_list, train_avg_loss_list, train_accuracy_list = [], [], []\n",
    "eval_avg_loss_list, eval_accuracy_list, conf_matrix_list = [], [], []\n",
    "\n",
    "for epoch in range(epochs):\n",
    "    \n",
    "    print('\\nStart epoch [{}/{}]'.format(epoch+1, epochs))\n",
    "    \n",
    "    train_losses, train_avg_loss, train_accuracy = model.train_model(train_iterator)\n",
    "    \n",
    "    train_losses_list.append(train_losses)\n",
    "    train_avg_loss_list.append(train_avg_loss)\n",
    "    train_accuracy_list.append(train_accuracy)\n",
    "    \n",
    "    _, eval_avg_loss, eval_accuracy, conf_matrix = model.evaluate_model(val_iterator)\n",
    "    \n",
    "    eval_avg_loss_list.append(eval_avg_loss)\n",
    "    eval_accuracy_list.append(eval_accuracy)\n",
    "    conf_matrix_list.append(conf_matrix)\n",
    "    \n",
    "    print('\\nEpoch [{}/{}]: Train accuracy: {:.3f}. Train loss: {:.4f}. Evaluation accuracy: {:.3f}. Evaluation loss: {:.4f}'\\\n",
    "          .format(epoch+1, epochs, train_accuracy, train_avg_loss, eval_accuracy, eval_avg_loss))\n",
    "    \n",
    "    if early_stop.stop(eval_avg_loss, model, delta=0.003):\n",
    "        break\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add the dataset initial loss\n",
    "train_avg_loss_list.insert(0, train_losses_list[0][0])\n",
    "eval_avg_loss_list.insert(0, train_losses_list[0][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfQAAAFQCAYAAABJW4xyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeXxU1f3/8dfJHsgCCSQBAiTsS0ICBgQERUABAakrgopal+Jetx9WbUVaWrV+61Zbl7ZqrVVQiyKKqKC4IgQN+74nhCRsIQSyn98fd8ABQiDJZCYJ7+fjMY/M3Hvn3M8k6HvOucsx1lpERESkYfPzdQEiIiJSewp0ERGRRkCBLiIi0ggo0EVERBoBBbqIiEgjoEAXERFpBBTo0mgZY7YaY4afZN1gY8w6b9fkTcaY14wxfzjNbav6XZ12O55mjJlrjLnOF/sWaWgU6HJGstZ+ba3t6r7MGHOVMeYHY0yhMSbX9fw2Y4xxrX/NGFNijDlojNlrjPnMGNPN7f1TjTH/OX5fxhhrjOlU95+q8bHWjrLWvu7rOkQaAgW6CGCMuQ94FvgzEAfEApOBc4Agt02ftNaGAW2ALOCfXi610TDGBPi6htpqDJ9BGg8FujR2fY0xq40x+4wxrxpjQgCMMUOMMZmu55HANOA2a+271toC6/jJWnu1tbb4+EattYeBmUBqbYpzDXU/YIxZ7hoZ+KcxJtY11FxgjPncGNPcbfuLjTGrjDH7jTFfGmO6u63rbYz50fW+GUDIcfsaY4zJcL33O2NMrxrWfNJ2jDEPGmM2uWpYbYy5xG3d9caYb40xTxtj9gJTXcu+McY85fobbTHGjHJ7z5fGmJvc3l/VtonGmK/cfm8vVDZi4rb9ONfnOOCqeaTb32S423ZHR16MMQmuEZcbjTHbgQXGmE+MMXcc1/YyY8ylrufdXKM5e40x64wxV7ptd5Hr91RgjMkyxtxfk7+JCCjQpfG7GhgBdAS6AI9Uss0AIBj44HQbNcY0BSYAGz1Q42XABa76xgJzgYeAFjj/jd7l2mcX4C3g10BL4GPgQ2NMkDEmCHgfeAOIAt5xtXuk3j7Av4BfAdHAS8BsY0xwdQo9jXY2AYOBSOAx4D/GmFZuTZwNbAZigOluy9a5Pu+TwD+PHOaoRFXb/hdY7KprKnBtFZ+jH/Bv4AGgGXAusPVUn9/NeUB3nH9b/8X5t3Ck7R5Ae+Aj17+Tz1zbxLi2+5sxpqdr838Cv7LWhgNJwIJq1CByDAW6NHZ/tdbusNbuxQmQCZVs0wLYba0tO7LA1fPcb4w5bIw5123b+40x+4ECYBBVhEY1PG+tzbHWZgFfAz+4RgeKgVlAb9d244GPrLWfWWtLgaeAUGAg0B8IBJ6x1pZaa98Flrjt42bgJWvtD9bactdx6WLX+6qjynaste9Ya3daayustTOADUA/t/fvtNY+b60tc41yAGyz1r5irS0HXgda4RzyqEyl2xpj2gF9gd9Za0ustd8As6v4HDcC/3L9LiustVnW2rXV+D1MtdYWuj7DLCDVGNPete5q4H+uv98YYKu19lXXZ/4ReA+43LVtKdDDGBNhrd3nWi9SIwp0aex2uD3fBrSuZJs9QAv346HW2oHW2maude7/nTzlWp4AHAbcT6wrwwnVo4wxR16XVlFjjtvzw5W8DnM9b+36DEdqrMD5fG1c67LssbMtbXN73h64z/UlZb/rS0lbKv99VKXKdowxk9yG4/fj9DpbuL1/x4lNssvtMx1yPQ2rZLuqtm0N7HVbdrJ9HdEWZzShpo62ba0tAD4CrnItugp40/W8PXD2cb+vq3HO0wBnFOUiYJsxZqExZkAtapIznAJdGru2bs/bATsr2eZ7nF7muNNt1Fq7HbgbeNYYE+pavB0n6N0lAuU4J9DV1k6cgADANdTc1tV2NtDmuKHqdm7PdwDTrbXN3B5NrLVvVbOGk7bj6qG+AtwBRLu++KwE3Guqq+kds4EoY0wTt2VtT7YxzufoeJJ1hYB7O3GVbHP853gLmOAK5FDgC7f9LDzu9xVmrb0VwFq7xFo7Dmc4/n2c8zJEakSBLo3d7caYeGNMFM5x6RnHb2Ct3Y9zvPdvxpjLjTFhxhg/Y0wq0PRkDVtrP8MJ2Vtciz4BuhpjrjXGBLr2+UfgXffh/FqYCYw2xgxz9fzvw/ki8h3Ol5Iy4C5jTIDrhCz3oe5XgMnGmLONo6kxZrQxJryaNVTVTlOcoMsDMMbcgNNDr3PW2m1AOs6JdkGuYB1bxVv+Cdzg+l36GWPamJ8vQcwArnL9DdP4eXi8Kh/jfNmaBsxwjZ4AzAG6uP2bCDTG9DXGdHfVebUxJtJ1COUAzpc/kRpRoEtj91/gU5wTsTYDld4gxVr7JHAv8P+AXJxh75eAKTiBeTJ/Bv6fMSbYWpuLM3z6K1cbK4F84FZPfBBr7TrgGuB5YDdOYI11HTMuAS4Frgf24Rxv/5/be9Nxjn//1bV+o2vb6tZw0nastauB/8P5cpEDJAPfVncftXA1zgmOe3D+zjNwvvCcwFq7GLgBeBrnb7SQn0c/fovTe9+H80Xvv6faset4+f+A4e7bu4bjL8QZht+Jc8jgCZyTMME5B2OrMeYAzmWS15zuhxU5njn2kJuISONgnEv31lprH/V1LSLeoB66iDQKrqHsjq4h9JE450S87+u6RLxFdzkSkcYiDmfYOxrIBG611v7k25JEvEdD7iIiIo2AhtxFREQaAQW6iIhII9Cgj6G3aNHCJiQk+LoMERERr1i6dOlua23LytY16EBPSEggPT3d12WIiIh4hTFm28nWachdRESkEVCgi4iINAIKdBERkUagQR9DFxGRqpWWlpKZmUlRUZGvS5FqCAkJIT4+nsDAwFNv7KJAFxFpxDIzMwkPDychIYFjZ9eV+spay549e8jMzCQxMfG036chdxGRRqyoqIjo6GiFeQNijCE6OrraoyoKdBGRRk5h3vDU5G+mQBcRkTqzZ88eUlNTSU1NJS4ujjZt2hx9XVJSclpt3HDDDaxbt67KbV544QXefPNNT5TMoEGDyMjI8Ehb3qRj6CIiUmeio6OPhuPUqVMJCwvj/vvvP2Ybay3WWvz8Ku9jvvrqq6fcz+233177Yhs49dBdsrJ28OOMP5Cbf9jXpYiINHobN24kKSmJyZMn06dPH7Kzs7nllltIS0ujZ8+eTJs27ei2R3rMZWVlNGvWjAcffJCUlBQGDBhAbm4uAI888gjPPPPM0e0ffPBB+vXrR9euXfnuu+8AKCws5LLLLiMlJYUJEyaQlpZ2yp74f/7zH5KTk0lKSuKhhx4CoKysjGuvvfbo8ueeew6Ap59+mh49epCSksI111zj8d/ZqaiH7rL96zcZsPbP3LqshKzWFzK0WwzDusWS1CZCx59EROrA6tWrefXVV3nxxRcBePzxx4mKiqKsrIzzzz+fyy+/nB49ehzznvz8fM477zwef/xx7r33Xv71r3/x4IMPntC2tZbFixcze/Zspk2bxieffMLzzz9PXFwc7733HsuWLaNPnz5V1peZmckjjzxCeno6kZGRDB8+nDlz5tCyZUt2797NihUrANi/fz8ATz75JNu2bSMoKOjoMm9SoLv0v+I+iv42mz8XvMUtnMuz8zfwzOcbiI0IZmi3GIZ2i+WcTtE0CdKvTEQapsc+XMXqnQc82maP1hE8OrZnjd7bsWNH+vbte/T1W2+9xT//+U/KysrYuXMnq1evPiHQQ0NDGTVqFABnnXUWX3/9daVtX3rppUe32bp1KwDffPMNU6ZMASAlJYWePauu+4cffmDo0KG0aNECgIkTJ/LVV18xZcoU1q1bx913381FF13EhRdeCEDPnj255pprGDduHL/4xS+q+duoPQ25uxj/QEJ+8QxhJbn8t8tC0h8ezv9dkUJa+yg+XJbNzf9OJ3XaZ1z/6mLeWLSNrP0amhcRqY2mTZsefb5hwwaeffZZFixYwPLlyxk5cmSll20FBQUdfe7v709ZWVmlbQcHB5+wjbW2WvWdbPvo6GiWL1/OoEGDeO655/jVr34FwLx585g8eTKLFy8mLS2N8vLyau2vttTddNe2H/S+Fhb9jejUiVx2VncuOyuekrIKlmzdy/w1ucxfm8Nv31/Jb4FuceEM7x7L0O4xpMQ3w99PQ/MiUn/VtCftDQcOHCA8PJyIiAiys7OZN28eI0eO9Og+Bg0axMyZMxk8eDArVqxg9erVVW7fv39/HnjgAfbs2UNkZCRvv/02999/P3l5eYSEhHDFFVeQmJjI5MmTKS8vJzMzk6FDhzJo0CDefPNNDh06RHh4uEc/Q1UU6Mcb/hisnQMf3Q/XzwFjCArw45xOLTinUwt+O6Y7m3cXMn9NDvPX5PL3hZv46xcbiW4axJCuMQzvHsOgzi0IDzn92/WJiJzp+vTpQ48ePUhKSqJDhw6cc845Ht/HnXfeyaRJk+jVqxd9+vQhKSmJyMjIk24fHx/PtGnTGDJkCNZaxo4dy+jRo/nxxx+58cYbsdZijOGJJ56grKyMiRMnUlBQQEVFBVOmTPFqmAOY6g5B1CdpaWm2TuZDT38V5vwaLn0Fel1Z5ab5h0pZuCGP+Wty+HJdHvmHSwn0N5ydGM2w7jFc1bcdoUH+nq9RROQ0rFmzhu7du/u6jHqhrKyMsrIyQkJC2LBhAxdeeCEbNmwgIKB+9m0r+9sZY5Zaa9Mq275+fgpf63Md/PQGzHsYuoyAkJN/g4tsEsjFKa25OKU1ZeUV/Lh9P/PX5rBgTS6Pfbia5Zn5PD0+1YvFi4hIZQ4ePMiwYcMoKyvDWstLL71Ub8O8JhrPJ/EkPz8Y/X/w8vnwxR9h1BOn9bYAfz/6JUbRLzGK34zqztOfrefZ+RsY0TOWkUmt6rhoERGpSrNmzVi6dKmvy6gzOsv9ZFr3hr43wuKXIXt5jZq4Y2gnkttE8tCsleQVFHu4QBERkZ8p0Ksy9BEIjYKP7oOKimq/PdDfj79cmcLB4jIemrWi2pdMiIiInC4FelVCm8OFv4fMxZBRs5v+d44N54ELu/LZ6hze+zHLwwWKiIg4FOinkjIB2g2Azx+FQ3tr1MQvByXSLyGKx2av0g1pRESkTijQT8UY5wS5w/th/rRTb18Jfz/DU1ekUG4tD7yzjIoKDb2LyJnD39//6JSpqampPP744zVqZ8iQIdT0UuUvv/zy6CQtAC+++CL//ve/a9SWu61bt5KUlFTrdjzBa2e5G2NGAs8C/sA/rLWPH7f+aeB818smQIy1tpm36qtSbE84ezIs+hv0uRbanFXtJtpFN+GR0T14aNYK/v39Vq4/J9HzdYqI1EOhoaE+n1/8yy+/JCwsjIEDBwIwefJkn9ZTF7zSQzfG+AMvAKOAHsAEY8wxd9y31t5jrU211qYCzwP/80Ztp23IgxAWC3PuhYqa3Z93Qr+2DOnaksc/WcvmvIMeLlBEpOGYO3cuV1758427vvzyS8aOHQvArbfeenQa1UcffbTS94eFhR19/u6773L99dcD8OGHH3L22WfTu3dvhg8fTk5ODlu3buXFF1/k6aefJjU1la+//pqpU6fy1FNPAZCRkUH//v3p1asXl1xyCfv27QOcEYEpU6bQr18/unTpctKJYI4oKirihhtuIDk5md69e/PFF18AsGrVKvr160dqaiq9evViw4YNFBYWMnr0aFJSUkhKSmLGjBk1+0W68daQez9go7V2s7W2BHgbGFfF9hOAt7xS2ekKiYAR0yE7A5a+WqMmjDE8cVkvggP8uXfmMsrKq3/mvIhIQ3P48OFjhtxnzJjBBRdcwKJFiygsLARgxowZjB8/HoDp06eTnp7O8uXLWbhwIcuXn/6lw4MGDWLRokX89NNPXHXVVTz55JMkJCQwefJk7rnnHjIyMhg8ePAx75k0aRJPPPEEy5cvJzk5mccee+zourKyMhYvXswzzzxzzPLKvPDCCwCsWLGCt956i+uuu46ioiJefPFF7r77bjIyMkhPTyc+Pp5PPvmE1q1bs2zZMlauXOmR+9Z7a8i9DbDD7XUmcHZlGxpj2gOJwIKTrL8FuAWgXbt2nq3yVJIugx9fd46ldx8HYS2r3URsRAjTxvXk7rczeOmrzdx+fqc6KFREpBJzH4RdKzzbZlwyjKr6mPjJhtxHjhzJhx9+yOWXX85HH33Ek08+CcDMmTN5+eWXKSsrIzs7m9WrV9OrV6/TKiczM5Px48eTnZ1NSUkJiYlVH97Mz89n//79nHfeeQBcd911XHHFFUfXVzYN68l888033HnnnQB069aN9u3bs379egYMGMD06dPJzMzk0ksvpXPnziQnJ3P//fczZcoUxowZc8KXjJrwVg+9smnITnZm2FXAu9baSse1rbUvW2vTrLVpLVtWP1BrxRi46P+g5JBz1nsNXZzSmtHJrXjm8/Uen5tYRKShGD9+PDNnzmTBggX07duX8PBwtmzZwlNPPcX8+fNZvnw5o0ePrnQaVWN+jhX39XfeeSd33HEHK1as4KWXXqr0vdVR2TSsJ3Oye41MnDiR2bNnExoayogRI1iwYAFdunRh6dKlJCcn85vf/IZp02p20rU7b/XQM4G2bq/jgZ0n2fYq4PY6r6imWnaBgXfAN09Dn0nQrn+1mzDG8PtfJLF4617unZnBB3ecQ3CAJnARkTp2ip60tw0ZMoQbb7yRV1555ehw+4EDB2jatCmRkZHk5OQwd+5chgwZcsJ7Y2NjWbNmDV27dmXWrFlHZzbLz8+nTZs2ALz++utHtw8PD+fAgRM7UJGRkTRv3pyvv/6awYMH88YbbxztrVfXueeey5tvvsnQoUNZv34927dvp2vXrmzevJkOHTpw1113sXnzZpYvX063bt2IiorimmuuISwsjNdee61G+3TnrR76EqCzMSbRGBOEE9qzj9/IGNMVaA5876W6aubcByCyrXMHufKqv7GdTFTTIJ64LJm1uwp45vMNHi5QRKT+OP4Y+oMPPgg4vd4xY8Ywd+5cxowZA0BKSgq9e/emZ8+e/PKXvzzpNKqPP/44Y8aMYejQobRq9fNcGVOnTuWKK65g8ODBtGjR4ujysWPHMmvWrKMnxbl7/fXXeeCBB+jVqxcZGRn87ne/q9HnvO222ygvLyc5OZnx48fz2muvERwczIwZM0hKSiI1NZW1a9cyadIkVqxYcfREuenTp/PII4/UaJ/uvDZ9qjHmIuAZnMvW/mWtnW6MmQakW2tnu7aZCoRYax88nTbrbPrU07HmQ5hxDYz4Ewy4rcbNTHl3Oe8s3cE7kwdwVvsoDxYoIqLpUxuy6k6f6rUby1hrP7bWdrHWdrTWTnct+92RMHe9nnq6Ye5z3cZApwuc2dgOZNe4mUfGdKdVZCj3zVzGoZKa9fZFRER0p7iaMgYuehLKS+DTmg+VhIcE8tQVKWzdc4jH5671YIEiInImUaDXRlQHGHQPrHwXNi+scTMDOkbzy3MS+ff32/hmw24PFigiImcKBXptDfo1NE+Aj++HspIaN/P/RnalY8umPPDuMvIPl3quPhE542nq5oanJn8zBXptBYbCqD/D7vXw/V9r3ExIoD9/uTKV3IJiHvtwlQcLFJEzWUhICHv27FGoNyDWWvbs2UNISEi13ue1yVkatS4XOifJffVnSL4CmrU99XsqkdK2GbcP6chzCzYyomccI3rGebhQETnTxMfHk5mZSV5enq9LkWoICQkhPj6+Wu/x2mVrdcGnl60db/92+Gs/6DQMrnqzxs2UlFVwyd++ZVd+EfPuOZcWYcEeLFJERBqyenHZWqPXrB2c9wCsnQMbPqtxM0EBfvzlylQKisp4eNYKDZOJiMhpUaB70oA7IbozfPwAlNb8/sFd48K578IuzFuVw6yfsjxYoIiINFYKdE8KCILRT8G+LfDtM7Vq6qbBHeib0JxHZ69i5/7DHipQREQaKwW6p3UYAj0vha//Ans317gZfz/DU1ekUF5hmfLecg29i4hIlRTodWHEdPAPhLlToBZB3D66KQ+P7s7XG3bzn0XbPFigiIg0Ngr0uhDRGob8BjZ8Cuvm1qqpif3acV6Xlvzx47Vs2V3ooQJFRKSxUaDXlbN/BdGdYP5jUFFe42aMMTxxWS8C/Q33zcygvEJD7yIiciIFel3xD4Shj0DeWlg+s1ZNxUWG8PtfJPHj9v28/FXNj8uLiEjjpUCvS93HQatUZ4rVsuJaNXVxSmtG9Izl2fnrydJZ7yIichwFel3y84Nhv4P87bD0tVo1ZYzhd2N7AjD9o9UeKE5ERBoTBXpd6zgUEgbDwieh+GCtmmrTLJQ7zu/Exyt2aZpVERE5hgK9rhkDwx6FQ7th0d9r3dxNgzvQProJj85eSUlZhQcKFBGRxkCB7g1t+zqzsX33HBzaW6umQgL9eXRsDzblFfLad1s8VKCIiDR0CnRvGfoIFBfAN3+pfVPdYhnWLYZnP99AzoGa3zNeREQaDwW6t8R0h5QJ8MPLkF/7CVd+N7YHpeWWP328xgPFiYhIQ6dA96YhD4KtgIVP1Lqp9tFN+dV5HXg/Yyc/bN7jgeJERKQhU6B7U/P20PdG+Ok/sHtjrZu7bUgn2jQL5dHZqygr1wlyIiJnMgW6tw2+HwJC4Is/1Lqp0CB/fjumO2t3FfDmD9s9UJyIiDRUCnRvC2sJA26HVbNgZ0atmxvRM47BnVvwf5+uY/fB2t2NTkREGi4Fui8MvANCm8P8abVuyhjDo2N7cqiknCc/WeuB4kREpCFSoPtCSCQMuhc2zYctX9e6uU4xYdw4KJGZ6Zn8tH2fBwoUEZGGRoHuK/1uhvDWzvSqtvZTot45rDMx4cH87oNVmmJVROQMpED3lcBQGDIFMpfAurm1bi4sOICHR3dnRVY+M9N3eKBAERFpSBTovpR6DUR3co6lV5TXurmLU1rTLzGKJz9Zy/5DJR4oUEREGgoFui/5B8D5D0PeGljxTq2bM8bw2MU9OVBUxv99ut4DBYqISEOhQPe1Hr+AVinwxXQoq32vunurCK7t3543f9jGyqx8DxQoIiINgQLd1/z8YNjvYP92WPqaR5q854IuNG8SxKOzV1GhE+RERM4ICvT6oOMwSBgMXz0JxQdr3VxkaCBTRnVj6bZ9zPqp9hPBiIhI/adArw+MgWGPQmEe/PB3jzR5eZ94Uts2409z13KgqNQjbYqISP2lQK8v2vaFrqPh2+fg0N5aN+fnZ5g2rid7Cot59vMNHihQRETqMwV6fTL0ESgugG+e9khzveKbcVXfdrz23VbW5xR4pE0REamfFOj1SWwPSLkKFr8MB3Z6pMkHRnQlPCSARz9YhfXAHelERKR+UqDXN0N+49xkZuETHmkuqmkQ91/Yle837+GjFdkeaVNEROofBXp907w9pP0SfnwD9mzySJMT+rWjZ+sIpn+0hsLiMo+0KSIi9YvXAt0YM9IYs84Ys9EY8+BJtrnSGLPaGLPKGPNfb9VW75x7PwSEwII/eKQ5f9cJctn5Rfz1i40eaVNEROoXrwS6McYfeAEYBfQAJhhjehy3TWfgN8A51tqewK+9UVu9FBYDA26DVf+D7GUeafKs9lFc1ieef3y9mc15tb/WXURE6hdv9dD7ARuttZuttSXA28C447a5GXjBWrsPwFqb66Xa6qeBd0Joc2fiFg+ZMqorIQH+TP1wtU6QExFpZLwV6G0A9zk9M13L3HUBuhhjvjXGLDLGjPRSbfVTSCQMuhc2fg5bv/FIkzHhIfz6gi58tT6Pz1bneKRNERGpH7wV6KaSZcd3EQOAzsAQYALwD2NMsxMaMuYWY0y6MSY9Ly/P44XWK/1uhvDW8Plj4KEe9aQB7ekSG8a0OaspKq39lK0iIlI/eCvQM4G2bq/jgeMvtM4EPrDWllprtwDrcAL+GNbal621adbatJYtW9ZZwfVCYCgMmQKZi2H9J55p0t+PqRf3JHPfYV5c6Jmz6EVExPe8FehLgM7GmERjTBBwFTD7uG3eB84HMMa0wBmC3+yl+uqv1GsgqqNzLL3CMz3qgR1bMKZXK/7+5SZ27D3kkTZFRMS3vBLo1toy4A5gHrAGmGmtXWWMmWaMudi12TxgjzFmNfAF8IC1do836qvX/AOcW8LmroYV73qs2YdHd3cuZ5uz2mNtioiI75iGfLZzWlqaTU9P93UZda+iAl4+D4ry4Y50CAjySLMvfLGRP89bx/9uG0ifds090qaIiNQdY8xSa21aZet0p7iGwM8Phj8K+7fBj697rNnrByYQ1TRIs7GJiDQCCvSGouMwaHs2LPq7x854bxocwE2DE1m4Po+MHfs90qaIiPiGAr2hMAb6TIK9m2DHYo81O2lAAs2aBPLs5+s91qaIiHifAr0h6TEOApvAMs/d5j4sOICbB3fgi3V5LFMvXUSkwVKgNyTB4dB9LKycBaWHPdbspAHtadYkkOfm61i6iEhDpUBvaFImQHE+rPvYY02GhwRy06BE5q/NZUVmvsfaFRER71GgNzSJ50JEG8h4y6PNXjcwgcjQQJ5VL11EpEFSoDc0fv7Qazxsmg8FuzzWbHhIIDcOSuTzNTmszFIvXUSkoVGgN0SpE8FWwPKZHm32+nMSiAgJ0LF0EZEGSIHeELXoDG3SYNlbHrsmHSAiJJBfDkrk09U5rNqpXrqISEOiQG+oUic493fPXubRZm84J5Fw9dJFRBocBXpD1fNS8A9yeukeFBkayA3nJDJvVQ5rsg94tG0REak7CvSGqkkUdB0FK96BshKPNn3jOYmEB6uXLiLSkCjQG7KUiXBoD2z8zKPNRjYJ5PpzEpi7chdrd6mXLiLSECjQG7JOw6BpS8jw3K1gj7hxUCJhwQE8P3+jx9sWERHPU6A3ZP6BkHwlrJ8Hh/Z6tOlmTYK4bmB7Pl6ZzfqcAo+2LSIinqdAb+hSJ0BFKax41+NN3zSoA00C/XUsXUSkAVCgN3RxyRCb7NEZ2I5o3jSI6wYm8NGKbDaoly4iUq8p0BuD1Amw8yfIXevxpm8a3IHQQH+eX6Bj6SIi9ZkCvTFIvgKMv8evSQeIahrEpAEJfLh8JxtzD3q8fRER8QwFemMQFgOdL4DlM6Ci3OPN3zw4kU+thZ8AACAASURBVJAAf/66QMfSRUTqKwV6Y5EyAQqyYfOXHm86OiyYSQPaM3vZTjbnqZcuIlIfKdAbiy4jISSyTobdAW4+twPBAf78VcfSRUTqJQV6YxEYAkmXwZo5UOT5u7u1CAvmmv7teD8jiy27Cz3evoiI1I4CvTFJmQhlh2H1+3XS/C3ndiQowE+9dBGRekiB3pjEp0F0J8iom2H3luHBXH12e97PyGLbHvXSRUTqEwV6Y2KMc3Lc9u9g75Y62cWvzutAgJ9RL11EpJ5RoDc2KVcBBpa9XSfNx4SHMPHsdvzvpyy27zlUJ/sQEZHqU6A3NpHxkHiuc7Z7RUWd7GLyeR3x9zP89Qtdly4iUl8o0Buj1Imwfxts/75Omo+NCGFiv3b878csduxVL11EpD5QoDdG3cdCUFidTNhyxK1DOuLnZ3jhCx1LFxGpDxTojVFQU+gxDlZ9ACV104OOjQhhQt+2vLs0U710EZF6QIHeWKVMgJICWDunznYxeUhH/Izhb19uqrN9iIjI6VGgN1btz4HIdpBRd8PurSJDGd+3Le8u3UHW/sN1th8RETk1BXpj5efnXMK2+UvIz6qz3dw6pCMAf9OxdBERn1KgN2YpVwHWmVa1jrRuFsqVaW2Zmb6Dneqli4j4jAK9MYvuCG37O9ekW1tnu7nt/E4A/F3H0kVEfEaB3tilToDd6yHrxzrbRZtmoVx+VltmLNlBdr566SIivqBAb+x6XgIBIXV6TTrAbUM6UmEtL6qXLiLiEwr0xi4kErqNhpXvQVlxne2mbVQTLj8rnrfUSxcR8QmvBboxZqQxZp0xZqMx5sFK1l9vjMkzxmS4Hjd5q7ZGL2UiHN4H6+fV6W5udx1L/8OcNXW6HxEROZFXAt0Y4w+8AIwCegATjDE9Ktl0hrU21fX4hzdqOyN0PB/C4pyT4+pQ26gm3Hl+Jz5akc0X63LrdF8iInIsb/XQ+wEbrbWbrbUlwNvAOC/tW/z8odeVsOFTKNxdp7u65bwOdGzZlN++v5LDJeV1ui8REfnZaQe6MeZeY0yq63l/Y8x2Y8xmY8yA03h7G2CH2+tM17LjXWaMWW6MedcY0/Z0a5PTkDIBKspgxTt1upvgAH+mX5JM5r7DPLdA06uKiHhLdXro9wBbXM//BPwFmA48cxrvNZUsO/7C6A+BBGttL+Bz4PVKGzLmFmNMujEmPS8v77QKFyC2B7RKqdNbwR7Rv0M0l58VzytfbWbdroI635+IiFQv0COttfnGmHAgBXjeWvtPoOtpvDcTcO9xxwM73Tew1u6x1h45DfsV4KzKGrLWvmytTbPWprVs2bIa5QspE2HXcshZVee7euii7oSHBPDQrBVUVNTdTW1ERMRRnUDfYYwZCFwFfGWtLTfGRACnc6B0CdDZGJNojAlytTHbfQNjTCu3lxcDOlXa05IvB78Ar/TSo5oG8dBF3Vm6bR8z0nec+g0iIlIr1Qn0B4B3gYeB37uWjQEWn+qN1toy4A5gHk5Qz7TWrjLGTDPGXOza7C5jzCpjzDLgLuD6atQmp6NpC+g8ApbPhPKyOt/d5WfFc3ZiFI/PXcvug3V3DbyIiICxtbjHtzEmEMBaW+qxiqohLS3Npqen+2LXDdeaD2HGNTDxHehyYZ3vbmPuQUY9+xVjerXm6fGpdb4/EZHGzBiz1FqbVtm66pzl3sMYE+t6HmaMeQz4DRDomTLFKzqPgNCoOr8V7BGdYsKYfF5HZv2Uxbcb6/aSORGRM1l1htz/CzRzPX8KOBcYALzk6aKkDgUEOcfS137s3D3OC24/vxPto5vwyPsrKSrVtekiInWhOoGeYK1dZ4wxwCXAFcDlwIg6qUzqTsoEKC+GVbO8sruQQH/+8Isktuwu5G+avEVEpE5UJ9CLXZes9QN2WGt3A8VASJ1UJnWndW9o2Q0y6vZWsO4Gd27JuNTWvPjlJjblHfTafkVEzhTVHXJfgHPDl9dcy/rw881mpKEwxumlZy6G3Ru9tttHRvcgJNCPh2etoDYnY4qIyIlOO9CttffgXLJ2q7X2r67FFTh3kJOGptd4MH51PmGLu5bhwUwZ1Y1Fm/fyvx+zvLZfEZEzQbUmZ7HWfgpsMsYMMMa0s9amW2sX1FFtUpciWkGH82HZ21DuvasOJ/RtR592zZj+8Rr2FZZ4bb8iIo1ddS5ba2WMWQhsAP4HbDTGLDTGtK6z6qRunT0ZDmTCkn96bZd+fobplySTf7iUx+eu9dp+RUQau+r00P8OLAOirLWtgOZABvBiXRQmXtD5AugwBL78Exza67Xddm8VwU2DEpmRvoPFW7y3XxGRxqw6gT4IuM9aWwjg+vn/gIF1UZh4gTFw4XQoyoevnvLqru8e3pk2zUJ5aNYKSsoqvLpvEZHGqDqBvg/ocdyyrsB+z5UjXheXBH2uhcUvwx7vXSPeJCiAaeN6sjH3IK98vdlr+xURaayqE+hPAp8bYx43xtxqjHkc+My1XBqy8x+BgGD47Hde3e2w7rGMSorjufkb2Lan0Kv7FhFpbKpz2dorwHigBTDW9fNanLnNpSELj4VB98DaObD1G6/u+tGxPQn09+O3H6zStekiIrVQ3cvWFlhrb7LWXmStvQlYiHNtujR0A26HiHiY9xBUeO+YdlxkCPdd2IWv1ucxZ3m21/YrItLYVCvQT8J4oA3xtcBQGD4VspfB8re9uutJAxJIbhPJtDmryT/sk5l4RUQaPE8EusZJG4uky6DNWTB/GpR475i2v5/hj5cks+dgMX+ep2vTRURq4pSBbowZerIHcL4XahRv8fODEX+Egmz47nmv7jo5PpJJAxJ484ft/LTdO9O6iog0JuZUJyIZY045+Yq1NtFjFVVDWlqaTU9P98WuG7eZ18GGT+HOpRDhvRsBFhSVcsFfvqJ50yA+vOMcAvw9MYAkItJ4GGOWWmvTKlt3yv9jWmsTT/XwfMniU8OnQkUZLPiDV3cbHhLI1It7sCb7AK9+u9Wr+xYRaejUBZITRSU693nP+C/szPDqrkf0jGNYtxj+8tl6svYf9uq+RUQaMgW6VO7c+6FJFHz6CHjx+nBjDI+N6wnAox+s1LXpIiKnSYEulQuJhPMfgq1fw7qPvbrr+OZN+PXwzny+Jpd5q3K8um8RkYZKgS4n1+d6aNHV6aWXeXfu8l8OSqRbXDhTZ6/iYHGZV/ctItIQKdDl5PwDYMR02LsZlvzDq7sO9Pdj+iXJ5BQU8eB7yykr14xsIiJVUaBL1ToNh45DYeETXp0zHeCs9s2ZMrIbc5Znc/eMDEoV6iIiJ6VAl6oZAxf+AYoPwELvT6w3+byOPHxRdz5ans0d//1Rc6eLiJyEAl1OLbYn9JkES16B3Ru9vvubz+3Ao2N7MG9VDre9uZTisnKv1yAiUt8p0OX0nP8wBIR6fc70I244J5Hf/yKJz9fkcsu/l1JUqlAXEXGnQJfTExYDg++FdR/Blq98UsK1/dvzxGXJfLUhj5teT+dwiUJdROQIBbqcvv63QWQ715zpvgnT8X3b8dTlKXy3aTc3vLaYQl3SJiICKNClOgJDYPijsGsFLHvLZ2VcdlY8T49PZcnWfVz3r8UUFGkOdRERBbpUT9JlEN8X5v8eig/6rIxxqW147qreZOzYz6R/LSb/sEJdRM5sCnSpHmOcOdMP7oLvnvNpKaN7teKFq/uwMiufa//5A/sPefdudiIi9YkCXaqvbT/oeSl8+xzkZ/m0lBE943jxmrNYm13AxFd+YG+hQl1EzkwKdKmZ4VPBVsCC3/u6EoZ1j+WV69LYlHeQCS8vYvfBYl+XJCLidQp0qZnm7aH/rc7JcVk/+roazuvSkn9d35dtewu56uVF5B4o8nVJIiJepUCXmht8HzRp4fU500/mnE4teO2Gfuzcf5irXl7ErnyFuoicORToUnMhETD0Ydj2Layd4+tqAOjfIZp//7IfuQXFjH/5e7L2H/Z1SSIiXqFAl9rpPQladodPfwtl9ePYdVpCFG/c2I+9hSWMf+l7duw95OuSRETqnAJdasc/AEb8AfZtgcWv+Lqao3q3a85/b+pPQVEZ41/6nq27C31dkohInVKgS+11Gu48Fj4JhXt8Xc1RyfGR/PfmszlcWs74l79nU57vboQjIlLXvBboxpiRxph1xpiNxpgHq9jucmOMNcakeas28YAL/wAlBbDwCV9XcoyerSN5+5YBlFdYxr+0iA05Bb4uSUSkTngl0I0x/sALwCigBzDBGNOjku3CgbuAH7xRl3hQTHc463pY8g/Y+q2vqzlG17hw3r6lP8bAVS8vYk32AV+XJCLicd7qofcDNlprN1trS4C3gXGVbPd74ElA1xs1ROc/DFGJ8MYvIMN3k7dUplNMODNu6U+gvx9XvPg97y3NxNaDS+1ERDzFW4HeBtjh9jrTtewoY0xvoK21tsrrn4wxtxhj0o0x6Xl5eZ6vVGquaQu48TNoeza8P9mZwKWiwtdVHdWhZRj/u20gPVpHcN87y7jjrZ/IP6RJXUSkcfBWoJtKlh3tHhlj/ICngftO1ZC19mVrbZq1Nq1ly5YeLFE8okkUXDsL+kyCr5+Cd2+Akvpz2VjrZqG8dXN//t/IrsxbuYuRz37Fd5t2+7osEZFa81agZwJt3V7HAzvdXocDScCXxpitQH9gtk6Ma6D8A2Hsc86Jcqs/gNdGQ0GOr6s6yt/PcNuQTsy67RxCA/25+h8/8Ke5aygpqz+jCSIi1eWtQF8CdDbGJBpjgoCrgNlHVlpr8621Lay1CdbaBGARcLG1Nt1L9YmnGQMD74Sr3oS8tfDKUNi1wtdVHSM5PpI5dw1iQr92vLRwM5f87Vs25uoseBFpmLwS6NbaMuAOYB6wBphprV1ljJlmjLnYGzWIj3QbDb/8xJmZ7V8jYd0nvq7oGE2CAvjjJcm8MimN7PwiRj/3DW98v1UnzIlIg2Ma8v+40tLSbHq6OvENwoFseOsq2LUcLpzuzNRmKju1wndyC4p44J3lLFyfx9BuMTxxWS9ahgf7uiwRkaOMMUuttZUejtad4sQ7IlrBDR87PfZ5v4E590B5/TrDPCY8hNdu6MvUsT34ZuNuRj37FQvW1p9j/yIiVVGgi/cENYUr/g2D7oGlr8KbV8Dh/b6u6hjGGK4/J5E5dw6iRVgwv3wtnd++v5LDJeW+Lk1EpEoKdPEuPz8YPhXGvQBbv4F/XgB7N/u6qhN0iQ3ngzvO4aZBibyxaBtjnv+alVn5vi5LROSkFOjiG72vgUnvQ2EevDIMtn3v64pOEBzgzyNjevCfG8/mYHEZl/ztW15cuImKioZ73omINF4KdPGdhEFw03znZjT/vhiWve3riio1qHMLPrn7XIZ3j+XxuWuZ+I9F7Nx/2NdliYgcQ4EuvhXd8efbxc76Vb27XewRzZsG8ber+/DkZb1YnpnPyGe+4sNlO0/9RhERL1Ggi+9VdrvY0vrXAzbGcGXftnx812A6tAzjzrd+4t6ZGRQU1a+z9UXkzKRAl/qhnt8u1l1Ci6a8M3kAdw3rzPs/ZTHq2a+Z9VMm5Tq2LiI+pECX+sP9drG5a1y3i13p66oqFejvx70XdOGdyQMICw7gnhnLuODphXyQkaVgFxGfUKBL/XPM7WJHwOJXoPigr6uq1Fnto/j4rsH8/eo+BPr5cffbGVyoYBcRH9CtX6X+OpAN71wPOxZBcASkTIC+N0HLLr6urFIVFZZPVu3imc/Xsz7nIJ1jwrhrWGdGJ7fCz69+3eZWRBqmqm79qkCX+s1ayFzi9NJXzYKKUugwBPreDF1Ggn+Arys8QUWF5eOV2Tz7+QY25B6kS2wYdw/rwqikOAW7iNSKAl0ah4O58OPrkP4qHMiCiHhIuwH6XAdhLX1d3QnKKywfrcjm2c/XsymvkK6x4dw9vDMjeyrYRaRmFOjSuJSXwfq5Tq99y0LwD4Kelzi99vi0ejeLW3mFZc7ynTw7fwOb8wrpFhfOr4d35sIeCnYRqR4FujReeetgyT8g4y0oKYBWqdDvZki6DAJDfV3dMcorLB8u28lz8zeweXch3VtFuII9FlPPvoSISP2kQJfGr7gAls+Axf+AvDUQ2ty5X3zajRCV6OvqjlFWXsFsV7Bv3XOIHq5gv0DBLiKnoECXM4e1zixuS16BNXOcS986X+j02jsOc2Z7qyfKyiv4IGMnzy3YwLY9h0hqE8Gvh3VhWPcYBbuIVEqBLmemAzth6WvO42AONE+EvjdCykRoGu3r6o4qK69g1k9ZPL9gI9v3HiK5TSSTz+vIhT1jCfSvP19ARMT3FOhyZisrgbUfOifRbXdN09qiC7TtB/H9nJ8tuvq8915aXsGsH7P46xdOsMdFhHD12e2YcHY7WoQF+7Q2EakfFOgiR+xaCes/ca5t37EYDu91lgdHQvxZroDvC23SILSZT0osr7B8sTaX17/fytcbdhPk78eYXq24bmACKW19U5OI1A8KdJHKWAt7NkHmYifcM5dA7mrnuDsGWnaF+L7O1K5t+0F0Z6/34jfmHuSN77fy7tJMCkvKSW3bjOsGtuei5FYEB/h7tRYR8T0FusjpKi6ArKWwY4kT9JlL4PA+Z11IpBPw7r34kAivlFVQVMp7SzP59/fb2Ly7kBZhQUzs146r+7cnNiLEKzWIiO8p0EVqylrYsxF2/ODWi18DWMBATHfnZjatUiA2GWJ7QnBYnZVTUWH5ZuNuXv9uKwvW5eJvDCOS4rh+YAJp7Zvr7HiRRk6BLuJJRfnH9eLToWi/a6VxrnuPS3Yesa6fEa09fge7bXsKeeP7bcxM38GBojJ6tIrg+oEJXJzampBADceLNEYKdJG6ZK1zb/ldK5yT7nYtd57v2/LzNqHNXSHfC2KTnOctu4J/YK13f6ikjPd/2snr321lXU4BzZoEMr5vW67t35745k1q3b6I1B8KdBFfKC6AnFWuoHc9cldDWZGz3j/ICXX3kI9LcsK/Bqy1LNq8l9e/28qnq3cBMLx7LNcPTGBAx2gNx4s0Agp0kfqivAz2bjo25HNWOje+OSIiHpq3h/A4CIuD8FgIbwVhrp/hsc788FUEdNb+w7y5aBtvL9nB3sISOrRoymVnxXNJ7za0bla/7nEvIqdPgS5S3xXkQI5ryD5nlTOEX5ANBbug9NCJ2wc2OTbgjw981+si/zDmrNjFzPQdLN6yF2NgYMdoLusTz8ikOJoE1b/55EXk5BToIg2Vtc7Q/cEcV8C7fh7MccK+YBccdP0sOXji+wNCnYAPi+NQcAvWFTZhye5ANhxqygH/KLp26sS5fZLp060TfgEKd5H6rqpA13/BIvWZMc617iER0KJz1dsWFziBfyTgC3b93Ms/mEOT/evpfTCH3qX5cORcvM3Ooxw/DgY2JzAyjuBmrV29/VjnZ1isa/g/xjkEEKQT7UTqIwW6SGMRHO48WnSqervSw04P/2Auxft2snbjRrZs3UTRvmxaFu0nYf9WWvkvI7RkD8aWn/j+oHBncpvAphAY4gz/B4ZCgNtz90fAkedNjtvefbsm0CQaAoLq5ncjcgZQoIucaQJDoXkCNE8guC2k9IIUIDv/MLN+yuKPSzPZlFdIaAD8omsol3b2p09UKf6FOa4vAjlQuNs5W7/0EJQWwcFct9eHnWWlh6CyLwRVCW3uGhWIgaYxPz8/+tP1vEk0+OlaexF3OoYuIsew1rIsM5/3lmYye9lO8g+XEhMezCW923DZWfF0iQ0//cbKS91C3vUoO3zs69LDzvH/Q3t+/sJwMO/n55WdFGj8oGlLV+hXEvhhMc5oRUAoBAQ7owcBwc6XGf9gn8+sJ1JTOilORGqkuKycBWtyee/HTL5Yl0d5hSW5TSSX9mnDqKRWxEV64T7yxQehMNcZBXAdKjj2eQ4Uur4AlJecXpv+QT+HfGWh7/76+PUBIc6hA/fXJ/w8fr3bc/8qBkatdb4EVZRV/igvhYpy1+sj25X/vB7jjFwYf9dPP7fXAZUsc1/nd9wy17YY51wOb93HwFrngXUmSrIVrmUVbsvcXoNTI7hqND//PGEZx64/2bJ6fM8GBbqI1Nrug8V8kLGT95Zmsjr7AAC92zVjVFIcI3u2ol20j0+Ws9a5Be+RkC8pdI0IFDuHA8qKndGBI69Li9yWF7k9TrG+oqx2dfoFuII9ECoqjg1nW+GZ30WdqiocTxKs7tufLJiPhHd9YfzcvtC4nhtz3PKq1rmeh0TCrd94riwFuoh40sbcg3yyMptPVu1iZZYT7j1aRTjhnhRH5+oMyzc05WVQXnxc0BdX8uXh+J/Hry/+udfsH+h67nrt5/baP8Bt3XEP/0DX9gFOrxpcXwzKnS8LttzpwR/56f78lOsq3MLWHvsTTlx2NEsqW+a2/ZHwOxp6lQWhOfl2xwdslfs9Rc0c+VHJZzw6OuD2xePIF46TrnN/n2u7wBAY83Tt/925KNBFpM7s2HuIeat2MXflLpZuc6aa7diyKSOT4hiV1IqerSN021kRD1Ggi4hX5Bwo4lNXuP+wZS/lFZb45qGM7BnHqOQ4erdtjp+fwl2kphToIuJ1ewtL+Hx1DnNXZvPNxt2UlltiwoMZ0TOOUUlx9EuMIsBfZ5uLVIcCXUR86kBRKV+szWXuil18uT6XotIKmjcJ5IIesYxKasXATtEEB+i6cpFTqReBbowZCTwL+AP/sNY+ftz6ycDtQDlwELjFWru6qjYV6CINz+GSchauz2Xuyl0sWJNLQXEZYcEB9IqPJKmN69E6goTophqeFzmOzwPdGOMPrAcuADKBJcAE98A2xkRYaw+4nl8M3GatHVlVuwp0kYatuKyc7zbuYf7aHFZk5rNmVwElZc6ZxGHBAfRoHUFym0iS2jg/E1uE4a+QlzNYfZicpR+w0Vq72VXQ28A44GigHwlzl6b8fMcAEWmkggP8Ob9bDOd3iwGgtLyCDTkHWZmVz8qd+azIyufNH7ZRVOqEfJMgf3q0ivi5J98mgk4tw3QsXgTvBXobYIfb60zg7OM3MsbcDtwLBAFDvVOaiNQXgf5+9GgdQY/WEVxJWwDKyivYlFfIyiwn4FftzGdm+g5e+24rAMEBfnRv9XNPPqlNJJ1jwgkKUMjLmcVbQ+5XACOstTe5Xl8L9LPW3nmS7Se6tr+uknW3ALcAtGvX7qxt27bVXeEiUi+VV1i27HZC/uegP8DBYucubkH+fpzdIYqLU1ozIimOiJDAU7Qo0jDUh2PoA4Cp1toRrte/AbDW/ukk2/sB+6y1kVW1q2PoInJERYVl295DrMzKZ3nmfuatymH73kMEBfgxrFsM41JbM6RrDCGBOpteGq76EOgBOCfFDQOycE6Km2itXeW2TWdr7QbX87HAoycr+ggFuoicjLWWjB37+SBjJ3OW72T3wRLCgwMYmRTHuNQ2DOgYrRPspMHxeaC7irgIeAbnsrV/WWunG2OmAenW2tnGmGeB4UApsA+4wz3wK6NAF5HTUVZewfeb9/D+TzuZt2oXB4vLaBkezJherRiX2oaU+EjdnlYahHoR6HVBgS4i1VVUWs6Ctbl8kJHFF2vzKCmvoH10E8altObi1DZ0ignzdYkiJ6VAFxGpRP7hUuat3MUHy7L4btMerIWerSMYl9qasSmtaRUZ6usSRY6hQBcROYXcA0V8uDyb2RlZLMvMxxjolxDFuNQ2XJQcR7MmQb4uUUSBLiJSHVt2FzI7YycfLMtic14hgf6G/h2i6RUfSXKbSHq2jiS+eaiOu4vXKdBFRGrAWsuqnQf4ICOLbzbuYX1OAeUVzv8zmzcJPHrHumTXQyEvda0+3PpVRKTBMcYcDW1wTqhbu6uAFVn5rMx0bmjzylebKXOFfGRo4NG71R0J+XZRTRTy4hUKdBGR0xQS6E9q22aktm12dFlRaTnrc1wh77pr3b++2UJpuRPyESEBbveed0K+fVQTzSQnHqdAFxGphZBAf3rFN6NX/M8hX1xWzvpdB49OMLMyK5/Xvt1KSbkzyUx4cADJ8ZGc26Ulw7rF0CkmTL14qTUdQxcR8YKSsgrW5xQcnUlu6bb9rMl2JplsF9WEod1iGN49ln6JUZpYRk5Kx9BFRHwsKMDvmOPxADv3H2bB2lwWrM3lrcXbee27rYQFBzC4cwuGdY9lSNeWtAgL9mHV0pCohy4iUg8cLinn2427mb82lwVrc8g5UIwxkNq2GcO7xzK0Wwzd4sI1NH+G02VrIiINyJHL5eavyWX+2hyWZ+YD0KZZKEO7xTC0ewwDOkRr5rgzkAJdRKQByz1QxBfrcpm/JpevN+zmcGk5oYH+DOrcgmHdYhjaLYaYiBBflyleoEAXEWkkikrLWbR5DwvWOgGftf8wAMltIukWF05cZAixESHERYQcfR7dNEiXyTUSCnQRkUbIWsu6nALmr8ll4bo8tu0tJK+gmIrj/rce6G+ICXcCPi7CFfiRwScEv4bw6z8FuojIGaKsvILdB0vYdaCIXflF7Mo/zK4DxeS4XuccKGLXgSIOlZSf8N7mTQJdYe8EfYeWTUlLiCKpdaQupasndNmaiMgZIsDfzwnkyBBoW/k21loKisvIyS8iO98J+JwjP12BvzIrn91LSgAICfQjtW0z+iVEkZYQRZ/2zQkLVnzUN/qLiIicYYwxRIQEEhESSOfY8JNul1tQRPrWfSzZupclW/fy1y82UmHBz0CP1hGktY+iX2IUaQnNiQnXSXm+piF3ERE5LQeLy/hp+z6WbNnLkq37+GnHPopKndvZto9uQt+EKPomNKdvQhSJLZrqmvk6oCF3ERH5/+3dfZBV9X3H8ffHhV3YZXncXXm0YEQ046Axam3jQyuOo5aS2NZOnHRGx6adZppGzSSpqZ08jNPGPHcm7cRpNG0nGhtrTKKZNkJMjX3yCRWFsKjIIk8LC2tgAXkQvv3j91tygQW5e86F5e7nNXPnnvuwH36He8/565oF2wAADNJJREFUnvM7555fYekqdu1cOrsdSJezXbZ+K891vckzXb08vnwjDy1eC8CklkYuyMX9wpkTeffUsYxs8HH4WvIeupmZlSIiWNmznWcruunX9Kaf1TU3NnDu9PGcNaWVsyePZc7kVs48tZXRjT6zvhreQzczs5qTxBkdrZzR0coNF50GQPfWXTy3updnV/Xywppf8sAzbxzoppdg5qQW5pzaypzJrZw1uZWzpozltInNNPh381VzQTczs5qZPG4U8+dOZf7cqQDs2x+80buTFd3b6Ozuo3NDHys29vHYL7rp7zAeNfIUzjw1Ffg5k8emQj+5lUkeqOao3OVuZmYn3Ft79vHqplTgO7v7WLFxG50b+tiyY8+B97SNacpFPu/NTx7LrPaWYfUTOne5m5nZkDa6sYG508czd/r4g57v6dvNiu4+OvMe/YruPu57ajW7395/4D3trU3MmtTCzLZmZra15OkWZk5qGVbH6F3QzcxsyGpvbaK9tYlLZrcdeG7f/mD1lh2s6O7j9c076Nq8g64tO/hZZw+bt6896O8njx3FrLZU4Ge1NTNzUguz2lqYMbG57i5164JuZmYnlYZTxOntYzi9fcxhr/Xt2svqLTtZlQv9qi3p/rFl3fRWdN9LMHXc6Fzsf1Xop00YTfuYJiY0n3wD2rigm5lZ3WgdNZJzpo3jnGnjDntt6869dG1Je/OrNu84UPQfeXE923a9fdB7R5wi2sY0Hegh6Gj91XT7mCY6xjbRPmYU7a1NQ6Zb3wXdzMyGhXHNIzm3eTznzjj4OH1E8ObOvazavIPurbvo6dtFz/bd9PTtZlNfGthm6bqtbN5++Eh2kC6409HaRNthBb+JjrGjuPzM9uMyfy7oZmY2rEliYksjE1saj/q+ffuD3h176OnbXVHwd6XHufgvX7+NJ/t207c77fGPGz2SJZ+96njMhgu6mZnZsWg4RQe63d/JW3v20dO3m2279h6HliUu6GZmZiUb3djAaZOaj+u/6Svlm5mZ1QEXdDMzszrggm5mZlYHXNDNzMzqgAu6mZlZHXBBNzMzqwMu6GZmZnXABd3MzKwOuKCbmZnVARd0MzOzOqCIAYaOOUlI6gFWlxjZBmwuMa8WmUM9rxaZwy2vFplDPa8WmcMtrxaZQz2vFplDPe/XImLA4dtO6oJeNknPRcQFQzlzqOfVInO45dUic6jn1SJzuOXVInOo59Uic6jnHY273M3MzOqAC7qZmVkdcEE/2D+eBJlDPa8WmcMtrxaZQz2vFpnDLa8WmUM9rxaZQz3viHwM3czMrA54D93MzKwOuKBnkq6WtELSa5JuLyHv25I2SVpaUvtmSPpPScslLZN0S8G8UZKekbQk532+pHY2SHpB0o9LyOqS9LKkFyU9V1L7xkt6SFJn/r/8jQJZc3Lb+m/bJN1asH235c9jqaQHJI0qmHdLzlo22LYN9F2WNFHSIkmv5vsJBfOuz23cL6nqM4KPkPnl/Dm/JOkHksYXzLszZ70oaaGkqUXyKl77hKSQ1FawfZ+TtK7i+3jtseYdrY2S/iKvG5dJ+lLBNn6von1dkl4smHeepKf61xGSLjrWvKNknivp//K651FJY6vIG3A9XWR5qUpEDPsb0ACsBE4HGoElwLsLZl4GnA8sLamNU4Dz83Qr8EqRNgICxuTpkcDTwMUltPPjwHeBH5eQ1QW0lfxZ/wvw4TzdCIwv8TvUTfqN6GAzpgGrgNH58YPATQXyzgGWAs3ACOCnwOxB5Bz2XQa+BNyep28Hvlgw72xgDvAEcEFJbbwKGJGnv1hCG8dWTH8MuLtIXn5+BvAY6Xoax/xdP0L7Pgd8osD3ZaDM387fm6b8uKPoPFe8/lXgMwXbtxC4Jk9fCzxRwjw/C1yep28G7qwib8D1dJHlpZqb99CTi4DXIuL1iNgD/Cvw/iKBEfEk0FtG43Lehoh4Pk/3ActJBWCweRER2/PDkflW6IQKSdOB3wHuKZJTK3lL+zLgXoCI2BMRvywpfh6wMiKKXuhoBDBa0ghSIV5fIOts4KmI2BkRbwM/B66rNuQI3+X3kzaOyPcfKJIXEcsjYkW1bXuHzIV5vgGeAqYXzNtW8bCFKpaXo6wPvg58qpqsd8gbtCNkfgS4KyJ25/dsKpgHgCQBfwg8UDAvgP496HFUubwcIXMO8GSeXgT8fhV5R1pPD3p5qYYLejINWFPxeC0FimWtSZoJvIe0V10kpyF3eW0CFkVEoTzg70grp/0Fc/oFsFDSYkl/WkLe6UAP8E/5sMA9klpKyAX4IFWsnAYSEeuArwBvABuArRGxsEDkUuAySZMkNZP2YGYUaWOFUyNiA6SVGNBRUm6t3Az8R9EQSX8jaQ3wIeAzBbMWAOsiYknRdlX4aD4s8O2SunXPBC6V9LSkn0u6sIRMgEuBjRHxasGcW4Ev58/kK8CnC7csLTcL8vT1DHKZOWQ9fVyWFxf0RAM8NyRP/5c0Bvg+cOshewxVi4h9EXEeac/lIknnFGjXfGBTRCwu0qZDvC8izgeuAf5c0mUF80aQute+GRHvAXaQur8KkdRIWgH8W8GcCaQt+VnAVKBF0h8NNi8ilpO6mhcBPyEdSnr7qH9UhyTdQZrv+4tmRcQdETEjZ320QJuagTsouFFwiG8C7wLOI20QfrWEzBHABOBi4JPAg3nvuqgbKLgBnH0EuC1/JreRe98Kupm0vllM6jbfU21AmevparigJ2s5eCtsOsW6OmtC0kjSl+T+iHi4rNzc7fwEcHWBmPcBCyR1kQ5ZXCHpvoLtWp/vNwE/IB0aKWItsLaiJ+IhUoEv6hrg+YjYWDDnSmBVRPRExF7gYeA3iwRGxL0RcX5EXEbqWiy6R9Rvo6QpAPn+mLtijydJNwLzgQ9FPoBZku9SRVfsAN5F2nBbkpeZ6cDzkiYPNjAiNuaN9P3Atyi+vEBaZh7Oh+ieIfW+HfPJewPJh5N+D/heCe27kbScQNqgLjzPEdEZEVdFxHtJGx0rq/n7I6ynj8vy4oKePAvMljQr7219EHjkBLfpIHmr+F5geUR8rYS89v6zfiWNJhWTzsHmRcSnI2J6RMwk/f/9LCIGvXcpqUVSa/806QSnQr8YiIhuYI2kOfmpecAvimRmZe1tvAFcLKk5f97zSMfgBk1SR74/jbQSLaOdkJaPG/P0jcCPSsotjaSrgb8EFkTEzhLyZlc8XECx5eXliOiIiJl5mVlLOpmqu0D7plQ8vI6Cy0v2Q+CKnH8m6UTSogONXAl0RsTagjmQdrwuz9NXUMIGa8Uycwrw18DdVfztkdbTx2d5qcWZdifjjXR88RXS1tgdJeQ9QOr22ktaWP+4YN4lpMMALwEv5tu1BfLmAi/kvKVUcbbpMWT/FgXPcicd716Sb8vK+Exy7nnAc3m+fwhMKJjXDGwBxpXUvs+TCsVS4Dvks4sL5P0XaaNlCTBvkBmHfZeBScDjpBXo48DEgnnX5endwEbgsRLa+Brp3Jj+5aWas9IHyvt+/lxeAh4FphXJO+T1Lqo7y32g9n0HeDm37xFgSgn/h43AfXm+nweuKDrPwD8Df1bS9/ASYHH+fj8NvLeEzFtIteAV4C7yBdiOMW/A9XSR5aWam68UZ2ZmVgfc5W5mZlYHXNDNzMzqgAu6mZlZHXBBNzMzqwMu6GZmZnXABd3MaiqPJHbGiW6HWb1zQTcbZvKwlW9J2l5x+/sT3S4zK2bEiW6AmZ0QvxsRPz3RjTCz8ngP3cwAkHSTpP+R9A1JWyV1SppX8fpUSY9I6pX0mqQ/qXitQdJfSVopqS+PkFc5PsKVkl6V9KakfyhpgA8zq+A9dDOr9OukQWvaSNd+f1jSrIjoJV0mcxlpJLizgEWSXo+Ix4GPk65p338J5blA5fXT5wMXksauXky6dOpPjsscmQ0TvvSr2TCTR/dq4+ChVD9Jup7135KuUR75vc8A3yCNxtcFjI+IvvzaF0jXC79J0grgUxFx2KATkgK4NCL+Oz9+kDQ63V01mUGzYcpd7mbD0wciYnzF7Vv5+XVx8Fb+atIe+VSgt7+YV7w2LU/P4OjDTFaOIrYTGFOs+WZ2KBd0M6s07ZDj26eRhqhcD0zsH9K24rV1eXoNaYxvMztBXNDNrFIH8DFJIyVdD5wN/HtErAH+F/iCpFGS5pKGmrw//909wJ2SZiuZK2nSCZkDs2HKJ8WZDU+PStpX8XgR8CPSmNKzgc2kccn/ICK25PfcANxN2lt/E/hsRCzKr30NaAIWko7Pd5LGODez48QnxZkZkH62Bnw4Ii450W0xs+q5y93MzKwOuKCbmZnVAXe5m5mZ1QHvoZuZmdUBF3QzM7M64IJuZmZWB1zQzczM6oALupmZWR1wQTczM6sD/w9zJHg9YNf+nQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the training and the validation learning curve\n",
    "plt.figure(figsize=(8,5))\n",
    "plt.plot(train_avg_loss_list, label='Training loss')\n",
    "plt.plot(eval_avg_loss_list, label='Evaluation loss')\n",
    "plt.xlabel('Epoch', size=12)\n",
    "plt.ylabel('Loss', size=12)\n",
    "plt.title('biGRU model learning curves')\n",
    "plt.xticks(ticks=range(21))\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAFZCAYAAAB0RP9xAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd7xU1bnG8d9DFQIKKhaaoGJsEQt2TewtttwYS+xXg93EEnvU2GJJbBEL9m7sGiNXMImJmqggahRsKCIgCiiigEo57/1j74PD5JSROVPO7Oebz/5kZu2y1hwP71nzrrXXVkRgZmbZ0KbSDTAzs/Jx0DczyxAHfTOzDHHQNzPLEAd9M7MMcdA3M8uQdpVugJlZtZk3/f2i5rK3X3ZltVRbWpp7+mZmGeKevplZvroFlW5ByTjom5nli7pKt6BkHPTNzPLVOeibmWVG1HBP3wO5ZmYZ4p6+mVk+p3fMzDKkhtM7DvpmZvk8ZdPMLENquKfvgVwzswxxT9/MLF8ND+S6p28tSlInSX+WNFPSA0VcZ39Jw1uybZUgaZikgyvdDvtuIuqK2qqZg35GSfq5pFGSZkmakganLVrg0nsBywPLRMTPFvciEXF3ROzQAu1ZhKStJIWkh/PKB6blzxR4nXMl3dXccRGxc0TcvpjNtUqpqytuq2IO+hkk6UTgSuAikgDdF7gW2KMFLr8S8E5EzG+Ba5XKNGAzScvklB0MvNNSFSjhf19WdfxLmTGSlgLOA46JiIcjYnZEzIuIP0fEr9NjOkq6UtJH6XalpI7pvq0kTZJ0kqSp6beEQ9N9vwXOBvZJv0Eclt8jltQv7VG3S98fIul9SV9KGi9p/5zy53LO20zSyDRtNFLSZjn7npF0vqTn0+sMl7RsEz+GucCjwL7p+W2BvYG7835WV0maKOkLSS9L2jIt3wk4I+dzvpbTjgslPQ/MAVZOyw5P918n6cGc618i6a+Sqnbt9cyKuuK2Kuagnz2bAksAjzRxzJnAJsC6wEBgI+CsnP0rAEsBvYDDgCGSukfEOSTfHv4UEV0i4uamGiLpe8DVwM4R0RXYDHi1geOWBv6SHrsMcDnwl7ye+s+BQ4HlgA7AyU3VDdwBHJS+3hEYA3yUd8xIkp/B0sA9wAOSloiI/8v7nANzzjkQGAx0BSbkXe8kYJ30D9qWJD+7gyOiqAd2WAnULShuq2IO+tmzDDC9mfTL/sB5ETE1IqYBvyUJZvXmpfvnRcSTwCzg+4vZnjpgbUmdImJKRIxp4JgfA+9GxJ0RMT8i7gXeAnbLOebWiHgnIr4C7icJ1o2KiH8BS0v6Pknwv6OBY+6KiE/TOv8AdKT5z3lbRIxJz5mXd705wAEkf7TuAo6LiEnNXM8qwT19qyGfAsvWp1ca0ZNFe6kT0rKF18j7ozEH6PJdGxIRs4F9gCOBKZL+Imn1AtpT36ZeOe8/Xoz23AkcC2xNA9980hTWm2lK6XOSbzdNpY0AJja1MyJeAt4HRPLHyaqRB3Kthvwb+BrYs4ljPiIZkK3Xl/9OfRRqNtA55/0KuTsj4qmI2B5YkaT3fmMB7alv0+TFbFO9O4GjgSfTXvhCafrlVJJcf/eI6AbMJAnWAI2lZJpM1Ug6huQbw0fAKYvfdLPF46CfMRExk2SwdYikPSV1ltRe0s6SLk0Puxc4S1KPdED0bJJ0xOJ4FfihpL7pIPLp9TskLS9p9zS3/w1JmqihhOiTwGrpNNN2kvYB1gSeWMw2ARAR44EfkYxh5OsKzCeZ6dNO0tnAkjn7PwH6fZcZOpJWAy4gSfEcCJwiqck0lFWI0ztWSyLicuBEksHZaSQpiWNJZrRAEphGAf8BXgdGp2WLU9cI4E/ptV5m0UDdhmRw8yPgM5IAfHQD1/gU2DU99lOSHvKuETF9cdqUd+3nIqKhbzFPAcNIpnFOIPl2lJu6qb/x7FNJo5urJ02n3QVcEhGvRcS7JDOA7qyfGWVVpIbTO/LEATOzRX392pNFBcYlBu5StdNw3dM3M8sQL7hmZpavyvPyxXDQNzPLV+V5+WI46JuZ5XNP38wsQ6p8KYVitJag7ylGZlaoqp05Uw1aS9Bn7oRmp0JbhnRYaX0AunTuX+GWWDWZNWd8y1yohtM7nrJpZpavTDdnSWor6RVJT6Tv+0t6UdK7kv4kqUNa3jF9Py7d3y/nGqen5W9L2rG5Oh30zczylW8Zhl8Cb+a8vwS4IiIGADNIlt8m/f8ZEbEqcEV6HJLWJHkuxFrATsC16fMhGuWgb2aWrww9fUm9SZYNvyl9L2AboP5BO7fz7cKIe6TvSfdvmx6/B3BfRHyTriU1juT5F41y0Dczq4wrSdaRqv8rsQzwec6y5ZP4dvnwXqRrP6X7Z6bHLyxv4JwGOeibmeUrsqcvabCkUTnb4NzLS9oVmBoRL+cWN9CSaGZfU+c0qNXM3jEzK5eI4ubpR8RQYGgTh2wO7C5pF5LHly5J0vPvJqld2pvvzbfPsZgE9AEmpSu2LkWyMm19eb3ccxrknr6ZWb4S5/Qj4vSI6B0R/UgGYv8WEfsDfwf2Sg87GHgsff14+p50/9/SZys/Duybzu7pDwwAXmqqbvf0zcyqx6nAfZIuAF4Bbk7LbyZ59sI4kh7+vgARMUbS/cBYkof+HBPNfE1x0Dczy1fGm7Mi4hngmfT1+zQw+yYivgZ+1sj5FwIXFlqfg76ZWT6vsmlmliE1vAyDg76ZWb4a7ul79o6ZWYa4p29mls/pHTOzDKnh9I6DvplZPgd9M7MMqeH0jgdyzcwyxD19M7N8Tu+YmWVIDad3HPTNzPLVcE/fOX0zswxxT9/MLJ/TO2ZmGVLD6R0HfTOzfA76ZmYZEk0+W7xV80CumVmGuKdvZpbP6R0zswxx0DczyxBP2TQzy5Aa7ul7INfMLEPc0zczy1fDUzYd9M3M8tVwesdB38wsXw0Hfef0zcwyxD19M7N8nrJpZpYdUeeBXDOz7KjhnL6DvplZvhpO73gg18wsQ9zTNzPL55y+mVmGOKdvZpYhDvpmZhlSw2vveCDXzCxD3NM3M8vn9I6ZWYZ49o6ZWYb45iwzM6sF7umbmeVzesfMLDvCA7lmZhninr6ZWYZ4INfMzGqBe/pmZvmc3jEzyxAP5JqZZYh7+mZmGeKBXDMzqwXu6VeJBQvq2PfYM1hu2aUZcv4pnPq7axj77vu0a9uWtVdfhbN/eTjt27Vj5pezOPsPNzBxyid07NCB8048ggH9+wDwxazZnHv5UN79YBISnHfSEay75moV/mTWEpZaqitDrr2ENddcjYjgqCNPYbvtf8ghh+7L9OmfAXDuOZcx/KlnaNeuHUOuvZh1112Ldu3acc89D/OH319X4U/Qyji9Y6V21yPD6N+3F7PnfAXAj7fdnItPOwaAU3/3Rx4e9nf22W17brr3MVZfZSWuOvck3v9wMhddcys3XXoWAJdcezubbziQy88+gXnz5vPVN99U7PNYy7r0snMYMeIfHLD/0bRv357OnZdgu+1/yDV/vIWrr7pxkWN/8j+70LFjBzbeaGc6dVqCUaNH8MD9j/Phh5Mr1PrWp5bvyC1rekdSx3LW11p8PO1Tnn3pFX6609YLy3640XpIQhJrf39VPkl7c+99OImN11sbgJX79mLyJ9OYPuNzZs2ew8uvv8X/pNdo374dS3b5Xvk/jLW4rl27sPkWG3H7bX8CYN68ecyc+WXjJ0TQ+Xudadu2LZ06LcHcufP48stZZWptjaiL4rYqVpagL2kjSa8D76bvB0r6Yznqbg0uve4OTjj857Rp89//OebNn88Tf32WzQcNBOD7K6/E08+NBOD1t8Yx5ZPpfDLtMyZ9PJXu3ZbkrN9fz8+OOo1zLh/KnK++LuvnsNLo178P06d/xvU3XMbz/36Ca669mM6dOwFwxJEH8cKLw7j2+kvo1m1JAB55ZBhzZs/hvfdf5M23n+fqq25kxoyZlfwIVkXK1dO/GtgV+BQgIl4Dtm7yjIz4xwujWbrbkqy12soN7r/wj7ewwQ9WZ4MfrA7AYfvszhezZrPXkadxz2NPsfqq/WjXti0LFizgzXfHs8+u2/PAdRfTaYmO3Pynx8v5UaxE2rVrx7rrrsVNN93N5pvuypzZczjp5KO46ca7+cFaP2LTTXbhk4+ncdHFZwIwaNBAFixYwKqrbMLaa/6Q444/nH79+lT4U7Qy7ukXX09ETMgrW9DUCZIGSxoladTQoUNL2LTKemXM2/z9hdHseOBx/Pqiq3np1TGcdvE1AFx354N89vmX/PqIAxce3+V7nbng5CN58PqLueiUo5kx8wt6rdCD5ZddhuV7LM06a6wKwPZbbsyb48ZX5DNZy5o8eQqTJ3/MqJGvAvDoI8MYuO5aTJ06nbq6OiKCW2+5l0EbJN8G995nD0aM+Cfz589n2rRPeeGFUay//jqV/AitT9QVt1WxcgX9iZI2AkJSW0m/At5p6oSIGBoRgyJi0ODBg8vTygr41WH78dd7hvDUnX/ksjOOZ6N11+Li047loWF/4/mX/8OlZxy3SNrni1mzmTdvPgAPDfsbG/xgDbp8rzPLLt2NFXosw/iJHwHw4itvsErf3hX5TNaypn4yncmTpjBgQPJtcKutN+OtN8ex/Ao9Fh6z2+47MnZs8k9q4sTJ/GirTQHo3LkTG224Hm+/8175G96a1XBPv1yzd44iSfH0BT4Bnk7LrBHnX3UzKy6/LAf88mwAtt1iQ4464Ke8/+Fkzrz0Otq0acMqK/Xityd++wfx9GMO4bSLr2He/Pn0XmF5zj/5iEo131rYSSedw823XkGH9h0Y/8GHHHXEr7ns9+eyzjprEAETPpzE8cedAcDQG+7k+hsuY+Sop5DEnXc+yJg33qrwJ2hdosoDdzEU0So+XMydMLrSbbAq0mGl9QHo0rl/hVti1WTWnPEAKvY6X/5qt6ICY9cr/9xkGyQtAfwT6EjS+X4wIs6RdDcwCJgHvAQcERHzJAm4CtgFmAMcEhGj02sdDJyVXvqCiLi9qbrL0tOXdCPwXz/EiKjdvI2ZtV6l7+l/A2wTEbMktQeekzQMuBs4ID3mHuBw4DpgZ2BAum2clm0saWngHJI/FAG8LOnxiJjRWMXlSu88nfN6CeAnwMQy1W1m9t2U+OasSFIs9TdPtE+3iIgn64+R9BJQPzC3B3BHet4LkrpJWhHYChgREZ+l54wAdgLubazusgT9iPhT7ntJdwIjylG3mdl3VmRPX9JgIDeTMTQihuYd0xZ4GVgVGBIRL+bsaw8cCPwyLerFoh3lSWlZY+WNqtQyDP2BlSpUt5lZ04oM+mmAb3KueUQsANaV1A14RNLaEfFGuvta4J8R8Wz6vqExgmiivFHlyunPyGlIG+Az4LRy1G1mVs0i4nNJz5CkZd6QdA7QA8idfjcJyL3DrjfwUVq+VV75M03VV/J5+umo80CSD9ED6B4RK0fE/aWu28xscUREUVtzJPVIe/hI6gRsB7wl6XBgR2C/iEXu8nocOEiJTYCZETEFeArYQVJ3Sd2BHdKyRpW8px8RIemRiNig1HWZmbWI0s/eWRG4Pc3rtwHuj4gnJM0HJgD/TvrLPBwR5wFPkkzXHEcyZfNQgIj4TNL5wMj0uufVD+o2plw5/ZckrV8/r9TMrKqVOOhHxH+A9RoobzAmp7N2jmlk3y3ALYXWXdKgL6ldRMwHtgB+Iek9YDbJ4ENExPqlrN/MzBZV6p7+S8D6wJ4lrsfMrMXU8jIMpQ76AogIr/ZkZq2Hg/5i6yHpxMZ2RsTlJa7fzOy7q+7VkYtS6qDfFuhCCyyAZGZWLk7vLL4p6XQjMzOrAmXJ6ZuZtSru6S+2bUt8fTOzluec/uJp7s4wM7NqVMs5/XI9I9fMzKpApZZWNjOrXk7vmJllRy2ndxz0zczyuadvZpYdUcNB3wO5ZmYZ4p6+mVm+Gu7pO+ibmeWp5fSOg76ZWT4HfTOz7Kjlnr4Hcs3MMsQ9fTOzPLXc03fQNzPL46BvZpYlUbuPAlmsnL6klST1bunGmJlZaRUU9CXdJWnT9PVBwNvAO5IOKWHbzMwqIuqK26pZoT39HYCX09cnAdsDmwBnlKJRZmaVFHUqaqtmheb0O0TEXEk9gR4R8SyApBVL1zQzs8qo9t56MQoN+q9J+jXQD/gLQPoH4IsStcvMrGLCA7kcDmwIdAPOSss2B+4tRaPMzKw0CurpR8S7wN55ZQ8AD5SiUWZmlZTJ9E46S6dZEXFHyzXHzKzyqn0wthhN9fR/UcD5ATjom1lNidp9RG7jQT8itixnQ8zMqkUt9/QLviNXUndJ+0k6MX2/QjqDx8zMWolC78jdEngHOAz4bVq8OnB9idplZlYxvjkLrgL2j4jhkmakZS8AG5WmWWZmlZPJnH6e/hExPH1d/+OYC7Rv+SaZmVVWtffWi1FoTv8tSdvllW0DvNHC7TEzsxIqtKd/MvCYpMeATpKGAD9JNzOzmlLLyzAUekfu85LWAw4kmZc/Bdg0IiaUsnFmZpWQyTty80XEROAiSd0jYkazJ5iZtVJ1NdzTL3TK5lKSbpU0B5guaU76vluJ22dmVnYRKmqrZoUO5N5CssLmxkD39P+XTMvNzKyVKDS9sw3QMyK+St+/ni7INrk0zTIzqxxP2YRxQN+8st7Auy3bHDOzyosobqtmhS6t/BQwXNLtwESgD3AQcGdpm2dmVn613NP/LksrfwhsnfN+IvCjFm+RmVmF1fLsHS+tbGaWIQXP0zczy4pqn3ZZjELn6feUdL+kTyQtyN1K3UAzs3Kr5YHcQmfvXJ8e+2NgFsmSyn8Bji5Ru8zMKqYuVNRWzQpN72wOrBQRsyRFRLws6VDgOeCG0jXPzMxaUqFBfwHJ+vkAMyX1AGaSzNU3M6sptZzTLzTojwR2Bh4DRgD3AHOA0SVql5lZxVR7Xr4YhQb9A/k2/388cCrQBbi8FI1qSIeV1i9XVdaKzJozvtJNsBpU7Xn5YhS6nv5nOa/nAOeUrEVmZhWWyfSOpLMLuUBEnNdyzWlcuw69ylGNtRLz5yZr/c2b/n6FW2LVpP2yK1e6CVWvqZ7+gALOr+HMl5llVSbTOxFxYDkbYmZWLUrdm5XUh+TRsysAdcDQiLgqZ//JwGVAj4iYLknAVcAuJJNoDomI0emxBwNnpadeEBG3N1W3l2EwM8tThp7+fOCkiBgtqSvwsqQRETE2/YOwPckil/V2Jsm+DCB5iNV1wMaSliYZYx1E8rfqZUmPN/VI20LvyDUzy4xSPy4xIqbU99Qj4kvgTaB+4PIK4BQW/cKxB3BHJF4AuklaEdgRGBERn6WBfgSwU1N1O+ibmVWQpH7AesCLknYHJkfEa3mH9SJZzr7epLSssfJGOb1jZpanrsjzJQ0GBucUDY2IoQ0c1wV4CPgVScrnTGCHhi7ZQFk0Ud6ogoO+pK2BfYHlI2JPSesDXSPiH4Vew8ysNYgGY+l3OD8J8P8V5HNJak8S8O+OiIcl/QDoD7yWjNvSGxgtaSOSHnyfnNN7Ax+l5VvllT/TVL2FLq18NHAzydeI+qdnzQUuLOR8M7PWpC6K25qTzsa5GXgzIi4HiIjXI2K5iOgXEf1IAvr6EfEx8DhwkBKbADMjYgrJo2x3kNRdUneSbwlPNVV3oT39k4DtIuJ9SSelZW8CaxR4vpmZfWtzkuVtXpf0alp2RkQ82cjxT5JM1xxHMmXzUEhWS5B0Psn6aADn5a6g0JBCg35XYEL6uv7vWDu+XXnTzKxm1BWZ3mlORDxHw/n43GP65bwO4JhGjrsFuKXQugudvfMccHJe2TGA8/lmVnMCFbVVs0J7+scBT0j6BdBV0hiSXv4uJWuZmVmFFDt7p5oVusrmZEkbAJsCfUkGdP8dEX5GrpnVnGrvrRej4CmbEVEHPJ9uZmbWChUU9CWNp5EJ/xHhtUzNrKZkPr0DHJ73fkWSPP+9LdscM7PKy3zQj4i/5pdJ+ivJ3NErW7pRZmaV5Jx+w74CnNoxs5pTV7sxv+Ccfv6jEzsDPwaGt3iLzMysZArt6ec/OnE2MAS4rUVbY2ZWBUp9R24lNRv0JbUlWZj//oj4uvRNMjOrrFp++HezyzCkN2D90QHfzLKirsitmhW69s5fJHnJBTOzVq7QnH4b4GFJz5EswbDw209E/G8pGmZmVil1ynBOP/UucFkpG2JmVi1qOaffZNCXtF9E3BsRvylXg8zMKq3a8/LFaC6nf0NZWmFmVkXqVNxWzZoL+lXefDMz+y6ay+m3lbQ1TQT/iPhbyzbJzKyysnxzVkeSJ7Y39hMIvP6OmdWYzA7kArO9Xr6ZZU215+WLUejNWWZmVgOa6+nX8N87M7OG1fKUzSaDfkR0LVdDzMyqRZZz+mZmmVPLOX0HfTOzPLWc3vFArplZhrinb2aWp5Z7+g76ZmZ5wjl9M7PscE/fzCxDajnoeyDXzCxD3NM3M8vjm7PMzDLEN2eZmWWIc/pmZlYT3NM3M8tTyz19B30zszweyDUzyxAP5JqZZUgtp3c8kGtmliHu6ZuZ5XFO38wsQ+pqOOw76JuZ5anlnL6DvplZntrt53sg18wsU9zTNzPL4/SOmVmG+OYsM7MMqeXZO87pm5lliIN+lRn3zgu8MvppRo0czgv/fhKAs39zIhPGj2LUyOGMGjmcnXfaZuHxp55yLG+NfY4xb/yTHbb/UaWabSWyYMEC9jrkGI7+9TmLlF90+bVsuN1PFr4f9err/OzQYxn4wx8z/O/PLix/65332H/wCeyx/xH85KCjGPb0P8rW9tYsityqmdM7VWi77X/Gp5/OWKTsqqtv5PIrblikbI01BrD33nuwzrrb0LPn8jw17D7WWGtL6upqeRgqW+564DFW7teXWbPnLCx74813+GLW7EWOW3H55bjgzJO47d6HFilfYomOXPSbk1mpTy+mTvuUvQ87js033oAlu3YpS/tbq1r+F+Sefiu2+247cv/9jzF37lw++GAi7733ARttuF6lm2Ut5OOp0/jnv17ip7vtuLBswYIF/GHIzZx09GGLHNtrxeX5/qr9aaNFRyD79e3NSn16AbBcj2VYuns3Znw+s/SNb+XqiKK2ala2oK/EAZLOTt/3lbRRuepvLSKCYU/ey4svDOPww/ZfWH70UYcy+uUR3Dj0D3TrthQAPXuuwMRJHy08ZtLkKfTstULZ22ylcclVN3Di0YchffvP9J6H/szWW2xCj2WX/s7Xe33s28ybN58+vVZsyWbWpFpO75Szp38tsCmwX/r+S2BIGetvFX641Z5stPFO7LrbARx11CFsucXGXH/DHay2+mZsMGgHPv54KpddejYA0n/PK4uo9l85K8Qzz7/I0t27sdbqAxaWTZ32KcP//iw/32v373y9adM/4/TzLuOCM06gTRt/wc+ycub0N46I9SW9AhARMyR1aOxgSYOBwQA33HBDY4fVnClTPgFg2rRPeeyxYWy44bo8+9yLC/ffdPPdPPbo7QBMnjyFPr17LtzXu9eKTPnok/I22Erilf+M5ZnnXuDZf4/km7nzmD17DnseeCTt27dnl33+F4Cvv/6Gnff+X4bdf0uT15o1ezZH//psjht8MAPXXqMczW/1ajmnX86gP09SW9JvP5J60MTPNiKGAkPr3x597G9L38IK69y5E23atGHWrNl07tyJ7bf7ERdceAUrrLAcH388FYA999iZMWPeBuDPTwznzjuGcMWVQ+nZc3lWXbU/L418pZIfwVrICUcdyglHHQrAS6P/w233PsS1ly36b2DD7X7SbMCfN28evzz9fHbfaVt23GbLkrW31lR7Xr4Y5Qz6VwOPAMtJuhDYCzirjPVXveWX78GDD9wMQLt2bbnvvkd5avgz3Hbr1QwcuCYRwYQJkzjq6FMBGDv2HR588M+8/trfmb9gAcf/8kzP3Mmo1998m1+dfj5ffDmLZ55/kSE33cVjd9/A//3tWV5+9Q0+n/kljz75NAAXnnkiq6+2SoVbXN1qN+SDypkDlrQ6sC0g4K8R8WaBp0a7Dr1K1zBrdebPnQzAvOnvV7glVk3aL7syJPGlKL/st29RgfGqD+6r2oUcyjl7ZxVgfEQMAd4AtpfUrVz1m5lVC0m3SJoq6Y288uMkvS1pjKRLc8pPlzQu3bdjTvlOadk4SacVUnc5h/EfAhZIWhW4CegP3FPG+s3MChJF/q8AtwE75RZI2hrYA1gnItYCfp+WrwnsC6yVnnOtpLbpGOkQYGdgTWC/9NgmlTOnXxcR8yX9D3BVRPyxfiaPmVk1KfXIWET8U1K/vOKjgIsj4pv0mKlp+R7AfWn5eEnjgPp7nMZFxPsAku5Ljx3bVN3l7OnPk7QfcBDwRFrWvoz1m5kVpNg7ciUNljQqZxtcQLWrAVtKelHSPyRtmJb3AibmHDcpLWusvEnl7OkfChwJXBgR4yX1B+4qY/1mZmWRN+W8UO2A7sAmwIbA/ZJWpuGB6aDhTnuzuaWyBf2IGAscn/N+PHBxueo3MytUhaZsTgIejmRK5UuS6oBl0/I+Ocf1BurXX2msvFElD/qSXqeJn2FErFPqNpiZfRcVujnrUWAb4BlJqwEdgOnA48A9ki4HegIDgJdIvgEMSLMmk0kGe3/eXCXl6OnvWoY6zMxaTKkHciXdC2wFLCtpEnAOcAtwSzqNcy5wcNrrHyPpfpIB2vnAMRGxIL3OscBTQFvglogY01zdJQ/6ETGh1HWYmbWkAqddLv71I/ZrZNcBjRx/IXBhA+VPAk9+l7rLeXPWJpJGSpolaa6kBZK+KFf9ZmZW3tk715DknB4ABpFM3Vy1jPWbmRWkllewKuvjEiNinKS2aT7qVkn/Kmf9ZmaFKHV6p5LKGfTnpOvnv5quKTEF+F4Z6zczK0gt9/TLeUfugWl9xwKzSeaX/rSM9ZuZFaQuoqitmpVjnn7fiPgwZxbP10DtPxHFzKwKlaOn/2j9C0kPlaE+M7Oi1PKD0cuR089dN2LlMtRnZlYUPy6xONHIazOzquTZO8UZmN6EJaBTzg1ZAiIilixDG8zMjPIsw9C21HWYmbWkWp6yWdabs8zMWgPn9M3MMsQ5fTOzDKnl9E4578g1M7MKc0/fzNr1dVkAAAhOSURBVCxPVPlSCsVw0Dczy+OBXDOzDKnlnL6DvplZnlqeveOBXDOzDHFP38wsj3P6ZmYZ4tk7ZmYZUssDuc7pm5lliHv6ZmZ5ann2joO+mVkeD+SamWWIB3LNzDKklnv6Hsg1M8sQ9/TNzPJ4INfMLEPqnNM3M8uO2g35DvpmZv/FA7lmZlYT3NM3M8tTyz19B30zszy+OcvMLENquafvnL6ZWYa4p29mlsc3Z5mZZYhz+mZmGVLLOX0HfTOzPLXc0/dArplZhrinb2aWx+kdM7MM8ewdM7MMqeWllZ3TNzPLEPf0zczyOL1jZpYhtZzecdA3M8vjnr6ZWYbUck/fA7lmZhninr6ZWR6nd6rA/LmTK90Eq0Ltl1250k2wGlTL6Z3WEvRV6QZUC0mDI2Jopdth1cW/Fy2rlnv6zum3PoMr3QCrSv69aEERdUVt1cxB38wsQ1pLesfMrGxqeZVN9/RbH+dtrSH+vWhBEVHUVghJJ0gaI+kNSfdKWkJSf0kvSnpX0p8kdUiP7Zi+H5fu77e4n81Bv5XxYJ01xL8XLauOKGprjqRewPHAoIhYG2gL7AtcAlwREQOAGcBh6SmHATMiYlXgivS4xeKgb2ZWGe2ATpLaAZ2BKcA2wIPp/tuBPdPXe6TvSfdvK2mxZjU6p18FJC0D/DV9uwKwAJiWvt8oIuZWpGFWMZIWAK/nFO0ZER80cmw/4Im0x2gtoNTPyI2IyZJ+D3wIfAUMB14GPo+I+elhk4Be6etewMT03PmSZgLLANO/a90O+lUgIj4F1gWQdC4wKyJ+n3tM+lddUe3zwaylfBUR61a6EVlV7M1Zkgaz6DTaobkpOEndSXrv/YHPgQeAnRu4VH1DGurVL1Yjnd6pYpJWTQd5rgdGA30kfZ6zf19JN6Wvl5f0sKRRkl6StEml2m2lIamfpGcljU63zRo4Zq30v/+rkv4jaUBafkBO+Q2S2pb/E7QeUez/IoZGxKCcLX/MZTtgfERMi4h5wMPAZkC3NN0D0Bv4KH09CegDkO5fCvhscT6bg371WxO4OSLWA5pai+Jq4NKIGATsDdxUjsZZyXRKA/Srkh5Jy6YC20fE+sA+JP/N8x0JXJV+SxgETJK0Rnr85mn5AmD/0n+E1qsMs3c+BDaR1Dn9Fr8tMBb4O7BXeszBwGPp68fT96T7/xaLmYNyeqf6vRcRIws4bjvg+zljO90ldYqIr0rXNCuhhtI77YFrJNUH7tUaOO/fwJmSegMPR8S7krYFNgBGpr8fnUj+gFiFRMSLkh4k+QY/H3iFZNrtX4D7JF2Qlt2cnnIzcKekcSQ9/H0Xt24H/eo3O+d1HYvm9pbIeS086FvrTgA+AQaSfEv/Ov+AiLhH0ovAj4GnJB1O8rtxe0ScXs7GtmbluDkrIs4Bzskrfh/YqIFjvwZ+1hL1Or3TiqSDuDMkDZDUBvhJzu6ngWPq36S9QastSwFT0t+DA0nmdi9C0srA+xFxNUlKYB2SmWF7SVouPWZpSSuVr9mtTzluzqoUB/3W51Tg/0j+IU/KKT8G2DwdvBsL/KISjbOSuhY4WNILJKmd2Q0csw/whqRXgdWBOyJiLHAWMFzSf4ARwIplanOrVBdR1FbNVO1/lczMyq17l1WLCowzZo2r2uXg3dM3M8sQD+SameWp5VU2HfTNzPLUctrbQd/MLE+1D8YWwzl9M7MMcdC3qpOuMRP1a5BIGibp4ObOa+A6fSXN8joz9l0Vu/ZONXPQt8Um6QNJX6WB9RNJt0rq0tL1RMTOEXF7c8el7dku57wPI6JLRCxo6TZZbavlefoO+las3SKiC7A+sCHJTUALKeHfM2tVfEeuWTMiYjIwDFhb0jOSLpT0PDAHWFnSUpJuljRF0mRJF9SnXSS1lfR7SdMlvU+ybsxC6fUOz3n/C0lvSvpS0lhJ60u6E+gL/Dn95nFKA2minpIel/RZ+qzRX+Rc81xJ90u6I73uGEmDcvafmrb7S0lvp4uYWY1yesesGZL6ALuQrAwIydowg4GuwASSR73NB1YF1gN2AOoD+S+AXdPyQXy7tGxD9fwMOBc4CFgS2B34NCIOJFmudrc0pXNpA6ffS7J0Rc+0jovygvfuwH1AN5J1a65J6/w+cCywYUR0BXYEPmj+p2JWfRz0rViPpg92eQ74B3BRWn5bRIxJH/22NMlTgX4VEbMjYirJw53rl4fdG7gyIiZGxGfA75qo73CS5waMjMS4iJjQXCPTP0pbAKdGxNcR8SrJMwcOzDnsuYh4Mh0DuJNkNUtIljHuCKwpqX1EfBAR7zVXp7VetZze8Tx9K9aeEfF0bkG6ZvvEnKKVSNaCn5Kz3n+bnGN65h3fVBDvAyxOwO0JfBYRX+bVMyjn/cc5r+cAS0hqFxHjJP2K5BvGWpKeAk6MiI+wmlTtgbsY7ulbqeT+q5kIfAMsGxHd0m3JiFgr3T+F9FFwqb5NXHcisEoBdeb7CFhaUte8epp6Gtm3F464JyK2IPkDFsAlhZxnrVMUuVUz9/St5CJiiqThwB8k/QaYRfJA6N4R8Q/gfuB4SU+QLBd8WhOXuwm4XNJzJE8dWgWYl6Z4PgFWbqQNEyX9C/idpJNJliY+DDigufanOf1ewPMkDy75CneYatr8uZOrdpXMYvkX18rlIKADyXNAZwAP8u2a7jcCTwGvkQTyhxu7SEQ8AFwI3AN8CTxKMmYAyVjAWZI+TwN7vv2AfiS9/keAcyJiRAFt7whcDEwnSQEtB5xRwHlmVcfr6ZuZZYh7+mZmGeKgb2aWIQ76ZmYZ4qBvZpYhDvpmZhnioG9mliEO+mZmGeKgb2aWIQ76ZmYZ8v/7U/1iOYclyQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x432 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Confusion matrix\n",
    "plt.figure(figsize=(6,6))\n",
    "ax = sns.heatmap(conf_matrix, fmt='d', annot=True, linewidths=1, square=True)\n",
    "ax.set_xlabel('Predictions', size=12)\n",
    "ax.set_ylabel('True labels', size=12) \n",
    "ax.set_title('Confusion Matrix', size=12); \n",
    "ax.xaxis.set_ticklabels(['True', 'False'])\n",
    "ax.yaxis.set_ticklabels(['True', 'False'])\n",
    "ax.set_ylim(2,0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model achieved the validation accuracy of 0.878, while the training accuracy was 0.908. The model's best state was saved to the *checkpoint.pt* file in the current directory. The training wasn't stopped by EarlyStopping object because the validation loss changes were too small and fluctuated near the same value.\n",
    "\n",
    "The training process that is presented above regards the model with the tuned hyperparameters. The steps we went through when doing hyperparameters fine-tuning are listed in the next section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameters fine-tuning\n",
    "\n",
    "After the model is created we have to tune its hyperparameters like hidden_size, embedding_dim, n_layers, dropout, spatial_dropout, learning_rate, batch_size, seq_length or min_word_count and then chose the set of parameters that enable the model to achieve the best performance (a model that returns the highest accuracy or lowest loss) on the validation set.\n",
    "\n",
    "Below we will show how the training and validation losses change during the training, do to so we have to disable the early stopping. We are interested in comparing training and validations losses in terms of their values but also in the directions in which they are heading during the training. This will help us discern the overfitting if it occurs.\n",
    "\n",
    "The initial parameters used in our model are the following:\n",
    "- hidden_size = 100\n",
    "- embedding_dim = 200\n",
    "- n_layers = 1\n",
    "- dropout = 0.2\n",
    "- learning_rate = 0.001\n",
    "- batch_size = 32\n",
    "- spatial_dropout = False\n",
    "- min_word_count = 5\n",
    "- max_seq_len = 0.8\n",
    "\n",
    "Then we will change one of the above hyperparameters, train the model and depict the training and validation loss at each epoch during the training."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Learning curves for the model with dropout  set to 0.2.\n",
    "\n",
    "dropout = 0.2\n",
    "\n",
    "<div>\n",
    "<img src=\"assets/learning_curves_droput02.png\" width=\"500\" />\n",
    "</div>\n",
    "\n",
    "With the lowest evaluation loss (at second epoch), the model archived evaluation accuracy of 0.861 while the training accuracy at that time was 0.883.\n",
    "\n",
    "We can notice on the preceding plot, that the training loss constantly decrease together with the validation loss, but at the second epoch, the validation loss reverse and begin to increase. This behaviour indicates that our model is overfitted, it learns too well how to map training input variables to the target, thus it cannot generalize its performance on some unseen data (validation set). One of the solutions to this problem is to increase the dropout probability, so the neural network will be forced to learn some more robust correlations in the data since the neurons will not depend anymore on each other, therefore the overfitting will be reduced.  The best remedy to the overfitting issue would be to gain more training examples, but our data set is finite, so we are not able to do that.."
   ]
  },
  {
   "attachments": {
    "learning_curves_droput05.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfQAAAFQCAYAAABJW4xyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd1hUV/rA8e9LEwTELgqo2GlWbLEbu6IpGo0pJpvEuGlu2ia7v1Sz2bjZ9LLRNGM2btRUjdGYYk9iwcRu7Cgo2MWKtPP74150QEDAYYbyfp5nHmbuPffe984MvJxzzz1HjDEopZRSqnzzcHcASimllLpymtCVUkqpCkATulJKKVUBaEJXSimlKgBN6EoppVQFoAldKaWUqgA0oSu3EpEFIjLO3XGUBSLyDxE5IiIppbT/3iKSVBr7diYRaSwiRkS8ilD2NhFZcYXH6yEi25wRTwHbX3GMShWFJvRKSkQSRKSfu+Mwxgw2xkx3dxzuJiJhwMNApDEm2En7NCLSzBn7qsiMMcuNMS1zXpeV343SdqX/4InIgyKSIiKpIvKhiFQpoFzOP0SnHR5PljxyVRBN6KrUlLRGU5a48BwaAUeNMYeKu2FFeJ8rq/L62YnIQOBx4GqgMdAEePYym1U3xgTYj+dKOcRKSRO6uoSIDBORdSJyQkR+EZHWDuseF5FdInJKRLaIyLUO624TkZ9F5FUROQY8k9PcKCIvichxEdkjIoMdtlkiInc6bF9Y2XARWWYf+0cReVtEPinkPEbY53HSjnmQvTxXDUxEnsnZj0Nt4g4R2QcsEpHvROS+PPteLyLX2c9bicgPInJMRLaJyA0O5YbY79MpEdkvIo/kE2c/4AeggV17+chePlxENtufwxIRiXDYJkFEHhORDcCZvIlBRJbZT9fb+xztsO5hETkkIskicrvD8ir2e79PRA6KyBQR8SvgvXX8rE+IyG4Rucpenmjvf5xD+SAR+VhEDovIXhF5QkQ87HWe9nGPiMhuYGieYwWJyAd2vPvFujThmV9cebabLiIP289D7M/1Hvt1M/vzEnGoqYrIf4GGwDf2+/ZXh13eZL83R0Tk/wo5bi0RmWt/71YDTfOsNyJyr4jsAHbYy64SkTVi1XbXiMhVDuWXiMgLIrLaXj9HRGo6rC/se5KrlUZEPrLfP39gARe/c6dFpMHl3lMH44APjDGbjTHHgeeA24qxvSoNxhh9VMIHkAD0y2d5e+AQ0BnwxPrFTQCq2OtHAQ2w/hkcDZwB6tvrbgMygfsBL8DPXpYB3GXv78/AAUDsbZYAdzpsX1jZX4GXAB+gO3AS+KSA8+sEpAL97VhDgFb5nTvwTM5+sGobBvgY8LfP4VbgZ4fykcAJoIpdJhG43T7n9sARIMoumwz0sJ/XANoXEG9vIMnhdQv7ve0PeAN/BXYCPg7nsA4IA/wK2KcBmuU5RiYwyd7nEOAsUMNe/xowF6gJBALfAC8UsO+cz/p2+7P6B7APeNt+XwYAp4AAu/zHwBx7v42B7cAd9roJwB/2udQEFtuxe9nrvwam2u91XWA1cLdDHCsKiPFPwDf287HALmCWw7o5Bbz3eb8fOd+J9+zvQxvgPBBRwHFnArPteKOB/Y4x2vv6wT5XP/vnceAWrO/QjfbrWg6/I/vtffkDX3Dx+3q570ne78BHwD/yO2+H9+lEIY+Gdrn1wGiH7Wrbx6qVz/uR8/7tB5KAaUBtd/8NrIgPtwegDzd98AUn9HeA5/Is2wb0KmA/64AR9vPbgH151t8G7HR4XdX+5Q62Xy8hd0LPtyxWrSkTqOqw/hMKTuhTgVeLcu7kn9CbOKwPtP9oNrJfPw98aD8fDSzP59hP28/3AXcD1S7zeeT64wo8Ccx2eO1h/0Hs7XAOf7rMPvNL6OewE6W97BDQBRD7HJs6rOsK7Clg37cBOxxex9jHq+ew7CjQFivhn8fqH5Cz7m5gif18ETDBYd0Ae19eQD17Wz+H9TcCix3iKCihN8VKQh7AFPuYSfa66cBDBbz3eb8fOd+JUIdlq4Ex+RzTE+uf0lYOy/7JpQm9r8PrW4DVefbzK3Cbw+/IZId1kUC6fazLfU+KldCL+sD652iQw2tv+1iN8ykbAMQ6fJ6fAwtLclx9FP7QJneVVyPgYbv57oSInMCqOTUAEJFb5WJz/AmsWkNth+0T89nnhV7bxpiz9tOAAo5fUNkGwDGHZQUdK0cY1h+dkrqwb2PMKeBbYIy9aAwww37eCOic5/26CeufEIDrsWrCe0VkqYh0LeLxGwB7HWLItmMKyS/GYjhqjMl0eH0W6/2tg/UP1FqH8/jOXl6Qgw7Pz9lx5l0WgPX98MHhfOznOefSgNzn4liuEVaySHaIaypWTb1QxphdwGmsfyp6APOAAyLSEugFLL3cPvJwvPsg533Lqw5W4irofHI4rm+QTxnH9ydv+b1Y70ntvNsW8D0pDaeBag6vc56fylvQGHPaGBNvjMm0vx/3AQNEpFresurKaEJXeSUCzxtjqjs8qhpjPhWRRljNjvdhNa1VBzZh1e5ymFKKKxmoKSJVHZaFFVI+kTzXLh2cwUpeOfLrVZ73PD4FbrQTsh9Ws3DOcZbmeb8CjDF/BjDGrDHGjMBKQF9jNcUWxQGsZAaAiAjW+e4vJMYrcQQrAUc5nEeQMaagf7yKu+8MHM4Hq8Ul51ySyf1ZNnR4nohVQ6/tEFc1Y0xUEY+9FBiJ1QS93359K9blj3UFbHMl7+thrJakgs4nv2Pk+qwdtnH8rPPuLwPrfb3c9+QsBX/XLzlPEblJcvdGz/vIOZfNWJcecrQBDhpjjuZzrnnlHFcKLaWKTRN65eYtIr4ODy+shD1BRDrbHYb8RWSoiARiXb8zWH+0EKtDVbQrAjXG7AXisTra+diJNa6QTT4AbheRq0XEw+4U1cpetw4YIyLeIhKL9Qf/cuZj/eGchHUdNttePg9oISK32PvzFpGOIhJhx3mTiAQZYzKwrvlnFfGUZwND7fi9sW5pOw/8UsTtwapBNylKQft83gNeFZG6cKEj2cBiHK+gfWdhnc/zIhJo/2P4ENYlE+x1D4hIqIjUwOo9nbNtMvA98LKIVLM/y6Yi0quIh1+K9Q9oTifBJVh9PFbYceWnyO9bXvY+v8T6nlYVkUisfiiFmY/1HRorIl5idWCMxPpu5bhZRCLtf2gnAZ87vK+FfU/WAWPF6ng4CKtlwvE8a4lIkEP8M8zFnuj5PfbZRT8G7rBjqgE8gdWcfwn7b0lL+7OrBbyBdbkl9TLviyomTeiV23ysWlnO4xljTDxWp7S3sDrm7MTuvWqM2QK8jHV97yDWddOfXRjvTVjXdY9idcKahfXH6xLGmNVYHbZexeoct5SLNZknsWrvx7Futfnf5Q5sjDmP9Ye6n2N5uzl+AFYz/AGsZtl/YXUMA+v6aIKInMTq/HVzUU7UGLPNLvsmVk0sDogzxqQXZXvbM8B0u6n6hssVBh7D+rxX2vH+CLQsfJMiux+rZWQ3sALrPfzQXvcesBCro9VvWO+zo1uxmuy3YH1mnwP1i3jcpVh9IHIS+gqsGuuyAreAF4An7PftkrsSiuA+rOb4FKwkN62wwnatdhhWMj6K1bFtmDHmiEOx/9r7SgF8gQfsbS/3PZloL8u5FPS1w3H/wGp52m2fa5F7uRtjvgNexGqp2ms/ns5Zb/e6v8l+2QTr8s0prBa981j9IJST5fQeVqrcEZFZwB/GmKcvW1ipckpElmB12nzf3bGosk1r6KrcsJuym9pNd4OAETjUOJRSqjIrl6MUqUorGKs5thbW/ax/Nsb87t6QlFKqbNAmd6WUUqoC0CZ3pZRSqgLQhK6UUkpVAOX6Gnrt2rVN48aN3R2GUkop5RJr1649YozJdwTHcp3QGzduTHx8vLvDUEoppVxCRPIbShjQJnellFKqQtCErpRSSlUAmtCVUkqpCqBcX0NXSilVuIyMDJKSkkhLS3N3KKoYfH19CQ0Nxdvbu8jbaEJXSqkKLCkpicDAQBo3bow1u6oq64wxHD16lKSkJMLDw4u8nTa5K6VUBZaWlkatWrU0mZcjIkKtWrWK3aqiCV0ppSo4TeblT0k+M03oSimlSs3Ro0dp27Ytbdu2JTg4mJCQkAuv09PTL78D4Pbbb2fbtm2Flnn77beZMWOGM0Kme/furFu3zin7ciW9hq6UUqrU1KpV60JyfOaZZwgICOCRRx7JVcYYgzEGD4/865jTpk277HHuvffeKw+2nNMauu1w4jZWzpiEyc52dyhKKVXh7dy5k+joaCZMmED79u1JTk5m/PjxxMbGEhUVxaRJky6UzakxZ2ZmUr16dR5//HHatGlD165dOXToEABPPPEEr7322oXyjz/+OJ06daJly5b88ssvAJw5c4brr7+eNm3acOONNxIbG3vZmvgnn3xCTEwM0dHR/P3vfwcgMzOTW2655cLyN954A4BXX32VyMhI2rRpw8033+z09+xytIZu27X6O7rseJn1a3rQpnMfd4ejlFIV3pYtW5g2bRpTpkwBYPLkydSsWZPMzEz69OnDyJEjiYyMzLVNamoqvXr1YvLkyTz00EN8+OGHPP7445fs2xjD6tWrmTt3LpMmTeK7777jzTffJDg4mC+++IL169fTvn37QuNLSkriiSeeID4+nqCgIPr168e8efOoU6cOR44cYePGjQCcOHECgBdffJG9e/fi4+NzYZkraUK3te03lsyNz7D355ma0JVSFdKz32xmy4GTTt1nZINqPB0XVaJtmzZtSseOHS+8/vTTT/nggw/IzMzkwIEDbNmy5ZKE7ufnx+DBgwHo0KEDy5cvz3ff11133YUyCQkJAKxYsYLHHnsMgDZt2hAVVXjcq1atom/fvtSuXRuAsWPHsmzZMh577DG2bdvGxIkTGTJkCAMGDAAgKiqKm2++mREjRnDNNdcU8924ctrkbvMNqkNy9ViiTixl64FUd4ejlFIVnr+//4XnO3bs4PXXX2fRokVs2LCBQYMG5Xvblo+Pz4Xnnp6eZGZm5rvvKlWqXFLGGFOs+AoqX6tWLTZs2ED37t154403uPvuuwFYuHAhEyZMYPXq1cTGxpKVlVWs410praE7qN1xJH4//JXJPy4m4lbX/3ellFKlqaQ1aVc4efIkgYGBVKtWjeTkZBYuXMigQYOceozu3bsze/ZsevTowcaNG9myZUuh5bt06cKjjz7K0aNHCQoKYubMmTzyyCMcPnwYX19fRo0aRXh4OBMmTCArK4ukpCT69u1L9+7dmTFjBmfPniUwMNCp51AYTegO/GKGww9/xWf7PPafGEhIdT93h6SUUpVC+/btiYyMJDo6miZNmtCtWzenH+P+++/n1ltvpXXr1rRv357o6GiCgoIKLB8aGsqkSZPo3bs3xhji4uIYOnQov/32G3fccQfGGESEf/3rX2RmZjJ27FhOnTpFdnY2jz32mEuTOYAUtwmiLImNjTXOng/9/NSr2bn/MF92msmTwyIvv4FSSpVhW7duJSIiwt1hlAmZmZlkZmbi6+vLjh07GDBgADt27MDLq2zWbfP77ERkrTEmNr/yZfMs3KhKzDVEJT/Bg6vXkNq3OUFViz4wvlJKqbLr9OnTXH311WRmZmKMYerUqWU2mZdExTkTZ2k1DL5/gp5Zq/hkVTfu7dPM3REppZRygurVq7N27Vp3h1FqtJd7XjXDITiGG/x/Z9rPCaRluLaXolJKKVUSmtDzEzGCFulb8Didwle/73d3NEoppdRlaULPT0QcALfX2sR7y3aTlV1+Ow4qpZSqHFyW0EVkkIhsE5GdInLJOH0i8qqIrLMf20XE9ePm5ajTEmo1Z2TV39l95Aw/bDnotlCUUkqponBJQhcRT+BtYDAQCdwoIrnuCTPGPGiMaWuMaQu8CXzpitjyJQIRcdQ+soaoGhlMXbar2CMMKaWUsnh6el6YMrVt27ZMnjy5RPvp3bs3Jb1VecmSJRcmaQGYMmUKH3/8cYn25SghIYHo6Ogr3o8zuKqXeydgpzFmN4CIzARGAAUN03Mj8LSLYstf5HBkxSv8vWkCN8V7E7/3OB0b13RrSEopVR75+fm5fX7xJUuWEBAQwFVXXQXAhAkT3BpPaXBVk3sIkOjwOsledgkRaQSEA4tcEFfB6reFoDC6pP1CjareTF26y63hKKVURbJgwQJuuOGGC6+XLFlCXJzVf+nPf/7zhWlUn346/7pdQEDAheeff/45t912GwDffPMNnTt3pl27dvTr14+DBw+SkJDAlClTePXVV2nbti3Lly/nmWee4aWXXgJg3bp1dOnShdatW3Pttddy/PhxwGoReOyxx+jUqRMtWrQocCKYHGlpadx+++3ExMTQrl07Fi9eDMDmzZvp1KkTbdu2pXXr1uzYsYMzZ84wdOhQ2rRpQ3R0NLNmzSrZG+nAVQld8llWUBv2GOBzY0y+94uJyHgRiReR+MOHDzstwHwOBBFxeO5ZzB0d6/Dj1kPsPHSq9I6nlFIV1Llz53I1uc+aNYv+/fuzcuVKzpw5A8CsWbMYPXo0AM8//zzx8fFs2LCBpUuXsmHDhiIfq3v37qxcuZLff/+dMWPG8OKLL9K4cWMmTJjAgw8+yLp16+jRo0eubW699Vb+9a9/sWHDBmJiYnj22WcvrMvMzGT16tW89tpruZbn5+233wZg48aNfPrpp4wbN460tDSmTJnCxIkTWbduHfHx8YSGhvLdd9/RoEED1q9fz6ZNm5wybr2rmtyTgDCH16HAgQLKjgHuLWhHxph3gXfBGvrVWQHmK2I4rPwP4+ps4y3vIN5dtpsXR7Yp1UMqpVSpWfA4pGx07j6DY2Bw4dfEC2pyHzRoEN988w0jR47k22+/5cUXXwRg9uzZvPvuu2RmZpKcnMyWLVto3bp1kcJJSkpi9OjRJCcnk56eTnh4eKHlU1NTOXHiBL169QJg3LhxjBo16sL6/KZhLciKFSu4//77AWjVqhWNGjVi+/btdO3aleeff56kpCSuu+46mjdvTkxMDI888giPPfYYw4YNu+SfjJJwVQ19DdBcRMJFxAcrac/NW0hEWgI1gF9dFFfhwjqBf10Cdy9gVIcwvvp9PwdPXjqdn1JKqeIbPXo0s2fPZtGiRXTs2JHAwED27NnDSy+9xE8//cSGDRsYOnRovtOoilxs+HVcf//993PfffexceNGpk6dmu+2xZHfNKwFKajz9NixY5k7dy5+fn4MHDiQRYsW0aJFC9auXUtMTAx/+9vfmDRp0hXFCS6qoRtjMkXkPmAh4Al8aIzZLCKTgHhjTE5yvxGYacpKl3IPT2g1FDbM5q47X2bGqr1M+zmBxwe3cndkSilVfJepSbta7969ueOOO3jvvfcuNLefPHkSf39/goKCOHjwIAsWLKB3796XbFuvXj22bt1Ky5Yt+eqrry7MbJaamkpIiNVFa/r06RfKBwYGcvLkyUv2ExQURI0aNVi+fDk9evTgv//974XaenH17NmTGTNm0LdvX7Zv386+ffto2bIlu3fvpkmTJjzwwAPs3r2bDRs20KpVK2rWrMnNN99MQEAAH330UYmO6chlY7kbY+YD8/MseyrP62dcFU+RRcTB2mk0PL6awTH1mbFyL/f2aUqgr07aopRSRZFzDT3HoEGDmDx5Mp6engwbNoyPPvroQvJt06YN7dq1IyoqqtBpVCdPnsywYcMICwsjOjqa06dPA/DMM88watQoQkJC6NKlC3v27AEgLi6OkSNHMmfOHN58881c+5o+fToTJkzg7NmzNGnShGnTppXoPO+55x4mTJhATEwMXl5efPTRR1SpUoVZs2bxySef4O3tTXBwME899RRr1qzh0UcfxcPDA29vb955550SHdORTp96OVkZ8O+m0HIIGzpOZvhbP/P3Ia0Y37Np6R5XKaWcQKdPLb+KO32qDv16OZ7e0HIIbJtP6/r+dG1Siw9XJJCeme3uyJRSSqkLNKEXRUQcpKVCwnLu7tWElJNpzF1fUCd9pZRSyvU0oRdF077g7Q9b5tKrRR1aBQfy7rJdZOukLUoppcoITehF4e0HzfvDH98iJpvxPZuw/eBplmw/5O7IlFLqsspzX6nKqiSfmSb0ooqIgzOHIHE1cW0a0CDIl6lLd7s7KqWUKpSvry9Hjx7VpF6OGGM4evQovr6+xdrOZbetlXvNB4CnD2z9Bu9GXflT93D+8e1Wft93nHYNa7g7OqWUyldoaChJSUmU6lDZyul8fX0JDQ0t1jaa0IvKt5p1LX3rNzDwecZ0asjrP+3g3WW7eefmDu6OTiml8uXt7X3Z4U9VxaBN7sUREQep+yB5HQFVvLilSyO+25xCwpEz7o5MKaVUJacJvThaDAbxtGrpwG3dGuPt4cF7y/VaulJKKffShF4c/rWgcbcLCb1uoC/XtQ/hs7VJHD513s3BKaWUqsw0oRdXxHA4sh0O/QHAXT2bkJGVzce/Jrg1LKWUUpWbJvTiajXM+mnX0pvWCaB/RD0+/nUvZ84XPrWeUkopVVo0oRdXtfoQ2gm2XpzO/e5eTUk9l8Hs+EQ3BqaUUqoy04ReEhFxkLIBjlnT8nVoVIPYRjV4f/keMrN00hallFKupwm9JCLirJ9/zLuw6O5eTdl/4hzfbkx2U1BKKaUqM03oJVEzHIJjLlxHB7i6VV2a1vFn6tLdOsSiUkopl9OEXlIRwyFxFZxKAcDDQxjfswlbkk+yYucRNwenlFKqstGEXlIRw62fDrX0a9qFUCewCu8u04FmlFJKuZYm9JKq0xJqNc+V0Kt4efKnbuEs33GETftT3RicUkqpykYTekmJWJ3jElbA2WMXFo/t3BB/H0+tpSullHIpTehXIiIOTBZsW3BhUZCfN2M7N+TbjckkHjvrxuCUUkpVJprQr0SDdhAUlmuQGYA/dQ9HgA9W7HFPXEoppSodTehXIqfZfdciOH/qwuL6QX6MaBvCrDWJHD+T7sYAlVJKVRaa0K9URBxkpcOO73MtHt+zCecysvjvyr1uCkwppVRlogn9SoV1Bv86sCV3s3vL4ED6tKzD9F8SSMvIclNwSimlKgtN6FfKw9OagW3HD5BxLtequ3s15eiZdD5fm+Sm4JRSSlUWmtCdISIOMs7ArsW5FncOr0mb0CDeW76brGwdDlYppVTp0YTuDI17gG9QrkFmAESEu3s1Ze/Rs3y/OcVNwSmllKoMNKE7g5cPtBwC2+ZDVkauVQOjgmlUqypTlu7SSVuUUkqVGk3ozhIRB2knIGF5rsWeHsKdPZqwPimVVXuOFbCxUkopdWU0oTtL077gXfWSZneAUR1CqeXvo8PBKqWUKjWa0J3F2w+a94et8yA7921qvt6ejLuqMYv+OMS2lFMF7EAppZQqOU3ozhQxHM4cgsTVl6y6pUsj/Lx10hallFKlw2UJXUQGicg2EdkpIo8XUOYGEdkiIptF5H+uis1pmg8AT598m91r+PswumMYc9btJzn1XD4bK6WUUiXnkoQuIp7A28BgIBK4UUQi85RpDvwN6GaMiQL+4orYnMq3GjTpYyX0fHq039E9HANM+znB5aEppZSq2FxVQ+8E7DTG7DbGpAMzgRF5ytwFvG2MOQ5gjDnkoticK3I4pO6D5HWXrAqrWZWhMfX536p9pJ7LyGdjpZRSqmRcldBDgESH10n2MkctgBYi8rOIrBSRQfntSETGi0i8iMQfPny4lMK9Ai0Gg3jm2+wO1qQtp89n8r9V+1wcmFJKqYrMVQld8lmWt03aC2gO9AZuBN4XkeqXbGTMu8aYWGNMbJ06dZwe6BXzrwWNuxWY0KNDgujerDbTft7D+UydtEUppZRzuCqhJwFhDq9DgQP5lJljjMkwxuwBtmEl+PInYjgc2Q6Ht+W7+u5eTTh06jxzfs/7FiillFIl46qEvgZoLiLhIuIDjAHm5inzNdAHQERqYzXBl897vFoNs35uyXuKlu7NahNZvxpTl+0iWydtUUqp8s0YOHME9q+FzV/Bz2/At4/AjBvg47zdxUqPlysOYozJFJH7gIWAJ/ChMWaziEwC4o0xc+11A0RkC5AFPGqMOeqK+JyuWn0I7QRb50KvRy9ZbU3a0oSJM9ex6I9D9Ius54YglVJKFUlOwj6xD07stX/aj9RE62fG2dzbVAmC6g2hZmNre8nvyrNzSXmeMCQ2NtbEx8e7O4z8/fwG/PAkTFwPNRpfsjojK5ve/15Cg+q+fDbhKtfHp5RSymIMnD5kJ+h9uRP2iX1wIhEy84wf4lvdStj5PYLCwO+SLmBOISJrjTGx+a1zSQ29UoqIsxL61nlw1X2XrPb29OCO7uFMmreFtXuP0aFRTTcEqZRSlUB2tjWK54UEvfdios6pZWem5d7GryZUD4M6La1BwxyTdfUwa8rsMkYTemmpGQ7BMVazez4JHWB0xzBe/2kHU5fu5t1bNaErpVSJZGfD6RSHJJ23WTwJss7n3qZqLStB14uEFgOheiOHWnYYVAl0z7lcAU3opSliOCx+Hk6lQGDwJav9q3hxa9dGvLV4J7sOn6ZpnQA3BKmUUuXQkR3ww1Nw+A87YafnXu9fx6pNB8dAq6F2om5kJeugMKhS8f7eakIvTRFxVkL/Yx50vDPfIuOuaszUZbt5f/luXriutYsDVEqpcmj9LJj3IHj5WMNtRwy3EnVOLTsoFHz83R2ly2lCL011WkGt5tbtawUk9NoBVRjZIZTP45N4sH8L6gb6ujhIpZQqJ9LPwoJH4fdPoOFVMPIDqNbA3VGVGTp9amkSsWrpCSvg7LECi93VowkZ2dlM/yXBdbEppVR5cugPeK8v/D4Dej4K477RZJ6HJvTSFhEHJgu2LSiwSHhtfwZFBfPfX/dy+nymC4NTSqly4PcZ8G5vOHsEbvkS+j4BntrAnJcm9NLWoJ3VAaOAsd1zjO/ZhJNpmcxcrZO2KKUUAOdPw1cTYM49EBoLE1ZA077ujqrM0oRe2nKa3XctgvOnCizWrmENOoXX5MMVe8jIynZhgEopVQYd3Azv9YH1M6H33+DWOfneLaQu0oTuChFx1j2QO74vtNiEXk04kJrGvA06aYtSqpIyBtZ+ZF0vT0uFcZInxVgAACAASURBVHOh9+Pg4enuyMo8TeiuENbZuifyMs3uvVvUpXndAKYu3U15HpJXKaVK5Pwp+OJO+GYiNOxqNbGH93R3VOWGJnRX8PC0BjbY/j1kpBVczEMY37MJf6ScYun2wy4MUCml3Cx5PUztCZu/hL5Pws1fQkBdd0dVrmhCd5WI4ZBxxrqWXogRbUOoV60K7y4rnzPHKqVUsRgDq9+D9/tDxjkYNw96PgIemp6KS98xV2ncwxrM/zLN7j5eHvypWzi/7DrKhqQTLgpOKaXcIC0VPhsH8x+xmtYnrIDG3dwdVbmlCd1VvHygxWDYNh+yMgotemPnhgRW8WKq1tKVUhXV/t+sJvat86DfszB2NvjXdndU5ZomdFeKHA5pJyBheaHFqvl6M7ZLQxZsTGbf0bMuCk4ppVzAGFg5BT4YAFmZcPsC6P4XbWJ3An0HXalpX/Cuetlmd4A/dQvH00N4f4XW0pVSFcS54zDrZvjuMWjWDyYsh4ad3R1VhaEJ3ZW8/aB5f6uJKTur0KL1qvlyTdsQZscncuxMeqFllVKqzEuKhyk9YftCGPhPuPFTqFrT3VFVKJrQXS1iOJw5BElrLlt0fM8mpGVk8/GvCaUellJKlQpj4Jc34cOBIMCfFkLXe61RNJVTaUJ3teYDwNPHmlL1ckXrBdIvoi7Tf0ngXHrhNXqllCpzzh6DT8fA909Ai0Fw93II7eDuqCosTeiu5lsNmvSxrqMXYTS48T2bcvxsBp+tTXRBcEop5ST7VsKUHtbYG4P/DaM/Ab/q7o6qQtOE7g4RcZC6zxoZ6TI6Nq5Bu4bVeW/5bjJ10halVFmXnQ3LX4FpQ8DTG+74HjqP1yZ2F9CE7g4th4B4Fqm3u4hwd8+mJB47x3ebU1wQnFJKldCZI/C/UfDTs9ZtuncvtaaQVi6hCd0d/GtZoyFtvfx1dID+kfUIr+2vk7YopcquhJ9hSnfYsxyGvgIjp1mjYyqX0YTuLhHD4ch2OLztskU9PYS7ejRh4/5Uft111AXBKaVUEWVnwdJ/w/Rh1jgbd/4IHe/QJnY30ITuLq2GWj+LWEu/rn0ItQOq8NpPO7SWrpQqG04fgk+ug8X/gOjrrSb2+q3dHVWlpQndXao1gNBORbp9DcDX25MH+zdn9Z5jzF1/oJSDU0qpy9i9FN7pZvVmH/4mXPceVAl0d1SVmiZ0d4qIg5QNcDyhSMXHdGxI69Ag/vHtVk6mFT7Bi1JKlYrsLFj8T/h4BPjVgLsWQ/tbtYm9DNCE7k4Rw6yfW+cVqbinh/DciGiOnD7Paz/sKMXAlFIqHyeTrUS+9F/Q5kYYvxjqRbo7KmXThO5ONZtAvZgi3b6Wo01YdW7s1JDpvyawNflk6cWmlFKOdv5k9WLfvxaueQeufQd8/N0dlXKgCd3dIodD4io4VfR7zP86sCXVfL148utN2kFOKVW6sjLhp0nwyfUQUNdqYm871t1RqXxoQne3iDjAwB9Fa3YHqF7Vh8cHtyJ+73G++G1/6cWmlKrcUvdbt6Mtfxna3wJ3/gR1W7k7KlUAL3cHUOnVaQW1mlnN7h3vLPJmozqEMXNNIi/M30r/yHoE+XmXYpBKqQov87zVUngq2Xqc2AcrXoOsdLjufWg9yt0RqstwWUIXkUHA64An8L4xZnKe9bcB/wZyqpxvGWPed1V8biNiDTLz8+vWzERFnB/Yw+4gN/ytFbz8/TYmjYgu5UCVUuVSdpY1JGtOoj6VbHVuO5WcO4GfzWfQqvpt4PoPoXYz18etis0lCV1EPIG3gf5AErBGROYaY7bkKTrLGHOfK2IqUyLiYMUrsG0BtLupyJtFhwRxS5dG/HflXm6IDSM6RIdZVKrSMAbSUu2kfOBicj7pkLhPpVgPk2f6ZfEA/7oQGAzVG0JYJwhsYL2uVh8C7YdfDb0drRxxVQ29E7DTGLMbQERmAiOAvAm9cmrQDqqFWs3uxUjoAA8NaMm3G5N54utNfPnnq/Dw0F8+pcq9jDSHhGwn65MHLibonGUZZy/d1jfoYnKu08r6mZOgc5K1f13w1CuuFY2rPtEQwHFC7ySgcz7lrheRnsB24EFjTOWYBFzEqqXHfwjnTxVrtKUgP2/+NjiChz9bz2drExndsWEpBqqUcoqMNNizFFIT7WSdnDuBnzt+6TZevnZybgD120LLBpcm64Bg8Knq+vNRZYKrEnp+1ca891t9A3xqjDkvIhOA6UDfS3YkMh4YD9CwYQVKXpHDYdU7sON7a0zkYriufQgz1+xj8oI/GBgVTPWqPqUUpFLqipw/BfHT4Ne34PRBa5l4QEA9KznXaAwNu+Ru9g6sb63T5m91Ga5K6ElAmMPrUCDXgOTGGMceGe8B/8pvR8aYd4F3AWJjYyvOTdhhncG/jtXsXsyELiJMGhHNsDdX8OLCbfzz2phSClIpVSJnj8GqqbBqCqSdgPBeMOJtqBdt/d5r87dyAld9i9YAzUUkHKsX+xgg18gEIlLfGJNsvxwObHVRbGWDh6c1A9uGz6zmOG/fYm0eUb8a47o2ZtovexgdG0absOqlFKhSqshOJlu18fhpkHEGWg2D7g9BaAd3R6YqIJcMLGOMyQTuAxZiJerZxpjNIjJJRIbbxR4Qkc0ish54ALjNFbGVKRFx1i/97sUl2vwv/ZtTO6AKT87ZRFZ2xWm8UKrcObYbvpkIr7eGle9Y8zbcsxLGzNBkrkqNlOehQ2NjY018fLy7w3CezHR4qRm0HGqNk1wCc9btZ+LMdTx/bTQ3dW7k5ACVUoU6uBmWvwKbvwQPb2h3M3R7wLo2rpQTiMhaY0xsfuv0wk1Z4uUDLQbDtvmQlQGexR/9bXibBny6eh8vfreNQVHB1AqoUgqBKqVySVxjDY+6fQH4BEDX+6DrvVZnNqVcRMdyL2si4qxOMwkrSrR5Tge5M+czefG7bU4OTil1gTGwaxF8NAw+6AeJK6H33+EvG2HAc5rMlctpDb2saXY1eFe1ers37VOiXbSoF8ifuofz7rLd3NAxjA6Najg5SKUqsexs2PatVSM/8Lt1W9nAf0L7cVAlwN3RqUpMa+hljbcfNO9vzb6WnV3i3TxwdXOCq/ny5NfaQU4pp8jKgPUz4T9dYNbNcO4ExL0OE9dbzeuazJWbaUIviyKGW4NOJK0u8S4CqnjxxLAItiSf5JOVe50YnFKVTMY5WP0evNEevrobPLzg+g/gvnjocBt4aT8VVTZok3tZ1HwAePpYze4Nu5R4N0Nj6jOzWSIvfb+NITH1qROof3iUKrK0k9ZwzL++DWcOQWhHGPJvaDFQR2xTZZLW0Msi32rQpA9smWt1vCkhEeHZEVGkZWTxwoLKNU6PUiV25igseh5ei4Yfn4Z6UTBuHtzxA7QcpMlclVlaQy+rIuJgx0JIXg8N2pZ4N03rBHBXjyb8Z8kuxnRsSKfwos23rlSlk7rfGtVt7UfWLGYRcdaobiHt3R2ZUkWiNfSyquUQa9KGrd9c8a7u69uMkOp+PDVnE5lZJe9op1SFdHQXzL0fXm9jjbceOQLuWQWjP9FkrsqVIid0EXlIRNraz7uIyD4R2S0iXUsvvErMvxY06uaUhF7Vx4snh0XyR8oppv+qHeSUAiBlI3x2O7wVC+tnQYdx8MDvcO0UqNvK3dEpVWzFqaE/COyxn78AvAI8D7zm7KCULXIEHNkGh698gJiBUfXo1aIOr/6wnYMn05wQnFLl1L5VMOMGmNIddvwAVz1gDQYz9GWoocMlq/KrOAk9yBiTKiKBQBvgTWPMB0DL0glN0Wqo9XPr3CvelYjw7PAo0jOz+ed87SCnKhljYOePMG0ofDgAktZAnyfgwY3Q/1kIrOfuCJW6YsVJ6IkichXW1KfLjDFZIlINyCqd0BTVGli3yjih2R2gcW1/JvRqwpx1B/hl1xGn7FOpMi07G7bMgXd7wSfXW7OgDZoMD26CXo+Cn46iqCqO4iT0R4HPgf8DnrOXDQNKPvqJuryI4VZP9+POufZ9T59mhNbw46k5m8nQDnKqosrKgHX/g/90htm3wvlTMPxNmLgOuvwZfPzdHaFSTlfk29aMMfOBBnkWf2Y/VGmJGAY/PGnV0q+674p35+vtyTNxUdz5cTwfrtjD3b2aOiFIpdwgMx1SE+F4AhzfY/9MgGP26/TTUC8GRn4IkdeAh6d741WqlBU5oYtIJHDUGHNQRAKwauxZwEtARinFp2o2sf4oOSmhA/SLrEe/iLq8/tMOhrdtQP0gP6fsVymnO3vsYqJ2TNrHEyA1CYxDK5NnFatTW43G0KgrNOtvzYugA8GoSqI4A8v8DxgNHMRK4i2BNGAqcIvzQ1MXRMTBkhfgVIrTpmR8Oi6Kfq8s5R/ztvL2TXqvrXKTrEyHWnbCpYk7LTV3ef86VsIO6wytx1jPazSGmuEQEAweOrSGqryKk9AbG2O2iYgA1wJRwDku3sqmSkvkcFjyT2sGto53OmWXYTWrcm+fZrzyw3bG7DhMj+Z1nLJfpS6Rlmo3he+5NGmfSATj0K/Ww/tiLTu0I9QIv5i0azTWGc2UKkRxEvp5+5a1SCDRGHNERLwA39IJTV1QpxXUamY1uzspoQOM79mEL35L4uk5m1nwlx5U8dJrjKoEsrPg5P6Ck/a547nLV61lJeeQDhB9fe6kXa2BXutWqoSK2+S+CAgE3rKXtUdr6KVPxGp2//kN65piVeeMx+7r7cmzw6O4bdoa3l++h3v7NHPKflUFl7QW1v/vYvI+sQ+yHbrReHhB9YZWgm7Qzk7WOUm7EfgGuSdupSq44vRyf1BEBgAZxpjF9uJsrBHkVGmLGA4rXoXt30HbsU7bbe+WdRkYVY83F+3gmnYhhFTXDnKqAOeOw0+TIH4a+ARA7WZQv7V1ScgxaVcLAU+d90kpVxNTzOk5RaQhEALsN8bsK5Woiig2NtbEx8e7MwTXMQZeaw3pp6DDbRB7B1QPc8qu9584x9UvL6FXizpMvSXWKftUFYgxsGEWfP8EnD0KnSdA779Z0/wqpVxKRNYaY/L9Q12cyVnqi8hSYAfwJbBTRJaKSN5701VpEIExM6wJW35+HV5vDbNugYQVVzRnOkBIdT/u79uchZsPsnjbIScFrCqEw9tgehx8dbdV+x6/FAa9oMlcqTKoyDV0Efka2Af8zRhzRkT8gX8C4caY4aUYY4EqVQ3d0fG9EP8BrJ0OaSegbhR0vhtiRoFP1RLtMj0zm0GvLyMr27DwLz3x9daOSZVa+llY9m/45U1rVLV+z0D7cXpbmFJuVlgNvTgJ/QhQ3xiT4bCsClbTe22nRFpMlTah50g/Cxs/g9XvwsFN4Fsd2t9q9YQvwaxRK3Yc4eYPVvFQ/xY8cHXzUghYlQvbvoP5j0LqPmgzFgY8B/5u+RVXSuXhlCZ34DjWLWuOWgInShqYukI+Va05nCesgNvmQ5Ne8Ovb8EZbmHkT7F5arOb47s1rM7R1fd5evJPEY2dLMXBVJp1ItL43n462vlu3zYdr39FkrlQ5UZyuqC8CP4rIB8BeoBFwO/BkaQSmikEEGnezHqlJsOYDWPuRNRBNnQjodBe0GVOkCSmeGBrB4j8O8ew3m3l/XMfSj125X1aG9Y/g0n9Zr/s9C13vBU9v98allCqWItfQjTHvYQ39WhuIs3/eAoSWTmiqRIJCod/T8NAWGPG29Uf524fglQhY+H/WvcOFqB/kx8Srm/Pj1kP8uOWgi4JWbrP3F5jSA358Gpr0gXtXQfe/aDJXqhwq9m1ruTa2rqGfNca4pQdVpb+GXhTGQOIqWDXVmhfaZEOLgdBpPDTtm+/EFRlZ2Qx5fTnnMrL48aFe2kGuIjpzBH54CtbNgKCGMORFaDnY3VEppS7DWdfQC9y/E/ahSosINOwCo6bBg5ug5yOQFA+fXAdvd4LV71lzRTvw9vRg0ohoko6f4z+Ld7opcFUqsrOtyzFvxVr3lnd/EO5dqclcqQpAa+iVUeZ52PwVrJoCB36HKtWg7U3WtfZaF+dHnzjzdxZsTOH7B3vSuPblr7+rMi5lI8x7CJJWW+MZDH0F6rZyd1RKqWK4otvWRKRvIat9gG81oZdTxli19dVTYfPX1njczfpb97Q3vZpDp9Pp+/JSOjSqwUe3d0R0Xuny6fwpWPyC9Q+cXw0Y8A+rk6R+nkqVO1ea0C87+YoxJryEsV0RTehOdCrFGqM7/kM4cwhqNoVO4/k4rRtPfbePKTd3YFC0c+ZiVy5ijNVv4ru/walka8jgq59y2uQ+SinXc8rAMmWRJvRSkJluJYFVU2B/PMYngG/oxadmIB88ehNVfXTSjXLh2G5rcJidP0JwDAx9FcL0NkSlyrvS7hRX1CAGicg2EdkpIo8XUm6kiBgR0VlC3MHLB1qPgrt+grsWIa2GMSzzBz7NeICDbw+xRhHLznZ3lKogmedh6Yvwn66wbxUMmgx3LdFkrlQl4JIauoh4AtuB/kASsAa40RizJU+5QOBbrGvz9xljCq1+aw3dRU4fYv70ybQ/9CXBctyapKPTeKsjnV91d0encuxeAt8+DEd3QtS1MPCfUE3nTlKqIikLNfROwE5jzG5jTDowExiRT7nnsEakS3NRXKooAurS8dYXGCxv82bNv2MCgmHh3+GVSJj3IBz6w90RVm6nUuDzO+DjEZCdBTd/CaM+0mSuVCXjqoQeAiQ6vE6yl10gIu2AMGPMvMJ2JCLjRSReROIPHz7s/EhVvuoEVuEvA6J4+UA033acZk2jGXUN/D4D/tMZpg+HP761EopyjewsWPUuvNURts6FXo/DPSuh2dXujkwp5Qau6uGU3/0xF9r6RcQDeBW47XI7Msa8C7wLVpO7k+JTRXBzl0bMjk/kuXlb6P1wbwKu+Q/0nwS/TbfGj585FgIbQEAdewuHj/3CLVKS/+tSKVPANlUCrfvtazWHWs2sh3/t8nUb1/611j3lyeusIVuHvpxrDAGlVOXjqoSeBIQ5vA4FDji8DgSigSX2vc7BwFwRGX656+jKdTw9hEkjorn+nV9446cd/H1IhJUIezwMV020JoPZ/KXVMStX3wz7+YVleV+XtAz5lzGm8P2cSoHtC6377nP4Bl1M7rkeTYs0qY3LnDsBi56z/oEKqAcjP4So68rXPyNKqVLhqoS+BmguIuHAfmAMMDZnpTEmFWuyFwBEZAnwiCbzsqdDoxqMjg3jwxV7GNkhlBb1Aq0Vnl5WE3zUNe4NsKiyMiE10epA5vhI+NkaEtVRYAOonU+yr97IOm9XMAY2zIbv/w/OHrUG/+nzf+BbzTXHV0qVeS75a2SMyRSR+4CFgCfwoTFms4hMAuKNMXNdEYdyjr8Oasl3m1N48utNzBzfpXyOIOfpBTXDrUfz/rnXpZ+17uM+uhOO7oCju6znm76EtBMXy3l4QY1wqN3cbsLPSfbNIaCu82rNh7dbM+YlLIeQDnDT59CgrXP2rZSqMHRgGVUiM1bt5f++2sTrY9oyom3I5TeoCIyBs8ccavQ77J+7rEfW+Ytlfezr9LWb526+r9XMuoZfFOlnYflL8PMb4FMVrn7aGu3NQ2e/U6qyKuy2NR32S5XImI4NmbUmkX98u5U+repSzbcSzJ8tAv61rEfDzrnXZWdBapJDgreTfeIq2Pg5Dn1AISD4YoJ3TPg1Gl+ch3z7Qpj/CJzYB63HwIDnrFq/UkoVQGvoqsTWJ57gmv/8zO1XhfNUXKS7wym7MtIcmvDzJPyzRy+WE08rqftVt3qx125p9V4P7+G20JVSZYvW0FWpaBNWnRs7NWT6rwmMig0lor520MqXty/Ui7QeeZ09ZiX7IzsuJvzURGsSla73W0PxKqVUEWhCV1fkrwNbsmBjMk/N2cTsu7uWzw5y7lS1pvUI1akLlFJXxmWTs6iKqXpVHx4f3Io1Ccf58rf97g5HKaUqLU3o6oqN6hBGu4bVeWHBVlLPZVx+A6WUUk6nCV1dMQ8P4bkR0Rw7k84r329zdzhKKVUpaUJXThEdEsQtXRrx35V7mfbzHjKzdM50pZRyJU3oymkeHtiSbs1q8+w3Wxj6xgp+3XX08hsppZRyCk3oymmq+Xrz8Z86MeXmDpxJz+TG91Zy7/9+48CJc+4OTSmlKjxN6MqpRIRB0cH8+FAvHuzXgh+3HKTvy0t486cdpGXoXOlKKVVaNKGrUuHr7cnEfs356eFe9GlZl5d/2M6AV5fxw5aDlOfRCZVSqqzShK5KVWiNqrxzcwdm3NmZKl4e3PVxPLdNW8Ouw6fdHZpSSlUomtCVS3RrVpv5E3vw5LBIftt7nEGvLeOF+Vs5fT7T3aEppVSFoAlduYy3pwd3dA9n0SO9uaZtCFOX7abPS0v48rckbYZXSqkrpAlduVydwCr8e1Qbvr63Gw2CfHlo9npGTvmVTftT3R2aUkqVW5rQldu0DavOV/d048XrW5Nw5Axxb63gb19u5NiZdHeHppRS5Y4mdOVWHh7CDR3DWPRIb26/KpzZ8Yn0eWkJH/+aoKPNKaVUMWhCV2VCkJ83T8VFsmBiD6IaVOOpOZsZ9uYKVu3W0eaUUqooNKGrMqVFvUBm3NmZd25qz6m0TEa/u5L7P/2d5FQdbU4ppQqjCV2VOSLC4Jj6/PhQLx64ujkLN6fQ96WlvL14J+czdbQ5pZTKjyZ0VWb5+XjyUP8W/PRQL3q2qM2/F25jwKvL+GnrQXeHppRSZY4mdFXmhdWsytRbYvnvHZ3w8hDumB7P7dNWs+fIGXeHppRSZYYmdFVu9GhehwUTe/J/QyJYk3CcAa8uZfKCPzijo80ppZQmdFW++Hh5cFfPJix6pBfD24QwZeku+r68hDnr9utoc0qpSk0TuiqX6gb68vINbfjiz1dRN9CXiTPXccPUX9l8QEebU0pVTprQVbnWoVEN5tzbjcnXxbDr8Bni3lzBE19v5LiONqeUqmQ0oatyz8NDGNOpIYsf7s2tXRvz6epE+ry8hE9W7iUrW5vhlVKVgyZ0VWEEVfXmmeFRfPtAd1oFB/LE15uIe3MFaxKOuTs0pZQqdZrQVYXTKrgan97VhbfGtuP42XRGTfmVv8z8nZTUNHeHppRSpUYTuqqQRIRhrRvw08O9uL9vM+ZvSqHvy0t4Z8kuHW1OKVUhSXm+1Sc2NtbEx8e7OwxVDuw7epZJ87bw49aD1A/yZWSHUK5vH0rj2v7uDk0ppYpMRNYaY2LzXacJXVUmS7cf5sMVe1i+4zDZBjo1rsnIDqEMaV2fgCpe7g5PKaUKVSYSuogMAl4HPIH3jTGT86yfANwLZAGngfHGmC2F7VMTuiqplNQ0vvw9ic/XJrH78Bn8vD0ZHBPMqA5hdA6viYeHuDtEpZS6hNsTuoh4AtuB/kASsAa40TFhi0g1Y8xJ+/lw4B5jzKDC9qsJXV0pYwy/7TvB52uTmLf+AKfOZxJW04/r21tN8mE1q7o7RKWUuqCwhO6qNsZOwE5jzG47oJnACOBCQs9J5jZ/oPxeC1DlhojQoVENOjSqwVPDIvl+SwqfxSfx+k87eO3HHXRtUouRHUIZHBNMVR9tkldKlV2u+gsVAiQ6vE4COuctJCL3Ag8BPkBf14SmlMXPx5MRbUMY0TaE/SfO8eXaJD7/LYmHP1vPU3M2MbR1fUbFhhHbqAYi2iSvlCpbXNXkPgoYaIy50359C9DJGHN/AeXH2uXH5bNuPDAeoGHDhh327t1beoGrSs8Yw5qE43y+NpFvNyRzJj2LxrWqMrJDKNe1D6VBdT93h6iUqkTKwjX0rsAzxpiB9uu/ARhjXiigvAdw3BgTVNh+9Rq6cqWz6Zks2JjCZ2sTWbn7GCLQvVltRnYIZWBUML7enu4OUSlVwZWFa+hrgOYiEg7sB8YAYx0LiEhzY8wO++VQYAdKlSFVfby4vkMo13cIJfHYWT5fm8QXvyUxceY6Aqt4MaxNA0bFhtIurLo2ySulXM6Vt60NAV7Dum3tQ2PM8yIyCYg3xswVkdeBfkAGcBy4zxizubB9ag1duVt2tmHlnqN8vjaJBRtTOJeRRZM6/hcGrqlXzdfdISqlKhC3N7mXFk3oqiw5fT6T+RuS+XxtEqsTjuEh0LNFHUZ2CKVfRD1tkldKXTFN6Eq5WMKRM3zxWxJfrE3iQGoaQX7eDLeb5GNCgrRJXilVIprQlXKTrGzDr7uO8tnaRL7blML5zGxa1AtgZIdQrmkXQt1AbZJXShWdJnSlyoCTaRnMW5/M52sT+W3fCTw9hN4t6jAqNpS+rerh46WTHyqlCqcJXakyZueh03zxWxJf/pbEwZPnqVHVmxFtQxjZIZTokELv1lRKVWKa0JUqo7KyDct3HOaztUn8sPkg6VnZRNSvxvA2DRgSE0yjWjq9q1LqIk3oSpUDJ86m8836A3z+237WJ54AIKpBNYbE1GdwdDBN6gS4OUKllLtpQleqnEk6fpbvNqWwYFMKa/ceB6BVcCCDo+szJCaY5vUC3RyhUsodNKErVY4lp55j4aYU5m9KYU3CMYyBZnUDGBIdzOCY+rQKDtTb4JSqJDShK1VBHDqZxsLNKczfmMKqPUfJNhBe25/B0cEMialPVINqmtyVqsA0oStVAR05fZ7vNx9kwaZkftl1lKxsQ1hNP4ZE12dwTH3ahOoANkpVNJrQlargjp9J54ctB5m/KZmfdx4hI8sQUt2PQdHBDIkJpl1YDTw8NLkrVd5pQleqEkk9m8GPW62a+7LtR0jPyqZetSoMjrZ6y8c2romnJnelyiVN6EpVUqfSMlj0xyHmb0xmybbDnM/MpnZAFQZF12NIdH06hdfEy1NHqFOqvNCErpTizPlMFm87xIKNKSz64xDnMrKo6e/DwKh6DI6uT9emtfDW5K5UmaYJXSmVy9n0TJZuO8z8TSks2nqQM+lZBPl5MyCyHkNi6tOtWW0dW16pMkgTulKqQGkZWSzbfpgFm1L4cctBTp3P46cYqgAAEV5JREFUJNDXi/4R9RgcU58ezWvrXO5KlRGFJXQvVwejlCpbfL09GRAVzICoYM5nZvHzziPM35jC95tT+PL3/fj7eHJ1RD2GxATTq0Vd/Hw0uStVFmkNXSmVr/TMbH7dfZQFG5NZuDmF42cz8PP2pG+rugyIqkfXprV0PnelXEyb3JVSVyQzK5tVe44x307uR06nA9Cktj+dm9Skc3gtOjepSf0gPzdHqlTFpgldKeU0WdmGjftTWbX7KKv2HGPNnmOcOp8JwP+3d+/BcZ3lHce/j7S6ra67lizZkmXJxpYDuTq+hIQmQICGNE2gU2jSoQNtgE4nlFA6ZSj9g2k7U9JphykDTFsa0mbaNAyE0CRtSGIgwJROfE1C7NiyHUu2ZVsXe2V7ZdmSZT394xytdmXZASzprFa/z8zOXs5m99FG3p/ey3nf1mScje1JNq5YxMb2JMuS8YirFSksCnQRmTUXxp3dx07zUhjwW7pSnDp7HoDmugo2rkhyU9iCb03GtRytyBVQoIvInBkfdzr70pkW/OauFKkzQRd9U005G1ck2dAedNOvbKhUwIv8EhToIhIZd2d//xAvdaUyIT+QHgGgvqos7KIPAn7V4iqtOS9yGQp0Eckb7k7X8TNB6z0M+GOnzgGQrCxlfVsiM8nuqqYaBbxIFp2HLiJ5w8xY0VDFioYq7tvQirtzOHWWl7pOsPlAis1dJ3h+Vx8AtRUlrG9LclPYgn/r0hptLCNyCQp0EYmUmdG6KE7rojgfXrcMgCMnzwat9zDgf7A7CPjqshjr2hKZWfRXN9dq/XmRkLrcRSTv9Z46x+auE5lu+jcGzgAQLy3mxuUJbgoD/tqWOq1BLwVNY+giUlAG0iNs6Qpa75sPpOjsSwNQXlLE2tZgDH5De5Lrl9VpqVopKAp0ESloqTOjOQG/u/c07lBSbFzdXMv6tiTr25KsW54gUVkadbkivzIFuogsKKeGz7Pj0CBbulNs607x6uFTjF4YB2DV4irWtSXZ0J5g3fIkLYkKnQsv84YCXUQWtHPnL/DznlNs7U6xtTvF9u7BzHK1S2rLg4BvS7CuLUlHY7VOlZO8pdPWRGRBKy8pZkN7sEIdBMvVdvam2XYwWKp2S9cJnnn1KAA15TFuXJ5gfXvQTX9tSy1lMY3DS/5TC11EFjx3p2fwbKYFv7V7kP39QwCUxoq4rmVyHH7t8gS1FSURVywLVV50uZvZHcBXgGLgYXd/aMrxzwIfB8aAAeAP3P3g5V5TgS4isyV1ZpRtWQG/88gpxsYdM+horGZDezLsqk/SVKt94WVuRB7oZlYM7AXeC/QAW4H73P31rOe8C9js7sNm9kfAO939dy73ugp0EZkrw6NjvHL4JFu7Btl2MMX2g4MMj14AoCVRwYa2ZGay3cqGKk20k1mRD2PoG4D97n4gLOhbwD1AJtDd/cWs578EfGSOahMReVPx0hg3r6zn5pX1AIxdGGf3sXRmJv1P9w3w5MtHAEjES1jXlmR9W4L1bVrRTubGXAV6M3A4634PsPEyz78f+P6sViQicgVixUVc01LLNS213P+Odtyd7hPDbO1KZcbiN70eLFlbXlLEDcsSQcC3J7mhNUFVmeYky8yaq9+o6fqepu3rN7OPAOuA2y5x/JPAJwFaW1tnqj4RkStiZrTXV9JeX8mH1wdr0venz7GtezAT8F97cT/jP4LiIqOjsZoVDZW0JuOZy7JknCW15cTUmpdfwVwFeg+wLOt+C3B06pPM7D3AXwC3ufvIdC/k7t8AvgHBGPrMlyoiMjMWV5dz5zVLuPOaJQAMjYyx4+Ag27pTvHz4JDuPnOK5nb2MjU9+lcWKjOZERSbgM2GfCK5r45phL9Obq0DfCqwys3bgCHAv8LvZTzCzG4B/Bu5w9/45qktEZM5UlcW4dXUDt65uyDx2Ydw5duosh1LDHE4Ncyg1zKFUcP+5nb2kzozmvEZNeSzYnW5K4Lcm4yytq9BY/QI2J4Hu7mNm9ingeYLT1h5x911m9lfANnd/Gvg7oAr4Tjg79JC73z0X9YmIRKW4yGhJxGlJxGHlxcfT585zODU18IfZ05vmB6/3Z5a0BSgyWFpXkdOFnx34dfESzb4vYFpYRkRknhofd/rS5zh0YviiwD+UOsvxodyRy+qyWCbklyVzu/WbExVaEW8eyIfT1kREZIYVFRlLaitYUlvBxhWLLjp+ZmSMnsGzmZCfCPz9A0P8qLOf0bHJ1r0ZLKkpz2nVtzdU0tFYTVt9pbry5wEFuohIgaosi9HRVE1HU/VFx8bHnYGhkSDsp7Twf7J3gP70ZOu+pNhY2VDF6sbgtVY3VtPRWE1LokIb2eQRBbqIyAJUVGQ01pTTWFPO+rbkRcfPjl7gwPEh9val6ewNrrcfHOTpVydPUKooKWZ145Sgb6pmcXWZxuojoEAXEZGLVJQW87altbxtaW3O4+lz59nXP8Te3jSdfWn29qV5sXOA72zvyTyntqKEjsZqVjdV0dFUE9xurKIuXjrXP8aCokAXEZFfWHV5CWtbE6xtTeQ8fmJohL19YYu+L83e3jRPvXKU9LlDmec01pRluutXNwXXqxqriJcqimaCPkUREblii6rKeHtVGW9fOTk5z93pPX2Ozt50Ttf9v790kJGsCXmtyXjYXT/Zfb+ivorSmCbi/TIU6CIiMivMJmfhv7NjcebxC+PO4dRwpiXf2ZemszfNjzv7M6vmxYqCpXQnWvITQd+ajFOsiXjTUqCLiMicKi4y2uoraauv5Nff1pR5fGTsAl3Hz+S06F/rOcX//PxY5jllsSJWTUzEC0N+TVMNjTWaiKdAFxGRvFAWK2ZNUw1rmmpyHh8eHWNf31BOi/5n+4/z5I4jmedMTMSbOE1vTVMwTl9TvnDWvlegi4hIXouXxrhuWR3XLavLefzk8CidYcDv6Q267f/r5SOkR8Yyz2muq8gJ+UIen1egi4jIvFQXL2XjikU5q+S5O0dOnqWzdzLkO3vT/HTvQM74/IqGSjqaaoKQD1v2LYmKed1tr0AXEZGCYTa52c3tVzVmHh8dG6fr+Bn29J7OhPyOg4M8k7VQTlVZjNWNVZNBH7bq58v58wp0EREpeKWxommXwU2fO8/erC77Pb1pnn3tGI9vyT1/PlggZzLs37K4ivKS/NrMRoEuIiILVnV5CTcuT3Lj8snlb92d/vRIGPKnM2H/6IETmQ1tigza6ivDLvuaTGu+NRmPbH17BbqIiEgWs8l17m9b3ZB5fOzCON0nhsMu+yDoXz96mu/v7GViJ/KJ9e0n1rZf01TDzSsXzUnIaz90ERGRK5A5rW5iIl5fME5/fGiUmvIYr37xfTM22U77oYuIiMySS51Wd3xohKMnz87ZzHkFuoiIyCyoryqjvqpszt6v8M6sFxERWYAU6CIiIgVAgS4iIlIAFOgiIiIFQIEuIiJSABToIiIiBUCBLiIiUgAU6CIiIgVAgS4iIlIAFOgiIiIFYF5vzmJmA8DBGXzJeuD4DL7ebFCNVy7f64P8rzHf64P8rzHf6wPVOBNmur7l7t4w3YF5Hegzzcy2XWoXm3yhGq9cvtcH+V9jvtcH+V9jvtcHqnEmzGV96nIXEREpAAp0ERGRAqBAz/WNqAv4BajGK5fv9UH+15jv9UH+15jv9YFqnAlzVp/G0EVERAqAWugiIiIFQIEeMrM7zKzTzPab2eejrmcqM3vEzPrNbGfUtUzHzJaZ2YtmttvMdpnZg1HXNJWZlZvZFjN7NazxL6OuaTpmVmxmL5vZf0ddy3TMrNvMXjOzV8xsW9T1TGVmdWb2hJntCX8f3x51TdnMrCP87CYup83sM1HXNZWZ/Un472SnmT1uZuVR15TNzB4Ma9uVL5/fdN/TZpY0s01mti+8TszW+yvQCb5Aga8D7wfeCtxnZm+NtqqL/BtwR9RFXMYY8KfufhVwE/BAHn6GI8C73f064HrgDjO7KeKapvMgsDvqIt7Eu9z9+jw9XegrwHPuvga4jjz7LN29M/zsrgduBIaB70VcVg4zawY+Daxz96uBYuDeaKuaZGZXA58ANhD8P77LzFZFWxUw/ff054Efuvsq4Ifh/VmhQA9sAPa7+wF3HwW+BdwTcU053P2nQCrqOi7F3Y+5+47wdprgS7Q52qpyeWAovFsSXvJqEomZtQC/ATwcdS3zkZnVALcC3wRw91F3PxltVZd1O/CGu8/kAlkzJQZUmFkMiANHI64n21XAS+4+7O5jwE+AD0Zc06W+p+8BHg1vPwp8YLbeX4EeaAYOZ93vIc/CaD4xszbgBmBztJVcLOzOfgXoBza5e77V+A/A54DxqAu5DAdeMLPtZvbJqIuZYgUwAPxrOGzxsJlVRl3UZdwLPB51EVO5+xHg74FDwDHglLu/EG1VOXYCt5rZIjOLA3cCyyKu6VIa3f0YBA0fYPFsvZECPWDTPJZXLbf5wsyqgO8Cn3H301HXM5W7Xwi7OluADWHXXV4ws7uAfnffHnUtb+IWd19LMET1gJndGnVBWWLAWuAf3f0G4Ayz2MV5JcysFLgb+E7UtUwVjvPeA7QDS4FKM/tItFVNcvfdwN8Cm4DngFcJhv0WNAV6oIfcv+5ayK/upXnBzEoIwvwxd38y6nouJ+yG/TH5NS/hFuBuM+smGPZ5t5n9R7QlXczdj4bX/QRjvxuirShHD9CT1fPyBEHA56P3AzvcvS/qQqbxHqDL3Qfc/TzwJHBzxDXlcPdvuvtad7+VoJt7X9Q1XUKfmS0BCK/7Z+uNFOiBrcAqM2sP/2q+F3g64prmFTMzgnHL3e7+5ajrmY6ZNZhZXXi7guBLa0+0VU1y9z939xZ3byP4HfyRu+dNqwjAzCrNrHriNvA+gu7PvODuvcBhM+sIH7odeD3Cki7nPvKwuz10CLjJzOLhv+3bybPJhWa2OLxuBX6L/P0snwY+Gt7+KPDUbL1RbLZeeD5x9zEz+xTwPMFszkfcfVfEZeUws8eBdwL1ZtYDfNHdvxltVTluAX4PeC0cowb4grs/G2FNUy0BHg3PaigCvu3ueXlqWB5rBL4XfMcTA/7T3Z+LtqSL/DHwWPjH+QHg9yOu5yLhuO97gT+MupbpuPtmM3sC2EHQlf0y+bci23fNbBFwHnjA3QejLmi672ngIeDbZnY/wR9KH5q199dKcSIiIvOfutxFREQKgAJdRESkACjQRURECoACXUREpAAo0EVERAqAAl1EZpWZuZm9Jeo6RAqdAl1kgQm3Pz1rZkNZl69FXZeIXBktLCOyMP2mu/8g6iJEZOaohS4iAJjZx8zsZ2b2VTM7ZWZ7zOz2rONLzexpM0uZ2X4z+0TWsWIz+4KZvWFm6XAntuz9Ed5jZvvMbNDMvh4uJyoiM0gtdBHJtpFgQ5N6gvWxnzSzdndPEayVvYtg9601wCYzO+DuPwQ+S7A2+Z3AXuBaYDjrde8C1gM1wHbgGYJdskRkhmjpV5EFJtzNrZ7c7Sb/jGBN7L8Bmj38YjCzLcBXCXam6wbq3D0dHvsSsMTdP2ZmncDn3P2ijSfMzIFfc/f/De9/m2CXsYdm5QcUWaDU5S6yMH3A3euyLv8SPn7Ec//KP0jQIl8KpCbCPOtYc3h7GfDGZd6vN+v2MFB1ZeWLyFQKdBHJ1jxlfLsVOBpekhNbp2YdOxLePgysnJsSRWQ6CnQRybYY+LSZlZjZh4CrgGfd/TDwf8CXzKzczK4F7gceC/+7h4G/NrNVFrg23NpSROaIJsWJLEzPmNmFrPubgKeAzcAq4DjQB/y2u58In3Mf8E8ErfVB4Ivuvik89mWgDHiBYHx+D/DB2f4hRGSSJsWJCBCctgZ83N3fEXUtIvLLU5e7iIhIAVCgi4iIFAB1uYuIiBQAtdBFREQKgAJdRESkACjQRURECoACXUREpAAo0EVERAqAAl1ERKQA/D9RljKubkU+CQAAAABJRU5ErkJggg=="
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Learning curves for the model with dropout set to 0.5.\n",
    "\n",
    "dropout = 0.5\n",
    "\n",
    "![assets/learning_curves_droput05.png](attachment:learning_curves_droput05.png)\n",
    "\n",
    "With dropout probability set to 0.5 the validation loss decrease longer than in the first case and reverses at 5th epoch (at the value of loss a bit greater than with dropout = 0.2). This denotes that the overfitting problem still occurs in our model. The validation accuracy is comparable to the previous model.\n",
    "\n",
    "In the next training procedure, we will use the spatial dropout instead of the traditional dropout after the embedding layer."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Learning curves for the model with spatial dropout set to 0.5.\n",
    "\n",
    "dropout = 0.5 <br>\n",
    "spatial_dropout = True\n",
    "\n",
    "<div>\n",
    "<img src=\"assets/learning_curves_spatial_droput.png\" width=\"500\" />\n",
    "</div>\n",
    "\n",
    "When we use the spatial dropout instead of normal dropout the evaluation loss decrease to the value around 0.36, similar to the first example where we used dropout = 0.2, but lower than with the dropout probability set to 0.5. With spatial dropout, the loss begin to reverse at the second epoch, but it increases much slower than in the previous cases (the gap between training loss and validation loss is smaller). As a summary, we can say that considering all regularization methods results, the spatial dropout provides the model best behaviour.\n",
    "\n",
    "The next hyperparameter that we are going to tune is the hidden_size."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Learning curves for the model with hidden_size = 64.\n",
    "\n",
    "There are many rules regarding the choice of the hidden size. For instance, we can use the following formula to calculate the number of hidden units $N_{h}$ [https://ai.stackexchange.com/questions/3156/how-to-select-number-of-hidden-layers-and-number-of-memory-cells-in-an-lstm]:\n",
    "<br>\n",
    "\n",
    "<p><center>$N_{h} = \\frac{N_{s}}{(\\alpha \\cdot (N_{i} + N_{0}))}$</center></p>\n",
    "\n",
    "Where: <br>\n",
    "$N_{s}$ - number of samples in training data set. <br>\n",
    "$\\alpha$ - an arbitrary scaling factor usually 2-10.<br>\n",
    "$N_{i}$ - number of input neurons.<br>\n",
    "$N_{0}$ - number of output neurons.<br>\n",
    "\n",
    "Nevertheless in most cases, the number of hidden units have to be figured out by series of trials.\n",
    "\n",
    "dropout = 0.5 <br>\n",
    "spatial_dropout = True <br>\n",
    "hidden_size = 64 <br>\n",
    "\n",
    "<div>\n",
    "<img src=\"assets/learning_curves_hidden64.png\" width=\"500\" />\n",
    "</div>\n",
    "\n",
    "As we can see reducing the number of hidden units makes the validation loss curve more flat after the first epoch than it was in the preceding examples.\n",
    "\n",
    "We will go further and decrease the hidden_size to 32."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Learning curves for the model with hidden_size = 32.\n",
    "\n",
    "dropout = 0.5 <br>\n",
    "spatial_dropout = True <br>\n",
    "hidden_size = 32 <br>\n",
    "\n",
    "<div>\n",
    "<img src=\"assets/learning_curves_hidden32.png\" width=\"500\" />\n",
    "</div>\n",
    "\n",
    "After decreasing the hidden_size to 32, the validation loss became even more flat and at the 3rd epoch, it begins to fluctuate around the value of 0.38. We can see that decreasing the number of hidden units helped and learning curves starting to look as expected, so we will diminish the hidden_size one more time."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6. Learning curves for the model with hidden_size = 16.\n",
    "\n",
    "dropout = 0.5 <br>\n",
    "spatial_dropout = True <br>\n",
    "hidden_size = 16 <br>\n",
    "\n",
    "<div>\n",
    "<img src=\"assets/learning_curves_hidden16.png\" width=\"500\" />\n",
    "</div>\n",
    "\n",
    "This time the validation learning curve takes a bit lower values and it is more smooth than the previous one."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 7. Learning curves for the model with hidden_size = 8.\n",
    "\n",
    "dropout = 0.5 <br>\n",
    "spatial_dropout = True <br>\n",
    "hidden_size = 8 <br>\n",
    "\n",
    "<div>\n",
    "<img src=\"assets/learning_curves_hidden8.png\" width=\"500\" />\n",
    "</div>\n",
    "\n",
    "The validation loss slowly, but constantly decreases during the training, so we can say that overfitting has been overcame."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 8. Learning curves for the model with n_layers = 2.\n",
    "\n",
    "dropout = 0.5 <br>\n",
    "spatial_dropout = True <br>\n",
    "hidden_size = 8 <br>\n",
    "n_layers = 2 <br>\n",
    "\n",
    "<div>\n",
    "<img src=\"assets/learning_curves_nlayers2.png\" width=\"500\" />\n",
    "</div>\n",
    "\n",
    "Using one addtional GRU layer didn't make a difference for our model's performance."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 9. Learning curves for the model with batch_size = 16.\n",
    "\n",
    "Now we are going to change the batch_size to 16, as reducing batch size might work as regularizer since losses calculated on small batches are noisier, thus model has a lower tendentious to overfitting. The drawback of this method is the processing time, which increases while batch size decreases. Notice that we will change back the n_layers to 1.\n",
    "\n",
    "dropout = 0.5 <br>\n",
    "spatial_dropout = True <br>\n",
    "hidden_size = 8 <br>\n",
    "n_layers = 1 <br>\n",
    "batch_size = 16 <br>\n",
    "\n",
    "<div>\n",
    "<img src=\"assets/learning_curves_batch16.png\" width=\"500\" />\n",
    "</div>\n",
    "\n",
    "Reducing the batch_size cause that the validation loss diminished to the value around 0.33 (among to the lowest of all our trials), but then the loss began to increase. That might be only some local peak, or the model started to overfit to the training dataset.\n",
    "\n",
    "The validation loss curve tends to be under the training curve for most of the time. The explanation of why it behaves like that is the following:\n",
    "- The validation error is smaller than the training error because the dropout is active during the training, while during the evaluation, dropout layer is inactive, thus we got more smooth results in the second case.  \n",
    "- The training loss is measured as the mean loss on the entire training set, and since the performance improves during the training, the general training loss depicted on the plot is worse than the lowest loss calculated at the end of the set. While the validation loss is calculated using the model's last state, which is associated with the best performance at the given time."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 10. Learning curves for the model with batch_size = 64.\n",
    "\n",
    "Increasing the batch_size will speed up the training process.\n",
    "<br>\n",
    "<div>\n",
    "<img src=\"assets/learning_curves_batch64.png\" width=\"500\" />\n",
    "</div>\n",
    "\n",
    "As we can see, we gained positive results by using larger batch_size, namely, the learning process was much faster and the validation loss is comparable to the previous results, it doesn't even raise at the end of the training."
   ]
  },
  {
   "attachments": {
    "learning_curves_batch128.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfQAAAFQCAYAAABJW4xyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3wUdf7H8dcnPaRBIEDovfeOAiI2mnIqiAUpYjsV/Z0NvfPuOO84Pc87+52NagV7A9shTenSe+iBAEkoIT2bfH9/zCQsIYEEkp3dzef5eOwju7Ozs5/dbPLe78xnZsQYg1JKKaV8W4DTBSillFLq4mmgK6WUUn5AA10ppZTyAxroSimllB/QQFdKKaX8gAa6Ukop5Qc00FWFEJH5IjLO6Tq8gYj8TURSRORwJS1/oIgkVsayK5KINBERIyJBZZh3vIgsvcjn6y8i2yuiHnv+mSLyt4upqYzPU6G/z/O9D8p/aaD7OBHZKyJXOl2HMWaIMWaW03U4TUQaAo8A7YwxdStomUZEWlTEsvyZMWaJMaZ14W0n/zZEZIqIvOvEcxd/H5wkIh1E5Dv7C64pdl+oiEwTkX0ickpE1orIkGLz3CQiW+37t4jIbzz7CnyLBro6r7KOaLyZB19DYyDVGHO0vA/0h/dZqWLygLnAxBLuCwIOAJcBMcAfgbki0gRAROoD7wIPA9HAY8D7IlK70qv2URrofkxEhovIOhE5ISK/iEgnt/ueEJFdbt98r3e7b7yI/CwiL4jIMWBK4SpREXleRI6LyB73b9MislBE7nR7/LnmbSoii+3n/lFEXjvXaEZERtivI82uebA9/YwRmPuoyG316kQR2Q8sEJFvReSBYsteLyI32NfbiMgPInJMRLaLyE1u8w2136dTInJQRB4toc4rgR+AeiKSLiIz7enXichm+/ewUETauj1mr4hMFpENQEbxUBeRxfbV9fYyR7vd94iIHBWRJBGZ4DY91H7v94vIERF5XUTCS3lv3X/XJ0Rkt4hcYk8/YC9/nNv8MSIyW0SS7ZHVUyISYN8XaD9viojsBoYVe64Ye0SWZL+HfxORwJLqKva4WSLyiH29vv17vc++3cL+fYm4rboWkXeARsBX9vv2uNsib7PfmxQR+cN5nr6W/Zk4JSKLRKSxW10v2e9RmoisEZH+9vTBwO+B0fZzr7enx4rIDBE5ZP9dfF7sdZb4+zzH+1LiZ7LY+1BYQ+ElR0QW2veV+XNyoYwx240x04DNJdyXYYyZYozZa4wpMMZ8DewButuzNABOGGPmG8s3QAbQvCJr9CvGGL348AXYC1xZwvRuwFGgNxAIjLPnDbXvHwXUw/pSNxrrDyXevm884AImYX2LDren5QF32cv7LXAIEPsxC4E73R5/rnmXAc8DIUA/IA14t5TX1ws4CVxl11ofaFPSawemFC4HaAIYYDYQYb+GscDPbvO3A04AofY8B4AJ9mvuBqQA7e15k4D+9vUaQLdS6h0IJLrdbmW/t1cBwcDjQAIQ4vYa1gENgfBSlmmAFsWewwU8bS9zKJAJ1LDvfxH4EogFooCvgGdKWXbh73qC/bv6G7AfeM1+X64GTgGR9vyzgS/s5TYBdgAT7fvuBbbZryUW+MmuPci+/3PgDfu9rg2sBO5xq2NpKTXeAXxlX78V2AXMcbvvi1Le++Kfj8LPxFv256EzkAO0LeV5Z9qvfYD9XrzkXiMwBqhpf14eAQ4DYcU/i27zfwPMsT8/wcBlZfl9nuNvv8TPZPH3wW3+aGCr23tens9JP6y/ldIu/c5TawvAnGeeOkA2p/++A4FFwHX29d8AiUBEZfwv9YeL4wXo5SJ/gaUH+n+Bvxabtr3wn0gJ868DRtjXxwP7i90/Hkhwu13N/udY1769kDMDvcR5sUZNLqCa2/3vFv/n53bfG8ALZXntlBzozdzuj8IK18b27anAdPv6aGBJCc/9Z/v6fuAeIPo8v48z/plir0Z0ux0AHAQGur2GO86zzJICPQs7KO1pR4E+gNivsbnbfX2BPaUsezyw0+12R/v56rhNSwW62P9Uc7D6AwrvuwdYaF9fANzrdt/V9rKCsP5Z5+D2pQW4BfjJrY7SAr05VmgEAK/bz5lo3zcLeLiU977456PwM9HAbdpK4OZSnncm8KHb7UggH2hYyvzHgc7FP4v27XiggBJC+ly/z/N8Lkr8TBZ/H9w+d18D/7Vvl+tzcrEXzhPoWF9kfgTeKDZ9IpCO9T8jExhWGfX5y0VXufuvxsAj9mrUEyJyAmvkVA9ARMbK6dXxJ4AOQC23xx8oYZlFXdvGmEz7amQpz1/avPWAY27TSnuuQg2xRmQXqmjZxphTWKOkm+1JNwPv2dcbA72LvV+3YX0JAbgRa+S0z1712reMz18P2OdWQ4FdU/2SaiyHVGOMy+12Jtb7G4f1BWqN2+v41p5emiNu17PsOotPi8T6fITg9nrs64WvpR5nvhb3+Rpj/dNOcqvrDayR+jkZY3Zh/VPvAvTHCqZDItIaa/vrovMtoxj3vQ8K37fSuH9+0oFjnP4bekSshq2T9uuJ4cy/IXcNsT73x0u5v7Tf57mU5zM5FesL7YP27Qv5nFQKe5PNO0Au8IDb9CuB57C+oIRg/a7fFpEunq7RV2gTjv86AEw1xkwtfoe9HfAt4ApgmTEmX0TWYX1rL2SKP66CJAGxIlLNLdQbnmP+A5S+zSwD659SoZK6you/jg+AP4u1bToca7Vw4fMsMsZcVdITGWNWASNEJBjrn87c89Rd6BDWqBcAERH7cQfPUePFSMEK4PbGmIPnm/kClp2HFc5b7GmNOP1akjjzPWnkdv0A1gi9VrHgKqtFwEisTRUHRWQR1iaUGlhrl0pSEe9r0esRkUis1dOH7O3lk7H+hjYbYwpE5Din/4aKP/cBrM99dWPMiQqoq8yfSRG5GWttSE9jTJ49uVyfE/v1zj/HLEOMMUvK+RIK/x6mYa3BGepWH1hf4BYbY1bbt1eJyArgSkr/nVdpOkL3D8EiEuZ2CcIK7HtFpLfdMBQhIsNEJAprG6YBkgHsBpwOnijUGLMPWI3VaBdijyquPcdDpgETROQKEQmwm6La2PetA24WkWAR6YH1D/985mEF0tNY22EL7OlfA61E5HZ7ecEi0lNE2tp13iYiMfY/nDSsVa9lMRcYZtcfjLWtNQf4pYyPB2sE3awsM9qv5y3gBbG7ge337JpyPF9py87Hej1TRSTK/mL4MNYmE+z7HhSRBiJSA3jC7bFJwPfAv0Qk2v5dNheRy8r49IuwQquwSXAhVo/HUruukpT5fTuHoSLST0RCgL8CK4wxB7BGuy6sv6EgEfkT1jZq9+duYo8+C1//fOA/IlLD/nwNuNCiyvqZFJGuwCvAb4wxyYXTy/s5MdaucJHnuJQY5vb/njCsETb2/6dQt1n+C7QFrjXGZBV7+Cqgf+GI3H4t/YEN53+HqiYNdP8wD+vbduFliv2t9i7gVaxtewlY2ykxxmwB/oXVnHYEawT5swfrvQ1re10qVhPWHKyQO4sxZiVWw9YLWM1xi7ACGazt082xXt9fgPfP98TGmBzgU6xv+e+7TT+Ftc33ZqxR9WHgH1jNUAC3A3tFJA2r+WtMWV6oMWa7Pe8rWKOia7H+eeWW5fG2KcAse9XoTeebGWvkmAAst+v9Eaio/ZInYa0Z2Q0sxXoPp9v3vQV8B6wHfsV6n92NxfrHvgXrd/Yx1rblsliEFaKFgb4Ua+3M4lIfAc8AT9nv21l7JZTR+8CfsVa1d8f67IL1OudjNQXuw2rmct/c8JH9M1VEfrWv3461hmMb1jby/7vAmgqV5TM5AmstxlI53eleONKuzM9JocZY/5MKu9yzsHp5CtcU3oM1Ej/sVt9tAMaYRVif/Y9F5BTwCfB3Y8z3FVyj3yjsOlbKMSIyB9hmjPmz07UopZSv0hG68jh7VXZze7XrYKxRxOfne5xSSqnSaaArJ9TF2gaaDrwM/NYYs9bRipTyQmIdkCi9hMtt53+0qmp0lbtSSinlB3SErpRSSvkBDXSllFLKD/j0gWVq1aplmjRp4nQZSimllEesWbMmxRhT4hH9fDrQmzRpwurVq88/o1JKKeUHRGRfaffpKnellFLKD2igK6WUUn5AA10ppZTyAz69DV0ppdS55eXlkZiYSHZ2ttOlqHIICwujQYMGBAcHl/kxGuhKKeXHEhMTiYqKokmTJlhnK1XezhhDamoqiYmJNG3atMyP01XuSinlx7Kzs6lZs6aGuQ8REWrWrFnutSoa6Eop5ec0zH3PhfzONNCVUkpVmtTUVLp06UKXLl2oW7cu9evXL7qdm5tbpmVMmDCB7du3n3Oe1157jffee68iSqZfv36sW7euQpblSboNXSmlVKWpWbNmUThOmTKFyMhIHn300TPmMcZgjCEgoOQx5owZM877PPfff//FF+vjdIRuO3DwIGvn/I19KenoGeiUUqpyJSQk0KFDB+699166detGUlISd999Nz169KB9+/Y8/fTTRfMWjphdLhfVq1fniSeeoHPnzvTt25ejR48C8NRTT/Hiiy8Wzf/EE0/Qq1cvWrduzS+//AJARkYGN954I507d+aWW26hR48e5x2Jv/vuu3Ts2JEOHTrw+9//HgCXy8Xtt99eNP3ll18G4IUXXqBdu3Z07tyZMWPGVPh7dj46QrcdXDybPtv/yR83JPNj5HX0aVaTPs1i6dOsJo1iq+k2KKWUqmBbtmxhxowZvP766wA8++yzxMbG4nK5uPzyyxk5ciTt2rU74zEnT57ksssu49lnn+Xhhx9m+vTpPPHEE2ct2xjDypUr+fLLL3n66af59ttveeWVV6hbty6ffPIJ69evp1u3buesLzExkaeeeorVq1cTExPDlVdeyddff01cXBwpKSls3LgRgBMnTgDw3HPPsW/fPkJCQoqmeZIGuq336MfJmLWaKfvfJap2L+buLOCztQcBiI8J04BXSvm8v3y1mS2H0ip0me3qRfPna9tf0GObN29Oz549i25/8MEHTJs2DZfLxaFDh9iyZctZgR4eHs6QIUMA6N69O0uWLClx2TfccEPRPHv37gVg6dKlTJ48GYDOnTvTvv25616xYgWDBg2iVq1aANx6660sXryYyZMns337dh566CGGDh3K1VdfDUD79u0ZM2YMI0aM4De/+U05342Lp4Fuk4BAIka/BW9exuNpz/DY7xaxKyOEZbuPsXx3Kkt2JhcFfL2igLcuDWPDNeCVUqqcIiIiiq7v3LmTl156iZUrV1K9enXGjBlT4m5bISEhRdcDAwNxuVwlLjs0NPSsecq7ObW0+WvWrMmGDRuYP38+L7/8Mp988glvvvkm3333HYsWLeKLL77gb3/7G5s2bSIwMLBcz3kxNNDdVYuFUbNg+jXIZ3fT4taPaFE7itv7NMYYw67k9KKAX7wzmU814JVSPuRCR9KekJaWRlRUFNHR0SQlJfHdd98xePDgCn2Ofv36MXfuXPr378/GjRvZsmXLOefv06cPjz32GKmpqcTExPDhhx/y6KOPkpycTFhYGKNGjaJp06bce++95Ofnk5iYyKBBg+jXrx/vvfcemZmZREVFVehrOBePBbqIDAZeAgKBt40xzxa7/wXgcvtmNaC2Maa6p+orUr8bDH4WvnkYlvwLLnussD5a1I7SgFdKqUrQrVs32rVrR4cOHWjWrBmXXnpphT/HpEmTGDt2LJ06daJbt2506NCBmJiYUudv0KABTz/9NAMHDsQYw7XXXsuwYcP49ddfmThxIsYYRIR//OMfuFwubr31Vk6dOkVBQQGTJ0/2aJgDiCc6ukUkENgBXAUkAquAW4wxJX49EpFJQFdjzB3nWm6PHj1MpZwP3Rj47B7YMBdu/wyaX16Gh5wZ8Ct2p5KSbu1jqQGvlHLK1q1badu2rdNleAWXy4XL5SIsLIydO3dy9dVXs3PnToKCvHNldUm/OxFZY4zpUdL8nnoVvYAEY8xuu6APgRFAaes7bgH+7KHaziYCw1+ApA3wyZ1w7xKIrneeh+gIXimlvFl6ejpXXHEFLpcLYwxvvPGG14b5hfDUK6kPHHC7nQj0LmlGEWkMNAUWeKCu0oVEwE2z4c2B8NF4GP8NBJb9rDclBXzC0XSW705l+e5jLNqhAa+UUp5UvXp11qxZ43QZlcZTgV5SOpW2rv9m4GNjTH6JCxK5G7gboFGjRhVTXWniWsF1L8MnE+HHKXDN1AtelIjQsk4ULetEcXvfJhrwSimlKpSnAj0RaOh2uwFwqJR5bwZKPYafMeZN4E2wtqFXVIGl6jgSDqyAZa9Cw97Q7roKWeyFBvxV7epQIyLkPEtXSilV1Xgq0FcBLUWkKXAQK7RvLT6TiLQGagDLPFRX2Vw9FQ7+Cl/cD3XaQ83mFf4UZQ34xgur8dl9lxKroa6UUsqNR47lboxxAQ8A3wFbgbnGmM0i8rSIuA95bwE+NN52MPWgEBg1EwICYe5YyM2s9KcsDPjb+zbhtdu6sfqpK3l3Ym8On8zmrtmryc4rcYuEUkqpKspjJ2cxxswzxrQyxjQ3xky1p/3JGPOl2zxTjDFnH5TXG1RvCDe8DUc2w7zHPP70IkK/lrX4901dWLPvOI9+tJ6CAu/63qOUUiUJDAwsOmVqly5dePbZZ8//oBIMHDiQC91VeeHChUUnaQF4/fXXmT179gUty93evXvp0KHDRS+nIvhPv74ntLwSBjwGi5+DRn2g2+0eL2FYp3gOHG/Ds/O30TC2GpMHt/F4DUopVR7h4eGOn1984cKFREZGcskllwBw7733OlpPZdDTp5bXwCeg2UCY96i1n7oD7hnQjFt6NeK/C3fxwcr9jtSglFIXY/78+dx0001FtxcuXMi1114LwG9/+9ui06j++c8lH5IkMjKy6PrHH3/M+PHjAfjqq6/o3bs3Xbt25corr+TIkSPs3buX119/nRdeeIEuXbqwZMkSpkyZwvPPPw/AunXr6NOnD506deL666/n+PHjgLVGYPLkyfTq1YtWrVqVeiKYQtnZ2UyYMIGOHTvStWtXfvrpJwA2b95Mr1696NKlC506dWLnzp1kZGQwbNgwOnfuTIcOHZgzZ86FvZFuNNDLKyAQbpwG4bHW9vQsz58iT0T464j2XNYqjqc+38TiHcker0EppcoqKyvrjFXuc+bM4aqrrmL58uVkZGQAMGfOHEaPHg3A1KlTWb16NRs2bGDRokVs2FD2wVO/fv1Yvnw5a9eu5eabb+a5556jSZMm3Hvvvfzud79j3bp19O/f/4zHjB07ln/84x9s2LCBjh078pe//KXoPpfLxcqVK3nxxRfPmF6S1157DYCNGzfywQcfMG7cOLKzs3n99dd56KGHWLduHatXr6ZBgwZ8++231KtXj/Xr17Np06YKOW69rnK/EBG1YNQMmDnM6nwf/a51dDkPCgoM4NVbuzLq9WXc996vfPzbvrSpG+3RGpRSPmb+E3B4Y8Uus25HGHLubeKlrXIfPHgwX331FSNHjuSbb77hueeeA2Du3Lm8+eabuFwukpKS2LJlC506dSpTOYmJiYwePZqkpCRyc3Np2rTpOec/efIkJ06c4LLLLgNg3LhxjBo1quj+kk7DWpqlS5cyadIkANq0aUPjxo3ZsWMHffv2ZerUqSQmJnLDDTfQsmVLOnbsyKOPPsrkyZMZPnz4WV8yLoSO0C9Uoz5w1dOw7WtrH3UHRIUFM2NCTyJCA7ljxiqOpJ19qkGllPJWo0ePZu7cuSxYsICePXsSFRXFnj17eP755/nf//7Hhg0bGDZsWImnUXU/2Jb7/ZMmTeKBBx5g48aNvPHGGyU+tjxKOg1raUrbQevWW2/lyy+/JDw8nGuuuYYFCxbQqlUr1qxZQ8eOHXnyySd5+umnL6pO0BH6xelzH+xfDj/8Ger3gMZ9PV5CfEw408b15KY3lnHHzFXMvacvEaH6a1VKleA8I2lPGzhwIBMnTuStt94qWt2elpZGREQEMTExHDlyhPnz5zNw4MCzHlunTh22bt1K69at+eyzz4rObHby5Enq168PwKxZs4rmj4qKIi0t7azlxMTEUKNGDZYsWUL//v155513ikbr5TVgwADee+89Bg0axI4dO9i/fz+tW7dm9+7dNGvWjAcffJDdu3ezYcMG2rRpQ2xsLGPGjCEyMpKZM2de0HO60xH6xRCBEa9BjSbW8d7TjzpSRof6Mbx2aze2JqXx4Adrydfd2ZRSXqT4NvQnnrD2Tg4MDGT48OHMnz+f4cOHA9C5c2e6du1K+/btueOOO0o9jeqzzz7L8OHDGTRoEPHx8UXTp0yZwqhRo+jfvz+1atUqmn7ttdfy2WefFTXFuZs1axaPPfYYnTp1Yt26dfzpT3+6oNd53333kZ+fT8eOHRk9ejQzZ84kNDSUOXPm0KFDB7p06cK2bdsYO3YsGzduLGqUmzp1Kk899dQFPac7j5w+tbJU2ulTy+vwJnj7CmjQE8Z+YTXOOeCdZXv54xebGde3MVOua6/Hf1dK6elTfVh5T5+qI/SKULcDDPs37F0CP/3dsTJu79uEu/o3ZdayfUz/ea9jdSillPI83dhaUbreBvuXwZLnoWEvaHWNI2U8OaQtB45l8bdvttCgRjjXtK/rSB1KKaU8S0foFWnoP61dOD69G044c8CXgADhhdFd6NSgOg99uJZ1Bzy/n7xSSinP00CvSMHhcNNsMMY66Iwrx5EywkMCeXtsD2pFhnLnrFUcOFb5J5NRSnkvX+6Vqqou5HemgV7RYpvBb/4Dh9bCd793rIy4qFBmTuhJrquACTNXcTIrz7FalFLOCQsLIzU1VUPdhxhjSE1NJSwsrFyP0y73yvL9U/DLK9YZ2jqNOv/8leSXXSmMm76Snk1imTmhFyFB+h1OqaokLy+PxMTEiz7AivKssLAwGjRoQHBw8BnTz9XlroFeWfLzYNZ1kLQe7loAtZ07K9onaxJ55KP1jOzegH+O7KS7symllI/S3dacEBgMI6dDSDVre3pOumOl3Ni9AQ9d0ZKP1yTy6oIEx+pQSilVeTTQK1N0vBXqqTvhqwetZjmH/N+VLbmha33+9cMOPl970LE6lFJKVQ4N9MrWdABc/gfY9AmsetuxMkSEZ27sSO+msTz+8QZW7E51rBallFIVTwPdE/o9DC2vgW+fhMQ1jpURGhTIm7f3oEFsOHe/s4Zdyc5tBlBKKVWxNNA9ISAArn8douLho3GQecyxUmKqBTNzfC+CAoQJM1aRmu7MvvJKKaUqlga6p1SLhZtmQfoR60hyBQWOldKoZjXeHteDI2nZ3DV7Ndl5+Y7VopRSqmJooHtS/W4w+BlI+AGW/svRUro2qsGLo7uw9sAJHp67jgI95apSSvk0DXRP6zEROo6yzsq2e6GjpQzpGM+TQ9owb+Nh/vHdNkdrUUopdXE00D1NBIa/CLVawSd3QtohR8u5q38zbuvdiDcW7eb9Fc6cUEYppdTF00B3QmikdRKX3Ez4aIJ1VDmHiAh/ua49A1vH8ccvNrFw+1HHalFKKXXhNNCdEtcarnsZDiyHH6c4WkpQYACv3tqN1nWieOD9tWw5lOZoPUoppcpPA91JHUdCz7tg2auw5UtHS4kMDWL6+J5EhgZxx8xVHD6pJ3JQSilfooHutGumQv3u8MX9kLrL0VLqxoQxfXxPTmXnccfMVaTnuBytRymlVNlpoDstKBRGzYSAQJg7DvKyHC2nXb1oXr2tG9uPnGLS+7/iynduf3mllFJlp4HuDao3ghvegiMbYd6jTlfD5a1r85fr2vPT9mT+8tUWfPkUu0opVVVooHuLllfBgMdg7bvw6ztOV8OYPo25Z0Az3lm+j2lL9zhdjlJKqfPQQPcmA5+EppdZo/TDG52uhsmD2zC0Y12mztvKt5uSnC5HKaXUOWige5OAQLhxGoTXgLljIfuks+UECP++qQtdGlbnoQ/XsXb/cUfrUUopVToNdG8TGWc1yR3fB5/fBw5vvw4LDuStsT2oHR3KnbNWc+BYpqP1KKWUKpkGujdq1Aeuehq2fQ3LXnO6GmpFhjJjfC9cBYbxM1ZyMtO5I9sppZQqmQa6t+p7P7S9Fn74E+xb5nQ1tKgdyRu3d2f/sUzufXcNuS7dnU0ppbyJBrq3EoERr0GNxvDxBEhPdroi+jSryXMjO7FsdypPfLpBd2dTSikvooHuzcJirJO4ZB2HTyZCQb7TFXF91wb87spWfPrrQV76306ny1FKKWXTQPd2dTvCsH/BnkWw8BmnqwHgwStacGO3Brz4404+/TXR6XKUUkqhge4buo6xLov/CTt/cLoaRIRnbuhI32Y1mfzJBpbtSnW6JKWUqvI00H3F0Ochrg189wfHd2UDCAkK4PUx3WlcM4J73llNwtF0p0tSSqkqTQPdVwSHQ98HIGU77F/udDUAxFQLZsb4noQEBTBh5kpS0nOcLkkppaosDXRf0uEGCI2GNTOcrqRIw9hqvD2uJ8mncrhz1mqy85xv3FNKqapIA92XhERAp5tg8+eQeczpaop0aVidF0d3ZX3iCX43Zx0FBc5vElBKqapGA93XdB8P+TmwYY7TlZxhcIe6/GFoW+ZvOswHq/Y7XY5SSlU5Gui+pm5HqN8DVs/wiuY4dxP7NaVH4xq88MNO0nNcTpejlFJViga6L+o+3qua4wqJCH8Y1paU9BzeXLzb6XKUUqpK0UD3RUXNcTOdruQsXRvVYFineN5avJsjadlOl6OUUlWGBrovKmqO+8yrmuMKTb6mDa6CAv79/Q6nS1FKqSpDA91XeWlzHECjmtUY27cJH605wLbDaU6Xo5RSVYLHAl1EBovIdhFJEJEnSpnnJhHZIiKbReR9T9Xmk+p2hPrdvbI5DmDSoBZEhgbxzLxtTpeilFJVgkcCXUQCgdeAIUA74BYRaVdsnpbAk8Clxpj2wP95ojaf1n2CVzbHAVSvFsIDg1qwaEcyS3Y6f+pXpZTyd54aofcCEowxu40xucCHwIhi89wFvGaMOQ5gjDnqodp8lxc3xwGM7duEBjXC+fu8beTrwWaUUqpSeSrQ6wMH3G4n2tPctQJaicjPIrJcRJtKh8IAACAASURBVAZ7qDbf5eXNcWHBgTx2TWu2JqXx2dqDTpejlFJ+zVOBLiVMKz5kCwJaAgOBW4C3RaT6WQsSuVtEVovI6uRkXZXrzc1xANd2qkfnBjH86/vtZOXqcd6VUqqyeCrQE4GGbrcbAIdKmOcLY0yeMWYPsB0r4M9gjHnTGNPDGNMjLi6u0gr2GYXNcWtmemVzXECA8PuhbUk6mc30n/c4XY5SSvktTwX6KqCliDQVkRDgZuDLYvN8DlwOICK1sFbB6+HGyqL7BEje5pXNcQC9m9XkyrZ1+O/CXXqKVaWUqiQeCXRjjAt4APgO2ArMNcZsFpGnReQ6e7bvgFQR2QL8BDxmjEn1RH0+r8MNEBLltc1xAE8MaUNWXj4v/bjT6VKUUsoveWw/dGPMPGNMK2NMc2PMVHvan4wxX9rXjTHmYWNMO2NMR2PMh56qzed5eXMcQIvakdzSqyHvr9zPruR0p8tRSim/o0eK8xc9Jnh1cxzA/13ZivDgQJ6drwebUUqpiqaB7i+8vDkOoFZkKPde1owfthxh5R7vXJOglFK+SgPdn3Qf79XNcQAT+zWjbnQYU7/ZQoEebEYppSqMBro/6XCj1zfHhYcE8sjVrVifeJKvNyY5XY5SSvkNDXR/4gPNcQA3dGtAm7pRPPftNnJcerAZpZSqCBro/sYHmuMCA4Q/DGtL4vEsZv+yz+lylFLKL2ig+xsfaI4D6N8yjgGt4nhlwU5OZOY6XY5SSvk8DXR/5APNcQBPDmnDqRwXry5IcLoUpZTyeRro/sgHmuMA2sZHM6p7A2Yt28v+1Eyny1FKKZ+mge6PfKQ5DuDhq1oTGCA8950ebEYppS6GBrq/8vLTqhaqGxPG3f2b8fWGJNbuP+50OUop5bM00P1VfCefaI4DuPuy5tSKDOHv87ZivLxWpZTyVhro/qywOe7ACqcrOafI0CD+78pWrNp7nO+3HHG6HKWU8kka6P6ssDlu9QynKzmvm3s2pHlcBM/O30ZefoHT5SillM/RQPdnPtQcFxQYwJND2rInJYMPVu53uhyllPI5Guj+zkea4wCuaFub3k1jefHHnaRl5zldjlJK+RQNdH/nQ81xItYhYY9l5PL6wl1Ol6OUUj5FA70q8JHmOIBODaozoks9pi3dw6ETWU6Xo5RSPkMDvSpof4PPNMcBPHp1awzw/PfbnS5FKaV8hgZ6VRAa6TPNcQANY6sx4ZImfLb2IJsOnnS6HKWU8gka6FVFUXPcXKcrKZP7Lm9BTHgwz8zXg80opVRZaKBXFUXNcTO8vjkOICY8mAcHteTnhFQW7kh2uhyllPJ6GuhViQ81xwGM6dOYxjWr8cy8rbj0YDNKKXVOGuhViY81x4UEBTB5cBt2HEnn4zWJTpejlFJeTQO9KvGx5jiAIR3q0q1Rdf79ww4yclxOl6OUUl5LA72q8bHmuMKDzRw9lcNbS3Y7XY5SSnktDfSqJr4T1OvmM81xAN0bxzKkQ13eXLybo6eynS5HKaW8kgZ6VdRjgk81xwE8PrgNua4CXvhhp9OlKKWUV9JAr4oKm+PWzHS6kjJrWiuCMX0aM2fVfnYcOeV0OUop5XU00Kui0EjoNMpqjss67nQ1ZfbgFS2JCAni2fnbnC5FKaW8jgZ6VdV9AriyYb33n1a1UGxECPdd3oIF247yS0KK0+UopZRX0UCvqnywOQ5gwqVNqF89nKnztlJQ4Dt1K6VUZdNAr8p8sDkuLDiQR69pxeZDaXyx/qDT5SillNfQQK/KfLA5DmBE5/p0qB/NP7/dTnZevtPlKKWUV9BAr8p8tDkuIED4/dC2HDqZzYyf9zpdjlJKeQUN9KrOB5vjAC5pXotBbWrzn58SSE3PcbocpZRynAZ6VVfUHDfTp5rjAJ4c0oaMXBevLEhwuhSllHKcBrqyT6u61aea4wBa1olidM9GvLt8H3tSMpwuRymlHKWBrqDDjT7ZHAfwu6taEhIUwD/0YDNKqSpOA135bHMcQO2oMO4Z0JxvNx9m9V7fOCWsUkpVBg10ZfHR5jiAuwY0pXZUKFPnbcX4WB+AUkpVFA10ZfHh5rhqIUE8cnUr1u4/wbyNh50uRymlHFHmQBeRh0Wki329j4jsF5HdItK38spTHuWjzXEAI7s3pHWdKP7x7TZyXQVOl6OUUh5XnhH674A99vVngH8DU4EXK7oo5RAfbo4LDBCeGNqG/ccyeWf5PqfLUUopjytPoMcYY06KSBTQGXjFGDMNaF05pSmP8+HmOICBreLo16IWryzYycnMPKfLUUopjypPoB8QkUuAm4HFxph8EYkG9GDa/qT7eJ9tjhMRnhzahpNZeby2UA82o5SqWsoT6I8BHwN/AP5qTxsOrKzoopSD4jv7bHMcQPt6MdzQtQEzf97LgWOZTpejlFIeU+ZAN8bMM8bUM8Y0McassSd/BFxXOaUpx/hwcxzAo9e0QgSe/36706UopZTHlKfLvZ2I1LGvR4rIX4AngeDKKk45xIeb4wDiY8KZ2K8pX6w7xIbEE06Xo5RSHlGeVe7vA9Xt688DA4C+wBsVXZRymI83xwH8dmBzakaEMPUbPdiMUqpqKE+gNzHGbBcRAa4HRgEjgWsqpTLlLB9ujgOICgvmoStbsmLPMX7cetTpcpRSqtKVJ9Bz7F3WegEHjDEpQA4QVpYHi8hgEdkuIgki8kQJ948XkWQRWWdf7ixHbaqi+XhzHMAtvRrRrFYEz87fiitfDzajlPJv5V3lvgCYBcy0p3Xj9MFmSiUigcBrwBCgHXCLiLQrYdY5xpgu9uXtctSmKkNRc5xv7sgQHBjA5CFt2JWcwYerDjhdjlJKVarydLn/DmuXtd8aY161JxdgHUHufHoBCcaY3caYXOBDYER5i1UeVtQcN8PpSi7Y1e3q0LNJDV78cQfpOS6ny1FKqUpTrpOzGGO+B3aJSF8RaWSMWW2MWVCGh9YH3IdIifa04m4UkQ0i8rGINCxpQSJyt4isFpHVycnJ5SlflZcfNMeJCL8f2paU9FzeWLTL6XKUUqrSlGe3tXgRWQTsBD4FEkRkkYjUK8vDS5hWfMPsV1iNd52AH7FW7Z/9IGPeNMb0MMb0iIuLK2v56kL5eHMcQNdGNRjeKZ63luzm8Mlsp8tRSqlKUZ4R+n+B9UCsMSYeqAGsA14vw2MTAfcRdwPgkPsMxphUY0yOffMtoHs5alOVxQ+a4wAmD25DQQH8Sw82o5TyU+UJ9H7AI8aYDAD75+PAJWV47CqgpYg0FZEQrOPBf+k+g4jEu928DthajtpUZfLx5jiAhrHVGNu3MR//msjWpDSny1FKqQpXnkA/jtWh7q41cN5DcRljXMADwHdYQT3XGLNZRJ4WkcJDxz4oIptFZD3wIDC+HLWpyuQHzXEADwxqQVRoEH/+crOeM10p5XekrEfREpG7gL8D04B9QGNgAvBHY8yblVbhOfTo0cOsXr3aiaeuer7+Hax7Hx7ZBuE1nK7mgn20+gCPfbyBYZ3iefnmrgQGlNTeoZRS3klE1hhjepR0X3l2W3sLGA3UAq61f96OtT1c+bvC5rgNc52u5KKM6tGQJ4e04ZsNSTz56QYKCny3L0AppdwFlWdmexe1ot3URCQUmA/8qYLrUt4mvjPU6wqrZ0Cvu0F8d2R7z2XNSc9x8cqCBCJCg/jT8HaID78epZSCcu6HXgr9T1hVdJ/g881xhR6+qhUTLm3CjJ/38sIPO5wuRymlLlpFBLqus6wqOtwIIZE+3xwH1gFn/jisHTf1aMDLCxL0oDNKKZ933lXuIjLoHHeHVGAtytuFRkLHUbD+Axj8jE83xwEEBAjP3NCJjNx8npm/jciwIG7r3djpspRS6oKUZRv6tPPcv78iClE+oscEa4S+YS70vsfpai5aYIDwwk1dyMrN56nPN1EtJJDru2qfp1LK95x3lbsxpun5Lp4oVHkJ9+Y4Hz5ynLuQoAD+c1s3ejeN5dGPNvDd5sNOl6SUUuVWEdvQVVXjR81xhcKCA3l7XE861o9h0vtrWbJTT/yjlPItGuiq/Iqa42Y6XUmFigwNYuaEnjSLi+Du2WtYvfeY0yUppVSZaaCr8itsjtv8qc+eVrU01auF8M7E3tSNCWPCjFVsOnjS6ZKUUqpMNNDVhekxwS+OHFeSuKhQ3r2zN9Hhwdw+bQU7j5xyuiSllDovDXR1YfywOc5d/erhvHdnb4ICAxgzbQX7UzOdLkkppc5JA11dOD84req5NKkVwbsTe5PjKuC2acs5fDLb6ZKUUqpUGujqwnUY6ZfNce5a141i1oReHEvP5ba3l5OanuN0SUopVSINdHXh/Lg5zl3nhtWZNr4nicezGDt9JSez8pwuSSmlzqKBri6OHzfHuevTrCZv3N6dHUdOccfMVWTmupwuSSmlzqCBri5OYXPcmpl+2RznbmDr2rx0c1fW7j/O3bPXkJ2X73RJSilVRANdXbzu4+HoFr9tjnM3tGM8z43szNKEFCZ9sJa8/AKnS1JKKUADXVWEKtAc525k9wb85br2/LDlCI9+tJ6CAv9eM6GU8g0a6OriVZHmOHfjLmnCY9e05ot1h3jqi00YP9/coJTyfhroqmJ0H18lmuPc3X95C347sDnvr9jP3+dt1VBXSjlKA11VjHpdqkxznLvHr2nN2L6NeWvJHl5ZkOB0OUqpKkwDXVWcwua4TZ84XYnHiAhTrm3PDd3q8+8fdjBt6R6nS1JKVVFBTheg/Ein0bD2PfjsXgiNglbXOF2RRwQECM/d2Ims3Hz++vUWIkMDGd2zkdNlKaWqGB2hq4oTHA63fQR12sGc22HXT05X5DFBgQG8eHMXLmsVxxOfbuSr9YecLkkpVcVooKuKFV4dbv8caraAD26BvT87XZHHhAYF8vqY7vRsHMvv5qzjf1uPOF2SUqoK0UBXFa9aLIz9Aqo3hPdvggOrnK7IY8JDApk2vgft6kXz2/d+5ZddKU6XpJSqIjTQVeWIjIOxX0JEHLx7Ixxa53RFHhMVFsysCb1oUrMad85aza/7q8a++UopZ2mgq8oTHQ/jvoSwaHjnejiy2emKPKZGRAjvTuxNXFQo46evZMuhNKdLUkr5OQ10VbmqN7JCPSgUZo+A5B1OV+QxtaPDeHdibyJCgxg7fQW7ktOdLkkp5cc00FXli21mrX4HmH0dHNvtbD0e1DC2Gu/e2RuAMW+v4MCxTIcrUkr5Kw105RlxraxGOVc2zLoOThxwuiKPaR4Xyew7epOR42LMtBUcTct2uiSllB/SQFeeU6c93P4ZZKfBrGshLcnpijymXb1oZt7Ri+RTOYyZtoLjGblOl6SU8jMa6Mqz6nWFMR9DRrK1+j092emKPKZboxq8PbYHe1MzGTdjJaey85wuSSnlRzTQlec17AW3zrVWu7/zG8g85nRFHnNJi1r897ZubDmUxsSZq8nKzXe6JKWUn9BAV85ocinc8j6k7LR2acs+6XRFHnNF2zq8MLoLq/Yd455315Dj0lBXSl08DXTlnOaD4KbZcGQTvDsSck45XZHHXNu5Hs/e0JHFO5L5vw/X4covcLokpZSP00BXzmo9GEZOh4Nr4P2bIbfq7NY1umcj/ji8HfM3HebxTzZQUFB1ziOvlKp4GujKee1GwPVvwL6fYc5tkFd1duua2K8pD1/Vik9/PciUrzZjjIa6UurC6PnQlXfoNMraR/3LB+Cj8daq+KAQp6vyiEmDWpCe4+LNxbuJDA3i8cFtnC5JKeWDNNCV9+h2uxXq8x6FT++EG6dDoP9/REWEJ4e0ISPHxX8W7iIiNIj7L2/hdFlKKR/j//8tlW/pdRe4cuD7P0DQffCb/0JAoNNVVToR4a8jOpCR4+Kf321nT0oGkwa1oHHNCKdLU0r5CA105X0ueQBcWbDgb9ZJXYa/BAH+3+4RECD8c1Rn4qJCmb1sH5+tPciILvV44PIWNIuLdLo8pZSX00BX3mnAY1Zz3JLnISgMhjwHIk5XVemCAwP4w7B23NW/GW8s3s17K/bx+dqDXNe5Hg8MakGL2lFOl6iU8lLiy121PXr0MKtXr3a6DFVZjIHvn4Jlr8Ilk+Cqv1aJUHeXfCqHt5fsZvayfWS78hneqR6TBrWgVR0NdqWqIhFZY4zpUeJ9GujKqxljNcmtehsGPA6D/uB0RY5ITc/h7aV7mP3LXjJy8xnasS6TBrWkbXy006UppTxIA135toIC+GoSrH0XrvgT9H/E6Yocczwjl2lL9zDzl72k57i4pn0dHryiJe3rxThdmlLKAzTQle8ryIfP7oGNH8E1z0Df+5yuyFEnM/OY/vMepv+8h1PZLq5sW4cHr2hBpwbVnS5NKVWJNNCVf8h3wcfjYetXMOzf0HOi0xU57mRWHrN+2cu0pXs4mZXH5a3jeOjKVnRpqMGulD/SQFf+w5ULc8bAzu9gxH+g621OV+QVTmXnMXvZPt5aspsTmXkMaBXHQ1e0pHvjGk6XppSqQOcKdI/t3Csig0Vku4gkiMgT55hvpIgYESmxYFXFBYVYh4Vtdrl1mNiNHztdkVeICgvm/stbsHTyICYPbsOmgye58b+/MObtFazcU3XON69UVeaREbqIBAI7gKuARGAVcIsxZkux+aKAb4AQ4AFjzDmH3zpCr8JyM+G9kbB/Odw0C9pe63RFXiUz18V7y/fzxuJdpKTn0qdZLA9d0Yq+zWs6XZpS6iJ4wwi9F5BgjNltjMkFPgRGlDDfX4HngKpzui11YUKqwa1zoH43+GgC7Pje6Yq8SrWQIO4a0Iwljw/ij8PbsSs5g1veWs5Nbyzj54QUPaubUn7IU4FeHzjgdjvRnlZERLoCDY0xX3uoJuXrQqPgto+hTjtru/qun5yuyOuEhwQysV9Tljx+OVOubce+1Axue3sFI19fxqIdyRrsSvkRTwV6SYf3KvpPIiIBwAvAeXcwFpG7RWS1iKxOTk6uwBKVTwqvDrd/DjWbwwe3wL5fnK7IK4UFBzL+0qYseuxy/jqiPYdOZDFu+kqu/88v/LTtqAa7Un7AU9vQ+wJTjDHX2LefBDDGPGPfjgF2Aen2Q+oCx4DrzrUdXbehqyLpR2HGUDiVBGO/gAbaU3kuOa58PllzkNd+SuDgiSw6NYjhwUEtuaJtbaSKHV5XKV/i+G5rIhKE1RR3BXAQqynuVmPM5lLmXwg8qk1xqlzSDsGMIZB1HMZ9BfGdna7I6+W6CvhsbSKv/pTAgWNZtK8XzaRBLbm6XR0CAjTYlfI2jjfFGWNcwAPAd8BWYK4xZrOIPC0i13miBlUFRNezgjw0Gmb/Bo5sOf9jqriQoABG92zEgkcG8s+RncjIcXHvu2sY+vIS5m1MoqBAV8Ur5Sv0wDLK/6Tusla/mwKYMB9qtXC6Ip/hyi/gqw2HeGVBAruTM2hVJ5JJg1oytGM8gTpiV8pxjq9yrywa6KpUydutUA8MgQnzILap0xX5lPwCw9d2sCccTadF7UgmDWrB8E71NNiVcpAGuqqaDm+CWcOt3dvGz4PqDZ2uyOcUFBjmbUrilf8lsP3IKZrViuD+y1swoks9ggI9dqBJpZRNA11VXYfWwqzrIKKWFerR8U5X5JMKCgzfbT7MS//bybbDp2gUW40JlzZhVI+GRIYGOV2eUlWGBrqq2vavgHeuh5gGMP4biIxzuiKfVVBg+GHrEd5YtItf958gKiyIm3s2ZGzfJjSMreZ0eUr5PQ10pfYssY79XrOF1QlfLdbpinze2v3Hmf7zXuZtTMIYw+AOdZnYryndGtXQfdmVqiQa6EoBJPwPPrgZIutAuxHQeig07A2Busr4Yhw6kcWsZXv5YMV+0rJddG4Qwx39mjK0YzzBup1dqQqlga5Uod2L4OeXYM9iKMiD8BrQ8hpoPQRaXGE10KkLkpnr4pM1iUz/eS97UjKoGx3G2Esac2uvRlSvFuJ0eUr5BQ10pYrLToNdC2DHt9Yl67i1i1uT/la4tx5ibXNX5VZQYFi44yjTlu7h54RUwoMDubF7fSZc2pTmcZFOl6eUT9NAV+pc8l2QuBK2z4Nt8+DYLmt63Y7WavnWQyC+C+h24XLbmpTGjJ/38Pm6Q+S6Cri8dRx39GtKvxa1dDu7UhdAA12p8kjZaYX79m/hwHLriHNR9aD1YCvgm/SH4DCnq/QpKek5vLd8P+8s30tKei6t60RxR78mjOhSn7DgQKfLU8pnaKArdaEyUmHn91bAJ/wP8jIgOAKaX26Fe6trrH3cVZnkuPL5ct0hpi3dw7bDp6gZEcJtvRsxpm9jakfplySlzkcDXamKkJcNe5fao/f5cOoQIFanfOshVsDXaqmr5svAGMOy3alMX7qH/207SnBAANd2rscd/ZrQvl6M0+Up5bU00JWqaMbA4Q1WsG+fB0nrremxzU831TXso7vElcGelAxm/ryHj9YkkpmbT59msdxxaVOuaFtHjxuvVDEa6EpVtpOJVrf89vnWLnH5uRBW3Vol33oINL8CwqKdrtKrnczMY87q/cz6ZR8HT2TRuGY1JlzShJF6eFmlimigK+VJOaesXeK2F+4SdwwCgqFpf3u7+2A9Ucw5uPIL+HbzYaYv3XPG4WXHXdKEBjX08LKqatNAV8opBflwYOXp7e6pO63pdTqeXjUf3wUC9IhqJdHDyyp1Jg10pbxFyk57u/t8t13i4q1Re+uh0HSA7hJXgrMOL9uwOndc2kQPL6uqHA10pbxR5rEzd4nLTYfgatB8ENTrCrVaWV3zsc0gKNTpar1CRo6LT3/Vw8uqqksDXSlv58qBvUuskfvO7+HE/tP3SQBUb3w64Gu2OH09Iq5K7ianh5dVVZUGulK+JucUpCZASoK13T1lx+nrruzT84XFQM2WdsC3OH09tmmVGdWXdHjZif2acWmLmrqdXfkdDXSl/EVBAaQlng74lB124CfYB7qxSQDUaGIHvH0pDPuIWn45qi9+eNm60WEMaFWLAa3i6Neilq6SV35BA12pqqBoVL/TvuywbqcmnD2qr9XKDvgWp6/HNoMg3w+9HFc+32xI4setR1i6M4W0bBcBAp0bVmdAyzgGtIqjc4MYgrSZTvkgDXSlqrKCAjh5wB7JFwv7U0mn55NAqGFvq6/Zwh7Z22Hvo6N6V34B6xNPsmhHMot3JLMh8QQFBqLDgujXslZRwNerHu50qUqViQa6Uqpk2WmnR/EpO04H/rFdxUb11d1W27eEms2tM9BFx0NkHQgMdu41lMOJzFyWJqSweEcyi3ekcDjNeo0takdyWSsr3Hs3jdUzwCmvpYGulCqfwlF9yk63pjw77NMPF5tZILK2tT99dD3rZ1S8Ffbu08JivGqUb4xh59F0Fu9IZtGOZFbsOUauq4DQoAB6NY0tCviWtSO1uU55DQ10pVTFyU6D43vg1GFIO2Stti/6mWT9zDp29uOCq5Ue9oU/I+s4th0/KzefFXtSWbwjhcU7k0k4mg5AfEwY/VvW4rJWtenXohYx1XxjbYTyTxroSinPysu2gr2ksHeflp9b7IFiba8vKezdvwiE16j00f7BE1ks2ZHM4p3J2lynvIYGulLK+xhjHS3v1KFzj/YzU85+bFCYW9jXLfkLQEQchFTMyVy0uU55Cw10pZTvcuVYge8e9u6BXzjNvYmvUFCYNZoPj4Vqsfb1GvZ192mxp6eFVz9vk19pzXUta0cyQJvrVCXSQFdK+TdjIOu4HfyHrLDPTLHWAGQdg8zj1v1Zx05PK3CVvrzQaLfgL/6FwD38a2DCa5CQHsLCvTksTkjR5jpVqTTQlVLKnTHWyXCKAv+YHfjHS5jm9iUg+2Tpy5RACK9OQVgspwKiOJxXjb2ZIezPCuO4icSE16B+fH1aNG5E+xZNiKpeC0IirEtgiFftAaC817kCPcjTxSillONEIDTKutRoXPbHFeRD1onSAz/rOAGZx4jJOkZM5nFam2MUmGMEuLLABRywL0uL1xNoBXtwNWu7f3CE/bNaKdPDS5mnhHmDwiFAG/eqAg10pZQqq4BAiKhpXcr6ELC6/rOO4UpPYdf+RHbs2c/exAMcP3GScHJoEg1tagbSNFqIDMiB3EzIy7AO55t+BPIy7WmZkJsBlHPNanC1kr8snPcLRDVr7UFQmLU7YVAYBIZaJ/4puoS5zRNqvUfKERroSilV2YLDILgeQdH1aF2vE637WJP3p2byzcYkZm9MYmOCtTq/c4MYhnWKZ0iHeBrGltClb4zVAFgY+mf8tAO/6AuA2/SSpp1KOnt6Qd7FvdaAoLNDP9At/M+aVsrtkr48BJbwRSIgyL4EWiclKroeaF8PcLtu/5QAv9zEodvQlVLKC+xPzWTepiS+2ZDExoOnw31ox3iGdiwl3CtDft7pLwV5WdaxAlzZ4LJ/Ft3OsS75Oaevu3LKNk+Jj3G7Xd41EBeiMPwl0A76QLfrJU0/z5eGMx5jf2kICISQSBjxasWVrU1xSinlO7wm3J1gjPWlIr+0LwluXxbyc6y9FQryrYuxfxa47OsFbtft+UyB2/XC6QVnzlPgsudzX9YFPkdIBNyzqMLeHg10pZTyUYXhPm9jEhsSq1i4q7NooCullB8oKdw7NYhhmIZ7laGBrpRSfubAMauhrni4D+0YzzANd7+lga6UUn7swLFM5m1M4hsNd7+nga6UUlWEhrt/00BXSqkqqDDc521MYr0d7h3rW/u5a7j7Jg10pZSq4koL98KRe6OaGu6+QANdKaVUEQ1336WBrpRSqkQHjmUy3z6ITWG4d6gfzbCO9ejTLJZakaHERYXqud29hAa6Ukqp8yop3AtFhgZRKzKEWpGh1iXK7XpkKHFutyNC9TQhlUUDXSmlVLkkHs9kW9IpUtJz7Esuyek5pJyybqdm5HIis+QTuYQHB54d+JEh1IoKdZtm3Y4KDUL88EQplUXPh66UUqpcGtSoRoMa596Wnusq4FhGLinp9Xz1RwAACvxJREFUOW5hn+v2JSCH/amZ/LrvOMcycylp/BgaFHA64M8z+o8JD9bwPwcNdKWUUhckJCiAujFh1I0JO++8rvwCjmXmknLqzMBPSc8l5ZT1heDQyWw2HDzJsYxc8gvOTv/gQKFmxOnAj4sMpUXtSNrER9O2bhRxUaFVOvA10JVSSlW6oMAAakeFUTvq/OFfUGA4npl7xmg/uYTR/+ZDaXy0JrHocbERIbSpG0WbutG0iY+ibd1oWtaJrDINfR4LdBEZDLwEBAJvG2OeLXb/vcD9QD6QDtxtjNniqfqUUkp5h4AAoWZkKDUjQ2lN1DnnPZ6Ry7bDp9h2OI1tSdbP91fuIzuvwFqWQNNaEUWj+Lbx0bSJj6ZeTJjfjeY90hQnIoHADuAqIBFYBdziHtgiEm2MSbOvXwfcZ4wZfK7lalOcUkqp4vILDPtSM6ygT0pjqx34B45lFc0TFRZEW3skXziib10nyus79L2hKa4XkGCM2W0X9CEwAigK9MIwt0UAvtt+r5RSyjGBAUKzuEiaxUUytGN80fRT2XnsOHKKrUmnR/Sf/nqQ9Jx9RfM0rlmtaLV9WzvsG8VWIyDA+0fzngr0+sABt9uJQO/iM4nI/cDDQAgwyDOlKaWUqgqiwoLp3jiW7o1ji6YZY0g8nlU0mt92+BRbD6fx/ZYjRV351UICaVUnqijg29SNok18NDHhwQ69kpJ5KtBL+mpz1gjcGPMa8JqI3Mr/t3fvMXKVdRjHvw9doBds1ra0abdbKbpCG0IBtYJoIxRIgcrFSFKMBgyCf4CAGgkSk0ZNBBNDNEgwyEUSawkWCMUgUKvWqJFLuYTWlpYi0O0dW2ihpND684/ztp3ZnS6Yzux79uzzSSZzLtM5z053z2/e9z0X+D5wSa83kq4ArgCYNGlSk2OamdlgIonOUcPpHDWcM6eO27f8nXf3sGpT0ZLf26L/w7KNzH9yf9u0o31YKu77W/RHjR5B25BDcvwo/VbQu4HOmvmJwPo+Xn8vcFujFRFxO3A7FGPozQpoZma217DDhjCts51pne37lkUEm7bvYkXNAXgrN+xgyaot7E6n2R3WdggfH3fEvpb8lPEjOeXo0f3SZd9fBf0poEvSZGAdMAf4cu0LJHVFxOo0ey6wGjMzs5KQtO+8+9OOGbtv+a7de1iz+e2iwG/cwYoN21myagsLlnYzcmgbz889q1/y9UtBj4jdkq4CHqM4be2uiFgu6YfA0xGxELhK0hnAe8A2GnS3m5mZlc3hbUOYOmEkUyeMrFv++lu7WP/GO/12epyv5W5mZjZA9HXaWp6RezMzM2sqF3QzM7MKcEE3MzOrABd0MzOzCnBBNzMzqwAXdDMzswpwQTczM6sAF3QzM7MKcEE3MzOrABd0MzOzChjQl36VtAV49X1f+MGNAV5v4vu1gjMevLLng/JnLHs+KH/GsucDZ2yGZuf7SEQc2WjFgC7ozSbp6QNdI7csnPHglT0flD9j2fNB+TOWPR84YzP0Zz53uZuZmVWAC7qZmVkFuKDXuz13gA/AGQ9e2fNB+TOWPR+UP2PZ84EzNkO/5fMYupmZWQW4hW5mZlYBLuiJpFmSXpT0kqTrc+fpSdJdkjZLWpY7SyOSOiX9WdIKScslXZM7U0+Shkp6UtLzKeMPcmdqRNIQSc9K+n3uLI1IekXSC5Kek/R07jw9SWqXtEDSyvT7eEruTLUkHZM+u72P7ZKuzZ2rJ0nfSn8nyyTNlzQ0d6Zakq5J2ZaX5fNrtJ+WNErSIkmr0/OHW7V9F3SKHShwK3A2MBW4WNLUvKl6+TUwK3eIPuwGvhMRU4CTgStL+BnuAk6PiGnACcAsSSdnztTINcCK3CHex2kRcUJJTxf6OfBoRBwLTKNkn2VEvJg+uxOATwA7gQczx6ojqQO4GvhkRBwHDAHm5E21n6TjgMuB6RT/x7MldeVNBTTeT18PLI6ILmBxmm8JF/TCdOCliHg5It4F7gXOz5ypTkT8FdiaO8eBRMSGiHgmTe+g2Il25E1VLwpvpdlD06NUB5FImgicC9yRO8tAJGkkMAO4EyAi3o2IN/Km6tNMYE1ENPMCWc3SBgyT1AYMB9ZnzlNrCvDPiNgZEbuBJcCFmTMdaD99PnBPmr4HuKBV23dBL3QAa2vmuylZMRpIJB0FnAg8kTdJb6k7+zlgM7AoIsqW8WfAdcB/cwfpQwCPS1oq6YrcYXo4GtgC3J2GLe6QNCJ3qD7MAebnDtFTRKwDfgq8BmwA3oyIx/OmqrMMmCFptKThwDlAZ+ZMBzIuIjZA0fABxrZqQy7oBTVYVqqW20Ah6QjgfuDaiNieO09PEbEndXVOBKanrrtSkDQb2BwRS3NneR+nRsRJFENUV0qakTtQjTbgJOC2iDgReJsWdnEeDEmHAecBv8udpac0zns+MBmYAIyQ9JW8qfaLiBXAT4BFwKPA8xTDfoOaC3qhm/pvdxMpV/fSgCDpUIpiPi8iHsidpy+pG/YvlOu4hFOB8yS9QjHsc7qk3+SN1FtErE/PmynGfqfnTVSnG+iu6XlZQFHgy+hs4JmI2JQ7SANnAP+OiC0R8R7wAPCZzJnqRMSdEXFSRMyg6OZenTvTAWySNB4gPW9u1YZc0AtPAV2SJqdvzXOAhZkzDSiSRDFuuSIibs6dpxFJR0pqT9PDKHZaK/Om2i8ivhcREyPiKIrfwT9FRGlaRQCSRkj60N5p4CyK7s9SiIiNwFpJx6RFM4F/ZYzUl4spYXd78hpwsqTh6W97JiU7uFDS2PQ8Cfgi5f0sFwKXpOlLgIdataG2Vr3xQBIRuyVdBTxGcTTnXRGxPHOsOpLmA58HxkjqBuZGxJ15U9U5Ffgq8EIaowa4ISIeyZipp/HAPemshkOA+yKilKeGldg44MFiH08b8NuIeDRvpF6+CcxLX85fBr6WOU8vadz3TOAbubM0EhFPSFoAPEPRlf0s5bsi2/2SRgPvAVdGxLbcgRrtp4GbgPskXUbxRemilm3fV4ozMzMb+NzlbmZmVgEu6GZmZhXggm5mZlYBLuhmZmYV4IJuZmZWAS7oZtZSkkLSx3LnMKs6F3SzQSbd/vQdSW/VPH6RO5eZHRxfWMZscPpCRPwxdwgzax630M0MAEmXSvq7pFskvSlppaSZNesnSFooaauklyRdXrNuiKQbJK2RtCPdia32/ghnSFotaZukW9PlRM2sidxCN7Nan6a4ockYiutjPyBpckRspbhW9nKKu28dCyyS9HJELAa+TXFt8nOAVcDxwM6a950NfAoYCSwFHqa4S5aZNYkv/Wo2yKS7uY2h/naT36W4JvaPgY5IOwZJTwK3UNyZ7hWgPSJ2pHU3AuMj4lJJLwLXRUSvG09ICuBzEfG3NH8fxV3GbmrJD2g2SLnL3WxwuiAi2msev0rL10X9t/xXKVrkE4Cte4t5zbqONN0JrOljextrpncCRxxcfDPryQXdzGp19BjfngSsT49Re2+dWrNuXZpeC3y0fyKaWSMu6GZWayxwtaRDJV0ETAEeiYi1wD+AGyUNlXQ8cBkwL/27O4AfSepS4fh0a0sz6yc+KM5scHpY0p6a+UXAQ8ATQBfwOrAJ+FJE/Ce95mLglxSt9W3A3IhYlNbdDBwOPE4xPr8SuLDVP4SZ7eeD4swMKE5bA74eEZ/NncXM/n/ucjczM6sAF3QzM7MKcJe7mZlZBbiFbmZmVgEu6GZmZhXggm5mZlYBLuhmZmYV4IJuZmZWAS7oZmZmFfA/FrJluwQ5sOcAAAAASUVORK5CYII="
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 11. Learning curves for the model with batch_size = 128.\n",
    "\n",
    "![assets/learning_curves_batch128.png](attachment:learning_curves_batch128.png)\n",
    "\n",
    "This time we can see the improvement, the validation error seems to constantly decrease, the curve is more smooth. As it turned out this could be the right way to one more time improve the model's predictive abilities."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 12. Learning curves for the model with batch_size = 256.\n",
    "<br>\n",
    "<div>\n",
    "<img src=\"assets/learning_curves_batch256.png\" width=\"500\" />\n",
    "</div>\n",
    "\n",
    "During the above training trial, we recorded the lowest validation error among all preceding trials that equals around 0.32."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 13. Learning curves for the model with min_word_count = 3 and max_seq_len = 0.9.\n",
    "\n",
    "As the last hyperparameters to tune we will use the min_word_count and max_seq_len.\n",
    "\n",
    "* min_word_count will be changed to the value of 3 so that the vocabulary will contain all the words which occur at least 3 times in the entire dataset.\n",
    "* max_seq_len is going to be changed to the values of 0.9, which corresponds to the 0.9 quantile of the distribution of the lengths, that will be the maximum lengths threshold.\n",
    "\n",
    "<div>\n",
    "<img src=\"assets/learning_curves_mwc3_msl09.png\" width=\"500\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The summary - final set of hyperparameters\n",
    "\n",
    "Considering all above training trials, we can draw the following conclusions:\n",
    "- increasing the dropout probability helps in reducing the model's overfitting.\n",
    "- spatial dropout works better in terms of decreasing the variance problem than the traditional dropout.\n",
    "- the most improvement in reducing overfitting is due to the reduction of hidden_size.\n",
    "- using stacked GRU doesn't improve in our case the model's performance.\n",
    "- reducing the batch_size doesn't significantly affect the model's learning ability what is rather unexpected while increasing the batch_size does improve a bit the model's performance.\n",
    "\n",
    "The following are the hyperparameters that will be used to finaly train our neural network:\n",
    "- hidden_size = 8 <br>\n",
    "- embedding_dim = 200 <br>\n",
    "- n_layers = 1 <br>\n",
    "- dropout = 0.5 <br>\n",
    "- learning_rate = 0.001 <br>\n",
    "- epochs = 20 <br>\n",
    "- spatial_dropout = True <br>\n",
    "- batch_size = 256 <br>\n",
    "- min_word_count = 3 <br>\n",
    "- max_seq_len = 0.9 <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we will use the *tensorboardX* to create the graph of our neural network model that has been depicted at the top of this notebook. You can encounter torch._C.Value issue while using *add.graph()* method, to tackle that I recommend following the *github* thread devoted to this topic:\n",
    "<br>https://github.com/lanpa/tensorboardX/issues/483<br>\n",
    "namely, you can try to build tensorboardX from source with:\n",
    "<br>*git clone https://github.com/lanpa/tensorboardX && cd tensorboardX && python setup.py install*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "graph(%self : ClassType<BiGRU>,\n",
      "      %input_seq : Long(16, 44),\n",
      "      %lengths.1 : Long(16)):\n",
      "  %1 : ClassType<Embedding> = prim::GetAttr[name=\"embedding\"](%self)\n",
      "  %weight.1 : Tensor = prim::GetAttr[name=\"weight\"](%1)\n",
      "  %5 : ClassType<GRU> = prim::GetAttr[name=\"gru\"](%self)\n",
      "  %6 : Tensor = prim::GetAttr[name=\"weight_ih_l0\"](%5)\n",
      "  %7 : Tensor = prim::GetAttr[name=\"weight_hh_l0\"](%5)\n",
      "  %8 : Tensor = prim::GetAttr[name=\"bias_ih_l0\"](%5)\n",
      "  %9 : Tensor = prim::GetAttr[name=\"bias_hh_l0\"](%5)\n",
      "  %10 : Tensor = prim::GetAttr[name=\"weight_ih_l0_reverse\"](%5)\n",
      "  %11 : Tensor = prim::GetAttr[name=\"weight_hh_l0_reverse\"](%5)\n",
      "  %12 : Tensor = prim::GetAttr[name=\"bias_ih_l0_reverse\"](%5)\n",
      "  %13 : Tensor = prim::GetAttr[name=\"bias_hh_l0_reverse\"](%5)\n",
      "  %14 : Tensor = prim::GetAttr[name=\"weight_ih_l1\"](%5)\n",
      "  %15 : Tensor = prim::GetAttr[name=\"weight_hh_l1\"](%5)\n",
      "  %16 : Tensor = prim::GetAttr[name=\"bias_ih_l1\"](%5)\n",
      "  %17 : Tensor = prim::GetAttr[name=\"bias_hh_l1\"](%5)\n",
      "  %18 : Tensor = prim::GetAttr[name=\"weight_ih_l1_reverse\"](%5)\n",
      "  %19 : Tensor = prim::GetAttr[name=\"weight_hh_l1_reverse\"](%5)\n",
      "  %20 : Tensor = prim::GetAttr[name=\"bias_ih_l1_reverse\"](%5)\n",
      "  %21 : Tensor = prim::GetAttr[name=\"bias_hh_l1_reverse\"](%5)\n",
      "  %22 : ClassType<Linear> = prim::GetAttr[name=\"linear\"](%self)\n",
      "  %weight : Tensor = prim::GetAttr[name=\"weight\"](%22)\n",
      "  %bias : Tensor = prim::GetAttr[name=\"bias\"](%22)\n",
      "  %27 : int = prim::Constant[value=0](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:74:0\n",
      "  %28 : int = aten::size(%input_seq, %27), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:74:0\n",
      "  %29 : Long() = prim::NumToTensor(%28), scope: BiGRU\n",
      "  %161 : int = aten::Int(%29), scope: BiGRU\n",
      "  %98 : int = aten::Int(%29), scope: BiGRU\n",
      "  %30 : int = prim::Constant[value=-1](), scope: BiGRU/Embedding[embedding] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py:1467:0\n",
      "  %31 : bool = prim::Constant[value=0](), scope: BiGRU/Embedding[embedding] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py:1467:0\n",
      "  %32 : bool = prim::Constant[value=0](), scope: BiGRU/Embedding[embedding] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py:1467:0\n",
      "  %emb_out.1 : Float(16, 44, 200) = aten::embedding(%weight.1, %input_seq, %30, %31, %32), scope: BiGRU/Embedding[embedding] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py:1467:0\n",
      "  %34 : int = prim::Constant[value=0](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:83:0\n",
      "  %35 : int = prim::Constant[value=2](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:83:0\n",
      "  %36 : int = prim::Constant[value=1](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:83:0\n",
      "  %37 : int[] = prim::ListConstruct(%34, %35, %36), scope: BiGRU\n",
      "  %input.1 : Float(16!, 200!, 44!) = aten::permute(%emb_out.1, %37), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:83:0\n",
      "  %39 : float = prim::Constant[value=0.5](), scope: BiGRU/Dropout2d[spatial_dropout1d] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py:844:0\n",
      "  %40 : bool = prim::Constant[value=0](), scope: BiGRU/Dropout2d[spatial_dropout1d] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py:844:0\n",
      "  %emb_out : Float(16!, 200!, 44!) = aten::feature_dropout(%input.1, %39, %40), scope: BiGRU/Dropout2d[spatial_dropout1d] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py:844:0\n",
      "  %42 : int = prim::Constant[value=0](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:86:0\n",
      "  %43 : int = prim::Constant[value=2](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:86:0\n",
      "  %44 : int = prim::Constant[value=1](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:86:0\n",
      "  %45 : int[] = prim::ListConstruct(%42, %43, %44), scope: BiGRU\n",
      "  %input.2 : Float(16, 44, 200) = aten::permute(%emb_out, %45), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:86:0\n",
      "  %47 : Device = prim::Constant[value=\"cpu\"](), scope: BiGRU # /root/anaconda3/lib/python3.7/site-packages/torch/nn/utils/rnn.py:265:0\n",
      "  %48 : int = prim::Constant[value=4](), scope: BiGRU # /root/anaconda3/lib/python3.7/site-packages/torch/nn/utils/rnn.py:265:0\n",
      "  %49 : bool = prim::Constant[value=0](), scope: BiGRU # /root/anaconda3/lib/python3.7/site-packages/torch/nn/utils/rnn.py:265:0\n",
      "  %50 : bool = prim::Constant[value=0](), scope: BiGRU # /root/anaconda3/lib/python3.7/site-packages/torch/nn/utils/rnn.py:265:0\n",
      "  %lengths.2 : Long(16) = aten::to(%lengths.1, %47, %48, %49, %50), scope: BiGRU # /root/anaconda3/lib/python3.7/site-packages/torch/nn/utils/rnn.py:265:0\n",
      "  %52 : bool = prim::Constant[value=1](), scope: BiGRU # /root/anaconda3/lib/python3.7/site-packages/torch/nn/utils/rnn.py:275:0\n",
      "  %input.3 : Float(704, 200), %batch_sizes : Long(44) = aten::_pack_padded_sequence(%input.2, %lengths.2, %52), scope: BiGRU # /root/anaconda3/lib/python3.7/site-packages/torch/nn/utils/rnn.py:275:0\n",
      "  %58 : int = prim::Constant[value=4](), scope: BiGRU/GRU[gru] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/rnn.py:691:0\n",
      "  %59 : int = prim::Constant[value=16](), scope: BiGRU/GRU[gru] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/rnn.py:691:0\n",
      "  %60 : int = prim::Constant[value=4](), scope: BiGRU/GRU[gru] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/rnn.py:691:0\n",
      "  %61 : int[] = prim::ListConstruct(%58, %59, %60), scope: BiGRU/GRU[gru]\n",
      "  %62 : int = prim::Constant[value=6](), scope: BiGRU/GRU[gru] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/rnn.py:691:0\n",
      "  %63 : int = prim::Constant[value=0](), scope: BiGRU/GRU[gru] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/rnn.py:691:0\n",
      "  %64 : Device = prim::Constant[value=\"cpu\"](), scope: BiGRU/GRU[gru] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/rnn.py:691:0\n",
      "  %65 : bool = prim::Constant[value=0](), scope: BiGRU/GRU[gru] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/rnn.py:691:0\n",
      "  %hx : Float(4, 16, 4) = aten::zeros(%61, %62, %63, %64, %65), scope: BiGRU/GRU[gru] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/rnn.py:691:0\n",
      "  %90 : Tensor[] = prim::ListConstruct(%6, %7, %8, %9, %10, %11, %12, %13, %14, %15, %16, %17, %18, %19, %20, %21), scope: BiGRU/GRU[gru]\n",
      "  %91 : bool = prim::Constant[value=1](), scope: BiGRU/GRU[gru] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/rnn.py:682:0\n",
      "  %92 : int = prim::Constant[value=2](), scope: BiGRU/GRU[gru] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/rnn.py:682:0\n",
      "  %93 : float = prim::Constant[value=0.5](), scope: BiGRU/GRU[gru] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/rnn.py:682:0\n",
      "  %94 : bool = prim::Constant[value=0](), scope: BiGRU/GRU[gru] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/rnn.py:682:0\n",
      "  %95 : bool = prim::Constant[value=1](), scope: BiGRU/GRU[gru] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/rnn.py:682:0\n",
      "  %96 : Float(704, 8), %hidden.1 : Float(4, 16, 4) = aten::gru(%input.3, %batch_sizes, %hx, %90, %91, %92, %93, %94, %95), scope: BiGRU/GRU[gru] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/rnn.py:682:0\n",
      "  %99 : int = prim::Constant[value=2](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:108:0\n",
      "  %100 : int = prim::Constant[value=2](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:108:0\n",
      "  %101 : int = prim::Constant[value=4](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:108:0\n",
      "  %102 : int[] = prim::ListConstruct(%99, %100, %98, %101), scope: BiGRU\n",
      "  %hidden : Float(2, 2, 16, 4) = aten::view(%hidden.1, %102), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:108:0\n",
      "  %104 : int = prim::Constant[value=0](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:109:0\n",
      "  %105 : int = prim::Constant[value=-1](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:109:0\n",
      "  %last_hidden.1 : Float(2, 16, 4) = aten::select(%hidden, %104, %105), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:109:0\n",
      "  %107 : int = prim::Constant[value=0](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:112:0\n",
      "  %108 : int[] = prim::ListConstruct(%107), scope: BiGRU\n",
      "  %109 : bool = prim::Constant[value=0](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:112:0\n",
      "  %110 : int? = prim::Constant(), scope: BiGRU\n",
      "  %last_hidden : Float(16, 4) = aten::sum(%last_hidden.1, %108, %109, %110), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:112:0\n",
      "  %112 : int = prim::Constant[value=0](), scope: BiGRU # /root/anaconda3/lib/python3.7/site-packages/torch/nn/utils/rnn.py:313:0\n",
      "  %113 : int = aten::size(%batch_sizes, %112), scope: BiGRU # /root/anaconda3/lib/python3.7/site-packages/torch/nn/utils/rnn.py:313:0\n",
      "  %max_seq_length : Long() = prim::NumToTensor(%113), scope: BiGRU\n",
      "  %115 : int = aten::Int(%max_seq_length), scope: BiGRU\n",
      "  %116 : bool = prim::Constant[value=1](), scope: BiGRU # /root/anaconda3/lib/python3.7/site-packages/torch/nn/utils/rnn.py:322:0\n",
      "  %117 : float = prim::Constant[value=0](), scope: BiGRU # /root/anaconda3/lib/python3.7/site-packages/torch/nn/utils/rnn.py:322:0\n",
      "  %gru_out.1 : Float(16!, 44!, 8), %lengths : Long(16) = aten::_pad_packed_sequence(%96, %batch_sizes, %116, %117, %115), scope: BiGRU # /root/anaconda3/lib/python3.7/site-packages/torch/nn/utils/rnn.py:322:0\n",
      "  %120 : int = prim::Constant[value=0](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %121 : int = prim::Constant[value=0](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %122 : int = prim::Constant[value=9223372036854775807](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %123 : int = prim::Constant[value=1](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %124 : Float(16!, 44!, 8) = aten::slice(%gru_out.1, %120, %121, %122, %123), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %125 : int = prim::Constant[value=1](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %126 : int = prim::Constant[value=0](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %127 : int = prim::Constant[value=9223372036854775807](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %128 : int = prim::Constant[value=1](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %129 : Float(16!, 44!, 8) = aten::slice(%124, %125, %126, %127, %128), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %130 : int = prim::Constant[value=2](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %131 : int = prim::Constant[value=0](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %132 : int = prim::Constant[value=4](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %133 : int = prim::Constant[value=1](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %134 : Float(16!, 44!, 4) = aten::slice(%129, %130, %131, %132, %133), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %135 : int = prim::Constant[value=0](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %136 : int = prim::Constant[value=0](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %137 : int = prim::Constant[value=9223372036854775807](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %138 : int = prim::Constant[value=1](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %139 : Float(16!, 44!, 8) = aten::slice(%gru_out.1, %135, %136, %137, %138), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %140 : int = prim::Constant[value=1](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %141 : int = prim::Constant[value=0](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %142 : int = prim::Constant[value=9223372036854775807](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %143 : int = prim::Constant[value=1](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %144 : Float(16!, 44!, 8) = aten::slice(%139, %140, %141, %142, %143), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %145 : int = prim::Constant[value=2](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %146 : int = prim::Constant[value=4](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %147 : int = prim::Constant[value=9223372036854775807](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %148 : int = prim::Constant[value=1](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %149 : Float(16!, 44!, 4) = aten::slice(%144, %145, %146, %147, %148), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %150 : int = prim::Constant[value=1](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %gru_out : Float(16!, 44!, 4) = aten::add(%134, %149, %150), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:121:0\n",
      "  %152 : int = prim::Constant[value=0](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:126:0\n",
      "  %153 : int = prim::Constant[value=2](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:126:0\n",
      "  %154 : int = prim::Constant[value=1](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:126:0\n",
      "  %155 : int[] = prim::ListConstruct(%152, %153, %154), scope: BiGRU\n",
      "  %input.4 : Float(16, 4!, 44!) = aten::permute(%gru_out, %155), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:126:0\n",
      "  %157 : int = prim::Constant[value=1](), scope: BiGRU # /root/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py:664:0\n",
      "  %158 : int[] = prim::ListConstruct(%157), scope: BiGRU\n",
      "  %159 : Float(16, 4, 1), %160 : Long(16, 4, 1) = aten::adaptive_max_pool1d(%input.4, %158), scope: BiGRU # /root/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py:664:0\n",
      "  %162 : int = prim::Constant[value=-1](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:126:0\n",
      "  %163 : int[] = prim::ListConstruct(%161, %162), scope: BiGRU\n",
      "  %max_pool : Float(16, 4) = aten::view(%159, %163), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:126:0\n",
      "  %165 : int = prim::Constant[value=1](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:131:0\n",
      "  %166 : int[] = prim::ListConstruct(%165), scope: BiGRU\n",
      "  %167 : bool = prim::Constant[value=0](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:131:0\n",
      "  %168 : int? = prim::Constant(), scope: BiGRU\n",
      "  %169 : Float(16, 4) = aten::sum(%gru_out, %166, %167, %168), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:131:0\n",
      "  %170 : int = prim::Constant[value=-1](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:131:0\n",
      "  %171 : int = prim::Constant[value=1](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:131:0\n",
      "  %172 : int[] = prim::ListConstruct(%170, %171), scope: BiGRU\n",
      "  %173 : Long(16, 1) = aten::view(%lengths, %172), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:131:0\n",
      "  %174 : Device = prim::Constant[value=\"cpu\"](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:131:0\n",
      "  %175 : int = prim::Constant[value=6](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:131:0\n",
      "  %176 : bool = prim::Constant[value=0](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:131:0\n",
      "  %177 : bool = prim::Constant[value=0](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:131:0\n",
      "  %178 : Float(16, 1) = aten::to(%173, %174, %175, %176, %177), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:131:0\n",
      "  %avg_pool : Float(16, 4) = aten::div(%169, %178), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:131:0\n",
      "  %180 : Tensor[] = prim::ListConstruct(%last_hidden, %max_pool, %avg_pool), scope: BiGRU\n",
      "  %181 : int = prim::Constant[value=1](), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:134:0\n",
      "  %input.5 : Float(16, 12) = aten::cat(%180, %181), scope: BiGRU # <ipython-input-9-9eacdc0fb8a8>:134:0\n",
      "  %183 : Float(12!, 2!) = aten::t(%weight), scope: BiGRU/Linear[linear] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py:1369:0\n",
      "  %184 : int = prim::Constant[value=1](), scope: BiGRU/Linear[linear] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py:1369:0\n",
      "  %185 : int = prim::Constant[value=1](), scope: BiGRU/Linear[linear] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py:1369:0\n",
      "  %input : Float(16, 2) = aten::addmm(%bias, %input.5, %183, %184, %185), scope: BiGRU/Linear[linear] # /root/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py:1369:0\n",
      "  %187 : int = prim::Constant[value=-1](), scope: BiGRU # /root/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py:1316:0\n",
      "  %188 : int? = prim::Constant(), scope: BiGRU\n",
      "  %189 : Float(16, 2) = aten::log_softmax(%input, %187, %188), scope: BiGRU # /root/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py:1316:0\n",
      "  return (%189)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "hidden_size = 4\n",
    "vocab_size = len(train_iterator.word2index)\n",
    "embedding_dim = 200\n",
    "n_layers = 2\n",
    "output_size = 2\n",
    "spatial_dropout = True\n",
    "dropout = 0.5\n",
    "\n",
    "writer = SummaryWriter('runs/exp-1')\n",
    "\n",
    "for batch in train_iterator:\n",
    "    input_seq, _, x_lengths = batch['input_seq'], batch['target'], batch['x_lengths']\n",
    "\n",
    "with SummaryWriter(comment='Model graph') as w:\n",
    "    w.add_graph(BiGRU(hidden_size, vocab_size, embedding_dim, output_size, n_layers, dropout,\n",
    "                      spatial_dropout, bidirectional=True), (input_seq, x_lengths), True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The generalization error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the dataset. Use clean_review and label columns\n",
    "test_dataset = pd.read_csv('dataset/datasets_feat_clean/test_feat_clean.csv',\n",
    "                          usecols=['clean_review', 'label'])\n",
    "\n",
    "# Change columns order\n",
    "test_dataset = test_dataset[['clean_review', 'label']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>clean_review</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>hear film till pop cable tv understand geena o...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>know film show local tv kid remember watch see...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>admire kiss mouth frankness  pubic hair cut m...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>see cure kid love year late get hold copy acci...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>movie true fact see documentary day early movi...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        clean_review  label\n",
       "0  hear film till pop cable tv understand geena o...      1\n",
       "1  know film show local tv kid remember watch see...      0\n",
       "2  admire kiss mouth frankness  pubic hair cut m...      0\n",
       "3  see cure kid love year late get hold copy acci...      1\n",
       "4  movie true fact see documentary day early movi...      0"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_dataset.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trimmed vocabulary using as minimum count threashold: count = 3.00\n",
      "13480/39104 tokens has been retained\n",
      "Trimmed input strings vocabulary\n",
      "Trimmed input sequences lengths to the length of: 187\n",
      "Mapped words to indices\n",
      "Batches created\n"
     ]
    }
   ],
   "source": [
    "test_iterator = BatchIterator(test_dataset, batch_size=256, vocab_created=False, vocab=None, target_col=None,\n",
    "                              word2index=train_iterator.word2index, sos_token='<SOS>', eos_token='<EOS>',\n",
    "                              unk_token='<UNK>', pad_token='<PAD>', min_word_count=3, max_vocab_size=None,\n",
    "                              max_seq_len=0.9, use_pretrained_vectors=False, glove_path='glove/',\n",
    "                              glove_name='glove.6B.100d.txt', weights_file_name='glove/weights.npy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "94b48c5d5a5c469da549f7c83cdb5172",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation', max=20, style=ProgressStyle(description_width='i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "_, test_avg_loss, test_accuracy, test_conf_matrix = model.evaluate_model(test_iterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 0.880. Test error: 0.331\n"
     ]
    }
   ],
   "source": [
    "print('Test accuracy: {:.3f}. Test error: {:.3f}'.format(test_accuracy, test_avg_loss))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAFZCAYAAAB0RP9xAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd5xU1d3H8c/XpaiAYBfpCGrUR43BEo1PNJbErimWGESDYuw1IZZYkhg1eYwlVYzGEmONvUSwxV5QsSAqiIUmFooKKCz7e/64d8k4LrvDzk7Zud+3r/ti5syZOWfW3d+c+Z1zz1VEYGZm2bBcpTtgZmbl46BvZpYhDvpmZhnioG9mliEO+mZmGeKgb2aWIR0q3QEzs2qz6MPJRa1l77jaQLVVX9qaR/pmZhnikb6ZWb6GxZXuQck46JuZ5YuGSvegZBz0zczyNTjom5llRtTwSN8TuWZmGeKRvplZPqd3zMwypIbTOw76Zmb5vGTTzCxDanik74lcM7MM8UjfzCxfDU/keqRvbUrSCpLulDRX0k1FvM6Bkka3Zd8qQdK9koZVuh+2bCIaijqqmYN+Rkn6oaSxkj6VNCMNTt9og5f+PrAmsGpE/KC1LxIR10bEzm3Qny+QtJ2kkHRLXvkmafnDBb7OWZL+0VK9iNglIq5qZXetUhoaijuqmIN+Bkk6EbgI+A1JgO4L/BnYqw1evh/wRkTUt8FrlcoHwNaSVs0pGwa80VYNKOG/L6s6/qXMGEndgV8CR0XELRExLyIWRcSdEfHTtE5nSRdJmp4eF0nqnD62naSpkk6S9H76LeGQ9LGzgTOA/dJvEMPzR8SS+qcj6g7p/YMlTZb0iaS3JB2YU/5YzvO2lvRsmjZ6VtLWOY89LOlXkh5PX2e0pNWa+TEsBG4D9k+fXwfsC1yb97O6WNIUSR9Lek7Stmn5d4BTc97nizn9OEfS48B8YGBadmj6+F8k3Zzz+udLekBS1e69nlnRUNxRxRz0s+frwPLArc3UOQ3YCtgU2ATYAjg95/G1gO5AL2A48CdJK0fEmSTfHm6IiK4RcXlzHZHUBbgE2CUiugFbA+OaqLcKcHdad1Xg98DdeSP1HwKHAGsAnYCTm2sbuBo4KL39bWA8MD2vzrMkP4NVgH8CN0laPiL+nfc+N8l5zlBgBNANeCfv9U4CNk4/0LYl+dkNi4iiLthhJdCwuLijijnoZ8+qwIctpF8OBH4ZEe9HxAfA2STBrNGi9PFFEXEP8CmwXiv70wBsJGmFiJgREeObqLMbMDEiromI+oi4DngN2COnzt8j4o2IWADcSBKslyoingBWkbQeSfC/uok6/4iIj9I2LwA60/L7vDIixqfPWZT3evOBH5F8aP0DOCYiprbwelYJHulbDfkIWK0xvbIUa/PFUeo7admS18j70JgPdF3WjkTEPGA/4CfADEl3S1q/gP409qlXzv33WtGfa4Cjge1p4ptPmsKakKaU5pB8u2kubQQwpbkHI+IZYDIgkg8nq0aeyLUa8iTwGbB3M3Wmk0zINurLl1MfhZoHrJhzf63cByPivojYCehJMnq/rID+NPZpWiv71Oga4EjgnnQUvkSafhlJkutfOSJ6AHNJgjXA0lIyzaZqJB1F8o1hOvCz1nfdrHUc9DMmIuaSTLb+SdLeklaU1FHSLpJ+m1a7Djhd0urphOgZJOmI1hgH/K+kvukk8imND0haU9KeaW7/c5I0UVMJ0XuAddNlph0k7QdsANzVyj4BEBFvAd8kmcPI1w2oJ1np00HSGcBKOY/PBPovywodSesCvyZJ8QwFfiap2TSUVYjTO1ZLIuL3wIkkk7MfkKQkjiZZ0QJJYBoLvAS8DDyflrWmrTHADelrPccXA/VyJJOb04FZJAH4yCZe4yNg97TuRyQj5N0j4sPW9CnvtR+LiKa+xdwH3EuyjPMdkm9HuambxhPPPpL0fEvtpOm0fwDnR8SLETGRZAXQNY0ro6yK1HB6R144YGb2RZ+9eE9RgXH5TXat2mW4HumbmWWIN1wzM8tX5Xn5Yjjom5nlq/K8fDEc9M3M8nmkb2aWIVW+lUIx2stEbvjw4cNHgUfVk9RH0kPpGd/jJR2Xlq8iaYykiem/K6flknSJpEmSXpK0Wc5rDUvrT1QB125oNyP9RTMmVLoLVkU69vxK8m+nXi3UtCxZtLDYk7RTpU/v1AMnRcTzkroBz0kaAxwMPBAR50n6OfBzkjPDdwEGp8eWwF+ALdPNCM8EhpB84D0n6Y6ImL20htvLSN/MrHxKfHJWurng8+ntT4AJJHtJ7QU0XnTnKv67XcpewNWReAroIaknyQ6xYyJiVhroxwDfaa7tdjPSNzMrmzJO5ErqD3wVeBpYMyJmQPLBIGmNtFovvnhG+NS0bGnlS+Wgb2aWr8glm5JGkFxXodGoiBjVRL2uwL+A4yPi42aup9PUA9FM+VI56JuZtbE0wH8pyOeS1JEk4F8bEY3XbJ4pqWc6yu8JvJ+WTwX65Dy9N8meVVOB7fLKH26uXef0zczylTinn14i83JgQroBYqM7SK7XTPrv7TnlB6WreLYC5qZpoPuAnSWtnK702TktWyqP9M3M8kSUfJ3+NiTba78sqfESoacC5wE3ShoOvAv8IH3sHmBXYBLJRYIOSfoZsyT9iuTSnpBc0W5Wcw076JuZ5SvxNgwR8RhN5+MBdmiifgBHLeW1rgCuKLRtp3fMzDLEI30zs3zee8fMLEO8y6aZWYZ4pG9mliE1PNL3RK6ZWYZ4pG9mls/pHTOzDKnh9I6DvplZPgd9M7MMqeH0jidyzcwyxCN9M7N8Tu+YmWVIDad3HPTNzPLV8EjfOX0zswzxSN/MLJ/TO2ZmGVLD6R0HfTOzfA76ZmYZElHpHpSMJ3LNzDLEI30zs3xO75iZZYiDvplZhnjJpplZhtTwSN8TuWZmGeKRvplZvhpesumgb2aWr4bTOw76Zmb5ajjoO6dvZpYhHumbmeXzkk0zs+yIBk/kmpllRw3n9B30zczy1XB6xxO5ZmYZ4pG+mVk+5/TNzDLEOX0zswxx0Dczy5Aa3nvHE7lmZhnikb6ZWT6nd8zMMsSrd8zMMsQnZ5mZWS3wSN/MLJ/TO2Zm2RGeyDUzyxCP9M3MMsQTuWZmVgs80jczy+f0jplZhngi18wsQzzSNzPLEE/kmplZLXDQr7AZ73/AIcefzh4HHc1eBx/DNTffCcB9Dz/OXgcfw/9svw+vvDbpC895/c23OfDIkex18DHsc8ixfP75QgDuffAx9vnxcex18DFc8Ncry/1WrEQ6d+7ME4/fxXNjxzBu3IOcccZJAPTv34fHH7uTV8c/xrXX/oWOHTsCcNDQfZk+7SXGPjuasc+O5seHHFDJ7rdPDVHcUcWc3qmwDnV1/PTIQ9hg3XWYN38B+444ia2HbMqgAX256Jc/5+wL/vyF+vX1i/n5ORdy7qnHs/6gAcyZ+zEdOtQxZ+7HXPDXK7lx1AWs0qM7p557MU899yJbfW2TCr0zayuff/45O+28L/PmzadDhw785+Fbue/fD3Hc8SO4+JLLuPHGO/jTH8/jx4ccwKWjrgbgppvu4LjjT69wz9uvWj4jt6wjfUmdy9lee7D6qquwwbrrANBlxRUY2K83Mz/8iHX69WFA315fqv/E2BdYd2B/1h80AIAe3Veirq6OKTNm0r/32qzSozsAW31tY8Y88mT53oiV1Lx58wHo2LEDHTt2JCLYfrtt+Ne/7gbgmmtuYs89v13JLtaWGh7plyXoS9pC0svAxPT+JpL+UI6225NpM2YyYeJkNv7Kukut886U6Ugw4qdn8YPDTuSK624BoG+vnrz17jSmzZhJff1iHnzsad57/8Nydd1KbLnllmPss6OZPu0l7n/gEd6c/DZz5sxl8eLFAEydNoO1e621pP4+++zK88+N4frrR9G799qV6rZVoXKN9C8Bdgc+AoiIF4Hty9R2uzB//gJOOPN8Rh49nK5dVlxqvfrFDbzw8gTOP+1Erv7DuTzw6NM89dyLdO/WlV+ceDgn//L/GHbsqfRaaw3q6urK+A6slBoaGhiy+c70HzCEzYd8lfXXH/zlSul1Xe+6ewyDBm/FZl/biQcfeJQrLr+ozL2tAR7pF99ORLyTV7a4uSdIGiFprKSxo0aNKmHXKm9RfT3Hn3k+u+34TXb63683W3fN1VdlyCYbsnKPlVhh+c5su9VmvDpxMgDbbb0F1/3ld1z75/Pp36cX/Xr3LEf3rYzmzv2Y/zzyBFtuuRk9enRf8sHeu1dPpk+fCcCsWbNZuDCZ3P/b5dey2Wb/U7H+tlvRUNxRxcoV9KdI2gIISXWSjgfeaO4JETEqIoZExJARI0aUp5cVEBGc8ds/MrBvb4btu1eL9bfZ4qu8MfkdFnz2OfX1ixk7bjzr9OsDwEez5wAw95NPuf62e/nebjuVtO9WHquttgrdu68EwPLLL88O39qW116bxMP/eYLvfW83AIYO/QF33jkagLXWWmPJc/fYY2dey1v9ZQWo4ZF+uVbvHEGS4ukLzATuT8sy74WXJ3Dn6IcZPLAf3xt+PADHHfYjFi6q59yLL2PW3LkcecqvWH/QAEb97iy6d+vKQT/Yk/1/cjJCbLvVZnzz60MAOO8Pl/P6m28B8JOD9qN/ny9PBFv707Pnmlxx+UXU1S2HlluOm2++k3vuuZ8JE97g2n/8mbPP+hnjXhzPFX+/DoCjj/4xu+++M4vrFzNr1hyGH3p8hd9B+xMlDtySriBJeb8fERvllB8DHA3UA3dHxM/S8lOA4SQZkmMj4r60/DvAxUAd8LeIOK/FtiOq+1MpFYtmTKh0H6yKdOz5leTfTv5gs/9atHAagIp9nU+O36OowNjtojub7YOk/wU+Ba5uDPqStgdOA3aLiM8lrRER70vaALgO2AJYm2TQ3Lja4w1gJ2Aq8CxwQES82lzbZRnpS7oM+NIPMSJqN29jZu1XiUf6EfGIpP55xUcA50XE52md99PyvYDr0/K3JE0i+QAAmBQRkwEkXZ/WrXzQJ/lkarQ8sA8wpUxtm5ktm8qcnLUusK2kc4DPgJMj4lmgF/BUTr2paRl8MY5OBbZsqZGyBP2IuCH3vqRrgDHlaNvMbJkVOdKXNALIzWSMioiWliF2AFYGtgI2B26UNJCm01VB0wtxWux4pbZhGAD0q1DbZmbNKzLopwF+WdeaTwVuiWSi9RlJDcBqaXmfnHq9genp7aWVL1W5zsidLWlWeswhGeWfWo62zczaiduAbwFIWhfoBHwI3AHsL6mzpAHAYOAZkonbwZIGSOoE7J/WbVbJR/qSBGwCTEuLGqKdLBkys2wqdYiSdB2wHbCapKnAmcAVwBWSXgEWAsPSWDle0o0kE7T1wFERsTh9naOB+0iWbF4REeNbbLsc8VfScxHxtSJewks27Qu8ZNOa0lZLNj8+bOeiAuNKl40uug+lUq4zcp+RtFmZ2jIzK47PyG0dSR0ioh74BnCYpDeBeSSfxBER/iAwMyujUuf0nwE2A/YucTtmZm2m1NswVFKpg74AIuLNErdjZtZ2HPRbbXVJJy7twYj4fYnbNzNbdtW9O3JRSh3064CutMFsuplZuTi903ozIuKXJW7DzMwKVJacvplZu+KRfqvtUOLXNzNre87pt05EzCrl65uZlUIt5/TLdUaumZlVgUptrWxmVr2c3jEzy45aTu846JuZ5fNI38wsO6KGg74ncs3MMsQjfTOzfDU80nfQNzPLU8vpHQd9M7N8DvpmZtlRyyN9T+SamWWIR/pmZnlqeaTvoG9mlsdB38wsS6J2LwXSqpy+pH6Serd1Z8zMrLQKCvqS/iHp6+ntg4DXgTckHVzCvpmZVUQ0FHdUs0JH+jsDz6W3TwJ2ArYCTi1Fp8zMKikaVNRRzQrN6XeKiIWS1gZWj4hHAST1LF3XzMwqo9pH68UoNOi/KOmnQH/gboD0A+DjEvXLzKxiwhO5HApsDvQATk/LtgGuK0WnzMysNAoa6UfERGDfvLKbgJtK0Skzs0rKZHonXaXTooi4uu26Y2ZWedU+GVuM5kb6hxXw/AAc9M2spkTtXiJ36UE/IrYtZ0fMzKpFLY/0Cz4jV9LKkg6QdGJ6f610BY+ZmbUThZ6Ruy3wBjAcODstXh/4a4n6ZWZWMT45Cy4GDoyI0ZJmp2VPAVuUpltmZpWTyZx+ngERMTq93fjjWAh0bPsumZlVVrWP1otRaE7/NUk75pV9C3iljftjZmYlVOhI/2Tgdkm3AytI+hOwT3qYmdWUWt6GodAzch+X9FVgKMm6/BnA1yPinVJ2zsysEjJ5Rm6+iJgC/EbSyhExu8UnmJm1Uw01PNIvdMlmd0l/lzQf+FDS/PR+jxL3z8ys7CJU1FHNCp3IvYJkh80tgZXTf1dKy83MrJ0oNL3zLWDtiFiQ3n853ZBtWmm6ZWZWOV6yCZOAvnllvYGJbdsdM7PKiyjuqGaFbq18HzBa0lXAFKAPcBBwTWm7Z2ZWfrU80l+WrZXfBbbPuT8F+Gab98jMrMJqefWOt1Y2M8uQgtfpm5llRbUvuyxGoev015Z0o6SZkhbnHqXuoJlZudXyRG6hq3f+mtbdDfiUZEvlu4EjS9QvM7OKaQgVdVSzQtM72wD9IuJTSRERz0k6BHgMuLR03TMzs7ZUaNBfTLJ/PsBcSasDc0nW6puZ1ZRazukXGvSfBXYBbgfGAP8E5gPPl6hfZmYVU+15+WIUGvSH8t/8/7HASKAr8PtSdKopHXt+pVxNWTuyaKF3ArG2V+15+WIUup/+rJzb84EzS9YjM7MKy2R6R9IZhbxARPyy7bqzdB069SpHM9ZO1Kcj/EUfTq5wT6yadFxtYKW7UPWaG+kPLuD5NZz5MrOsymR6JyKGlrMjZmbVopZHs4WenGVmlhmlPjlL0hWS3pf0Sk7Z7yS9JuklSbfmXplQ0imSJkl6XdK3c8q/k5ZNkvTzQt6bg76ZWZ4yXC7xSuA7eWVjgI0iYmPgDeAUAEkbAPsDG6bP+bOkOkl1wJ9IltNvAByQ1m2Wg76ZWZlFxCPArLyy0RFRn959iv+e/LoXcH1EfB4Rb5Fc1GqL9JgUEZMjYiFwfVq3WQ76ZmZ5Goo8JI2QNDbnGLGMXfgxcG96uxfJ9UsaTU3LllberIK3Vpa0PclXjDUjYm9JmwHdIuI/hb6GmVl7EBS3eiciRgGjWvNcSacB9cC1jUVNNUHTg/YW56AL3Vr5SOBykk+VxqtnLQTOKeT5ZmbtSUMUd7SWpGHA7sCBEUs2g5hKconaRr2B6c2UN6vQ9M5JwI4R8WuSby8AEwDvjWBm1gYkfYdki5s9050PGt0B7C+ps6QBJOdQPUOyJ9pgSQMkdSLJxNzRUjuFpne6Ae+ktxs/fTrw3503zcxqRkOR6Z2WSLoO2A5YTdJUkq1tTgE6A2MkATwVET+JiPGSbgReJUn7HBURi9PXORq4D6gDroiI8S21XWjQfww4GTg/p+wowPl8M6s5xeb0W3z9iAOaKL68mfrn0EQ6PSLuAe5ZlrYLDfrHAHdJOgzoJmk8ySh/12VpzMysPWhouUq7Vegum9MkfQ34OtCXZEL3ycavGGZmtaTUI/1KKnjJZkQ0AI+nh5mZtUMFBX1Jb7GU9Z8R4b1MzaymZD69Axyad78nSZ7/urbtjplZ5WU+6EfEA/llkh4gmTW+qK07ZWZWSc7pN20B4NSOmdWchtqN+QXn9PMvnbgisBswus17ZGZmJVPoSD//0onzSPZxvrJNe2NmVgVKfUZuJbUY9NON+scAN0bEZ6XvkplZZWX6conpCVh/cMA3s6wodj/9alboLpt3S/KWC2Zm7VyhOf3lgFskPUayBcOSbz8R8eNSdMzMrFIalOGcfmoi8LtSdsTMrFrUck6/2aAv6YCIuC4iflGuDpmZVVq15+WL0VJO/9Ky9MLMrIo0qLijmrUU9Ku8+2ZmtixayunXSdqeZoJ/RDzYtl0yM6usLJ+c1ZnkEl5L+wkE3n/HzGpMZidygXneL9/Msqba8/LFKPTkLDMzqwEtjfRr+PPOzKxptbxks9mgHxHdytURM7NqkeWcvplZ5tRyTt9B38wsTy2ndzyRa2aWIR7pm5nlqeWRvoO+mVmecE7fzCw7PNI3M8uQWg76nsg1M8sQj/TNzPL45CwzswzxyVlmZhninL6ZmdUEj/TNzPLU8kjfQd/MLI8ncs3MMsQTuWZmGVLL6R1P5JqZZYhH+mZmeZzTNzPLkIYaDvsO+mZmeWo5p++gb2aWp3bH+Z7INTPLFI/0zczyOL1jZpYhPjnLzCxDann1jnP6ZmYZ4qBfRXr3Xpv7R9/Eyy89zIvjHuSYo4cDsPHGG/DYI3fwwvP3c9utV9KtW1cAdtxhW55+6l5eeP5+nn7qXrbfbptKdt/a0IyZH3DI0SPZ44cj2OvAw7nmxtsAuO/BR9nrwMP5n2/syisT3lhSf9GiRZx+zu/ZZ+gRfHfYkTzz/EtLHjv8xNP57rAj2evAwzn7t39g8eLFZX8/7U0UeVQzp3eqSH19PT/92dm8MO4VunbtwjNP/5v7H3iES//6O0aO/BWPPPoUBw/bj5NPOoIzz/odH340i733OZgZM2ay4Ybrcc9d19JvwJBKvw1rAx3q6vjpMYexwXqDmDdvPvsOP5atN/8qgwb246Lf/IKzf3fJF+rffMe/Abj1mr/w0ew5HHHSL7j+bxez3HLLccGvTqFrly5EBCecdg73PfQou+64XQXeVftRyxO5HulXkffee58Xxr0CwKefzuO11ybSa+21WG/ddXjk0acAuP+BR9lnn10BGDduPDNmzARg/PjXWX755enUqVNlOm9tavXVVmGD9QYB0KXLigzs14eZH3zEOv37MqBf7y/Vf/Ptd9lyyKYArLpyD7p17cL41yYC0LVLFwDqFy9mUf0iRA3PUraRBqKoo5qVLegr8SNJZ6T3+0raolzttzf9+vVm00024ulnXmD8+NfZY4+dAfj+93anT++1v1T/u9/djXHjXmHhwoXl7qqV2LQZM5kw8U023nC9pdZZb9AAHnr0SerrFzN1+nu8+vok3pv5wZLHR5xwGt/c/QC6rLgiO2//jXJ0u12r5fROOUf6fwa+DhyQ3v8E+FMZ2283unRZkRtvuIwTTz6TTz75lENHnMiRPzmYp5+6l27durBw4aIv1N9gg3U595xTOeKokRXqsZXK/PkLOOG0XzPy2MOXjNibss9u32bN1Vdjv+HHcv7Fl7LpRl+hrkPdksdHXXgOD91+LQsXLuLp514sR9etSpUzp79lRGwm6QWAiJgtaam5CEkjgBEAl156aZm6WHkdOnTgphsu47rrbuW22+4F4PXX32SX3X4IwODBA9l1lx2W1O/Vqyc333Q5h/z4OCZPfqcifbbSWFRfz/Gn/Zrddt6enVqYpO/QoY6Rxx2+5P6Bh59Iv7xvhJ07d2L7b2zJQ48+xdZbbFaSPtcK5/TbxiJJdaTffiStTjM/24gYFRFDImLIiBEjytXHirts1AVMeG0SF108aknZ6quvCoAkTj3lOC4ddQ0A3buvxB23X81pp5/LE0+OrUh/rTQigjPOvYiB/fowbP/vtlh/wWefMX/BZwA88czzdKirY50B/Zg/fwEffDgLgPr6xTzy5Ngm5wTsi2o5p1/Okf4lwK3AGpLOAb4PnF7G9qveNltvztAffZ+XXn6Vsc+OBuAXvziPQYMGcMQRBwNw2233cOVVNwBw1JGHMGid/px26vGcdurxAOyy6wF88MFHFem/tZ0XXhrPnf9+gMHr9Od7w44C4LjDh7Fw0SLOvfAvzJozlyN/eibrDx7IqAvPYdbsuRx+wmloueVYc/VVOfeMkwGY/9lnHD3yLBYuWkTD4ga2/Nom7Lv3bpV8a+1CdYft4iiifG9P0vrADoCAByJiQoFPjQ6depWuY9bu1C+cBsCiDydXuCdWTTquNhAofnnScf33LyowXvz29VW7RKqcq3fWAd6KiD8BrwA7SepRrvbNzKy8Of1/AYslDQL+BgwA/lnG9s3MChJF/lcISSdIGi/pFUnXSVpe0gBJT0uaKOmGxsUukjqn9yelj/dv7XsrZ9BviIh64LvAxRFxAtCzjO2bmRWkocijJZJ6AccCQyJiI6AO2B84H7gwIgYDs4Hh6VOGA7MjYhBwYVqvVcq9eucA4CDgrrSsYxnbNzMrSJlW73QAVpDUAVgRmAF8C7g5ffwqYO/09l7pfdLHd5DUqnmDcgb9Q0hOzjonIt6SNAD4RxnbNzOrChExDfg/4F2SYD8XeA6Yk2ZEAKYCjStYegFT0ufWp/VXbU3bZVuyGRGvknydabz/FnBeudo3MytUsWsac08uTY2KiFE5j69MMnofAMwBbgJ2aaYrTY3qW9XNkgd9SS/TTOciYuNS98HMbFkUe4JVGuBHNVNlR5LVjB8ASLoF2BroIalDOprvDUxP608F+gBT03RQd2BWa/pWjpH+7mVow8yszZRhG4Z3ga0krQgsIDl/aSzwEMmJq9cDw4Db0/p3pPefTB9/MFp5klXJg35EeEMYM2tXCl122erXj3ha0s3A80A98ALJN4O7gesl/Totuzx9yuXANZImkYzw929t22XL6UvaCvgD8BWgE8kSpXkRsVK5+mBmVi0i4kzgzLziycCXtpyPiM+AH7RFu+Xce+ePJJ9ONwFDSJZuDipj+2ZmBanlXTbLernEiJgkqS4iFgN/l/REOds3MytEqdM7lVTOoD8/PaV4nKTfkqxNXfpVIczMKqSWR/rlPDlraNre0cA8kuVH3ytj+2ZmBWmIKOqoZuVYp983It7NWcXzGXB2qds1M7MvK8dI/7bGG5L+VYb2zMyKUssXRi9HTj/39OGBZWjPzKwo1X7Jw2KUI+jHUm6bmVUlr94pziaSPiYZ8a+Q3ia9Hz45y8ysfMqxDUNdqdswM2tLtbxks6wnZ5mZtQfO6ZuZZYhz+mZmGVLL6Z1ynpFrZmYV5pG+mVmeVl6fpF1w0Dczy+OJXDOzDKnlnL6DvplZnlpeveOJXDOzDPFI38wsj3P6ZmYZ4tU7ZmYZUssTuc7pm5lliEf6ZmZ5aucDjdAAAAfDSURBVHn1joO+mVkeT+SamWWIJ3LNzDKklkf6nsg1M8sQj/TNzPJ4ItfMLEManNM3M8uO2g35DvpmZl/iiVwzM6sJHumbmeWp5ZG+g76ZWR6fnGVmliG1PNJ3Tt/MLEM80jczy+OTs8zMMsQ5fTOzDKnlnL6DvplZnloe6Xsi18wsQzzSNzPL4/SOmVmGePWOmVmG1PLWys7pm5lliEf6ZmZ5nN4xM8uQWk7vOOibmeXxSN/MLENqeaTviVwzswzxSN/MLI/TO1WgfuG0SnfBqlDH1QZWugtWg2o5vdNegr4q3YFqIWlERIyqdD+suvj3om3V8kjfOf32Z0SlO2BVyb8XbSiioaijmjnom5llSHtJ75iZlY132bRq4rytNcW/F22oli+iolp+c2ZmrdF7lY2KCoxTZ71StYtPnNM3M8sQB/0qIGlVSePS4z1J03Lud6p0/6z8JC3O+R0YJ6l/M3X7S3qlfL2rfRFR1FEoSXWSXpB0V3p/gKSnJU2UdEPj37+kzun9Senj/Vv73hz0q0BEfBQRm0bEpsBfgQsb70fEQgAl/P8rOxbk/A5sGhFvV7pDWdIQUdSxDI4DJuTcP5/k738wMBsYnpYPB2ZHxCDgwrReqziIVDFJgyS9IumvwPNAH0lzch7fX9Lf0ttrSrpF0lhJz0jaqlL9ttJIR/SPSno+PbZuos6G6f//cZJekjQ4Lf9RTvmlkurK/w7ajyjyv0JI6g3sBjT+DQv4FnBzWuUqYO/09l7pfdLHd0jrLzMH/eq3AXB5RHwVaG4vikuA30bEEGBf0l8ka7dWyEnt3JqWvQ/sFBGbAfuR/D/P9xPg4vRb4xBgqqSvpPW3ScsXAweW/i20X8WmdySNSAdgjUdTJ89dBPwMaDyba1VgTkTUp/enAr3S272AKWnf6oG5af1l5iWb1e/NiHi2gHo7AuvlfPivLGmFiFhQuq5ZCS1IA3SujsAfJTUG7nWbeN6TwGnpKPKWiJgoaQfga8Cz6e/HCiQfIFYi6ZYYS11GK2l34P2IeE7Sdo3FTb1UAY8tEwf96jcv53YDX/yfv3zObQFbNM4BWE06AZgJbELyLf2z/AoR8U9JT5OkDe6TdCjJ78ZVEXFKOTvbnpXh5KxtgD0l7Uryd7wSyci/h6QO6Wi+NzA9rT8V6EPyza0D0B2Y1ZqGnd5pRyLZ1GO2pMHppO4+OQ/fDxzVeCcdDVpt6Q7MSH8PhgJfystLGghMjohLgDuAjYEHgO9LWiOts4qkfuXrdvtT6tU7EXFKRPSOiP7A/sCDEXEg8BDw/bTaMOD29PYd6X3Sxx+MVp5k5aDf/owE/k3yhzw1p/woYJt08u5V4LBKdM5K6s/AMElPkaR25jVRZz/gFUnjgPWBqyPiVeB0YLSkl4AxQM8y9bldKuPqnXwjgRMlTSLJ2V+ell8OrJqWnwj8vLUN+IxcM7M8K3cdVFRgnP3pJJ+Ra2ZmleeJXDOzPN5l08wsQ2o57e2gb2aWp5avkeucvplZhjjoW9VJ95iJ9CQUJN0raVhLz2vidfpK+tT7zNiyKsfeO5XioG+tJultSQvSwDpT0t8ldW3rdiJil4i4qqV6aX92zHneuxHRNSIWt3WfrLZVcJ1+yTnoW7H2iIiuwGbA5iQnAS3hLaGtPSrXfvqV4D9GaxMRMQ24F9hI0sOSzpH0ODAfGCipu6TLJc1ILxLz68a0S3ohif+T9KGkyST7xiyRvt6hOfcPkzRB0ieSXpW0maRrgL7Anek3j581kSZaW9IdkmalF6M4LOc1z5J0o6Sr09cdL2lIzuMj035/Iun1dBMzq1FO75i1QFIfYFfghbRoKDAC6Aa8Q7IXeD0wCPgqsDPQGMgPA3ZPy4fw371HmmrnB8BZwEEkm1TtCXwUEUOBd0m/eUTEb5t4+nUkW1esnbbxm7zgvSdwPdCDZK+TP6ZtrgccDWweEd2AbwNvt/xTMas+DvpWrNuUXNjlMeA/wG/S8isjYny6W+AqwC7A8RExLyLeJ7n6z/5p3X2BiyJiSkTMAs5tpr1DSa4b8GwkJkXEOy11Mv1Q+gYwMiI+i4hxJNccGJpT7bGIuCedA7iGZDdLSLYx7gxsIKljRLwdEW+21Ka1X7Wc3vE6fSvW3hFxf25Bumf7lJyifiR7wc/I2e9/uZw6a+fVby6I9wFaE3DXBmZFxCd57QzJuf9ezu35wPLpNreTJB1P8g1jQ0n3ASdGxHSsJlV74C6GR/pWKrl/NVOAz4HVIqJHeqwUERumj88gCeaN+jbzulOAdQpoM990YBVJ3fLaae5qZP994Yh/RsQ3SD7AgiKuUWrVL4o8qplH+lZyETFD0mjgAkm/AD4FBgC9I+I/wI3AsZLuItkuuLltY/8G/F7SYyTXDV4HWJSmeGYCA5fShymSngDOlXQyydbEw4EftdT/NKffC3ic5MIlC/CAqabVL5xWtbtkFsu/uFYuBwGdgFeB2SQXd27c0/0y4D7gRZJAfsvSXiQibgLOAf4JfALcRjJnAMlcwOmS5qSBPd8BQH+SUf+twJkRMaaAvncGzgM+JEkBrQGcWsDzzKqO99M3M8sQj/TNzDLEQd/MLEMc9M3MMsRB38wsQxz0zcwyxEHfzCxDHPTNzDLEQd/MLEMc9M3MMuT/AZykuxWdoHJNAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x432 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Confusion matrix\n",
    "plt.figure(figsize=(6,6))\n",
    "ax = sns.heatmap(test_conf_matrix, fmt='d', annot=True, linewidths=1, square=True)\n",
    "ax.set_xlabel('Predictions', size=12)\n",
    "ax.set_ylabel('True labels', size=12) \n",
    "ax.set_title('Confusion Matrix', size=12); \n",
    "ax.xaxis.set_ticklabels(['True', 'False'])\n",
    "ax.yaxis.set_ticklabels(['True', 'False'])\n",
    "ax.set_ylim(2,0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The generalization accuracy of the biGRU model equals 0.880. As we can see on the above plot of the confusion matrix the both, positive and negative classes were similarly numerous, and the prediction mistakes amount (TN, FP) is also very similar, so model learned both classes in the same detail. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}