transformer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "s_qNSzzyaCbD"
   },
   "source": [
    "##### Copyright 2019 The TensorFlow Authors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2020-11-02T02:23:39.100332Z",
     "iopub.status.busy": "2020-11-02T02:23:39.099742Z",
     "iopub.status.idle": "2020-11-02T02:23:39.102016Z",
     "shell.execute_reply": "2020-11-02T02:23:39.101527Z"
    },
    "id": "jmjh290raIky"
   },
   "outputs": [],
   "source": [
    "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "# https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "J0Qjg6vuaHNt"
   },
   "source": [
    "# Transformer model for language understanding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AOpGoE2T-YXS"
   },
   "source": [
    "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://www.tensorflow.org/tutorials/text/transformer\">\n",
    "    <img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />\n",
    "    View on TensorFlow.org</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/text/transformer.ipynb\">\n",
    "    <img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />\n",
    "    Run in Google Colab</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/tutorials/text/transformer.ipynb\">\n",
    "    <img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />\n",
    "    View source on GitHub</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/tutorials/text/transformer.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "M-f8TnGpE_ex"
   },
   "source": [
    "This tutorial trains a <a href=\"https://arxiv.org/abs/1706.03762\" class=\"external\">Transformer model</a> to translate Portuguese to English. This is an advanced example that assumes knowledge of [text generation](text_generation.ipynb) and [attention](nmt_with_attention.ipynb).\n",
    "\n",
    "The core idea behind the Transformer model is *self-attention*—the ability to attend to different positions of the input sequence to compute a representation of that sequence. Transformer creates stacks of self-attention layers and is explained below in the sections *Scaled dot product attention* and *Multi-head attention*.\n",
    "\n",
    "A transformer model handles variable-sized input using stacks of self-attention layers instead of [RNNs](text_classification_rnn.ipynb) or [CNNs](../images/intro_to_cnns.ipynb). This general architecture has a number of advantages:\n",
    "\n",
    "* It make no assumptions about the temporal/spatial relationships across the data. This is ideal for processing a set of objects (for example, [StarCraft units](https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-ii/#block-8)).\n",
    "* Layer outputs can be calculated in parallel, instead of a series like an RNN.\n",
    "* Distant items can affect each other's output without passing through many RNN-steps, or convolution layers (see [Scene Memory Transformer](https://arxiv.org/pdf/1903.03878.pdf) for example).\n",
    "* It can learn long-range dependencies. This is a challenge in many sequence tasks.\n",
    "\n",
    "The downsides of this architecture are:\n",
    "\n",
    "* For a time-series, the output for a time-step is calculated from the *entire history* instead of only the inputs and current hidden-state. This _may_ be less efficient.   \n",
    "* If the input *does* have a  temporal/spatial relationship, like text, some positional encoding must be added or the model will effectively see a bag of words. \n",
    "\n",
    "After training the model in this notebook, you will be able to input a Portuguese sentence and return the English translation.\n",
    "\n",
    "<img src=\"https://www.tensorflow.org/images/tutorials/transformer/attention_map_portuguese.png\" width=\"800\" alt=\"Attention heatmap\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:23:39.106807Z",
     "iopub.status.busy": "2020-11-02T02:23:39.106149Z",
     "iopub.status.idle": "2020-11-02T02:23:45.915265Z",
     "shell.execute_reply": "2020-11-02T02:23:45.915750Z"
    },
    "id": "XFG0NDRu5mYQ"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting matplotlib==3.2.2\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Downloading matplotlib-3.2.2-cp36-cp36m-manylinux1_x86_64.whl (12.4 MB)\r\n",
      "\u001b[?25l\r",
      "\u001b[K     |                                | 10 kB 32.4 MB/s eta 0:00:01\r",
      "\u001b[K     |                                | 20 kB 1.7 MB/s eta 0:00:08\r",
      "\u001b[K     |                                | 30 kB 2.1 MB/s eta 0:00:06"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[K     |                                | 40 kB 1.7 MB/s eta 0:00:08\r",
      "\u001b[K     |▏                               | 51 kB 2.0 MB/s eta 0:00:07\r",
      "\u001b[K     |▏                               | 61 kB 2.2 MB/s eta 0:00:06\r",
      "\u001b[K     |▏                               | 71 kB 2.4 MB/s eta 0:00:06\r",
      "\u001b[K     |▏                               | 81 kB 2.7 MB/s eta 0:00:05\r",
      "\u001b[K     |▎                               | 92 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▎                               | 102 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▎                               | 112 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▎                               | 122 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▍                               | 133 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▍                               | 143 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▍                               | 153 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▍                               | 163 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▌                               | 174 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▌                               | 184 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▌                               | 194 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▌                               | 204 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▌                               | 215 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▋                               | 225 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▋                               | 235 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▋                               | 245 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▋                               | 256 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▊                               | 266 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▊                               | 276 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▊                               | 286 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▊                               | 296 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▉                               | 307 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▉                               | 317 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▉                               | 327 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |▉                               | 337 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█                               | 348 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█                               | 358 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█                               | 368 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█                               | 378 kB 2.8 MB/s eta 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[K     |█                               | 389 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█                               | 399 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█                               | 409 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█                               | 419 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█                               | 430 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▏                              | 440 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▏                              | 450 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▏                              | 460 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▏                              | 471 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▎                              | 481 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▎                              | 491 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▎                              | 501 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▎                              | 512 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▍                              | 522 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▍                              | 532 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▍                              | 542 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▍                              | 552 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▌                              | 563 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▌                              | 573 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▌                              | 583 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▌                              | 593 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▋                              | 604 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▋                              | 614 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▋                              | 624 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▋                              | 634 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▋                              | 645 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▊                              | 655 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▊                              | 665 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▊                              | 675 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▊                              | 686 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▉                              | 696 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▉                              | 706 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▉                              | 716 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |█▉                              | 727 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██                              | 737 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██                              | 747 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██                              | 757 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██                              | 768 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██                              | 778 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██                              | 788 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██                              | 798 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██                              | 808 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▏                             | 819 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▏                             | 829 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▏                             | 839 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▏                             | 849 kB 2.8 MB/s eta 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[K     |██▏                             | 860 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▎                             | 870 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▎                             | 880 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▎                             | 890 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▎                             | 901 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▍                             | 911 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▍                             | 921 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▍                             | 931 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▍                             | 942 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▌                             | 952 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▌                             | 962 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▌                             | 972 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▌                             | 983 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▋                             | 993 kB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▋                             | 1.0 MB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▋                             | 1.0 MB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▋                             | 1.0 MB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▋                             | 1.0 MB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▊                             | 1.0 MB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▊                             | 1.1 MB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▊                             | 1.1 MB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▊                             | 1.1 MB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▉                             | 1.1 MB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▉                             | 1.1 MB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▉                             | 1.1 MB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |██▉                             | 1.1 MB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |███                             | 1.1 MB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |███                             | 1.1 MB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |███                             | 1.1 MB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |███                             | 1.2 MB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |███                             | 1.2 MB 2.8 MB/s eta 0:00:05\r",
      "\u001b[K     |███                             | 1.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███                             | 1.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███                             | 1.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▏                            | 1.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▏                            | 1.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▏                            | 1.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▏                            | 1.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▏                            | 1.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▎                            | 1.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▎                            | 1.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▎                            | 1.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▎                            | 1.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▍                            | 1.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▍                            | 1.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▍                            | 1.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▍                            | 1.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▌                            | 1.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▌                            | 1.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▌                            | 1.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▌                            | 1.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▋                            | 1.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▋                            | 1.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▋                            | 1.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▋                            | 1.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▊                            | 1.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▊                            | 1.4 MB 2.8 MB/s eta 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[K     |███▊                            | 1.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▊                            | 1.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▊                            | 1.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▉                            | 1.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▉                            | 1.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▉                            | 1.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███▉                            | 1.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████                            | 1.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████                            | 1.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████                            | 1.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████                            | 1.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████                            | 1.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████                            | 1.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████                            | 1.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████                            | 1.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▏                           | 1.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▏                           | 1.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▏                           | 1.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▏                           | 1.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▎                           | 1.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▎                           | 1.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▎                           | 1.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▎                           | 1.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▎                           | 1.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▍                           | 1.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▍                           | 1.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▍                           | 1.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▍                           | 1.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▌                           | 1.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▌                           | 1.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▌                           | 1.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▌                           | 1.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▋                           | 1.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▋                           | 1.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▋                           | 1.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▋                           | 1.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▊                           | 1.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▊                           | 1.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▊                           | 1.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▊                           | 1.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▉                           | 1.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▉                           | 1.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▉                           | 1.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▉                           | 1.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████▉                           | 1.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████                           | 1.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████                           | 1.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████                           | 1.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████                           | 1.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████                           | 1.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████                           | 2.0 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████                           | 2.0 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████                           | 2.0 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▏                          | 2.0 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▏                          | 2.0 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▏                          | 2.0 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▏                          | 2.0 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▎                          | 2.0 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▎                          | 2.0 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▎                          | 2.0 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▎                          | 2.1 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▎                          | 2.1 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▍                          | 2.1 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▍                          | 2.1 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▍                          | 2.1 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▍                          | 2.1 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▌                          | 2.1 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▌                          | 2.1 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▌                          | 2.1 MB 2.8 MB/s eta 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[K     |█████▌                          | 2.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▋                          | 2.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▋                          | 2.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▋                          | 2.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▋                          | 2.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▊                          | 2.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▊                          | 2.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▊                          | 2.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▊                          | 2.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▉                          | 2.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▉                          | 2.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▉                          | 2.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▉                          | 2.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████▉                          | 2.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████                          | 2.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████                          | 2.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████                          | 2.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████                          | 2.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████                          | 2.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████                          | 2.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████                          | 2.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████                          | 2.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▏                         | 2.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▏                         | 2.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▏                         | 2.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▏                         | 2.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▎                         | 2.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▎                         | 2.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▎                         | 2.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▎                         | 2.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▍                         | 2.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▍                         | 2.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▍                         | 2.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▍                         | 2.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▍                         | 2.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▌                         | 2.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▌                         | 2.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▌                         | 2.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▌                         | 2.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▋                         | 2.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▋                         | 2.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▋                         | 2.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▋                         | 2.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▊                         | 2.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▊                         | 2.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▊                         | 2.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▊                         | 2.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▉                         | 2.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▉                         | 2.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▉                         | 2.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████▉                         | 2.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████                         | 2.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████                         | 2.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████                         | 2.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████                         | 2.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████                         | 2.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████                         | 2.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████                         | 2.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████                         | 2.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████                         | 2.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▏                        | 2.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▏                        | 2.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▏                        | 2.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▏                        | 2.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▎                        | 2.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▎                        | 2.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▎                        | 2.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▎                        | 2.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▍                        | 2.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▍                        | 2.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▍                        | 2.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▍                        | 2.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▌                        | 2.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▌                        | 2.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▌                        | 2.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▌                        | 2.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▌                        | 2.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▋                        | 2.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▋                        | 2.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▋                        | 3.0 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▋                        | 3.0 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▊                        | 3.0 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▊                        | 3.0 MB 2.8 MB/s eta 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[K     |███████▊                        | 3.0 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▊                        | 3.0 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▉                        | 3.0 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▉                        | 3.0 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▉                        | 3.0 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |███████▉                        | 3.1 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████                        | 3.1 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████                        | 3.1 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████                        | 3.1 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████                        | 3.1 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████                        | 3.1 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████                        | 3.1 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████                        | 3.1 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████                        | 3.1 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████                        | 3.1 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▏                       | 3.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▏                       | 3.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▏                       | 3.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▏                       | 3.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▎                       | 3.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▎                       | 3.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▎                       | 3.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▎                       | 3.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▍                       | 3.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▍                       | 3.2 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▍                       | 3.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▍                       | 3.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▌                       | 3.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▌                       | 3.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▌                       | 3.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▌                       | 3.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▌                       | 3.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▋                       | 3.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▋                       | 3.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▋                       | 3.3 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▋                       | 3.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▊                       | 3.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▊                       | 3.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▊                       | 3.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▊                       | 3.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▉                       | 3.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▉                       | 3.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▉                       | 3.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |████████▉                       | 3.4 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████                       | 3.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████                       | 3.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████                       | 3.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████                       | 3.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████                       | 3.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████                       | 3.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████                       | 3.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████                       | 3.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████                       | 3.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▏                      | 3.5 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▏                      | 3.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▏                      | 3.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▏                      | 3.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▎                      | 3.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▎                      | 3.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▎                      | 3.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▎                      | 3.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▍                      | 3.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▍                      | 3.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▍                      | 3.6 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▍                      | 3.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▌                      | 3.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▌                      | 3.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▌                      | 3.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▌                      | 3.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▋                      | 3.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▋                      | 3.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▋                      | 3.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▋                      | 3.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▋                      | 3.7 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▊                      | 3.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▊                      | 3.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▊                      | 3.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▊                      | 3.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▉                      | 3.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▉                      | 3.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▉                      | 3.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |█████████▉                      | 3.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████████                      | 3.8 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████████                      | 3.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████████                      | 3.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████████                      | 3.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████████                      | 3.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████████                      | 3.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████████                      | 3.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████████                      | 3.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████████                      | 3.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████████▏                     | 3.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████████▏                     | 3.9 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████████▏                     | 4.0 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████████▏                     | 4.0 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████████▎                     | 4.0 MB 2.8 MB/s eta 0:00:04\r",
      "\u001b[K     |██████████▎                     | 4.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████▎                     | 4.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████▎                     | 4.0 MB 2.8 MB/s eta 0:00:03"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[K     |██████████▍                     | 4.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████▍                     | 4.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████▍                     | 4.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████▍                     | 4.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████▌                     | 4.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████▌                     | 4.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████▌                     | 4.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████▌                     | 4.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████▋                     | 4.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████▋                     | 4.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████▋                     | 4.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████▋                     | 4.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████▋                     | 4.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████▊                     | 4.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████▊                     | 4.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████▊                     | 4.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████▊                     | 4.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████▉                     | 4.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████▉                     | 4.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████▉                     | 4.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████▉                     | 4.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████                     | 4.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████                     | 4.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████                     | 4.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████                     | 4.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████                     | 4.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████                     | 4.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████                     | 4.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████                     | 4.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▏                    | 4.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▏                    | 4.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▏                    | 4.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▏                    | 4.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▏                    | 4.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▎                    | 4.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▎                    | 4.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▎                    | 4.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▎                    | 4.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▍                    | 4.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▍                    | 4.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▍                    | 4.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▍                    | 4.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▌                    | 4.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▌                    | 4.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▌                    | 4.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▌                    | 4.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▋                    | 4.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▋                    | 4.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▋                    | 4.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▋                    | 4.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▊                    | 4.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▊                    | 4.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▊                    | 4.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▊                    | 4.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▊                    | 4.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▉                    | 4.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▉                    | 4.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▉                    | 4.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████▉                    | 4.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████                    | 4.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████                    | 4.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████                    | 4.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████                    | 4.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████                    | 4.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████                    | 4.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████                    | 4.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████                    | 4.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▏                   | 4.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▏                   | 4.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▏                   | 4.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▏                   | 4.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▎                   | 4.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▎                   | 4.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▎                   | 4.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▎                   | 4.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▎                   | 4.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▍                   | 4.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▍                   | 4.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▍                   | 4.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▍                   | 4.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▌                   | 4.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▌                   | 4.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▌                   | 4.9 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▌                   | 4.9 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▋                   | 4.9 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▋                   | 4.9 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▋                   | 4.9 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▋                   | 4.9 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▊                   | 4.9 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▊                   | 4.9 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▊                   | 4.9 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▊                   | 4.9 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▊                   | 5.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▉                   | 5.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▉                   | 5.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▉                   | 5.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████▉                   | 5.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████                   | 5.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████                   | 5.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████                   | 5.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████                   | 5.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████                   | 5.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████                   | 5.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████                   | 5.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████                   | 5.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▏                  | 5.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▏                  | 5.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▏                  | 5.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▏                  | 5.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▎                  | 5.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▎                  | 5.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▎                  | 5.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▎                  | 5.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▎                  | 5.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▍                  | 5.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▍                  | 5.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▍                  | 5.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▍                  | 5.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▌                  | 5.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▌                  | 5.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▌                  | 5.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▌                  | 5.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▋                  | 5.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▋                  | 5.3 MB 2.8 MB/s eta 0:00:03"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[K     |█████████████▋                  | 5.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▋                  | 5.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▊                  | 5.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▊                  | 5.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▊                  | 5.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▊                  | 5.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▉                  | 5.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▉                  | 5.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▉                  | 5.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▉                  | 5.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████▉                  | 5.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████                  | 5.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████                  | 5.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████                  | 5.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████                  | 5.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████                  | 5.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████                  | 5.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████                  | 5.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████                  | 5.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▏                 | 5.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▏                 | 5.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▏                 | 5.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▏                 | 5.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▎                 | 5.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▎                 | 5.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▎                 | 5.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▎                 | 5.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▍                 | 5.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▍                 | 5.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▍                 | 5.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▍                 | 5.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▍                 | 5.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▌                 | 5.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▌                 | 5.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▌                 | 5.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▌                 | 5.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▋                 | 5.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▋                 | 5.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▋                 | 5.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▋                 | 5.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▊                 | 5.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▊                 | 5.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▊                 | 5.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▊                 | 5.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▉                 | 5.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▉                 | 5.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▉                 | 5.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |██████████████▉                 | 5.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████                 | 5.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████                 | 5.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████                 | 5.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████                 | 5.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████                 | 5.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████                 | 5.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████                 | 5.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████                 | 5.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████                 | 5.9 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▏                | 5.9 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▏                | 5.9 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▏                | 5.9 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▏                | 5.9 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▎                | 5.9 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▎                | 5.9 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▎                | 5.9 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▎                | 5.9 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▍                | 5.9 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▍                | 6.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▍                | 6.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▍                | 6.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▍                | 6.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▌                | 6.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▌                | 6.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▌                | 6.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▌                | 6.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▋                | 6.0 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▋                | 6.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▋                | 6.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▋                | 6.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▊                | 6.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▊                | 6.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▊                | 6.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▊                | 6.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▉                | 6.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▉                | 6.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▉                | 6.1 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |███████████████▉                | 6.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████                | 6.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████                | 6.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████                | 6.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████                | 6.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████                | 6.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████                | 6.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████                | 6.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████                | 6.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████                | 6.2 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▏               | 6.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▏               | 6.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▏               | 6.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▏               | 6.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▎               | 6.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▎               | 6.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▎               | 6.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▎               | 6.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▍               | 6.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▍               | 6.3 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▍               | 6.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▍               | 6.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▌               | 6.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▌               | 6.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▌               | 6.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▌               | 6.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▌               | 6.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▋               | 6.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▋               | 6.4 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▋               | 6.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▋               | 6.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▊               | 6.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▊               | 6.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▊               | 6.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▊               | 6.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▉               | 6.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▉               | 6.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▉               | 6.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |████████████████▉               | 6.5 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████               | 6.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████               | 6.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████               | 6.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████               | 6.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████               | 6.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████               | 6.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████               | 6.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████               | 6.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████               | 6.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████▏              | 6.6 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████▏              | 6.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████▏              | 6.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████▏              | 6.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████▎              | 6.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████▎              | 6.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████▎              | 6.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████▎              | 6.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████▍              | 6.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████▍              | 6.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████▍              | 6.7 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████▍              | 6.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████▌              | 6.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████▌              | 6.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████▌              | 6.8 MB 2.8 MB/s eta 0:00:03\r",
      "\u001b[K     |█████████████████▌              | 6.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████▌              | 6.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████▋              | 6.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████▋              | 6.8 MB 2.8 MB/s eta 0:00:02"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[K     |█████████████████▋              | 6.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████▋              | 6.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████▊              | 6.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████▊              | 6.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████▊              | 6.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████▊              | 6.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████▉              | 6.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████▉              | 6.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████▉              | 6.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████▉              | 6.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████              | 6.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████              | 7.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████              | 7.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████              | 7.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████              | 7.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████              | 7.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████              | 7.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████              | 7.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████              | 7.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▏             | 7.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▏             | 7.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▏             | 7.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▏             | 7.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▎             | 7.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▎             | 7.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▎             | 7.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▎             | 7.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▍             | 7.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▍             | 7.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▍             | 7.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▍             | 7.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▌             | 7.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▌             | 7.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▌             | 7.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▌             | 7.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▋             | 7.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▋             | 7.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▋             | 7.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▋             | 7.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▋             | 7.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▊             | 7.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▊             | 7.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▊             | 7.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▊             | 7.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▉             | 7.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▉             | 7.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▉             | 7.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████▉             | 7.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████             | 7.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████             | 7.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████             | 7.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████             | 7.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████             | 7.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████             | 7.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████             | 7.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████             | 7.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▏            | 7.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▏            | 7.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▏            | 7.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▏            | 7.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▏            | 7.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▎            | 7.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▎            | 7.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▎            | 7.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▎            | 7.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▍            | 7.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▍            | 7.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▍            | 7.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▍            | 7.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▌            | 7.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▌            | 7.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▌            | 7.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▌            | 7.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▋            | 7.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▋            | 7.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▋            | 7.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▋            | 7.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▊            | 7.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▊            | 7.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▊            | 7.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▊            | 7.7 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▊            | 7.7 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▉            | 7.7 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▉            | 7.7 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▉            | 7.7 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████▉            | 7.7 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████            | 7.7 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████            | 7.7 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████            | 7.7 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████            | 7.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████            | 7.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████            | 7.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████            | 7.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████            | 7.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▏           | 7.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▏           | 7.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▏           | 7.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▏           | 7.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▏           | 7.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▎           | 7.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▎           | 7.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▎           | 7.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▎           | 7.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▍           | 7.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▍           | 7.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▍           | 7.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▍           | 7.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▌           | 7.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▌           | 7.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▌           | 8.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▌           | 8.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▋           | 8.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▋           | 8.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▋           | 8.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▋           | 8.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▊           | 8.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▊           | 8.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▊           | 8.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▊           | 8.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▊           | 8.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▉           | 8.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▉           | 8.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▉           | 8.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████▉           | 8.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████           | 8.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████           | 8.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████           | 8.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████           | 8.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████           | 8.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████           | 8.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████           | 8.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████           | 8.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▏          | 8.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▏          | 8.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▏          | 8.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▏          | 8.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▎          | 8.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▎          | 8.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▎          | 8.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▎          | 8.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▎          | 8.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▍          | 8.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▍          | 8.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▍          | 8.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▍          | 8.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▌          | 8.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▌          | 8.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▌          | 8.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▌          | 8.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▋          | 8.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▋          | 8.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▋          | 8.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▋          | 8.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▊          | 8.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▊          | 8.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▊          | 8.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▊          | 8.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▉          | 8.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▉          | 8.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▉          | 8.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▉          | 8.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |█████████████████████▉          | 8.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████          | 8.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████          | 8.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████          | 8.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████          | 8.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████          | 8.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████          | 8.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████          | 8.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████          | 8.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▏         | 8.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▏         | 8.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▏         | 8.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▏         | 8.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▎         | 8.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▎         | 8.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▎         | 8.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▎         | 8.7 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▍         | 8.7 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▍         | 8.7 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▍         | 8.7 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▍         | 8.7 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▍         | 8.7 MB 2.8 MB/s eta 0:00:02"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[K     |██████████████████████▌         | 8.7 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▌         | 8.7 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▌         | 8.7 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▌         | 8.7 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▋         | 8.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▋         | 8.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▋         | 8.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▋         | 8.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▊         | 8.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▊         | 8.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▊         | 8.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▊         | 8.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▉         | 8.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▉         | 8.8 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▉         | 8.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▉         | 8.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |██████████████████████▉         | 8.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████         | 8.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████         | 8.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████         | 8.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████         | 8.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████         | 8.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████         | 8.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████         | 8.9 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████         | 9.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▏        | 9.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▏        | 9.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▏        | 9.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▏        | 9.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▎        | 9.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▎        | 9.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▎        | 9.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▎        | 9.0 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▍        | 9.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▍        | 9.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▍        | 9.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▍        | 9.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▍        | 9.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▌        | 9.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▌        | 9.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▌        | 9.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▌        | 9.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▋        | 9.1 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▋        | 9.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▋        | 9.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▋        | 9.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▊        | 9.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▊        | 9.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▊        | 9.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▊        | 9.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▉        | 9.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▉        | 9.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▉        | 9.2 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |███████████████████████▉        | 9.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████        | 9.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████        | 9.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████        | 9.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████        | 9.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████        | 9.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████        | 9.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████        | 9.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████        | 9.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████        | 9.3 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▏       | 9.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▏       | 9.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▏       | 9.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▏       | 9.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▎       | 9.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▎       | 9.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▎       | 9.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▎       | 9.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▍       | 9.4 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▍       | 9.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▍       | 9.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▍       | 9.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▌       | 9.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▌       | 9.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▌       | 9.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▌       | 9.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▌       | 9.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▋       | 9.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▋       | 9.5 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▋       | 9.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▋       | 9.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▊       | 9.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▊       | 9.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▊       | 9.6 MB 2.8 MB/s eta 0:00:02\r",
      "\u001b[K     |████████████████████████▊       | 9.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████▉       | 9.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████▉       | 9.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████▉       | 9.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████▉       | 9.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████       | 9.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████       | 9.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████       | 9.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████       | 9.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████       | 9.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████       | 9.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████       | 9.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████       | 9.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████       | 9.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▏      | 9.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▏      | 9.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▏      | 9.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▏      | 9.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▎      | 9.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▎      | 9.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▎      | 9.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▎      | 9.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▍      | 9.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▍      | 9.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▍      | 9.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▍      | 9.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▌      | 9.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▌      | 9.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▌      | 9.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▌      | 9.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▌      | 9.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▋      | 9.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▋      | 9.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▋      | 9.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▋      | 10.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▊      | 10.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▊      | 10.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▊      | 10.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▊      | 10.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▉      | 10.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▉      | 10.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▉      | 10.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▉      | 10.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████      | 10.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████      | 10.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████      | 10.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████      | 10.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████      | 10.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████      | 10.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████      | 10.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████      | 10.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████      | 10.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▏     | 10.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▏     | 10.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▏     | 10.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▏     | 10.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▎     | 10.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▎     | 10.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▎     | 10.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▎     | 10.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▍     | 10.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▍     | 10.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▍     | 10.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▍     | 10.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▌     | 10.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▌     | 10.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▌     | 10.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▌     | 10.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▋     | 10.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▋     | 10.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▋     | 10.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▋     | 10.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▋     | 10.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▊     | 10.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▊     | 10.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▊     | 10.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▊     | 10.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▉     | 10.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▉     | 10.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▉     | 10.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▉     | 10.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████     | 10.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████     | 10.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████     | 10.5 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████     | 10.5 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████     | 10.5 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████     | 10.5 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████     | 10.5 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████     | 10.5 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▏    | 10.5 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▏    | 10.5 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▏    | 10.5 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▏    | 10.5 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▏    | 10.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▎    | 10.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▎    | 10.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▎    | 10.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▎    | 10.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▍    | 10.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▍    | 10.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▍    | 10.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▍    | 10.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▌    | 10.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▌    | 10.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▌    | 10.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▌    | 10.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▋    | 10.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▋    | 10.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▋    | 10.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▋    | 10.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▋    | 10.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▊    | 10.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▊    | 10.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▊    | 10.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▊    | 10.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▉    | 10.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▉    | 10.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▉    | 10.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▉    | 10.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████    | 10.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████    | 10.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████    | 10.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████    | 10.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████    | 10.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████    | 10.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████    | 10.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████    | 10.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▏   | 10.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▏   | 10.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▏   | 10.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▏   | 10.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▏   | 10.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▎   | 11.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▎   | 11.0 MB 2.8 MB/s eta 0:00:01"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[K     |████████████████████████████▎   | 11.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▎   | 11.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▍   | 11.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▍   | 11.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▍   | 11.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▍   | 11.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▌   | 11.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▌   | 11.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▌   | 11.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▌   | 11.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▋   | 11.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▋   | 11.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▋   | 11.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▋   | 11.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▊   | 11.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▊   | 11.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▊   | 11.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▊   | 11.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▊   | 11.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▉   | 11.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▉   | 11.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▉   | 11.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▉   | 11.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████   | 11.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████   | 11.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████   | 11.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████   | 11.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████   | 11.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████   | 11.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████   | 11.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████   | 11.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▏  | 11.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▏  | 11.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▏  | 11.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▏  | 11.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▎  | 11.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▎  | 11.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▎  | 11.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▎  | 11.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▎  | 11.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▍  | 11.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▍  | 11.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▍  | 11.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▍  | 11.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▌  | 11.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▌  | 11.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▌  | 11.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▌  | 11.5 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▋  | 11.5 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▋  | 11.5 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▋  | 11.5 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▋  | 11.5 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▊  | 11.5 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▊  | 11.5 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▊  | 11.5 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▊  | 11.5 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▉  | 11.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▉  | 11.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▉  | 11.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▉  | 11.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▉  | 11.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████  | 11.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████  | 11.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████  | 11.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████  | 11.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████  | 11.6 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████  | 11.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████  | 11.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████  | 11.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▏ | 11.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▏ | 11.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▏ | 11.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▏ | 11.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▎ | 11.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▎ | 11.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▎ | 11.7 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▎ | 11.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▎ | 11.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▍ | 11.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▍ | 11.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▍ | 11.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▍ | 11.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▌ | 11.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▌ | 11.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▌ | 11.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▌ | 11.8 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▋ | 11.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▋ | 11.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▋ | 11.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▋ | 11.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▊ | 11.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▊ | 11.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▊ | 11.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▊ | 11.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▉ | 11.9 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▉ | 12.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▉ | 12.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▉ | 12.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▉ | 12.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████ | 12.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████ | 12.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████ | 12.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████ | 12.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████ | 12.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████ | 12.0 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████ | 12.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████ | 12.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▏| 12.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▏| 12.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▏| 12.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▏| 12.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▎| 12.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▎| 12.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▎| 12.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▎| 12.1 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▍| 12.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▍| 12.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▍| 12.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▍| 12.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▍| 12.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▌| 12.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▌| 12.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▌| 12.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▌| 12.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▋| 12.2 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▋| 12.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▋| 12.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▋| 12.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▊| 12.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▊| 12.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▊| 12.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▊| 12.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▉| 12.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▉| 12.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▉| 12.3 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▉| 12.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████████| 12.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████████| 12.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████████| 12.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████████| 12.4 MB 2.8 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████████| 12.4 MB 2.8 MB/s eta 0:00:01"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[K     |████████████████████████████████| 12.4 MB 2.8 MB/s \r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?25hRequirement already satisfied: numpy>=1.11 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from matplotlib==3.2.2) (1.18.5)\r\n",
      "Requirement already satisfied: cycler>=0.10 in /home/kbuilder/.local/lib/python3.6/site-packages (from matplotlib==3.2.2) (0.10.0)\r\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /home/kbuilder/.local/lib/python3.6/site-packages (from matplotlib==3.2.2) (1.2.0)\r\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /home/kbuilder/.local/lib/python3.6/site-packages (from matplotlib==3.2.2) (2.4.7)\r\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /home/kbuilder/.local/lib/python3.6/site-packages (from matplotlib==3.2.2) (2.8.1)\r\n",
      "Requirement already satisfied: six in /home/kbuilder/.local/lib/python3.6/site-packages (from cycler>=0.10->matplotlib==3.2.2) (1.15.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Installing collected packages: matplotlib\r\n",
      "  Attempting uninstall: matplotlib\r\n",
      "    Found existing installation: matplotlib 3.3.2\r\n",
      "    Not uninstalling matplotlib at /home/kbuilder/.local/lib/python3.6/site-packages, outside environment /tmpfs/src/tf_docs_env\r\n",
      "    Can't uninstall 'matplotlib'. No files were found to uninstall.\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully installed matplotlib-3.2.2\r\n"
     ]
    }
   ],
   "source": [
    "!pip install -q tfds-nightly\n",
    "\n",
    "# Pin matplotlib version to 3.2.2 since in the latest version\n",
    "# transformer.ipynb fails with the following error:\n",
    "# https://stackoverflow.com/questions/62953704/valueerror-the-number-of-fixedlocator-locations-5-usually-from-a-call-to-set\n",
    "!pip install matplotlib==3.2.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:23:45.920611Z",
     "iopub.status.busy": "2020-11-02T02:23:45.919935Z",
     "iopub.status.idle": "2020-11-02T02:23:53.342882Z",
     "shell.execute_reply": "2020-11-02T02:23:53.342273Z"
    },
    "id": "JjJJyJTZYebt"
   },
   "outputs": [],
   "source": [
    "import tensorflow_datasets as tfds\n",
    "import tensorflow as tf\n",
    "\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fd1NWMxjfsDd"
   },
   "source": [
    "## Setup input pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "t4_Qt8W1hJE_"
   },
   "source": [
    "Use [TFDS](https://www.tensorflow.org/datasets) to load the [Portugese-English translation dataset](https://github.com/neulab/word-embeddings-for-nmt) from the [TED Talks Open Translation Project](https://www.ted.com/participate/translate).\n",
    "\n",
    "This dataset contains approximately 50000 training examples, 1100 validation examples, and 2000 test examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:23:53.350740Z",
     "iopub.status.busy": "2020-11-02T02:23:53.349965Z",
     "iopub.status.idle": "2020-11-02T02:24:16.829151Z",
     "shell.execute_reply": "2020-11-02T02:24:16.829598Z"
    },
    "id": "8q9t4FmN96eN"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mDownloading and preparing dataset ted_hrlr_translate/pt_to_en/1.0.0 (download: 124.94 MiB, generated: Unknown size, total: 124.94 MiB) to /home/kbuilder/tensorflow_datasets/ted_hrlr_translate/pt_to_en/1.0.0...\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shuffling and writing examples to /home/kbuilder/tensorflow_datasets/ted_hrlr_translate/pt_to_en/1.0.0.incomplete9RVVAD/ted_hrlr_translate-train.tfrecord\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shuffling and writing examples to /home/kbuilder/tensorflow_datasets/ted_hrlr_translate/pt_to_en/1.0.0.incomplete9RVVAD/ted_hrlr_translate-validation.tfrecord\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shuffling and writing examples to /home/kbuilder/tensorflow_datasets/ted_hrlr_translate/pt_to_en/1.0.0.incomplete9RVVAD/ted_hrlr_translate-test.tfrecord\n",
      "\u001b[1mDataset ted_hrlr_translate downloaded and prepared to /home/kbuilder/tensorflow_datasets/ted_hrlr_translate/pt_to_en/1.0.0. Subsequent calls will reuse this data.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "examples, metadata = tfds.load('ted_hrlr_translate/pt_to_en', with_info=True,\n",
    "                               as_supervised=True)\n",
    "train_examples, val_examples = examples['train'], examples['validation']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RCEKotqosGfq"
   },
   "source": [
    "Create a custom subwords tokenizer from the training dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:24:16.835394Z",
     "iopub.status.busy": "2020-11-02T02:24:16.834726Z",
     "iopub.status.idle": "2020-11-02T02:26:24.694629Z",
     "shell.execute_reply": "2020-11-02T02:26:24.695136Z"
    },
    "id": "KVBg5Q8tBk5z"
   },
   "outputs": [],
   "source": [
    "tokenizer_en = tfds.deprecated.text.SubwordTextEncoder.build_from_corpus(\n",
    "    (en.numpy() for pt, en in train_examples), target_vocab_size=2**13)\n",
    "\n",
    "tokenizer_pt = tfds.deprecated.text.SubwordTextEncoder.build_from_corpus(\n",
    "    (pt.numpy() for pt, en in train_examples), target_vocab_size=2**13)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:24.700665Z",
     "iopub.status.busy": "2020-11-02T02:26:24.700009Z",
     "iopub.status.idle": "2020-11-02T02:26:24.702248Z",
     "shell.execute_reply": "2020-11-02T02:26:24.702679Z"
    },
    "id": "4DYWukNFkGQN"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokenized string is [7915, 1248, 7946, 7194, 13, 2799, 7877]\n",
      "The original string: Transformer is awesome.\n"
     ]
    }
   ],
   "source": [
    "sample_string = 'Transformer is awesome.'\n",
    "\n",
    "tokenized_string = tokenizer_en.encode(sample_string)\n",
    "print ('Tokenized string is {}'.format(tokenized_string))\n",
    "\n",
    "original_string = tokenizer_en.decode(tokenized_string)\n",
    "print ('The original string: {}'.format(original_string))\n",
    "\n",
    "assert original_string == sample_string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "o9KJWJjrsZ4Y"
   },
   "source": [
    "The tokenizer encodes the string by breaking it into subwords if the word is not in its dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:24.707214Z",
     "iopub.status.busy": "2020-11-02T02:26:24.706529Z",
     "iopub.status.idle": "2020-11-02T02:26:24.708820Z",
     "shell.execute_reply": "2020-11-02T02:26:24.709240Z"
    },
    "id": "bf2ntBxjkqK6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7915 ----> T\n",
      "1248 ----> ran\n",
      "7946 ----> s\n",
      "7194 ----> former \n",
      "13 ----> is \n",
      "2799 ----> awesome\n",
      "7877 ----> .\n"
     ]
    }
   ],
   "source": [
    "for ts in tokenized_string:\n",
    "  print ('{} ----> {}'.format(ts, tokenizer_en.decode([ts])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:24.712950Z",
     "iopub.status.busy": "2020-11-02T02:26:24.712287Z",
     "iopub.status.idle": "2020-11-02T02:26:24.714005Z",
     "shell.execute_reply": "2020-11-02T02:26:24.714450Z"
    },
    "id": "bcRp7VcQ5m6g"
   },
   "outputs": [],
   "source": [
    "BUFFER_SIZE = 20000\n",
    "BATCH_SIZE = 64"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kGi4PoVakxdc"
   },
   "source": [
    "Add a start and end token to the input and target. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:24.719434Z",
     "iopub.status.busy": "2020-11-02T02:26:24.718759Z",
     "iopub.status.idle": "2020-11-02T02:26:24.720965Z",
     "shell.execute_reply": "2020-11-02T02:26:24.720470Z"
    },
    "id": "UZwnPr4R055s"
   },
   "outputs": [],
   "source": [
    "def encode(lang1, lang2):\n",
    "  lang1 = [tokenizer_pt.vocab_size] + tokenizer_pt.encode(\n",
    "      lang1.numpy()) + [tokenizer_pt.vocab_size+1]\n",
    "\n",
    "  lang2 = [tokenizer_en.vocab_size] + tokenizer_en.encode(\n",
    "      lang2.numpy()) + [tokenizer_en.vocab_size+1]\n",
    "  \n",
    "  return lang1, lang2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Tx1sFbR-9fRs"
   },
   "source": [
    "You want to use `Dataset.map` to apply this function to each element of the dataset.  `Dataset.map` runs in graph mode.\n",
    "\n",
    "* Graph tensors do not have a value. \n",
    "* In graph mode you can only use TensorFlow Ops and functions. \n",
    "\n",
    "So you can't `.map` this function directly: You need to wrap it in a `tf.py_function`. The `tf.py_function` will pass regular tensors (with a value and a `.numpy()` method to access it), to the wrapped python function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:24.725590Z",
     "iopub.status.busy": "2020-11-02T02:26:24.724939Z",
     "iopub.status.idle": "2020-11-02T02:26:24.726798Z",
     "shell.execute_reply": "2020-11-02T02:26:24.727223Z"
    },
    "id": "Mah1cS-P70Iz"
   },
   "outputs": [],
   "source": [
    "def tf_encode(pt, en):\n",
    "  result_pt, result_en = tf.py_function(encode, [pt, en], [tf.int64, tf.int64])\n",
    "  result_pt.set_shape([None])\n",
    "  result_en.set_shape([None])\n",
    "\n",
    "  return result_pt, result_en"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6JrGp5Gek6Ql"
   },
   "source": [
    "Note: To keep this example small and relatively fast, drop examples with a length of over 40 tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:24.731110Z",
     "iopub.status.busy": "2020-11-02T02:26:24.730465Z",
     "iopub.status.idle": "2020-11-02T02:26:24.732249Z",
     "shell.execute_reply": "2020-11-02T02:26:24.732655Z"
    },
    "id": "2QEgbjntk6Yf"
   },
   "outputs": [],
   "source": [
    "MAX_LENGTH = 40"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:24.736845Z",
     "iopub.status.busy": "2020-11-02T02:26:24.736227Z",
     "iopub.status.idle": "2020-11-02T02:26:24.737975Z",
     "shell.execute_reply": "2020-11-02T02:26:24.738396Z"
    },
    "id": "c081xPGv1CPI"
   },
   "outputs": [],
   "source": [
    "def filter_max_length(x, y, max_length=MAX_LENGTH):\n",
    "  return tf.logical_and(tf.size(x) <= max_length,\n",
    "                        tf.size(y) <= max_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:24.743511Z",
     "iopub.status.busy": "2020-11-02T02:26:24.742805Z",
     "iopub.status.idle": "2020-11-02T02:26:24.863582Z",
     "shell.execute_reply": "2020-11-02T02:26:24.862990Z"
    },
    "id": "9mk9AZdZ5bcS"
   },
   "outputs": [],
   "source": [
    "train_dataset = train_examples.map(tf_encode)\n",
    "train_dataset = train_dataset.filter(filter_max_length)\n",
    "# cache the dataset to memory to get a speedup while reading from it.\n",
    "train_dataset = train_dataset.cache()\n",
    "train_dataset = train_dataset.shuffle(BUFFER_SIZE).padded_batch(BATCH_SIZE)\n",
    "train_dataset = train_dataset.prefetch(tf.data.experimental.AUTOTUNE)\n",
    "\n",
    "\n",
    "val_dataset = val_examples.map(tf_encode)\n",
    "val_dataset = val_dataset.filter(filter_max_length).padded_batch(BATCH_SIZE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:24.867883Z",
     "iopub.status.busy": "2020-11-02T02:26:24.867130Z",
     "iopub.status.idle": "2020-11-02T02:26:24.945263Z",
     "shell.execute_reply": "2020-11-02T02:26:24.945694Z"
    },
    "id": "_fXvfYVfQr2n"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<tf.Tensor: shape=(64, 38), dtype=int64, numpy=\n",
       " array([[8214,  342, 3032, ...,    0,    0,    0],\n",
       "        [8214,   95,  198, ...,    0,    0,    0],\n",
       "        [8214, 4479, 7990, ...,    0,    0,    0],\n",
       "        ...,\n",
       "        [8214,  584,   12, ...,    0,    0,    0],\n",
       "        [8214,   59, 1548, ...,    0,    0,    0],\n",
       "        [8214,  118,   34, ...,    0,    0,    0]])>,\n",
       " <tf.Tensor: shape=(64, 40), dtype=int64, numpy=\n",
       " array([[8087,   98,   25, ...,    0,    0,    0],\n",
       "        [8087,   12,   20, ...,    0,    0,    0],\n",
       "        [8087,   12, 5453, ...,    0,    0,    0],\n",
       "        ...,\n",
       "        [8087,   18, 2059, ...,    0,    0,    0],\n",
       "        [8087,   16, 1436, ...,    0,    0,    0],\n",
       "        [8087,   15,   57, ...,    0,    0,    0]])>)"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pt_batch, en_batch = next(iter(val_dataset))\n",
    "pt_batch, en_batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nBQuibYA4n0n"
   },
   "source": [
    "## Positional encoding\n",
    "\n",
    "Since this model doesn't contain any recurrence or convolution, positional encoding is added to give the model some information about the relative position of the words in the sentence. \n",
    "\n",
    "The positional encoding vector is added to the embedding vector. Embeddings represent a token in a d-dimensional space where tokens with similar meaning will be closer to each other. But the embeddings do not encode the relative position of words in a sentence. So after adding the positional encoding, words will be closer to each other based on the *similarity of their meaning and their position in the sentence*, in the d-dimensional space.\n",
    "\n",
    "See the notebook on [positional encoding](https://github.com/tensorflow/examples/blob/master/community/en/position_encoding.ipynb) to learn more about it. The formula for calculating the positional encoding is as follows:\n",
    "\n",
    "$$\\Large{PE_{(pos, 2i)} = sin(pos / 10000^{2i / d_{model}})} $$\n",
    "$$\\Large{PE_{(pos, 2i+1)} = cos(pos / 10000^{2i / d_{model}})} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:24.950742Z",
     "iopub.status.busy": "2020-11-02T02:26:24.949856Z",
     "iopub.status.idle": "2020-11-02T02:26:24.951933Z",
     "shell.execute_reply": "2020-11-02T02:26:24.952383Z"
    },
    "id": "WhIOZjMNKujn"
   },
   "outputs": [],
   "source": [
    "def get_angles(pos, i, d_model):\n",
    "  angle_rates = 1 / np.power(10000, (2 * (i//2)) / np.float32(d_model))\n",
    "  return pos * angle_rates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:24.958548Z",
     "iopub.status.busy": "2020-11-02T02:26:24.957602Z",
     "iopub.status.idle": "2020-11-02T02:26:24.960152Z",
     "shell.execute_reply": "2020-11-02T02:26:24.959567Z"
    },
    "id": "1Rz82wEs5biZ"
   },
   "outputs": [],
   "source": [
    "def positional_encoding(position, d_model):\n",
    "  angle_rads = get_angles(np.arange(position)[:, np.newaxis],\n",
    "                          np.arange(d_model)[np.newaxis, :],\n",
    "                          d_model)\n",
    "  \n",
    "  # apply sin to even indices in the array; 2i\n",
    "  angle_rads[:, 0::2] = np.sin(angle_rads[:, 0::2])\n",
    "  \n",
    "  # apply cos to odd indices in the array; 2i+1\n",
    "  angle_rads[:, 1::2] = np.cos(angle_rads[:, 1::2])\n",
    "    \n",
    "  pos_encoding = angle_rads[np.newaxis, ...]\n",
    "    \n",
    "  return tf.cast(pos_encoding, dtype=tf.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:24.964937Z",
     "iopub.status.busy": "2020-11-02T02:26:24.964299Z",
     "iopub.status.idle": "2020-11-02T02:26:25.185963Z",
     "shell.execute_reply": "2020-11-02T02:26:25.186470Z"
    },
    "id": "1kLCla68EloE"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 50, 512)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEKCAYAAAD+XoUoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd5xU1fmHn/femdneKywsvSpSRBCxYe8aE1s0lhhNYokaE2OKJjHFmKIxicaoMZpmjwb8YbCAoiDFQkfaUndh2b47u9PunfP7494ZZpeFHWAXWTzP53Oc2++ZdThz5/ue9/uKUgqNRqPRfD4wPusOaDQajebgoQd9jUaj+RyhB32NRqP5HKEHfY1Go/kcoQd9jUaj+RyhB32NRqP5HNGjg76IbBKR5SKyREQ+dLfli8ibIrLOfc3ryT5oNBrNZ4WIPCUiO0VkxR72i4j8QUTWi8gyEZmQsO8ad5xcJyLXdFefDsaT/jSl1Dil1ER3/W7gbaXUMOBtd12j0WgOR54GztrL/rOBYW67EfgzOA/HwI+BycAk4Mfd9YD8Wcg7FwLPuMvPABd9Bn3QaDSaHkcpNReo38shFwJ/Vw4LgFwR6QOcCbyplKpXSjUAb7L3L4+k8XTHRfaCAt4QEQX8RSn1OFCilNru7t8BlHR2oojciPPNR0Z62tFtKp1xowawZPVmxo0sZ+snKxlwxCCWbGkiIy+Hfi3baWgMUTr+CJat2443LZ0jCk2UbbGmxUNbQx1FfUsoU01Ubawh1RAKRw5kY6vQsLMO0+ujsCiXmuo6VDRKVkE+QwrSCG2toL4ugK0gN91LZnkJbd5sNlW3UJKfTkEKWDU7aN3ZQosVBSDdNMjITSGlqAA7LYfKT1biEyEjxSQ1Nw1vXh7R1CxawjYNrWHaAhZWKIgdCUPUZsKQIqKtzYRb2oi0hgmHo4SiClspooAApoBHhIKyXKxACCtoYYdswtEoVpT4sbF8awPIPmIkYVsRtqKELZuwFSVqK6dFo6io7TZneWypD/F4EdMLhokyTOcVIarAVqCU0681FdsRERBBcF8NY9e6YSBiICJ4U0yUApRCuddw1kE5/yGWKa5UlMysVEQEAQwR3NsgCIbg7HO3VVXWxd+1ci7Q4RO5a33IoD5I7PPm/kfctfbrDqvXb0vmMw/AkcP6d7pdZPdty9dsSfq6AEeNLO/82p1sW/pp8tcet4frdsaSfbiuc+0B+3Dtzclfd1T76y5ZvRkVqKtVShUlfZEOGNn9FFYwqWNVoG4lkHjw4+44lyxlwNaE9W3utj1tP2B6etA/XilVKSLFwJsi8mniTqWUcr8QdsP9wz0OcPRRR6jl5mTmzXuUnCk3Mff9R/hOxigeeekJCm6ZxXFfOof7Z/+MV2as43vz5tH3gvspPeJoFl6fgd1Ux0nvFvDRi//i8nvv4P7wa/z0K08wPNPHtS89wVcWpfLKI0+TWTqQa288lz8/9G8iwVZOuPpSXrrySDbe9hX+9c/lNEWiXDyqlOP++B2Wlp3C1Q+9x51XjOXqwSa1j/2ChX+ay5yaNgAm5KQy+fxhDLnxGlqOPJsfZo+mb4qHKQNzGHHRUfT90pdoHXkK725u4rkPt7JsWTU7N6zFv2MTVtDPohdvpG3hG1S+u4SqxZVs3tLMprYI9WGbcFRhCuR4TQp9JtfcdSG1yzZQt6aWhopGKv1hakI2DRGbgB3Fdv+6PkM4+6U32NIUZFNtK5vrWqmqa6O1OURbU4hgW5hQSyPhtiasgB8r2Mq875RjFpRi5hVDRi7RlCyiablEzBTaIlFaI1EClqI5ZHHK5T/B9PowPD4MjxfD48NMScP0+OLLhseHx+el37ACrHAUK2JjRWxsK4oViRK1oth2FNuKErWj2JZF1Aoz5eQR+DwGPo/pvJoGKR7D3da+3fPjp1FR2/kMuV9ezrLzGnVfAR792w8wBEwRDBFMw/lS6bguAgbC0Rfc1e5ae2PGGw8Buwb52E9qcTcYCSP0gGm3dnm9RN5+90+dDvBGJxuLT7gl6eu++/4j7dY7u0eM/Kk3J31dgPfnPZr0sbnH3ZT0sfM6XDdnyk1Elvwt+W+NzrCCeEZckNShkSV/CyZI172CHpV3lFKV7utO4BUcbara/fmC+7qzJ/ug0Wg0+4QIYphJtW6gEkj8WdjP3ban7QdMjw36IpIhIlmxZeAMYAUwHYhFoq8B/ttTfdBoNJp9R9xfrF23bmA6cLU7i+dYoMmVv2cBZ4hInhvAPcPddsD0pLxTArzi/pz1AP9WSv1PRBYDL4jI9cBm4NIe7INGo9HsG+6TfvdcSp4FTgYKRWQbzowcL4BS6jFgJnAOsB5oA65z99WLyM+Axe6l7lNK7S0gnDQ9NugrpSqAsZ1srwNO3ZdrrdoZZsp3r+adkZOZcuvDLDjmRC4dU8yl851v2unn53HbTav50S/O5ew/LyTYVMvf7ziBOWefQeTl11g281eUTzmPB84czNsjniVgK07/+hQWpo7m3ddfRkVtRp94DC/NXENbXRUDjjufu04bTvTNJ1k6fS01IZsJuamMunQi1rhz+ef/1jF+fB/OGFpAdNG/2fTmSpY3hQhHFf3TvAwemkfZieNg2GRW1QTI9BgMyvBSPKaYwolHoMrHsKU5zEdbG9m4rZnm2gaCDdVYQT8A4YqVNK7dSuPGBhq3+6kJ2fitKOGoI9D7DCHDNMj3mbRW1tK2009bbYCmoIXfitJqO8fG9HxTnNYQiNDQFqauNUydP0woYBEOWIRDFpFgG3Y4gB0KELXCqKiNkZ6FkZqB+NKIelJRvnSUJ4WwpZyAcFQRtqOErChixn7yGohhYnh9GO5PYMPjQwwT0+NBRLBj2r0dRUWdQLKKKqLKeVVKEY2quHZuGoJpGM6riLveSUuIkqpodO+fT9u9dpJ6/q7rdq3n7wudBXb3h870fDmAi3dTt3olAojZPYO+UuqKLvYroNMAiVLqKeCpbulIAj0dyNVoNJrehQhGNz3pH4roQV+j0Wg60F3yzqGIHvQ1Go0mkW7U9A9F9KCv0Wg0CQiC4fF+1t3oMXqFy2aopZG3TwnyxrZmZp9r8srqGo5dOJfXHnmSn//set468csck5dG7bW/ZOFzLzDp0ksYPe8RXlldw+2PfICybe65/hhqH7idmZXNnN0/mz7f/infe3EZtWsXU3zEVO654AgqP55DRlF/zjltKMemN7Ly8ddY3BAkx2swYUoZRRdfyVsbG3l38VYun9ifstaNVL4+m7UraqgOWaSZwuhsH/2OG0jG5FPYYeQyb3M9JSke+g/MpXTiUFLHTKHBV8CS7S18vLmB+u0ttO7cQri1CQDTl0Zw0wYa11fRvK2ZmpBNs+UkWoETxM30GOR43UDujjr81a201QdoikTjAV87IfPUFMFnCPXBCDubQ9T5QwQDEcKBCOGQ5SRIhQJYYSeIG7Ui2JEwRkY2RkYWUV8aUW8aypNCROEEcKMKy4a2iE3QisaDtrEgriQGcc1YMFcwPQYqihOwjSpsO+oGbVU8OUu5QVwVtVG2HQ/U+kwnASuWmNUxkGuItAu07i0xCzoPfu6JfYmJxu6XTGLW/vB5DrIeFA7uPP2Djn7S12g0mg701gE9GfSgr9FoNImIdNuUzUMRPehrNBpNAsLh/aTfKzT9/uV9+OVxt3DvHy/jwYnX893vnsQxP3yTPuNP4/rqV3m1ooEvT/8pF/98NukFfXntm5N57uZ/MjwzhY3vT+eocy/gqvwaZjz8Hvk+kxPvv4RnNipWvv0evowcTj/rSE5Or8UOBxg8eQq3nTCQxmf/xIJ52/BbUY7NT2PUV6ZRlX8kT83fRNXqT5k2MIfA3FfY+NYG1vrD2AoGpvvoP6GUPtOOxRpwNB9VtTBn9U6GZnrpM6EPOePGEelzBOsbgny4uYGqrU207NxO2N9A1AojhokvI4eGtVtp2txMfV0gnpiVaJwWS8xKz0+jZbuftroA9WGbpohN0NXbExOzfIaQZhrU+8PUt4ZpjCVmhWwiIQsr4McOB4hGXD0/lpyVkYXyZaC86eBNJepJIRhLzLIVQStK0IrSFrHbGa2JYWIkJGUZHh+GIZimgWka8cQs24qiosS1/Li2H9P0bUfXjxmtmYbgSdDw2+n6Ipiu2N3diVkSv27yiVl70vM7O+ZA0YlZ3YwYmB5fUq03op/0NRqNJhE5vJ/09aCv0Wg0CQh6nr5Go9F8rjicB/1eoelnN1RSmurh0eFfBWDZNQ+wbs4rzP7VOTx85aNcfWI5D1sT2Dx/Bt++8xKqbr+SxQ1Brrj3LLL7DefpGybxyc3fZWlTkAtPGUjrOXfwu2eX4q/exMBjp/Gj04ay/c+/pWDoBL55/ijKKz9g6ZPvs7olRP80L0d+YRS+067m1TU1rPxkO83b1pK27j02/PcDlm1poj5sk+8zGVmaQf+TR+MbP431zYp31tVSWdFA3zHFlE4ejWf0sVSGTD7e3szSTfXUV/sJNOyIz9H3pGWSklNI4/pqmrc1syMYm6O/y2gt0+Po+TmpHjJK0mmtbqWlKURTJEowqgjYu4zZYuf4DCHVkPgc/VDAIhSIEAlZRIJB7LAzR9925+nHtHRJy0L50lDeFKLeNEJWNK7nh21FW8SmLRIlZEfjRmsx47W4nu/O2TdMA8NjIIYQtZyCKUopp2BKB6O1mOFbrHVptObO0TcMiev5Xc3Rj7EnPb8jyc6t70r3j13nUNXzP2sOia7refoajUbzeULLOxqNRvO5QUQwvL1zZk4y6EFfo9FoEtGGaxqNRvP5Qg/6nzE7qv1ct+NDsi/4Lf5Fj1N4+5+ZdsP1tN56Ga12lAmvv865F/2aISdfxN19qvjB00v44sgCgl/9BZcPqqB87mP89K2NHJOXyvgHf8K1M1azaf4scspHccsXj6Rs9f8x/YkFjPvp9Vx1ZCEbbr+D9ysaMEU4bkQ+A666lCWBLJ599xNq1nxE1Aqzc8YrVMzdwtZABJ8hDM/0UT61H3knnExj7hDeX1XDh2tqaKisos/EgWSMm0IgfzArNjUxf10ttZUttNZsIdTS4CRCeXz40rNJLyij8aMmqlvCNER2BXFNgUyPQbYbyM0oySCzJIP6dQ3Uh50ErsTqWtA+iJtmGtS3hmhpDRMKRggHLCIha5fRmpuYFU0IoCqfY7KmvOlYGITtqNucIG7IjhKybKdyVoLBmtkhiGt6DEyP4QRKPUY8Ecu2VNx4LZaYlWi01i6Q2yExq12ClpuYFauctbcgbiwxCzoP2MZITMza3yBudxutHQx6QRcPCkZv+J+1n/SK2TsajUZzsBARxEiuJXm9s0RkjYisF5G7O9n/kIgscdtaEWlM2Gcn7JveHe+vVzzpazQazcHENLvneVhETOAR4HRgG7BYRKYrpVbFjlFK3ZFw/K3A+IRLBJRS47qlMy76SV+j0WgSEbrzSX8SsF4pVaGUCgPPARfu5fgrgGe74V3skV4x6JcUpDH+NyspO/pUTp0lmClpvH4aPPLcKr7zyBWc9Nv5WMFWXv3+yfzvzG/hM4RTXvw1lz22kAdPKWbmLc8QjirO+e6pvCUjePOVeQCMPX0K143MYNn9TzC3to2fnTsa+78Pseg/q9kRtJiQm8qY646nbex5PLFgMxs/WUNbXRVpeaWsn7GUpU0hAraib6qHYaMK6X/aRBg5lU92tPLGyh1Ub2nEX72JoinjiQ4az4aGEIs2N7BhUyNN1bUEG6qxgn4AfBk5pOWVkpWfSeN2PzuCVjuNPs004kZrOfmpZBank1GaS1PQoikSpdWO7ma0lmi2lukR6mJGawGLcMgiEmzDDgewQ4F4QlQ0sisxKupNR/nSiXpTCcWSsqKKsB0l5BqtxV5jGr5htE/OMj0eTNNJynKSs5wCKlHb1fLdBK1YYlaing+OTm6K7LFwSiypynATtPZGop4Pe07Miun5iexr0tDe/mElXutA/gEebkZrh0RiFjGXzW4b9MuArQnr29xtu99XZAAwCJidsDlVRD4UkQUictF+vqV2aHlHo9Fo2tH1A0QChSLyYcL640qpx/fzxpcDLymlEp9OBiilKkVkMDBbRJYrpTbs5/UBPehrNBpNe1x5J0lqlVIT97K/EuifsN7P3dYZlwM3J25QSlW6rxUi8g6O3n9Ag36vkHc0Go3mYNKN8s5iYJiIDBIRH87AvtssHBEZCeQBHyRsyxORFHe5EJgKrOp47r7SKwb9QMkA1r/7GssfPIf5f3+G6X+8gScnX88XRxbw+sRv8skrz3LFLVeR8cidzNjWzFdvOY7H/UNY8t//sP62G3hrZytfOLoPObf/jrv//hH1FUspn3Q6D3/xKBqf+BlvvbsFgPHhtXz0+5ksbghSmuph4lmDyf3i1/jPp7W898EWGjatwPD4yB86gZVr6tkRtMjxGozJT2PAqSNJn3IOmyMZvL22hg3r62jcupZgUw2+o05kB9ks3NbEB+tqqdvhzNGPG62lOkZrGYWl5BalUxmwaLaiuxVDz/eZFKZ4yCjOILNvFpllRdSHbVrt6G5Ga6Y4Wn6qYZDpcVpba5hwIOKarYWxAv7diqHH9HzANVtL21U0pZ3R2q4WiNi7jNU8Pqd53VdXzzdNI15IJT5P325vvJZospbYErV8Xwdt30iYo29K8kZrKmp3abR2IHP0d11jz3P0u1vP780cKno+OH0xPZJU6wqllAXcAswCVgMvKKVWish9InJBwqGXA88ppVTCtlHAhyKyFJgD/Cpx1s/+ouUdjUaj6UB3OpUqpWYCMztsu7fD+k86OW8+MKbbOuKiB32NRqNJQNzZYIcretDXaDSaDuxDILfXoQd9jUaj6cDhPOj3ikDups07+MEv7+CdkZOZctXVFPzyBja1RThpwSxu+dE/KJ9yHo9NtPjLA7O5cEAOafc8xs8eeh1fRg7PvrCKsTmpHPfYvdw+41PWzH6d7H7Duenyoxi+ZTYLfvc2m9oiTC1Io+Kh3/LOsp0AnDgsn2E3fJlV0pen3t7AjpUfYYcDZPcbzpCjSlnrD2EKDM/0MWjaAIpPO5Xm4tG8u6me91ZUU7NxG221VaioTaB4BMuqW3l/XQ07tzXTvH0TwaZaolYYw+MjJSuP9IIycoszGNgni1rXQM1WToJVmilkewyKUkwyStLJ6ptJZlkRGWVFNEU6C+I656S6AeBMj5CZ4iHYFiHkGq3Fgrh2KIAdDmJ3qFYFoLzpRMRDyIoSdKtmBSNR2iK7ErOCdpRA2HYTsXY3WosFb02PgWE6CVq2pZwA7h6M1oB2/ejMaC1eTUuIJ2bFfpJ3FlRNTMzaW3WrRKO1jtv2xL4EcXsyYHmwKmbtwxz23ok47zGZ1hvRT/oajUaTgOA8nByu6EFfo9FoEpHD21pZD/oajUbTgd5cXL4resVvGG96Fjetfpw3tjUz+2x46ImPufuxK5n6+48JNdXy+k9OZ+bx12GKcMbMh7nojx9Qu3YxJ15+Pn4rysU/OJ0308Yz/fl3UVGbo88+gW8ekcnSn/6Rt3a20j/Ny+TrjuH9Z5dT5Rqtjb3xJAITv8DDcyuo+PhTWmu2kpZXSv8jR/OVKQMI2Ir+aV5GjSlmwNmTYcwpfLi9ldeWbWf7xnr81Zuwgn4Mj4/1DSHmVdSxtqKBhsodnRqtZRfmUFSSyVH9c3czWsv2mHGjtaw+mWSU5pJZVoSnqMxNzGpvtBYrnhIzWsvxmqRkpxAOWE5iVoLRmh0O7ma0FiNmtBZMMFqLJWTFjNYCYafF9fwORmuGx4gbrcUKqezJaC2xD3HTtw7JWfEkLdOI6/hew3C0/Q7/UGOJWXvS87syWjOkezX4Q9Vo7bPmUOu6Y7iWXOuN9Hi3RcQUkU9E5DV3fZCILHQLCjzvpiZrNBrNoUFsckASrTdyML6rbsNJP47xAPCQUmoo0ABcfxD6oNFoNEkiGKaRVOuN9GivRaQfcC7wpLsuwCnAS+4hzwDd4hGt0Wg03YHoJ/0D4vfAXUDUXS8AGl0TIth7QYEb3eIBH5akBPnx7S/z40ev4MFJN3LlsWX8feR1LH31OW77/vXIfdfz2vYWvn7vmfxqe1+W/PclBp94IS9cPZ7Lpw3E880H+O6Ti6mvWMrgqWfxyCVHUfPwPcycsxlT4LSp/eh/07dZ3BCgf5qXY78wguxLbuLZFTt5f95mGjatwPSlUTRyImccW87ZQ/PJ95mMK81k0FljSD3ufNYHU5m5qpoNa+to3PIpwaYaxDBJyyvhg62NLFhXS21Vs1sMvR5wjNZS80rIKu5DfkkGR5blMKIoczejtaIUk6J0LxnFGWT1yyGrvISU0lI8peW7zdGPafkZpmOyluM18aV7Scn2EQpECAcCWAE/kaDfNVoL72a0FsOZn++YrIUsRUtod6M1f9CiLWzv1WjN9LivrsafrNFarEh7MkZrhtHecG1PRmuJdGW0Ftvccd5+IgdqtNYdWvzB1PO7e276oabnx+jOGrmHGj026IvIecBOpdRH+3O+UupxpdREpdTEwoKCbu6dRqPRdI4InScDdtJ6Iz05ZXMqcIGInAOkAtnAw0CuiHjcp/29FRTQaDSaz4TeOqAnQ4896Sulvq+U6qeUGojjFT1bKXUlji/0l9zDrgH+21N90Gg0mn1FSO4pv7d+MXwWyVnfA54TkZ8DnwB//Qz6oNFoNJ0iAj5tw3BgKKXeAd5xlyuASftyfu2KNVwyYTyPDLkWHy8x7H9vcM75P2bMeZdyT9rH3P3YYq48toz6a+/nwev/SGbpQJ6643gqv3M1E5/8Pef/awnr332NgqET+PG1R9Nv8T958Y9zqQpanN8vm3Hfv475dj98hnDyhFKG3vwN5vmzeGrWx2xf/gF2OEDh8GMYO7GMqyb0o7B6CUdmpzDkjCEUnXEONblDeXNFNfOX76B240ba6hyjtZSsfDJLBvHWqmp2bG6keXsFwaZaJzjpSyM1p5CMonLySjIZ0T+XMWU5DM1PZ6ZrtJbpMcjzmhSlmGT1zSS7n1MtK6NvMZ6Scsgp3i2I6zN2Ga3leA3SfCapeamk5aUSCkR2VcuKhB2jtYgTzO0skOtUyooSsnavltWakJgViNjtgrimxzFYixmtiUg8Scs0jd2M1qJWGGW3D+LCLtO1dklZHiMevPUmJGglGmAlBnH3x2gt8QFuf4zW4ud2YrTW3UHcZO/fPdf7nARxxTH5O1zRNgwajUaTgHB4a/p60NdoNJpEpPfq9clw+ApXGo1Gsx84T/pGUi2p64mcJSJrXOuZuzvZf62I1IjIErd9LWHfNSKyzm3XdMf76xVP+raCAf97g7PPvRv/oscZcudrZBT1Z/73pvBY30mMykph8uuvMOae2bTVVXHXfbcy7qO/8eu/fkz2ZZnMf+lFfBk5XPrlk/hi9k7evftvzKsLMDYnlWO/dyY14y7m3r9/zJ3FmYy/40K29p/KAy8tZ+OHHxNsqiGrzxAGTxjJ144byHCjjtrpLzDi2DL6n38q1uhTmLuugekfVbK9Yif+6k3Y4QCe1EwySwZSWF5CxYZ6mqoqaaurwg4HEMPEl5FDRlE5uUUZlPXN4qj+OYwszKA0w/lfkqjn5xRnkN0vi+zyYrLKSzBLyjGKy7GzSnYzWktzk7IyPQbZXpM0V89PzUvFCvixwwH3tfPCKYmELOUYrlnRBD0/SshyCqfEErNiRVQMj29X0ZSY2ZopcX3fMATTI9hWlKgdxbaseOGUzhKzYrQzXBPBa+zS8WNGa6bs/pN8b3q+Eytob7S2p8IpHXX+feWzKJxyqOv5hzrd9aQvIibwCHA6TjLqYhGZrpRa1eHQ55VSt3Q4Nx/4MTARUMBH7rkNB9In/aSv0Wg0CRiyKwO8q5YEk4D1SqkKpVQYeA64MMmunAm8qZSqdwf6N4Gz9utNJaAHfY1Go+mAM0Os6wYUxuxi3HZjh0uVAVsT1vdkPfNFEVkmIi+JSP99PHef6BXyjkaj0RwspBOpcC/UKqUmHuAtZwDPKqVCIvJ1HCPKUw7wmnukVzzplx4xmClff4qyo0/l1FlC9fK5zHjoauYddzpVwQjXvnYfZz79KRvfn87kyy/lx8P8vHDjX2mK2Pzu0bcJNFQz/vyzeeDMway46/vMXF1LaaqH0686isxrf8QvZm9g9XufcPStJ8I5t/D79zax7L3VNG9bS2pOEf2OGs9XTh7MtPJMwrP/xbpXP2bYxVMwJp3P4u1tvLqkki1ramnasopQSz2Gx0d6YV9y+5UzeEg+dZW1+Ks3EWltApzCKekFfckpKaS4LJsJA/IYXZRJv2wfmaF6txC6o+cX5KWS2TeTrH55ZJWX4CsbgLfvQKKZhYR8WUCini9kmM78/ByvQUqOj1RXz0/Ny8AK+okEdhmtdVY4JYYYJkHbKYjuD1v43fn4bREbf8japedHbAJhC8Prw/R4HMvZ2Jx8j7Sbr2+YjmVtYuGUvRmtxfT+uOFawrz8jkZrsXn6nRVO2RPJFE7ZV6O1xOt0PP9gGa31hoknh3qIoBszciuB/gnru1nPKKXqlFIhd/VJ4Ohkz90fesWgr9FoNAeLWHJWMi0JFgPD3OJRPhxLmunt7yd9ElYvYFf9kVnAGSKSJyJ5wBnutgNCyzsajUaTgCDdZsOglLJE5BacwdoEnlJKrRSR+4APlVLTgW+JyAWABdQD17rn1ovIz3C+OADuU0rVH2if9KCv0Wg0Ceyjpt8lSqmZwMwO2+5NWP4+8P09nPsU8FS3dQY96Gs0Gk07Dncbhl6h6a+qiRBqqWf5g+cw/+/PcNd9t5J7/w28sHwn3/75ufyqbSwf/OvfDD7xQv73jUm8c9FNLKgPcMVpg6j5dAHDpl3I3645mtoHbue/M9ZhK8U5J5Uz6Hv38MTyembOXEl9xVIKr/8uTy3Zzsy31lO7djGmL43i0ZM5/6RBfGFkITL/BdY+P5dlK2rIOOWLrLeyeWlpFctX7KR+4yoCDdXxalm5/YfTd1Ae00YV01K1vl21rLSCvmSX9qOgTyYTBuQxpk82g3JTyTdCeOo3k+MmZRWle8nul0VOeS7ZA/uQ2r8/3r4DsbNLsTOLaBIaYMMAACAASURBVAg6wcTOqmWlZ6eQmusmZuWmkZKbRSToJGfFjNb2FsQVw+y0WlZrePcgbrxylplotLaHJC2P0a5aVmIwubMgbqLhWmK1rFiClje23Q3odkZniVm7vee9VMsyhHa2a10FcROvGWNPQdz9HVt0taweRBdR0Wg0ms8PMT/9wxU96Gs0Gk0H9KCv0Wg0nxOMw7yISq94Z8HmRt548nbeGTmZKVddzV31L/H7v3zINy4ewfIv3Mvv7n+G/MFj+e8Pp7Huq1/kheU7uWhwHhOe+jOlY6fx+69PpuTNh5nx8HtUBS3OGZbP+J/dzuv+Yv784gqql8/Fm5HD67WpPDljNVVL56KiNoXDj+H44wdyzdH9yN80j00vzGDF+1tZ6w9RmTWE6aurmbekiup1a2it2RovnJLdbwSlA/I45YgSpvTLI9BQHS+ckpZXQlbJAArLshkzMJ+x/XIYUZhOn3QDT90mwhUrKfSZlKZ6HD2/XzbZg/qQUV6Gt89AVF5folnFNISiNAbteOGUXXq+QUaap53RWlpBDqkF2dihAFErstfCKTE9XwwzruO3hHcVTvEHLcdsLWThD0bihmsxvd7jNXcZrCUUTolr/YbssXBKRz0/hs9j4DWMPRZOMY32RVS6MlqLv9e9FE5J1PP3dH6y6MIpuzjk9XzQmr5Go9F8nhDivjqHJXrQ12g0mg4czlbSetDXaDSaBAT2OP33cKBXDPr9+pdi3Hwpb2xrZvbZ8INxT3HB0Hzyn3iZs274K2IYPHLPRWQ8ciePvbiaY/PTOO2l+/nJ0ig//OaJnLhzDv93x7MsbQpyWnEGxz9wDSv7nshPn1zEpkWzEcOk/JhpPDB9FZsWzSfS2kT+4LGMnjKMW08YzODWdVQ+9yxrZqxlRXOIgK14fV0dMxZuZfvazfh3bCJqhfFm5JBdNpzSgUUcN7qY4wfmM6IghagVxvD4SM0pJLN0EPl9shhWnsuEAbkcUZxJWaYXb916rI0raF2/ztHzy7LIHZhD9qBSsgf2wdN3EFLYDyurhCbLoCFos70lFDdZi+n5OameuMlaemE6qa6en1qQ48zR30vhlEQ9XwwTf9jGH7Z2Ga0FnTn6LSErPj8/ELaxIraj5Scaq8U0/IQ5+x7Xgzym50e7KOISL4yeYK5miOA1pV3hlMTlZPV82F3P78x8DZxBwBDZJz2/swfFjnp+d8/RP9T1/F6D+1k7XOkVg75Go9EcLATwJlkKsTeiB32NRqNJQMs7Go1G83nCnRJ8uKIHfY1Go0kgFsM5XOkVwlVu03aeeG0dP370Ch6cdCOjslI4+eM5TPvBLJqrNvDDe67l9KVP8JcHZtM31ctlf/smf7dH85fHXuOGwmre/doDzKpuZUJuKqf97EJ2Tv0qtz23hLXvvoMV8FM6dhpXnTeST+d+QFtdFVl9hjDs2KP49qnDGOupofbFv7HqhSUsbgjQFIlSlGLy3Aeb2fppJY1bV2MF/XhSM8nuM4SSwWVMGF3MtGGFHFmcTtrONYhhOkHckkEUluUzeEAukwfnM7Ykm/5ZXlIbt2BvWU1g/ac0rttKfkkGuQOyyRlYQs6QMrz9hmKWDsLO6UOrpNIQsqlqCVHZEiQtIYib53OSstIL00kvSCO1ICsexPXk5mO71bJiAdTOiAVxTa+PlpBTMavFNVmLG62F7fhrOGxjW9HdjdViCVkep1qWJ6GYdMekrD0ZrcXaLnM1I14lKzFRa1flrF3vI1mTtcTlWBC3XXD3wD66e/wH1v1B1+6+XvcPer1pHHWM/bpuvRH9pK/RaDQJiPtAcbiiB32NRqNJ4HCXd/Sgr9FoNB3ordJNMvSK3zDbd7TwvTtP4JEh1wJw1dKXmPTzeWxb/D+uuuOrfMuez59u/AemCDc8+CXeGX4Z9z70Jk1bVvPBNXfy6po6hmf6OP+uU7Gv+BG3vLyc5W/OJdCwg+LRU7ng7BHcPLkfLds3kFHUn6HHTuL2s0Ywrcii5dW/svKfC1lU1UJNyCbHazAhN5WNK7bTuGkFkdYmTF8amaUDKR4ymDGjijljZDHj+2SS07SR8Ip5pOYUkVkyiIL+xfQfkMtxwwoZ3yebgbk+MlqrUVtXE1y7goa1W2lYV0Pe4FxyBhWTM7QMX7/BePoOxs4ppc2TSV3AZkdLmO0tIbY1BMj2GOT7TPJ9pmOuVphGemEaaYVZpBbkkF6chzcvDyO7YK96fmJSlun1xZOz2iVlBS38IYuWYCSu51sRGysSxfAYeLztTdc8XiNeWCWm56d4jF2Ga13o+TES9XyPaSRo+Lv0fK+5yy8lGT0/fm3pWs83RPZLj+7uwil7vE8vGKB604OzsMvAr6uW1PVEzhKRNSKyXkTu7mT/t0VklYgsE5G3RWRAwj5bRJa4bXrHc/cH/aSv0Wg0iXRjjVwRMYFHgNOBbcBiEZmulFqVcNgnwESlVJuIfBP4NXCZuy+glBrXLZ1x6RVP+hqNRnOwcDT95FoSTALWK6UqlFJh4DngwsQDlFJzlFJt7uoCoF83vp3d0IO+RqPRJBCzYUimAYUi8mFCu7HD5cqArQnr29xte+J64PWE9VT3ugtE5KLueH+9Qt4pzkvl/S/fz8+/eT/+RY9z/N93sHrWS5z5zRt4dMROnjjplzREbG7/6dmsO+u7fPO+N9i5ah5DTr6I5/9wG31TvVx80xRybv8dX395BR/MeBd/9SYKhx/DGeeO5funDCZl9pOk5ZUyePIUbjp3JOcNSCX48u9Z/vRcPljfQFXQItPj6PnDTh1IQ8VSgk01GB6fq+cPZ+TIQs46ooRj+mZR2FaFtWIetQs+JqNoInllpfQtz2XqsEIm9MlhcG4K2cFaZNsqguuXUf/pZurX7KBhYyNDzhpB3vD+pA4Ygrd8OFZOXwIpedS2Wezwh6lsDrKloY3NdW0c53Xm6KfnO1p+emE6aQWZpBXlOXp+bi5GbjFmXlGXhdANjw8xY8te/GHLLZayS88PhJ0iKoGgFdfzrYjtmKolzM83TInr+Wk+M67n+zxmUvPzIWa4Fu1Uz/cmLDtF0WWfTNFU1G5XCB32rOfvD8nq+QecB9ADWvnhPHMlKQT2YcZmrVJqYrfcVuQqYCJwUsLmAUqpShEZDMwWkeVKqQ0Hcp8ee9IXkVQRWSQiS0VkpYj81N0+SEQWukGN50XE11N90Gg0mn0lNmWzmwK5lUD/hPV+7rb29xQ5DfghcIFSKhTbrpSqdF8rgHeA8fv9xlx6Ut4JAacopcYC44CzRORY4AHgIaXUUKAB5+eMRqPRHCKIa+fddUuCxcAw92HXB1wOtJuFIyLjgb/gDPg7E7bniUiKu1wITAUSA8D7RY8N+srB76563aaAU4CX3O3PAN2iU2k0Gk130J1P+kopC7gFmAWsBl5QSq0UkftE5AL3sN8AmcCLHaZmjgI+FJGlwBzgVx1m/ewXParpu9OVPgKG4kxb2gA0un8I2EtQww2I3AjQJz21J7up0Wg0cRwbhu6LayilZgIzO2y7N2H5tD2cNx8Y020dcenR2TtKKdudY9oPZ+rSyH0493Gl1ESl1MSMQcP5xrd+R9nRp3LqLOGjF//F1Guu5b+nmvzzlNtY6w9x850nUX/t/VzxqzlUfTSLAcedz+O3TiXfZ3LZV8fT554/cOf/rWHWy3Np3raW/MFjOfncifz4jGHkfvAvPv71iww69nhuPG8Ul4/Mxfq/R1n+1znMX1HD1kCETI/B2JwURp48gMEXTyPQsCMhiDuSEaOLuHBsX47rn0NJuBpr+VxqP1hM1cIK8vv3p+9AJ4h7dFkOQ/NTyY00INtWEVr7CfUrNtKwZjsNFY3U1AbIG15O6kAniGvn9CWUXkBdwGJna5itTQG2NAbYXNfGtvo28n0mmXmppBemkVGSQUZxFmlFeaQV5OAryMfMK8bMKcDIyidqhXf7O3cM4poeH4bHi+Hx4Q9ZNLVF2gVxW4IWoYSkLCtiE7Wi8cpZHp+JYUo8QSsxKcvnMXdVzkoyiAvEq2btKYjrjVXP6uTTvKeKXLAriBuroBX/m7ivsSe5A4lrfpZB3P25/uc+iOsiklzrjRyU2TtKqUYRmQNMAXJFxOM+7Xca1NBoNJrPEuOAv5IPXXpy9k6RiOS6y2k4GWmrcbSpL7mHXQP8t6f6oNFoNPuKoJ/095c+wDOurm/gBDBeE5FVwHMi8nOc9OO/9mAfNBqNZp/pDX5G+0uPDfpKqWV0MqfUnW86aV+uVbFpB+VfnsryB88hZ8pNTLnqat66KJt/TbyCpU1BvnX78bTd/jAX/3w2Wz54jfIp5/GXO45n4qrnKL12HP3vf5w7Z23mlWffoXHTCnIHHslJ50/h/nNHUfzh83x8/z95+6PtfP23o7lmTCH2jD+w5JFZzF9Szaa2CGmmMDYnhTEnlTPskml4T/gShudXZJYOpGjoaIaNLuKicWVMLc+lT6SG6Iq51M5bQNXCDVSvqKH0wlxOHFHElAF5jCpMp8BqwKhcRXjtJ9Qt20Dd6krq1jWwc2crO4IWaUOG4Rs4EjuvP6GMonhS1pamIFsaA1TUtLK5tpXmxiCZealkFGc4mr6r56cX55FSXOjo+XnFGDmFRNNydvu77k3PN7y+dnq+PxiJ6/nhkIUViRK1nGZFoqSmezvV89N8Zjs932ca+6Tnq6jtGq5Jl3p+Rz16b3p+jEQ935A96/n785NY6/m9lF78FJ8MSX2WReRiEVknIk0i0iwiLSLS3NOd02g0moONdO88/UOOZJ/0fw2cr5Ra3ZOd0Wg0mkMBLe9AtR7wNRrN54XDeMxPetD/UESeB17FsVcAQCn1nx7plUaj0XxG6HKJDtlAG3BGwjYFHJRB35OWyeqHz2XOyMlMufVh5nwhk78f7QRxb7vzRNru+CMX3vc2m+fPoHzKefz1zhOZtPLfTP/aY1y0YT63u0Hc+oql5A8ey0nnT+E3F4x2gri/eIa3FlVRFbS466giojP+wCd/nMl7n+yIB3En5KZy1CkDGXbZqXhPvJRPrZx4EHf0mBIuGlfGiQNy6WvVEF3+DjXvzady/nqqV9SwpiXMtFHFuwVxQ6sWdRrErQ3b8SBuMKOIGjeIu6khsFsQt605REZxRrukrD0FcTsGcvcWxDVT0jA9vi6DuLEELduKJh3E9XmMfQriAkkHcRM1Vh3E1RwIh/GYn9ygr5S6rqc7otFoNIcKh3OhkWRn7/QTkVdEZKfbXhaRHq3uotFoNJ8F4pZLTKb1RpL9Qvsbjh1oX7fNcLdpNBrNYYfOyIUipVTiIP+0iNzeEx3qjCP7ZfH6oInMrW1j9tnw5NFXsdYf5jv3nEHN1x7gC/e+QeXimQw+8UL+ceeJHPHBY7x089+ZVxfg9RkV/N/zb9O0ZTUFQydw5heO4xdnjyB/3tMs/sW/eXtJNTuCFgPTvURe+g2fPPIG7y3fZbI2ITeVMacNZOhlp2OeeBmrgxm8sLSKkmFHcORRJXxhfBnH98+hNLQda+kcat5fyLb569mxqpb1/gjVIYvzB+YzojCNgnCdUylr1SJql22gblUV9evrqakNUBmwaIjY+K0oVv4AQukF1LRZVDY7Jmsb651KWYl6fltLqJ2en9GnYJfJWkEpkl1INDXL0fRTsuJ/z2T0/JjhWmd6fsxkLabn23Y0aT0/xWPsk57vVLhKTs+PafHJ6Pmw90pZHfV82c9/4V3p+d39sNhLx6FDCkHLOwB1InKViJhuuwqo68mOaTQazWeFiCTVeiPJDvpfBS4FdgDbcQzTdHBXo9Ecfri/AJNpvZFkZ+9sBi7o8kCNRqPp5QhODYfDlb0O+iJyl1Lq1yLyR5x5+e1QSn2rx3qWQP3yNSww+vDjR6/gwUk30mzZfP+hL/LJmXdx3d2vUr1iLqPO/BLP33E8pf/9Ff/83n/4uDHItKJ0vvn31/BXb6J49FQuuvgY7jtjKKn/+xMLfvkys1fXUhOyGZLh48QpZSz+7UzeX1dPVdAix2twTF4aR5wzhEGXnYcx5WKWNpk8+8lW3ltSxYQJfbjILZpS1LqFyCezqX5vEVULNlL1aR3r/WGqQxYBWzG6KJ3cQDVsWU5g9cdxPb9ufQM76wPsCNpxPT8cVbSl5lPbalHZHGJLU5CNda1xPd/fGKS1OUTAHyLU0kxmn5wEPb8AI7cYM6/I0fPTclBpOdgpmbRFHK08Uc83vT532YvpS8Pw+jA9PmfZ46OxLUwgbBMIWu2Kplhhm6itsN25+nasiIqr5ScWTUnzmfjM2LrT9kXPB9rp+V5zl37fUc83jeT1fOhcz+8uLT/x+jG0nt976K3STTJ0Je/ErBc+xCl72LFpNBrNYYWTkdt98o6InCUia0RkvYjc3cn+FBF53t2/UEQGJuz7vrt9jYic2R3vb69P+kqpGe5im1LqxQ4dvaQ7OqDRaDSHGt31nO/WE3kEp4jUNmCxiEzvUOD8eqBBKTVURC4HHgAuE5HRwOXAEThT5d8SkeFKqc5/uiZJsoHc7ye5TaPRaHo5jlyYTEuCScB6pVSFUioMPAdc2OGYC4Fn3OWXgFPF0ZcuBJ5TSoWUUhuB9exjLZLO6ErTPxs4BygTkT8k7MoGrAO9uUaj0Rxy7FviVaGIfJiw/rhS6vGE9TJga8L6NmByh2vEj1FKWSLSBBS42xd0OLcs6Z7tga5m71Th6PkX0F7DbwHuONCbJ0s4qvjpa3fzO+/J+HiJH7xwG8/2vYi77/oHzdvWcvQlV/LKzcdi//7bPPGbOWxoDXN+v2xOeeRrXP3T5ZQdcw7XXTKGu44vJ/iPnzH3gdd5a0sTfivKkdkpTJ02gFHf+BK/uOgBakI2RSkmk/PTGXnxKPp/6ULUpIt4v6qN5z7ezKIlVVSvq+C+yy5hUlkWuXVrCX30Ftvnfkjlgi1sq2hkvT9MbdgmHFWYAnn+rUQ3LqVt5RLqVlZQu6qahopGdjQG2RHclZRlu6Hy6jYniLupMcDmujYqavxU1QfiQdxgW5hQSzORtibSRxeQUVqAtyBmslYEmQVxkzXbm05rOEpbJLorIcswMb1OAlZipSyPG8CNJWn5gxbhsN1pENcK29i2k5wVtaP43ACuz2OQ7jPbJWUlBnF9HqNdEHdX0HZXEDcx8BqN2kkHcTt78tpTEDdGskHcfQ266iBu70WUQrr43CRQq5Sa2JP96W660vSXAktF5F9KKf1kr9FoPheIinbXpSqB/gnr/dxtnR2zTUQ8QA5O8msy5+4ze9X0ReQFd/ETEVmW0JaLyLIDvblGo9EceihQ0eRa1ywGhonIIBHx4QRmp3c4Zjpwjbv8JWC2Ukq52y93Z/cMAoYBiw703XUl79zmvp53oDfSaDSaXoPaLS1pPy+jLBG5BZgFmMBTSqmVInIf8KFSajrwV+AfIrIeqMf5YsA97gVgFU4M9eYDnbkDXcs7293FWiCglIqKyHBgJPD6gd48WfocMYgrq8Yy8y8P41/0ON9dV8hf73qMqBXmrK9fy/OXj6Li1iv497Mr8VtRvjypL1P+cBeLS05g6EnzuevKcXy5XLHz17fzwaPvM7e2DYCpBWkcc9EIhtxwLY2jzqAm9Ev6p3mZNCCbkV8cR58vXkLr8JOYXdHIcx9uZcWyanZu+BT/jk2cNCCH1K0f0brgTSrnLqFqURUbtzWzNWBRn6Dn53hN7E8X0rJiKXUrNlK3ppaGikYq/WFqQk5SVsDepef7DKGiPsCWpiCbalupqPFT3RCgtTlEW1OINn+ISGsT4bYmrICfzLIivIUlGG7RFDJyiabmEE3NJmKm0Ba2aY1EaYuoPer5iSZrZoqj63t8XkIhCyscK5Ziu8lYTgGVRD3ftqwELd/Yq57vSzRcS9DzOyZkRRM0Va9pYAhd6vkdJf1k9PyuDNYOVHvv7HSt5x/iKJXsU3ySl1MzgZkdtt2bsBwEOp0Cr5T6BfCLbusMyU/ZnAukikgZ8AbwFeDp7uyIRqPRHCqIiibVeiPJDvqilGoDLgYeVUpdgpMwoNFoNIcZCqJWcq0XkqyfvojIFOBKnOwxcPQpjUajObxQdKu8c6iR7KB/O04G7itucGEwMKfnutWe1XU2y/74F8qnnMeps4QF//4TmaUDueP2i7l7sJ/5p5/Di4uqyPeZfPWSUYz+9QP8uyaPX/1hPo/ePIUTqGDt93/JnJc+ZWlTkByvwQmFGYz96iTKrvs6FdmjeHreZkZlpTDxqGJGXDqJvPOvZHvOcP63cifPLdrKplU7qa9YQVtdFVErjG/V29TPm03l+6vY/tEO1te2URW0aIrY2MrR5nO8Bn1TvdQvXEjdik3Ur2+gbnMTlQGnAHpTxNH+E/X8TI/B2rpWKna2srmulbqGAG3NIWd+fmuQcEs9kaAfK+DHDgfxFg/BLCjFzCuOF0tRaTkE8dAWjtIaiRKworSErLihWuLcfEe/T2uv57vmaZGQHZ+P72j6Kl5AJabpq6hN1ArH9fw0n6ddwZSYjm8aspumD13r+cq22+n5Hefqwy4930hQt7vS82PnQXJ6/v74b/X03PzO7qHpDhREP+eDvlLqXeBdEckUkUylVAVwUBw2NRqN5mDTW/X6ZEi2MPoYEfkEWAmsEpGPRERr+hqN5vCk++bpH3IkK+/8Bfi2UmoOgIicDDwBHNdD/dJoNJrPBqUgeRuGXkeyg35GbMAHUEq9IyIZPdQnjUaj+Uw5nOWdZAf9ChG5B/iHu34VUNEzXdqdQFMDU+/4Cm/cOoWcKTdRPuU8Hv/2CUz59AVeOfZR3trZyticVC78zjTyvvMQ3521nhdfeovqFXM59pxaFv7yad78oJKqoEXfVA8nH1XM2BtPIf2CG5nXksHjs9awaOE2nj99IMMvPxXvSZfyqZ3Pyx9V8vribVSuraJpy2oCDTsASMnKp/q16VR+sI4dS3eypsWpkuW3nA9KmikU+jyUpXnoU5DGjoXrqFvXwM6drXGDtaaIUyULnNJssSButsdkZWUzm2tbaW4M0tYcoq0lRKjVT6S1qV0Q1woF8JSUY+QUxg3WoilZtFmKtohNqxUlEInSFLRoClm7BXHjAVy3apbHl4JhGnh8Jh6viZVgtma7wdt2iVlW2AnkRsJOANdNyOoYxI0308AQ2WuVrFgQV9kJyVmGsdeErFgAVyS5AG7i/boK4u5vAaVkgrgHUp1JB3B7ku5NzjrU2JfC6EXAf4CXgUJ3m0aj0Rx+fF41fRFJBb4BDAWWA3cqpSIHo2MajUbzmdDNNgyHGl3JO88AEeA94GxgFM6cfY1GozksET7fmv5opdQYABH5K91g67k/9O1XypwzIrw5cjLH3fYHXr3hGBp+8nUefHQBVcEIXxiWz0l/upmKoy7hy39eyLJZ7+Kv3kRO+Sjeuu4h5uzwE7CjTMhNZcpZgxl+w+WEj72Ef6+u5al3lrPh4w00bF7B6N/ciD3+XGZvbuaFTzbw4ZLtVK9bR3PVBqygH8PjIzWnkOx+I1g341G2bmpkY2ukXcGUTI9BSYqj5xeXZZE/LJ+qRVVUNYfYEbRpttoXTDEF0kyDTI9Bntck32cws6oZf1OQQEuYgD9EqKUxbrBmh4NY4QDRSJioFUby+2CnuQZrnjTawlEClqI1EqU1bNMUsmgKRvCHbUxf6u56foLBmmkaeLwmhsfA4zUIh6w9GqzFtPxYclaaz9yjwZppCD7TwGsIhiF7LZgC7fV8FbW71PPjunySQneinr+3gimJknuyOmhnHG56/gF0vZegwD58Z+909VmOSzn7WkRFRPqLyBwRWSUiK0XkNnd7voi8KSLr3Ne8/ei3RqPR9AwxG4bDVNPvatAfKyLNbmsBjooti0hzF+daODGA0cCxwM1udfe7gbeVUsOAt911jUajOWQ4nF02u/LT329TNdeLf7u73CIiq3GK+l4InOwe9gzwDvC9/b2PRqPRdC+f70ButyAiA4HxwEKgJKE4yw6gZA/n3AjcCFCWk8kDU26mNmzx9pmKd487mZdX7KQkxcOtXx3HkJ//jqc2e3jwF7PZsuhNAMqnnMcl545kxnmPkO8zOa08j6OuO5aSq77OurTBPP7mBt6ct5mqFUvwV29CRW22Dz+TmUt28MKCLWz5tIb6imW01VWhojae1EwyivuTXz6M0oG5rHilnq2BSFyf9xlCvs+kJMVDeZaPvMG5FIwoJG94f96bVRE3WAvYuyry7Jqbb5Dj6vk5WSk01rQS8IfjBmvhtibsUAAr2IrtavkxPdzOKkKlZBFQJoEEg7XGgIU/7MzP94csmkNWgn7fucGax2vi8Rlxbb+1ObTb3PyYhh/T8+OavtfcTc+Pmax5DQNTnGIoXkO6NFiLL7vbvYaxW/HzRD3fkOR05o5z+JMxWDuUtPx9v3/33uvw1/ITOIwH/QP5TCeFiGTizO2/XSnVThJy60B2WpdMKfW4UmqiUmpiQUZaT3dTo9FoHGI2DMm0XkiPDvoi4sUZ8P+llPqPu7laRPq4+/sAO3uyDxqNRrNvKJQVSaodCMlMahGRcSLygTsZZpmIXJaw72kR2SgiS9w2Lpn79tigL87v2L8Cq5VSDybsSqz8fg3w357qg0aj0ewzioP1pJ/MpJY24Gql1BHAWcDvRSQ3Yf93lVLj3LYkmZv2pKY/FaeW7nIRiXXmB8CvgBdE5HpgM3BpD/ZBo9Fo9gmFahdb6kG6nNSilFqbsFwlIjtxLHEa9/emPTboK6XeZ895JKfuy7Wqqpooys3n5t9fwoOTbmRDa5jz+2Vzyh+vo3Lq1zj7+aUsmfU+zdvWktVnCKOnHcePLjiCU7ObeCw7hanTBjDqG19CnXw1L66p4y+vLmHDks3Urf+YSGsTntRMcvoN51dzNrDgkyp2rN1A8/YNRFqbEMMkvaAvWX2GUjywlKFD8jl5ZDGr/eF4QlaO14gbrJX2LrxafAAAH7lJREFUySR/WB75w/uSN2oAKYNGsjUwY48JWdkeg3yfSWGKh/TCNDJKMmhpCBBqaSbS1kS4tWm3hKzEgKSVlk9rJEpbvEKWTUvYoilo4Q/bNIUi+IMWTW0RvKmZTnKWG8TtLCErFtA1PYZbLWvPCVnxQG7UjlfO2lNCViyY6zGNpBKyEukqIatj1azO6MyIbV8SsvY1APtZBnG7O4ALn7cgLvtSOatQRD5MWH9cKfV4kucmNaklhohMAnzAhoTNvxCRe3F/KSilQl3d9KDM3tFoNJrewz756dcqpSbuaaeIvAWUdrLrh+3uqJQSkU4ntbjX+f/2zjw8jrvM85+3qrullmTrlixbjuX4NgkJORxCBiYkgQSWHJsNIYFhmF0yHpb7AYYkZGFgnp1nAzObsCwsYG52MjAQyEOAgElCjuUIwUnsxI7t2PER35Zlqa2jpe7q+u0f9etWtdwttXxIavf7eZ56uupX1VX1s1tvV3/fq4OgyvF7jMmFFt1J8GURA9YQ/Er4x4luWI2+oihKGGNO2kk7eipzVbF9InJIRDqMMQfGC2oRkdnAL4G7jDFPhc6d/ZUwIiLfAT5Ryj2d9pBNRVGU8sLkpMuJlpNkwqAWEYkBDwDfN8bcP2ZfNgpSgBuAjaVctCye9FsbqvnPm3/JFzZliHE/n/zo6+j8zD3cs76fb3z6N+x75mGcSIyz33A9775uBR+4pJPqJ7/H+i/fzzs++1Yab17NizKXr/5iK0/+4RUOvPgcg917AKhr76J50bmc/ao2fvXrLfTteoFk7yGMnyFaW8+s9i4aOrvoWNjIZctauWxhE69qq2WDb4i7QmPUZU51hPn1VTQtaaJpcTONKxZQt3gx0a4V+M0LSKRH9cG4K8TdUS2/KeYyq76KurZaalri1HXMZqjnUF6BtbEJWWF6hzM5PT/bLGXAavqDqUDL7xtKMzDi4cbieQlZkZgbaPqhhKywtp/T9EPNUsIJWX7owx8Pafqu1fCjblAkbVTXl5zePFFCVng76oY0/AIJWWGNfywT/WGeai2/EOOdo9QicaWiCVmngGz0zumnYFCLiFwEvM8Yc5sdewPQLCJ/Y9/3NzZS5z4RaSXwna4nKIM/IWVh9BVFUaYOMxlH7olfxZgeCgS1GGPWAbfZ9X8F/rXI+684keuq0VcURQljmKqQzWlBjb6iKEoek4reKTvKwuh7nQu54J4tbH/iIQaeXsMj7krefs+zvPTEI6QGE7Qufy2XvelcPveW5SzpeZadd/w3nvnRRp46muQT9z3Ivc8f4P4nnmb3hhdJ7H2JTCpJdX0rDV3nMH/5PK56zVzetqKdN3zv38ikkrixODXNc5nduYw5XY2cv7SF1y9q5oK5s+maHSV6aOtocbWaCM0L6mlZ1kzD0k4ali0k2rUc6ViE19BJbyb4J445QtwVat1RLb+xNkq8pYa6thpq22uJtzVSO6eJ5K8O5hqfF9PyxXERx6VveDQuP6vn948EWv7AcLA+MJymf9gjWls/Goefa4DuWB1/jL4fcfBS6eD6mfy4fONnyGS37RNRVtPPavhR2wQ96gY6ftQRXKvplxKbH94eW1xt7Bgcr42X4mQbq+ePp+WfqPZeTM9XLX8Gcwqjd2YiZWH0FUVRpg590lcURakcpi56Z1pQo68oihLCYHJ9nM9E1OgriqKE0Sf96eflXQeJPvZzOi9+M1euFZ5f+zUGDu2i/qwVXHTjdXz22pVcVnWIQ1/7e379zT/y+8ODHE1lmFMd4Z3f/jM71u/i6M4NpAcTRGvraew6h7nLz+ay8+dy7TlzuKijltmHX8T4mZwDt21BG0sXNfGXy9q4pLOeRY1VxHt34a/bwLGN6zmvvoq2ebOChCxbXC3WtRx33lIyjZ0cc2roHvTYe2yIukhQXK3RdsdqjEWoba+hpqWG2rYaatrqqe1oJt7aSLS1ndRPthcsrpZFHBcnEkMclwMDI/Tbzlj9qVEHbl8yzcBwmqFUhoFhj1QqQ6wqkpeAlUvOirq4EcFxHWKhJKvMSLJgcbWcQzcTSs6KugWLq2U7ZjkiufVSHbhZ3FBnq3BxtTzHLqPOzFIzJUtJyKo0By5UuBMXAkduOjXdd3HaKAujryiKMnVMTXLWdKFGX1EUZSwq7yiKolQIxpyKYmozlrIw+pHqWm7/7x/ljr/sov7S9zOrYxGrbnk3d12/kqsaBjj6/c/x6Dd/z+9eSdA9kqG1yuVtHbNYfuMK/vmBn+UapTQvvoA5Sxdx8XkdXHduB5d2zqLh6DZG1j7MzifX0br8GlrOamfpkmYuX97GqnkNLGqMUde/D/+55xjc8jxHnt/OkRcPsez18/MapbidgZafcOvoTnrsOzbIrr4ku3uGmFsdOa5RSlbLDxKymok2t+A2tuE2tuIl10+o5bvRoBnKgf6RoMBaMk1iKEjCGhjx6B9O57R8L53BS/vE4tHjGqVEos5xWn5VxCEei5BJJSfU8rP3WRVxxtXys4labhHdvdgfmfEzE2r5MNpgZbJ/rKVq+Scrc6uWX15o9I6iKEqlYAwmo0ZfURSlIjDG4Ke96b6N04YafUVRlDAGfdKfbs6ZP5sPb/sWj//db3jdR77EZ65dyRviRzj8nc/yyDf/wP87MMDRVKDlX9s5mxU3nUPnTTfgX3At5orbaVl6MR1LF3Kpjcu/eG4d9Ue2MPLrR9j5xDr2/3kvu1/u5fX3fDwXl7+woYraxCv4zz3HwOZAy+/Z2s3RbUfZf2yEm794C1WLVubi8vucGrqHPPYeG+SVRJId3YPs7hlk75EhPj6r6jgtPxeX39yC2zwHt7ENahvx4/UFi6uN1fKdSBQnEuOV3qGgsNqwRyKZyovLT48Een4m4+OlMlTXRseNyw+am9tt1zmuUUohLT+rfVa7TtG4fEeCWPtsg/Pw/MbT8rNk/QDjafkw+TZw2eOnU8s/kfOfDj1fyUeNvqIoSoVgjMHXevqKoiiVw5kcvaON0RVFUcLY6J1SlpNBRJpE5GER2WZfG4sclxGR9XZ5MDS+UET+JCLbReTfbRP1CVGjryiKEiIbvVPKcpLcATxqjFkCPGq3C5E0xpxvl+tC458H7jXGLAZ6gfeWctGykHeOPr+Fz3y4l5gjPHq1YecXP8BPbGesZMbQVRPlqmXNLL/5QtpvfAd981fx0x29/PC+Dbzm+htynbHOba0muvNPDPzoEbY+sYH9zxxkx/5+9iTTHE1l+MzVy3KdsdK/f4beTRvp2bSTni099O7oY89Qmu4Rj2OeT/yKt5Np6KQ7E6F7yGN3Xz97Ekl2WgfuwZ4hBo+NMHhshDnnt+V1xoq3NRJpbMVpbCPSPAe/pgG/ahZ+vJ6UE3xZZztjiePiRGM41pmbdeC6VXHcSIy9vclcZ6yBYS9IxEr5NiHLOnI9g+/51M6uzuuMFY+5VFknbtiBmx3zUslccbRCDtzR9aDgWrYz1miXrHwHbuDcHb8oWsGktCKF1cY6cIsVOStGMQduobNMNrnqdDhwlanDnxpH7vXA5Xb9e8DjwO2lvFGCD+8VwDtD7/8s8NWJ3qtP+oqiKGFsyGaJ8k6LiKwLLasncaV2Y8wBu34QaC9yXLU991MicoMdawb6jDHZnxt7gXmlXLQsnvQVRVGmjMll5B4xxlxUbKeIPALMKbDrrvxLGiMipshpFhhj9onI2cBvReQFIFHqDY5Fjb6iKEoIw6mL3jHGXFVsn4gcEpEOY8wBEekADhc5xz77ukNEHgdeA/wEaBCRiH3a7wT2lXJPZWH0U77h7Rd0cP7qN3LPqtW8PJgi7grn1Vdz3uvns+zWNxK9/Ba2mWa+s+kgDz34FHu27CPxymbW/+hOOv0e/I0/58h3fs++P27j4IbDbB9IsX/YY8AL/nPjrrD44FOkHn+G/S9s58jGPfRs66Wne5B9SY/edIZE2iflB1/Ge6rP4lBPml19A+w6OsSO7kH2Hh0i0TfM4LFhkv0pkv39pAcTdFyymJq2RqpamnKJWE59C368Hq96Fn51PUOeYSjlk/Q8q93HENfFDen4TjRGJBYPNP1YHCcaY/eRQUZCRdW80HrG88lkfHz7Wl0bJZan5Y/q+NlCa7HQ4qdTebp99g8hPAbg+xmqIzYhy2r5UcfJ0/HDun6pxdayuFntfgIt/2R197FvP9VF0lTHLxOMwU9NSRmGB4H3AHfb15+NPcBG9AwZY0ZEpAW4DPiC/WXwGHAT8MNi7y+EavqKoihhDPi+X9JyktwNvElEtgFX2W1E5CIR+aY9ZgWwTkQ2AI8BdxtjXrT7bgc+JiLbCTT+b5Vy0bJ40lcURZkqDFNTZdMY0wNcWWB8HXCbXf8DcG6R9+8AVk32umr0FUVRwhjy+jifaZSF0e9YuYCFax/mK+v3E+N+brmwgxU3X0Trje/icOu5/OTlXn7wwCu8vGkDR17exGD3HnwvhRuL0/Djf2Lr717gwLMH2X5gkP3DQUx+xkDMEVqrXNqrIpxVE2XbF7/MkS099O5KsC/p5WLykxmfjPWrxxwh7gq/3HaEHYeDmPwjvUkG+oYZGkgxPJgi1X+U1FACLzlAJjVM8yUX5hqk+DUN+NX1ZKpnkXJiDKZ9hgY9kmlDYiRN/0iGaLwuF5PvVlkNP6Tju7F4rhHKscRIwZj8bKE14xsynofvpWiui+Vi8uNRN0/Hdx3J0/OjTlBwDY6PyYdAx89iMhmqIk7BmPzwdrgRSvhc4xE0UTm+qFohHf9E6pCVGpM/2RyAia6hzGSMlmE4EUTk2yJyWEQ2hsZKSjtWFEWZNiYXp192nE5H7neBa8aMlZp2rCiKMi0YY8ikvJKWcuS0GX1jzJPA0THD1xOkC2Nfb0BRFGVGYaykOfFSjky1pl9q2jE2nXk1wFkdRQ9TFEU5tWjnrNPDBGnHGGPWAGsAauctNZes/haJvS8x8PQa+uav4uEdvfzw8T1s3fQo3dtfZPDwHjKpJG4sTm3rfGZ3LqP9rAZ+/KkP5gqqZUyQ6FMfzTpvIzQvqKdlWTMNSzv52b88VtR5WxcRal2HpphLU8zl63/YnSuoVsh5640k8b0guSn6qr/Cr64nbQuqDaZ9hoZ9kuk0/SmPxLBHYsRjIOXRP+IRq2vMFVQr5Lx1XYdIzCUSdRhIJHPO20wmSMjyM37OeWsymdx9NNVV5RVUG7u4tlhatvOV76WD/4siztvcejg5q4jzNlw0bSIH7tj92eSs8Zy3J/KTNexgPZOct9pY6yQxYDJFTVPZM9VGv6S0Y0VRlOnCYKaqyua0MNUZudm0Y5hE2rCiKMqUYcD4pqSlHDltT/oi8gOCWtEtIrIX+AeCNOMfich7gd3Azafr+oqiKCeCMZBJaXLWpDHG3Fpk13FpxxOR7Osl1n+UeRdeyZVrhd2bH6Jv1wsM9ewPNPPaembNXUTTWYuY09XApUtbuezsZs5tq+V/fHrYJmFFmFsdYV5djKYljTQtbqZpxQLqliwm1rUcv6WLDZ/+Ve6acVeIuw6zIw71UZfWKpdZ9VXUNMepa69l16b9pAcTeTp+Jp3K6edhXbq/cRGDaUMy6TOUTuVp+AMjHsdGPBJDthHKiEe8cU4uKSsSdYnEsjq+bYASdXEiDpGow+FXEqNavr12tlCa8QM937frbbOqRvV7m4wVdRyiruT0fMexr7Yw2ng6fpiaqJtXEC2s44/q7lJUbx5P5xeR0SYqofc7Y46ZLMcVXBvnHKe6+JpzioV31fFPIcaopq8oilJJ+Gr0FUVRKgQN2VQURakcDOCXqZO2FNToK4qihDFGHbnTzZx57fz0Gx/hvPYa6i99P24sTryxnbkXXk37WQ28emkLr1/cwsXzZtM1O0r00FbSL/2C/l9v5Or2WpoX1NO0uJGmFWfRsGwh0a7lSMciMg2d9GYidA957O4dpj7q5CVgNdZGibfUUNdWQ217LfG2RmpaG6jpaKb3GxvyErDGOiLFcXPLlp5hEsOB4zYxEiRgJYbSDAwH6wPD1ok77OGlM9S1tOQlYDmhpCw3IoFz13bA2r1pb14CVnbJZLczo9UxW2dXHZeAFXUDp23UyXa9Gl3PpFO5+UzU7SrqOHkJWOGKmnnjRd4/Hq712I7nuD1RR2sx5606bisXo8lZiqIoFYQafUVRlEpCM3IVRVEqhynKyC2lv4iIvFFE1oeWYRG5we77rojsDO07v5TrlsWTfluym6qP3MJvnznI6z72v/OSrzoiw7gHNpPa8hhHf76FbVv2cGRLDz37B9iX9Fj9bx/OJV95DfPoHvLoHvTY1TvE7p2j3a96e4f5dFdDLvmqpq2OmrZGauY0EW9twmlsI9I8B6ehFT9ez/C/fD7vHsMavhMNOl05kShOJMYfXunNS77KavhDVsP30j5eKpPrdlXfXJNLvsoWWcsmVVVFHOKxSLDtOjzVfzSXfJXV8MNdroIleGppqo7mJV+5Y9YdIdD83dHkrCyFNPjwWMTNT75yZFS/DydtFTvXeDjka+/HJVVN6myh941zzuOOneS5T7WGH0b1/NOLYcri9LP9Re4WkTvs9u1592LMY8D5EHxJANuB34QO+XtjzP2TuWhZGH1FUZQpwxj8qYneuZ6gVA0E/UUeZ4zRH8NNwK+MMUMnc1GVdxRFUUIYEzzpl7KcJCX3F7HcAvxgzNg/icjzInKviFSVclF90lcURRnDJLpitYjIutD2GtsLBAAReQSYU+B9d+Vdb4L+IrYU/bnA2tDwnQRfFjGC3iO3A/840Q2XhdHft7ePr+99ibgrPHq1YWTzQxz94VZ6Nu/l5S09dB8c4OBwhiMpjwHPJxn6Bt746neysy/J7q1D7Ojeyu4jgyT6hhk8NkyyP8Xw4FCucNpFH76SeHsrbmMrbmNbTr/34/X4VbMY8IXBtGEo7eNEYgX1eycaIxILiqU5kRhuVZzHNh8uqt97qaD5SbgJyooL5hKLONTEXGIRN6ffZzX9cOOT1GACOF6/D+v6EDRAaYxH8/T7qOPkmp0Uan4ykaYfJuZkG5zk6/fZn5KFGqCUiht609i3n0w8fbH3qmRe4ZhJPcUfMcZcVPxU5qpi+0RkMv1FbgYeMMakQ+fO/koYEZHvAJ8o5YZV3lEURQlj4/RLWU6SyfQXuZUx0o79okCCJ6obgI2lXLQsnvQVRVGmCsOUFVwr2F9ERC4C3meMuc1udwHzgSfGvP8+EWkl+HG6HnhfKRdVo68oihLGGDKp02/0jTE9FOgvYoxZB9wW2t4FzCtw3BUncl01+oqiKCGMAd9oGYZppXV2FZ/8L6+jaUUX96xaTW86w4Dnk7IZca4EjsS6iMPc6ihNMYfWqgg1TXFu+z9/ZKh/hJHBgcBhO5jAGx7E91J4I8lcdymAWX91N5nq2QykfQbTPknPJ5n2SRz1SIz0MzBiO16NeMyau8g6cGO4sbh14FaFulq5ueJor+zoJWMdtV4qgzEm1+lqbJcr42c4Z96KvO5WuSVUJC3qOLgC3vAgkO+whcJdrhrj0YIO27GF0UpJojq+4Fr2HPkO22KdriaDUNjpeiLdssaet1S0YFplkVGjryiKUhkY4Ayut6ZGX1EUZSz6pK8oilIh+IacdHwmUhZG3z/rbJ766y+w8+gQMe5nUW2UppjLrOYaalri1LbXUts2i5o5zdS0NRJrbsJt7sBtbGXrbT/NafZhcsXRbAKVG4lx/84UiZGDgXZvC6QlkmmSKY/+YY9kKMFqzrKVOc0+2/DEce22bXCSTaZ6cu3zeZp9oQJp4WSq5R2zcpp9xA1eAy1/dD1bLC3rlxhLobHZVZE8zT5bIG1sg5Osfj2ZwmgRV4o2OTnZhiTumBOc6gYnwTlVs1dGUXlHURSlQjAYlXcURVEqBXXkKoqiVBhq9KeZbbsP8bcf+p/46RQDT6+B2sagCFr1bNKROENpn6Rn6Ev77EtlSIx4JIbTDKQyxBvbjyuE5lYFr5FYNNDjbWz9vQ++aIuh5RdA8zM+Gc8L9HgbV3/1tRfkYufHFkHLxdi7DlFHeOj7O/MKoYW18kJx9UuaascthBZuVpJJJUv6NzR+hrpYoLqPLYIGhePqJ0OsBN39ROPqww1ZTiWT0fFVo68cjNHoHUVRlIrBoNE7iqIoFYNq+oqiKBWGyjuKoigVQqDpT/ddnD7Kwui7sWraVl6GG3G4cq3gpY7ipbttolSGjGfwPT/Xjcr4hozn4Xsp3vzO/2Cdqy7xqJvXfWpsQbNP/8N3Q0lSfsHuU1luvfA6HOE4R2shx+tw4kjufaUkPJ1VH7S6LKX71GQSqGqjwZkK+SRPNuEp6uaf4FT6Pd3T5EVV56xSDH3SVxRFqRAMMCUtVKYJNfqKoighDEajdxRFUSqFIHpHjf60cs6CJn7/pbcBUH/p+yf13u9+7e0lH/ux7j0lH3vZ/FklH1uo4Nt4tNWenv+WmuiJtjGZmMjpqIJmUe1dmVLOcEfu6bMC4yAi14jIVhHZLiJ3TMc9KIqiFCL7pF/KcjKIyNtFZJOI+LYZerHjCtpLEVkoIn+y4/8uIrFSrjvlRl9EXOArwFuAlcCtIrJyqu9DURSlGBlT2nKSbARuBJ4sdsAE9vLzwL3GmMVAL/DeUi46HU/6q4DtxpgdxpgU8EPg+mm4D0VRlOPwCcowlLKcDMaYzcaYrRMcVtBeShC/fQVwvz3ue8ANpVxXzBQ7LETkJuAaY8xtdvvdwCXGmA+OOW41sNpunkPwrXim0AIcmfCo8uFMmw+ceXOqpPksMMa0nuiJReTX9vylUA0Mh7bXGGPWTPJ6jwOfMMasK7CvoL0EPgs8ZZ/yEZH5wK+MMedMdL0Z68i1/3BrAERknTGmqOZVbuh8Zj5n2px0PqVjjLnmVJ1LRB4B5hTYdZcx5men6jqTYTqM/j5gfmi7044piqKcURhjrjrJUxSzlz1Ag4hEjDEek7Cj06Hp/xlYYj3PMeAW4MFpuA9FUZSZTkF7aQJd/jHgJnvce4CSfjlMudG330ofBNYCm4EfGWM2TfC2SWlkZYDOZ+Zzps1J5zPDEJH/KCJ7gUuBX4rIWjs+V0Qeggnt5e3Ax0RkO9AMfKuk6061I1dRFEWZPqYlOUtRFEWZHtToK4qiVBAz2uiXa7kGEfm2iBwWkY2hsSYReVhEttnXRjsuIvIlO8fnReSC6bvzwojIfBF5TERetGnjH7HjZTknEakWkadFZIOdz+fseMG0dhGpstvb7f6u6bz/YoiIKyLPicgv7Ha5z2eXiLwgIutFZJ0dK8vP3Exixhr9Mi/X8F1gbKzvHcCjxpglwKN2G4L5LbHLauCrU3SPk8EDPm6MWQm8FviA/b8o1zmNAFcYY84DzgeuEZHXUjyt/b1Arx2/1x43E/kIgbMvS7nPB+CNxpjzQzH55fqZmzkYY2bkQuDRXhvavhO4c7rvaxL33wVsDG1vBTrsegew1a5/Hbi10HEzdSEIDXvTmTAnoAZ4liDL8QgQseO5zx9B5MSldj1ij5Ppvvcx8+gkMIJXAL8gaF5WtvOx97YLaBkzVvafueleZuyTPjAPCNc63mvHypV2Y8wBu34QaLfrZTVPKwW8BvgTZTwnK4WsBw4DDwMvA30mCJGD/HvOzcfuTxCEyM0kvgh8ktGmT82U93wgKHj5GxF5xpZlgTL+zM0UZmwZhjMZY4wRkbKLlRWROuAnwEeNMcckVOi+3OZkjMkA54tIA/AAsHyab+mEEZG3AYeNMc+IyOXTfT+nkL8wxuwTkTbgYRHZEt5Zbp+5mcJMftI/08o1HBKRDgD7etiOl8U8RSRKYPDvM8b81A6X9ZwAjDF9BJmNl2LT2u2u8D3n5mP31xOkwc8ULgOuE5FdBFUYrwD+F+U7HwCMMfvs62GCL+ZVnAGfuelmJhv9M61cw4MEqdKQnzL9IPDXNvrgtUAi9PN1RiDBI/23gM3GmHtCu8pyTiLSap/wEZE4gX9iM8XT2sPzvAn4rbHC8UzAGHOnMabTGNNF8HfyW2PMuyjT+QCISK2IzMquA28mqLRblp+5GcV0OxXGW4C3Ai8R6K13Tff9TOK+fwAcANIE2uJ7CTTTR4FtwCNAkz1WCKKUXgZeAC6a7vsvMJ+/INBXnwfW2+Wt5Ton4NXAc3Y+G4HP2PGzgaeB7cCPgSo7Xm23t9v9Z0/3HMaZ2+XAL8p9PvbeN9hlU/bvv1w/czNp0TIMiqIoFcRMlncURVGUU4wafUVRlApCjb6iKEoFoUZfURSlglCjryiKUkGo0VemHRHJ2EqKm2zly4+LyAl/NkXkU6H1LglVO1WUSkeNvjITSJqgkuKrCBKl3gL8w0mc71MTH6IolYkafWVGYYKU+9XAB212pSsi/ywif7Z10v8OQEQuF5EnReSXEvRc+JqIOCJyNxC3vxzus6d1ReQb9pfEb2wWrqJUJGr0lRmHMWYH4AJtBNnMCWPMxcDFwN+KyEJ76CrgQwT9FhYBNxpj7mD0l8O77HFLgK/YXxJ9wH+autkoysxCjb4y03kzQU2V9QTlnJsJjDjA08aYHSaomPkDgnIRhdhpjFlv158h6HWgKBWJllZWZhwicjaQIaigKMCHjDFrxxxzOUE9oDDFaoqMhNYzgMo7SsWiT/rKjEJEWoGvAV82QWGotcB/taWdEZGltuoiwCpbhdUB3gH8zo6ns8cripKPPukrM4G4lW+iBP14/y+QLeH8TQI55llb4rkbuMHu+zPwZWAxQRnhB+z4GuB5EXkWuGsqJqAo5YJW2VTKEivvfMIY87bpvhdFKSdU3lEURakg9ElfURSlgtAnfUVRlApCjb6iKEoFoUZfURSlglCjryiKUkGo0VcURakg/j+7fyjNRp+DjgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pos_encoding = positional_encoding(50, 512)\n",
    "print (pos_encoding.shape)\n",
    "\n",
    "plt.pcolormesh(pos_encoding[0], cmap='RdBu')\n",
    "plt.xlabel('Depth')\n",
    "plt.xlim((0, 512))\n",
    "plt.ylabel('Position')\n",
    "plt.colorbar()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "a_b4ou4TYqUN"
   },
   "source": [
    "## Masking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "s42Uydjkv0hF"
   },
   "source": [
    "Mask all the pad tokens in the batch of sequence. It ensures that the model does not treat padding as the input. The mask indicates where pad value `0` is present: it outputs a `1` at those locations, and a `0` otherwise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:25.191500Z",
     "iopub.status.busy": "2020-11-02T02:26:25.190865Z",
     "iopub.status.idle": "2020-11-02T02:26:25.192659Z",
     "shell.execute_reply": "2020-11-02T02:26:25.193052Z"
    },
    "id": "U2i8-e1s8ti9"
   },
   "outputs": [],
   "source": [
    "def create_padding_mask(seq):\n",
    "  seq = tf.cast(tf.math.equal(seq, 0), tf.float32)\n",
    "  \n",
    "  # add extra dimensions to add the padding\n",
    "  # to the attention logits.\n",
    "  return seq[:, tf.newaxis, tf.newaxis, :]  # (batch_size, 1, 1, seq_len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:25.197206Z",
     "iopub.status.busy": "2020-11-02T02:26:25.196569Z",
     "iopub.status.idle": "2020-11-02T02:26:25.201194Z",
     "shell.execute_reply": "2020-11-02T02:26:25.201613Z"
    },
    "id": "A7BYeBCNvi7n"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 1, 1, 5), dtype=float32, numpy=\n",
       "array([[[[0., 0., 1., 1., 0.]]],\n",
       "\n",
       "\n",
       "       [[[0., 0., 0., 1., 1.]]],\n",
       "\n",
       "\n",
       "       [[[1., 1., 1., 0., 0.]]]], dtype=float32)>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = tf.constant([[7, 6, 0, 0, 1], [1, 2, 3, 0, 0], [0, 0, 0, 4, 5]])\n",
    "create_padding_mask(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Z0hzukDBgVom"
   },
   "source": [
    "The look-ahead mask is used to mask the future tokens in a sequence. In other words, the mask indicates which entries should not be used.\n",
    "\n",
    "This means that to predict the third word, only the first and second word will be used. Similarly to predict the fourth word, only the first, second and the third word will be used and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:25.206009Z",
     "iopub.status.busy": "2020-11-02T02:26:25.205344Z",
     "iopub.status.idle": "2020-11-02T02:26:25.207194Z",
     "shell.execute_reply": "2020-11-02T02:26:25.207586Z"
    },
    "id": "dVxS8OPI9uI0"
   },
   "outputs": [],
   "source": [
    "def create_look_ahead_mask(size):\n",
    "  mask = 1 - tf.linalg.band_part(tf.ones((size, size)), -1, 0)\n",
    "  return mask  # (seq_len, seq_len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:25.211387Z",
     "iopub.status.busy": "2020-11-02T02:26:25.210780Z",
     "iopub.status.idle": "2020-11-02T02:26:25.214859Z",
     "shell.execute_reply": "2020-11-02T02:26:25.215313Z"
    },
    "id": "yxKGuXxaBeeE"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 3), dtype=float32, numpy=\n",
       "array([[0., 1., 1.],\n",
       "       [0., 0., 1.],\n",
       "       [0., 0., 0.]], dtype=float32)>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = tf.random.uniform((1, 3))\n",
    "temp = create_look_ahead_mask(x.shape[1])\n",
    "temp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xluDl5cXYy4y"
   },
   "source": [
    "## Scaled dot product attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vsxEE_-Wa1gF"
   },
   "source": [
    "<img src=\"https://www.tensorflow.org/images/tutorials/transformer/scaled_attention.png\" width=\"500\" alt=\"scaled_dot_product_attention\">\n",
    "\n",
    "The attention function used by the transformer takes three inputs: Q (query), K (key), V (value). The equation used to calculate the attention weights is:\n",
    "\n",
    "$$\\Large{Attention(Q, K, V) = softmax_k(\\frac{QK^T}{\\sqrt{d_k}}) V} $$\n",
    "\n",
    "The dot-product attention is scaled by a factor of square root of the depth. This is done because for large values of depth, the dot product grows large in magnitude pushing the softmax function where it has small gradients resulting in a very hard softmax. \n",
    "\n",
    "For example, consider that `Q` and `K` have a mean of 0 and variance of 1. Their matrix multiplication will have a mean of 0 and variance of `dk`. Hence, *square root of `dk`* is used for scaling (and not any other number) because the matmul of `Q` and `K` should have a mean of 0 and variance of 1, and you get a gentler softmax.\n",
    "\n",
    "The mask is multiplied with -1e9 (close to negative infinity). This is done because the mask is summed with the scaled matrix multiplication of Q and K and is applied immediately before a softmax. The goal is to zero out these cells, and large negative inputs to softmax are near zero in the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:25.222000Z",
     "iopub.status.busy": "2020-11-02T02:26:25.221312Z",
     "iopub.status.idle": "2020-11-02T02:26:25.223205Z",
     "shell.execute_reply": "2020-11-02T02:26:25.223597Z"
    },
    "id": "LazzUq3bJ5SH"
   },
   "outputs": [],
   "source": [
    "def scaled_dot_product_attention(q, k, v, mask):\n",
    "  \"\"\"Calculate the attention weights.\n",
    "  q, k, v must have matching leading dimensions.\n",
    "  k, v must have matching penultimate dimension, i.e.: seq_len_k = seq_len_v.\n",
    "  The mask has different shapes depending on its type(padding or look ahead) \n",
    "  but it must be broadcastable for addition.\n",
    "  \n",
    "  Args:\n",
    "    q: query shape == (..., seq_len_q, depth)\n",
    "    k: key shape == (..., seq_len_k, depth)\n",
    "    v: value shape == (..., seq_len_v, depth_v)\n",
    "    mask: Float tensor with shape broadcastable \n",
    "          to (..., seq_len_q, seq_len_k). Defaults to None.\n",
    "    \n",
    "  Returns:\n",
    "    output, attention_weights\n",
    "  \"\"\"\n",
    "\n",
    "  matmul_qk = tf.matmul(q, k, transpose_b=True)  # (..., seq_len_q, seq_len_k)\n",
    "  \n",
    "  # scale matmul_qk\n",
    "  dk = tf.cast(tf.shape(k)[-1], tf.float32)\n",
    "  scaled_attention_logits = matmul_qk / tf.math.sqrt(dk)\n",
    "\n",
    "  # add the mask to the scaled tensor.\n",
    "  if mask is not None:\n",
    "    scaled_attention_logits += (mask * -1e9)  \n",
    "\n",
    "  # softmax is normalized on the last axis (seq_len_k) so that the scores\n",
    "  # add up to 1.\n",
    "  attention_weights = tf.nn.softmax(scaled_attention_logits, axis=-1)  # (..., seq_len_q, seq_len_k)\n",
    "\n",
    "  output = tf.matmul(attention_weights, v)  # (..., seq_len_q, depth_v)\n",
    "\n",
    "  return output, attention_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FiqETnhCkoXh"
   },
   "source": [
    "As the softmax normalization is done on K, its values decide the amount of importance given to Q.\n",
    "\n",
    "The output represents the multiplication of the attention weights and the V (value) vector. This ensures that the words you want to focus on are kept as-is and the irrelevant words are flushed out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:25.228177Z",
     "iopub.status.busy": "2020-11-02T02:26:25.227454Z",
     "iopub.status.idle": "2020-11-02T02:26:25.229676Z",
     "shell.execute_reply": "2020-11-02T02:26:25.229174Z"
    },
    "id": "n90YjClyInFy"
   },
   "outputs": [],
   "source": [
    "def print_out(q, k, v):\n",
    "  temp_out, temp_attn = scaled_dot_product_attention(\n",
    "      q, k, v, None)\n",
    "  print ('Attention weights are:')\n",
    "  print (temp_attn)\n",
    "  print ('Output is:')\n",
    "  print (temp_out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:25.235475Z",
     "iopub.status.busy": "2020-11-02T02:26:25.234773Z",
     "iopub.status.idle": "2020-11-02T02:26:25.238644Z",
     "shell.execute_reply": "2020-11-02T02:26:25.239081Z"
    },
    "id": "yAzUAf2DPlNt"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attention weights are:\n",
      "tf.Tensor([[0. 1. 0. 0.]], shape=(1, 4), dtype=float32)\n",
      "Output is:\n",
      "tf.Tensor([[10.  0.]], shape=(1, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "np.set_printoptions(suppress=True)\n",
    "\n",
    "temp_k = tf.constant([[10,0,0],\n",
    "                      [0,10,0],\n",
    "                      [0,0,10],\n",
    "                      [0,0,10]], dtype=tf.float32)  # (4, 3)\n",
    "\n",
    "temp_v = tf.constant([[   1,0],\n",
    "                      [  10,0],\n",
    "                      [ 100,5],\n",
    "                      [1000,6]], dtype=tf.float32)  # (4, 2)\n",
    "\n",
    "# This `query` aligns with the second `key`,\n",
    "# so the second `value` is returned.\n",
    "temp_q = tf.constant([[0, 10, 0]], dtype=tf.float32)  # (1, 3)\n",
    "print_out(temp_q, temp_k, temp_v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:25.243123Z",
     "iopub.status.busy": "2020-11-02T02:26:25.242423Z",
     "iopub.status.idle": "2020-11-02T02:26:25.246814Z",
     "shell.execute_reply": "2020-11-02T02:26:25.246309Z"
    },
    "id": "zg6k-fGhgXra"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attention weights are:\n",
      "tf.Tensor([[0.  0.  0.5 0.5]], shape=(1, 4), dtype=float32)\n",
      "Output is:\n",
      "tf.Tensor([[550.    5.5]], shape=(1, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# This query aligns with a repeated key (third and fourth), \n",
    "# so all associated values get averaged.\n",
    "temp_q = tf.constant([[0, 0, 10]], dtype=tf.float32)  # (1, 3)\n",
    "print_out(temp_q, temp_k, temp_v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:25.250955Z",
     "iopub.status.busy": "2020-11-02T02:26:25.250254Z",
     "iopub.status.idle": "2020-11-02T02:26:25.253722Z",
     "shell.execute_reply": "2020-11-02T02:26:25.254132Z"
    },
    "id": "UAq3YOzUgXhb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attention weights are:\n",
      "tf.Tensor([[0.5 0.5 0.  0. ]], shape=(1, 4), dtype=float32)\n",
      "Output is:\n",
      "tf.Tensor([[5.5 0. ]], shape=(1, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# This query aligns equally with the first and second key, \n",
    "# so their values get averaged.\n",
    "temp_q = tf.constant([[10, 10, 0]], dtype=tf.float32)  # (1, 3)\n",
    "print_out(temp_q, temp_k, temp_v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aOz-4_XIhaTP"
   },
   "source": [
    "Pass all the queries together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:25.258596Z",
     "iopub.status.busy": "2020-11-02T02:26:25.257867Z",
     "iopub.status.idle": "2020-11-02T02:26:25.307528Z",
     "shell.execute_reply": "2020-11-02T02:26:25.307944Z"
    },
    "id": "6dlU8Tm-hYrF"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attention weights are:\n",
      "tf.Tensor(\n",
      "[[0.  0.  0.5 0.5]\n",
      " [0.  1.  0.  0. ]\n",
      " [0.5 0.5 0.  0. ]], shape=(3, 4), dtype=float32)\n",
      "Output is:\n",
      "tf.Tensor(\n",
      "[[550.    5.5]\n",
      " [ 10.    0. ]\n",
      " [  5.5   0. ]], shape=(3, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "temp_q = tf.constant([[0, 0, 10], [0, 10, 0], [10, 10, 0]], dtype=tf.float32)  # (3, 3)\n",
    "print_out(temp_q, temp_k, temp_v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kmzGPEy64qmA"
   },
   "source": [
    "## Multi-head attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fz5BMC8Kaoqo"
   },
   "source": [
    "<img src=\"https://www.tensorflow.org/images/tutorials/transformer/multi_head_attention.png\" width=\"500\" alt=\"multi-head attention\">\n",
    "\n",
    "\n",
    "Multi-head attention consists of four parts:\n",
    "*    Linear layers and split into heads.\n",
    "*    Scaled dot-product attention.\n",
    "*    Concatenation of heads.\n",
    "*    Final linear layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JPmbr6F1C-v_"
   },
   "source": [
    "Each multi-head attention block gets three inputs; Q (query), K (key), V (value). These are put through linear (Dense) layers and split up into multiple heads. \n",
    "\n",
    "The `scaled_dot_product_attention` defined above is applied to each head (broadcasted for efficiency). An appropriate mask must be used in the attention step.  The attention output for each head is then concatenated (using `tf.transpose`, and `tf.reshape`) and put through a final `Dense` layer.\n",
    "\n",
    "Instead of one single attention head, Q, K, and V are split into multiple heads because it allows the model to jointly attend to information at different positions from different representational spaces. After the split each head has a reduced dimensionality, so the total computation cost is the same as a single head attention with full dimensionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:25.319284Z",
     "iopub.status.busy": "2020-11-02T02:26:25.318566Z",
     "iopub.status.idle": "2020-11-02T02:26:25.320448Z",
     "shell.execute_reply": "2020-11-02T02:26:25.320855Z"
    },
    "id": "BSV3PPKsYecw"
   },
   "outputs": [],
   "source": [
    "class MultiHeadAttention(tf.keras.layers.Layer):\n",
    "  def __init__(self, d_model, num_heads):\n",
    "    super(MultiHeadAttention, self).__init__()\n",
    "    self.num_heads = num_heads\n",
    "    self.d_model = d_model\n",
    "    \n",
    "    assert d_model % self.num_heads == 0\n",
    "    \n",
    "    self.depth = d_model // self.num_heads\n",
    "    \n",
    "    self.wq = tf.keras.layers.Dense(d_model)\n",
    "    self.wk = tf.keras.layers.Dense(d_model)\n",
    "    self.wv = tf.keras.layers.Dense(d_model)\n",
    "    \n",
    "    self.dense = tf.keras.layers.Dense(d_model)\n",
    "        \n",
    "  def split_heads(self, x, batch_size):\n",
    "    \"\"\"Split the last dimension into (num_heads, depth).\n",
    "    Transpose the result such that the shape is (batch_size, num_heads, seq_len, depth)\n",
    "    \"\"\"\n",
    "    x = tf.reshape(x, (batch_size, -1, self.num_heads, self.depth))\n",
    "    return tf.transpose(x, perm=[0, 2, 1, 3])\n",
    "    \n",
    "  def call(self, v, k, q, mask):\n",
    "    batch_size = tf.shape(q)[0]\n",
    "    \n",
    "    q = self.wq(q)  # (batch_size, seq_len, d_model)\n",
    "    k = self.wk(k)  # (batch_size, seq_len, d_model)\n",
    "    v = self.wv(v)  # (batch_size, seq_len, d_model)\n",
    "    \n",
    "    q = self.split_heads(q, batch_size)  # (batch_size, num_heads, seq_len_q, depth)\n",
    "    k = self.split_heads(k, batch_size)  # (batch_size, num_heads, seq_len_k, depth)\n",
    "    v = self.split_heads(v, batch_size)  # (batch_size, num_heads, seq_len_v, depth)\n",
    "    \n",
    "    # scaled_attention.shape == (batch_size, num_heads, seq_len_q, depth)\n",
    "    # attention_weights.shape == (batch_size, num_heads, seq_len_q, seq_len_k)\n",
    "    scaled_attention, attention_weights = scaled_dot_product_attention(\n",
    "        q, k, v, mask)\n",
    "    \n",
    "    scaled_attention = tf.transpose(scaled_attention, perm=[0, 2, 1, 3])  # (batch_size, seq_len_q, num_heads, depth)\n",
    "\n",
    "    concat_attention = tf.reshape(scaled_attention, \n",
    "                                  (batch_size, -1, self.d_model))  # (batch_size, seq_len_q, d_model)\n",
    "\n",
    "    output = self.dense(concat_attention)  # (batch_size, seq_len_q, d_model)\n",
    "        \n",
    "    return output, attention_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0D8FJue5lDyZ"
   },
   "source": [
    "Create a `MultiHeadAttention` layer to try out. At each location in the sequence, `y`, the `MultiHeadAttention` runs all 8 attention heads across all other locations in the sequence, returning a new vector of the same length at each location."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:25.326077Z",
     "iopub.status.busy": "2020-11-02T02:26:25.325340Z",
     "iopub.status.idle": "2020-11-02T02:26:25.359646Z",
     "shell.execute_reply": "2020-11-02T02:26:25.360072Z"
    },
    "id": "Hu94p-_-2_BX"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(TensorShape([1, 60, 512]), TensorShape([1, 8, 60, 60]))"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "temp_mha = MultiHeadAttention(d_model=512, num_heads=8)\n",
    "y = tf.random.uniform((1, 60, 512))  # (batch_size, encoder_sequence, d_model)\n",
    "out, attn = temp_mha(y, k=y, q=y, mask=None)\n",
    "out.shape, attn.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RdDqGayx67vv"
   },
   "source": [
    "## Point wise feed forward network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gBqzJXGfHK3X"
   },
   "source": [
    "Point wise feed forward network consists of two fully-connected layers with a ReLU activation in between."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:25.364864Z",
     "iopub.status.busy": "2020-11-02T02:26:25.364204Z",
     "iopub.status.idle": "2020-11-02T02:26:25.366219Z",
     "shell.execute_reply": "2020-11-02T02:26:25.366676Z"
    },
    "id": "ET7xLt0yCT6Z"
   },
   "outputs": [],
   "source": [
    "def point_wise_feed_forward_network(d_model, dff):\n",
    "  return tf.keras.Sequential([\n",
    "      tf.keras.layers.Dense(dff, activation='relu'),  # (batch_size, seq_len, dff)\n",
    "      tf.keras.layers.Dense(d_model)  # (batch_size, seq_len, d_model)\n",
    "  ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:25.373368Z",
     "iopub.status.busy": "2020-11-02T02:26:25.372687Z",
     "iopub.status.idle": "2020-11-02T02:26:25.437666Z",
     "shell.execute_reply": "2020-11-02T02:26:25.437066Z"
    },
    "id": "mytb1lPyOHLB"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([64, 50, 512])"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_ffn = point_wise_feed_forward_network(512, 2048)\n",
    "sample_ffn(tf.random.uniform((64, 50, 512))).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7e7hKcxn6-zd"
   },
   "source": [
    "## Encoder and decoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yScbC0MUH8dS"
   },
   "source": [
    "<img src=\"https://www.tensorflow.org/images/tutorials/transformer/transformer.png\" width=\"600\" alt=\"transformer\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MfYJG-Kvgwy2"
   },
   "source": [
    "The transformer model follows the same general pattern as a standard [sequence to sequence with attention model](nmt_with_attention.ipynb). \n",
    "\n",
    "* The input sentence is passed through `N` encoder layers that generates an output for each word/token in the sequence.\n",
    "* The decoder attends on the encoder's output and its own input (self-attention) to predict the next word. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QFv-FNYUmvpn"
   },
   "source": [
    "### Encoder layer\n",
    "\n",
    "Each encoder layer consists of sublayers:\n",
    "\n",
    "1.   Multi-head attention (with padding mask) \n",
    "2.    Point wise feed forward networks. \n",
    "\n",
    "Each of these sublayers has a residual connection around it followed by a layer normalization. Residual connections help in avoiding the vanishing gradient problem in deep networks.\n",
    "\n",
    "The output of each sublayer is `LayerNorm(x + Sublayer(x))`. The normalization is done on the `d_model` (last) axis. There are N encoder layers in the transformer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:25.446072Z",
     "iopub.status.busy": "2020-11-02T02:26:25.445347Z",
     "iopub.status.idle": "2020-11-02T02:26:25.447785Z",
     "shell.execute_reply": "2020-11-02T02:26:25.447205Z"
    },
    "id": "ncyS-Ms3i2x_"
   },
   "outputs": [],
   "source": [
    "class EncoderLayer(tf.keras.layers.Layer):\n",
    "  def __init__(self, d_model, num_heads, dff, rate=0.1):\n",
    "    super(EncoderLayer, self).__init__()\n",
    "\n",
    "    self.mha = MultiHeadAttention(d_model, num_heads)\n",
    "    self.ffn = point_wise_feed_forward_network(d_model, dff)\n",
    "\n",
    "    self.layernorm1 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n",
    "    self.layernorm2 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n",
    "    \n",
    "    self.dropout1 = tf.keras.layers.Dropout(rate)\n",
    "    self.dropout2 = tf.keras.layers.Dropout(rate)\n",
    "    \n",
    "  def call(self, x, training, mask):\n",
    "\n",
    "    attn_output, _ = self.mha(x, x, x, mask)  # (batch_size, input_seq_len, d_model)\n",
    "    attn_output = self.dropout1(attn_output, training=training)\n",
    "    out1 = self.layernorm1(x + attn_output)  # (batch_size, input_seq_len, d_model)\n",
    "    \n",
    "    ffn_output = self.ffn(out1)  # (batch_size, input_seq_len, d_model)\n",
    "    ffn_output = self.dropout2(ffn_output, training=training)\n",
    "    out2 = self.layernorm2(out1 + ffn_output)  # (batch_size, input_seq_len, d_model)\n",
    "    \n",
    "    return out2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:25.459795Z",
     "iopub.status.busy": "2020-11-02T02:26:25.459178Z",
     "iopub.status.idle": "2020-11-02T02:26:25.582504Z",
     "shell.execute_reply": "2020-11-02T02:26:25.582945Z"
    },
    "id": "AzZRXdO0mI48"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([64, 43, 512])"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_encoder_layer = EncoderLayer(512, 8, 2048)\n",
    "\n",
    "sample_encoder_layer_output = sample_encoder_layer(\n",
    "    tf.random.uniform((64, 43, 512)), False, None)\n",
    "\n",
    "sample_encoder_layer_output.shape  # (batch_size, input_seq_len, d_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6LO_48Owmx_o"
   },
   "source": [
    "### Decoder layer\n",
    "\n",
    "Each decoder layer consists of sublayers:\n",
    "\n",
    "1.   Masked multi-head attention (with look ahead mask and padding mask)\n",
    "2.   Multi-head attention (with padding mask). V (value) and K (key) receive the *encoder output* as inputs. Q (query) receives the *output from the masked multi-head attention sublayer.*\n",
    "3.   Point wise feed forward networks\n",
    "\n",
    "Each of these sublayers has a residual connection around it followed by a layer normalization. The output of each sublayer is `LayerNorm(x + Sublayer(x))`. The normalization is done on the `d_model` (last) axis.\n",
    "\n",
    "There are N decoder layers in the transformer.\n",
    "\n",
    "As Q receives the output from decoder's first attention block, and K receives the encoder output, the attention weights represent the importance given to the decoder's input based on the encoder's output. In other words, the decoder predicts the next word by looking at the encoder output and self-attending to its own output. See the demonstration above in the scaled dot product attention section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:25.592814Z",
     "iopub.status.busy": "2020-11-02T02:26:25.592125Z",
     "iopub.status.idle": "2020-11-02T02:26:25.593830Z",
     "shell.execute_reply": "2020-11-02T02:26:25.594250Z"
    },
    "id": "9SoX0-vd1hue"
   },
   "outputs": [],
   "source": [
    "class DecoderLayer(tf.keras.layers.Layer):\n",
    "  def __init__(self, d_model, num_heads, dff, rate=0.1):\n",
    "    super(DecoderLayer, self).__init__()\n",
    "\n",
    "    self.mha1 = MultiHeadAttention(d_model, num_heads)\n",
    "    self.mha2 = MultiHeadAttention(d_model, num_heads)\n",
    "\n",
    "    self.ffn = point_wise_feed_forward_network(d_model, dff)\n",
    " \n",
    "    self.layernorm1 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n",
    "    self.layernorm2 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n",
    "    self.layernorm3 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n",
    "    \n",
    "    self.dropout1 = tf.keras.layers.Dropout(rate)\n",
    "    self.dropout2 = tf.keras.layers.Dropout(rate)\n",
    "    self.dropout3 = tf.keras.layers.Dropout(rate)\n",
    "    \n",
    "    \n",
    "  def call(self, x, enc_output, training, \n",
    "           look_ahead_mask, padding_mask):\n",
    "    # enc_output.shape == (batch_size, input_seq_len, d_model)\n",
    "\n",
    "    attn1, attn_weights_block1 = self.mha1(x, x, x, look_ahead_mask)  # (batch_size, target_seq_len, d_model)\n",
    "    attn1 = self.dropout1(attn1, training=training)\n",
    "    out1 = self.layernorm1(attn1 + x)\n",
    "    \n",
    "    attn2, attn_weights_block2 = self.mha2(\n",
    "        enc_output, enc_output, out1, padding_mask)  # (batch_size, target_seq_len, d_model)\n",
    "    attn2 = self.dropout2(attn2, training=training)\n",
    "    out2 = self.layernorm2(attn2 + out1)  # (batch_size, target_seq_len, d_model)\n",
    "    \n",
    "    ffn_output = self.ffn(out2)  # (batch_size, target_seq_len, d_model)\n",
    "    ffn_output = self.dropout3(ffn_output, training=training)\n",
    "    out3 = self.layernorm3(ffn_output + out2)  # (batch_size, target_seq_len, d_model)\n",
    "    \n",
    "    return out3, attn_weights_block1, attn_weights_block2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:25.607195Z",
     "iopub.status.busy": "2020-11-02T02:26:25.606534Z",
     "iopub.status.idle": "2020-11-02T02:26:25.804635Z",
     "shell.execute_reply": "2020-11-02T02:26:25.805053Z"
    },
    "id": "Ne2Bqx8k71l0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([64, 50, 512])"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_decoder_layer = DecoderLayer(512, 8, 2048)\n",
    "\n",
    "sample_decoder_layer_output, _, _ = sample_decoder_layer(\n",
    "    tf.random.uniform((64, 50, 512)), sample_encoder_layer_output, \n",
    "    False, None, None)\n",
    "\n",
    "sample_decoder_layer_output.shape  # (batch_size, target_seq_len, d_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SE1H51Ajm0q1"
   },
   "source": [
    "### Encoder\n",
    "\n",
    "The `Encoder` consists of:\n",
    "1.   Input Embedding\n",
    "2.   Positional Encoding\n",
    "3.   N encoder layers\n",
    "\n",
    "The input is put through an embedding which is summed with the positional encoding. The output of this summation is the input to the encoder layers. The output of the encoder is the input to the decoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:25.814730Z",
     "iopub.status.busy": "2020-11-02T02:26:25.814013Z",
     "iopub.status.idle": "2020-11-02T02:26:25.816267Z",
     "shell.execute_reply": "2020-11-02T02:26:25.815698Z"
    },
    "id": "jpEox7gJ8FCI"
   },
   "outputs": [],
   "source": [
    "class Encoder(tf.keras.layers.Layer):\n",
    "  def __init__(self, num_layers, d_model, num_heads, dff, input_vocab_size,\n",
    "               maximum_position_encoding, rate=0.1):\n",
    "    super(Encoder, self).__init__()\n",
    "\n",
    "    self.d_model = d_model\n",
    "    self.num_layers = num_layers\n",
    "    \n",
    "    self.embedding = tf.keras.layers.Embedding(input_vocab_size, d_model)\n",
    "    self.pos_encoding = positional_encoding(maximum_position_encoding, \n",
    "                                            self.d_model)\n",
    "    \n",
    "    \n",
    "    self.enc_layers = [EncoderLayer(d_model, num_heads, dff, rate) \n",
    "                       for _ in range(num_layers)]\n",
    "  \n",
    "    self.dropout = tf.keras.layers.Dropout(rate)\n",
    "        \n",
    "  def call(self, x, training, mask):\n",
    "\n",
    "    seq_len = tf.shape(x)[1]\n",
    "    \n",
    "    # adding embedding and position encoding.\n",
    "    x = self.embedding(x)  # (batch_size, input_seq_len, d_model)\n",
    "    x *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))\n",
    "    x += self.pos_encoding[:, :seq_len, :]\n",
    "\n",
    "    x = self.dropout(x, training=training)\n",
    "    \n",
    "    for i in range(self.num_layers):\n",
    "      x = self.enc_layers[i](x, training, mask)\n",
    "    \n",
    "    return x  # (batch_size, input_seq_len, d_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:25.824870Z",
     "iopub.status.busy": "2020-11-02T02:26:25.824233Z",
     "iopub.status.idle": "2020-11-02T02:26:26.357891Z",
     "shell.execute_reply": "2020-11-02T02:26:26.358338Z"
    },
    "id": "8QG9nueFQKXx"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(64, 62, 512)\n"
     ]
    }
   ],
   "source": [
    "sample_encoder = Encoder(num_layers=2, d_model=512, num_heads=8, \n",
    "                         dff=2048, input_vocab_size=8500,\n",
    "                         maximum_position_encoding=10000)\n",
    "temp_input = tf.random.uniform((64, 62), dtype=tf.int64, minval=0, maxval=200)\n",
    "\n",
    "sample_encoder_output = sample_encoder(temp_input, training=False, mask=None)\n",
    "\n",
    "print (sample_encoder_output.shape)  # (batch_size, input_seq_len, d_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "p-uO6ls8m2O5"
   },
   "source": [
    "### Decoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZtT7PKzrXkNr"
   },
   "source": [
    " The `Decoder` consists of:\n",
    "1.   Output Embedding\n",
    "2.   Positional Encoding\n",
    "3.   N decoder layers\n",
    "\n",
    "The target is put through an embedding which is summed with the positional encoding. The output of this summation is the input to the decoder layers. The output of the decoder is the input to the final linear layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:26.368605Z",
     "iopub.status.busy": "2020-11-02T02:26:26.367591Z",
     "iopub.status.idle": "2020-11-02T02:26:26.369596Z",
     "shell.execute_reply": "2020-11-02T02:26:26.369997Z"
    },
    "id": "d5_d5-PLQXwY"
   },
   "outputs": [],
   "source": [
    "class Decoder(tf.keras.layers.Layer):\n",
    "  def __init__(self, num_layers, d_model, num_heads, dff, target_vocab_size,\n",
    "               maximum_position_encoding, rate=0.1):\n",
    "    super(Decoder, self).__init__()\n",
    "\n",
    "    self.d_model = d_model\n",
    "    self.num_layers = num_layers\n",
    "    \n",
    "    self.embedding = tf.keras.layers.Embedding(target_vocab_size, d_model)\n",
    "    self.pos_encoding = positional_encoding(maximum_position_encoding, d_model)\n",
    "    \n",
    "    self.dec_layers = [DecoderLayer(d_model, num_heads, dff, rate) \n",
    "                       for _ in range(num_layers)]\n",
    "    self.dropout = tf.keras.layers.Dropout(rate)\n",
    "    \n",
    "  def call(self, x, enc_output, training, \n",
    "           look_ahead_mask, padding_mask):\n",
    "\n",
    "    seq_len = tf.shape(x)[1]\n",
    "    attention_weights = {}\n",
    "    \n",
    "    x = self.embedding(x)  # (batch_size, target_seq_len, d_model)\n",
    "    x *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))\n",
    "    x += self.pos_encoding[:, :seq_len, :]\n",
    "    \n",
    "    x = self.dropout(x, training=training)\n",
    "\n",
    "    for i in range(self.num_layers):\n",
    "      x, block1, block2 = self.dec_layers[i](x, enc_output, training,\n",
    "                                             look_ahead_mask, padding_mask)\n",
    "      \n",
    "      attention_weights['decoder_layer{}_block1'.format(i+1)] = block1\n",
    "      attention_weights['decoder_layer{}_block2'.format(i+1)] = block2\n",
    "    \n",
    "    # x.shape == (batch_size, target_seq_len, d_model)\n",
    "    return x, attention_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:26.377767Z",
     "iopub.status.busy": "2020-11-02T02:26:26.376310Z",
     "iopub.status.idle": "2020-11-02T02:26:26.755762Z",
     "shell.execute_reply": "2020-11-02T02:26:26.755141Z"
    },
    "id": "a1jXoAMRZyvu"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(TensorShape([64, 26, 512]), TensorShape([64, 8, 26, 62]))"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_decoder = Decoder(num_layers=2, d_model=512, num_heads=8, \n",
    "                         dff=2048, target_vocab_size=8000,\n",
    "                         maximum_position_encoding=5000)\n",
    "temp_input = tf.random.uniform((64, 26), dtype=tf.int64, minval=0, maxval=200)\n",
    "\n",
    "output, attn = sample_decoder(temp_input, \n",
    "                              enc_output=sample_encoder_output, \n",
    "                              training=False,\n",
    "                              look_ahead_mask=None, \n",
    "                              padding_mask=None)\n",
    "\n",
    "output.shape, attn['decoder_layer2_block2'].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y54xnJnuYgJ7"
   },
   "source": [
    "## Create the Transformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uERO1y54cOKq"
   },
   "source": [
    "Transformer consists of the encoder, decoder and a final linear layer. The output of the decoder is the input to the linear layer and its output is returned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:26.763467Z",
     "iopub.status.busy": "2020-11-02T02:26:26.762728Z",
     "iopub.status.idle": "2020-11-02T02:26:26.764477Z",
     "shell.execute_reply": "2020-11-02T02:26:26.764889Z"
    },
    "id": "PED3bIpOYkBu"
   },
   "outputs": [],
   "source": [
    "class Transformer(tf.keras.Model):\n",
    "  def __init__(self, num_layers, d_model, num_heads, dff, input_vocab_size, \n",
    "               target_vocab_size, pe_input, pe_target, rate=0.1):\n",
    "    super(Transformer, self).__init__()\n",
    "\n",
    "    self.encoder = Encoder(num_layers, d_model, num_heads, dff, \n",
    "                           input_vocab_size, pe_input, rate)\n",
    "\n",
    "    self.decoder = Decoder(num_layers, d_model, num_heads, dff, \n",
    "                           target_vocab_size, pe_target, rate)\n",
    "\n",
    "    self.final_layer = tf.keras.layers.Dense(target_vocab_size)\n",
    "    \n",
    "  def call(self, inp, tar, training, enc_padding_mask, \n",
    "           look_ahead_mask, dec_padding_mask):\n",
    "\n",
    "    enc_output = self.encoder(inp, training, enc_padding_mask)  # (batch_size, inp_seq_len, d_model)\n",
    "    \n",
    "    # dec_output.shape == (batch_size, tar_seq_len, d_model)\n",
    "    dec_output, attention_weights = self.decoder(\n",
    "        tar, enc_output, training, look_ahead_mask, dec_padding_mask)\n",
    "    \n",
    "    final_output = self.final_layer(dec_output)  # (batch_size, tar_seq_len, target_vocab_size)\n",
    "    \n",
    "    return final_output, attention_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:26.772608Z",
     "iopub.status.busy": "2020-11-02T02:26:26.771855Z",
     "iopub.status.idle": "2020-11-02T02:26:27.696061Z",
     "shell.execute_reply": "2020-11-02T02:26:27.696483Z"
    },
    "id": "tJ4fbQcIkHW1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([64, 36, 8000])"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_transformer = Transformer(\n",
    "    num_layers=2, d_model=512, num_heads=8, dff=2048, \n",
    "    input_vocab_size=8500, target_vocab_size=8000, \n",
    "    pe_input=10000, pe_target=6000)\n",
    "\n",
    "temp_input = tf.random.uniform((64, 38), dtype=tf.int64, minval=0, maxval=200)\n",
    "temp_target = tf.random.uniform((64, 36), dtype=tf.int64, minval=0, maxval=200)\n",
    "\n",
    "fn_out, _ = sample_transformer(temp_input, temp_target, training=False, \n",
    "                               enc_padding_mask=None, \n",
    "                               look_ahead_mask=None,\n",
    "                               dec_padding_mask=None)\n",
    "\n",
    "fn_out.shape  # (batch_size, tar_seq_len, target_vocab_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wsINyf1VEQLC"
   },
   "source": [
    "## Set hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zVjWCxFNcgbt"
   },
   "source": [
    "To keep this example small and relatively fast, the values for *num_layers, d_model, and dff* have been reduced. \n",
    "\n",
    "The values used in the base model of transformer were; *num_layers=6*, *d_model = 512*, *dff = 2048*. See the [paper](https://arxiv.org/abs/1706.03762) for all the other versions of the transformer.\n",
    "\n",
    "Note: By changing the values below, you can get the model that achieved state of the art on many tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:27.701337Z",
     "iopub.status.busy": "2020-11-02T02:26:27.700717Z",
     "iopub.status.idle": "2020-11-02T02:26:27.702463Z",
     "shell.execute_reply": "2020-11-02T02:26:27.702871Z"
    },
    "id": "lnJn5SLA2ahP"
   },
   "outputs": [],
   "source": [
    "num_layers = 4\n",
    "d_model = 128\n",
    "dff = 512\n",
    "num_heads = 8\n",
    "\n",
    "input_vocab_size = tokenizer_pt.vocab_size + 2\n",
    "target_vocab_size = tokenizer_en.vocab_size + 2\n",
    "dropout_rate = 0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xYEGhEOtzn5W"
   },
   "source": [
    "## Optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GOmWW--yP3zx"
   },
   "source": [
    "Use the Adam optimizer with a custom learning rate scheduler according to the formula in the [paper](https://arxiv.org/abs/1706.03762).\n",
    "\n",
    "$$\\Large{lrate = d_{model}^{-0.5} * min(step{\\_}num^{-0.5}, step{\\_}num * warmup{\\_}steps^{-1.5})}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:27.709107Z",
     "iopub.status.busy": "2020-11-02T02:26:27.708470Z",
     "iopub.status.idle": "2020-11-02T02:26:27.710049Z",
     "shell.execute_reply": "2020-11-02T02:26:27.710587Z"
    },
    "id": "iYQdOO1axwEI"
   },
   "outputs": [],
   "source": [
    "class CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):\n",
    "  def __init__(self, d_model, warmup_steps=4000):\n",
    "    super(CustomSchedule, self).__init__()\n",
    "    \n",
    "    self.d_model = d_model\n",
    "    self.d_model = tf.cast(self.d_model, tf.float32)\n",
    "\n",
    "    self.warmup_steps = warmup_steps\n",
    "    \n",
    "  def __call__(self, step):\n",
    "    arg1 = tf.math.rsqrt(step)\n",
    "    arg2 = step * (self.warmup_steps ** -1.5)\n",
    "    \n",
    "    return tf.math.rsqrt(self.d_model) * tf.math.minimum(arg1, arg2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:27.715008Z",
     "iopub.status.busy": "2020-11-02T02:26:27.714374Z",
     "iopub.status.idle": "2020-11-02T02:26:27.716089Z",
     "shell.execute_reply": "2020-11-02T02:26:27.716593Z"
    },
    "id": "7r4scdulztRx"
   },
   "outputs": [],
   "source": [
    "learning_rate = CustomSchedule(d_model)\n",
    "\n",
    "optimizer = tf.keras.optimizers.Adam(learning_rate, beta_1=0.9, beta_2=0.98, \n",
    "                                     epsilon=1e-9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:27.720844Z",
     "iopub.status.busy": "2020-11-02T02:26:27.720237Z",
     "iopub.status.idle": "2020-11-02T02:26:27.866519Z",
     "shell.execute_reply": "2020-11-02T02:26:27.865886Z"
    },
    "id": "f33ZCgvHpPdG"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 0, 'Train Step')"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEGCAYAAACtqQjWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU9fno8c+ThCQkIYGsLAESIIBBcYvUfaMKaiutxQr19qc/ablttZu9P6u3vdbrr/5+tZvWVmut4nZVoNRWbFXc6q5A3JBFIJmAEJZMAgQSIJDw3D/ONzCESTJJZjKTzPN+vfLKme/5nu95ZgJ5cs73nOeIqmKMMcaEQ0K0AzDGGNN/WFIxxhgTNpZUjDHGhI0lFWOMMWFjScUYY0zYJEU7gGjKzc3VoqKiaIdhjDF9yvvvv1+rqnnB1sV1UikqKqK8vDzaYRhjTJ8iIhvbW2env4wxxoSNJRVjjDFhY0nFGGNM2FhSMcYYEzaWVIwxxoRNRJOKiEwXkbUiUiEiNwdZnyIiC9z6pSJSFLDuFte+VkSmBbTPE5EaEVnZzj5/JCIqIrmReE/GGGPaF7GkIiKJwL3AJUApMFtEStt0mwPsVNVxwF3AnW7bUmAWMAmYDtznxgN4xLUF2+dI4GLgs7C+GWOMMSGJ5JHKFKBCVX2qegCYD8xo02cG8KhbXgRMFRFx7fNVtUlVq4AKNx6q+gawo5193gXcBPTLev6qysLlm2hoao52KMYYE1Qkk8oIYFPA682uLWgfVW0G6oGcELc9iojMAKpV9eNO+s0VkXIRKff7/aG8j5jx0aZd3PTXFfx40Ypoh2KMMUH1i4l6EUkD/jdwa2d9VfUBVS1T1bK8vKBVBmLWZzv2AvDSmu1RjsQYY4KLZFKpBkYGvC50bUH7iEgSkAXUhbhtoLFAMfCxiGxw/T8QkaE9iD/mVPobATjQfIhNLsEYY0wsiWRSWQ6UiEixiCTjTbwvbtNnMXCNW54JvKre840XA7Pc1WHFQAmwrL0dqeonqpqvqkWqWoR3uuwUVd0W3rcUXZX+BkS85edXbo1uMMYYE0TEkoqbI7kBWAKsARaq6ioRuV1ELnfdHgJyRKQCuBG42W27ClgIrAZeAK5X1RYAEXkKeBeYICKbRWROpN5DrPH5GzlvfB6Thmfy/Mp+lS+NMf1ERKsUq+pzwHNt2m4NWN4PXNnOtncAdwRpnx3Cfou6GmusO3RIqapt4MyxOZxWlM2vlqxla/0+hmUNjHZoxhhzWL+YqI8HW+r3sf/gIcbkpXPJ8d5U0Qt2tGKMiTGWVPoIn5ukH5uXwZi8DCYOHcQ/Vti8ijEmtlhS6SMq/Q0AjMlLB2DGSSN4f+NONtY1RjMsY4w5iiWVPsLnb2RQahJ5GSkAzDhpOCLw9w+3RDkyY4w5wpJKH1Hpb2BMXgbirikePnggpxfn8LcPN+NdhW2MMdFnSaWP8PkbGZubflTbl08ZwYa6vXy4aVeUojLGmKNZUukDGpqa2bZ7P2PzM45qv+T4oaQkJfC3DzoqNmCMMb3HkkofUOWu/BrT5khlUOoALiot4NkVW2hqbolGaMYYcxRLKn2Ar9a78qvtkQrAlWUj2bX3IC+usiKTxpjos6TSB1TWNJAgMDon7Zh154zLpXDIQJ5cas8lM8ZEnyWVPqCytpHCIWmkJCUesy4hQZg9ZRTv+urwuXtZjDEmWiyp9AGVNQ2MzUtvd/2VZYUkJQjzl29qt48xxvQGSyox7tAhZUNdI2Pyjp1PaZU/KJWLSgtY9P5mm7A3xkSVJZUY11pIcmwHSQVg9pRR7Gg8YEUmjTFRZUklxrU+7XFMB6e/AM4el0txbjrz3t5gd9gbY6LGkkqMa5187+xIJSFBuO6sIj7etIsPPtvZG6EZY8wxLKnEuEp/A4NSk8jNSO6071dOLSRr4AAefLOqFyIzxphjWVKJcT5/41GFJDuSlpzE7CmjWLJqG5t27O2F6Iwx5miWVGKcz9/Y4eXEbV1z5mgSRHjknQ2RC8oYY9oR0aQiItNFZK2IVIjIzUHWp4jIArd+qYgUBay7xbWvFZFpAe3zRKRGRFa2GetXIvKpiKwQkb+JyOBIvrfecLiQZCfzKYGGZQ3kssnDWLB8E/V7D0YwOmOMOVbEkoqIJAL3ApcApcBsESlt020OsFNVxwF3AXe6bUuBWcAkYDpwnxsP4BHX1tZLwPGqOhlYB9wS1jcUBVWHHyEc+pEKwLfOG0tDUzMPv2NzK8aY3hXJI5UpQIWq+lT1ADAfmNGmzwzgUbe8CJgq3uTBDGC+qjapahVQ4cZDVd8AdrTdmaq+qKrN7uV7QGG431BvO/II4dCPVACOG5bJRaUFzHurij377WjFGNN7IplURgCBdUM2u7agfVxCqAdyQty2I9cBzwdbISJzRaRcRMr9fn8Xhux9Pn/7hSQ7890Lx7F7fzOPv7cxApEZY0xw/W6iXkR+AjQDTwRbr6oPqGqZqpbl5eX1bnBdVOlvZGR28EKSnZlcOJjzxufx4JtV7D3Q3PkGxhgTBpFMKtXAyIDXha4taB8RSQKygLoQtz2GiFwLfAG4WvvBbeWV/oZjHszVFd+bOo4djQd44j0ri2+M6R2RTCrLgRIRKRaRZLyJ98Vt+iwGrnHLM4FXXTJYDMxyV4cVAyXAso52JiLTgZuAy1W1z9+kceiQUlXb2KUrv9o6dXQ255Tkct9rFey2uRVjTC+IWFJxcyQ3AEuANcBCVV0lIreLyOWu20NAjohUADcCN7ttVwELgdXAC8D1qtoCICJPAe8CE0Rks4jMcWP9ARgEvCQiH4nI/ZF6b72hetc+mpoPdXmSvq0fT5/Izr0H+fMbvjBFZowx7UuK5OCq+hzwXJu2WwOW9wNXtrPtHcAdQdpnt9N/XI+CjTG+2u5dTtzW8SOy+MLkYTz4ZhVfP2M0+YNSwxGeMcYE1e8m6vuLypruXU4czI8unsDBlkP84dWKHo9ljDEdsaQSo3y1oReS7ExxbjpXnTaSJ5d+xgZ3BGSMMZFgSSVGeTW/QiskGYrvTy0hJSmBn/9zTVjGM8aYYCypxKhKf0OnD+bqivzMVL47tYSX12zntbU1YRvXGGMCWVKJQQ1NzWzf3dSjy4mD+feziijOTef2Z1dzoPlQWMc2xhiwpBKTjjztMXxHKgApSYnc+sVSfLWNPGLFJo0xEWBJJQb5Dj+XPrxHKgAXTMhn6sR8fvfyerbV7w/7+MaY+GZJJQZV9qCQZChu/WIpLar8n2dW0g+q2RhjYogllRjk60EhyVCMzknnh58fz0urt/P8ym0R2YcxJj5ZUolBlf6GsE/StzXn7GKOH5HJrc+ssidEGmPCxpJKjGktJNmT6sShSEpM4M6vTGbn3gPc8dzqiO7LGBM/LKnEmNZCkmPzI3ukAjBpeBZzzx3DwvLN/MvuXTHGhIEllRhz+BHCET5SafX9qSVMKBjETYtWUNfQ1Cv7NMb0X5ZUYkwkLycOJnVAInfPOon6vQe55elP7GowY0yPWFKJMb7aBjLDVEgyVMcNy+Sm6RN4cfV2FpZv6rX9GmP6H0sqMaayppExYSwkGarrzirmzLE5/N9nVx++o98YY7rKkkqM8dVG/nLiYBIShN989URSkhL4zhMfsO9AS6/HYIzp+yypxJA9+w+yfXdTWKsTd8WwrIHcddVJrN2+h5/+3e62N8Z0nSWVGFIVpkcI98T5E/L57oUl/PWDzSxYbvMrxpiuiWhSEZHpIrJWRCpE5OYg61NEZIFbv1REigLW3eLa14rItID2eSJSIyIr24yVLSIvich6931IJN9bJFQerk7c+6e/An1/agnnlORy6+JVrKyuj2osxpi+JWJJRUQSgXuBS4BSYLaIlLbpNgfYqarjgLuAO922pcAsYBIwHbjPjQfwiGtr62bgFVUtAV5xr/sUn7+RBIFRESokGarEBOHuq04iNz2Zbz5WTs0eq2ZsjAlNJI9UpgAVqupT1QPAfGBGmz4zgEfd8iJgqniXPc0A5qtqk6pWARVuPFT1DWBHkP0FjvUo8KVwvpne4PM3MiqChSS7IicjhT9fU8auvQf55mPvs/+gTdwbYzoXyaQyAgg8Kb/ZtQXto6rNQD2QE+K2bRWo6la3vA0oCNZJROaKSLmIlPv9/lDeR6/xHiEc3VNfgSYNz+LuWSfx8aZd/MeiFTZxb4zpVL+cqFfvt1/Q34Cq+oCqlqlqWV5eXi9H1r4WV0gympP0wUybNJSbpk/g2Y+3cM8rFdEOxxgT4yKZVKqBkQGvC11b0D4ikgRkAXUhbtvWdhEZ5sYaBvSpColbXCHJWDpSafXt88ZyxSkjuOvldSy0K8KMMR2IZFJZDpSISLGIJONNvC9u02cxcI1bngm86o4yFgOz3NVhxUAJsKyT/QWOdQ3wTBjeQ6/p7UKSXSEi/OKKyZxTksvNT6/gpdXbox2SMSZGRSypuDmSG4AlwBpgoaquEpHbReRy1+0hIEdEKoAbcVdsqeoqYCGwGngBuF5VWwBE5CngXWCCiGwWkTlurF8AF4nIeuDz7nWf0VpIsjdK3ndHclIC9/+PUzmhcDA3PPkBy6qCXSthjIl3Es+Tr2VlZVpeXh7tMAD4yd8+4dmPt/Dxzy7u9bpfXbGj8QAz738H/54mFsw9g9LhmdEOyRjTy0TkfVUtC7auX07U90U+fyNj83u/kGRXZacn8/icz5GRksTVD77Hmq27ox2SMSaGWFKJEZX+Bsbkxuapr7ZGDB7IU988nZSkRK5+cClrt+2JdkjGmBhhSSUG7Nl/kJo90Ssk2R1Fuek8Nfd0BiQKX/vze6zbbonFGGNJJSYcnqSPwcuJO1Kcm85T3zydxAQvsdipMGOMJZUY4KttLSTZd45UWo3Jy+CpuaeTlJDAVX96l/c32lVhxsSzTpOKiIwXkVdaqwKLyGQR+WnkQ4sfPn8jiQkS9UKS3TU2L4NF3z6DnIwUrn5wKa+t7VP3nRpjwiiUI5U/A7cABwFUdQXejYwmTCr9DYwcMjAmCkl2V+GQNP7yrTMYk5vBNx8r59mPt0Q7JGNMFISSVNJUte3d7M2RCCZe+fyNfW4+JZjcjBTm/8/TOXnkEL43/0MeeKPSilAaE2dCSSq1IjIWV6BRRGYCWzvexISq5ZDiq23sU1d+dSQzdQCPzZnCpccP47+e+5T//bdPONhyKNphGWN6SVIIfa4HHgAmikg1UAVcHdGo4siWXfs4EKOFJLsrdUAiv599MkW5adz7r0o+27GX+64+layBA6IdmjEmwkI5UlFV/TyQB0xU1bND3M6EIFYeIRxuCQnCf0ybyK9mTmZZ1Q6uuO9tNtQ2RjssY0yEhZIc/gqgqo2q2nqH26LIhRRfKt09Kv3l9FdbV5aN5PE5n6Ou8QBf/MNbvGwVjo3p19pNKiIyUUS+AmSJyBUBX9cCqb0WYT/n8zeQNXAAOenJ0Q4lYk4fk8OzN5zN6Jw0vvFYOb95cS0th2wC35j+qKM5lQnAF4DBwBcD2vcA34xkUPHEe4RweswXkuypkdlpLPrWmdz6zEp+/2oFH2+u53dXncSQfpxMjYlH7SYVVX0GeEZEzlDVd3sxprji8zdyTknsPNY4klIHJPLLmSdy8qgh/OyZVVx2z5vcPetkphRnRzs0Y0yYhDKn8qGIXC8i94nIvNaviEcWB1oLSY7N75/zKe2ZPWUUi759BslJCcx64F1+++Jamu2yY2P6hVCSyuPAUGAa8Dre8+KtJG0YtBaS7Csl78NpcuFg/vG9c7jilELuebWCr/7pXTbt2BvtsIwxPRRKUhmnqv8HaFTVR4HLgM9FNqz40FpIclycHam0ykhJ4tdXnsg9s09m/fYGLv3dmyws32R34RvTh4WSVA6677tE5HggC8iPXEjxo7LGFZLMjs+k0uryE4fz3PfP4bhhmdy0aAXXPrycLbv2RTssY0w3hJJUHhCRIcBPgcXAauDOiEYVJ3y1DYzKTiM5ye4lHZmdxvy5p3PbF0tZVrWDaXe9wfxln9lRizF9TKe/zVT1QVXdqapvqOoYVc0Hng9lcBGZLiJrRaRCRG4Osj5FRBa49UtFpChg3S2ufa2ITOtsTBGZKiIfiMhHIvKWiIwLJcZoqqxpZExufB+lBEpIEK49q5glPziXSSMyufnpT/i3ecvYWGd34hvTV3SYVETkDBGZKSL57vVkEXkSeLuzgUUkEbgXuAQoBWaLSGmbbnOAnao6DrgLdwTk+s0CJgHTgftEJLGTMf8IXK2qJwFP4h1ZxayWQ0pVXf8pJBlOo3LSePIbp/OfXzqeDzbu5OK73uCeV9bT1NwS7dCMMZ3o6I76XwHzgK8A/xSRnwMvAkuBkhDGngJUqKpPVQ8A84EZbfrMAB51y4uAqeLdBTgDmK+qTapaBVS48ToaU4FMt5wFxPQDPVoLSfa3ml/hkpAgfP300bzyo/O5qLSA3760jul3v8mb6/3RDs0Y04GO7qi/DDhZVfe7OZVNwPGquiHEsUe4bVpt5tirxg73UdVmEakHclz7e222HeGW2xvzG8BzIrIP2A2cHiwoEZkLzAUYNWpUiG8l/CpcIcn+VJ04EoZmpfKHr53CVaf5ufWZVXz9oWVcNnkYP7n0OIYPHhjt8IwxbXR0+mu/qu4HUNWdwPouJJRo+CFwqaoWAg8Dvw3WSVUfUNUyVS3Ly4veneyt96j0xefSR8M5JXm88INz+NFF43l59XYu+PVr/ObFtTQ02fPijIklHR2pjBGRxQGviwNfq+rlnYxdDYwMeF3o2oL12SwiSXinreo62faYdhHJA05U1aWufQHwQifxRVWlKySZbbWvQpaSlMh3p5bw5VNG8Ksla/n9qxU8tWwTP7p4PF8tG0liQv+un2ZMX9BRUmk7//GbLo69HCgRkWK8hDAL+FqbPouBa4B3gZnAq6qqLnk9KSK/BYbjzeEsA6SdMXfiVVMer6rrgIuANV2Mt1f54qSQZCQUDknjd7NO5t/PKubn/1jNLU9/wqPvbOCWS4/j3JJc+0yNiaKOCkq+3pOB3RzJDcASIBGYp6qrROR2oFxVFwMPAY+LSAWwAy9J4PotxLsnphm4XlVbAIKN6dq/CfxVRA7hJZnrehJ/pFX6GzlvfHwUkoyUk0YO5i/fOoPnV27jv59fwzXzljGlKJsfXTyez43JiXZ4xsQlieeby8rKyrS8vLzX97tn/0FOuO1Fbpo+ge+cH/O30/QJTc0tLFy+id+/WkHNnibOHpfLjReP55RRQ6IdmjH9joi8r6plwdbZrdxRcGSS3q78CpeUpES+fkYRb9x0AT+97DjWbN3NFfe9w5xHlrOyuj7a4RkTNyypRMGR59LblV/hljogkW+cM4Y3brqA/5g2geUbdvCF37/FNfOWsdRXZ2VfjImwjibqARCRZ/FuLAxUD5QDf2q97NiEzue3QpKRlp6SxPUXjOPrZ4zm/723kYferOKqB97j1NFDuP6CsVwwId8m9I2JgFCOVHxAA/Bn97Ub73kq491r00WVfisk2VsyUwfwnfPH8fbNF3L7jElsq9/PdY+Uc8nv3uTvH1ZzoNkeDmZMOHV6pAKcqaqnBbx+VkSWq+ppIrIqUoH1Zz6/FZLsbakDEvm3M4qYPWUUiz/awh9fr+QHCz7iv59fw9dPH83XPjfa7hkyJgxC+VM5Q0QO1zNxy60zzAciElU/1lpIcmy+TdJHw4DEBL5yaiEv/uBcHv730xhfMIhfv7iOM/77FX68aAWfbtsd7RCN6dNCOVL5EfCWiFTi3XxYDHxHRNI5UgzShKh6p1dI0o5UoishQbhgQj4XTMhn/fY9PPzOBp7+YDMLyjdx5tgcvn76aD5fWsCARDtFaUxXdJpUVPU5ESkBJrqmtQGT83dHLLJ+qtI9QtiOVGJHScEg/uvLJ3DTtAk8tWwTj7+7gW8/8QG5GSl8tayQ2VNGMTI7LdphGtMnhHKkAnAqUOT6nygiqOpjEYuqH6uscdWJ7Ugl5gxOS+bb549l7rljeH1dDU8u/Yz7X6/kvtcqOackl69NGWVHL8Z0IpRLih8HxgIfAa1PSVLAkko3+GobGZxmhSRjWWKCcOHEAi6cWMCWXftYWL6JBcs3HT56ueKUEVxxyggmDs3sfDBj4kwoRyplQKnaXWNhUVnTwJhcKyTZVwwfPJAffH48N1wwjtfX+Xlq2SbmvVXFA2/4KB2WyRWnjGDGSSPIG5QS7VCNiQmhJJWVwFBga4RjiQu+Wisk2RclJSYw9bgCph5XQF1DE89+vIWnP6zm5/9cw38//ynnluRyxSmFXFRaQOqAxGiHa0zUhJJUcoHVIrIMaGptDOF5KqaN3fsP4t/TZDW/+ricjBSuPauYa88qZv32PTz9YTV/+6Ca7z71IRkpSXz+uHy+MHk454zPJSXJEoyJL6EkldsiHUS8aC0kOcZqfvUbJQWD+PH0ifyviyfwbmUdz368hRdWbePvH21hUEoSF00q4IuTh3PWuFyroGDiQiiXFPfouSrmCN/hQpJ2pNLfJCYIZ5fkcnZJLv/5peN5u7KWf67YypJV23j6g2oyU5OYNmkol5wwlDPH5topMtNvtZtUROQtVT1bRPZwdEFJAVRV7dKXLqr0N7hCknbPQ3+WnJRw+MbKO758PG+t9xLM8yu38Zf3N5OWnMh54/O4qLSACyfmMzjNrgQ0/UdHT348230f1Hvh9G8+f6MVkowzKUmJhyf4m5pbeLeyjhdXb+fl1dt5fuU2EhOEKUXZXDypgItKCygcYn9wmL4tpCc/ikgiUEBAElLVzyIYV6/o7Sc/XnzX64zKTuPBa07rvLPp1w4dUlZU1/Piqm28tHo7691NsROHDuK8CXmcPz6fsqIhdqOliUkdPfkxlJsfvwv8DNgOtNYJV2By2CKMAy2HlA11ezl/Qn60QzExICFBOGnkYE4aOZibpk+kqraRl1Zv41+f+pn3VhV/et1HRkoSZ43L4fwJ+Zw3Po/hgwdGO2xjOhXK1V/fByaoal1XBxeR6cDvgETgQVX9RZv1KXh35p8K1AFXqeoGt+4WYA7eXfzfU9UlHY0p3t2EPweudNv8UVXv6WrMkdJaSNKe9miCKc5NZ+65Y5l77lgampp5u6KW19b6eX1tDUtWbQdgfEEG50/I59ySPMqKhthkv4lJoSSVTXhPeuwSd8rsXuAiYDOwXEQWq+rqgG5zgJ2qOk5EZgF3AleJSCkwC5gEDAdeFpHxbpv2xrwWGAlMVNVDIhJThwStjxAeY1d+mU5kpHhXik2bNBRVZX1NA6+treG1tX4eftu7mz85KYFTRw3hrHE5nDkul8kjskiyU2UmBoSSVHzAayLyT46++fG3nWw3BahQVR+AiMwHZgCBSWUGR+6DWQT8wR1xzADmq2oTUCUiFW48Ohjz28DXVPWQi68mhPfWayrtcmLTDSLC+IJBjC8YdPgoZnnVDt6uqOXtyjp+/eI6eHEdg1KS+NyYbM4cm8uZ43KYUDDISgGZqAglqXzmvpLdV6hG4B3ltNoMfK69PqraLCL1QI5rf6/NtiPccntjjsU7yvky4Mc7Zba+bVAiMheYCzBq1Ki2qyOm0m+FJE3PZaQkccHEfC6Y6B2I1zU08Z5vB29X1vJORS0vr/H+lspJT+a0omxOK85mSlE2xw0bZEcypld0mFTcKazxqnp1L8XTEynAflUtE5ErgHnAOW07qeoDwAPgXf3VW8H5/A1W7t6EXU5GCpdNHsZlk4cBUL1rH29X1LLUt4NlG+p4YdU2ANKTEzll9BCmFGUzpTibE0cOtjkZExEdJhVVbRGR0SKSrKpdfXRwNd4cR6tC1xasz2YRSQKy8CbsO9q2vfbNwNNu+W/Aw12MN6J8tY2cb4UkTYSNGDyQr5aN5Ktl3n+TbfX7WbZhB8uq6lhetZPfvLQOgOTEBCYXZnFacTZlo4dw0sjB5GRYpWXTc6HOqbwtIouBxtbGEOZUlgMlIlKM94t/FvC1Nn0WA9cA7wIzgVdVVd2+nhSR3+JN1JcAy/Du5m9vzL8DFwBVwHnAuhDeW69oLSRpk/Smtw3NSuXyE4dz+YnDAdjZeIDyjTtZvmEHy6p28Oc3fPzxkHfAPio7jZNHDebkkYM5adQQSodl2o26pstCSSqV7isBCPnuejdHcgOwBO/y33mqukpEbgfKVXUx8BDwuJuI34GXJHD9FuJNwDcD16tqC0CwMd0ufwE8ISI/BBqAb4Qaa6S1FpK0y4lNtA1JT+aiUu/ufYC9B5pZWb2bDz/byYef7eI9Xx3PfLQF8MrNHD88k5NHDeHkUd49NSMGD7QLAEyHQrqjvr/qrTvq//r+Zn70l495+cbzGGfPpjcxbmv9Pj78bNfhRPNJdT1Nzd59z3mDUpg8IovjR2RxgvtekJliiSbO9PSO+jzgJrx7RlJb21X1wrBF2M/5aq2QpOk7hmUNZNgJA7n0BG/y/2DLIT7duocPN+3ko892saK6nlfX1tD692huRgonjMg8nGROKMxiaGaqJZo4FcrpryeABcAXgG/hzYH4IxlUf1NZ08hoKyRp+qgBiQmcUOgli387w2trbGpmzdbdfFJdzyfV9aysruf1dX7c9Aw56ckcPyKLScMzmTgsk9JhgyjKSbfLmuNAKEklR1UfEpHvu2ervC4iyyMdWH/iq22wB3OZfiU9JYmyomzKirIPt+094CWaldW7DyeatytqaXaZJiUpgfEFgzhu2CAmDs3kuGGZHDdskJX+72dCSSoH3fetInIZsAXI7qC/CdBySNlQu5cLrJCk6efSkpM4dXQ2p44+8uuhqbmFipoGPt26hzVbd/Pptj28sqaGheWbD/cZlpXKxKGDOG6Yd1QzoWAQxbnpdmTfR4WSVH4uIlnAj4DfA5nADyMaVT+yeedeDrQcsiMVE5dSkhKZNDyLScOzDrepKv6GpqMSzZqtu3lz/ZGjmsQEoSgnjfEFgyjJz6CkYBAlBRkU56aTkmQ3bcayUB4n/A+3WI93H4jpgiOXE9tVX8aAV88sf1Aq+YNSOTfghuADzYeoqGlgfc0e1m9vYN32Pazdtoclq7YdnqtJTBBG56QxPt9LMuPyMxjvjs56Mi0AABPqSURBVGysQkBsCOXqr/HAH4ECVT1eRCYDl6vqzyMeXT9g1YmNCU1yUgKlwzMpHX70k8r3H2yhqraRddv3UFHjJZt1NXt4ac12Wly2SRAYMWQgY3K9o5mxeekU52YwJi+doZmpJCTYlWi9JZTTX38G/gP4E4CqrhCRJ/GeXWI6YYUkjemZ1AGJblL/6GTT1Owlm/XbG6ioacBX20hVbQPlG3bQeKDlcL+BAxIpyk1nTG46Y/K8r9aEk5k6oLffTr8XSlJJU9Vlba45b45QPP2Oz99gp76MiYCUpEQmDs1k4tCjk42qUrOniUp/A1W1jfj8jVTVNrJqSz0vrNp2+OgGIDcjmTG5GYzOSWN0ThqjctIZne0t21Vp3RNKUqkVkbF4jxBGRGYCWyMaVT9S6W/kgglWSNKY3iIiFGSmUpCZypljc49ad6D5EJ/t2IsvIOH4aht4fZ2fmj1NR/XNTE1idE46o3LSDieaUdnpjM5Js1NqHQglqVyPVyp+oohU4xVs7Aul8KOuft9BahuaGGulWYyJCclJCYzLzwhaLmnfgRY+27GXjXWN7vteNu7Yy8rqepas3Hb4yrTWcUYOGUiRSzqFQ9IoHDLQfaWRNTB+T6uFcvWXD/i8iKQDCaq6R0R+ANwd8ej6OF/rJL09R8WYmDcwOZEJQwcxYeixdXObWw6xZdd+Nu5oZGPd3sPJZ2PdXt711bE3YA4HYFBKEiNcgjmSbI68zho4oN+WsQnlSAUAVW0MeHkjllQ61Xo5sV35ZUzflpSYwKicNEblpHFOydHrVJWdew9SvXMfm3fuZbP7Xr3L+/6er46GpqOnodOTE49KOK0JaFhWKsMHDyQ3I4XEPnp6LeSk0kbffLe9rNLfQJK7rt4Y0z+JCNnpyWSnJ3NCYdYx61WV3fua2XRMwvG+lm3YwZ79RyedpARvXmhYVipDXaIZlpXqvgYybHAquekpMTmv092kEr/18rvA529kVHYaA6yInjFxS0TIShtAVppXxTmY+n3ekc7W+n1srd/vfd+1ny31+1hZXc+Lq7dzwD1+oNWARC/xDHdJZmiWW3aJZ2hWKjnpyb2eeNpNKiKyh+DJQ4CBEYuoH/EKSdqpL2NMx7IGDiBr4IBjbvxsparsaDzgEo6XdLbs2s+2+n1sqd/PB5/tZFv9fg62HP0re0CiV72gIDOFoVleFYOhWakMzUzlzLE55GemBt1fT7SbVFQ15Kc8mmNZIUljTLiICDkZKeRkpLR7tHPokFLXeOBwwtm+ez/bdu9ne733/dNte3h9rf/wjaGPXTeld5OK6ZnWQpJ246MxpjckJAh5g1K8p3MWtt+voamZbfX7GZYV/oQCllQi5kjNL7uc2BgTOzJSkiL6WPOIziCLyHQRWSsiFSJyc5D1KSKywK1fKiJFAetuce1rRWRaF8a8R0QaIvWeQmWXExtj4lHEkoqIJAL3ApcApcBsESlt020OsFNVxwF3AXe6bUuBWcAkYDpwn4gkdjamiJQBQyL1nrqi0t/IECskaYyJM5E8UpkCVKiqT1UPAPOBGW36zAAedcuLgKni3WY6A5ivqk2qWgVUuPHaHdMlnF8BN0XwPYWs0m9Xfhlj4k8kk8oIYFPA682uLWgfVW3GexBYTgfbdjTmDcBiVe2w2KWIzBWRchEp9/v9XXpDXeHzNzLW5lOMMXGmX9yVJyLDgSvxHnfcIVV9QFXLVLUsLy8y1YNbC0nakYoxJt5EMqlUAyMDXhe6tqB9RCQJyALqOti2vfaTgXFAhYhsANJEpCJcb6SrrJCkMSZeRTKpLAdKRKRYRJLxJt4Xt+mzGLjGLc8EXlVVde2z3NVhxUAJsKy9MVX1n6o6VFWLVLUI2Osm/6OisvW59Fby3hgTZyJ2n4qqNovIDcASIBGYp6qrROR2oFxVFwMPAY+7o4odeEkC128hsBrvKZPXq2oLQLAxI/UeusvnCkmOyrZCksaY+BLRmx9V9TnguTZttwYs78ebCwm27R3AHaGMGaRPVA8RfP5GRuVYIUljTPyx33oRUOlvYEyunfoyxsQfSyph1txyiI11exmbb5P0xpj4Y0klzDbv3OcVkrQjFWNMHLKkEma+WiskaYyJX5ZUwqy1kKSVvDfGxCNLKmFW6W9gSNoAhlghSWNMHLKkEmaV/kY7SjHGxC1LKmHm8zfYfIoxJm5ZUgmj+r0HqW04YIUkjTFxy5JKGFW6K7/s9JcxJl5ZUgmjI48QttNfxpj4ZEkljKyQpDEm3llSCaNKf4MVkjTGxDX77RdGPruc2BgT5yyphElzyyE21DXafIoxJq5ZUgmTzTv3cbBFrZCkMSauWVIJk9ZCklby3hgTzyyphElljbuc2I5UjDFxzJJKmPhqG8hOT7ZCksaYuBbRpCIi00VkrYhUiMjNQdaniMgCt36piBQFrLvFta8VkWmdjSkiT7j2lSIyT0QGRPK9tVVZ08iYXDv1ZYyJbxFLKiKSCNwLXAKUArNFpLRNtznATlUdB9wF3Om2LQVmAZOA6cB9IpLYyZhPABOBE4CBwDci9d6C8dVaIUljjInkkcoUoEJVfap6AJgPzGjTZwbwqFteBEwVEXHt81W1SVWrgAo3Xrtjqupz6gDLgMIIvrejtBaStHtUjDHxLpJJZQSwKeD1ZtcWtI+qNgP1QE4H23Y6pjvt9XXghR6/gxBVHn6EsCUVY0x8648T9fcBb6jqm8FWishcESkXkXK/3x+WHR55hLCd/jLGxLdIJpVqYGTA60LXFrSPiCQBWUBdB9t2OKaI/AzIA25sLyhVfUBVy1S1LC8vr4tvKbhKV0hypBWSNMbEuUgmleVAiYgUi0gy3sT74jZ9FgPXuOWZwKtuTmQxMMtdHVYMlODNk7Q7poh8A5gGzFbVQxF8X8fw+RsYbYUkjTGGpEgNrKrNInIDsARIBOap6ioRuR0oV9XFwEPA4yJSAezASxK4fguB1UAzcL2qtgAEG9Pt8n5gI/CuN9fP06p6e6TeX6BKf6PNpxhjDBFMKuBdkQU816bt1oDl/cCV7Wx7B3BHKGO69oi+l/Y0txxiY10jU4/Lj8bujTEmptj5mh46XEjSjlSMMcaSSk9V+lufS29XfhljjCWVHjr8XHorJGmMMZZUeqrSb4UkjTGmlSWVHvL5rZCkMca0sqTSQ5X+BpukN8YYx5JKD9TvPUhd4wGrTmyMMY4llR5oLSRpRyrGGOOxpNIDlTWt1YntSMUYY8CSSo/4ahsZkGiFJI0xppUllR6orGlgVLYVkjTGmFb227AHfLVWSNIYYwJZUumm1kKSNklvjDFHWFLppk2ukKRN0htjzBGWVLrJ57fLiY0xpi1LKt1k1YmNMeZYllS6yedvJCc9mcFpVkjSGGNaWVLppkp/g82nGGNMG5ZUusmrTmzzKcYYE8iSSjfs2nuAusYDjM23IxVjjAkU0aQiItNFZK2IVIjIzUHWp4jIArd+qYgUBay7xbWvFZFpnY0pIsVujAo3ZsQmOyrtaY/GGBNUxJKKiCQC9wKXAKXAbBEpbdNtDrBTVccBdwF3um1LgVnAJGA6cJ+IJHYy5p3AXW6snW7siDh8OXG+JRVjjAkUySOVKUCFqvpU9QAwH5jRps8M4FG3vAiYKiLi2uerapOqVgEVbrygY7ptLnRj4Mb8UqTeWKXfFZIcMjBSuzDGmD4pkkllBLAp4PVm1xa0j6o2A/VATgfbtteeA+xyY7S3LwBEZK6IlItIud/v78bbgqKcNL588giSrJCkMcYcJe5+K6rqA6papqpleXl53Rpj1pRR/HLmiWGOzBhj+r5IJpVqYGTA60LXFrSPiCQBWUBdB9u2114HDHZjtLcvY4wxERbJpLIcKHFXZSXjTbwvbtNnMXCNW54JvKqq6tpnuavDioESYFl7Y7pt/uXGwI35TATfmzHGmCCSOu/SParaLCI3AEuARGCeqq4SkduBclVdDDwEPC4iFcAOvCSB67cQWA00A9eragtAsDHdLn8MzBeRnwMfurGNMcb0IvH+yI9PZWVlWl5eHu0wjDGmTxGR91W1LNi6uJuoN8YYEzmWVIwxxoSNJRVjjDFhY0nFGGNM2MT1RL2I+IGN3dw8F6gNYzjhYnF1jcXVNRZX18RqXNCz2EaratC7x+M6qfSEiJS3d/VDNFlcXWNxdY3F1TWxGhdELjY7/WWMMSZsLKkYY4wJG0sq3fdAtANoh8XVNRZX11hcXROrcUGEYrM5FWOMMWFjRyrGGGPCxpKKMcaYsLGk0g0iMl1E1opIhYjc3Av72yAin4jIRyJS7tqyReQlEVnvvg9x7SIi97jYVojIKQHjXOP6rxeRa9rbXyexzBORGhFZGdAWtlhE5FT3XivcttKDuG4TkWr3uX0kIpcGrLvF7WOtiEwLaA/6s3WPW1jq2he4Ry90FtNIEfmXiKwWkVUi8v1Y+Lw6iCuqn5fbLlVElonIxy62/9vReOI9HmOBa18qIkXdjbmbcT0iIlUBn9lJrr03/+0nisiHIvKPWPisUFX76sIXXsn9SmAMkAx8DJRGeJ8bgNw2bb8EbnbLNwN3uuVLgecBAU4Hlrr2bMDnvg9xy0O6Ecu5wCnAykjEgvfcnNPdNs8Dl/QgrtuA/xWkb6n7uaUAxe7nmdjRzxZYCMxyy/cD3w4hpmHAKW55ELDO7Tuqn1cHcUX183J9BchwywOApe79BR0P+A5wv1ueBSzobszdjOsRYGaQ/r35b/9G4EngHx199r31WdmRStdNASpU1aeqB4D5wIwoxDEDeNQtPwp8KaD9MfW8h/dEzGHANOAlVd2hqjuBl4DpXd2pqr6B9+ybsMfi1mWq6nvq/Wt/LGCs7sTVnhnAfFVtUtUqoALv5xr0Z+v+YrwQWBTkPXYU01ZV/cAt7wHWACOI8ufVQVzt6ZXPy8WjqtrgXg5wX9rBeIGf5SJgqtt/l2LuQVzt6ZWfpYgUApcBD7rXHX32vfJZWVLpuhHApoDXm+n4P2Q4KPCiiLwvInNdW4GqbnXL24CCTuKLZNzhimWEWw5njDe40w/zxJ1m6kZcOcAuVW3ublzuVMPJeH/hxszn1SYuiIHPy53O+QiowfulW9nBeIdjcOvr3f7D/v+gbVyq2vqZ3eE+s7tEJKVtXCHuv7s/y7uBm4BD7nVHn32vfFaWVPqGs1X1FOAS4HoROTdwpfvLJiauDY+lWIA/AmOBk4CtwG+iEYSIZAB/BX6gqrsD10Xz8woSV0x8XqraoqonAYV4fy1PjEYcbbWNS0SOB27Bi+80vFNaP+6teETkC0CNqr7fW/sMhSWVrqsGRga8LnRtEaOq1e57DfA3vP9o290hM+57TSfxRTLucMVS7ZbDEqOqbne/CA4Bf8b73LoTVx3e6YukNu2dEpEBeL+4n1DVp11z1D+vYHHFwucVSFV3Af8CzuhgvMMxuPVZbv8R+38QENd0dypRVbUJeJjuf2bd+VmeBVwuIhvwTk1dCPyOaH9WnU262Ncxk2JJeJNrxRyZvJoUwf2lA4MClt/Bmwv5FUdP9v7SLV/G0ROEy1x7NlCFNzk4xC1ndzOmIo6eEA9bLBw7WXlpD+IaFrD8Q7zzxgCTOHpi0oc3Kdnuzxb4C0dPfn4nhHgE79z43W3ao/p5dRBXVD8v1zcPGOyWBwJvAl9obzzgeo6efF7Y3Zi7GdewgM/0buAXUfq3fz5HJuqj+1l155dKvH/hXdmxDu9c708ivK8x7of5MbCqdX9450JfAdYDLwf8wxTgXhfbJ0BZwFjX4U3CVQD/3s14nsI7NXIQ7xzrnHDGApQBK902f8BVfehmXI+7/a4AFnP0L82fuH2sJeAqm/Z+tu7nsMzF+xcgJYSYzsY7tbUC+Mh9XRrtz6uDuKL6ebntJgMfuhhWArd2NB6Q6l5XuPVjuhtzN+N61X1mK4H/x5ErxHrt377b9nyOJJWoflZWpsUYY0zY2JyKMcaYsLGkYowxJmwsqRhjjAkbSyrGGGPCxpKKMcaYsLGkYkwXiUhOQFXabXJ0Zd8Oq/GKSJmI3NPF/V3nqteuEJGVIjLDtV8rIsN78l6MCTe7pNiYHhCR24AGVf11QFuSHqm91NPxC4HX8aoK17vSKnmqWiUir+FVFS4Px76MCQc7UjEmDNxzNe4XkaXAL0Vkioi8655z8Y6ITHD9zg947sVtrnDjayLiE5HvBRk6H9gDNACoaoNLKDPxbpZ7wh0hDXTP43jdFR5dElAK5jUR+Z3rt1JEpgTZjzFhYUnFmPApBM5U1RuBT4FzVPVk4Fbgv9rZZiJeOfQpwM9cTa5AHwPbgSoReVhEvgigqouAcuBq9YocNgO/x3u2x6nAPOCOgHHSXL/vuHXGRERS512MMSH6i6q2uOUs4FERKcEridI2WbT6p3rFCJtEpAavDP7hEuiq2iIi0/Gq4E4F7hKRU1X1tjbjTACOB17yHpFBIl7ZmlZPufHeEJFMERmsXmFEY8LKkoox4dMYsPyfwL9U9cvumSWvtbNNU8ByC0H+T6o38bkMWCYiL+FVw72tTTcBVqnqGe3sp+3kqU2mmoiw01/GREYWR8qEX9vdQURkuAQ83xzvWScb3fIevMcBg1cIME9EznDbDRCRSQHbXeXazwbqVbW+uzEZ0xE7UjEmMn6Jd/rrp8A/ezDOAODX7tLh/YAf+JZb9whwv4jsw3vmyEzgHhHJwvu/fTdeZWuA/SLyoRvvuh7EY0yH7JJiY/o5u/TY9CY7/WWMMSZs7EjFGGNM2NiRijHGmLCxpGKMMSZsLKkYY4wJG0sqxhhjwsaSijHGmLD5/8mvsInwVResAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "temp_learning_rate_schedule = CustomSchedule(d_model)\n",
    "\n",
    "plt.plot(temp_learning_rate_schedule(tf.range(40000, dtype=tf.float32)))\n",
    "plt.ylabel(\"Learning Rate\")\n",
    "plt.xlabel(\"Train Step\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YgkDE7hzo8r5"
   },
   "source": [
    "## Loss and metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oxGJtoDuYIHL"
   },
   "source": [
    "Since the target sequences are padded, it is important to apply a padding mask when calculating the loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:27.870775Z",
     "iopub.status.busy": "2020-11-02T02:26:27.870132Z",
     "iopub.status.idle": "2020-11-02T02:26:27.871766Z",
     "shell.execute_reply": "2020-11-02T02:26:27.872206Z"
    },
    "id": "MlhsJMm0TW_B"
   },
   "outputs": [],
   "source": [
    "loss_object = tf.keras.losses.SparseCategoricalCrossentropy(\n",
    "    from_logits=True, reduction='none')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:27.879108Z",
     "iopub.status.busy": "2020-11-02T02:26:27.878422Z",
     "iopub.status.idle": "2020-11-02T02:26:27.880191Z",
     "shell.execute_reply": "2020-11-02T02:26:27.880573Z"
    },
    "id": "67oqVHiT0Eiu"
   },
   "outputs": [],
   "source": [
    "def loss_function(real, pred):\n",
    "  mask = tf.math.logical_not(tf.math.equal(real, 0))\n",
    "  loss_ = loss_object(real, pred)\n",
    "\n",
    "  mask = tf.cast(mask, dtype=loss_.dtype)\n",
    "  loss_ *= mask\n",
    "  \n",
    "  return tf.reduce_sum(loss_)/tf.reduce_sum(mask)\n",
    "\n",
    "\n",
    "def accuracy_function(real, pred):\n",
    "  accuracies = tf.equal(real, tf.argmax(pred, axis=2))\n",
    "  \n",
    "  mask = tf.math.logical_not(tf.math.equal(real, 0))\n",
    "  accuracies = tf.math.logical_and(mask, accuracies)\n",
    "\n",
    "  accuracies = tf.cast(accuracies, dtype=tf.float32)\n",
    "  mask = tf.cast(mask, dtype=tf.float32)\n",
    "  return tf.reduce_sum(accuracies)/tf.reduce_sum(mask)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:27.890929Z",
     "iopub.status.busy": "2020-11-02T02:26:27.890255Z",
     "iopub.status.idle": "2020-11-02T02:26:27.898638Z",
     "shell.execute_reply": "2020-11-02T02:26:27.899073Z"
    },
    "id": "phlyxMnm-Tpx"
   },
   "outputs": [],
   "source": [
    "train_loss = tf.keras.metrics.Mean(name='train_loss')\n",
    "train_accuracy = tf.keras.metrics.Mean(name='train_accuracy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aeHumfr7zmMa"
   },
   "source": [
    "## Training and checkpointing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:27.904511Z",
     "iopub.status.busy": "2020-11-02T02:26:27.903860Z",
     "iopub.status.idle": "2020-11-02T02:26:28.059188Z",
     "shell.execute_reply": "2020-11-02T02:26:28.059633Z"
    },
    "id": "UiysUa--4tOU"
   },
   "outputs": [],
   "source": [
    "transformer = Transformer(num_layers, d_model, num_heads, dff,\n",
    "                          input_vocab_size, target_vocab_size, \n",
    "                          pe_input=input_vocab_size, \n",
    "                          pe_target=target_vocab_size,\n",
    "                          rate=dropout_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:28.065224Z",
     "iopub.status.busy": "2020-11-02T02:26:28.064487Z",
     "iopub.status.idle": "2020-11-02T02:26:28.066163Z",
     "shell.execute_reply": "2020-11-02T02:26:28.066631Z"
    },
    "id": "ZOJUSB1T8GjM"
   },
   "outputs": [],
   "source": [
    "def create_masks(inp, tar):\n",
    "  # Encoder padding mask\n",
    "  enc_padding_mask = create_padding_mask(inp)\n",
    "  \n",
    "  # Used in the 2nd attention block in the decoder.\n",
    "  # This padding mask is used to mask the encoder outputs.\n",
    "  dec_padding_mask = create_padding_mask(inp)\n",
    "  \n",
    "  # Used in the 1st attention block in the decoder.\n",
    "  # It is used to pad and mask future tokens in the input received by \n",
    "  # the decoder.\n",
    "  look_ahead_mask = create_look_ahead_mask(tf.shape(tar)[1])\n",
    "  dec_target_padding_mask = create_padding_mask(tar)\n",
    "  combined_mask = tf.maximum(dec_target_padding_mask, look_ahead_mask)\n",
    "  \n",
    "  return enc_padding_mask, combined_mask, dec_padding_mask"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Fzuf06YZp66w"
   },
   "source": [
    "Create the checkpoint path and the checkpoint manager. This will be used to save checkpoints every `n` epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:28.072057Z",
     "iopub.status.busy": "2020-11-02T02:26:28.071394Z",
     "iopub.status.idle": "2020-11-02T02:26:28.073614Z",
     "shell.execute_reply": "2020-11-02T02:26:28.073034Z"
    },
    "id": "hNhuYfllndLZ"
   },
   "outputs": [],
   "source": [
    "checkpoint_path = \"./checkpoints/train\"\n",
    "\n",
    "ckpt = tf.train.Checkpoint(transformer=transformer,\n",
    "                           optimizer=optimizer)\n",
    "\n",
    "ckpt_manager = tf.train.CheckpointManager(ckpt, checkpoint_path, max_to_keep=5)\n",
    "\n",
    "# if a checkpoint exists, restore the latest checkpoint.\n",
    "if ckpt_manager.latest_checkpoint:\n",
    "  ckpt.restore(ckpt_manager.latest_checkpoint)\n",
    "  print ('Latest checkpoint restored!!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0Di_Yaa1gf9r"
   },
   "source": [
    "The target is divided into tar_inp and tar_real. tar_inp is passed as an input to the decoder. `tar_real` is that same input shifted by 1: At each location in `tar_input`, `tar_real` contains the  next token that should be predicted.\n",
    "\n",
    "For example, `sentence` = \"SOS A lion in the jungle is sleeping EOS\"\n",
    "\n",
    "`tar_inp` =  \"SOS A lion in the jungle is sleeping\"\n",
    "\n",
    "`tar_real` = \"A lion in the jungle is sleeping EOS\"\n",
    "\n",
    "The transformer is an auto-regressive model: it makes predictions one part at a time, and uses its output so far to decide what to do next. \n",
    "\n",
    "During training this example uses teacher-forcing (like in the [text generation tutorial](./text_generation.ipynb)). Teacher forcing is passing the true output to the next time step regardless of what the model predicts at the current time step.\n",
    "\n",
    "As the transformer predicts each word, *self-attention* allows it to look at the previous words in the input sequence to better predict the next word.\n",
    "\n",
    "To prevent the model from peeking at the expected output the model uses a look-ahead mask."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:28.077441Z",
     "iopub.status.busy": "2020-11-02T02:26:28.076745Z",
     "iopub.status.idle": "2020-11-02T02:26:28.078903Z",
     "shell.execute_reply": "2020-11-02T02:26:28.078389Z"
    },
    "id": "LKpoA6q1sJFj"
   },
   "outputs": [],
   "source": [
    "EPOCHS = 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:28.086043Z",
     "iopub.status.busy": "2020-11-02T02:26:28.085376Z",
     "iopub.status.idle": "2020-11-02T02:26:28.087078Z",
     "shell.execute_reply": "2020-11-02T02:26:28.087521Z"
    },
    "id": "iJwmp9OE29oj"
   },
   "outputs": [],
   "source": [
    "# The @tf.function trace-compiles train_step into a TF graph for faster\n",
    "# execution. The function specializes to the precise shape of the argument\n",
    "# tensors. To avoid re-tracing due to the variable sequence lengths or variable\n",
    "# batch sizes (the last batch is smaller), use input_signature to specify\n",
    "# more generic shapes.\n",
    "\n",
    "train_step_signature = [\n",
    "    tf.TensorSpec(shape=(None, None), dtype=tf.int64),\n",
    "    tf.TensorSpec(shape=(None, None), dtype=tf.int64),\n",
    "]\n",
    "\n",
    "@tf.function(input_signature=train_step_signature)\n",
    "def train_step(inp, tar):\n",
    "  tar_inp = tar[:, :-1]\n",
    "  tar_real = tar[:, 1:]\n",
    "  \n",
    "  enc_padding_mask, combined_mask, dec_padding_mask = create_masks(inp, tar_inp)\n",
    "  \n",
    "  with tf.GradientTape() as tape:\n",
    "    predictions, _ = transformer(inp, tar_inp, \n",
    "                                 True, \n",
    "                                 enc_padding_mask, \n",
    "                                 combined_mask, \n",
    "                                 dec_padding_mask)\n",
    "    loss = loss_function(tar_real, predictions)\n",
    "\n",
    "  gradients = tape.gradient(loss, transformer.trainable_variables)    \n",
    "  optimizer.apply_gradients(zip(gradients, transformer.trainable_variables))\n",
    "  \n",
    "  train_loss(loss)\n",
    "  train_accuracy(accuracy_function(tar_real, predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qM2PDWGDJ_8V"
   },
   "source": [
    "Portuguese is used as the input language and English is the target language."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T02:26:28.094989Z",
     "iopub.status.busy": "2020-11-02T02:26:28.094309Z",
     "iopub.status.idle": "2020-11-02T04:16:37.754510Z",
     "shell.execute_reply": "2020-11-02T04:16:37.753863Z"
    },
    "id": "bbvmaKNiznHZ"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 Batch 0 Loss 9.0198 Accuracy 0.0000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 Batch 50 Loss 8.9566 Accuracy 0.0060\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 Batch 100 Loss 8.8567 Accuracy 0.0306\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 Batch 150 Loss 8.7498 Accuracy 0.0381\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 Batch 200 Loss 8.6213 Accuracy 0.0431\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 Batch 250 Loss 8.4676 Accuracy 0.0528\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 Batch 300 Loss 8.2941 Accuracy 0.0630\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 Batch 350 Loss 8.1103 Accuracy 0.0710\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 Batch 400 Loss 7.9283 Accuracy 0.0773\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 Batch 450 Loss 7.7631 Accuracy 0.0825\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 Batch 500 Loss 7.6183 Accuracy 0.0881\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 Batch 550 Loss 7.4868 Accuracy 0.0939\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 Batch 600 Loss 7.3640 Accuracy 0.1007\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 Batch 650 Loss 7.2441 Accuracy 0.1076\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 Batch 700 Loss 7.1291 Accuracy 0.1146\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 Loss 7.1246 Accuracy 0.1149\n",
      "Time taken for 1 epoch: 356.5545802116394 secs\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2 Batch 0 Loss 5.3808 Accuracy 0.2259\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2 Batch 50 Loss 5.4921 Accuracy 0.2191\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2 Batch 100 Loss 5.4470 Accuracy 0.2231\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2 Batch 150 Loss 5.4008 Accuracy 0.2274\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2 Batch 200 Loss 5.3479 Accuracy 0.2311\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2 Batch 250 Loss 5.3001 Accuracy 0.2356\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2 Batch 300 Loss 5.2542 Accuracy 0.2389\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2 Batch 350 Loss 5.2172 Accuracy 0.2424\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2 Batch 400 Loss 5.1819 Accuracy 0.2459\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2 Batch 450 Loss 5.1527 Accuracy 0.2487\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2 Batch 500 Loss 5.1213 Accuracy 0.2519\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2 Batch 550 Loss 5.0927 Accuracy 0.2546\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2 Batch 600 Loss 5.0653 Accuracy 0.2572\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2 Batch 650 Loss 5.0412 Accuracy 0.2595\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2 Batch 700 Loss 5.0185 Accuracy 0.2620\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2 Loss 5.0170 Accuracy 0.2621\n",
      "Time taken for 1 epoch: 329.01824021339417 secs\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 3 Batch 0 Loss 4.6641 Accuracy 0.2917\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 3 Batch 50 Loss 4.6428 Accuracy 0.2983\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 3 Batch 100 Loss 4.6202 Accuracy 0.3008\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 3 Batch 150 Loss 4.6122 Accuracy 0.3018\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 3 Batch 200 Loss 4.5992 Accuracy 0.3031\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 3 Batch 250 Loss 4.5891 Accuracy 0.3047\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 3 Batch 300 Loss 4.5715 Accuracy 0.3064\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 3 Batch 350 Loss 4.5585 Accuracy 0.3076\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 3 Batch 400 Loss 4.5441 Accuracy 0.3094\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 3 Batch 450 Loss 4.5279 Accuracy 0.3112\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 3 Batch 500 Loss 4.5145 Accuracy 0.3128\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 3 Batch 550 Loss 4.5004 Accuracy 0.3142\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 3 Batch 600 Loss 4.4880 Accuracy 0.3159\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 3 Batch 650 Loss 4.4744 Accuracy 0.3177\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 3 Batch 700 Loss 4.4601 Accuracy 0.3193\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 3 Loss 4.4592 Accuracy 0.3194\n",
      "Time taken for 1 epoch: 329.6653571128845 secs\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 4 Batch 0 Loss 4.1498 Accuracy 0.3394\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 4 Batch 50 Loss 4.1801 Accuracy 0.3493\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 4 Batch 100 Loss 4.1397 Accuracy 0.3521\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 4 Batch 150 Loss 4.1312 Accuracy 0.3539\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 4 Batch 200 Loss 4.1132 Accuracy 0.3569\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 4 Batch 250 Loss 4.1008 Accuracy 0.3584\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 4 Batch 300 Loss 4.0835 Accuracy 0.3612\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 4 Batch 350 Loss 4.0667 Accuracy 0.3635\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 4 Batch 400 Loss 4.0514 Accuracy 0.3653\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 4 Batch 450 Loss 4.0328 Accuracy 0.3679\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 4 Batch 500 Loss 4.0187 Accuracy 0.3701\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 4 Batch 550 Loss 4.0050 Accuracy 0.3720\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 4 Batch 600 Loss 3.9889 Accuracy 0.3741\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 4 Batch 650 Loss 3.9719 Accuracy 0.3764\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 4 Batch 700 Loss 3.9573 Accuracy 0.3787\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 4 Loss 3.9571 Accuracy 0.3787\n",
      "Time taken for 1 epoch: 329.5320518016815 secs\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 5 Batch 0 Loss 3.5488 Accuracy 0.4171\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 5 Batch 50 Loss 3.6103 Accuracy 0.4169\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 5 Batch 100 Loss 3.5825 Accuracy 0.4221\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 5 Batch 150 Loss 3.5914 Accuracy 0.4221\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 5 Batch 200 Loss 3.5879 Accuracy 0.4233\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 5 Batch 250 Loss 3.5773 Accuracy 0.4248\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 5 Batch 300 Loss 3.5681 Accuracy 0.4263\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 5 Batch 350 Loss 3.5531 Accuracy 0.4286\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 5 Batch 400 Loss 3.5416 Accuracy 0.4302\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 5 Batch 450 Loss 3.5302 Accuracy 0.4319\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 5 Batch 500 Loss 3.5215 Accuracy 0.4332\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 5 Batch 550 Loss 3.5112 Accuracy 0.4347\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 5 Batch 600 Loss 3.5006 Accuracy 0.4361\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 5 Batch 650 Loss 3.4935 Accuracy 0.4369\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 5 Batch 700 Loss 3.4868 Accuracy 0.4376\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving checkpoint for epoch 5 at ./checkpoints/train/ckpt-1\n",
      "Epoch 5 Loss 3.4866 Accuracy 0.4377\n",
      "Time taken for 1 epoch: 329.0690224170685 secs\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 6 Batch 0 Loss 3.2325 Accuracy 0.4480\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 6 Batch 50 Loss 3.2148 Accuracy 0.4647\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 6 Batch 100 Loss 3.1823 Accuracy 0.4689\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 6 Batch 150 Loss 3.1713 Accuracy 0.4697\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 6 Batch 200 Loss 3.1619 Accuracy 0.4717\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 6 Batch 250 Loss 3.1487 Accuracy 0.4737\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 6 Batch 300 Loss 3.1402 Accuracy 0.4750\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 6 Batch 350 Loss 3.1383 Accuracy 0.4749\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 6 Batch 400 Loss 3.1291 Accuracy 0.4761\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 6 Batch 450 Loss 3.1239 Accuracy 0.4766\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 6 Batch 500 Loss 3.1190 Accuracy 0.4773\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 6 Batch 550 Loss 3.1155 Accuracy 0.4775\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 6 Batch 600 Loss 3.1079 Accuracy 0.4786\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 6 Batch 650 Loss 3.0996 Accuracy 0.4798\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 6 Batch 700 Loss 3.0910 Accuracy 0.4809\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 6 Loss 3.0901 Accuracy 0.4810\n",
      "Time taken for 1 epoch: 329.31779623031616 secs\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 7 Batch 0 Loss 2.6800 Accuracy 0.5217\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 7 Batch 50 Loss 2.7518 Accuracy 0.5169\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 7 Batch 100 Loss 2.7756 Accuracy 0.5133\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 7 Batch 150 Loss 2.7687 Accuracy 0.5150\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 7 Batch 200 Loss 2.7671 Accuracy 0.5156\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 7 Batch 250 Loss 2.7552 Accuracy 0.5175\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 7 Batch 300 Loss 2.7445 Accuracy 0.5193\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 7 Batch 350 Loss 2.7383 Accuracy 0.5202\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 7 Batch 400 Loss 2.7309 Accuracy 0.5213\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 7 Batch 450 Loss 2.7273 Accuracy 0.5217\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 7 Batch 500 Loss 2.7227 Accuracy 0.5224\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 7 Batch 550 Loss 2.7165 Accuracy 0.5233\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 7 Batch 600 Loss 2.7098 Accuracy 0.5242\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 7 Batch 650 Loss 2.7059 Accuracy 0.5248\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 7 Batch 700 Loss 2.7015 Accuracy 0.5255\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 7 Loss 2.7009 Accuracy 0.5255\n",
      "Time taken for 1 epoch: 328.4230201244354 secs\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 8 Batch 0 Loss 2.4337 Accuracy 0.5464\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 8 Batch 50 Loss 2.3840 Accuracy 0.5613\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 8 Batch 100 Loss 2.3731 Accuracy 0.5639\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 8 Batch 150 Loss 2.3805 Accuracy 0.5639\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 8 Batch 200 Loss 2.3819 Accuracy 0.5637\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 8 Batch 250 Loss 2.3868 Accuracy 0.5625\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 8 Batch 300 Loss 2.3867 Accuracy 0.5625\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 8 Batch 350 Loss 2.3840 Accuracy 0.5631\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 8 Batch 400 Loss 2.3814 Accuracy 0.5637\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 8 Batch 450 Loss 2.3806 Accuracy 0.5638\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 8 Batch 500 Loss 2.3784 Accuracy 0.5640\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 8 Batch 550 Loss 2.3755 Accuracy 0.5646\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 8 Batch 600 Loss 2.3763 Accuracy 0.5646\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 8 Batch 650 Loss 2.3762 Accuracy 0.5649\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 8 Batch 700 Loss 2.3753 Accuracy 0.5652\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 8 Loss 2.3750 Accuracy 0.5653\n",
      "Time taken for 1 epoch: 328.0940201282501 secs\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 9 Batch 0 Loss 2.1380 Accuracy 0.5907\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 9 Batch 50 Loss 2.1332 Accuracy 0.5948\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 9 Batch 100 Loss 2.1317 Accuracy 0.5945\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 9 Batch 150 Loss 2.1257 Accuracy 0.5951\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 9 Batch 200 Loss 2.1219 Accuracy 0.5961\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 9 Batch 250 Loss 2.1259 Accuracy 0.5958\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 9 Batch 300 Loss 2.1301 Accuracy 0.5954\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 9 Batch 350 Loss 2.1341 Accuracy 0.5952\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 9 Batch 400 Loss 2.1332 Accuracy 0.5954\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 9 Batch 450 Loss 2.1361 Accuracy 0.5950\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 9 Batch 500 Loss 2.1391 Accuracy 0.5945\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 9 Batch 550 Loss 2.1402 Accuracy 0.5945\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 9 Batch 600 Loss 2.1391 Accuracy 0.5948\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 9 Batch 650 Loss 2.1411 Accuracy 0.5946\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 9 Batch 700 Loss 2.1419 Accuracy 0.5947\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 9 Loss 2.1421 Accuracy 0.5947\n",
      "Time taken for 1 epoch: 328.67867064476013 secs\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 10 Batch 0 Loss 2.1447 Accuracy 0.5654\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 10 Batch 50 Loss 1.9188 Accuracy 0.6214\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 10 Batch 100 Loss 1.9275 Accuracy 0.6203\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 10 Batch 150 Loss 1.9255 Accuracy 0.6217\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 10 Batch 200 Loss 1.9352 Accuracy 0.6202\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 10 Batch 250 Loss 1.9391 Accuracy 0.6197\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 10 Batch 300 Loss 1.9473 Accuracy 0.6189\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 10 Batch 350 Loss 1.9503 Accuracy 0.6185\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 10 Batch 400 Loss 1.9528 Accuracy 0.6179\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 10 Batch 450 Loss 1.9583 Accuracy 0.6170\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 10 Batch 500 Loss 1.9615 Accuracy 0.6168\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 10 Batch 550 Loss 1.9638 Accuracy 0.6169\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 10 Batch 600 Loss 1.9646 Accuracy 0.6169\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 10 Batch 650 Loss 1.9658 Accuracy 0.6169\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 10 Batch 700 Loss 1.9674 Accuracy 0.6167\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving checkpoint for epoch 10 at ./checkpoints/train/ckpt-2\n",
      "Epoch 10 Loss 1.9675 Accuracy 0.6167\n",
      "Time taken for 1 epoch: 328.0170452594757 secs\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 11 Batch 0 Loss 1.9735 Accuracy 0.6258\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 11 Batch 50 Loss 1.7728 Accuracy 0.6423\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 11 Batch 100 Loss 1.7802 Accuracy 0.6410\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 11 Batch 150 Loss 1.7852 Accuracy 0.6403\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 11 Batch 200 Loss 1.7945 Accuracy 0.6387\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 11 Batch 250 Loss 1.7977 Accuracy 0.6379\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 11 Batch 300 Loss 1.7942 Accuracy 0.6385\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 11 Batch 350 Loss 1.8017 Accuracy 0.6374\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 11 Batch 400 Loss 1.8012 Accuracy 0.6378\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 11 Batch 450 Loss 1.8063 Accuracy 0.6371\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 11 Batch 500 Loss 1.8089 Accuracy 0.6369\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 11 Batch 550 Loss 1.8126 Accuracy 0.6365\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 11 Batch 600 Loss 1.8170 Accuracy 0.6360\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 11 Batch 650 Loss 1.8226 Accuracy 0.6354\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 11 Batch 700 Loss 1.8250 Accuracy 0.6352\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 11 Loss 1.8256 Accuracy 0.6351\n",
      "Time taken for 1 epoch: 327.8001685142517 secs\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 12 Batch 0 Loss 1.6151 Accuracy 0.6718\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 12 Batch 50 Loss 1.6312 Accuracy 0.6627\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 12 Batch 100 Loss 1.6509 Accuracy 0.6588\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 12 Batch 150 Loss 1.6678 Accuracy 0.6562\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 12 Batch 200 Loss 1.6765 Accuracy 0.6549\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 12 Batch 250 Loss 1.6849 Accuracy 0.6536\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 12 Batch 300 Loss 1.6853 Accuracy 0.6532\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 12 Batch 350 Loss 1.6876 Accuracy 0.6529\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 12 Batch 400 Loss 1.6880 Accuracy 0.6531\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 12 Batch 450 Loss 1.6916 Accuracy 0.6527\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 12 Batch 500 Loss 1.6950 Accuracy 0.6522\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 12 Batch 550 Loss 1.6998 Accuracy 0.6516\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 12 Batch 600 Loss 1.7023 Accuracy 0.6512\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 12 Batch 650 Loss 1.7068 Accuracy 0.6507\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 12 Batch 700 Loss 1.7100 Accuracy 0.6503\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 12 Loss 1.7105 Accuracy 0.6502\n",
      "Time taken for 1 epoch: 329.3431029319763 secs\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 13 Batch 0 Loss 1.5410 Accuracy 0.6902\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 13 Batch 50 Loss 1.5296 Accuracy 0.6748\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 13 Batch 100 Loss 1.5512 Accuracy 0.6718\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 13 Batch 150 Loss 1.5605 Accuracy 0.6706\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 13 Batch 200 Loss 1.5643 Accuracy 0.6705\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 13 Batch 250 Loss 1.5695 Accuracy 0.6700\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 13 Batch 300 Loss 1.5756 Accuracy 0.6691\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 13 Batch 350 Loss 1.5782 Accuracy 0.6689\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 13 Batch 400 Loss 1.5816 Accuracy 0.6681\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 13 Batch 450 Loss 1.5883 Accuracy 0.6668\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 13 Batch 500 Loss 1.5922 Accuracy 0.6663\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 13 Batch 550 Loss 1.5949 Accuracy 0.6661\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 13 Batch 600 Loss 1.5997 Accuracy 0.6655\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 13 Batch 650 Loss 1.6056 Accuracy 0.6645\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 13 Batch 700 Loss 1.6087 Accuracy 0.6644\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 13 Loss 1.6087 Accuracy 0.6644\n",
      "Time taken for 1 epoch: 329.06107211112976 secs\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 14 Batch 0 Loss 1.3944 Accuracy 0.6995\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 14 Batch 50 Loss 1.4568 Accuracy 0.6866\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 14 Batch 100 Loss 1.4724 Accuracy 0.6843\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 14 Batch 150 Loss 1.4806 Accuracy 0.6835\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 14 Batch 200 Loss 1.4826 Accuracy 0.6830\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 14 Batch 250 Loss 1.4882 Accuracy 0.6812\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 14 Batch 300 Loss 1.4904 Accuracy 0.6811\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 14 Batch 350 Loss 1.4919 Accuracy 0.6809\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 14 Batch 400 Loss 1.4956 Accuracy 0.6806\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 14 Batch 450 Loss 1.5005 Accuracy 0.6799\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 14 Batch 500 Loss 1.5039 Accuracy 0.6795\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 14 Batch 550 Loss 1.5082 Accuracy 0.6788\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 14 Batch 600 Loss 1.5137 Accuracy 0.6779\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 14 Batch 650 Loss 1.5200 Accuracy 0.6770\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 14 Batch 700 Loss 1.5251 Accuracy 0.6763\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 14 Loss 1.5255 Accuracy 0.6762\n",
      "Time taken for 1 epoch: 328.8318474292755 secs\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 15 Batch 0 Loss 1.5352 Accuracy 0.6574\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 15 Batch 50 Loss 1.3418 Accuracy 0.7061\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 15 Batch 100 Loss 1.3626 Accuracy 0.7019\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 15 Batch 150 Loss 1.3738 Accuracy 0.6999\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 15 Batch 200 Loss 1.3871 Accuracy 0.6971\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 15 Batch 250 Loss 1.3942 Accuracy 0.6962\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 15 Batch 300 Loss 1.4037 Accuracy 0.6945\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 15 Batch 350 Loss 1.4079 Accuracy 0.6940\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 15 Batch 400 Loss 1.4144 Accuracy 0.6931\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 15 Batch 450 Loss 1.4214 Accuracy 0.6919\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 15 Batch 500 Loss 1.4255 Accuracy 0.6914\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 15 Batch 550 Loss 1.4304 Accuracy 0.6906\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 15 Batch 600 Loss 1.4359 Accuracy 0.6896\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 15 Batch 650 Loss 1.4403 Accuracy 0.6889\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 15 Batch 700 Loss 1.4479 Accuracy 0.6878\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving checkpoint for epoch 15 at ./checkpoints/train/ckpt-3\n",
      "Epoch 15 Loss 1.4481 Accuracy 0.6878\n",
      "Time taken for 1 epoch: 329.04406332969666 secs\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 16 Batch 0 Loss 1.3499 Accuracy 0.6903\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 16 Batch 50 Loss 1.2827 Accuracy 0.7132\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 16 Batch 100 Loss 1.2885 Accuracy 0.7136\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 16 Batch 150 Loss 1.3014 Accuracy 0.7119\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 16 Batch 200 Loss 1.3165 Accuracy 0.7090\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 16 Batch 250 Loss 1.3267 Accuracy 0.7072\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 16 Batch 300 Loss 1.3326 Accuracy 0.7057\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 16 Batch 350 Loss 1.3404 Accuracy 0.7041\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 16 Batch 400 Loss 1.3477 Accuracy 0.7027\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 16 Batch 450 Loss 1.3547 Accuracy 0.7017\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 16 Batch 500 Loss 1.3585 Accuracy 0.7012\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 16 Batch 550 Loss 1.3641 Accuracy 0.7003\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 16 Batch 600 Loss 1.3696 Accuracy 0.6994\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 16 Batch 650 Loss 1.3757 Accuracy 0.6987\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 16 Batch 700 Loss 1.3806 Accuracy 0.6979\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 16 Loss 1.3809 Accuracy 0.6979\n",
      "Time taken for 1 epoch: 329.09080934524536 secs\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 17 Batch 0 Loss 1.2575 Accuracy 0.7162\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 17 Batch 50 Loss 1.2620 Accuracy 0.7161\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 17 Batch 100 Loss 1.2560 Accuracy 0.7187\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 17 Batch 150 Loss 1.2598 Accuracy 0.7174\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 17 Batch 200 Loss 1.2668 Accuracy 0.7160\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 17 Batch 250 Loss 1.2760 Accuracy 0.7143\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 17 Batch 300 Loss 1.2784 Accuracy 0.7143\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 17 Batch 350 Loss 1.2853 Accuracy 0.7133\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 17 Batch 400 Loss 1.2901 Accuracy 0.7123\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 17 Batch 450 Loss 1.2948 Accuracy 0.7119\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 17 Batch 500 Loss 1.3009 Accuracy 0.7108\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 17 Batch 550 Loss 1.3065 Accuracy 0.7097\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 17 Batch 600 Loss 1.3117 Accuracy 0.7089\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 17 Batch 650 Loss 1.3161 Accuracy 0.7082\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 17 Batch 700 Loss 1.3205 Accuracy 0.7075\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 17 Loss 1.3213 Accuracy 0.7074\n",
      "Time taken for 1 epoch: 330.29657912254333 secs\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 18 Batch 0 Loss 0.9995 Accuracy 0.7531\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 18 Batch 50 Loss 1.1740 Accuracy 0.7312\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 18 Batch 100 Loss 1.1851 Accuracy 0.7302\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 18 Batch 150 Loss 1.1958 Accuracy 0.7280\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 18 Batch 200 Loss 1.2056 Accuracy 0.7266\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 18 Batch 250 Loss 1.2129 Accuracy 0.7251\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 18 Batch 300 Loss 1.2198 Accuracy 0.7237\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 18 Batch 350 Loss 1.2270 Accuracy 0.7223\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 18 Batch 400 Loss 1.2320 Accuracy 0.7216\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 18 Batch 450 Loss 1.2352 Accuracy 0.7208\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 18 Batch 500 Loss 1.2431 Accuracy 0.7193\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 18 Batch 550 Loss 1.2500 Accuracy 0.7181\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 18 Batch 600 Loss 1.2569 Accuracy 0.7169\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 18 Batch 650 Loss 1.2627 Accuracy 0.7158\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 18 Batch 700 Loss 1.2681 Accuracy 0.7152\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 18 Loss 1.2685 Accuracy 0.7152\n",
      "Time taken for 1 epoch: 329.7291362285614 secs\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 19 Batch 0 Loss 1.0314 Accuracy 0.7674\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 19 Batch 50 Loss 1.1367 Accuracy 0.7379\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 19 Batch 100 Loss 1.1340 Accuracy 0.7396\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 19 Batch 150 Loss 1.1466 Accuracy 0.7365\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 19 Batch 200 Loss 1.1553 Accuracy 0.7347\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 19 Batch 250 Loss 1.1660 Accuracy 0.7327\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 19 Batch 300 Loss 1.1708 Accuracy 0.7319\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 19 Batch 350 Loss 1.1795 Accuracy 0.7299\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 19 Batch 400 Loss 1.1847 Accuracy 0.7291\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 19 Batch 450 Loss 1.1905 Accuracy 0.7279\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 19 Batch 500 Loss 1.1967 Accuracy 0.7267\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 19 Batch 550 Loss 1.2032 Accuracy 0.7255\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 19 Batch 600 Loss 1.2076 Accuracy 0.7249\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 19 Batch 650 Loss 1.2144 Accuracy 0.7238\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 19 Batch 700 Loss 1.2202 Accuracy 0.7228\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 19 Loss 1.2207 Accuracy 0.7227\n",
      "Time taken for 1 epoch: 330.0889730453491 secs\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 20 Batch 0 Loss 0.9899 Accuracy 0.7659\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 20 Batch 50 Loss 1.0937 Accuracy 0.7445\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 20 Batch 100 Loss 1.1030 Accuracy 0.7432\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 20 Batch 150 Loss 1.1085 Accuracy 0.7418\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 20 Batch 200 Loss 1.1173 Accuracy 0.7404\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 20 Batch 250 Loss 1.1255 Accuracy 0.7387\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 20 Batch 300 Loss 1.1306 Accuracy 0.7377\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 20 Batch 350 Loss 1.1335 Accuracy 0.7372\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 20 Batch 400 Loss 1.1396 Accuracy 0.7362\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 20 Batch 450 Loss 1.1451 Accuracy 0.7350\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 20 Batch 500 Loss 1.1503 Accuracy 0.7342\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 20 Batch 550 Loss 1.1561 Accuracy 0.7330\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 20 Batch 600 Loss 1.1630 Accuracy 0.7317\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 20 Batch 650 Loss 1.1674 Accuracy 0.7310\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 20 Batch 700 Loss 1.1748 Accuracy 0.7298\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving checkpoint for epoch 20 at ./checkpoints/train/ckpt-4\n",
      "Epoch 20 Loss 1.1756 Accuracy 0.7297\n",
      "Time taken for 1 epoch: 330.0001685619354 secs\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(EPOCHS):\n",
    "  start = time.time()\n",
    "  \n",
    "  train_loss.reset_states()\n",
    "  train_accuracy.reset_states()\n",
    "  \n",
    "  # inp -> portuguese, tar -> english\n",
    "  for (batch, (inp, tar)) in enumerate(train_dataset):\n",
    "    train_step(inp, tar)\n",
    "    \n",
    "    if batch % 50 == 0:\n",
    "      print ('Epoch {} Batch {} Loss {:.4f} Accuracy {:.4f}'.format(\n",
    "          epoch + 1, batch, train_loss.result(), train_accuracy.result()))\n",
    "      \n",
    "  if (epoch + 1) % 5 == 0:\n",
    "    ckpt_save_path = ckpt_manager.save()\n",
    "    print ('Saving checkpoint for epoch {} at {}'.format(epoch+1,\n",
    "                                                         ckpt_save_path))\n",
    "    \n",
    "  print ('Epoch {} Loss {:.4f} Accuracy {:.4f}'.format(epoch + 1, \n",
    "                                                train_loss.result(), \n",
    "                                                train_accuracy.result()))\n",
    "\n",
    "  print ('Time taken for 1 epoch: {} secs\\n'.format(time.time() - start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QfcsSWswSdGV"
   },
   "source": [
    "## Evaluate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y6APsFrgImLW"
   },
   "source": [
    "The following steps are used for evaluation:\n",
    "\n",
    "* Encode the input sentence using the Portuguese tokenizer (`tokenizer_pt`). Moreover, add the start and end token so the input is equivalent to what the model is trained with. This is the encoder input.\n",
    "* The decoder input is the `start token == tokenizer_en.vocab_size`.\n",
    "* Calculate the padding masks and the look ahead masks.\n",
    "* The `decoder` then outputs the predictions by looking at the `encoder output` and its own output (self-attention).\n",
    "* Select the last word and calculate the argmax of that.\n",
    "* Concatentate the predicted word to the decoder input as pass it to the decoder.\n",
    "* In this approach, the decoder predicts the next word based on the previous words it predicted.\n",
    "\n",
    "Note: The model used here has less capacity to keep the example relatively faster so the predictions maybe less right. To reproduce the results in the paper, use the entire dataset and base transformer model or transformer XL, by changing the hyperparameters above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T04:16:37.762810Z",
     "iopub.status.busy": "2020-11-02T04:16:37.762085Z",
     "iopub.status.idle": "2020-11-02T04:16:37.764376Z",
     "shell.execute_reply": "2020-11-02T04:16:37.763905Z"
    },
    "id": "5buvMlnvyrFm"
   },
   "outputs": [],
   "source": [
    "def evaluate(inp_sentence):\n",
    "  start_token = [tokenizer_pt.vocab_size]\n",
    "  end_token = [tokenizer_pt.vocab_size + 1]\n",
    "  \n",
    "  # inp sentence is portuguese, hence adding the start and end token\n",
    "  inp_sentence = start_token + tokenizer_pt.encode(inp_sentence) + end_token\n",
    "  encoder_input = tf.expand_dims(inp_sentence, 0)\n",
    "  \n",
    "  # as the target is english, the first word to the transformer should be the\n",
    "  # english start token.\n",
    "  decoder_input = [tokenizer_en.vocab_size]\n",
    "  output = tf.expand_dims(decoder_input, 0)\n",
    "    \n",
    "  for i in range(MAX_LENGTH):\n",
    "    enc_padding_mask, combined_mask, dec_padding_mask = create_masks(\n",
    "        encoder_input, output)\n",
    "  \n",
    "    # predictions.shape == (batch_size, seq_len, vocab_size)\n",
    "    predictions, attention_weights = transformer(encoder_input, \n",
    "                                                 output,\n",
    "                                                 False,\n",
    "                                                 enc_padding_mask,\n",
    "                                                 combined_mask,\n",
    "                                                 dec_padding_mask)\n",
    "    \n",
    "    # select the last word from the seq_len dimension\n",
    "    predictions = predictions[: ,-1:, :]  # (batch_size, 1, vocab_size)\n",
    "\n",
    "    predicted_id = tf.cast(tf.argmax(predictions, axis=-1), tf.int32)\n",
    "    \n",
    "    # return the result if the predicted_id is equal to the end token\n",
    "    if predicted_id == tokenizer_en.vocab_size+1:\n",
    "      return tf.squeeze(output, axis=0), attention_weights\n",
    "    \n",
    "    # concatentate the predicted_id to the output which is given to the decoder\n",
    "    # as its input.\n",
    "    output = tf.concat([output, predicted_id], axis=-1)\n",
    "\n",
    "  return tf.squeeze(output, axis=0), attention_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T04:16:37.772962Z",
     "iopub.status.busy": "2020-11-02T04:16:37.772298Z",
     "iopub.status.idle": "2020-11-02T04:16:37.774529Z",
     "shell.execute_reply": "2020-11-02T04:16:37.774013Z"
    },
    "id": "CN-BV43FMBej"
   },
   "outputs": [],
   "source": [
    "def plot_attention_weights(attention, sentence, result, layer):\n",
    "  fig = plt.figure(figsize=(16, 8))\n",
    "  \n",
    "  sentence = tokenizer_pt.encode(sentence)\n",
    "  \n",
    "  attention = tf.squeeze(attention[layer], axis=0)\n",
    "  \n",
    "  for head in range(attention.shape[0]):\n",
    "    ax = fig.add_subplot(2, 4, head+1)\n",
    "    \n",
    "    # plot the attention weights\n",
    "    ax.matshow(attention[head][:-1, :], cmap='viridis')\n",
    "\n",
    "    fontdict = {'fontsize': 10}\n",
    "    \n",
    "    ax.set_xticks(range(len(sentence)+2))\n",
    "    ax.set_yticks(range(len(result)))\n",
    "    \n",
    "    ax.set_ylim(len(result)-1.5, -0.5)\n",
    "        \n",
    "    ax.set_xticklabels(\n",
    "        ['<start>']+[tokenizer_pt.decode([i]) for i in sentence]+['<end>'], \n",
    "        fontdict=fontdict, rotation=90)\n",
    "    \n",
    "    ax.set_yticklabels([tokenizer_en.decode([i]) for i in result \n",
    "                        if i < tokenizer_en.vocab_size], \n",
    "                       fontdict=fontdict)\n",
    "    \n",
    "    ax.set_xlabel('Head {}'.format(head+1))\n",
    "  \n",
    "  plt.tight_layout()\n",
    "  plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T04:16:37.779773Z",
     "iopub.status.busy": "2020-11-02T04:16:37.779103Z",
     "iopub.status.idle": "2020-11-02T04:16:37.781360Z",
     "shell.execute_reply": "2020-11-02T04:16:37.780888Z"
    },
    "id": "lU2_yG_vBGza"
   },
   "outputs": [],
   "source": [
    "def translate(sentence, plot=''):\n",
    "  result, attention_weights = evaluate(sentence)\n",
    "  \n",
    "  predicted_sentence = tokenizer_en.decode([i for i in result \n",
    "                                            if i < tokenizer_en.vocab_size])  \n",
    "\n",
    "  print('Input: {}'.format(sentence))\n",
    "  print('Predicted translation: {}'.format(predicted_sentence))\n",
    "  \n",
    "  if plot:\n",
    "    plot_attention_weights(attention_weights, sentence, result, plot)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T04:16:37.785148Z",
     "iopub.status.busy": "2020-11-02T04:16:37.784487Z",
     "iopub.status.idle": "2020-11-02T04:16:39.025891Z",
     "shell.execute_reply": "2020-11-02T04:16:39.025281Z"
    },
    "id": "YsxrAlvFG8SZ"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input: este é um problema que temos que resolver.\n",
      "Predicted translation: so this is a problem that we have to solve the global challenges .\n",
      "Real translation: this is a problem we have to solve .\n"
     ]
    }
   ],
   "source": [
    "translate(\"este é um problema que temos que resolver.\")\n",
    "print (\"Real translation: this is a problem we have to solve .\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T04:16:39.030035Z",
     "iopub.status.busy": "2020-11-02T04:16:39.029378Z",
     "iopub.status.idle": "2020-11-02T04:16:39.764247Z",
     "shell.execute_reply": "2020-11-02T04:16:39.763740Z"
    },
    "id": "7EH5y_aqI4t1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input: os meus vizinhos ouviram sobre esta ideia.\n",
      "Predicted translation: my neighbors heard about this idea .\n",
      "Real translation: and my neighboring homes heard about this idea .\n"
     ]
    }
   ],
   "source": [
    "translate(\"os meus vizinhos ouviram sobre esta ideia.\")\n",
    "print (\"Real translation: and my neighboring homes heard about this idea .\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T04:16:39.769311Z",
     "iopub.status.busy": "2020-11-02T04:16:39.768606Z",
     "iopub.status.idle": "2020-11-02T04:16:42.049202Z",
     "shell.execute_reply": "2020-11-02T04:16:42.049631Z"
    },
    "id": "J-hVCTSUMlkb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input: vou então muito rapidamente partilhar convosco algumas histórias de algumas coisas mágicas que aconteceram.\n",
      "Predicted translation: so i 'm going to share with you a lot of magic stories that happened to be some very magical stuff that happened .\n",
      "Real translation: so i 'll just share with you some stories very quickly of some magical things that have happened .\n"
     ]
    }
   ],
   "source": [
    "translate(\"vou então muito rapidamente partilhar convosco algumas histórias de algumas coisas mágicas que aconteceram.\")\n",
    "print (\"Real translation: so i 'll just share with you some stories very quickly of some magical things that have happened .\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_1MxkSZvz0jX"
   },
   "source": [
    "You can pass different layers and attention blocks of the decoder to the `plot` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-11-02T04:16:42.054286Z",
     "iopub.status.busy": "2020-11-02T04:16:42.053550Z",
     "iopub.status.idle": "2020-11-02T04:16:43.727809Z",
     "shell.execute_reply": "2020-11-02T04:16:43.728233Z"
    },
    "id": "t-kFyiOLH0xg"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input: este é o primeiro livro que eu fiz.\n",
      "Predicted translation: so this is the first book i made .\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABHgAAAIrCAYAAACUOC4gAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzde5ytdVn//9c1+8AG9kZQUBTiIKKoyMmdiqGR5/xav9TMPP1SVExTU/NrUn4zy/xmlj3SDoYoaJpFHgoz8gwokMp5g4KZmImaSgobkH2Yub5/rDWbNcM+zJ71+dz3fNa8no/Hesyse9Zc9zX3rPXe91z7vu8VmYkkSZIkSZLaNdV3A5IkSZIkSRqPAx5JkiRJkqTGOeCRJEmSJElqnAMeSZIkSZKkxjngkSRJkiRJapwDHkmSJEmSpMY54JEkSZIkSWqcAx5JkiRJkqTGOeCRJEmSJElqnAMeSZIkSZKkxq3suwEtfRGxGrjv8O51mbmlz34kTT5zR1KXzBxJXTJzVEtkZt89aAmLiJOB9wDfAAL4CeBXMvOCHtuSNMHMHUldMnMkdcnMUU0OeLRTEXEp8MzMvG54/77ABzLzwf12JmlSmTuSumTmSOqSmaOavAaPdmXVbPgAZOZXgVU99iNp8pk7krpk5kjqkpmjaiZqwBMD/xgR9++7lwlyaUScEREnD2/vBC7puylpqTB3qjB3pB0wc6owc6QdMHOqMHNUzUSdohURjwfeDfxdZv5G3/1MgojYA/g14KThos8Bf5mZm/rrSlo6zJ3yzB1px8yc8swcacfMnPLMHNU0aQOes4EzgT8DHpCZW3tuqWkRsQK4JjOP6rsXaakyd8oyd6SdM3PKMnOknTNzyjJzVNvEnKIVEfsDD8zMc4FPAb/Qc0vNy8xp4LqIOKTvXqSlyNwpz9yRdszMKc/MkXbMzCnPzFFtEzPgAZ4DfGD4+ZnAC3rsZZLsB1wTEZ+OiHNmb303pX5FxJMjYm3ffSwB5k4d5o7mMHO2MXPqMHM0h5mzjZlTh5mjOymVOxNzilZEbACekJk3DO9fCTwpM/+r387aFhE/vb3lmXl+171oaYiII4BrgZdl5jv67qdP5k4d5o5GmTl3MHPqMHM0ysy5g5lTh5mj+UrmzkQMeCJiX+DpmfnXI8seC/wgMy/vrzNp8kTEG4efPi4zH9JrMz0yd6RumDkDZo7UDTNnwMyRulMydybiFK3M/BFw9bxlnwT26qej9kXE54cfN0bEzSO3jRFxc9/9qR/DC8M9DXgzcFNEHNtzS70xd8ozdzSfmXMHM6c8M0fzmTl3MHPKM3O0PaVzZyIGPENvX+AyLUBmnjT8uC4z9xm5rcvMffruT715IvBvmbmRwVtmPr/nfvpm7hRk7mg7zJy5zJyCzBxth5kzl5lTkJmjHSiaOyuLtNSjiDgReDhwQES8auRL+wAr+ulqskTEScCRmXnm8Gr66zLz+r77Ui+eD7x1+PlHgDdGxKszc3OPPXXO3KnP3NGQmYOZ0wUzR0NmDmZOF8wcjSiaO5NwBM9qYC2DYdW6kdvNwC/22NdEiIjXA78JnDZctBp4X38dqS/Dc7H3zcwLADLzduCDwKN6bawf5k5F5o7AzJnHzKnIzBGYOfOYORWZOZpVI3cm5SLLK4CzM/OpffcyaSLiCuB44LLMPH647KrMPKbfzqR+mTv1mDvSnZk59Zg50p2ZOfWYOaqp+VO0ADJzOiLu1XcfE2pzZmZEJEBE7N13Q+peRJyws69n5mVd9bJUmDtVmTvLnJlzZ2ZOVWbOMmfm3JmZU5WZo2q5MxEDnqErIuIc4B+AW2cXZuaH+2tpIpwdEX8N7BsRLwROAd7Zc0/q3p8MP64B1gNXAgEcA1wCnNhTX30zd+owd2TmbJ+ZU4eZIzNn+8ycOswcQaXcmYhTtAAi4sztLM7MPKXzZiZMRDwWeByDJ9zHh2+RqGUoIj4MvD4zNwzvHw38bmYuy/OxzZ16zB2BmTOfmVOPmSMwc+Yzc+oxczSrdO5MzIBHdUXEPowc8ZWZ/9NjO+pJRFyTmQ/c1TKpBHNHZo66ZObIzFGXzBxB+dyZmFO0ImINg7cYeyCDw5wAcMI8noh4EfAG4HZghsGUOYF799mXenNVRJzBHVf6fxZwVY/99MrcqcPc0QgzZ4SZU4eZoxFmzggzpw4zR/MUzZ1JeJv0WX8DHAg8HjgfOBjYOE7BiLhHRLwrIs4d3n9ARDx/7E7b8mrg6Mw8LDPvnZmHZ+bY4RMRB0fERyLi+xHxvYj4UEQcXKBf1fU84Brg14e3Lw+XLVdFc8fM2aZ47pg5zTJz5nJfpw73dTTLzJnLzKnDzNGoorkzMadoRcTlmXn87FvMRcQq4HOZ+bAxap4LnAn8dmYeGxErgcsz80Gl+l7qIuJfgadk5m2F634S+FsG/3AAPBt4VmY+tuR6pJpK546ZM1Ajd8wcTQL3depwX0faPjOnDjNHNU3MKVrAluHHHw0vTPRd4O5j1tw/M8+OiNMAMnNrREyPWbM1pwEXRcQXgE2zCzPz5WPWPSAzRy/cdlZEvGLMmgBExKHAkZn5qYjYE1iZmWP9b4MGIuKngN8FDmXuOcPL9ZDS0rlj5gzUyB0zp0Fmzp24r1NHU/s6Zk49Zs6dmDl1NJU5YO7UVDp3JmnAc3pE7Ae8DjgHWAv8nzFr3hoRd2NwTiQR8TDgpjFrtuavgc8AGxicI1rKjRHxbOADw/vPAG4ct2gM3mrwVOCuwBEMDiV9B/DocWsLgHcBrwQuBZbbP8bbUzp3zJyBGrlj5rTJzJnLfZ06mtnXMXOqM3PmMnPqaCZzwNzpQNHcmaRTtA7PzOt3tWw3a54AvB04GrgaOAB4WmZeOVazDZk9NLNC3UMZbNsTGQT8RcDLM/ObY9a9AngI8IXZviNiw3I67LOmiPhCZj607z6WitK5Y+YM1MgdM6dNZs5c7uvU0dK+jplTl5kzl5lTR0uZM6xr7lRUOncm6QieDwEnzFv2QeDBY9S8Bvhp4H4Mrm5+HZN1YeqFODciTgU+ytxDCBf9Nn4RsQJ4U2b+fIH+5tuUmZsjYnZdKxn+D4GK+GxEvAX4MHOfD5f111KvSueOmTNQNHfMnKaZOXO5r1NHS/s6Zk5dZs5cZk4dLWUOmDu1Fc2d5gc8EXEUg7fuu0tEPGXkS/sw8nZ+i3RxZp7AIIhm13cZdw66SfaM4cfTRpaN9TZ+mTkdEYdGxOrM3DxWd3d2fkT8FrBnRDwWeAmD8FQZs9Pl9SPLEnhUD730pmLumDkDRXPHzGmamYP7Oh1oaV/HzKnLzMHM6UBLmQPmTm1Fc6f5AQ+D6e+TgH2BnxtZvhF44WIKRsSBwEEMnsTHM5guwyDU9lp8q+3JzMMrlf46cGFEnAPcOrK+t45Z97XA8xmc0/oi4F+AM8asqaHM/Jm+e1giiuaOmTNXpdwxcxpk5mzjvk5Fje3rmDkVmTnbmDkVNZY5YO5UVTp3JukaPCdm5sWFav0K8FwGU7QvcUcAbQTOyswPl1jPUhYRj8rMz8yb2m8z7jaIiNfvoO4bxqmruiLiHsCbgHtl5s9GxAOAEzPzXT231otSuWPmDNTMHTOnTWbOXO7rlOW+juYzc+Yyc8oyc7Q9pXNnkgY8fwS8Efgx8K/AMcArM/N9Y9R8amZ+qFCLTYmIN2Tm6yPizO18OTPzlDHrn1DjfOaIuJ7tnBM67ttbDrfD9uqOtR1aExHnAmcCv52Zxw7Pwb18uV5krXTuLOfMgbq5Y+a0ycyZy32dslrc1zFz6jJz5jJzymoxc4Z1i+eOmXOH0rkzCadozXpcZr4mIp4MfAN4CnABsOgAAg6OiH0YTJbfyeDc0Ndm5ifGbXapG4bPFHBuZp5dYRV/MjxU84PA32fm1YXqjp67uAZ4GoO39BvXP8+r+2Tg2+MWjYhHABdl5vTIsirhXMj+mXl2RJwGkJlbI2I5v41o6dxZtpkD1XPHzMHMmQDu6xTU6L6OmVOXmTOXmVNQo5kDdXLHzLlD2dzJzIm4AdcMP54BPGH4+ZVj1rxy+PHxwEcYXGzssr5/1o636yUVax8IvBy4kME5na+rtJ5LK9ScYhAe49a5DTgfuPvIsiX7HAPOA+422yPwMOD8vvvqcXsUzR0zZ9t2qJI7Zo6Z0/rNfZ1q27XpfR0zp+jPbebM3R5mTp3t2nTmDNdTNHeWa+YM+yuaO5N0BM9HI+JaBocQvjgiDgBuH7Pm7LmhTwTem5nXRETs7Bsm0Kci4tXA3zP3Yl2Lfhu/kRrfBd4WEZ8FXgP8DoPDQBctIkavwD/FYOJc43l+JHD3AnWuA97C4Or0z8/Mi7jjebcUvQo4BzgiIi4EDgB+sd+WelU6d8ycgSq5Y+YAZk7r3Nepo5l9HTOnOjNnLjOnjmYyBzrLneWaOVA4dybmGjwAEXFX4KYcvE3c3sC64ZN8sfXOZHC198OBY4EVwHmZ+eAiDTdgeM7lfJnjn+t9f+DpwFOBGxkE3Icy83tj1v3syN2tDA4n/ePMvG7MuhsZnCcaw4/fBU7LMc8hjojLMvOEiDiSwTZ4N3BKDt4+ckkanhd6Pwbb4rrM3NJzS70qmTtmzkCN3DFzttU1cxrnvk55Le3rmDn1mTlzmTnltZQ5w7rFc8fMmatk7kzEgCci9gKOzMwrR5YdAkxn5g1j1J0CjgO+npk/ioi7AQdl5lVjN73MRcTFDF50Z2fm2OdbtioiLs/M44efr2UQQk/JzCV3dF2t11mramwPM6ceM2fAzGmX+zrtMXfMnJaZOe0xc9rKHKj098SEDHhWAdcCx2TmrcNlnwB+KzMvGaNuAM8C7p2Zvzfc2Adm5hfH7LdK3WHtY4FHDO9+bvTJssh6a4CXACcxmK5+DnhHZo57eGYVEfGqnX09M9+6yLqzv7PDM/P3S/7OtrOuQzLzm6XrjqvW66xVNbaHmbOtZjO5Y+bUY+bM1dK+jplTj5lTj5kzV0uZU7nuss4cqJM7Zs5AjdfZVMH+ejM8hOkjwC/BtqnXAQXC+C+BE4FnDO9vBP5iMYUi4qSIWFG67rx1/DrwfgbnL94deF9EvGzMsu9lcPGztwN/Pvz8b8bo8ezhxw0RcdXIbUNElJjcrwdezODQz4OAX2Vwdf51w9tizf7Onjm8P9bvLCJeM/z49oh42+gNePUYfVZT8XXWpErbw8wZKJY7Zs6AmdO+pb6vY+bM6bNm7pg5lZg5cy31zBn2VDV3zJxtauTOss8cqPQ6yyVw5egSN+Ao4ILh568DXl6g5uyVrC8fWbaoK8cDDwdOL1133jquAvYeub83cNWYNb+8kGW7Ue+ew4+Hbu9WYBtcwODc4Nn762afF0vluTD83huHH18B/Mr827j91rrVeJ21fCu9PcycbXWK5Y6Zs+17zZwJuNXYHqWea2bOnO+tljtmTt2bmVN/e5R8rtXOHTNnW/3iuWPmzOm96OtsSZ6LthiZeW0M3Bf4Ze44lG4cW4ZT4QSIwZXjZxbZ30URcVvpuvMEMD1yf3q4bByXRcTDMvPfACLiocCiJ4qZ+Z3hx/8cs68duQeweeT+5uGycZX+nf13RNwLeB5wMuP/njpR6XXWrArbw8wZKJY7Zs42Zs4EWMr7OmbOHSrnjplTkZkz11LOnGF/tXPHzBmokTtmzlDp19nEDHiG3gWcAWzIzB8WqPc2BodM3T0i/oDB25W9brHFMvOKGnVHnAl8ISI+Mrz/Cwy2yTgeDFwUEbPnLR4CXBcRGxhc7f2Y3SkWd1wx/U5fGtbbZ6xuB4c8fnHeNjhrzJpQ/nf2V8CngXsDl44sn72S/FhX0d+eiDgwx3jXgxGlX2etK7k9zJyBYrlj5mxj5kyOJbuvY+YMVM4dM2cnCuWOmTPXks0cqJ47Zs5Ajdwxc+Yq9jqbiIssz4rBVai/Azw1Mz9VqOZRwKMZPDk+nZlfWeJ1T2BwwS4YXAjs8jHrHbqzr1ecFC/acBvMTj4vGHcbjNQt/juLiL/KzBeP3dzC1vWxzPxfBeoUf521rPT2WO6ZM6zZVO6YOTtcl5lTQSv7OmZOPWbOTtc3du6YOXO1kjm16po5AzVyx8yZU6fY62yiBjySJEmSJEnL0US8i5YkSZIkSdJyNpEDnog41bp16rbUa2t1W+q1Zt1W+Zxoq25LvbZWt6VeW9bSdm6p19bqttRra3XNnLla+t3VqttSr63VbanXpV53Igc8QK1Atm5bvbZWt6Vea9Ztlc+Jtuq21GtrdVvqtWUtbeeWem2tbku9tlbXzJmrpd9drbot9dpa3ZZ6XdJ1J3XAI0mSJEmStGw0c5Hl1bEm94y9F/TYzWxiNXss6LFHPujWBffw/RunOeBuKxb02H//8sLfjW7zzI9ZPbXnrh84M7PgmgCb83ZWx5pdPi53o+4WNrFqgdt2d1i3rV53t+5GfviDzDygeBOVrY49cg0Ly53d2s577fp1CbBl622sWrnXwmoCMbOwPN88fRurVyysbm7atOD1b8lNrIoFboMjVy247uabfszquywgIwG+umVBD1sKr4tJrbsUer2dW9mcm6J4E5VVy5y1C3z9AJu33MrqVQvrYfrAhe0/bL35Nlbus7DMWfG13cic3dgGscfCn5PVMnJCX2/sxittt/6d2I0/UfretmbOXPsfvfDXxcb/2cq6u65c0GN/8I19F1x3wVl2y48XXHNJvN4mtG5LvS6Vujv6+2phr6YlYM/Ym4eteWLxuud+/N+K1wR44vGPK14zN95SvCbAzO0LD+HdKzxdp64gKu1DVBr4fio/uOTe7nEh1rA3D11R/rUcDziqeE2Aqdu3Fq85c91/FK8JMP2Oe1apO/Xo/6pSV235Qn667xYWpVbmzJxwTPGaADe9tvx+yX7/69+L1wRYcdgRVepO//vXq9St9e8xUwv7j8rdEVN19klyup39yC/MtPlu6mvYm4fGo4vXfd6H6+z2vevUXyhec+r8sd/tW+rFjv6+8hQtSZIkSZKkxjngkSRJkiRJapwDHkmSJEmSpMY54JEkSZIkSWqcAx5JkiRJkqTGOeCRJEmSJElqnAMeSZIkSZKkxjngkSRJkiRJalyVAU9E7B0RH4uIKyPi6oh4+nD5oyPi8ojYEBHvjog9aqxf0vJi5kjqmrkjqUtmjqSFqHUEzxOAb2fmsZl5NPCvEbEGOAt4emY+CFgJvHhnRSLi1Ii4JCIu2cymSq1KmgBFMgfm5s4Wc0fSjhXf1zFzJO2EmSNpl2oNeDYAj42IN0fEIzLzJuB+wPWZ+dXhY94DPHJnRTLz9Mxcn5nrV+MwWtIOFckcmJs7q8wdSTtWfF/HzJG0E2aOpF2qMuAZhswJDILojRHxOzXWI0lg5kjqnrkjqUtmjqSFqHUNnnsBt2Xm+4C3MAij64DDIuI+w4c9Bzi/xvolLS9mjqSumTuSumTmSFqIlZXqPgh4S0TMAFuAF2fm7RHxPOAfImIl8CXgHZXWL2l5MXMkdc3ckdQlM0fSLlUZ8GTmx4GPb2f5p4Hja6xT0vJl5kjqmrkjqUtmjqSFqHWRZUmSJEmSJHXEAY8kSZIkSVLjHPBIkiRJkiQ1zgGPJEmSJElS4xzwSJIkSZIkNa7W26QXl3uvYfq4o4rXPfzjDyxeE+ABe36/eM3bH/QTxWsC7HHxtVXq1jJz6619t9C/qDSbzek6dVuWM8VLTt2+tXhNgLi5ndfGYWv/p0rdb06tqFKXGV8bqi9WrmDFvncpXnfl1/+7eE2AH204tHjN/eJrxWsCfO+n716l7v7X/1eVurllc5W6MRU1ipavCcTqOnmem+tsW93h9R/85Sp1tzx/U/GaR37O/QZNFo/gkSRJkiRJapwDHkmSJEmSpMY54JEkSZIkSWqcAx5JkiRJkqTGOeCRJEmSJElqnAMeSZIkSZKkxjngkSRJkiRJatyiBzwRsW9EvGTk/skR8c87eOwZEfGAxa5LkswcSV0zdyR1ycyRNK5xjuDZF3jJLh8FZOYLMvPLY6xLkswcSV0zdyR1ycyRNJZxBjx/CBwREVdExFuGy9ZGxAcj4tqIeH9EBEBEnBcR6yNiRUScFRFXR8SGiHjl2D+BpOXCzJHUNXNHUpfMHEljWTnG974WODozj4PBIYTA8cADgW8DFwI/BXx+5HuOAw7KzKOH37PvzlYQEacCpwKs2eMuY7QqaQJUz5zhY+7IHfYq2L6kBnW7rzO1tnD7khrTbea4nyNNnNIXWf5iZn4rM2eAK4DD5n3968C9I+LtEfEE4OadFcvM0zNzfWauX7Vq78KtSpoARTMH5uUOe5TvWFLrqu3rrJ5aU6djSS2r9/eV+znSxCk94Nk08vk0844QyswfAscC5wG/CpxReP2SlhczR1LXzB1JXTJzJC3YOKdobQTW7c43RMT+wObM/FBEXAe8b4z1S1pezBxJXTN3JHXJzJE0lkUPeDLzxoi4MCKuBs4FPraAbzsIODMiZo8cOm2x65e0vJg5krpm7kjqkpkjaVzjHMFDZj5z3qLzRr720pHPTx55zAnjrFPS8mXmSOqauSOpS2aOpHGUvgaPJEmSJEmSOuaAR5IkSZIkqXEOeCRJkiRJkhrngEeSJEmSJKlxDngkSZIkSZIaN9a7aHUpNm9l1bduLF73wE8cVLwmwDd/6eDiNQ/5m68XrwmQh9yrTt1vfbdKXaZWVCkbK8rXndr3LsVrArBlc5Wy0zfdXKUuWadsbbFqJSv3v3vxurcdvK54TYD/fMba4jXv+4I6r+MbnrhHlbor71Fn2279TqU8k0atWkXes3zmcMN/l68JHPqx24vXnFpbPscAZursOsDRR9ape8WXq5SNleV3/XN6pnhNgKm99qpSt0q3m6JG1W5E+d6PePM1xWsC/Mu1FxSv+fiZ44rXrKrS30HMTNepq855BI8kSZIkSVLjHPBIkiRJkiQ1zgGPJEmSJElS4xzwSJIkSZIkNc4BjyRJkiRJUuMc8EiSJEmSJDWukwFPRFzUxXokaZa5I6lLZo6kLpk5krankwFPZj68i/VI0ixzR1KXzBxJXTJzJG1PV0fw3DL8eM+IuCAiroiIqyPiEV2sX9LyY+5I6pKZI6lLZo6k7VnZ8fqeCXw8M/8gIlYAe3W8fknLj7kjqUtmjqQumTmStul6wPMl4N0RsQr4x8y8YmcPjohTgVMB1qxY10F7kibQGLmztoP2JE2YxWfOqn06aE/ShFl85jgLkiZOp++ilZkXAI8EbgDOioj/fxePPz0z12fm+tUr9uykR0mTZazcmTJ3JO2esTJn5d6d9ChpcoyTOavYo5MeJXWn0wFPRBwK/HdmvhM4Azihy/VLWn7MHUldMnMkdcnMkTSq61O0Tgb+d0RsAW4BdjphlqQCTsbckdSdkzFzJHXnZMwcSUOdDHgyc+3w43uA93SxTknLm7kjqUtmjqQumTmStqfTU7QkSZIkSZJUngMeSZIkSZKkxjngkSRJkiRJapwDHkmSJEmSpMY54JEkSZIkSWpc12+TvuTc9fPfqlJ3v1tuLV7zlC9cWrwmwDsfdP8qdXPL1ip1mZmuUjYr1J3+wQ+K1wQgs05dzZFbp5n+4Y+K193zki3FawLc97zyuRN77FG8JsDjLvh6lbof/8l7ValLRJ26NV7LLfWquWZmiE2bi5edvunm4jUBVnzxy8Vr5nSdf+Pv8+yvVql7y4fq/F/pdKXX28zmOv/+1DBz222VCptls2L1KlYeeFDxujPfr7P/+7P3eXjxmt9447HFawLc+81XV6l766Pq/N229zc2Vqk7c+VXyhettJ8TK1dVqctUpf2y23ewujprkyRJkiRJUlcc8EiSJEmSJDXOAY8kSZIkSVLjHPBIkiRJkiQ1zgGPJEmSJElS4xzwSJIkSZIkNc4BjyRJkiRJUuPGHvBExL4R8ZKR+ydHxD+PW1eStsfMkdQ1c0dSl8wcSYtV4giefYGX7PJRklSGmSOpa+aOpC6ZOZIWpcSA5w+BIyLiioh4y3DZ2oj4YERcGxHvj4gAiIgHR8T5EXFpRHw8Iu5ZYP2SlhczR1LXzB1JXTJzJC3KygI1XgscnZnHweAQQuB44IHAt4ELgZ+KiC8Abwf+v8z8fkQ8HfgD4JQdFY6IU4FTAdasWFegVUkToFrmDOvdkTvsVetnkNSWbvZ1Vu5T82eQ1A7/vpK0KCUGPNvzxcz8FkBEXAEcBvwIOBr45HDgvAL4zs6KZObpwOkAd9njHlmpV0ntK5I5MDd39pm6m7kjaUfK7+usOdDMkbQj/n0laZdqDXg2jXw+PVxPANdk5omV1ilp+TJzJHXN3JHUJTNH0i6VuAbPRmAhx/ddBxwQEScCRMSqiHhggfVLWl7MHEldM3ckdcnMkbQoYw94MvNG4MKIuHrkImDbe9xm4BeBN0fElcAVwMPHXb+k5cXMkdQ1c0dSl8wcSYtV5BStzHzmvEXnjXztpSOfXwE8ssQ6JS1fZo6krpk7krpk5khajBKnaEmSJEmSJKlHDngkSZIkSZIa54BHkiRJkiSpcQ54JEmSJEmSGueAR5IkSZIkqXFF3kWrC7llC9M3fKd43RUH3qN4TQCmonjJM447unhNgD+49rwqdX/7vidVqZsz01XqtiRWra5SN7dsrlK3WZnkpk3Fy05XqFlLjZ8f4BX7faNK3XNv27dK3aZk9t2BFik3bWb637/edxsLlpva+ff47Ht/ukrdx3//uCp1q2loH6ql51eruZubt7D1WzdUKNzO9jjij79cpe6vXf6lKnX/4mcPqFJ3+mvXV6lbRaXn16T8HeQRPJIkSZIkSY1zwCNJkiRJktQ4BzySJEmSJEmNc8AjSZIkSZLUOAc8kiRJkiRJjXPAI0mSJEmS1DgHPJIkSZIkSY1b9IAnIl4eEV+JiPdHxM9HxGt343sPi4hnLnbdkpYfM0dSl8wcSV0ycySVsHKM730J8JjM/Nbw/jnzHxARKzNz63a+9zDgmcDfjrF+ScuLmSOpS2aOpC6ZOZLGtqgBT0S8A7g3cG5EvBv4IbA+M18aEWcBtwPHAxdGxD8Bfzb81gQeCfwhcP+IuAJ4T2b+6Xg/hqRJZuZI6pKZI6lLZo6kUv8p284AACAASURBVBY14MnMX42IJwA/k5k/iIjnznvIwcDDM3M6Ij4K/FpmXhgRaxkE1GuBV2fmk3a2nog4FTgVYA17LaZVSROgq8wBc0eSmSOpW2aOpFJqXWT5HzJzevj5hcBbI+LlwL47OKxwuzLz9Mxcn5nrV8UeVRqVNBGKZA7Myx3MHUnbZeZI6pKZI2lBag14bp39JDP/EHgBsCeDwwqPqrROScuXmSOpS2aOpC6ZOZIWZJyLLC9IRByRmRuADRHxk8BRwH8B62qvW9LyY+ZI6pKZI6lLZo6knal1BM+oV0TE1RFxFbAFOBe4CpiOiCsj4pUd9CBp+TBzJHXJzJHUJTNH0g4t+giezDxs5POzgLOGnz933uNetoMSj1rsuiUtP2aOpC6ZOZK6ZOZIKqGLI3gkSZIkSZJUkQMeSZIkSZKkxjngkSRJkiRJapwDHkmSJEmSpMY54JEkSZIkSWrcot9Fq2sxNcXUXnsVr7v1WzcUrwlARPGSU3vuWbwmwG8f9Ygqdd/2tc9UqfuK455Upe7Mxo3Fa8bKOi+xmU2bqtSVuvL4ex1Xpe6/3HBZlbpPPOiEKnWlOSKIPfYoXjZb+jejwv4TwOMPfnCVumd98/wqdZ97aJ19MzLr1K2h0nOhqW2guaZWFC+Zt9fJx7946MOr1D398vdWqfv8WpmjznkEjyRJkiRJUuMc8EiSJEmSJDXOAY8kSZIkSVLjHPBIkiRJkiQ1zgGPJEmSJElS4xzwSJIkSZIkNc4BjyRJkiRJUuN2OeCJiMMi4uoSK4uIb0TE/iVqSZpc5o6kLpk5krpk5kiqxSN4JEmSJEmSGrfQAc/KiHh/RHwlIj4YEXsBRMSjI+LyiNgQEe+OiD12tnxWROwZEedGxAsL/zySJoe5I6lLZo6kLpk5kopb6IDnfsBfZub9gZuBl0TEGuAs4OmZ+SBgJfDiHS0fqbUW+Cjwgcx8585WGhGnRsQlEXHJ5pnbd+PHkjQBes+dLWwq/TNJWrr6z5x0X0daRvrPHPdzpImz0AHPf2XmhcPP3wecxCCUrs/Mrw6Xvwd45E6Wz/on4MzMfO+uVpqZp2fm+sxcv3pqzQJblTQhes+dVeyxq4dLmhz9Z064ryMtI/1njvs50sRZ6IAnd3F/d1wIPCEiYowakiafuSOpS2aOpC6ZOZKKW+iA55CIOHH4+TOBzwPXAYdFxH2Gy58DnL+T5bN+B/gh8BfjNC5p4pk7krpk5kjqkpkjqbiFDniuA34tIr4C7Af8VWbeDjwP+IeI2ADMAO/Y0fJ59X4d2DMi/qjEDyFpIpk7krpk5kjqkpkjqbiVu3pAZn4DOGoHX/s0cPxuLD9s5O7zFtqkpOXF3JHUJTNHUpfMHEm1LPQIHkmSJEmSJC1RDngkSZIkSZIa54BHkiRJkiSpcQ54JEmSJEmSGueAR5IkSZIkqXG7fBetpSNgKvpuYuGi/Oxs5vZNxWsCkDNVyr78F06tUnfqXlur1I24R/GaX3vOfsVrAhz5h9dWqZs/UX4bAHBlnbLVRRCrVhcvm1s2F6/Zmv94y4lV6v78Qw+qUjdW/neVulNHHl6+6P/cVL4mMHNInXyYuv7bxWvGD1cUr9mFWDHF1Nq9i9ed3lRp/yHa2S9bUWG7Apxy38dUqRsr6+ybTd3n0OI1r33xXYvXBDjqz39QpS4ryu+jx9c/X7xmF2L1alYefEjxuluv/8/iNYEqf7PEEeVfEwDT11xXpe6pj3pOlbrX/+0+Veoe/qyri9eMVQ2NMICpfepsW763g/XVWZskSZIkSZK64oBHkiRJkiSpcQ54JEmSJEmSGueAR5IkSZIkqXEOeCRJkiRJkhrngEeSJEmSJKlxvQ14IuKivtYtaXkydyR1ycyR1CUzR1JvA57MfHhf65a0PJk7krpk5kjqkpkjqc8jeG7pa92SlidzR1KXzBxJXTJzJHkNHkmSJEmSpMat7LuBnYmIU4FTAdZM7d1zN5KWgzm5w149dyNp0s3d11nbczeSJt2czFm5ruduJJW2pI/gyczTM3N9Zq5fHXv23Y6kZWA0d1bFmr7bkTTh5uzrTJk5kuqamzn+R5Y0aZb0gEeSJEmSJEm75oBHkiRJkiSpcX2+TbonmkvqlLkjqUtmjqQumTmSPIJHkiRJkiSpcQ54JEmSJEmSGueAR5IkSZIkqXEOeCRJkiRJkhrngEeSJEmSJKlxK/tuYKFyeprpH93UdxsLNzPddwe9y8uvqVJ3empFlbpx/FHFa/7vn/+n4jUB/vEP7lOl7tQPb6lSt1mZ5LSv5RqO+M0vVqm7NWeq1CWzStm47fbiNdd9eGvxmgA3P/fWKnVz0+byRWfq/L5qy+kZZm5uKIcrvS5qmN64sUrdWFFnn6SaqfL/t3vP+36/eE0Avndjnbr3OKBO3Qbl5s1svf4/+25j4SpkzvQ11xWvWdP0166vUvfIV9+rSt2ZNXsUr7nxwwcWrwmw7infrVL31oceXqUuH93+Yo/gkSRJkiRJapwDHkmSJEmSpMY54JEkSZIkSWqcAx5JkiRJkqTGOeCRJEmSJElqnAMeSZIkSZKkxjngkSRJkiRJalynA56IOC8i1ne5TknLl5kjqWvmjqQumTmSRnkEjyRJkiRJUuN2OeCJiMMi4tqIOCsivhoR74+Ix0TEhRHx7xHxkOHt4oi4PCIuioj7Db93z4j4u4j4SkR8BNhzpO7jht9zWUT8Q0SsrfhzSmqEmSOpa+aOpC6ZOZJqWegRPPcB/gQ4anh7JnAS8Grgt4BrgUdk5vHA7wBvGn7fi4HbMvP+wOuBBwNExP7A64DHZOYJwCXAq+avNCJOjYhLIuKSLWxa3E8oqUW9ZM7wseaOtDz1v6+Tt1f88SQtMf1njvs50sRZucDHXZ+ZGwAi4hrg05mZEbEBOAy4C/CeiDgSSGDV8PseCbwNIDOvioirhssfBjwAuDAiAFYDF89faWaeDpwOsE/cNXf7p5PUql4yZ/h95o60PPW/rzN1NzNHWj76zxz3c6SJs9ABz+h4d2bk/sywxu8Dn83MJ0fEYcB5u6gXwCcz8xkL7lTScmLmSOqauSOpS2aOpOJKXWT5LsANw8+fO7L8AgaHGxIRRwPHDJf/G/BTEXGf4df2joj7FupF0uQzcyR1zdyR1CUzR9JuKzXg+SPg/0bE5cw9KuivgLUR8RXg94BLATLz+wyC6gPDwwovZnDuqSQthJkjqWvmjqQumTmSdtsuT9HKzG8AR4/cf+4OvjY6IX7d8Os/Bn55B3U/A/zkbvYracKZOZK6Zu5I6pKZI6mWUkfwSJIkSZIkqScOeCRJkiRJkhrngEeSJEmSJKlxDngkSZIkSZIa54BHkiRJkiSpcbt8F60lI4LYY4/iZXPz5uI1B4WzTl1Vs+K/f1S85j/97PriNQFue8Q9qtSNV3yvSl0eU6dsdXutIY4q/w6jefk1xWs2Z2a67w52T0SVslu/+a3iNTd87MTiNQEO/cFXqtS94YUPKl5z8/s+WbxmF7bcfS++/ezy/27c808vLl4TqLKvM7VuXfGaADO33FKlbs7U2d+bWr2qSt2Z6/6jeM19nrK6eE2Am550TJW6q24t/+/PzHdWFK/ZhVi5ghX73a143ekf3Fi8poZq7Y/c8O0qdWv0u/bnyu87AfzHe+9fpe7eF3Y7cvEIHkmSJEmSpMY54JEkSZIkSWqcAx5JkiRJkqTGOeCRJEmSJElqnAMeSZIkSZKkxjngkSRJkiRJapwDHkmSJEmSpMY54JEkSZIkSWqcAx5JkiRJkqTGOeCRJEmSJElq3Mq+G9iZiDgVOBVgDXv13I2k5WBO7qy+S8/dSJp0o5mzat1+PXcjadLN2c+ZWttzN5JKW9JH8GTm6Zm5PjPXr4o1fbcjaRmYkzsrHSxLqms0c1bstXff7UiacKOZs3rKv6+kSbOkBzySJEmSJEnaNQc8kiRJkiRJjVsSA56IOCMi1vfdh6TlwcyR1DVzR1KXzBxpeVoSF1nOzBf03YOk5cPMkdQ1c0dSl8wcaXlaEkfwSJIkSZIkafEc8EiSJEmSJDXOAY8kSZIkSVLjHPBIkiRJkiQ1zgGPJEmSJElS4yIz++5hQSLi+8B/LvDh+wM/qNCGddvqtbW6LfW6u3UPzcwDKvRQ1RLInaXwu5vUui312lrdpdCrmbN4k/qcmNS6LfXaWl0zZ66Wfne16rbUa2t1W+p1qdTdbu40M+DZHRFxSWaut275ui312lrdlnqtWbdVPifaqttSr63VbanXlrW0nVvqtbW6LfXaWl0zZ66Wfne16rbUa2t1W+p1qdf1FC1JkiRJkqTGOeCRJEmSJElq3KQOeE63brW6LfXaWt2Weq1Zt1U+J9qq21KvrdVtqdeWtbSdW+q1tbot9dpaXTNnrpZ+d7XqttRra3Vb6nVJ153Ia/CoWxFxS2auHbn/XGB9Zr60QO3zgFdn5iXzlr8UeAVwBHBAZta4yJWkJaqn3Hk/sB7YAnwReFFmbhl3fZKWvp4y510MMieArwLPzcxbxl2fpKWvj8wZ+frbgFNG1692TOoRPJp8FwKPYeFX/pekcb0fOAp4ELAn8IJ+25E04V6Zmcdm5jHAN4Gx/7CTpJ2JiPXAfn33ocVzwKOqIuKAiPhQRHxpePup4fKHRMTFEXF5RFwUEfcbLt8zIv4uIr4SER9h8EfUnWTm5Zn5je5+EkmtqJg7/5JDDI7gObizH0rSklUxc24ePj6Gj/Gwe0nVMiciVgBvAV7T2Q+j4lb23YAmwp4RccXI/bsC5ww//zPgTzPz8xFxCPBx4P7AtcAjMnNrRDwGeBPwVODFwG2Zef+IOAa4rLOfQlJLesudiFgFPAf49aI/kaSlrJfMiYgzgScCXwZ+o/QPJWnJ6iNzXgqck5nfGcyV1SIHPCrhx5l53Oyd2XNEh3cfAzxgJCT2iYi1wF2A90TEkQz+R2rV8OuPBN4GkJlXRcRV9duX1KA+c+cvgQsy83MlfhBJTeglczLzecP/VX878HTgzGI/kaSlrNPMiYh7AU8DTi7+k6hTDnhU2xTwsMy8fXRhRPw58NnMfHJEHAac131rkiZUtdyJiNcDBwAvGr9NSROi6r5OZk5HxN8xOG3CAY+kGplzPHAf4GvDwdFeEfG1zLxPkY7VGa/Bo9o+Abxs9k5EzE6i7wLcMPz8uSOPvwB45vCxRwPH1G9R0oSpkjsR8QLg8cAzMnOmbMuSGlY8c2LgPrOfAz/P4PQLSSqeOZn5scw8MDMPy8zDGJzS5XCnQQ54VNvLgfURcVVEfBn41eHyPwL+b0Rcztwjyf4KWBsRXwF+D7h0e0Uj4uUR8S0GFzm9KiLOqPYTSGpNldwB3gHcA7g4Iq6IiN+p076kxtTInGBwqsUGYANwz+FjJanWfo4mQAzeDESSJEmSJEmt8ggeSZIkSZKkxjngkSRJkiRJapwDHkmSJEmSpMY54JEkSZIkSWqcAx5JkiRJkqTGOeCRJEmSJElqnAMeSZIkSZKkxjngkSRJkiRJapwDHkmSJEmSpMY54JEkSZIkSWqcAx5JkiRJkqTGOeCRJEmSJElqnAMeSZIkSZKkxjngkSRJkiRJapwDHkmSJEmSpMY54JEkSZIkSWqcAx5JkiRJkqTGOeCRJEmSJElqnAMeSZIkSZKkxjngkSRJkiRJapwDHkmSJEmSpMY54JEkSZIkSWqcAx5JkiRJkqTGrey7AS19EbEauO/w7nWZuaXPfiRNPnNHUpfMHEldMnNUS2Rm3z1oCYuIk4H3AN8AAvgJ4Fcy84Ie25I0wcwdSV0ycyR1ycxRTQ54tFMRcSnwzMy8bnj/vsAHMvPB/XYmaVKZO5K6ZOZI6pKZo5q8Bo92ZdVs+ABk5leBVT32I2nymTuSumTmSOqSmaNqJmrAEwP/GBH377uXCXJpRJwREScPb+8ELum7KWmpMHeqMHekHTBzqjBzpB0wc6owc1TNRJ2iFRGPB94N/F1m/kbf/UyCiNgD+DXgpOGizwF/mZmb+utKWjrMnfLMHWnHzJzyzBxpx8yc8swc1TRpA56zgTOBPwMekJlbe26paRGxArgmM4/quxdpqTJ3yjJ3pJ0zc8oyc6SdM3PKMnNU28ScohUR+wMPzMxzgU8Bv9BzS83LzGnguog4pO9epKXI3CnP3JF2zMwpz8yRdszMKc/MUW0TM+ABngN8YPj5mcALeuxlkuwHXBMRn46Ic2ZvfTelfkXEkyNibd99LAHmTh3mjuYwc7Yxc+owczSHmbONmVOHmaM7KZU7E3OKVkRsAJ6QmTcM718JPCkz/6vfztoWET+9veWZeX7XvWhpiIgjgGuBl2XmO/rup0/mTh3mjkaZOXcwc+owczTKzLmDmVOHmaP5SubORAx4ImJf4OmZ+dcjyx4L/CAzL++vM2nyRMQbh58+LjMf0mszPTJ3pG6YOQNmjtQNM2fAzJG6UzJ3JuIUrcz8EXD1vGWfBPbqp6P2RcTnhx83RsTNI7eNEXFz3/2pH8MLwz0NeDNwU0Qc23NLvTF3yjN3NJ+ZcwczpzwzR/OZOXcwc8ozc7Q9pXNnIgY8Q29f4DItQGaeNPy4LjP3Gbmty8x9+u5PvXki8G+ZuZHBW2Y+v+d++mbuFGTuaDvMnLnMnILMHG2HmTOXmVOQmaMdKJo7K4u01KOIOBF4OHBARLxq5Ev7ACv66WqyRMRJwJGZeebwavrrMvP6vvtSL54PvHX4+UeAN0bEqzNzc489dc7cqc/c0ZCZg5nTBTNHQ2YOZk4XzByNKJo7k3AEz2pgLYNh1bqR283AL/bY10SIiNcDvwmcNly0Gnhffx2pL8NzsffNzAsAMvN24IPAo3ptrB/mTkXmjsDMmcfMqcjMEZg585g5FZk5mlUjdyblIssrgLMz86l99zJpIuIK4Hjgssw8frjsqsw8pt/OpH6ZO/WYO9KdmTn1mDnSnZk59Zg5qqn5U7QAMnM6Iu7Vdx8TanNmZkQkQETs3XdD6l5EnLCzr2fmZV31slSYO1WZO8ucmXNnZk5VZs4yZ+bcmZlTlZmjarkzEQOeoSsi4hzgH4BbZxdm5of7a2kinB0Rfw3sGxEvBE4B3tlzT+renww/rgHWA1cCARwDXAKc2FNffTN36jB3ZOZsn5lTh5kjM2f7zJw6zBxBpdyZiFO0ACLizO0szsw8pfNmJkxEPBZ4HIMn3MeHb5GoZSgiPgy8PjM3DO8fDfxuZi7L87HNnXrMHYGZM5+ZU4+ZIzBz5jNz6jFzNKt07kzMgEd1RcQ+jBzxlZn/02M76klEXJOZD9zVMqkEc0dmjrpk5sjMUZfMHEH53JmYU7QiYg2Dtxh7IIPDnABwwjyeiHgR8AbgdmCGwZQ5gXv32Zd6c1VEnMEdV/p/FnBVj/30ytypw9zRCDNnhJlTh5mjEWbOCDOnDjNH8xTNnUl4m/RZfwMcCDweOB84GNg4TsGIuEdEvCsizh3ef0BEPH/sTtvyauDozDwsM++dmYdn5tjhExEHR8RHIuL7EfG9iPhQRBxcoF/V9TzgGuDXh7cvD5ctV0Vzx8zZpnjumDnNMnPmcl+nDvd1NMvMmcvMqcPM0aiiuTMxp2hFxOWZefzsW8xFxCrgc5n5sDFqngucCfx2Zh4bESuByzPzQaX6Xuoi4l+Bp2TmbYXrfhL4Wwb/cAA8G3hWZj625HqkmkrnjpkzUCN3zBxNAvd16nBfR9o+M6cOM0c1TcwpWsCW4ccfDS9M9F3g7mPW3D8zz46I0wAyc2tETI9ZszWnARdFxBeATbMLM/PlY9Y9IDNHL9x2VkS8YsyaAETEocCRmfmpiNgTWJmZY/1vgwYi4qeA3wUOZe45w8v1kNLSuWPmDNTIHTOnQWbOnbivU0dT+zpmTj1mzp2YOXU0lTlg7tRUOncmacBzekTsB7wOOAdYC/yfMWveGhF3Y3BOJBHxMOCmMWu25q+BzwAbGJwjWsqNEfFs4APD+88Abhy3aAzeavBU4K7AEQwOJX0H8OhxawuAdwGvBC4Flts/xttTOnfMnIEauWPmtMnMmct9nTqa2dcxc6ozc+Yyc+poJnPA3OlA0dyZpFO0Ds/M63e1bDdrngC8HTgauBo4AHhaZl45VrMNmT00s0LdQxls2xMZBPxFwMsz85tj1r0CeAjwhdm+I2LDcjrss6aI+EJmPrTvPpaK0rlj5gzUyB0zp01mzlzu69TR0r6OmVOXmTOXmVNHS5kzrGvuVFQ6dybpCJ4PASfMW/ZB4MFj1LwG+Gngfgyubn4dk3Vh6oU4NyJOBT7K3EMIF/02fhGxAnhTZv58gf7m25SZmyNidl0rGf4PgYr4bES8Bfgwc58Pl/XXUq9K546ZM1A0d8ycppk5c7mvU0dL+zpmTl1mzlxmTh0tZQ6YO7UVzZ3mBzwRcRSDt+67S0Q8ZeRL+zDydn6LdHFmnsAgiGbXdxl3DrpJ9ozhx9NGlo31Nn6ZOR0Rh0bE6szcPFZ3d3Z+RPwWsGdEPBZ4CYPwVBmz0+X1I8sSeFQPvfSmYu6YOQNFc8fMaZqZg/s6HWhpX8fMqcvMwczpQEuZA+ZObUVzp/kBD4Pp75OAfYGfG1m+EXjhYgpGxIHAQQyexMczmC7DINT2Wnyr7cnMwyuV/jpwYUScA9w6sr63jln3tcDzGZzT+iLgX4Azxqypocz8mb57WCKK5o6ZM1el3DFzGmTmbOO+TkWN7euYORWZOduYORU1ljlg7lRVOncm6Ro8J2bmxYVq/QrwXAZTtC9xRwBtBM7KzA+XWM9SFhGPyszPzJvabzPuNoiI1++g7hvGqau6IuIewJuAe2Xmz0bEA4ATM/NdPbfWi1K5Y+YM1MwdM6dNZs5c7uuU5b6O5jNz5jJzyjJztD2lc2eSBjx/BLwR+DHwr8AxwCsz831j1HxqZn6oUItNiYg3ZObrI+LM7Xw5M/OUMeufUON85oi4nu2cEzru21sOt8P26o61HVoTEecCZwK/nZnHDs/BvXy5XmStdO4s58yBurlj5rTJzJnLfZ2yWtzXMXPqMnPmMnPKajFzhnWL546Zc4fSuTMJp2jNelxmviYingx8A3gKcAGw6AACDo6IfRhMlt/J4NzQ12bmJ8Ztdqkbhs8UcG5mnl1hFX8yPFTzg8DfZ+bVheqOnru4Bngag7f0G9c/z6v7ZODb4xaNiEcAF2Xm9MiyKuFcyP6ZeXZEnAaQmVsjYjm/jWjp3Fm2mQPVc8fMwcyZAO7rFNTovo6ZU5eZM5eZU1CjmQN1csfMuUPZ3MnMibgB1ww/ngE8Yfj5lWPWvHL48fHARxhcbOyyvn/WjrfrJRVrHwi8HLiQwTmdr6u0nksr1JxiEB7j1rkNOB+4+8iyJfscA84D7jbbI/Aw4Py+++pxexTNHTNn23aokjtmjpnT+s19nWrbtel9HTOn6M9t5szdHmZOne3adOYM11M0d5Zr5gz7K5o7k3QEz0cj4loGhxC+OCIOAG4fs+bsuaFPBN6bmddEROzsGybQpyLi1cDfM/diXYt+G7+RGt8F3hYRnwVeA/wOg8NAFy0iRq/AP8Vg4lzjeX4kcPcCda4D3sLg6vTPz8yLuON5txS9CjgHOCIiLgQOAH6x35Z6VTp3zJyBKrlj5gBmTuvc16mjmX0dM6c6M2cuM6eOZjIHOsud5Zo5UDh3JuYaPAARcVfgphy8TdzewLrhk3yx9c5kcLX3w4FjgRXAeZn54CINN2B4zuV8meOf631/4OnAU4EbGQTchzLze2PW/ezI3a0MDif948y8bsy6GxmcJxrDj98FTssxzyGOiMsy84SIOJLBNng3cEoO3j5ySRqeF3o/Btviuszc0nNLvSqZO2bOQI3cMXO21TVzGue+Tnkt7euYOfWZOXOZOeW1lDnDusVzx8yZq2TuTMSAJyL2Ao7MzCtHlh0CTGfmDWPUnQKOA76emT+KiLsBB2XmVWM3vcxFxMUMXnRnZ+bY51u2KiIuz8zjh5+vZRBCT8nMJXd0Xa3XWatqbA8zpx4zZ8DMaZf7Ou0xd8yclpk57TFz2socqPT3xIQMeFYB1wLHZOatw2WfAH4rMy8Zo24AzwLunZm/N9zYB2bmF8fst0rdYe1jgUcM735u9MmyyHprgJcAJzGYrn4OeEdmjnt4ZhUR8aqdfT0z37rIurO/s8Mz8/dL/s62s65DMvObpeuOq9brrFU1toeZs61mM7lj5tRj5szV0r6OmVOPmVOPmTNXS5lTue6yzhyokztmzkCN19lUwf56MzyE6SPAL8G2qdcBBcL4L4ETgWcM728E/mIxhSLipIhYUbruvHX8OvB+Bucv3h14X0S8bMyy72Vw8bO3A38+/Pxvxujx7OHHDRFx1chtQ0SUmNyvB17M4NDPg4BfZXB1/nXD22LN/s6eObw/1u8sIl4z/Pj2iHjb6A149Rh9VvP/2rv3KFvPuj7g39+ZOfeTiyEBoilEErlI5HpE7kQBYdlWpWpR1sJCF80SSmntkhZbFu2yXlCWusRWbEQIrdQLUErV2lRpkRKugYQkGi4VcHE3xEAuJzmXmad/zD5hzjHnZJ8z7/POvDOfz1qzzt579v6+z96z93fe+Z139nR8nU1Sp8dD56wYrHd0zgqdM30bfV9H5xyzzp69o3M60TnH2uidM1tT197ROXfr0TtbvnOSTq+ztgHeOXqIjyQPTfLu2elXJnnZAJlH38n6mlWXndY7xyd5YpLLh849bhvXJdm76vzeJNetMfPP57nsFPLOn/37wHv6GOAxeHdWfjf46Pkzjj4vNspzYXbbm2f//rMk/+D4j7Wut9dHj9fZlD+Gfjx0zt05g/WOzrn7tjpnE3z0eDyGeq7pnGNu2613dE7fD53T//EY8rnWu3d0zt35g/eOzjlm7YO+zjbk76Kdjtbax2rFg5P8cL5+KN1aHJ5NhVuS1Mo7xy+f5vreDRR/RwAAH65JREFUW1UHhs49TiVZWnV+aXbZWnykqh7fWnt/klTVdyQ57Ylia+2Ls3//co3rOpH7JTm06vyh2WVrNfTX7MtV9Y1JXpjk0qz96zSKTq+zyerweOicFYP1js65m87ZBDbyvo7O+brOvaNzOtI5x9rInTNbX+/e0TkrevSOzpkZ+nW2aQY8M7+Z5PVJrm+t3TJA3muzcsjUfavqZ7Ly58peebphrbVre+Su8sYkH6iqt8/Of39WHpO1eGyS91bV0d9bfECSj1fV9Vl5t/dHnEpYff0d0//Gp2Z5Z65ptSuHPH7wuMfgijVmJsN/zV6X5J1JHpTkw6suP/pO8mt6F/17UlX3b2v4qwerDP06m7ohHw+ds2Kw3tE5d9M5m8eG3dfROSs6947OOYmBekfnHGvDdk7SvXd0zooevaNzjjXY62xTvMnyUbXyLtRfTPIDrbU/GSjzoUmenpUnxztbazdu8NzHZOUNu5KVNwK7Zo15DzzZ5ztOik/b7DE4Ovl891ofg1W5g3/Nqup1rbUXr3lx823rD1trf3uAnMFfZ1M29OOx1Ttnljmp3tE5J9yWzulgKvs6OqcfnXPS7a25d3TOsabSOb1ydc6KHr2jc47JGex1tqkGPAAAAABb0ab4K1oAAAAAW9mmHPBU1WVy++ROaa1Ty53SWnvmTpXnxLRyp7TWqeVOaa1TNqXHeUprnVrulNY6tVydc6wpfe165U5prVPLndJaN3ruphzwJOlVyHKntdap5U5prT1zp8pzYlq5U1rr1HKntNYpm9LjPKW1Ti13SmudWq7OOdaUvna9cqe01qnlTmmtGzp3sw54AAAAALaMybzJ8o7a2XZl71zXPZyD2Z6dc133wY84MPcabrp5KefdZ2Gu637iuj1z557Kek9Fj9wprXVquVNa66nm3pZbvtJaO2/wRXTWq3cOXjhfPyzddkcWzphv+0mycKDmut6RO+/I4u75crffNH9HHsrB7JjzMTiV7zyH213ZXrvmu/Kc39M2wutis+ZuhLXelTtyqB2c7wWxgWzWfZ2N8JzYrLlTWuvUcrdM59ScndMOZnvN+Rjv2T33Gg4fviPbt8+3hjqyNHfuoaU7s2NhjnUcOTJ/5vJd2bFtvv2RQxfumDt36dYDWThzvj7d8amDc+ee0n7ZKcwE1vv1JnfFiX6+Whx8VZ3syt58Rz198Nwrr7x28MwkedYFjx0+dHn+UoON5E/aWzfcn3ucx67szXdse8bguZ/4Nx36IclZ18y/MzGv83/jI4NnJqe2I3FKuYfn31E7teDlTrkdHodt8/1wfsom9D3oA+2d672E02JfJ9N6rcHMZDun9ubxi88aPHf5kQ8fPDNJFm+6dfjQv7p5+Mwkn/3FC7rkXvAjn+qSu3xw/sHRKdG9SfWZ/f7J8lvu8ecrv6IFAAAAMHEGPAAAAAATZ8ADAAAAMHEGPAAAAAATZ8ADAAAAMHEGPAAAAAATZ8ADAAAAMHEGPAAAAAAT12XAU1V7q+oPq+qjVXVDVT13dvnTq+qaqrq+qt5QVTt7bB/YWnQOMDa9A4xJ5wDz6HUEz7OTfKG19sjW2iVJ/mdV7UpyRZLntta+LclikhefLKSqLquqq6vq6sM52GmpwCYwSOckegeYm30dYEzDd07TObDZ9BrwXJ/kmVX181X1lNba15I8JMmnW2ufmF3nTUmeerKQ1trlrbX9rbX922MYDZzQIJ2T6B1gbvZ1gDEN3zkO9oFNp8uAZ1Yyj8lKEf10Vb2qx3YAEp0DjE/vAGPSOcA8er0HzzcmOdBa+60kr8lKGX08yYVVdfHsas9P8qc9tg9sLToHGJveAcakc4B5LHbK/bYkr6mq5SSHk7y4tXZXVb0wyVuqajHJh5L8eqftA1uLzgHGpneAMekc4F51GfC01q5McuU9XP7OJI/usU1g69I5wNj0DjAmnQPMo9ebLAMAAAAwEgMeAAAAgIkz4AEAAACYOAMeAAAAgIkz4AEAAACYuF5/Jn0yltpyl9zaVoNntjZ85iy4Ty5MXVVqcfvgsQs3D5+ZJNvvGP61XBecP3hmkrTPfLZLbo/uTZK23On/Qzosd9vePcOHJlm+7bYuuRxn28LgkX9x+PbBM5OkFoZf67bdewfPTJKlXs9f+1BMWC0sZts3fMPguQtfvGXwzCTJtg7fi889Z/jMJHd8/owuufWAb+qSm0/8RZ/cHqrTz8SbhCN4AAAAACbOgAcAAABg4gx4AAAAACbOgAcAAABg4gx4AAAAACbOgAcAAABg4gx4AAAAACbutAc8VXV2Vb1k1flLq+oPTnDd11fVt57utgB0DjA2vQOMSecAa7WWI3jOTvKSe71Wktbai1prf76GbQHoHGBsegcYk84B1mQtA55XJ7moqq6tqtfMLttXVW+tqo9V1ZurqpKkqt5VVfuraqGqrqiqG6rq+qr68TXfA2Cr0DnA2PQOMCadA6zJ4hpu+4okl7TWHpWsHEKY5NFJHp7kC0muSvKkJO9ZdZtHJfmm1tols9ucfbINVNVlSS5Lkl3Zs4alAptA986ZXUfvAEfZ1wHGNG7nbNs38PKB9Tb0myx/sLX2udbacpJrk1x43Oc/leRBVfWrVfXsJLeeLKy1dnlrbX9rbf/27Bx4qcAmMGjnJMf1Tu0afsXA1NnXAcbUrXN2bNvdZ8XAuhl6wHNw1emlHHeEUGvtliSPTPKuJD+W5PUDbx/YWnQOMDa9A4xJ5wBzW8uvaN2W5IxTuUFVnZvkUGvtbVX18SS/tYbtA1uLzgHGpneAMekcYE1Oe8DTWru5qq6qqhuS/FGSP5zjZt+U5I1VdfTIoZ883e0DW4vOAcamd4Ax6RxgrdZyBE9aa8877qJ3rfrcS1edvnTVdR6zlm0CW5fOAcamd4Ax6RxgLYZ+Dx4AAAAARmbAAwAAADBxBjwAAAAAE2fAAwAAADBxBjwAAAAAE7emv6I1qqrUzp2Dx/7dS54+eGaSfPWHHzJ45tlvuWbwzCTZdvGFXXI/+aPndMl90Cve3yV3Ulpb7xWwBhf/5Ie65Nbu3YNntsOHB89Mkid/5PYuuVc9vk/vVFWX3Haow+Pb6WvWTY/HdqIVWbt2ZuGiiwbPfdnTvmnwzCTZ9r+ODJ659PQvD56ZJKk+/6e5cJ+zu+Qu3fzXXXK77D906kf7OiNYXk47cGD42K9+bfDMJFm477mDZ7YDdw6emSSPeWSf/Zw7Dwz/83BXXb7Hd+qGXl02MkfwAAAAAEycAQ8AAADAxBnwAAAAAEycAQ8AAADAxBnwAAAAAEycAQ8AAADAxI0y4Kmq946xHYCj9A4wJp0DjEnnAPdklAFPa+2JY2wH4Ci9A4xJ5wBj0jnAPRnrCJ7bZ/+eX1Xvrqprq+qGqnrKGNsHth69A4xJ5wBj0jnAPVkceXvPS3Jla+1nqmohyZ6TXbmqLktyWZLsOvlVAU5E7wBjOv3O2X7mCMsDNpnT75zaO8LygDGNPeD5UJI3VNX2JP+ttXbtya7cWrs8yeVJcua2+7QR1gdsPnoHGNNpd85Zu8/XOcCpOv3OWThX58AmM+pf0WqtvTvJU5N8PskVVfWjY24f2Hr0DjAmnQOMSecAq4064KmqByb5cmvtN5K8Psljxtw+sPXoHWBMOgcYk84BVhv7V7QuTfLyqjqc5PYkJsxAb5dG7wDjuTQ6BxjPpdE5wMwoA57W2r7Zv29K8qYxtglsbXoHGJPOAcakc4B7MuqvaAEAAAAwPAMeAAAAgIkz4AEAAACYOAMeAAAAgIkz4AEAAACYuLH/TPrGs9BnxnX2735k8Mzlb3/Y4JlJ8tmn7e2Se9YnWpfcbtrE1kt3laQ6dMS2c88bPDNJ2u23D5552zMfOXhmkrztPy50yb3f4au75G7bvatLbp3ToX/POWv4zCT52P/rk6t779YOHsryX/zl4Lm1Y8fgmUnSnn1o8Myf/Yv3DZ6ZJP/qmx/XJXfpKzd3yYXRdOjgXt8zs7Q0eOTyxRcMnpkkH33fvi653/yAA11y6wtf6pLbx/J6L+CU1OL2PsEn+BbsCB4AAACAiTPgAQAAAJg4Ax4AAACAiTPgAQAAAJg4Ax4AAACAiTPgAQAAAJg4Ax4AAACAiVvzgKeqzq6ql6w6f2lV/cFacwHuic4BxqZ3gDHpHOB0DXEEz9lJXnKv1wIYhs4BxqZ3gDHpHOC0DDHgeXWSi6rq2qp6zeyyfVX11qr6WFW9uaoqSarqsVX1p1X14aq6sqrOH2D7wNaic4Cx6R1gTDoHOC2LA2S8IsklrbVHJSuHECZ5dJKHJ/lCkquSPKmqPpDkV5N8X2vtpqp6bpKfSfIPTxRcVZcluSxJdmXPAEsFNoFunTPL+3rv1N5e9wGYFvs6wJjG6Rz7ObDpDDHguScfbK19Lkmq6tokFyb5apJLkvzxbOC8kOSLJwtprV2e5PIkOXPbfVqntQLTN0jnJMf2zll6Bzgx+zrAmAbvnLMWztU5sMn0GvAcXHV6abadSvJnrbUndNomsHXpHGBsegcYk84B7tUQ78FzW5Iz5rjex5OcV1VPSJKq2l5VDx9g+8DWonOAsekdYEw6Bzgtax7wtNZuTnJVVd2w6k3A7ul6h5L8YJKfr6qPJrk2yRPXun1ga9E5wNj0DjAmnQOcrkF+Rau19rzjLnrXqs+9dNXpa5M8dYhtAluXzgHGpneAMekc4HQM8StaAAAAAKwjAx4AAACAiTPgAQAAAJg4Ax4AAACAiTPgAQAAAJi4Qf6K1ihaSzt4cPDYpQ6ZvWz70I1dct/7e3/aJfe5F13aJbe11iUXjtday/Jddw2eu/y5zw+e2cue//qBLrn/9wvXdsl91q89ukvu0q2HuuTm1luHz/zyXw2fyThaSzt8ZPjcpaXhM5O0I8Ov9V8/9CmDZybJf/7s/+6S+/y/9aQuuZNiv2yy2vJylg8cWO9lzK/H98wvfXn4zCSffEen/Zx/8aguufTTDnfahzwBR/AAAAAATJwBDwAAAMDEGfAAAAAATJwBDwAAAMDEGfAAAAAATJwBDwAAAMDEGfAAAAAATNxpD3iq6mVVdWNVvbmqvreqXnEKt72wqp53utsGth6dA4xJ5wBj0jnAEBbXcNuXJHlGa+1zs/P//fgrVNVia+3IPdz2wiTPS/Jf1rB9YGvROcCYdA4wJp0DrNlpDXiq6teTPCjJH1XVG5LckmR/a+2lVXVFkruSPDrJVVX1jiS/MrtpS/LUJK9O8rCqujbJm1prv7y2uwFsZjoHGJPOAcakc4ChnNaAp7X2Y1X17CTf2Vr7SlW94LirXJDkia21par6/ST/uLV2VVXty0pBvSLJT7TW/s7JtlNVlyW5LEl2Zc/pLBXYBMbqnETvADoHGJfOAYbS602W39JaW5qdvirJL1XVy5KcfYLDCu9Ra+3y1tr+1tr+7dnZZaHApjBI5yR6B5iLzgHGpHOAufQa8Nxx9ERr7dVJXpRkd1YOK3xop20CW5fOAcakc4Ax6RxgLmt5k+W5VNVFrbXrk1xfVd+e5KFJPpvkjN7bBrYenQOMSecAY9I5wMn0OoJntX9WVTdU1XVJDif5oyTXJVmqqo9W1Y+PsAZg69A5wJh0DjAmnQOc0GkfwdNau3DV6SuSXDE7/YLjrvdPThDxXae7bWDr0TnAmHQOMCadAwxhjCN4AAAAAOjIgAcAAABg4gx4AAAAACbOgAcAAABg4gx4AAAAACbutP+KFuNrhw91yf37D3hyl9wrP/eBLrnP+sZHdcmFe1Q1fGZrw2cmybaF4TOXl4bPTPI9T/+hLrnPuv7DXXL/+DvO75K7fOddw4e25eEzk37PW47V4evXjkzna9cOH+mS+6Pf/LQuuW//3Hu75D7ngsd1yYW/YUL7Dj3Wum3H9sEzk+TZD+zzGn7NZ97dJffl3/yELrn2HcbnCB4AAACAiTPgAQAAAJg4Ax4AAACAiTPgAQAAAJg4Ax4AAACAiTPgAQAAAJg4Ax4AAACAibvXAU9VXVhVNwyxsar6TFWdO0QWsHnpHWBMOgcYk84BenEEDwAAAMDEzTvgWayqN1fVjVX11qrakyRV9fSquqaqrq+qN1TVzpNdflRV7a6qP6qqfzTw/QE2D70DjEnnAGPSOcDg5h3wPCTJr7XWHpbk1iQvqapdSa5I8tzW2rclWUzy4hNdviprX5LfT/LbrbXfONlGq+qyqrq6qq4+nIOncLeATUDvAGPSOcCYdA4wuHkHPJ9trV01O/1bSZ6clVL6dGvtE7PL35TkqSe5/Kh3JHlja+0/3dtGW2uXt9b2t9b2b8/Oe7s6sLnoHWBMOgcYk84BBjfvgKfdy/lTcVWSZ1dVrSED2Pz0DjAmnQOMSecAg5t3wPOAqnrC7PTzkrwnyceTXFhVF88uf36SPz3J5Ue9KsktSf7DWhYObHp6BxiTzgHGpHOAwc074Pl4kn9cVTcm+YYkr2ut3ZXkhUneUlXXJ1lO8usnuvy4vH+aZHdV/cIQdwLYlPQOMCadA4xJ5wCDW7y3K7TWPpPkoSf43DuTPPoULr9w1dkXzrtIYGvRO8CYdA4wJp0D9DLvETwAAAAAbFAGPAAAAAATZ8ADAAAAMHEGPAAAAAATZ8ADAAAAMHH3+le0Noyq1PYdw8fu2jl4ZpIs33Fg+NC2PHxmkiwvdYn9nkc+s0vuwpkHu+R++UcePnjmudfcPnhmkix8tcPzK8mBi87pkpv/8dY+uWNobb1XML9Or+Uelm78ZJfcKy85s0tuckeX1Pu9b/j1fvo1Dxs8M0n2vP2DXXIn9RrrrHbvyraLHzJ4bvv4pwbPTJJsq+EzL7l4+Mwkt110RpfcH3ryN3bJXbygU58vDP9/u+2Wrw2emSTZsb1LbC0O/+NPfWU6P1KtVtu3Z/H+9x88d+mmrwyemSTbdu8aPnRhYfjMJF/+oeG7PEle8cwLuuQunHtbl9zat2fwzHZ7n5+DcrDPz5i573365J5gV9oRPAAAAAATZ8ADAAAAMHEGPAAAAAATZ8ADAAAAMHEGPAAAAAATZ8ADAAAAMHHrNuCpqveu17aBrUnvAGPSOcCYdA6wbgOe1toT12vbwNakd4Ax6RxgTDoHWM8jeG5fr20DW5PeAcakc4Ax6Rxgcb0XcDJVdVmSy5JkV/as82qArUDvAGM6pnO2n7nOqwE2u2M6Z+GMdV4NMLQN/SbLrbXLW2v7W2v7t9eu9V4OsAUc0zvZud7LATa51Z2zY3Hvei8H2OSO6Zxtu9d7OcDANvSABwAAAIB7Z8ADAAAAMHEGPAAAAAATt55/Jn3fem0b2Jr0DjAmnQOMSecAjuABAAAAmDgDHgAAAICJM+ABAAAAmDgDHgAAAICJM+ABAAAAmLjF9V7A3FpLO3xo+NgOmaxot93WJbd27eySe7/3/PXgmWdeftPgmUlyy5OGX2uS7L3tji65U1W7dmbhQd8yeO7SjZ8cPLOX2r6jT+5Cn/9faEeOdMnNwkKX2Jv/3p7BM8848+bBM5Pk5uc/vkvuUoen2JG3v3/40BG0O+/K8g0fW+9lrK8P/1mX2H0f7hKbTo3TTe0cfh/qHZ96z+CZSfK9FzyuS25a6xA5tWfCTFtOu/PO4WMPHhw8M0mWOuX2cN6vv69L7lJVl9xa3N4ld2HPrsEzv/Sb5wyemST3/wd/1SW37enzs+uJOIIHAAAAYOIMeAAAAAAmzoAHAAAAYOIMeAAAAAAmzoAHAAAAYOIMeAAAAAAmzoAHAAAAYOJGHfBU1buqav+Y2wS2Lp0DjE3vAGPSOcBqjuABAAAAmLh7HfBU1YVV9bGquqKqPlFVb66qZ1TVVVX1yap63OzjfVV1TVW9t6oeMrvt7qr6naq6sarenmT3qtzvnt3mI1X1lqra1/F+AhOhc4Cx6R1gTDoH6GXeI3guTvKLSR46+3hekicn+Ykk/yrJx5I8pbX26CSvSvKzs9u9OMmB1trDkvybJI9Nkqo6N8krkzyjtfaYJFcn+efHb7SqLquqq6vq6sM5eHr3EJiidemc2XXv7p1DSwc63T1gA7KvA4xp3Tvn0PJdHe8esB4W57zep1tr1ydJVf1Zkne21lpVXZ/kwiRnJXlTVX1LkpZk++x2T03y2iRprV1XVdfNLn98km9NclVVJcmOJO87fqOttcuTXJ4kZ9Y57ZTvHTBV69I5s9vd3Ttn7T5f78DWYV8HGNO6d85Z28/TObDJzDvgWf1fSsurzi/PMv5dkv/TWntOVV2Y5F33kldJ/ri19iNzrxTYSnQOMDa9A4xJ5wCDG+pNls9K8vnZ6ResuvzdWTncMFV1SZJHzC5/f5InVdXFs8/traoHD7QWYPPTOcDY9A4wJp0DnLKhBjy/kOTnquqaHHtU0OuS7KuqG5P8VJIPJ0lr7aasFNVvzw4rfF9WfvcUYB46Bxib3gHGpHOAU3avv6LVWvtMkktWnX/BCT63ekL8ytnn70zywyfI/d9Jvv0U1wtscjoHGJveAcakc4BehjqCBwAAAIB1YsADAAAAMHEGPAAAAAATZ8ADAAAAMHEGPAAAAAATd69/RWvDqErt3Dl4bDt4cPBMVizfdVeX3IV9e7vktk9/dvDMW79v9+CZSdKe8Mguubf921u75Oa7+8T2dmTfYm56wrmD555z4ycHz+ylHT7UKbdLbDcLZ5/VJffIF780eGZ9ZcfgmUmy/MTzuuQu3tkGz6zlwSNHUbt3ZduDh/+rxsvXf3zwzCRJdfh/wuWl4TOTpKpPbhv++Zski+ffv0tuOzR8pz/n0ucOnpkkt/39+3bJvfM+wz9vj/ze+wfPHMXiYnK/4fdzcvNfD59JV9s67ee0vcP/LLTnTWcPnpkktz+1z2Ow6yt99qVPxBE8AAAAABNnwAMAAAAwcQY8AAAAABNnwAMAAAAwcQY8AAAAABNnwAMAAAAwcQY8AAAAABNnwAMAAAAwcQY8AAAAABNnwAMAAAAwcYvrvYCTqarLklyWJLuyZ51XA2wFq3tnx95vWOfVAJvdMfs6289c59UAm53Ogc1tQx/B01q7vLW2v7W2f3vtWu/lAFvA6t5Z3L13vZcDbHKrO2fHos4B+jqmcxb8BzpsNht6wAMAAADAvTPgAQAAAJi4DTHgqarXV9X+9V4HsDXoHGBsegcYk86BrWlDvMlya+1F670GYOvQOcDY9A4wJp0DW9OGOIIHAAAAgNNnwAMAAAAwcQY8AAAAABNnwAMAAAAwcQY8AAAAABNXrbX1XsNcquqmJH8559XPTfKVDsuQO621Ti13Sms91dwHttbO67CGrjZA72yEr91mzZ3SWqeWuxHWqnNO32Z9TmzW3CmtdWq5OudYU/ra9cqd0lqnljultW6U3HvsnckMeE5FVV3dWtsvd/jcKa11arlTWmvP3KnynJhW7pTWOrXcKa11yqb0OE9prVPLndJap5arc441pa9dr9wprXVquVNa60bP9StaAAAAABNnwAMAAAAwcZt1wHO53G65U1rr1HKntNaeuVPlOTGt3CmtdWq5U1rrlE3pcZ7SWqeWO6W1Ti1X5xxrSl+7XrlTWuvUcqe01g2duynfg4dxVdXtrbV9q86/IMn+1tpLB8h+V5KfaK1dfdzlVyR5WpKvzS56QWvt2rVuD5iGdeqdSvLTSX4oyVKS17XWXrvW7QEb3zp1zv9Ncsbs7H2TfLC19v1r3R6w8a1T5zw9yWuychDI7Vn5+er/rXV7jGtxvRcAa/Dy1tpb13sRwJbxgiR/K8lDW2vLVXXfdV4PsIm11p5y9HRVvS3JO9ZxOcDm97ok39dau7GqXpLklVnZ92FCNuuvaLFBVNV5VfW2qvrQ7ONJs8sfV1Xvq6prquq9VfWQ2eW7q+p3qurGqnp7kt3regeAyenYOy9O8lOtteUkaa391Sh3CNjQeu/rVNWZSb4ryX/rfmeADa9j57QkZ85On5XkC93vDINzBA9D2F1Vq3896pwk/312+leS/HJr7T1V9YAkVyZ5WJKPJXlKa+1IVT0jyc8m+YGs/AB1oLX2sKp6RJKPnGS7P1NVr0ryziSvaK0dHPZuARvYevTORUmeW1XPSXJTkpe11j45+D0DNqL12tdJku9P8s7W2q0D3h9gY1uPznlRkv9RVXcmuTXJ4we/V3RnwMMQ7mytPeromaO/Izo7+4wk37ry1hVJkjOral9WpsJvqqpvycq0ePvs809N8tokaa1dV1XXnWCbP5nkS0l2ZOXNqP5lkp8a6g4BG9569M7OJHe11vZX1d9L8oYkTznBdYHNZT0656gfSfL6Ie4EMBnr0Tk/nuR7WmsfqKqXJ/mlrAx9mBADHnrbluTxrbW7Vl9YVf8+yf9prT2nqi5M8q5TCW2tfXF28mBVvTHJT6x9qcAm0aV3knwuyX+dnX57kjeubZnAJtGrc1JV5yZ5XJLnrH2ZwCYxeOdU1XlJHtla+8Dsot9N8j8HWS2j8h489Pa/kvyTo2eq6ugk+qwkn5+dfsGq6787yfNm170kySPuKbSqzp/9W1k5dPmGIRcNTFqX3snK+1985+z005J8YpjlAhPXq3OS5AeT/MHxP8gBW1qPzrklyVlV9eDZ+WcmuXG4JTMWAx56e1mS/VV1XVX9eZIfm13+C0l+rqquybFHkr0uyb6qujErv3L14RPkvrmqrk9yfZJzs/KniwGSfr3z6iQ/MOuen4vDloEVvTonSX44yW93WDMwXYN3TmvtSJJ/lORtVfXRJM9P8vKO94FOqrW23msAAAAAYA0cwQMAAAAwcQY8AAAAABNnwAMAAAAwcQY8AAAAABNnwAMAAAAwcQY8AAAAABNnwAMAAAAwcf8ftLSL5s5jwS8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1152x576 with 8 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Real translation: this is the first book i've ever done.\n"
     ]
    }
   ],
   "source": [
    "translate(\"este é o primeiro livro que eu fiz.\", plot='decoder_layer4_block2')\n",
    "print (\"Real translation: this is the first book i've ever done.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RqQ1fIsLwkGE"
   },
   "source": [
    "## Summary\n",
    "\n",
    "In this tutorial, you learned about positional encoding, multi-head attention, the importance of masking and how to create a transformer.\n",
    "\n",
    "Try using a different dataset to train the transformer. You can also create the base transformer or transformer XL by changing the hyperparameters above. You can also use the layers defined here to create [BERT](https://arxiv.org/abs/1810.04805) and train state of the art models. Futhermore, you can implement beam search to get better predictions."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "s_qNSzzyaCbD"
   ],
   "name": "transformer.ipynb",
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}