NLP-preprocessing/Word_vectors.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UuPdnyWXh7vb"
      },
      "source": [
        "# Natural Language Processing | Word Vectors\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qOE58pxDZqah"
      },
      "source": [
        "To demonstrate word vectors, we're going to use **Gensim** which we first encountered in the [topic modelling](https://www.nlpdemystified.org/course/topic-modelling) module.<br>\n",
        "\n",
        "At the time of this recording, the default Gensim version in Colab was 3.X, so we'll first upgrade to 4.X."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "2vNBSTzIKkRT"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: gensim==4.* in c:\\users\\akg96\\anaconda3\\envs\\text-summarizer\\lib\\site-packages (4.3.2)\n",
            "Requirement already satisfied: numpy>=1.18.5 in c:\\users\\akg96\\anaconda3\\envs\\text-summarizer\\lib\\site-packages (from gensim==4.*) (1.22.4)\n",
            "Requirement already satisfied: scipy>=1.7.0 in c:\\users\\akg96\\anaconda3\\envs\\text-summarizer\\lib\\site-packages (from gensim==4.*) (1.10.1)\n",
            "Requirement already satisfied: smart-open>=1.8.1 in c:\\users\\akg96\\anaconda3\\envs\\text-summarizer\\lib\\site-packages (from gensim==4.*) (5.2.1)\n"
          ]
        }
      ],
      "source": [
        "# Upgrade gensim just in case.\n",
        "!pip install -U gensim==4.*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "kIMAMpBaoqkP"
      },
      "outputs": [],
      "source": [
        "import collections\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import spacy\n",
        "import tensorflow as tf\n",
        "\n",
        "from gensim.models.keyedvectors import KeyedVectors\n",
        "from sklearn.decomposition import PCA\n",
        "from sklearn.model_selection import train_test_split\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras import layers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SBQlbyiVY3WJ"
      },
      "source": [
        "**NOTE**<br>\n",
        "In this notebook, we won't train standalone word embeddings from scratch. Rather, we'll:\n",
        "1. Use *pretrained* embeddings in one model.\n",
        "2. Train embeddings alongside another model.\n",
        "<br>\n",
        "\n",
        "If you want to try training standalone word embeddings, coding Skip-Gram With Negative Sampling (SGNS) from scratch shouldn't be too hard now that you know all the details. But I recommend just using the **Gensim** library instead:<br>\n",
        "https://radimrehurek.com/gensim/models/word2vec.html<br>\n",
        "https://radimrehurek.com/gensim/auto_examples/tutorials/run_word2vec.html\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A_rgKP1vJ2kT"
      },
      "source": [
        "# Using Pretrained, Third-Party Vectors"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RHMDs5FPKXUx"
      },
      "source": [
        "There are a variety of pretrained, static word vector packages out there. In this section, we'll use the **Google News** vectors, a collection of three million, 300-dimension word vectors trained from three billion words from a Google News corpus (circa 2015)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lQEAXBoBMidI"
      },
      "source": [
        "We'll need to first download the actual word vectors. It's over a gigabyte in size but will fit within our environment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "dIKSA-KMbj8E"
      },
      "outputs": [],
      "source": [
        "# !gdown \"https://drive.google.com/uc?id=1BpfbHu4denceXiv8yfdY3EHgjKIcULku\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "6FtVSlPQbj4F"
      },
      "outputs": [],
      "source": [
        "embedding_file = './GoogleNews-vectors-negative300.bin.gz'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BXQRngPaPWhE"
      },
      "source": [
        "Next, we'll have **gensim** load the vectors through the **KeyedVectors** module which will enable us to look up vectors by tokens and indices.<br>\n",
        "https://radimrehurek.com/gensim/models/keyedvectors.html\n",
        "<br><br>\n",
        "To save time and space, we'll limit ourselves to 200,000 word vectors for now."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "dmRUiKPObjqM"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CPU times: total: 3.55 s\n",
            "Wall time: 6.35 s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "word_vectors = KeyedVectors.load_word2vec_format(embedding_file, binary=True, limit=200000)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ir22T0t3QqPS"
      },
      "source": [
        "Retrieving a word's vector is a matter of using a token as a key."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "wm9lDs3Vfv12"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Vector dimension: (300,)\n",
            "[-1.25976562e-01  2.53906250e-02  1.66992188e-01  5.50781250e-01\n",
            " -7.66601562e-02  1.28906250e-01  1.02539062e-01 -3.94821167e-04\n",
            "  1.21582031e-01  4.32128906e-02  1.72851562e-01 -6.83593750e-02\n",
            "  3.41796875e-01  8.39843750e-02  6.68945312e-02  2.67578125e-01\n",
            " -3.71093750e-02 -5.56640625e-02  1.80664062e-01  1.90429688e-02\n",
            " -5.07812500e-02  9.03320312e-03  1.76757812e-01  6.49414062e-02\n",
            " -6.25000000e-02 -9.42382812e-02 -9.71679688e-02  4.00390625e-01\n",
            "  1.15234375e-01  1.02539062e-01 -1.86767578e-02 -2.69531250e-01\n",
            "  1.80664062e-01  1.25000000e-01 -3.17382812e-02 -5.49316406e-02\n",
            "  3.45703125e-01 -1.57470703e-02  1.82390213e-05  2.07031250e-01\n",
            " -1.25976562e-01 -2.83203125e-01  2.00195312e-01  8.34960938e-02\n",
            " -4.73632812e-02 -3.11279297e-02 -2.61718750e-01  1.69921875e-01\n",
            " -2.02636719e-02  1.53320312e-01 -1.20605469e-01  3.75000000e-01\n",
            " -5.68847656e-02 -4.76074219e-03 -1.95312500e-01 -2.03125000e-01\n",
            "  3.00781250e-01 -1.01074219e-01 -3.18359375e-01 -9.03320312e-02\n",
            " -1.19140625e-01  1.95312500e-01 -8.78906250e-02  1.58203125e-01\n",
            "  1.51977539e-02 -1.60156250e-01 -3.30078125e-01 -4.66796875e-01\n",
            "  1.68945312e-01  2.23388672e-02  1.55273438e-01  1.08398438e-01\n",
            " -3.56445312e-02  9.13085938e-02 -8.69140625e-02 -1.20117188e-01\n",
            " -3.08593750e-01 -2.61230469e-02 -7.22656250e-02 -4.80468750e-01\n",
            "  3.78417969e-02 -1.35742188e-01 -1.02539062e-01 -2.91015625e-01\n",
            " -1.93359375e-01 -4.21875000e-01 -1.06445312e-01  3.55468750e-01\n",
            "  1.66992188e-01 -3.63159180e-03 -7.42187500e-02 -3.22265625e-01\n",
            " -7.51953125e-02 -8.25195312e-02 -2.91015625e-01 -1.25976562e-01\n",
            "  1.68457031e-02  5.00488281e-02  1.27929688e-01 -7.42187500e-02\n",
            " -1.30859375e-01 -2.46093750e-01  6.49414062e-02  1.53320312e-01\n",
            "  2.59765625e-01 -1.05468750e-01  3.57421875e-01 -4.29687500e-02\n",
            " -1.58203125e-01  8.20312500e-02 -5.98144531e-02 -2.34375000e-01\n",
            " -3.22265625e-01 -1.25976562e-01  5.39550781e-02 -1.87500000e-01\n",
            "  1.35742188e-01 -6.59179688e-02  8.36181641e-03 -1.84570312e-01\n",
            " -2.96875000e-01 -1.84570312e-01 -4.73632812e-02 -1.06445312e-01\n",
            " -6.93359375e-02  3.83300781e-02 -3.19824219e-02  3.63769531e-02\n",
            " -1.20117188e-01  1.76757812e-01 -1.15722656e-01  1.98974609e-02\n",
            "  8.64257812e-02  6.07910156e-02 -1.40625000e-01  3.30078125e-01\n",
            "  1.94335938e-01 -1.56250000e-01  3.92578125e-01  1.80816650e-03\n",
            "  7.27539062e-02 -2.53906250e-01 -3.54003906e-02  2.86865234e-03\n",
            " -1.72851562e-01  9.76562500e-03 -1.55639648e-02  3.23486328e-03\n",
            " -1.69921875e-01  1.55273438e-01  7.17773438e-02  4.10156250e-01\n",
            " -2.10937500e-01  1.31835938e-01  7.62939453e-03  4.78515625e-02\n",
            " -4.54101562e-02  7.32421875e-02 -4.06250000e-01 -2.06298828e-02\n",
            " -4.04296875e-01 -1.00585938e-01 -2.03125000e-01  1.55273438e-01\n",
            " -1.89453125e-01  6.59179688e-02  6.54296875e-02 -2.05078125e-01\n",
            "  5.46875000e-02 -3.06396484e-02 -1.54296875e-01 -2.61718750e-01\n",
            "  3.81469727e-03 -8.20312500e-02 -3.20312500e-01  2.84423828e-02\n",
            "  2.69531250e-01  1.73828125e-01 -1.66992188e-01  2.22656250e-01\n",
            "  6.34765625e-02 -1.96289062e-01  1.45507812e-01 -1.56250000e-02\n",
            "  2.60009766e-02 -6.29882812e-02  2.94189453e-02  3.28125000e-01\n",
            " -4.68750000e-02 -1.52343750e-01  6.98242188e-02  3.18359375e-01\n",
            " -1.08398438e-01  3.66210938e-02 -1.99218750e-01  1.64031982e-03\n",
            "  6.40869141e-03 -1.47460938e-01 -6.25000000e-02 -4.36401367e-03\n",
            " -2.75390625e-01  8.54492188e-02 -5.00488281e-02 -3.12500000e-01\n",
            " -1.33789062e-01 -1.99218750e-01  5.17578125e-02 -9.27734375e-02\n",
            " -2.40234375e-01 -7.86132812e-02 -1.54296875e-01 -6.64062500e-02\n",
            " -1.97265625e-01  1.76757812e-01 -1.57226562e-01 -1.63085938e-01\n",
            "  6.00585938e-02 -5.85937500e-02 -2.22656250e-01 -6.59179688e-02\n",
            " -9.37500000e-02 -4.14062500e-01  2.55859375e-01 -1.76757812e-01\n",
            "  2.51953125e-01  1.48437500e-01 -1.04492188e-01 -8.60595703e-03\n",
            " -1.22558594e-01 -9.22851562e-02  4.41894531e-02 -1.70898438e-01\n",
            " -1.98242188e-01  1.92382812e-01  2.85156250e-01 -4.34570312e-02\n",
            "  1.08398438e-01 -5.37109375e-02 -2.09960938e-02  1.46484375e-01\n",
            "  3.82812500e-01  2.31933594e-02 -8.83789062e-02  7.32421875e-02\n",
            " -1.01074219e-01 -1.06445312e-01  4.12109375e-01  2.10937500e-01\n",
            "  2.79296875e-01 -2.08740234e-02  2.07031250e-01  9.81445312e-02\n",
            "  2.39257812e-01  7.66601562e-02  2.02148438e-01 -6.07910156e-02\n",
            " -2.63977051e-03 -1.83593750e-01 -1.57470703e-02 -3.20312500e-01\n",
            "  9.03320312e-02  1.02050781e-01 -4.96093750e-01 -9.71679688e-02\n",
            " -8.10546875e-02 -1.80664062e-01 -1.45507812e-01  8.64257812e-02\n",
            " -2.04101562e-01 -2.02148438e-01 -5.46875000e-02  2.53906250e-01\n",
            "  2.08740234e-02 -1.15722656e-01  2.02148438e-01 -8.05664062e-02\n",
            " -1.04980469e-01 -7.95898438e-02  1.96533203e-02 -2.49023438e-01\n",
            "  1.30859375e-01  2.89062500e-01 -2.25585938e-01  4.55078125e-01\n",
            " -2.73437500e-01 -2.57812500e-01 -3.14941406e-02  4.04296875e-01\n",
            " -2.67578125e-01  2.89062500e-01 -1.83593750e-01 -1.48437500e-01\n",
            " -1.06933594e-01  1.27929688e-01  5.46875000e-01 -8.69140625e-02\n",
            " -1.48315430e-02  6.98242188e-02 -8.49609375e-02 -1.55273438e-01]\n"
          ]
        }
      ],
      "source": [
        "pizza = word_vectors['pizza']\n",
        "print(f'Vector dimension: {pizza.shape}')\n",
        "\n",
        "# The embedding for the word 'pizza'.\n",
        "print(pizza)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ORLN9U9fq8hp"
      },
      "source": [
        "We can get the cosine similarity between two words using the *similarity* method.\n",
        "https://radimrehurek.com/gensim/models/keyedvectors.html#gensim.models.keyedvectors.KeyedVectors.similarity"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G-7Iuo2orp60"
      },
      "source": [
        "In this case, we see words which we expect to be related have a higher similarity measure..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "r1i6H88Cfp1P"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "0.35505623\n",
            "0.36959887\n",
            "0.47083578\n"
          ]
        }
      ],
      "source": [
        "print(word_vectors.similarity('pizza', 'tomato'))\n",
        "print(word_vectors.similarity('pizza', 'sauce'))\n",
        "print(word_vectors.similarity('pizza', 'cheese'))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6UWdQJzaryQL"
      },
      "source": [
        "...compared to words we don't expect to be related."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "YUXR8uQ0ftWF"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "0.108518735\n",
            "0.11185601\n",
            "0.14119941\n"
          ]
        }
      ],
      "source": [
        "print(word_vectors.similarity('pizza', 'gorilla'))\n",
        "print(word_vectors.similarity('pizza', 'tree'))\n",
        "print(word_vectors.similarity('pizza', 'yoga'))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g3cD4P1tr4fT"
      },
      "source": [
        "Out-of-vocabulary (OOV) words thrown an exception. Bear in mind that looking up even common words here can result in an exception because we loaded only a subset of the vocabulary."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "zLf9bpYihVz7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\"Key 'womblyboo' not present\"\n"
          ]
        }
      ],
      "source": [
        "try:\n",
        "  word_vectors['womblyboo']\n",
        "except KeyError as e:\n",
        "  print(e)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XCQcU72-q0V5"
      },
      "source": [
        "We can compare two sentences by using *n_similarity* which computes the cosine similarity of two **sets** of words.\n",
        "https://radimrehurek.com/gensim/models/keyedvectors.html#gensim.models.keyedvectors.KeyedVectors.n_similarity<br><br>\n",
        "*n_similarity* expects a sentence as a list of words, hence the use of '.split()'. It'll take this list of words, calculate the average of its word vectors, and use the result as an embedding for the whole sentence."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "nwvDT25miXk0"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0.5314661"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "word_vectors.n_similarity(\"dog bites man\".split(), \"canine nips human\".split())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "_rgATfqBisrn"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0.14505778"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "word_vectors.n_similarity(\"martian dolphins are hostile\".split(), \"i flunked calligraphy school\".split())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BZLcvAU4vf6P"
      },
      "source": [
        "One downside of this approach is that word order is thrown out, so two sentences with identical words which mean different things would score a perfect similarity score."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "yikNOChTinp0"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "1.0"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "word_vectors.n_similarity(\"dog bites man\".split(), \"man bites dog\".split())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DLLpgr06vxTQ"
      },
      "source": [
        "Doing quick-and-dirty similarity measures like this is probably best if your corpus is domain-specific and similarity is based more on keywords. The more specific, the better.<br><br>\n",
        "For example, a corpus of business news headlines would probably work well."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "EejHDg24iyum"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0.52682734"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "s1 = \"Volkswagen intends to double electric car sales in China\".lower().split()\n",
        "s2 = \"First Toyota with solid state battery will be hybrid\".lower().split()\n",
        "word_vectors.n_similarity(s1, s2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aWsFWJUXQ9RR"
      },
      "source": [
        "The *most_similar* method returns the words with the closest vectors.<br>\n",
        "https://radimrehurek.com/gensim/models/keyedvectors.html#gensim.models.keyedvectors.KeyedVectors.most_similar"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "2f2CXSQNxZgf"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[('cells', 0.8180485367774963),\n",
              " ('Cells', 0.6146953105926514),\n",
              " ('Cell', 0.532131552696228),\n",
              " ('cellular', 0.5272858738899231),\n",
              " ('tumor_cells', 0.5097643136978149),\n",
              " ('ES_cells', 0.4866572618484497),\n",
              " ('immune_cells', 0.48319196701049805),\n",
              " ('epithelial_cells', 0.4781806170940399),\n",
              " ('molecule', 0.4775747060775757),\n",
              " ('proteins', 0.47494736313819885)]"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "word_vectors.most_similar(positive=['cell'], topn=10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1YN45ETCReXK"
      },
      "source": [
        "We can also combine vectors first, and retrieve the words most similar to their mean. Here, we're combining the vectors for 'cell' and 'phone' and retrieving the vectors closest to that result."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "wvOkR7UWx1su"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[('cell_phone', 0.6861391067504883),\n",
              " ('cellular', 0.6757438778877258),\n",
              " ('cellphone', 0.6633196473121643),\n",
              " ('telephone', 0.6474039554595947),\n",
              " ('Phone', 0.6051893830299377),\n",
              " ('phones', 0.5896332263946533),\n",
              " ('cells', 0.585094690322876),\n",
              " ('cordless_phone', 0.5711769461631775),\n",
              " ('Cell', 0.5617641806602478),\n",
              " ('landline', 0.5467295050621033)]"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "word_vectors.most_similar(positive=['cell', 'phone'], topn=10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PjMdhqJvR7ez"
      },
      "source": [
        "Given a collection of words, the *doesn't_match* method returns the word that \"doesn't go\" with the rest (i.e. with the vector that's furthest away from the mean of all the other vectors).<br>\n",
        "https://radimrehurek.com/gensim/models/keyedvectors.html#gensim.models.keyedvectors.KeyedVectors.doesnt_match"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "oA2NVKHPzExZ"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'hamburger'"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "word_vectors.doesnt_match([\"apple\", \"orange\", \"hamburger\", \"banana\", \"kiwi\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xajz9UFQR91b"
      },
      "source": [
        "We can see the power of context in this example with 'Toyota' being correctly identified as the odd one out."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "evIAtmcVygyn"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'Toyota'"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "word_vectors.doesnt_match([\"Microsoft\", \"Apple\", \"Toyota\", \"Amazon\", \"Netflix\", \"Google\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h6UyREbVUS6p"
      },
      "source": [
        "Visualizing word vectors is straight-forward and can offer insights into what kind of contexts the training algorithm picked up.<br><br>\n",
        "Because these word vectors have a dimension of 300, we need to reduce them down to two dimensions to plot them on a regular graph. This can be done through **Principal Components Analysis (PCA)**:<br>\n",
        "https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html<br>\n",
        "<br>\n",
        "Here, we're plotting the words we considered in the slides."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "wC1e7EF0-b-s"
      },
      "outputs": [],
      "source": [
        "def display_pca_scatterplot(model, words):\n",
        "    word_vectors = np.array([model[w] for w in words])\n",
        "\n",
        "    twodim = PCA().fit_transform(word_vectors)[:,:2]\n",
        "\n",
        "    plt.figure(figsize=(10,10))\n",
        "    plt.scatter(twodim[:,0], twodim[:,1], edgecolors='k', c='r', s=128)\n",
        "    for word, (x,y) in zip(words, twodim):\n",
        "        plt.text(x+0.05, y+0.05, word)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "cqmNesyaEhq_"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1sAAAMtCAYAAABpR2vZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABSp0lEQVR4nO3de5xXdYH/8fd3QGFGZoa8cUkQNUMIFPMW6iqtlJcyra01pbyseUfFQUt/5Q1r2RLQ8EZ3rWhtu2hmZhkJlpKm4q4JumquWAG6qcwIAypzfn9gs5GADHHmO+Dz+Xh8H4/m+z2f7/fzPY/ZWV+ccz6nUhRFEQAAADaommpPAAAAYFMktgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAErQvdoT2NDa2trypz/9KfX19alUKtWeDgAAUCVFUaSlpSX9+/dPTU3nH2fa5GLrT3/6UwYMGFDtaQAAAF3EM888k+22267TP3eTi636+vokK3doQ0NDlWcDAABUS3NzcwYMGNDeCJ1tk4utv5w62NDQILYAAICqXV5kgQwAAIASiC0AAIASiC0AAIASiC0AAIASiC0AAIASiC0AAIASiC0AAIASiC0AAIASiC0AAIASiC0AAIASiC0AAIASiC0AAIASiC0AqmbUqFEZN25ctacBAKUQWwAAACUQWwAAACUQWwBU1auvvpqxY8emsbExW2+9dS688MIURZEkqVQqufnmm1fZvnfv3rn++uuTJC+//HLGjh2bfv36pWfPntl+++0zceLETv4GALB63as9AQDe3G644YaceOKJue+++3L//ffn5JNPzsCBA3PSSSe94dipU6fmlltuyX/8x39k4MCBeeaZZ/LMM890wqwB4I2JLQCqasCAAbniiitSqVQyePDgPPzww7niiivWKbbmz5+fnXfeOfvvv38qlUq23377TpgxAKwbpxECUFXvete7UqlU2n8eOXJkHn/88axYseINxx5//PF56KGHMnjw4Jx11ln5+c9/XuZUAaBDxBYAXValUmm/fusvXnnllfb//c53vjNPPfVULrvssrS2tuaf//mf8+EPf7izpwkAq+U0QgCq6t57713l59/85jfZeeed061bt2yzzTZZsGBB+2uPP/54li5dusr2DQ0NOeqoo3LUUUflwx/+cA455JA8//zz2XLLLTtl/gCwJmILgKqaP39+mpqacsopp+TBBx/MVVddlcmTJydJ/vEf/zFXX311Ro4cmRUrVuRTn/pUNttss/axU6ZMSb9+/bL77runpqYm3/ve99K3b9/07t27St8GAP6P2AKgqo499ti0trZm7733Trdu3XL22Wfn5JNPTpJMnjw5J5xwQv7hH/4h/fv3zxe/+MU88MAD7WPr6+vzhS98IY8//ni6deuWvfbaK7fddltqapwlD0D1VYq/PRl+I9fc3JzGxsYsXrw4DQ0N1Z4OwJtOURRpbW1NbW3tKgtfAEBnq3Yb+Kc/AP5uLS0tue666zJi2LBs1r17tthii2zWvXtGDBuWadOmpaWlpdpTBIBOJ7YAWG9FUWTy5Mnp36dPxp5xRnaYOzdXtbXl20muamvLDnPn5ozTT89b+/bN5MmTX7eyIABsylyzBcB6KYoi54wbly9OnZozk5yXZMDfbHNaUWR+kklLl+bcc8/NM/Pn54orr3R6IQBvCmILgPUyZcqUfHHq1Fyb5LS1bDcwydQkuyQ5Y+rUDBg4MOPHj++UOQJANVkgA4AOa2lpSf8+fXJCa2umdmDcmUluqKvLnxYtSq9evcqaHgAkqX4buGYLgA6bPn16li5blvM6OO68JEtaWzN9+vQypgUAXYrYAqDDpl19dT6Q11+j9UYGJjn8tfEAsKkTWwB0SFtbWx6eNy/vXc+z0A8uijw8d66VCQHY5IktADpk2bJlaWtry/qe+V6fZEVbW1pbWzfktACgyxFbAHRIbW1tutXUpHk9x7ck6VZTk9ra2g05LQDocsQWAB1SqVQybMiQ/Hw975X1s0olw4cOda8tADZ5YguADjt17NjckuSZDo6bn+THr40HgE2d2AKgw8aMGZO6nj1zeQfHXZ5ki9rajBkzpoxpAUCXIrYA6LD6+vpcctlluSrJdes45tokVye5eMIENzQG4E1BbAGwXpqamnL2WWfl9CRnZuUpgqsz/7XXz0hy9llnpampqbOmCABV1b3aEwBg41SpVHLFlVdmwMCBufSii3Jta2sOz8r7aNVn5aqDP6tU8uOsPHVw0oQJaWpqsjAGAG8alWITu6tkc3NzGhsbs3jx4jQ0rO9dYADoiJaWlkyfPj1fuuaaPDx3bla0taVbTU2GDx2aU8eOzTHHHJP6+vpqTxOAN5lqt4HYAmCDKooira2tqa2tdRQLgKqqdhs4jRCADapSqaSurq7a0wCAqrNABgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAlKja2JEydmr732Sn19fbbddtsceeSReeyxx95w3Pe+973ssssu6dmzZ4YPH57bbrutzGkCAABscKXG1qxZs3LGGWfkN7/5Te6444688soree9735slS5asccw999yTo48+OieeeGLmzJmTI488MkceeWR+97vflTlVAACADapSFEXRWR/23HPPZdttt82sWbNywAEHrHabo446KkuWLMmtt97a/ty73vWujBgxItOmTXvDz2hubk5jY2MWL16choaGDTZ3AABg41LtNujUa7YWL16cJNlyyy3XuM3s2bMzevToVZ47+OCDM3v27NVuv3z58jQ3N6/yAAAAqLZOi622traMGzcu++23X4YNG7bG7RYuXJg+ffqs8lyfPn2ycOHC1W4/ceLENDY2tj8GDBiwQecNAACwPjotts4444z87ne/y4033rhB3/eCCy7I4sWL2x/PPPPMBn1/AACA9dG9Mz5k7NixufXWW3PXXXdlu+22W+u2ffv2zaJFi1Z5btGiRenbt+9qt+/Ro0d69OixweYKAACwIZR6ZKsoiowdOzY33XRTfvnLX2aHHXZ4wzEjR47MjBkzVnnujjvuyMiRI8uaJgAAwAZX6pGtM844I9/5znfyox/9KPX19e3XXTU2Nqa2tjZJcuyxx+atb31rJk6cmCQ5++yzc+CBB2by5Ml53/velxtvvDH3339/vvzlL5c5VQAAgA2q1CNb1113XRYvXpxRo0alX79+7Y/vfve77dvMnz8/CxYsaP953333zXe+8518+ctfzm677Zbvf//7ufnmm9e6qAYAAEBX06n32eoM1V5LHwAA6Bqq3Qadep8tAACANwuxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAAUAKxBQAbyP/8z/+kUqnkoYceqvZUAOgCxBYAAEAJxBYAbzovv/xytacAwJuA2AJgkzdq1KiMHTs248aNy9Zbb52DDz44s2bNyt57750ePXqkX79+Of/88/Pqq6+2j7n99tuz//77p3fv3tlqq63y/ve/P08++eQq73vfffdl9913T8+ePbPnnntmzpw5nf3VAOjCxBYAbwo33HBDNt9889x999255JJLcthhh2WvvfbKf/7nf+a6667L1772tXz2s59t337JkiVpamrK/fffnxkzZqSmpiYf/OAH09bWliR56aWX8v73vz9Dhw7NAw88kEsuuSTnnntutb4eAF1Q92pPAAA6w84775wvfOELSZJvfvObGTBgQK6++upUKpXssssu+dOf/pRPfepTueiii1JTU5N/+qd/WmX817/+9WyzzTaZO3duhg0blu985ztpa2vL1772tfTs2TPveMc78oc//CGnnXZaNb4eAF2QI1sAvCnsscce7f973rx5GTlyZCqVSvtz++23X1566aX84Q9/SJI8/vjjOfroo7PjjjumoaEhgwYNSpLMnz+//T123XXX9OzZs/09Ro4c2QnfBICNhSNbALwpbLHFFh3a/vDDD8/222+fr3zlK+nfv3/a2toybNgwi2sAsM4c2QLgTWfIkCGZPXt2iqJof+7uu+9OfX19tttuu/z5z3/OY489ls985jM56KCDMmTIkLzwwguve4//+q//yrJly9qf+81vftNp3wGArk9sAfCmc/rpp+eZZ57JmWeemUcffTQ/+tGPcvHFF6epqSk1NTV5y1vekq222ipf/vKX88QTT+SXv/xlmpqaVnmPY445JpVKJSeddFLmzp2b2267LZMmTarSNwKgKxJbALzpvPWtb81tt92W++67L7vttltOPfXUnHjiifnMZz6TJKmpqcmNN96YBx54IMOGDcs555yTyy+/fJX36NWrV3784x/n4Ycfzu67755Pf/rT+fznP1+NrwNAF1Up/vocik1Ac3NzGhsbs3jx4jQ0NFR7OgCUpCiKtLa2pra2dpWFLgDgL6rdBo5sAbDRaGlpyXXXXZcRw4Zls+7ds8UWW2Sz7t0zYtiwTJs2LS0tLdWeIgC0E1sAdHlFUWTy5Mnp36dPxp5xRnaYOzdXtbXl20muamvLDnPn5ozTT89b+/bN5MmTs4mdtAHARsrS7wB0aUVR5Jxx4/LFqVNzZpLzkgz4m21OK4rMTzJp6dKce+65eWb+/Fxx5ZVOLwSgqsQWAF3alClT8sWpU3NtktPWst3AJFOT7JLkjKlTM2DgwIwfP75T5ggAq2OBDAC6rJaWlvTv0ycntLZmagfGnZnkhrq6/GnRovTq1aus6QHQxVW7DVyzBUCXNX369CxdtizndXDceUmWtLZm+vTpZUwLANaJ2AKgy5p29dX5QF5/jdYbGZjk8NfGA0C1iC0AuqS2trY8PG9e3rueZ7sfXBR5eO5cKxMCUDViC4AuadmyZWlra8v6nmFfn2RFW1taW1s35LQAYJ2JLQC6pNra2nSrqUnzeo5vSdKtpia1tbUbcloAsM7EFgBdUqVSybAhQ/Lz9bxX1s8qlQwfOtS9tgCoGrEFQJd16tixuSXJMx0cNz/Jj18bDwDVIrYA6LLGjBmTup49c3kHx12eZIva2owZM6aMaQHAOhFbAHRZ9fX1ueSyy3JVkuvWccy1Sa5OcvGECW5oDEBViS0AurSmpqacfdZZOT3JmVl5iuDqzH/t9TOSnH3WWWlqauqsKQLAanWv9gQAYG0qlUquuPLKDBg4MJdedFGubW3N4Vl5H636rFx18GeVSn6clacOTpowIU1NTRbGAKDqSj2yddddd+Xwww9P//79U6lUcvPNN691+5kzZ6ZSqbzusXDhwjKnCUAXV6lUMn78+Pxx4cJcc+21efod78iZNTX5eJIza2ry9DvekWuvuy5/XLgw48ePF1oAdAmlHtlasmRJdtttt/zLv/xLPvShD63zuMceeywNDf93G8ttt922jOkBsJGpr6/PqaeemlNPPTVFUaS1tTW1tbXiCoAuqdTYOvTQQ3PooYd2eNy2226b3r17b/gJAbDJqFQqqaurq/Y0AGCNuuQCGSNGjEi/fv3ynve8J3ffffdat12+fHmam5tXeQAAAFRbl4qtfv36Zdq0afnBD36QH/zgBxkwYEBGjRqVBx98cI1jJk6cmMbGxvbHgAEDOnHGAAAAq1cpiqLolA+qVHLTTTflyCOP7NC4Aw88MAMHDsy3vvWt1b6+fPnyLF++vP3n5ubmDBgwIIsXL17lui8AAODNpbm5OY2NjVVrgy6/9Pvee++dX//612t8vUePHunRo0cnzggAAOCNdanTCFfnoYceSr9+/ao9DQAAgA4p9cjWSy+9lCeeeKL956eeeioPPfRQttxyywwcODAXXHBB/vjHP+ab3/xmkuTKK6/MDjvskHe84x1ZtmxZvvrVr+aXv/xlfv7zn5c5TQAAgA2u1Ni6//778+53v7v956ampiTJcccdl+uvvz4LFizI/Pnz219/+eWXV9608o9/TF1dXXbdddf84he/WOU9AAAANgadtkBGZ6n2RXAAAEDXUO026PLXbAEAAGyMxBYAAEAJxBYA6+zll1+u9hQAYKMhtgA2cW1tbfnCF76Qt73tbenRo0cGDhyYz33uc0mST33qU3n729+eurq67LjjjrnwwgvzyiuvtI+95JJLMmLEiHz1q1/NDjvskJ49e1brawDARqfL39QYgL/PBRdckK985Su54oorsv/++2fBggV59NFHkyT19fW5/vrr079//zz88MM56aSTUl9fn09+8pPt45944on84Ac/yA9/+MN069atWl8DADY6ViME2IS1tLRkm222ydVXX51PfOITb7j9pEmTcuONN+b+++9PsvLI1r/+67/mj3/8Y7bZZpuypwsAG1S128CRLYBN2Lx587J8+fIcdNBBq339u9/9bqZOnZonn3wyL730Ul599dXX/T+j7bffXmgBwHpwzRbAJqy2tnaNr82ePTtjxozJYYcdlltvvTVz5szJpz/96dctgrHFFluUPU0A2CSJLYBN2M4775za2trMmDHjda/dc8892X777fPpT386e+65Z3beeec8/fTTVZglAGyanEYIsAnr2bNnPvWpT+WTn/xkNt988+y333557rnn8sgjj2TnnXfO/Pnzc+ONN2avvfbKT37yk9x0003VnjIAbDIc2QLYxF144YUZP358LrroogwZMiRHHXVUnn322XzgAx/IOeeck7Fjx2bEiBG55557cuGFF1Z7ugCwybAaIQAAsEmqdhs4sgWwESqKIkuXLs0m9u9lALBJEVsAG4mWlpZcd911GTFsWDbr3j1bbLFFNuvePSOGDcu0adPS0tJS7SkCAH9FbAF0cUVRZPLkyenfp0/GnnFGdpg7N1e1teXbSa5qa8sOc+fmjNNPz1v79s3kyZMd7QKALsJqhABdWFEUOWfcuHxx6tScmeS8JAP+ZpvTiiLzk0xaujTnnntunpk/P1dceWUqlUrnTxgAaCe2ALqwKVOm5ItTp+baJKetZbuBSaYm2SXJGVOnZsDAgRk/fnynzBEAWD2rEQJ0US0tLenfp09OaG3N1A6MOzPJDXV1+dOiRenVq1dZ0wOALq/abeCaLYAuavr06Vm6bFnO6+C485IsaW3N9OnTy5gWALCOxBZAFzXt6qvzgbz+Gq03MjDJ4a+NBwCqR2wBdEFtbW15eN68vHc9z/Q+uCjy8Ny5ViYEgCoSWwBd0LJly9LW1pb1Pbu8PsmKtra0trZuyGkBAB0gtgC6oNra2nSrqUnzeo5vSdKtpia1tbUbcloAQAeILYAuqFKpZNiQIfn5et4r62eVSoYPHepeWwBQRWILoIs6dezY3JLkmQ6Om5/kx6+NBwCqR2wBdFFjxoxJXc+eubyD4y5PskVtbcaMGVPGtACAdSS2ALqo+vr6XHLZZbkqyXXrOObaJFcnuXjCBDc0BoAqE1sAXVhTU1POPuusnJ7kzKw8RXB15r/2+hlJzj7rrDQ1NXXWFAGANehe7QkAsGaVSiVXXHllBgwcmEsvuijXtrbm8Ky8j1Z9Vq46+LNKJT/OylMHJ02YkKamJgtjAEAXUCk2sTteNjc3p7GxMYsXL05Dw/reoQag62lpacn06dPzpWuuycNz52ZFW1u61dRk+NChOXXs2BxzzDGpr6+v9jQBoMuodhuILYCNUFEUaW1tTW1traNYALAG1W4DpxECbIQqlUrq6uqqPQ0AYC0skAEAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFACsQUAAFCCUmPrrrvuyuGHH57+/funUqnk5ptvfsMxM2fOzDvf+c706NEjb3vb23L99deXOUUAAIBSlBpbS5YsyW677ZZrrrlmnbZ/6qmn8r73vS/vfve789BDD2XcuHH5xCc+kZ/97GdlThMAAGCD617mmx966KE59NBD13n7adOmZYcddsjkyZOTJEOGDMmvf/3rXHHFFTn44INXO2b58uVZvnx5+8/Nzc1/36QBAAA2gC51zdbs2bMzevToVZ47+OCDM3v27DWOmThxYhobG9sfAwYMKHuaAAAAb6hLxdbChQvTp0+fVZ7r06dPmpub09rautoxF1xwQRYvXtz+eOaZZzpjqgAAAGtV6mmEnaFHjx7p0aNHtacBAACwii51ZKtv375ZtGjRKs8tWrQoDQ0Nqa2trdKsAAAAOq5LxdbIkSMzY8aMVZ674447MnLkyCrNCAAAYP2UGlsvvfRSHnrooTz00ENJVi7t/tBDD2X+/PlJVl5vdeyxx7Zvf+qpp+b3v/99PvnJT+bRRx/Ntddem//4j//IOeecU+Y0AQAANrhSY+v+++/P7rvvnt133z1J0tTUlN133z0XXXRRkmTBggXt4ZUkO+ywQ37yk5/kjjvuyG677ZbJkyfnq1/96hqXfQcAAOiqKkVRFNWexIbU3NycxsbGLF68OA0NDdWeDgAAUCXVboMudc0WAADApkJsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlEBsAQAAlKBTYuuaa67JoEGD0rNnz+yzzz6577771rjt9ddfn0qlssqjZ8+enTFNAACADab02Prud7+bpqamXHzxxXnwwQez22675eCDD86zzz67xjENDQ1ZsGBB++Ppp58ue5oAAAAbVOmxNWXKlJx00kk54YQTMnTo0EybNi11dXX5+te/vsYxlUolffv2bX/06dOn7GkCAABsUKXG1ssvv5wHHnggo0eP/r8PrKnJ6NGjM3v27DWOe+mll7L99ttnwIABOeKII/LII4+scdvly5enubl5lQcAAEC1lRpb//u//5sVK1a87shUnz59snDhwtWOGTx4cL7+9a/nRz/6Ub797W+nra0t++67b/7whz+sdvuJEyemsbGx/TFgwIAN/j0AAAA6qsutRjhy5Mgce+yxGTFiRA488MD88Ic/zDbbbJMvfelLq93+ggsuyOLFi9sfzzzzTCfPGAAA4PW6l/nmW2+9dbp165ZFixat8vyiRYvSt2/fdXqPzTbbLLvvvnueeOKJ1b7eo0eP9OjR4++eKwAAwIZU6pGtzTffPHvssUdmzJjR/lxbW1tmzJiRkSNHrtN7rFixIg8//HD69etX1jQBAAA2uFKPbCVJU1NTjjvuuOy5557Ze++9c+WVV2bJkiU54YQTkiTHHnts3vrWt2bixIlJkgkTJuRd73pX3va2t+XFF1/M5Zdfnqeffjqf+MQnyp4qAADABlN6bB111FF57rnnctFFF2XhwoUZMWJEbr/99vZFM+bPn5+amv87wPbCCy/kpJNOysKFC/OWt7wle+yxR+65554MHTq07KkCAABsMJWiKIpqT2JDam5uTmNjYxYvXpyGhoZqTwcAAKiSardBl1uNEAAAYFMgtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAErQKbF1zTXXZNCgQenZs2f22Wef3HfffWvd/nvf+1522WWX9OzZM8OHD89tt93WGdMEAADYYEqPre9+97tpamrKxRdfnAcffDC77bZbDj744Dz77LOr3f6ee+7J0UcfnRNPPDFz5szJkUcemSOPPDK/+93vyp4qAADABlMpiqIo8wP22Wef7LXXXrn66quTJG1tbRkwYEDOPPPMnH/++a/b/qijjsqSJUty6623tj/3rne9KyNGjMi0adPe8POam5vT2NiYxYsXp6GhYcN9EQAAYKNS7TYo9cjWyy+/nAceeCCjR4/+vw+sqcno0aMze/bs1Y6ZPXv2KtsnycEHH7zG7ZcvX57m5uZVHgAAANVWamz97//+b1asWJE+ffqs8nyfPn2ycOHC1Y5ZuHBhh7afOHFiGhsb2x8DBgzYMJMHAAD4O2z0qxFecMEFWbx4cfvjmWeeqfaUAAAA0r3MN996663TrVu3LFq0aJXnFy1alL59+652TN++fTu0fY8ePdKjR48NM2EAAIANpNQjW5tvvnn22GOPzJgxo/25tra2zJgxIyNHjlztmJEjR66yfZLccccda9weAACgKyr1yFaSNDU15bjjjsuee+6ZvffeO1deeWWWLFmSE044IUly7LHH5q1vfWsmTpyYJDn77LNz4IEHZvLkyXnf+96XG2+8Mffff3++/OUvlz1VAACADab02DrqqKPy3HPP5aKLLsrChQszYsSI3H777e2LYMyfPz81Nf93gG3ffffNd77znXzmM5/J//t//y8777xzbr755gwbNqzsqQIAAGwwpd9nq7NVey19AACga6h2G2z0qxECAAB0RWILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBGILAACgBKXG1vPPP58xY8akoaEhvXv3zoknnpiXXnpprWNGjRqVSqWyyuPUU08tc5oAAAAbXPcy33zMmDFZsGBB7rjjjrzyyis54YQTcvLJJ+c73/nOWseddNJJmTBhQvvPdXV1ZU4TAABggysttubNm5fbb789v/3tb7PnnnsmSa666qocdthhmTRpUvr377/GsXV1denbt29ZUwMAAChdaacRzp49O717924PrSQZPXp0ampqcu+996517PTp07P11ltn2LBhueCCC7J06dI1brt8+fI0Nzev8gAAAKi20o5sLVy4MNtuu+2qH9a9e7bccsssXLhwjeOOOeaYbL/99unfv3/+67/+K5/61Kfy2GOP5Yc//OFqt584cWIuvfTSDTp3AACAv1eHY+v888/P5z//+bVuM2/evPWe0Mknn9z+v4cPH55+/frloIMOypNPPpmddtrpddtfcMEFaWpqav+5ubk5AwYMWO/PBwAA2BA6HFvjx4/P8ccfv9Ztdtxxx/Tt2zfPPvvsKs+/+uqref755zt0PdY+++yTJHniiSdWG1s9evRIjx491vn9AAAAOkOHY2ubbbbJNtts84bbjRw5Mi+++GIeeOCB7LHHHkmSX/7yl2lra2sPqHXx0EMPJUn69evX0akCAABUTWkLZAwZMiSHHHJITjrppNx33325++67M3bs2Hz0ox9tX4nwj3/8Y3bZZZfcd999SZInn3wyl112WR544IH8z//8T2655ZYce+yxOeCAA7LrrruWNVUAAIANrtSbGk+fPj277LJLDjrooBx22GHZf//98+Uvf7n99VdeeSWPPfZY+2qDm2++eX7xi1/kve99b3bZZZeMHz8+//RP/5Qf//jHZU4TAABgg6sURVFUexIbUnNzcxobG7N48eI0NDRUezoAAECVVLsNSj2yBQAA8GYltgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtgAAAEogtnhTGDVqVMaNG1ftaQAA8CYitgAAAEogtgAAAEogttjkLFmyJMcee2x69eqVfv36ZfLkyau8/sILL+TYY4/NW97yltTV1eXQQw/N448/vso2X/nKVzJgwIDU1dXlgx/8YKZMmZLevXt34rcAAGBjJ7bY5Jx33nmZNWtWfvSjH+XnP/95Zs6cmQcffLD99eOPPz73339/brnllsyePTtFUeSwww7LK6+8kiS5++67c+qpp+bss8/OQw89lPe85z353Oc+V62vAwDARqpSFEVR7UlsSM3NzWlsbMzixYvT0NBQ7enQyV566aVstdVW+fa3v52PfOQjSZLnn38+2223XU4++eScccYZefvb35677747++67b5Lkz3/+cwYMGJAbbrghH/nIR/LRj340L730Um699db29/3Yxz6WW2+9NS+++GI1vhYAAOuh2m3gyBablCeffDIvv/xy9tlnn/bnttxyywwePDhJMm/evHTv3n2V17faaqsMHjw48+bNS5I89thj2XvvvVd537/9GQAA3ojYAgAAKIHYYpOy0047ZbPNNsu9997b/twLL7yQ//7v/06SDBkyJK+++uoqr//5z3/OY489lqFDhyZJBg8enN/+9rervO/f/gwAAG+ke7UnABtSr169cuKJJ+a8887LVlttlW233Taf/vSnU1Oz8t8Vdt555xxxxBE56aST8qUvfSn19fU5//zz89a3vjVHHHFEkuTMM8/MAQcckClTpuTwww/PL3/5y/z0pz9NpVKp5lcDAGAj48gWm5zLL788//AP/5DDDz88o0ePzv7775899tij/fVvfOMb2WOPPfL+978/I0eOTFEUue2227LZZpslSfbbb79MmzYtU6ZMyW677Zbbb78955xzTnr27FmtrwQAwEbIaoSwDk466aQ8+uij+dWvflXtqQAAsI6q3QZOI2SjUBRFWltbU1tb2ymn802aNCnvec97ssUWW+SnP/1pbrjhhlx77bWlfy4AAJsOpxHSZbW0tOS6667LiGHDsln37tliiy2yWffuGTFsWKZNm5aWlpbSPvu+++7Le97zngwfPjzTpk3L1KlT84lPfKK0zwMAYNPjNEK6nKIoMmXKlFxy4YVZumxZPpDkvUWRhiTNSX5eqeSWJFvU1ubiCRPS1NRk8QoAAF6n2m3gNEK6lKIocs64cfni1Kk5M8l5SQb8zTanFUXmJ5m0dGnOPffcPDN/fq648krBBQBAlyK26FKmTJmSL06dmmuTnLaW7QYmmZpklyRnTJ2aAQMHZvz48Z0yRwAAWBdOI6TLaGlpSf8+fXJCa2umdmDcmUluqKvLnxYtSq9evcqaHgAAG5lqt4EFMugypk+fnqXLluW8Do47L8mS1tZMnz69jGkBAMB6EVt0GdOuvjofyOuv0XojA5Mc/tp4AADoKsQWXUJbW1senjcv713Ps1oPLoo8PHduNrGzYgEA2IiJLbqEZcuWpa2tLet7Jm19khVtbWltbd2Q0wIAgPUmtugSamtr062mJs3rOb4lSbeamtTW1m7IaQEAwHoTW3QJlUolw4YMyc/X815ZP6tUMnzoUPfaAgCgyxBbdBmnjh2bW5I808Fx85P8+LXxAADQVYgtuowxY8akrmfPXN7BcZcn2aK2NmPGjCljWgAAsF7EFl1GfX19LrnsslyV5Lp1HHNtkquTXDxhghsaAwDQpYgtupSmpqacfdZZOT3JmVl5iuDqzH/t9TOSnH3WWWlqauqsKQIAwDrpXu0JwF+rVCq54sorM2DgwFx60UW5trU1h2flfbTqs3LVwZ9VKvlxVp46OGnChDQ1NVkYAwCALqdSbGJ3gW1ubk5jY2MWL16chob1vWsTXUFLS0umT5+eL11zTR6eOzcr2trSraYmw4cOzaljx+aYY45JfX19tacJAEAXVe02EFtsFIqiSGtra2prax3FAgBgnVS7DZxGyEahUqmkrq6u2tMAAIB1ZoEMAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACAEogtAACgdDNnzkylUsmLL77Y6Z89ffr09O7du9M/t1IURdHpn1qi5ubmNDY2ZvHixWloaKj2dAAAgCQvv/xynn/++fTp0yeVSqVTPvMvbbBw4cJUKpVsu+22nfK5f9G9Uz8NAAB4U9p8883Tt2/fqnx2bW1tVQ7EOI0QAABYq+9///sZPnx4amtrs9VWW2X06NH5z//8z9TU1OS5555Lkjz//POpqanJRz/60fZxn/3sZ7P//vsnef1phNdff3169+6dW2+9NYMHD05dXV0+/OEPZ+nSpbnhhhsyaNCgvOUtb8lZZ52VFStWtL/noEGD8tnPfjbHHntsevXqle233z633HJLnnvuuRxxxBHp1atXdt1119x///3tY/72NMJLLrkkI0aMyLe+9a0MGjQojY2N+ehHP5qWlpb2bVpaWjJmzJhsscUW6devX6644oqMGjUq48aNW+f9JrYAAIA1WrBgQY4++uj8y7/8S+bNm5eZM2fmQx/6UHbcccdstdVWmTVrVpLkV7/61So/J8msWbMyatSoNb730qVLM3Xq1Nx44425/fbbM3PmzHzwgx/Mbbfdlttuuy3f+ta38qUvfSnf//73Vxl3xRVXZL/99sucOXPyvve9Lx//+Mdz7LHH5mMf+1gefPDB7LTTTjn22GOztiumnnzyydx888259dZbc+utt2bWrFn5t3/7t/bXm5qacvfdd+eWW27JHXfckV/96ld58MEHO7TvxBYAALBGCxYsyKuvvpoPfehDGTRoUIYPH57TTz899fX1OeCAAzJz5swkK49cnXDCCVm+fHkeffTRvPLKK7nnnnty4IEHrvG9X3nllVx33XXZfffdc8ABB+TDH/5wfv3rX+drX/tahg4dmve///1597vfnTvvvHOVcYcddlhOOeWU7LzzzrnooovS3NycvfbaKx/5yEfy9re/PZ/61Kcyb968PPvss2v87La2tlx//fUZNmxY/uEf/iEf//jHM2PGjCQrj2rdcMMNmTRpUg466KAMGzYs3/jGN1Y5wrYuxFYXUhRFTj755Gy55ZapVCp56KGH1rr9mg7FAgDAhrLbbrvloIMOyvDhw/ORj3wkX/nKV/LCCy8kSQ488MD22Jo1a1b+8R//sT3Afvvb3+aVV17Jfvvtt8b3rqury0477dT+c58+fTJo0KD06tVrlef+Npp23XXXVV5PkuHDh7/uub+c4rg6gwYNSn19ffvP/fr1a/+c3//+93nllVey9957t7/e2NiYwYMHr/H9VkdsdSG33357rr/++tx6661ZsGBBhg0b1qHxRx11VP77v/+7pNkBAPBm1K1bt9xxxx356U9/mqFDh+aqq67K4MGD89RTT2XUqFGZO3duHn/88cydOzf7779/Ro0alZkzZ2bWrFnZc889U1dXt8b33myzzVb5uVKprPa5tra2NY77y8qGq3vub8e90Wevbfv1UVpsfe5zn8u+++6burq6dT7aUhRFLrroovTr1y+1tbUZPXp0Hn/88bKm2OU8+eST6devX/bdd9/07ds33bt3bLHI2traTl/OEgCATV+lUsl+++2XSy+9NHPmzMnmm2+em266KcOHD89b3vKWfPazn82IESPSq1evjBo1KrNmzcrMmTPXer1WV7bjjjtms802y29/+9v25xYvXtzhAxulxdbLL7+cj3zkIznttNPWecwXvvCFTJ06NdOmTcu9996bLbbYIgcffHCWLVtW1jS7jOOPPz5nnnlm5s+fn0qlkkGDBqWtrS0TJ07MDjvskNra2uy2226vuzjwr/3taYTrsspKRz8DAIA3l3vvvTf/+q//mvvvvz/z58/PD3/4wzz33HMZMmRIKpVKDjjggEyfPr09rHbdddcsX748M2bMWOv1Wl1ZfX19jjvuuJx33nm5884788gjj+TEE09MTU1Nh+4RVlpsXXrppTnnnHNWOXdybYqiyJVXXpnPfOYzOeKII7Lrrrvmm9/8Zv70pz/l5ptvXuO45cuXp7m5eZXHxuiLX/xiJkyYkO222y4LFizIb3/720ycODHf/OY3M23atDzyyCM555xz8rGPfWyVFV7eyButsrIhPgMAgE1XQ0ND7rrrrhx22GF5+9vfns985jOZPHlyDj300CQrr9tasWJFe2zV1NTkgAMOaD8atrGaMmVKRo4cmfe///0ZPXp09ttvvwwZMiQ9e/Zc9zcpSvaNb3yjaGxsfMPtnnzyySJJMWfOnFWeP+CAA4qzzjprjeMuvvjiIsnrHosXL/47Z975rrjiimL77bcviqIoli1bVtTV1RX33HPPKtuceOKJxdFHH10URVHceeedRZLihRdeKIri9fv64osvLurq6orm5ub2584777xin332WefPAACAjdXixYs3WBu89NJLRWNjY/HVr351ncd07KKgEi1cuDDJ/60c8hd9+vRpf211LrjggjQ1NbX/3NzcnAEDBpQzyU70xBNPZOnSpXnPe96zyvMvv/xydt9993V+n7WtsrKhPgMAgI1LURRpbW1NbW1th06LezOZM2dOHn300ey9995ZvHhxJkyYkCQ54ogj1vk9OhRb559/fj7/+c+vdZt58+Zll1126cjb/l169OiRHj16dNrndZaXXnopSfKTn/wkb33rW1d5rSPfd22rrKzrZ1xyySW5+eab33ApegAAuq6WlpZ8+9vfzpeuuSa/mzcvK9ra0q2mJsOGDMmpY8dmzJgxq/wjPcmkSZPy2GOPZfPNN88ee+yRX/3qV9l6663XeXyHYmv8+PE5/vjj17rNjjvu2JG3bNe3b98kyaJFi9KvX7/25xctWpQRI0as13tuzIYOHZoePXpk/vz5pV1Y2BmfAQBAdRVFkSlTpuSSCy/M0mXL8oEkpxRFGpI0t7Xl53Pn5ozTT88nx4/PxRMmpKmpydGuJLvvvnseeOCBv+s9OhRb22yzTbbZZpu/6wPXZIcddkjfvn0zY8aM9rhqbm7Ovffe26EVDTcV9fX1Offcc3POOeekra0t+++/fxYvXpy77747DQ0NOe644zr0fm1tbZk0aVK+8IUv5M9//nMGDhyYU045Jeeee25OOeWUnHfeeXnxxRez1VZbtd+w7sQTT8z111+fSy+9NMn/3a/gG9/4xhtGNwAA1VcURc4ZNy5fnDo1ZyY5L8nfXnBzWlFkfpJJS5fm3HPPzTPz5+eKK68UXBtAaddszZ8/P88//3zmz5+fFStWtJ+C9ra3va39jtC77LJLJk6cmA9+8IOpVCoZN25cPvvZz2bnnXfODjvskAsvvDD9+/fPkUceWdY0u7TLLrss22yzTSZOnJjf//736d27d975znfm//2//9fh97rgggvyla98JYccckjuuuuufOc738mjjz6aE088MXPmzMnvfve7tLW1ZcmSJbnzzjszaNCgnHjiiTnqqKPyu9/9Lrfffnt+8YtfJFl592wAALq+KVOm5ItTp+baJGs7fDEwydQkuyQ5Y+rUDBg4MOPHj++UOW7KKkVRFGW88fHHH58bbrjhdc/feeed7ctCViqVVY6SFEWRiy++OF/+8pfz4osvZv/998+1116bt7/97ev8uc3NzWlsbMzixYvT0NCwIb7K36UrXHzY0tKSbbbZJldffXU+8YlPvOH2kyZNyo033pj7778/iWu2AAA2Ri0tLenfp09OaG3N1A6MOzPJDXV1+dOiRe0HSTZW1W6D0u6zdf3116coitc9/vou0kVRrHI6WqVSyYQJE7Jw4cIsW7Ysv/jFLzoUWl1FS0tLrrvuuowYNiybde+eLbbYIpt1754Rw4Zl2rRpq9xUuDPMmzcvy5cvz0EHHbTa17/73e9mv/32S9++fdOrV6985jOfyfz58zt1jgAAbFjTp0/P0mXLcl4Hx52XZElra6ZPn17GtN5USoutN6OiKDJ58uT079MnY884IzvMnZur2try7SRXtbVlh9cuPnxr376ZPHlySjqo+Dq1tbVrfG327NkZM2ZMDjvssNx6662ZM2dOPv3pT+fll1/ulLkBAFCOaVdfnQ/k9ddovZGBSQ5/bTx/ny5zn62NXVe++HDnnXdObW1tZsyY8brTCO+5555sv/32+fSnP93+3NNPP73KNptvvnlWrFhR6hwBANhw2tra8vC8eTllPf9x/+CiyJlz56YoCgtl/B3E1gbSlS8+7NmzZz71qU/lk5/8ZDbffPPst99+ee655/LII49k5513zvz583PjjTdmr732yk9+8pPcdNNNq4wfNGhQnnrqqTz00EPZbrvtUl9fv0ne2wwAYFOxbNmytLW1ZX2vUqpPsqKtLa2tramrq9uQU3tTcRrhBtDS0pJLLrwwZ2btofXXTk8yNsmlF13UfnPhMl144YUZP358LrroogwZMiRHHXVUnn322XzgAx/IOeeck7Fjx2bEiBG55557cuGFF64y9p/+6Z9yyCGH5N3vfne22Wab/Pu//3vp8wUAYP3V1tamW01NmtdzfEuSbjU1a70chTdW2mqE1VKNFUemTZuWM04/Pf9TFB06J3Z+kh0qlVx73XU55ZRTOvy5XWGlQwAAuqYRw4Zlh7lzc9N6/Of+kZVKnn7HOzLn4YdLmFnn2WRXI3wz6cyLD7vaSocAAHRNp44dm1uSPNPBcfOT/Pi18fx9xNbf6S8XH77377j48OHXLj5cm6660iEAAF3TmDFjUtezZy7v4LjLk2xRW5sxY8aUMa03FbH1d9qQFx+uyV9WOjz33HNzQmtr/qcoclNR5LQkY7LyOrGbiiJPFUWOf22lw3PGjRNcAABvYvX19bnksstyVZLr1nHMtUmuTnLxhAkbzQ2NZ86cmUqlkhdffLHaU3kdsfV36oyLD/96pcOpWfPpin9Z6fCaJF+cOjVTpkxZz1kBALApaGpqytlnnZXTk5yZlacIrs78114/I8nZZ52Vpqamzpri323ffffNggUL0tjYWO2pvI4FMjaAMi8+bGlpSf8+fXJCa2umduB9z0xyQ11d/rRo0UbzrxIAAGx4RVFkypQpufSii7KktTWHZ+WlLPVZ+Q//P6tU8uOsPHXw4gkT0tTUtMksvmaBjE1AmRcfTp8+PUuXLct5HXzv85IsaW3N9OnTOzgSAIBNSaVSyfjx4/PHhQtzzbXX5ul3vCNn1tTk40nOrKnJ0+94R6697rr8ceHCjB8/vlND6/vf/36GDx+e2trabLXVVhk9enT+8z//MzU1NXnuueeSJM8//3xqamry0Y9+tH3cZz/72ey///5JXn8a4fXXX5/evXvn1ltvzR577JEk+fjHP56lS5fmhhtuyKBBg/KWt7wlZ511VlasWFHq9xNbG0CZFx925kqHAABsuurr63PqqadmzsMP55VXX82SJUvyyquvZs7DD+eUU05JfX19p85nwYIFOfroo/Mv//IvmTdvXmbOnJkPfehD2XHHHbPVVltl1qxZSZJf/epXq/ycJLNmzcqoUaPW+N5Lly7N1KlT8/Wvfz1J8utf/zof/OAHc9ttt+W2227Lt771rXzpS1/K97///VK/o9jaAMq6+LCzVjoEAODNpVKppK6urqqnCy5YsCCvvvpqPvShD2XQoEEZPnx4Tj/99NTX1+eAAw7IzJkzk6w8cnXCCSdk+fLlefTRR/PKK6/knnvuyYEHHrjG937llVdy3XXXZbfddkuSHHHEEfn1r3+dr33taxk6dGje//73593vfnfuvPPOUr+j2NpAyrj4sDNWOgQAgGrYbbfdctBBB2X48OH5yEc+kq985St54YUXkiQHHnhge2zNmjUr//iP/9geYL/97W/zyiuvZL/99lvje9fV1WWnnXZq/3nbbbfNoEGDVjnI0adPnzz77LPlfLnXiK0NpFKp5Iorr8ykSZNyQ11ddqhUcmSlkuuSfDsrj3gdWalkh0olN9TVZdKkSbniyivX+q8JnbHSIQAAVEO3bt1yxx135Kc//WmGDh2aq666KoMHD85TTz2VUaNGZe7cuXn88cczd+7c7L///hk1alRmzpyZWbNmZc8990xdXd0a33uzzTZb5edKpbLa59ra2kr5bn8htjagDX3xYaVSybAhQ/Lz9Ty8+7NKJcOHDt1kVpMBAGDTUqlUst9+++XSSy/NnDlzsvnmm+emm27K8OHD85a3vCWf/exnM2LEiPTq1SujRo3KrFmzMnPmzLVer9WViK0SbMiLD8tc6RAAAKrl3nvvzb/+67/m/vvvz/z58/PDH/4wzz33XIYMGZJKpZIDDjgg06dPbw+rXXfdNcuXL8+MGTPWer1WVyK2Svb3XnxY5kqHAABQLQ0NDbnrrrty2GGH5e1vf3s+85nPZPLkyTn00EOTrLxua8WKFe2xVVNTkwMOOKD9aNjGwE2NNwKTJ0/Oueeem2uTnLYO21+blQtwTJo0KePHjy93cgAA8FeKokhra2tqa2urfjlLtdvAka2NQBkrHQIAwIbS0tKS6667LiOGDctm3btniy22yGbdu2fEsGGZNm1aWlpaqj3FqnBkayNRFEWmTJmSSy+6KEtaW3N4Vt5Hqz4rVx38WaWSH2flqYMXT5iQpqamqv9LAgAAm7a//DfqJRdemKXLluUDSd5bFGlI0pzk55VKbkn1/hu12m0gtjYyLS0tmT59er50zTV5eO7crGhrS7eamgwfOjSnjh2bY445ptPv/g0AwJtPURQ5Z9y4fHHq1JyZ5LwkA1az3fwkk5JclZVnX73R7Y82pGq3gdjaiHWl82EBAHhz2RjWFah2G4gtAACgQ1paWtK/T5+c0NqaqR0Yd2aSG+rq8qdFi9KrV6+ypteu2m1ggQwAAKBDpk+fnqXLluW8Do47L8mS1tZMnz69jGl1OWILAADokGlXX50PZPXXaK3NwCSHvzb+zUBsAQAA66ytrS0Pz5uX967n1UgHF0Uenjs3m9jVTKsltgAAgHW2bNmytLW1ZX2vgKpPsqKtLa2trRtyWl2S2AIAANZZbW1tutXUpHk9x7ck6VZTk9ra2g05rS5JbAEAAOusUqlk2JAh+fl63nroZ5VKhg8d+qa4dZHYAgAAOuTUsWNzS5JnOjhufpIfvzb+zUBsAQAAHTJmzJjU9eyZyzs47vIkW9TWZsyYMWVMq8sRWwAAQIfU19fnkssuy1VJrlvHMdcmuTrJxRMmdMoNjbsCsQUAAHRYU1NTzj7rrJye5MysPEVwdea/9voZSc4+66w0NTV11hSrrnu1JwAAAGx8KpVKrrjyygwYODCXXnRRrm1tzeFZeR+t+qxcdfBnlUp+nJWnDk6aMCFNTU1vioUx/qJSbGJ3E2tubk5jY2MWL16chob1Xf0fAABYVy0tLZk+fXq+dM01eXju3Kxoa0u3mpoMHzo0p44dm2OOOSb19fWdPq9qt4HYAgAANpiiKNLa2pra2tqqH8Wqdhs4jRAAANhgKpVK6urqqj2NLsECGQAAACUQWwAAACUQWwAAACUQWwAAACUQWwAAACUQWwAAACUQWwAAACUQWwAAACUQWwAAACUQWwAAACUQWwAAACUQWwAAACUQWwAAACUQWwAAACUQWwAAACUQWwAAACUQWwAAACUQWwAAACUQWwAAACUQWwAAACUQWwAAACUQWwAAACUQWwAAACXoXu0JbGhFUSRJmpubqzwTAACgmv7SBH9phM62ycVWS0tLkmTAgAFVngkAANAVtLS0pLGxsdM/t1JUK/NK0tbWlj/96U+pr69PpVKp9nQ2Os3NzRkwYECeeeaZNDQ0VHs6mxT7thz2a3ns23LYr+Wxb8thv5bHvi3HX+/X+vr6tLS0pH///qmp6fwrqDa5I1s1NTXZbrvtqj2NjV5DQ4P/oy+JfVsO+7U89m057Nfy2LflsF/LY9+W4y/7tRpHtP7CAhkAAAAlEFsAAAAlEFusokePHrn44ovTo0ePak9lk2PflsN+LY99Ww77tTz2bTns1/LYt+XoSvt1k1sgAwAAoCtwZAsAAKAEYgsAAKAEYgsAAKAEYgsAAKAEYgsAAKAEYot87nOfy7777pu6urr07t17ncYcf/zxqVQqqzwOOeSQcie6EVqffVsURS666KL069cvtbW1GT16dB5//PFyJ7qRef755zNmzJg0NDSkd+/eOfHEE/PSSy+tdcyoUaNe9zt76qmndtKMu65rrrkmgwYNSs+ePbPPPvvkvvvuW+v23/ve97LLLrukZ8+eGT58eG677bZOmunGpSP79frrr3/d72bPnj07cbYbh7vuuiuHH354+vfvn0qlkptvvvkNx8ycOTPvfOc706NHj7ztbW/L9ddfX/o8N0Yd3bczZ8583e9spVLJwoULO2fCG4mJEydmr732Sn19fbbddtsceeSReeyxx95wnL+za7c++7Waf2fFFnn55ZfzkY98JKeddlqHxh1yyCFZsGBB++Pf//3fS5rhxmt99u0XvvCFTJ06NdOmTcu9996bLbbYIgcffHCWLVtW4kw3LmPGjMkjjzySO+64I7feemvuuuuunHzyyW847qSTTlrld/YLX/hCJ8y26/rud7+bpqamXHzxxXnwwQez22675eCDD86zzz672u3vueeeHH300TnxxBMzZ86cHHnkkTnyyCPzu9/9rpNn3rV1dL8mSUNDwyq/m08//XQnznjjsGTJkuy222655ppr1mn7p556Ku973/vy7ne/Ow899FDGjRuXT3ziE/nZz35W8kw3Ph3dt3/x2GOPrfJ7u+2225Y0w43TrFmzcsYZZ+Q3v/lN7rjjjrzyyit573vfmyVLlqxxjL+zb2x99mtSxb+zBbzmG9/4RtHY2LhO2x533HHFEUccUep8NiXrum/b2tqKvn37Fpdffnn7cy+++GLRo0eP4t///d9LnOHGY+7cuUWS4re//W37cz/96U+LSqVS/PGPf1zjuAMPPLA4++yzO2GGG4+99967OOOMM9p/XrFiRdG/f/9i4sSJq93+n//5n4v3ve99qzy3zz77FKecckqp89zYdHS/duRvLyslKW666aa1bvPJT36yeMc73rHKc0cddVRx8MEHlzizjd+67Ns777yzSFK88MILnTKnTcWzzz5bJClmzZq1xm38ne24ddmv1fw768gW623mzJnZdtttM3jw4Jx22mn585//XO0pbfSeeuqpLFy4MKNHj25/rrGxMfvss09mz55dxZl1HbNnz07v3r2z5557tj83evTo1NTU5N57713r2OnTp2frrbfOsGHDcsEFF2Tp0qVlT7fLevnll/PAAw+s8rtWU1OT0aNHr/F3bfbs2atsnyQHH3yw382/sj77NUleeumlbL/99hkwYECOOOKIPPLII50x3U2a39fyjRgxIv369ct73vOe3H333dWeTpe3ePHiJMmWW265xm383nbcuuzXpHp/Z8UW6+WQQw7JN7/5zcyYMSOf//znM2vWrBx66KFZsWJFtae2UfvL+e59+vRZ5fk+ffo4F/41CxcufN2pKt27d8+WW2651n10zDHH5Nvf/nbuvPPOXHDBBfnWt76Vj33sY2VPt8v63//936xYsaJDv2sLFy70u/kG1me/Dh48OF//+tfzox/9KN/+9rfT1taWfffdN3/4wx86Y8qbrDX9vjY3N6e1tbVKs9o09OvXL9OmTcsPfvCD/OAHP8iAAQMyatSoPPjgg9WeWpfV1taWcePGZb/99suwYcPWuJ2/sx2zrvu1mn9nu5f+CVTF+eefn89//vNr3WbevHnZZZdd1uv9P/rRj7b/7+HDh2fXXXfNTjvtlJkzZ+aggw5ar/fcWJS9b9+s1nW/rq+/vqZr+PDh6devXw466KA8+eST2Wmnndb7feHvNXLkyIwcObL953333TdDhgzJl770pVx22WVVnBms3uDBgzN48OD2n/fdd988+eSTueKKK/Ktb32rijPrus4444z87ne/y69//etqT2WTsq77tZp/Z8XWJmr8+PE5/vjj17rNjjvuuME+b8cdd8zWW2+dJ554YpOPrTL3bd++fZMkixYtSr9+/dqfX7RoUUaMGLFe77mxWNf92rdv39ctNPDqq6/m+eefb99/62KfffZJkjzxxBNvytjaeuut061btyxatGiV5xctWrTG/di3b98Obf9mtD779W9tttlm2X333fPEE0+UMcU3jTX9vjY0NKS2trZKs9p07b333kJiDcaOHdu+mNN222231m39nV13Hdmvf6sz/86KrU3UNttsk2222abTPu8Pf/hD/vznP68SCJuqMvftDjvskL59+2bGjBntcdXc3Jx77723w6tFbmzWdb+OHDkyL774Yh544IHsscceSZJf/vKXaWtraw+odfHQQw8lyZvid3Z1Nt988+yxxx6ZMWNGjjzyyCQrT8eYMWNGxo4du9oxI0eOzIwZMzJu3Lj25+64445V/rXwzW599uvfWrFiRR5++OEcdthhJc500zdy5MjXLZnt97U8Dz300Jv27+maFEWRM888MzfddFNmzpyZHXbY4Q3H+Dv7xtZnv/6tTv07W5VlOehSnn766WLOnDnFpZdeWvTq1auYM2dOMWfOnKKlpaV9m8GDBxc//OEPi6IoipaWluLcc88tZs+eXTz11FPFL37xi+Kd73xnsfPOOxfLli2r1tfokjq6b4uiKP7t3/6t6N27d/GjH/2o+K//+q/iiCOOKHbYYYeitbW1Gl+hSzrkkEOK3Xffvbj33nuLX//618XOO+9cHH300e2v/+EPfygGDx5c3HvvvUVRFMUTTzxRTJgwobj//vuLp556qvjRj35U7LjjjsUBBxxQra/QJdx4441Fjx49iuuvv76YO3ducfLJJxe9e/cuFi5cWBRFUXz84x8vzj///Pbt77777qJ79+7FpEmTinnz5hUXX3xxsdlmmxUPP/xwtb5Cl9TR/XrppZcWP/vZz4onn3yyeOCBB4qPfvSjRc+ePYtHHnmkWl+hS2ppaWn/G5qkmDJlSjFnzpzi6aefLoqiKM4///zi4x//ePv2v//974u6urrivPPOK+bNm1dcc801Rbdu3Yrbb7+9Wl+hy+rovr3iiiuKm2++uXj88ceLhx9+uDj77LOLmpqa4he/+EW1vkKXdNpppxWNjY3FzJkziwULFrQ/li5d2r6Nv7Mdtz77tZp/Z8UWxXHHHVcked3jzjvvbN8mSfGNb3yjKIqiWLp0afHe97632GabbYrNNtus2H777YuTTjqp/T8k+D8d3bdFsXL59wsvvLDo06dP0aNHj+Kggw4qHnvssc6ffBf25z//uTj66KOLXr16FQ0NDcUJJ5ywSsA+9dRTq+zn+fPnFwcccECx5ZZbFj169Cje9ra3Feedd16xePHiKn2DruOqq64qBg4cWGy++ebF3nvvXfzmN79pf+3AAw8sjjvuuFW2/4//+I/i7W9/e7H55psX73jHO4qf/OQnnTzjjUNH9uu4cePat+3Tp09x2GGHFQ8++GAVZt21/WW58b99/GVfHnfcccWBBx74ujEjRowoNt9882LHHXdc5W8t/6ej+/bzn/98sdNOOxU9e/Ysttxyy2LUqFHFL3/5y+pMvgtb3T792/+f7+9sx63Pfq3m39nKa5MGAABgA7L0OwAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAnEFgAAQAn+PwvKyCffYACxAAAAAElFTkSuQmCC",
            "text/plain": [
              "<Figure size 1000x1000 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "display_pca_scatterplot(word_vectors, ['swim', 'swimming', 'cat', 'dog', 'feline', 'road', 'car', 'bus'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EXE5AEEUSnEN"
      },
      "source": [
        "We can even solve analogies (to a limited extent) with vector arithmetic.<br><br>\n",
        "Here, we're solving the analogy:<br>\n",
        "_Rome is to Italy as London is to __________.<br><br>\n",
        "Arithmetically, this is Italy + London - Rome.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "VCJZYi-1zNUI"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[('UK', 0.6230568885803223),\n",
              " ('Britain', 0.6016390323638916),\n",
              " ('United_Kingdom', 0.5343457460403442)]"
            ]
          },
          "execution_count": 20,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "word_vectors.most_similar(positive=['Italy', 'London'], negative=['Rome'], topn=3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vunBH2R5XRWe"
      },
      "source": [
        "Visualizing it can help with geometric intuition."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "RlH9W8q2V0Dm"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1UAAAMtCAYAAAB3jltqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABFn0lEQVR4nO3df7TVdZ3v8dc+oHBAzkET+aGQmI2/UKD8MTizEkcKrHFi1WosuSZe0/EHKuGPtDthal1GEzUUpTtNoRM12S2tsbmaktiMkZpGmqiTjgYaYFmec/ipcb73D+tMJCqHD/scfjwea+21PPt8P3u/t9/2Oj77fvd316qqqgIAAMBmaejuAQAAALZlogoAAKCAqAIAACggqgAAAAqIKgAAgAKiCgAAoICoAgAAKNCzuwfY0trb2/PLX/4y/fr1S61W6+5xAACAblJVVdra2jJkyJA0NNTveNJ2F1W//OUvM3To0O4eAwAA2EosXbo0e+21V90ef7uLqn79+iV59V9cU1NTN08DAAB0l9bW1gwdOrSjEeplu4uqP5zy19TUJKoAAIC6fyzIhSoAAAAKiCoAAIACogoAAKCAqAIAACggqgAAAAqIKgAAgAKiCgAAoICoAgAAKCCqAAAACogqAACAAqIKeEO1Wi233XZbd48BALDVElWwDZk8eXImTpzY3WMAAPBHRBUAAEABUQXbiXvvvTeHH354evXqlcGDB+eiiy7K7373u47fjx07Nuecc04uvPDC7Lbbbhk0aFA+/elPb/AYP//5z/Oud70rvXv3zoEHHpi77rrrNc/z6KOP5q/+6q/S2NiYt7zlLTnttNOycuXKjt//4WjaVVddlcGDB+ctb3lLzjrrrLzyyit1e+0AAN1JVMF24Pnnn8973/veHHbYYfnpT3+aG2+8Mf/0T/+Uz3zmMxtsd9NNN6Vv3765//77c+WVV+ayyy7rCKf29vZ84AMfyM4775z7778/c+bMySc+8YkN1q9atSrjx4/PrrvumgcffDDf+MY3cvfdd2fKlCkbbHfPPffk6aefzj333JObbropc+fOzdy5c+v67wAAoLv07O4BgHI33HBDhg4dmuuvvz61Wi37779/fvnLX+YTn/hEpk+fnoaGV///k0MOOSSXXHJJkuTtb397rr/++syfPz/vfve7c/fdd+eJJ57InXfemSFDhiRJ/vf//t859thjO57nq1/9atauXZubb745ffv2TZJcf/31Oe6443LFFVdk4MCBSZJdd901119/fXr06JH9998/73vf+zJ//vyceuqpXfmvBQCgSzhSBduBxx9/PGPGjEmtVuu47y/+4i+ycuXKPPfccx33HXLIIRusGzx4cF544YWOxxg6dGhHUCXJmDFjXvM8I0eO7AiqPzxPe3t7nnzyyY77DjrooPTo0WOjzwMAsL0RVbAD2WmnnTb4uVarpb29fZt9HgCArYGogu3AAQcckIULF6aqqo777rvvvvTr1y977bXXJj/G0qVLs2zZso77fvSjH71mm5/+9KdZtWrVBs/T0NCQ/fbbr/BVAABsm0QVbGNaWlqyaNGiDW6nnXZali5dmrPPPjtPPPFEvv3tb+eSSy7JtGnTOj5P9WbGjRuXP/uzP8tJJ52Un/70p/n3f//3/K//9b822GbSpEnp3bt3TjrppPzsZz/LPffck7PPPjsnnnhix+epAAB2NC5UAduYBQsWZPTo0Rvcd8opp+Tf/u3fcsEFF2TkyJHZbbfdcsopp+Tv//7vN/lxGxoacuutt+aUU07J4Ycfnr333juzZs3KhAkTOrbp06dP7rzzzpx77rk57LDD0qdPn3zwgx/M1VdfvcVeHwDAtqZW/fH5QtuB1tbWNDc3p6WlJU1NTd09DgAA0E26qg2c/gcAAFBAVEE3qaoqq1evznZ2sBgAYIcjqqALtbW15cYbb8yoESOyU8+e6du3b3bq2TOjRozInDlz0tbW1t0jAgDQSaIKukBVVZk5c2aGDByYKWedleGLF+e69vZ8Jcl17e0ZvnhxzjrzzOw5aFBmzpzp6BUAwDbE1f+gzqqqysenTs3nZ83K2UkuSDL0T7Y5o6qyJMlVq1fn/PPPz9IlS3LNtdemVqt1/cAAAHSKqII6u/rqq/P5WbNyQ5Iz3mC7YUlmJdk/yVmzZmXosGE577zzumRGAAA2n0uqQx21tbVlyMCBOXnNmszqxLqzk9zUp09+uWJFdtlll3qNBwCwXXNJddgOzJs3L6vXrs0FnVx3QZJVa9Zk3rx59RgLAIAtqK5R9YMf/CDHHXdchgwZklqtlttuu+0Nt1+wYEFqtdprbsuXL6/nmFA3c66/Pn+T136G6s0MS3Lc79cDALB1q2tUrVq1KiNHjszs2bM7te7JJ5/MsmXLOm577LFHnSaE+mlvb8+jjz+e92zmGbbjqyqPLl7sSoAAAFu5ul6o4thjj82xxx7b6XV77LFH+vfvv+UHgi60du3atLe3Z3PP3u2XZH17e9asWZM+ffpsydEAANiCtsrPVI0aNSqDBw/Ou9/97tx3331vuO26devS2tq6wQ22Bo2NjenR0JDN/V9kW5IeDQ1pbGzckmMBALCFbVVRNXjw4MyZMyff/OY3881vfjNDhw7N2LFj8/DDD7/umhkzZqS5ubnjNnRoZz+9AvVRq9Uy4oAD8r3N/K6pO2u1HHzggb6rCgBgK9dll1Sv1Wq59dZbM3HixE6tO+qoozJs2LD88z//80Z/v27duqxbt67j59bW1gwdOtQl1dkqzJkzJ2edeWaerapOXaxiSZLhtVpuuPHG/N3f/V29xgMA2K65pPrvHX744Xnqqade9/e9evVKU1PTBjfYWkyaNCl9evfO5zq57nNJ+jY2ZtKkSfUYCwCALWirj6pFixZl8ODB3T0GbJZ+/frl05dfnuuS3LiJa25Icn2SSy67zBf/AgBsA+p69b+VK1ducJTpmWeeyaJFi7Lbbrtl2LBhufjii/P888/n5ptvTpJce+21GT58eA466KCsXbs2X/ziF/P9738/3/ve9+o5JtTVtGnTsnTJkpw5a1YW59Uv9h22ke2W5NUjVNcnOfecczJt2rSuHBMAgM1U16j68Y9/nKOPPrrj5z/8R+JJJ52UuXPnZtmyZVmyZEnH719++eWcd955ef7559OnT58ccsghufvuuzd4DNjW1Gq1XHPttRk6bFgunT49N6xZk+Py6vdQ9curV/m7s1bLv+bVU/6uuuyyTJs2zQUqAAC2EV12oYqu0lUfRoPN0dbWlnnz5uULs2fn0cWLs769PT0aGnLwgQfm9ClTcsIJJ6Rfv37dPSYAwHahq9pAVEE3qaoqa9asSWNjo6NSAAB10FVtUNfT/4DXV6vV0qdPn+4eAwCAQlv91f8AAAC2ZqIKAACggKgCAAAoIKoAAAAKiCoAAIACogoAAKCAqAIAACggqgAAAAqIKgAAgAKiCgAAoICoAgAAKCCqAAAACogqAACAAqIKAACggKgCAAAoIKoAAAAKiCoAAIACogoAAKCAqAIAACggqgAAAAqIKgAAgAKiCgAAoICoAgAAKCCqAAAACogqAACAAqIKAACggKgCAAAoIKoAAAAKiCoAAIACogoAAKCAqAIAACggqgAAAAqIKgAAgAKiCgAAoICoAgAAKCCqAAAACogqAACAAqIKAACggKgCAAAoIKoAAAAKiCoAAIACogoAAKCAqAIAACggqgAAAAqIKgAAgAKiCgAAoICoAgAAKCCqAAAACogqAACAAqIKAACggKgCAAAoIKoAAAAKiCoAAIACogoAAKCAqAIAACggqgAAAAqIKgAAgAKiCgAAoICoAgAAKCCqAAAACogqAACAAqIKAACggKgCAAAoIKoAAAAKiCoAAIACogoAAKCAqAIAACggqgAAAAqIKgAAgAKiCgAAoICoAgAAKCCqAAAACogqAACAAqIKAACggKgCAAAoIKoAAAAKiCoAAIACogoAAKCAqAIAACggqgAAAAqIKgAAgAKiCgAAoICoAgAAKCCqAAAACogqAACAAqIKAACggKgCAAAoIKoAAAAKiCoAAIACogoAAKCAqAIAACggqgAAAAqIKgAAgAKiCgAAoICoAgAAKCCqAAAACogqAACAAqIKAACggKgCAAAoIKoAAAAKiCoAAIACogoAAKCAqAIAACggqgAAAAqIKgAAgAKiCgAAoICoAgAAKCCqAAAACtQ1qn7wgx/kuOOOy5AhQ1Kr1XLbbbe96ZoFCxbkHe94R3r16pV99903c+fOreeIAAAAReoaVatWrcrIkSMze/bsTdr+mWeeyfve974cffTRWbRoUaZOnZqPfexjufPOO+s5JgAAwGbrWc8HP/bYY3Psscdu8vZz5szJ8OHDM3PmzCTJAQcckP/4j//INddck/Hjx9drTAAAgM22VX2mauHChRk3btwG940fPz4LFy583TXr1q1La2vrBjcAAICuslVF1fLlyzNw4MAN7hs4cGBaW1uzZs2aja6ZMWNGmpubO25Dhw7tilEBAACSbGVRtTkuvvjitLS0dNyWLl3a3SMBAAA7kLp+pqqzBg0alBUrVmxw34oVK9LU1JTGxsaNrunVq1d69erVFeMBAAC8xlZ1pGrMmDGZP3/+BvfdddddGTNmTDdNBAAA8MbqGlUrV67MokWLsmjRoiSvXjJ90aJFWbJkSZJXT9376Ec/2rH96aefnv/6r//KhRdemCeeeCI33HBDbrnllnz84x+v55gAAACbra5R9eMf/zijR4/O6NGjkyTTpk3L6NGjM3369CTJsmXLOgIrSYYPH57vfve7ueuuuzJy5MjMnDkzX/ziF11OHQAA2GrVqqqqunuILam1tTXNzc1paWlJU1NTd48DAAB0k65qg63qM1UAAADbGlEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAQLeaPHlyarVaarVadtpppwwfPjwXXnhh1q5d292jbZKe3T0AAADAhAkT8uUvfzmvvPJKHnrooZx00kmp1Wq54oorunu0N+VIFQAA0O169eqVQYMGZejQoZk4cWLGjRuXu+66K0mybt26nHPOOdljjz3Su3fv/OVf/mUefPDBjrULFixIrVbLnXfemdGjR6exsTF/9Vd/lV/96ldJksMOOyxNTU054YQTsnr16o517e3tmTFjRoYPH57GxsaMHDky//f//t9Ozy6qAACArcrPfvaz/PCHP8zOO++cJLnwwgvzzW9+MzfddFMefvjh7Lvvvhk/fnx+85vfbLDu05/+dK6//vr88Ic/zNKlSzN58uQkyRe/+MV897vfzfe+971cd911HdvPmDEjN998c+bMmZPHHnssH//4x/M//sf/yL333tupeWtVVVVlL3nr0tramubm5rS0tKSpqam7xwEAAN7E5MmT85WvfCW9e/fO7373u6xbty4NDQ255ZZbMmHChOy6666ZO3duTjjhhCTJK6+8kr333jtTp07NBRdckAULFuToo4/O3XffnWOOOSZJ8g//8A+5+OKLk6SjDU4//fQ8++yzueOOO7Ju3brstttuufvuuzNmzJiOWT72sY9l9erV+epXv7rJ8/tMFQAA0O2OPvro3HjjjVm1alWuueaa9OzZMx/84AfzyCOP5JVXXslf/MVfdGy700475fDDD8/jjz++wWMccsghHf88cODA9OnTZ4PT/QYOHJgHHnggSfLUU09l9erVefe7373BY7z88ssZPXp0p2YXVQAAQLfr27dv9t133yTJl770pYwcOTL/9E//lMMOO2yTH2OnnXbq+Oc/XEnwj9VqtbS3tydJVq5cmST57ne/mz333HOD7Xr16tWp2UUVAACwVWloaMgnP/nJTJs2LU899VR23nnn3HfffXnrW9+a5NXT/x588MFMnTp1s5/jwAMPTK9evbJkyZIcddRRRfOKKgAAYKvzoQ99KBdccEFuvPHGnHHGGbnggguy2267ZdiwYbnyyiuzevXqnHLKKZv9+P369cv555+fj3/842lvb89f/uVfpqWlJffdd1+amppy0kknbfJjiSoAAGCr07Nnz0yZMiVXXnllnnnmmbS3t+fEE09MW1tbDj300Nx5553Zddddi57j8ssvz4ABAzJjxoz813/9V/r37593vOMd+eQnP9mpx3H1PwAAYLvUVW3ge6oAAAAKiCoAAIACogoAANhiqqrK6tWrs519yugNiSoAAKBIW1tbbrzxxowaMSI79eyZvn37ZqeePTNqxIjMmTMnbW1t3T1iXYkqAABgs1RVlZkzZ2bIwIGZctZZGb54ca5rb89XklzX3p7hixfnrDPPzJ6DBmXmzJnb7dErl1QHAAA6raqqfHzq1Hx+1qycneSCJEP/ZJszqipLkly1enXOP//8LF2yJNdce21qtVrXD1xHogoAAOi0q6++Op+fNSs3JDnjDbYblmRWkv2TnDVrVoYOG5bzzjuvS2bsKr6nCgAA6JS2trYMGTgwJ69Zk1mdWHd2kpv69MkvV6zILrvsUq/xOvieKgAAYKs0b968rF67Nhd0ct0FSVatWZN58+bVY6xuI6oAAIBOmXP99fmbvPYzVG9mWJLjfr9+eyKqAACATdbe3p5HH38879nMTxGNr6o8unjxdnUlQFEFAABssrVr16a9vT2b+wmlfknWt7dnzZo1W3KsbiWqAACATdbY2JgeDQ1p3cz1bUl6NDSksbFxS47VrUQVAACwyWq1WkYccEC+t5nfNXVnrZaDDzxwu/quKlEFAAB0yulTpuQ7SZZ2ct2SJP/6+/XbE1EFAAB0yqRJk9Knd+98rpPrPpekb2NjJk2aVI+xuo2oAgAAOqVfv3759OWX57okN27imhuSXJ/kkssu65Iv/u1KogoAAOi0adOm5dxzzsmZSc7Oq6f2bcyS3//+rCTnnnNOpk2b1lUjdpme3T0AAACw7anVarnm2mszdNiwXDp9em5YsybH5dXvoeqXV6/yd2etln/Nq6f8XXXZZZk2bdp2dYGKP6hV29O3biVpbW1Nc3NzWlpa0tS0uVfPBwAANlVbW1vmzZuXL8yenUcXL8769vb0aGjIwQcemNOnTMkJJ5yQfv36dflcXdUGogoAANhiqqrKmjVr0tjY2O1HpbqqDZz+BwAAbDG1Wi19+vTp7jG6lAtVAAAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAW6JKpmz56dvffeO717984RRxyRBx544HW3nTt3bmq12ga33r17d8WYAAAAnVb3qPr617+eadOm5ZJLLsnDDz+ckSNHZvz48XnhhRded01TU1OWLVvWcfvFL35R7zEBAAA2S92j6uqrr86pp56ak08+OQceeGDmzJmTPn365Etf+tLrrqnVahk0aFDHbeDAgfUeEwAAYLPUNapefvnlPPTQQxk3btx/P2FDQ8aNG5eFCxe+7rqVK1fmrW99a4YOHZr3v//9eeyxx15323Xr1qW1tXWDGwAAQFepa1T9+te/zvr1619zpGngwIFZvnz5Rtfst99++dKXvpRvf/vb+cpXvpL29vYceeSRee655za6/YwZM9Lc3NxxGzp06BZ/HQAAAK9nq7v635gxY/LRj340o0aNylFHHZVvfetbGTBgQL7whS9sdPuLL744LS0tHbelS5d28cQAAMCOrGc9H3z33XdPjx49smLFig3uX7FiRQYNGrRJj7HTTjtl9OjReeqppzb6+169eqVXr17FswIAAGyOuh6p2nnnnfPOd74z8+fP77ivvb098+fPz5gxYzbpMdavX59HH300gwcPrteYAAAAm62uR6qSZNq0aTnppJNy6KGH5vDDD8+1116bVatW5eSTT06SfPSjH82ee+6ZGTNmJEkuu+yy/Pmf/3n23XffvPTSS/nc5z6XX/ziF/nYxz5W71EBAAA6re5Rdfzxx+dXv/pVpk+fnuXLl2fUqFG54447Oi5esWTJkjQ0/PcBs9/+9rc59dRTs3z58uy666555zvfmR/+8Ic58MAD6z0qAABAp9Wqqqq6e4gtqbW1Nc3NzWlpaUlTU1N3jwMAAHSTrmqDre7qfwAAANsSUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABTokqiaPXt29t577/Tu3TtHHHFEHnjggTfc/hvf+Eb233//9O7dOwcffHD+7d/+rSvGBAAA6LS6R9XXv/71TJs2LZdcckkefvjhjBw5MuPHj88LL7yw0e1/+MMf5iMf+UhOOeWU/OQnP8nEiRMzceLE/OxnP6v3qAAAAJ1Wq6qqqucTHHHEETnssMNy/fXXJ0na29szdOjQnH322bnoootes/3xxx+fVatW5fbbb++478///M8zatSozJkz5zXbr1u3LuvWrev4ubW1NUOHDk1LS0uamprq8IoAAIBtQWtra5qbm+veBnU9UvXyyy/noYceyrhx4/77CRsaMm7cuCxcuHCjaxYuXLjB9kkyfvz4191+xowZaW5u7rgNHTp0y70AAACAN1HXqPr1r3+d9evXZ+DAgRvcP3DgwCxfvnyja5YvX96p7S+++OK0tLR03JYuXbplhgcAANgEPbt7gFK9evVKr169unsMAABgB1XXI1W77757evTokRUrVmxw/4oVKzJo0KCNrhk0aFCntgcAAOhOdY2qnXfeOe985zszf/78jvva29szf/78jBkzZqNrxowZs8H2SXLXXXe97vYAAADdqe6n/02bNi0nnXRSDj300Bx++OG59tprs2rVqpx88slJko9+9KPZc889M2PGjCTJueeem6OOOiozZ87M+973vvzLv/xLfvzjH+f//J//U+9RAQAAOq3uUXX88cfnV7/6VaZPn57ly5dn1KhRueOOOzouRrFkyZI0NPz3AbMjjzwyX/3qV/P3f//3+eQnP5m3v/3tue222zJixIh6jwoAANBpdf+eqq7WVdeiBwAAtm7bxfdUAQAAbO9EFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVdKm5c+emf//+W83jAABAKVHFZpk8eXJqtVrH7S1veUsmTJiQRx555A3XHX/88fnP//zPjp8//elPZ9SoUZ1+/j99HAAA6C6iis02YcKELFu2LMuWLcv8+fPTs2fP/PVf//Xrbv/KK6+ksbExe+yxR/Fzb6nHAQCAUqKKzdarV68MGjQogwYNyqhRo3LRRRdl6dKl+dWvfpVnn302tVotX//613PUUUeld+/emTdv3gan7c2dOzeXXnppfvrTn3Yc8Zo7d26S5Oqrr87BBx+cvn37ZujQoTnzzDOzcuXKjuf+09P//nDE65//+Z+z9957p7m5OR/+8IfT1tbWhf9GAADYEYkqtoiVK1fmK1/5Svbdd9+85S1v6bj/oosuyrnnnpvHH38848eP32DN8ccfn/POOy8HHXRQxxGv448/PknS0NCQWbNm5bHHHstNN92U73//+7nwwgvfcIann346t912W26//fbcfvvtuffee/MP//APW/7FAgDAH+nZ3QOw7br99tuzyy67JElWrVqVwYMH5/bbb09Dw3+3+tSpU/OBD3xgo+sbGxuzyy67pGfPnhk0aNAGv5s6dWrHP++99975zGc+k9NPPz033HDD687T3t6euXPnpl+/fkmSE088MfPnz89nP/vZzX2JAADwphypYrMdffTRWbRoURYtWpQHHngg48ePz7HHHptf/OIXHdsceuihm/XYd999d4455pjsueee6devX0488cS8+OKLWb169euu2XvvvTuCKkkGDx6cF154YbOeHwAANpWoYrP17ds3++67b/bdd98cdthh+eIXv5hVq1blH//xHzfYprOeffbZ/PVf/3UOOeSQfPOb38xDDz2U2bNnJ0lefvnl11230047bfBzrVZLe3t7p58fAAA6w+l/bDG1Wi0NDQ1Zs2bNJq/Zeeeds379+g3ue+ihh9Le3p6ZM2d2nEp4yy23bNFZAQBgSxFVbLZ169Zl+fLlSZLf/va3uf7667Ny5cocd9xxm/wYe++9d5555pksWrQoe+21V/r165d99903r7zySq677rocd9xxue+++zJnzpx6vQwAACji9D822x133JHBgwdn8ODBOeKII/Lggw/mG9/4RsaOHbvJj/HBD34wEyZMyNFHH50BAwbka1/7WkaOHJmrr746V1xxRUaMGJF58+ZlxowZ9XshAABQoFZVVdXdQ2xJra2taW5uTktLS5qamrp7HAAAoJt0VRs4UgUAAFBAVJGqqrJ69epsZwctAQCgS4iqrczYsWM3+OLbP5g7d2769++fJPn0pz+dUaNGbfD7f//3f0///v0zderUTYqjtra23HjjjRk1YkR26tkzffv2zU49e2bUiBGZM2dO2tratsCrAQCA7Z+o2g5897vfzfjx4zNt2rRce+21qdVqr7ttVVWZOXNmhgwcmClnnZXhixfnuvb2fCXJde3tGb54cc4688zsOWhQZs6c6egVAAC8CZdU38Z99atfzcknn5yZM2dmypQpb7htVVX5+NSp+fysWTk7yQVJhv7JNmdUVZYkuWr16px//vlZumRJrnmTUAMAgB2ZqNqGzZ49O9OmTcuXvvSlTJo06U23v/rqq/P5WbNyQ5Iz3mC7YUlmJdk/yVmzZmXosGE577zztszQAACwnXH63zbq8ccfz5QpU3LjjTduUlC1tbXl05/6VM7OGwfVHzszyZQkl06fnpUrVxZMCwAA2y9RtY3aa6+98o53vCOf+9znsmzZsjfdft68eVm9dm0u6OTzXJBk1Zo1mTdv3mbNCQAA2ztRtZVpampKS0vLa+5/6aWX0tzc3PFzv379cvfdd6dv3745+uij3zSs5lx/ff4mr/0M1ZsZluS4368HAABeS1RtZfbbb788/PDDr7n/4Ycfzp/92Z9tcN+uu+6au+++O01NTRk7dmx++ctfbvQx29vb8+jjj+c9m3klv/FVlUcXL3YlQAAA2AhRtZU544wz8p//+Z8555xz8sgjj+TJJ5/M1Vdfna997WsbvVhE//79c9ddd2XXXXd93bBau3Zt2tvb07SZM/VLsr69PWvWrNnMRwAAgO2XqNrK7LPPPvnBD36QJ554IuPGjcsRRxyRW265Jd/4xjcyYcKEja5pbm7O9773vey+++456qij8vzzz2/w+8bGxvRoaEjrZs7UlqRHQ0MaGxs38xEAAGD7Vau2s3O6Wltb09zcnJaWljQ1be6xme3PqBEjMnzx4ty6Gbt7Yq2WXxx0UH7y6KN1mAwAAOqjq9rAkaodxOlTpuQ7SZZ2ct2SJP/6+/UAAMBriaodxKRJk9Knd+98rpPrPpekb2PjJn0XFgAA7IhEVZ1VVZXVq1d3+5Xz+vXrl09ffnmuS3LjJq65Icn1SS657LLssssu9RsOAAC2YaKqDtra2nLjjTdm1IgR2alnz/Tt2zc79eyZUSNGZM6cOWlra+uWuaZNm5ZzzzknZyY5O6+e2rcxS37/+7OSnHvOOZk2bVpXjQgAANscF6rYgqqqytVXX51Pf+pTWb12bf4myXuqKk1JWpN8r1bLd/Lq6XSXXHZZpk2bllqt1i0zXjp9elatWZPj8ur3UPXLq1f5u7NWy79284wAALAldFUbiKotpKqqfHzq1Hx+1qycneSCJEM3st2SJFcluS6vHgW65tpruyVa2traMm/evHxh9uw8unhx1re3p0dDQw4+8MCcPmVKTjjhhPTr16/L5wIAgC1FVG2m7oqqmTNn5vzzz88NSc7YhO1vyKun11111VUb/VLfrlRVVdasWZPGxkZHpQAA2G6Iqs3UHVHV1taWIQMH5uQ1azKrE+vOTnJTnz755YoVLgQBAABbmO+p2obMmzcvq9euzQWdXHdBklVr1mTevHn1GAsAAOgCdY2q3/zmN5k0aVKamprSv3//nHLKKVm5cuUbrhk7dmxqtdoGt9NPP72eYxabc/31+Zts/DNUb2RYkuN+vx4AANg21TWqJk2alMceeyx33XVXbr/99vzgBz/Iaaed9qbrTj311CxbtqzjduWVV9ZzzCLt7e159PHH857NPItyfFXl0cWLu/17rAAAgM3Ts14P/Pjjj+eOO+7Igw8+mEMPPTRJct111+W9731vrrrqqgwZMuR11/bp0yeDBg2q12hb1Nq1a9Pe3p7NPUOzX5L17e1Zs2ZN+vTpsyVHAwAAukDdjlQtXLgw/fv37wiqJBk3blwaGhpy//33v+HaefPmZffdd8+IESNy8cUXZ/Xq1a+77bp169La2rrBrSs1NjamR0NDNvdZ25L0aGhIY2PjlhwLAADoInU7UrV8+fLsscceGz5Zz57Zbbfdsnz58tddd8IJJ+Stb31rhgwZkkceeSSf+MQn8uSTT+Zb3/rWRrefMWNGLr300i06e2fUarWMOOCAfG/x4pyxGafw3Vmr5eADD3QpcwAA2EZ1+kjVRRdd9JoLSfzp7YknntjsgU477bSMHz8+Bx98cCZNmpSbb745t956a55++umNbn/xxRenpaWl47Z06dLNfu7NdfqUKflOks4+85Ik//r79QAAwLap00eqzjvvvEyePPkNt9lnn30yaNCgvPDCCxvc/7vf/S6/+c1vOvV5qSOOOCJJ8tRTT+Vtb3vba37fq1ev9OrVa5Mfrx4mTZqUC6ZNy+c6+T1Vn0vSt7ExkyZNqtdoAABAnXU6qgYMGJABAwa86XZjxozJSy+9lIceeijvfOc7kyTf//73097e3hFKm2LRokVJksGDB3d21C7Tr1+/fPryy3P++efngCRnbMKaG5Jcn+Sqyy7zxb8AALANq9uFKg444IBMmDAhp556ah544IHcd999mTJlSj784Q93XPnv+eefz/77758HHnggSfL000/n8ssvz0MPPZRnn3023/nOd/LRj34073rXu3LIIYfUa9QtYtq0aTn3nHNyZpKz8+qpfRuz5Pe/PyvJueeck2nTpnXViAAAQB3U7UIVyatX8ZsyZUqOOeaYNDQ05IMf/GBmzfrvE+ReeeWVPPnkkx1X99t5551z991359prr82qVasydOjQfPCDH8zf//3f13PMLaJWq+Waa6/N0GHDcun06blhzZocl1e/h6pfXr3K3521Wv41r57yd9Vll2XatGkuUAEAANu4WrWdfetsa2trmpub09LSkqamzf32qDJtbW2ZN29evjB7dh5dvDjr29vTo6EhBx94YE6fMiUnnHBC+vXr1y2zAQDAjqKr2kBU1VlVVVmzZk0aGxsdlQIAgC7UVW1Q19P/ePW0wD59+nT3GAAAQJ3U7UIVAAAAOwJRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAXqFlWf/exnc+SRR6ZPnz7p37//Jq2pqirTp0/P4MGD09jYmHHjxuXnP/95vUYEAAAoVreoevnll/OhD30oZ5xxxiavufLKKzNr1qzMmTMn999/f/r27Zvx48dn7dq19RoTAACgSK2qqqqeTzB37txMnTo1L7300htuV1VVhgwZkvPOOy/nn39+kqSlpSUDBw7M3Llz8+EPf3iTnq+1tTXNzc1paWlJU1NT6fgAAMA2qqvaYKv5TNUzzzyT5cuXZ9y4cR33NTc354gjjsjChQtfd926devS2tq6wQ0AAKCrbDVRtXz58iTJwIEDN7h/4MCBHb/bmBkzZqS5ubnjNnTo0LrOCQAA8Mc6FVUXXXRRarXaG96eeOKJes26URdffHFaWlo6bkuXLu3S5wcAAHZsPTuz8XnnnZfJkye/4Tb77LPPZg0yaNCgJMmKFSsyePDgjvtXrFiRUaNGve66Xr16pVevXpv1nAAAAKU6FVUDBgzIgAED6jLI8OHDM2jQoMyfP78jolpbW3P//fd36gqCAAAAXalun6lasmRJFi1alCVLlmT9+vVZtGhRFi1alJUrV3Zss//+++fWW29NktRqtUydOjWf+cxn8p3vfCePPvpoPvrRj2bIkCGZOHFivcYEAAAo0qkjVZ0xffr03HTTTR0/jx49Oklyzz33ZOzYsUmSJ598Mi0tLR3bXHjhhVm1alVOO+20vPTSS/nLv/zL3HHHHendu3e9xgQAAChS9++p6mq+pwoAAEh2wO+pAgAA2BaJKgAAgAKiCgAAoICoAgAAKCCqAAAACogqAACAAqIKAACggKgCAAAoIKoAAAAKiCoAAIACogoAAKCAqAIAACggqgAAAAqIKgAAgAKiCgAAoICoAgAAKCCqAAAACogqAACAAqIKAACggKgCAAAoIKoAAAAKiCoAAIACogoAAKCAqAIAACggqgAAAAqIKgAAgAKiCgAAoICoAgAAKCCqAAAACogqAACAAqIKAACggKgCAAAoIKoAAAAKiCoAAIACogoAAKCAqAIAACggqgAAAAqIKgAAgAKiCgAAoICoAgAAKCCqAAAACogqAACAAqIKAACggKgCAAAoIKoAAAAKiCoAAIACogoAAKCAqAIAACggqgAAAAqIKgAAgAKiCgAAoICoAgAAKCCqAAAACogqAACAAqIKAACggKgCAAAoIKoAAAAKiCoAAIACogoAAKCAqAIAACggqgAAAAqIKgAAgAKiCgAAoICoAgBgi5g8eXImTpzY3WNAlxNVAABsEERjx47N1KlTu3Ue2JaIKgAAgAKiCgCADpMnT869996bz3/+86nVaqnVann22Wezfv36nHLKKRk+fHgaGxuz33775fOf//zrPs7NN9+ct7zlLVm3bt0G90+cODEnnnhivV8GdClRBQBAh89//vMZM2ZMTj311CxbtizLli3L0KFD097enr322ivf+MY3snjx4kyfPj2f/OQnc8stt2z0cT70oQ9l/fr1+c53vtNx3wsvvJDvfve7+Z//83921cuBLtGzuwcAAGDr0dzcnJ133jl9+vTJoEGDOu7v0aNHLr300o6fhw8fnoULF+aWW27J3/7t377mcRobG3PCCSfky1/+cj70oQ8lSb7yla9k2LBhGTt2bN1fB3QlUQUAwCaZPXt2vvSlL2XJkiVZs2ZNXn755YwaNep1tz/11FNz2GGH5fnnn8+ee+6ZuXPnZvLkyanVal03NHQBp/8BAPCm/uVf/iXnn39+TjnllHzve9/LokWLcvLJJ+fll19+3TWjR4/OyJEjc/PNN+ehhx7KY489lsmTJ3fd0NBFHKkCAGADO++8c9avX7/Bfffdd1+OPPLInHnmmR33Pf3002/6WB/72Mdy7bXX5vnnn8+4ceMydOjQLT4vdDdHqgAA2MDee++d+++/P88++2x+/etfp729PW9/+9vz4x//OHfeeWf+8z//M5/61Kfy4IMPvuljnXDCCXnuuefyj//4jy5QwXZLVAEAsIHzzz8/PXr0yIEHHpgBAwZkyZIl+bu/+7t84AMfyPHHH58jjjgiL7744gZHrV5Pc3NzPvjBD2aXXXbp+HJh2N7UqqqqunuILam1tTXNzc1paWlJU1NTd48DALDDO+aYY3LQQQdl1qxZ3T0KO5iuagOfqQIAoC5++9vfZsGCBVmwYEFuuOGG7h4H6kZUAQBsZ6qqypo1a9LY2Nitly8fPXp0fvvb3+aKK67Ifvvt121zQL35TBUAwHagra0tN954Y0aNGJGdevZM3759s1PPnhk1YkTmzJmTtra2Lp/p2WefTUtLS84///wuf27oSqIKAGAbVlVVZs6cmSEDB2bKWWdl+OLFua69PV9Jcl17e4YvXpyzzjwzew4alJkzZ2Y7+zg9bBWc/gcAsI2qqiofnzo1n581K2cnuSDJn34L1BlVlSVJrlq9Oueff36WLlmSa669tltPC4TtjagCANhGXX311fn8rFm5IckZb7DdsCSzkuyf5KxZszJ02LCcd955XTIj7AhcUh0AYBvU1taWIQMH5uQ1a9KZC5WfneSmPn3yyxUrsssuu9RrPNgqdFUb+EwVAMA2aN68eVm9dm0u6OS6C5KsWrMm8+bNq8dYsEMSVQAA26A511+fv8lrP0P1ZoYlOe7364EtQ1QBAGxj2tvb8+jjj+c9m/kpjvFVlUcXL3YlQNhCRBUAwDZm7dq1aW9vz+Z+QqRfkvXt7VmzZs2WHAt2WKIKAGAb09jYmB4NDWndzPVtSXo0NKSxsXFLjgU7LFEFALCNqdVqGXHAAfneZn7X1J21Wg4+8EDfVQVbiKgCANgGnT5lSr6TZGkn1y1J8q+/Xw9sGaIKAGAbNGnSpPTp3Tuf6+S6zyXp29iYSZMm1WMs2CGJKgCAbVC/fv3y6csvz3VJbtzENTckuT7JJZdd5ot/YQsSVQAA26hp06bl3HPOyZlJzs6rp/ZtzJLf//6sJOeec06mTZvWVSPCDqFndw8AAMDmqdVquebaazN02LBcOn16blizJsfl1e+h6pdXr/J3Z62Wf82rp/xdddllmTZtmgtUwBZWq7azb31rbW1Nc3NzWlpa0tS0ud/eAACwbWlra8u8efPyhdmz8+jixVnf3p4eDQ05+MADc/qUKTnhhBPSr1+/7h4TulRXtYGoAgDYzlRVlTVr1qSxsdFRKXZoXdUGTv8DANjO1Gq19OnTp7vHgB2GC1UAAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABQQVQAAAAVEFQAAQAFRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAF6hZVn/3sZ3PkkUemT58+6d+//yatmTx5cmq12ga3CRMm1GtEAACAYj3r9cAvv/xyPvShD2XMmDH5p3/6p01eN2HChHz5y1/u+LlXr171GA8AAGCLqFtUXXrppUmSuXPndmpdr169MmjQoDpMBAAAsOVtdZ+pWrBgQfbYY4/st99+OeOMM/Liiy++4fbr1q1La2vrBjcAAICuslVF1YQJE3LzzTdn/vz5ueKKK3Lvvffm2GOPzfr16193zYwZM9Lc3NxxGzp0aBdODAAA7Og6FVUXXXTRay4k8ae3J554YrOH+fCHP5y/+Zu/ycEHH5yJEyfm9ttvz4MPPpgFCxa87pqLL744LS0tHbelS5du9vMDAAB0Vqc+U3Xeeedl8uTJb7jNPvvsUzLPax5r9913z1NPPZVjjjlmo9v06tXLxSwAAIBu06moGjBgQAYMGFCvWV7jueeey4svvpjBgwd32XMCAAB0Rt0+U7VkyZIsWrQoS5Ysyfr167No0aIsWrQoK1eu7Nhm//33z6233pokWblyZS644IL86Ec/yrPPPpv58+fn/e9/f/bdd9+MHz++XmMCAAAUqdsl1adPn56bbrqp4+fRo0cnSe65556MHTs2SfLkk0+mpaUlSdKjR4888sgjuemmm/LSSy9lyJAhec973pPLL7+8U6f3VVWVJK4CCAAAO7g/NMEfGqFealW9n6GLPffcc64ACAAAdFi6dGn22muvuj3+dhdV7e3tefLJJ3PggQdm6dKlaWpq6u6ReAOtra0ZOnSofbWVs5+2HfbVtsF+2nbYV9sO+2rb0NX7qaqqtLW1ZciQIWloqN+3SdXt9L/u0tDQkD333DNJ0tTU5E21jbCvtg3207bDvto22E/bDvtq22FfbRu6cj81NzfX/Tm2qi//BQAA2NaIKgAAgALbZVT16tUrl1xyiS8F3gbYV9sG+2nbYV9tG+ynbYd9te2wr7YN2+t+2u4uVAEAANCVtssjVQAAAF1FVAEAABQQVQAAAAVEFQAAQAFRBQAAUGC7iarPfvazOfLII9OnT5/0799/k9ZMnjw5tVptg9uECRPqO+gObnP2U1VVmT59egYPHpzGxsaMGzcuP//5z+s7KPnNb36TSZMmpampKf37988pp5ySlStXvuGasWPHvuY9dfrpp3fRxDuO2bNnZ++9907v3r1zxBFH5IEHHnjD7b/xjW9k//33T+/evXPwwQfn3/7t37po0h1bZ/bT3LlzX/Pe6d27dxdOu+P6wQ9+kOOOOy5DhgxJrVbLbbfd9qZrFixYkHe84x3p1atX9t1338ydO7fuc+7oOrufFixY8Jr3VK1Wy/Lly7tm4B3UjBkzcthhh6Vfv37ZY489MnHixDz55JNvum57+Du13UTVyy+/nA996EM544wzOrVuwoQJWbZsWcfta1/7Wp0mJNm8/XTllVdm1qxZmTNnTu6///707ds348ePz9q1a+s4KZMmTcpjjz2Wu+66K7fffnt+8IMf5LTTTnvTdaeeeuoG76krr7yyC6bdcXz961/PtGnTcskll+Thhx/OyJEjM378+Lzwwgsb3f6HP/xhPvKRj+SUU07JT37yk0ycODETJ07Mz372sy6efMfS2f2UJE1NTRu8d37xi1904cQ7rlWrVmXkyJGZPXv2Jm3/zDPP5H3ve1+OPvroLFq0KFOnTs3HPvax3HnnnXWedMfW2f30B08++eQG76s99tijThOSJPfee2/OOuus/OhHP8pdd92VV155Je95z3uyatWq112z3fydqrYzX/7yl6vm5uZN2vakk06q3v/+99d1HjZuU/dTe3t7NWjQoOpzn/tcx30vvfRS1atXr+prX/taHSfcsS1evLhKUj344IMd9/2///f/qlqtVj3//POvu+6oo46qzj333C6YcMd1+OGHV2eddVbHz+vXr6+GDBlSzZgxY6Pb/+3f/m31vve9b4P7jjjiiOrv/u7v6jrnjq6z+6kzf7uonyTVrbfe+obbXHjhhdVBBx20wX3HH398NX78+DpOxh/blP10zz33VEmq3/72t10yExv3wgsvVEmqe++993W32V7+Tm03R6o214IFC7LHHntkv/32yxlnnJEXX3yxu0fijzzzzDNZvnx5xo0b13Ffc3NzjjjiiCxcuLAbJ9u+LVy4MP3798+hhx7acd+4cePS0NCQ+++//w3Xzps3L7vvvntGjBiRiy++OKtXr673uDuMl19+OQ899NAG74eGhoaMGzfudd8PCxcu3GD7JBk/frz3Tx1tzn5KkpUrV+atb31rhg4dmve///157LHHumJcOsl7atsyatSoDB48OO9+97tz3333dfc4O5yWlpYkyW677fa622wv76me3T1Ad5owYUI+8IEPZPjw4Xn66afzyU9+Mscee2wWLlyYHj16dPd4JB3nPg8cOHCD+wcOHOi86Dpavnz5a06R6NmzZ3bbbbc3/Pd+wgkn5K1vfWuGDBmSRx55JJ/4xCfy5JNP5lvf+la9R94h/PrXv8769es3+n544oknNrpm+fLl3j9dbHP203777ZcvfelLOeSQQ9LS0pKrrroqRx55ZB577LHstddeXTE2m+j13lOtra1Zs2ZNGhsbu2ky/tjgwYMzZ86cHHrooVm3bl2++MUvZuzYsbn//vvzjne8o7vH2yG0t7dn6tSp+Yu/+IuMGDHidbfbXv5ObdVRddFFF+WKK654w20ef/zx7L///pv1+B/+8Ic7/vnggw/OIYcckre97W1ZsGBBjjnmmM16zB1RvfcTW86m7qvN9cefuTr44IMzePDgHHPMMXn66afztre9bbMfF7Z3Y8aMyZgxYzp+PvLII3PAAQfkC1/4Qi6//PJunAy2Tfvtt1/222+/jp+PPPLIPP3007nmmmvyz//8z9042Y7jrLPOys9+9rP8x3/8R3eP0iW26qg677zzMnny5DfcZp999tliz7fPPvtk9913z1NPPSWqOqGe+2nQoEFJkhUrVmTw4MEd969YsSKjRo3arMfckW3qvho0aNBrPlD/u9/9Lr/5zW869smmOOKII5IkTz31lKjaAnbffff06NEjK1as2OD+FStWvO5+GTRoUKe2p9zm7Kc/tdNOO2X06NF56qmn6jEiBV7vPdXU1OQo1Vbu8MMP32H+A7+7TZkypeMiV292tH17+Tu1VUfVgAEDMmDAgC57vueeey4vvvjiBv/xzpur534aPnx4Bg0alPnz53dEVGtra+6///5OX+mRTd9XY8aMyUsvvZSHHnoo73znO5Mk3//+99Pe3t4RSpti0aJFSeI9tYXsvPPOeec735n58+dn4sSJSV49vWL+/PmZMmXKRteMGTMm8+fPz9SpUzvuu+uuuzY4KsKWtTn76U+tX78+jz76aN773vfWcVI2x5gxY15zuWfvqW3DokWL/D2qs6qqcvbZZ+fWW2/NggULMnz48Ddds938neruK2VsKb/4xS+qn/zkJ9Wll15a7bLLLtVPfvKT6ic/+UnV1tbWsc1+++1Xfetb36qqqqra2tqq888/v1q4cGH1zDPPVHfffXf1jne8o3r7299erV27trtexnavs/upqqrqH/7hH6r+/ftX3/72t6tHHnmkev/7318NHz68WrNmTXe8hB3GhAkTqtGjR1f3339/9R//8R/V29/+9uojH/lIx++fe+65ar/99qvuv//+qqqq6qmnnqouu+yy6sc//nH1zDPPVN/+9rerffbZp3rXu97VXS9hu/Qv//IvVa9evaq5c+dWixcvrk477bSqf//+1fLly6uqqqoTTzyxuuiiizq2v++++6qePXtWV111VfX4449Xl1xySbXTTjtVjz76aHe9hB1CZ/fTpZdeWt15553V008/XT300EPVhz/84ap3797VY4891l0vYYfR1tbW8bcoSXX11VdXP/nJT6pf/OIXVVVV1UUXXVSdeOKJHdv/13/9V9WnT5/qggsuqB5//PFq9uzZVY8ePao77riju17CDqGz++maa66pbrvtturnP/959eijj1bnnntu1dDQUN19993d9RJ2CGeccUbV3NxcLViwoFq2bFnHbfXq1R3bbK9/p7abqDrppJOqJK+53XPPPR3bJKm+/OUvV1VVVatXr67e8573VAMGDKh22mmn6q1vfWt16qmndvzBoz46u5+q6tXLqn/qU5+qBg4cWPXq1as65phjqieffLLrh9/BvPjii9VHPvKRapdddqmampqqk08+eYP4feaZZzbYd0uWLKne9a53VbvttlvVq1evat99960uuOCCqqWlpZtewfbruuuuq4YNG1btvPPO1eGHH1796Ec/6vjdUUcdVZ100kkbbH/LLbdUf/Znf1btvPPO1UEHHVR997vf7eKJd0yd2U9Tp07t2HbgwIHVe9/73urhhx/uhql3PH+49Paf3v6wf0466aTqqKOOes2aUaNGVTvvvHO1zz77bPA3i/ro7H664oorqre97W1V7969q912260aO3Zs9f3vf797ht+BbGwf/el/122vf6dqVVVVXXBADAAAYLu0w39PFQAAQAlRBQAAUEBUAQAAFBBVAAAABUQVAABAAVEFAABQQFQBAAAUEFUAAAAFRBUAAEABUQUAAFBAVAEAABT4/60UEE95GcDVAAAAAElFTkSuQmCC",
            "text/plain": [
              "<Figure size 1000x1000 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "display_pca_scatterplot(word_vectors, ['Rome', 'Italy', 'London', 'Britain', 'UK'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iuDWv6NXXjoF"
      },
      "source": [
        "# Using Pretrained Word Vectors for Classification"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Am-9bvppYKuS"
      },
      "source": [
        "In this section, we'll train a **Keras** model to use these Google News vectors to perform sentiment analysis on a bunch of **Yelp** reviews.\n",
        "<br><br>\n",
        "For this model, we'll increase the number of word vectors loaded to 1,000,000.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "id": "VVLj4fSMzvHJ"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CPU times: total: 17 s\n",
            "Wall time: 27.1 s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "word_vectors = KeyedVectors.load_word2vec_format(embedding_file, binary=True, limit=1000000)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vdZQNavVYxfP"
      },
      "source": [
        "The dataset we'll use is *Yelp Polarity Reviews*, a collection of ~600,000 reviews for both training and testing.<br><br>\n",
        "The original Yelp reviews use a five-star rating system. The ratings in this dataset have been modified to simply be negative (label==1) or positive (label==2).<br>\n",
        "https://www.tensorflow.org/datasets/catalog/yelp_polarity_reviews<br><br>\n",
        "Tensorflow comes with a datasets loader but we're going to download the file manually and process the data ourselves for completeness."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "id": "RcMEE11bzz_j"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "'wget' is not recognized as an internal or external command,\n",
            "operable program or batch file.\n"
          ]
        }
      ],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wnTnApF4aq7k"
      },
      "source": [
        "Unzipping the archive results in *train.csv* and *test.csv* files placed in the default *contents* folder of our environment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "id": "-UK1y9kjzz7y"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "x yelp_review_polarity_csv/\n",
            "x yelp_review_polarity_csv/train.csv\n",
            "x yelp_review_polarity_csv/readme.txt\n",
            "x yelp_review_polarity_csv/test.csv\n",
            "'pwd' is not recognized as an internal or external command,\n",
            "operable program or batch file.\n"
          ]
        }
      ],
      "source": [
        "!tar xvzf \"./yelp_review_polarity_csv.tgz\"\n",
        "\n",
        "# Show current working directory.\n",
        "!pwd"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OaJCSM3lbK-E"
      },
      "source": [
        "The **Pandas** library makes it simple to load a CSV file into memory and manipulate the data.<br>\n",
        "https://pandas.pydata.org/<br>\n",
        "https://pandas.pydata.org/docs/<br>\n",
        "https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html?highlight=read_csv"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xIG4UVMkbq9E"
      },
      "source": [
        "Here, we're loading the CSV into a Pandas **dataframe** (sort of like an in-memory table) and explicitly naming the columns.<br>\n",
        "https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html?highlight=dataframe#pandas.DataFrame"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "id": "tdLoK1Jozz0Z"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "(560000, 2)\n"
          ]
        }
      ],
      "source": [
        "yelp_train = pd.read_csv('yelp_review_polarity_csv/train.csv', names=['sentiment', 'review'])\n",
        "print(yelp_train.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lGgJl1QCb738"
      },
      "source": [
        "We can get a quick view of the data through the *head* method.<br>\n",
        "https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.head.html"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "id": "jVHtVqOBzzwS"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentiment</th>\n",
              "      <th>review</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>Unfortunately, the frustration of being Dr. Go...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>Been going to Dr. Goldberg for over 10 years. ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1</td>\n",
              "      <td>I don't know what Dr. Goldberg was like before...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>I'm writing this review to give you a heads up...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2</td>\n",
              "      <td>All the food is great here. But the best thing...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   sentiment                                             review\n",
              "0          1  Unfortunately, the frustration of being Dr. Go...\n",
              "1          2  Been going to Dr. Goldberg for over 10 years. ...\n",
              "2          1  I don't know what Dr. Goldberg was like before...\n",
              "3          1  I'm writing this review to give you a heads up...\n",
              "4          2  All the food is great here. But the best thing..."
            ]
          },
          "execution_count": 34,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "yelp_train.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9wS0o4gEclCT"
      },
      "source": [
        "To save on training time, we'll train on 100,000 reviews rather than the full set. To do that, we'll shuffle the dataset using the *sample* method and *copy* the first 100,000 entries. The reason to shuffle first is to ensure we get a mix of reviews from a variety of businesses (in case the data is sorted in some way).<br>\n",
        "https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.sample.html<br>\n",
        "https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.copy.html\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "id": "YacSH4F0c5aU"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "(100000, 2)\n"
          ]
        }
      ],
      "source": [
        "TRAIN_SIZE = 100000\n",
        "yelp_train = yelp_train.sample(frac=1, random_state=1)[:TRAIN_SIZE].copy()\n",
        "print(yelp_train.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DLraxnTMd9iE"
      },
      "source": [
        "The next thing to do is adjust the labels. This is a **binary classification problem**, so our model's output layer will be a single unit with a **sigmoid** activation function. This function's output will be between 0 and 1 which is then compared against the training label. But the labels are currently 1 for negative, and 2 for positive, which is going to cause problems when calculating the loss.<br><br>\n",
        "So we'll simply replace the 1s with 0s, and 2s with 1s using the *replace* method.<br>\n",
        "https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.replace.html\n",
        "<br><br>\n",
        "Alternatively, we could keep the labels as-is and treat this as a **multiclassification** problem with two labels and use a **softmax**, but we would then need to **one-hot encode** the labels (at least based on what we've learnt so far).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {
        "id": "kbmYKm-SjGnm"
      },
      "outputs": [],
      "source": [
        "yelp_train['sentiment'].replace(to_replace=1, value=0, inplace=True)\n",
        "yelp_train['sentiment'].replace(to_replace=2, value=1, inplace=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "metadata": {
        "id": "LlzWLECijrg_"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentiment</th>\n",
              "      <th>review</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>39658</th>\n",
              "      <td>1</td>\n",
              "      <td>omg, the grandma g's pizza with rosemary chick...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>330033</th>\n",
              "      <td>0</td>\n",
              "      <td>Microwaved food over over over seasoned gyro m...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>238898</th>\n",
              "      <td>0</td>\n",
              "      <td>While the food is good, the service leaves a l...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>453536</th>\n",
              "      <td>0</td>\n",
              "      <td>Called the Chandler PD about the sign spinner ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>555973</th>\n",
              "      <td>1</td>\n",
              "      <td>Just took Ewok, my approx. 6 month old shih tz...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        sentiment                                             review\n",
              "39658           1  omg, the grandma g's pizza with rosemary chick...\n",
              "330033          0  Microwaved food over over over seasoned gyro m...\n",
              "238898          0  While the food is good, the service leaves a l...\n",
              "453536          0  Called the Chandler PD about the sign spinner ...\n",
              "555973          1  Just took Ewok, my approx. 6 month old shih tz..."
            ]
          },
          "execution_count": 37,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "yelp_train.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 56,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "\"omg, the grandma g's pizza with rosemary chicken and potatoes was soooo good! I came here with my parents and we also shared a mediterranean salad, which was also delicious. Tried the Four Peaks Kilt Lifter on draft and it was great. Seems hard to go wrong here. There was a huge line, but totally worth the wait.\\\\n\\\\nOh yeah, I got carded w/ my parents and I'm turning 31 in four months. lol! Apparently I look *way* young. YESSSSSS ;P\""
            ]
          },
          "execution_count": 56,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "yelp_train.iloc[0]['review']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rILnYdQMgIlh"
      },
      "source": [
        "As we've done throughout this course, we'll create train/validation splits.<br>\n",
        "https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {
        "id": "CC-VSNcn7Cpq"
      },
      "outputs": [],
      "source": [
        "yelp_train_split, yelp_val_split = train_test_split(yelp_train, train_size=0.85, random_state=1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 39,
      "metadata": {
        "id": "9Xj4WyLb7Wdr"
      },
      "outputs": [],
      "source": [
        "# Set up training data.\n",
        "train_reviews = yelp_train_split['review']\n",
        "y_train = np.array(yelp_train_split['sentiment'])\n",
        "\n",
        "# Set up validation data.\n",
        "val_reviews = yelp_val_split['review']\n",
        "y_val = np.array(yelp_val_split['sentiment'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CK9V0dJAgbSJ"
      },
      "source": [
        "A quick sanity check to see how our data is distributed (e.g. balanced or skewed)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {
        "id": "MHBWB2vc7ie1"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Counter({0: 42395, 1: 42605})"
            ]
          },
          "execution_count": 40,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "collections.Counter(y_train)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M0fM5vRygqfq"
      },
      "source": [
        "Because we're relying more on richer encodings (in this case, word vectors), we won't perform as much preprocessing this time around. We'll stick with using the regular Keras **tokenizer** and just filter out numbers and certain symbols.<br>\n",
        "https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text/Tokenizer<br><br>\n",
        "We'll also have the tokenizer limit itself to tokenizing only the most frequent 20,000 words. This way, the model will focus on the most frequent descriptive sentiment words."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 41,
      "metadata": {
        "id": "UI4FgHPa4p2T"
      },
      "outputs": [],
      "source": [
        "tokenizer = keras.preprocessing.text.Tokenizer(num_words=20000,\n",
        "                                               filters='0123456789!\"#$%&()*+,-./:;<=>?@[\\]^_`{|}~',\n",
        "                                               lower=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gXQXX82chivF"
      },
      "source": [
        "Build the vocabulary."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 42,
      "metadata": {
        "id": "ayGLszCqzzoK"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CPU times: total: 9.5 s\n",
            "Wall time: 12.2 s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "tokenizer.fit_on_texts(train_reviews)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v5zDD6RCBzxH"
      },
      "source": [
        "The next step is to vectorize our reviews. In the [_Neural Network Foundations_](https://github.com/futuremojo/nlp-demystified/blob/main/notebooks/nlpdemystified_neural_networks_foundations.ipynb) notebook, we used the *texts_to_matrix* method to turn text into binary bags of words.<br><br>\n",
        "Here, we're going to use the *text_to_sequences* method to turn each review into a sequence of integers, with each integer representing its corresponding token.<br>\n",
        "https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text/Tokenizer#texts_to_sequences\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 43,
      "metadata": {
        "id": "OAd431jp6Ydg"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CPU times: total: 6.88 s\n",
            "Wall time: 8.89 s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "X_train = tokenizer.texts_to_sequences(train_reviews)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 44,
      "metadata": {
        "id": "PPo0mNRB6YZL"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[49, 14, 2224, 4199, 1981, 11, 2312, 16, 6, 59, 3042, 138, 5, 145, 1, 294, 1, 378, 370, 7, 8, 37, 6, 1, 120, 944, 16362, 1026, 33, 3014, 21, 1170, 12144, 3036, 1427, 5513, 56, 3214, 2, 106, 5514, 136, 886, 1, 2224, 268, 3, 90, 22, 1278, 944, 16362, 51, 183, 36, 60, 107, 3214, 9, 14, 795, 26, 351, 3, 148, 5, 150, 107, 65, 12, 47, 74, 1966, 1935, 42, 7, 1, 132, 5352, 24, 1918, 36, 5481, 21, 1, 11637, 1378, 1116, 6722, 13, 64, 262, 19603, 2, 312, 4, 1253, 3278, 5, 14, 362, 116, 6, 10, 1, 128, 271, 137, 2100, 7, 16, 14, 362, 2, 3, 148, 4, 156, 1694, 1047, 17, 147, 86, 323, 598, 1, 1103, 207, 1055, 51, 544, 783, 21, 1, 3872, 15, 26, 147, 17728, 2, 27, 148, 39, 15, 102, 993, 18, 1, 1364, 55, 1, 6722, 5481, 2, 60, 39, 249, 102, 30, 282, 8, 161, 4, 43, 189, 46, 9, 944, 16362, 5, 1338, 65, 10, 674, 3, 102, 598, 2, 47, 64, 90, 279, 48, 64, 86, 124, 174, 1, 128, 18, 4, 545, 303, 1072, 13078, 12751, 17, 18, 1, 766, 13, 14, 362, 2, 3, 24, 282, 172, 360, 10, 815, 9, 107, 3, 78, 113, 39, 1, 5352, 24, 68, 4628, 51, 2968, 13, 15, 86, 598, 1, 1103, 2, 99, 79, 944, 16362, 888, 48, 288, 1, 2348, 10, 65, 19, 5274, 41, 1317, 5, 150, 1, 1364, 17, 47, 944, 16362, 299, 90, 1338, 15, 26, 137, 5, 240, 38, 1453, 116, 24, 282, 172, 360, 2, 26, 165, 10, 1966, 9, 8, 32, 6, 57, 889, 8276, 57, 12145, 35, 5, 118, 3, 6, 792, 22, 73, 4, 1318, 40, 4, 18584, 64, 148, 238, 38, 20, 1559, 59, 9752, 6722, 12, 13, 1388, 2, 47, 15, 215, 5, 1243, 64, 6032, 6723, 107, 2636, 2, 6842, 12, 13, 1388, 15, 975, 13, 10, 489, 57, 8, 6, 22, 1388, 42, 7, 1, 5352, 24, 628, 5, 561, 1, 6722, 9, 14, 104, 5, 150, 1512, 13, 24, 134, 4343, 21, 1, 6722, 2, 13, 12, 60, 15, 24, 4273, 15, 26, 565, 2, 1290, 5, 2193, 5, 59, 9752, 6722, 39, 1, 448, 18585, 944, 16362, 99, 14, 362, 9, 73, 222, 25, 1, 6722, 3344, 10, 7655, 7, 1, 489, 13, 8, 6, 42, 7, 1, 5352, 116, 2663, 399, 175, 845, 5, 8, 944, 16362, 137, 8, 1096, 5, 67, 13, 1, 9595, 28, 22, 9, 20, 1, 1737, 5, 655, 20, 78, 113, 118, 5, 655, 77, 8, 817, 266, 7, 1462, 5, 1916, 118, 5, 655, 250, 9595, 1462, 20, 78, 23, 2, 73, 39, 20, 26, 2956, 19, 53, 147, 30, 1248, 5, 655, 3, 547, 718, 7, 250, 515, 24, 9595, 544, 2, 271, 15, 79, 533, 13, 15755, 944, 16362, 137, 8, 540, 35, 194, 15, 24, 15756, 1, 5352, 65, 25, 13, 401, 15, 1266, 8080, 67, 13, 1, 5352, 28, 94, 2956, 6407, 2, 113, 103, 74, 55, 250, 238, 100, 20, 82, 2, 64, 106, 37, 16, 10, 270, 7, 77, 10, 1, 209, 450, 2901, 15, 975, 13, 13, 6, 91, 4, 971, 10, 1490, 13, 15, 91, 15756, 56, 37, 15, 26, 1254, 9, 6, 69, 14728, 10, 4, 1103, 399, 5, 41, 845, 75, 1, 5352, 2, 137, 36, 5, 30, 42, 9, 10065, 2833, 35, 4, 1737, 3, 141, 3, 23, 1, 144, 5, 269, 1047, 2, 1317, 5, 150, 4, 1364, 3, 24, 1, 1875, 944, 16362, 6, 21, 1, 7085, 1, 579, 46, 2, 60, 137, 238, 74, 9, 33, 6, 1, 489, 13, 64, 260, 5, 1711, 8, 37, 48, 18, 4, 302, 7, 11886, 271, 64, 918, 820, 1152, 67, 108, 944, 16362, 260, 5, 361, 107, 281, 18, 60, 407, 38, 4, 12752, 51, 7, 14, 5784, 5, 150, 107, 35, 64, 6, 286, 565, 5, 383, 36, 7, 1, 128, 944, 16362, 24, 4, 1283, 9061, 18, 2855, 815, 191, 3, 145, 8, 280, 3, 1513, 145, 8, 363, 40, 145, 8, 72, 235, 1, 302, 7, 352, 3, 117, 99, 64, 264, 33, 1151, 10, 1, 869, 2, 106, 233, 115, 5, 23, 4, 1037, 957, 2, 356, 36, 184, 267, 13, 24, 576, 21, 13, 132, 73, 47, 944, 16362, 102, 22, 141, 7, 1, 238, 64, 6, 5990, 5, 2, 24, 5, 30, 1743, 7, 75, 1, 5352, 184, 267, 64, 24, 106, 28, 238, 3, 278, 102, 23, 2902, 17, 22, 13, 22, 10, 4, 31, 3, 24, 68, 2489, 3, 102, 494, 8000, 49, 3, 24, 91, 538, 6781, 136, 2, 3, 6, 22, 828, 9, 1, 13837, 7, 3942, 3, 6, 142, 49, 1, 46, 3, 24, 419, 25, 1, 2224, 5, 121, 2, 2019, 5, 429, 238, 694, 17, 118, 102, 477, 2019, 80, 169, 1026, 38, 13, 1, 7, 3442, 28, 1248, 227, 10, 1427, 17, 80, 376, 59, 5112, 18, 944, 16362, 3, 6, 531, 5381, 36, 55, 169, 4867, 5, 10, 376, 4, 120, 2, 37, 3, 102, 99, 141, 55, 6, 1, 15231, 7, 3942, 64, 161, 226, 14, 803, 9, 4, 893, 2, 3308, 33, 27, 103, 15232, 13, 3, 531, 48, 9, 1, 615, 7, 60, 53, 147, 30, 4, 43, 189, 3442, 107, 1882, 7, 33, 6, 10, 69, 2132, 74, 18586, 100, 1, 3942, 674, 53, 442, 244, 39, 1055]\n"
          ]
        }
      ],
      "source": [
        "# The first review in the training set, vectorized.\n",
        "print(X_train[0])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 52,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "910\n",
            "25\n",
            "80\n"
          ]
        }
      ],
      "source": [
        "print(len(X_train[0]))\n",
        "print(len(X_train[1]))\n",
        "print(len(X_train[2]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kH4I4SeoqVDn"
      },
      "source": [
        "We can look up the corresponding tokens using the tokenizer's *index_word* dict. Here are the tokens corresponding to the first three integers from the first vectorized review."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 57,
      "metadata": {
        "id": "jkOI7jYFqb44"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['from', 'my', 'hospital']"
            ]
          },
          "execution_count": 57,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "[tokenizer.index_word[x] for x in X_train[0][:3]]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ahgfyZhDrHDJ"
      },
      "source": [
        "We can also convert the integer sequence back to text using the *sequences_to_texts* method, and compare it against the original text.<br>\n",
        "https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text/Tokenizer#sequences_to_texts"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 58,
      "metadata": {
        "id": "RPlvLdz8rGZZ"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'from my hospital feedback form n nwhile this was an unpleasant experience to say the least the worst part of it all was the way dr haslett treated me doctors on previous shifts introduced themselves discussed their plans and said goodbye before leaving the hospital however i did not meet dr haslett '"
            ]
          },
          "execution_count": 58,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Review excerpt reconstructed from integer sequence.\n",
        "tokenizer.sequences_to_texts([X_train[0]])[0][:300]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 59,
      "metadata": {
        "id": "ZirHR4_OC87H"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'From my hospital feedback form:\\\\n\\\\nWhile this was an unpleasant experience to say the least, the worst part of it all was the way Dr. Haslett treated me.  Doctors on previous shifts introduced themselves, discussed their plans and said goodbye before leaving the hospital.  However, I did not meet Dr'"
            ]
          },
          "execution_count": 59,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Original review text.\n",
        "train_reviews.iloc[0][:300]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bEqDNjactNk1"
      },
      "source": [
        "Some models and situations require us to **pad** our sequences to the same length. While that's not the case here, it can still be beneficial to have all our inputs (and consequently, our batches) to be of uniform size to help with optimizations.<br><br>\n",
        "In this case, we'll make all our reviews 200 tokens in length (in practice, you can choose a number based on some analysis). So the reviews longer than 200 tokens will be truncated, while the reviews shorter than 200 will be padded with zeroes.<br>\n",
        "https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/sequence/pad_sequences"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 60,
      "metadata": {
        "id": "2lTEjMgDqIPp"
      },
      "outputs": [],
      "source": [
        "MAX_REVIEW_LEN = 200\n",
        "X_train = keras.preprocessing.sequence.pad_sequences(X_train, maxlen=MAX_REVIEW_LEN)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 61,
      "metadata": {
        "id": "d2tHPfAVv37k"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[ 1037   957     2   356    36   184   267    13    24   576    21    13\n",
            "   132    73    47   944 16362   102    22   141     7     1   238    64\n",
            "     6  5990     5     2    24     5    30  1743     7    75     1  5352\n",
            "   184   267    64    24   106    28   238     3   278   102    23  2902\n",
            "    17    22    13    22    10     4    31     3    24    68  2489     3\n",
            "   102   494  8000    49     3    24    91   538  6781   136     2     3\n",
            "     6    22   828     9     1 13837     7  3942     3     6   142    49\n",
            "     1    46     3    24   419    25     1  2224     5   121     2  2019\n",
            "     5   429   238   694    17   118   102   477  2019    80   169  1026\n",
            "    38    13     1     7  3442    28  1248   227    10  1427    17    80\n",
            "   376    59  5112    18   944 16362     3     6   531  5381    36    55\n",
            "   169  4867     5    10   376     4   120     2    37     3   102    99\n",
            "   141    55     6     1 15231     7  3942    64   161   226    14   803\n",
            "     9     4   893     2  3308    33    27   103 15232    13     3   531\n",
            "    48     9     1   615     7    60    53   147    30     4    43   189\n",
            "  3442   107  1882     7    33     6    10    69  2132    74 18586   100\n",
            "     1  3942   674    53   442   244    39  1055]\n",
            "[    0     0     0     0     0     0     0     0     0     0     0     0\n",
            "     0     0     0     0     0     0     0     0     0     0     0     0\n",
            "     0     0     0     0     0     0     0     0     0     0     0     0\n",
            "     0     0     0     0     0     0     0     0     0     0     0     0\n",
            "     0     0     0     0     0     0     0     0     0     0     0     0\n",
            "     0     0     0     0     0     0     0     0     0     0     0     0\n",
            "     0     0     0     0     0     0     0     0     0     0     0     0\n",
            "     0     0     0     0     0     0     0     0     0     0     0     0\n",
            "     0     0     0     0     0     0     0     0     0     0     0     0\n",
            "     0     0     0     0     0     0     0     0     0     0     0     0\n",
            "     0     0     0     0     0     0     0     0     0     0     0     0\n",
            "     0     0     0     0     0     0     0     0     0     0     0     0\n",
            "     0     0     0     0     0     0     0     0     0     0     0     0\n",
            "     0     0     0     0     0     0     0     0     0     0     0     0\n",
            "     0     0     0     0     0     0     0    38    16    31  2281    17\n",
            "     3   141     1   133   619     5  5785     2     1  1955     4   208\n",
            "   855  1465     1   539 11887    26   122  1351]\n"
          ]
        }
      ],
      "source": [
        "print(X_train[0])\n",
        "print(X_train[1])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nBAg3laKwDWO"
      },
      "source": [
        "Our training set is prepared. We can now also vectorize and pad our validation set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 62,
      "metadata": {
        "id": "E9P72JJYHqKp"
      },
      "outputs": [],
      "source": [
        "X_val = tokenizer.texts_to_sequences(val_reviews)\n",
        "X_val = keras.preprocessing.sequence.pad_sequences(X_val, maxlen=MAX_REVIEW_LEN)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P8dy9OWMwafM"
      },
      "source": [
        "Now we need to incorporate the Google News vectors (currently loaded into gensim) into our Keras model. What we'll do is create an embedding matrix that maps each tokenizer integer to its respective word vector.<br><br>\n",
        "For example, here's the index for the word \"good\" from the Keras tokenizer and the word vector for \"good\" from gensim. We want a matrix which maps the index to the vector.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 63,
      "metadata": {
        "id": "5m7OW6-pwaJc"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "34\n"
          ]
        }
      ],
      "source": [
        "print(tokenizer.word_index['good'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 64,
      "metadata": {
        "id": "3rOAy_IkHqfx"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[ 0.04052734  0.0625     -0.01745605  0.07861328  0.03271484 -0.01263428\n",
            "  0.00964355  0.12353516 -0.02148438  0.15234375 -0.05834961 -0.10644531\n",
            "  0.02124023  0.13574219 -0.13183594  0.17675781  0.27148438  0.13769531\n",
            " -0.17382812 -0.14160156 -0.03076172  0.19628906 -0.03295898  0.125\n",
            "  0.25390625  0.12695312 -0.15234375  0.03198242  0.01135254 -0.01361084\n",
            " -0.12890625  0.01019287  0.23925781 -0.08447266  0.140625    0.13085938\n",
            " -0.04516602  0.06494141  0.02539062  0.05615234  0.24609375 -0.20507812\n",
            "  0.23632812 -0.00860596 -0.02294922  0.05078125  0.10644531 -0.03564453\n",
            "  0.08740234 -0.05712891]\n"
          ]
        }
      ],
      "source": [
        "# Part of the vector for the word 'good'.\n",
        "print(word_vectors['good'][:50])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "elzisAtxyO-o"
      },
      "source": [
        "We'll create this embedding matrix by first initializing a matrix of zeros, then looping over every word in the tokenizer vocabulary and:\n",
        "1. Checking if the word has a corresponding vector in gensim.\n",
        "2. If it does, then copy the vector into the matrix row corresponding to the word's index."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 66,
      "metadata": {
        "id": "Iw25Irs4J-JG"
      },
      "outputs": [],
      "source": [
        "# + 1 to account for padding token.\n",
        "num_tokens = len(tokenizer.word_index) + 1\n",
        "\n",
        "# Initialize a matrix of zeroes of size: vocabulary x embedding dimension.\n",
        "embedding_dim = 300\n",
        "embedding_matrix = np.zeros((num_tokens, embedding_dim))\n",
        "\n",
        "for word, i in tokenizer.word_index.items():\n",
        "  if word_vectors.has_index_for(word):\n",
        "    embedding_matrix[i] = word_vectors[word].copy()\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 67,
      "metadata": {
        "id": "uncvu4lcJ-OI"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[ 0.04052734  0.0625     -0.01745605  0.07861328  0.03271484 -0.01263428\n",
            "  0.00964355  0.12353516 -0.02148438  0.15234375 -0.05834961 -0.10644531\n",
            "  0.02124023  0.13574219 -0.13183594  0.17675781  0.27148438  0.13769531\n",
            " -0.17382812 -0.14160156 -0.03076172  0.19628906 -0.03295898  0.125\n",
            "  0.25390625  0.12695312 -0.15234375  0.03198242  0.01135254 -0.01361084\n",
            " -0.12890625  0.01019287  0.23925781 -0.08447266  0.140625    0.13085938\n",
            " -0.04516602  0.06494141  0.02539062  0.05615234  0.24609375 -0.20507812\n",
            "  0.23632812 -0.00860596 -0.02294922  0.05078125  0.10644531 -0.03564453\n",
            "  0.08740234 -0.05712891]\n"
          ]
        }
      ],
      "source": [
        "# Quick visual check.\n",
        "print(embedding_matrix[tokenizer.word_index['good']][:50])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CmzNcyFu0Noo"
      },
      "source": [
        "We're ready to build our first model using pretrained word vectors. The first layer we'll add is a Keras **embedding** layer which is essentially a trainable lookup table/matrix.<br>\n",
        "https://keras.io/api/layers/base_layer/#layer-class<br>\n",
        "https://keras.io/api/layers/core_layers/embedding/<br><br>\n",
        "In this case, we'll populate the **embedding** layer with the embedding matrix we created, and set *trainable* to True. This means we'll allow the learning algorithm training the classification model to adjust/fine-tune the word vectors as needed for greater performance. This corresponds to one of the scenarios we covered in the slides."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 68,
      "metadata": {
        "id": "ALs6rkTnJ-aj"
      },
      "outputs": [],
      "source": [
        "embedding_layer = layers.Embedding(\n",
        "    num_tokens,\n",
        "    embedding_dim,\n",
        "    embeddings_initializer=keras.initializers.Constant(embedding_matrix),\n",
        "    input_length=MAX_REVIEW_LEN,\n",
        "    trainable=True\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OqRgEWpb2iuI"
      },
      "source": [
        "We'll use a simple architecture for this model. Each training example is a sequence of *integers* which gets converted to a sequence of *vectors* (embeddings), but subsequent layers are expecting one vector per review. So we're inserting a **GlobalAveragePooling1D** layer after the embedding layer to average out all the word vectors into a single vector, before sending it further into the network. For classification, this can be pretty effective as a base model approach.<br>\n",
        "https://keras.io/api/layers/pooling_layers/global_average_pooling1d/<br>\n",
        "\n",
        "There was no science behind choosing 128 units in the first hidden layer and 64 units in the second hidden layer. The intuition was that the signal would be distilled from 300 dimensions down to 128 dimensions, then down to 64 dimensions before going to output.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 69,
      "metadata": {
        "id": "1RrLd4gzbhVS"
      },
      "outputs": [],
      "source": [
        "tf.random.set_seed(0)\n",
        "\n",
        "model = keras.Sequential()\n",
        "\n",
        "# This layer will output a sequence of 300-dimension *vectors*, one for each element in the input sequence.\n",
        "model.add(embedding_layer)\n",
        "\n",
        "# This layer will calculate an average of those vectors.\n",
        "model.add(layers.GlobalAveragePooling1D())\n",
        "\n",
        "model.add(layers.Dense(128, activation='relu', kernel_initializer=tf.keras.initializers.random_normal(seed=1)))\n",
        "model.add(layers.Dense(64, activation='relu', kernel_initializer=tf.keras.initializers.random_normal(seed=1)))\n",
        "model.add(layers.Dense(1, activation='sigmoid', kernel_initializer=tf.keras.initializers.random_normal(seed=1)))\n",
        "\n",
        "model.compile(optimizer='adam',\n",
        "              loss='binary_crossentropy',\n",
        "              metrics=['accuracy'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dlmis9Oz4_jy"
      },
      "source": [
        "Here's an example of what's going to happen under the hood to turn review text into a single vector for the dense layers:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 70,
      "metadata": {
        "id": "1r5K-Qqb1d-H"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Review: fantastic papaya steak\n",
            "Review as sequence of integers: [[543, 5844, 305]]\n",
            "Review embeddings shape: (Batch size: 1, Sequence length: 3, Embedding size: 300)\n",
            "Average of embeddings (shape): (1, 300)\n"
          ]
        }
      ],
      "source": [
        "review = \"fantastic papaya steak\"\n",
        "print(f\"Review: {review}\")\n",
        "\n",
        "review_sequence = tokenizer.texts_to_sequences([review])\n",
        "print(f\"Review as sequence of integers: {review_sequence}\")\n",
        "\n",
        "review_embeddings = embedding_layer(np.array(review_sequence))\n",
        "print(f\"Review embeddings shape: (Batch size: {review_embeddings.shape[0]}, \\\n",
        "Sequence length: {review_embeddings.shape[1]}, \\\n",
        "Embedding size: {review_embeddings.shape[2]})\")\n",
        "\n",
        "# How our document will be presented to the rest of the neural network.\n",
        "print(f\"Average of embeddings (shape): {np.mean(review_embeddings, axis=1).shape}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "546ouDEv41Ea"
      },
      "source": [
        "When we call the model's *summary* method, note how there are no params for the **GlobalAveragePooling1D** layer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 71,
      "metadata": {
        "id": "Pz31QQcb5nq5"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " embedding (Embedding)       (None, 200, 300)          28023900  \n",
            "                                                                 \n",
            " global_average_pooling1d (  (None, 300)               0         \n",
            " GlobalAveragePooling1D)                                         \n",
            "                                                                 \n",
            " dense (Dense)               (None, 128)               38528     \n",
            "                                                                 \n",
            " dense_1 (Dense)             (None, 64)                8256      \n",
            "                                                                 \n",
            " dense_2 (Dense)             (None, 1)                 65        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 28070749 (107.08 MB)\n",
            "Trainable params: 28070749 (107.08 MB)\n",
            "Non-trainable params: 0 (0.00 Byte)\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AwGYMD028zhO"
      },
      "source": [
        "We won't use **early stopping** for this run. This way, we'll be able to compare metrics between the train and validation sets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "oU6QBey79cis"
      },
      "outputs": [
        {
          "ename": "NameError",
          "evalue": "name 'model' is not defined",
          "output_type": "error",
          "traceback": [
            "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[1;32md:\\Github\\NLP-Model-Zoo\\NLP-preprocessing\\Word_vectors.ipynb Cell 100\u001b[0m line \u001b[0;36m1\n\u001b[1;32m----> <a href='vscode-notebook-cell:/d%3A/Github/NLP-Model-Zoo/NLP-preprocessing/Word_vectors.ipynb#Y166sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m history \u001b[39m=\u001b[39m model\u001b[39m.\u001b[39mfit(X_train, y_train, epochs\u001b[39m=\u001b[39m\u001b[39m20\u001b[39m, batch_size\u001b[39m=\u001b[39m\u001b[39m512\u001b[39m, validation_data\u001b[39m=\u001b[39m(X_val, y_val))\n",
            "\u001b[1;31mNameError\u001b[0m: name 'model' is not defined"
          ]
        }
      ],
      "source": [
        "history = model.fit(X_train, y_train, epochs=20, batch_size=512, validation_data=(X_val, y_val))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Neard5tEGwnG"
      },
      "outputs": [],
      "source": [
        "def plot_train_vs_val_performance(history):\n",
        "  training_losses = history.history['loss']\n",
        "  validation_losses = history.history['val_loss']\n",
        "\n",
        "  training_accuracy = history.history['accuracy']\n",
        "  validation_accuracy = history.history['val_accuracy']\n",
        "\n",
        "  epochs = range(1, len(training_losses) + 1)\n",
        "\n",
        "  import matplotlib.pyplot as plt\n",
        "  fig, (ax1, ax2) = plt.subplots(2)\n",
        "  fig.set_figheight(15)\n",
        "  fig.set_figwidth(15)\n",
        "  fig.tight_layout(pad=5.0)\n",
        "\n",
        "  # Plot training vs. validation loss.\n",
        "  ax1.plot(epochs, training_losses, 'bo', label='Training Loss')\n",
        "  ax1.plot(epochs, validation_losses, 'b', label='Validation Loss')\n",
        "  ax1.title.set_text('Training vs. Validation Loss')\n",
        "  ax1.set_xlabel('Epoch')\n",
        "  ax1.set_ylabel('Loss')\n",
        "  ax1.legend()\n",
        "\n",
        "  # PLot training vs. validation accuracy.\n",
        "  ax2.plot(epochs, training_accuracy, 'bo', label='Training Accuracy')\n",
        "  ax2.plot(epochs, validation_accuracy, 'b', label='Validation Accuracy')\n",
        "  ax2.title.set_text('Training vs. Validation Accuracy')\n",
        "  ax2.set_xlabel('Epoch')\n",
        "  ax2.set_ylabel('Accuracy')\n",
        "  ax2.legend()\n",
        "\n",
        "  plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lbSk2qE1cEfS"
      },
      "outputs": [],
      "source": [
        "plot_train_vs_val_performance(history)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xdeEdLD6W91U"
      },
      "source": [
        "We'll initialize a new embedding layer and model and train for epochs equalling the point where we saw the validation loss diverge from the training loss.<br>\n",
        "\n",
        "**NOTE**: We need to initialize a new embedding layer here because we set the *learnable* parameter to **True** in the previous embedding layer. This means the previous embeddings were almost certainly updated by the learning algorithm. So we're re-training a new model now with the original embeddings."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "V_nWiYy6cEbJ"
      },
      "outputs": [],
      "source": [
        "embedding_layer = layers.Embedding(\n",
        "    num_tokens,\n",
        "    embedding_dim,\n",
        "    embeddings_initializer=keras.initializers.Constant(embedding_matrix),\n",
        "    input_length=MAX_REVIEW_LEN,\n",
        "    trainable=True\n",
        ")\n",
        "\n",
        "model = keras.Sequential()\n",
        "model.add(embedding_layer)\n",
        "model.add(layers.GlobalAveragePooling1D())\n",
        "model.add(layers.Dense(128, activation='relu'))\n",
        "model.add(layers.Dense(64, activation='relu'))\n",
        "model.add(layers.Dense(1, activation='sigmoid'))\n",
        "\n",
        "model.compile(optimizer='adam',\n",
        "              loss='binary_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "model.fit(X_train, y_train, epochs=3, batch_size=512, validation_data=(X_val, y_val))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_4X_sdScYPAP"
      },
      "source": [
        "Now that we have a trained model, let's try it on the test data. As we did with the training data, we'll:\n",
        "1. Replace the labels with 0 for negative sentiment, and 1 for positive.\n",
        "2. Convert the reviews into a sequence of integers and pad/truncate each review to a fixed length."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AUsripS7SZdk"
      },
      "outputs": [],
      "source": [
        "yelp_test = pd.read_csv('yelp_review_polarity_csv/test.csv', names=['sentiment', 'review'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bm4Day_jYbTe"
      },
      "outputs": [],
      "source": [
        "yelp_test['sentiment'].replace(to_replace=1, value=0, inplace=True)\n",
        "yelp_test['sentiment'].replace(to_replace=2, value=1, inplace=True)\n",
        "yelp_test.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Qks0uXyzYbTe"
      },
      "outputs": [],
      "source": [
        "y_test = np.array(yelp_test['sentiment'])\n",
        "print(y_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-UyBts6EYbTe"
      },
      "outputs": [],
      "source": [
        "X_test = tokenizer.texts_to_sequences(yelp_test['review'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "loh2rh3WYbTe"
      },
      "outputs": [],
      "source": [
        "X_test = keras.preprocessing.sequence.pad_sequences(X_test, maxlen=MAX_REVIEW_LEN)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yKmqmLUkYbTf"
      },
      "outputs": [],
      "source": [
        "model.evaluate(X_test, y_test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9GBMe-YKZCDP"
      },
      "source": [
        "Not bad for a conceptually simple model where we average out a review's word vectors, run it through a few plain hidden layers, and out through a sigmoid function with no regularization and just using defaults for model components (e.g. optimizer settings).<br><br>\n",
        "We can now use the model for predictions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rNbNyr8PVJMv"
      },
      "outputs": [],
      "source": [
        "def sentiment(reviews):\n",
        "  seqs = tokenizer.texts_to_sequences(reviews)\n",
        "  seqs = keras.preprocessing.sequence.pad_sequences(seqs, maxlen=MAX_REVIEW_LEN)\n",
        "  return model.predict(seqs)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Jy-4MWV1VJRA"
      },
      "outputs": [],
      "source": [
        "# Real reviews from Google Reviews.\n",
        "pos_review = \"The best seafood joint in East Village San Diego!  Great lobster roll, great fish, great oysters, great bread, great cocktails, and such amazing service.  The atmosphere is top notch and the location is so much fun being located just a block away from Petco Park (San Diego Padres Stadium).\"\n",
        "neg_review = \"A thoroughly disappointing experience. When you book a Marriott you expect a certain standard. Albany falls way short. Room cleaning has to be booked 24 hours in advance but nobody thought to mention this at check in. The hotel is tired and needs a face-lift. The only bright light in a sea of mediocrity were the pancakes at breakfast. Sadly they weren't enough to save the experience. If you travel to Albany, then do yourself a big favour and book the Westin.\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wH8b-Z3GVJU3"
      },
      "outputs": [],
      "source": [
        "print(sentiment([pos_review, neg_review]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EUX0AZqYbt1y"
      },
      "source": [
        "# Training New Embeddings and a Model at the Same Time"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-ek2bnbCVJbv"
      },
      "source": [
        "For this last model, rather than using pretrained embeddings, we'll start with a **random** embedding matrix and let the model come up with its own vectors simultaneously while fitting the training data.<br><br>\n",
        "We'll also use **early stopping**, but otherwise keep everything else the same."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wCy90PyxVJfK"
      },
      "outputs": [],
      "source": [
        "tf.random.set_seed(0)\n",
        "\n",
        "model = keras.Sequential()\n",
        "\n",
        "# The 'trainable' property is True by default.\n",
        "model.add(layers.Embedding(input_dim=num_tokens,\n",
        "                           output_dim=embedding_dim,\n",
        "                           input_length=MAX_REVIEW_LEN))\n",
        "\n",
        "\n",
        "model.add(layers.GlobalAveragePooling1D())\n",
        "model.add(layers.Dense(128, activation='relu', kernel_initializer=tf.keras.initializers.random_normal(seed=1)))\n",
        "model.add(layers.Dense(64, activation='relu', kernel_initializer=tf.keras.initializers.random_normal(seed=1)))\n",
        "model.add(layers.Dense(1, activation='sigmoid', kernel_initializer=tf.keras.initializers.random_normal(seed=1)))\n",
        "\n",
        "model.compile(optimizer='adam',\n",
        "              loss='binary_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "es_callback = keras.callbacks.EarlyStopping(monitor='val_loss', patience=3)\n",
        "history = model.fit(X_train, y_train, epochs=20, batch_size=512, validation_data=(X_val, y_val), callbacks=[es_callback])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5dL0GwLUVJio"
      },
      "outputs": [],
      "source": [
        "model.evaluate(X_test, y_test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n3Uy8zP-0WUI"
      },
      "source": [
        "It looks like in this case, we get comparable performance between fine-tuning pretrained vectors and training embeddings from scratch as part of the model; likely because of the nature of the data and amount of it."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Si2Nx7NSfl1V"
      },
      "source": [
        "# Try This"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FsQYls6a0zH2"
      },
      "source": [
        "In our first model, we used pretrained vectors in the **embedding layer** and set the *trainable* property to **True**, allowing the model to fine-tune the word vectors.<br><br>\n",
        "Instantiate the same model but this time, set the *trainable* property in the **embedding layer** to **False**. What happens to training performance? Does the training speed increase or decrease? What happens if you try to add some regularization like dropout?<br>\n",
        "\n",
        "Other things you can try to see what happens: reduce the number of units, use a slower learning rate, reduce the number of hidden layers, reduce vocabulary, reduce embedding dimensions, use regularization, use a shorter sequence length."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "28u6ssTzflcE"
      },
      "outputs": [],
      "source": [
        "# Instantiate the embedding layer.\n",
        "\n",
        "\n",
        "model = keras.Sequential()\n",
        "\n",
        "# Add layers.\n",
        "\n",
        "\n",
        "# Compile model.\n",
        "\n",
        "\n",
        "# Call fit.\n",
        "\n",
        "\n",
        "# Evaluate the model.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TgJM0Flw6Imq"
      },
      "source": [
        "# Alternative Static Embedding Algorithms"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xcq27-hYKdMh"
      },
      "source": [
        "## GloVe\n",
        "**GloVe (Global Vectors for Word Representation)** is another algorithm for creating static word vectors. You can read the original GloVe paper and download pretrained word vectors here:<br>\n",
        "https://nlp.stanford.edu/projects/glove/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "331buv_FK4nQ"
      },
      "source": [
        "## Doc2Vec\n",
        "An algorithm which represents a document as a dense vector which addresses weaknesses of bag-of-words models.<br>\n",
        "https://arxiv.org/abs/1405.4053<br>\n",
        "https://radimrehurek.com/gensim/models/doc2vec.html<br>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aRq8HklIg2Gu"
      },
      "source": [
        "## fastText\n",
        "An alternative approach to creating embeddings. Instead of assigning a vector to each _word_ (e.g. a separate vector each for \"dog\" and \"dogs\"), a vector is assigned to each _subword_. For fastText, a subword is defined as a character n-gram.\n",
        "<br><br>\n",
        "So if n=3, then a word like \"hello\" would result in vectors for \"<he\", \"hel\", \"ell\", \"llo\", \"lo>\" (note that \"<\" and \">\" are special characters). The vector for \"hello\" would be the sum of all the above vectors. This helps deal with OOV situations because vectors can still be assigned to unseen words as long as the n-grams exist in the vocabulary.<br>\n",
        "https://fasttext.cc/<br>\n",
        "https://radimrehurek.com/gensim/models/fasttext.html\n",
        "<br><br>\n",
        "**We'll cover subword tokenization in greater detail later in the course.**"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "nlpdemystified-word-vectors.ipynb",
      "private_outputs": true,
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.17"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}