Martin_Lukan_Word_embedding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/mlukan/GDA3B2021/blob/main/Martin_Lukan_Word_embedding.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eqw_NvWnNHq6"
   },
   "source": [
    "# Word embedding manipulation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "svsBTr3zNPu4"
   },
   "source": [
    "## Load pre-trained embeddings\n",
    "\n",
    "You could train your own word embedding (using library like [gensim](https://radimrehurek.com/gensim/models/word2vec.html))  if you want, however you would need a lot of text and you would have to determine a ton of parameters (What is the size of your context, how big do you want your embedding, which algorithm to use, etc.).\n",
    "\n",
    "Why go through all that hassle when you can just use embeddings that specialist in the field already trained on huge corpus?\n",
    "\n",
    "[SpaCy](https://spacy.io/usage/models) is a library for NLP that provide such embeddings.\n",
    "\n",
    "### Run the code bellow :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "l3YRZwRkM98k",
    "outputId": "43384b36-5dc9-48be-8097-af4ea1ab22c4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting en_core_web_md==2.2.5\n",
      "\u001b[?25l  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_md-2.2.5/en_core_web_md-2.2.5.tar.gz (96.4MB)\n",
      "\u001b[K     |████████████████████████████████| 96.4MB 1.3MB/s \n",
      "\u001b[?25hRequirement already satisfied: spacy>=2.2.2 in /usr/local/lib/python3.7/dist-packages (from en_core_web_md==2.2.5) (2.2.4)\n",
      "Requirement already satisfied: srsly<1.1.0,>=1.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_md==2.2.5) (1.0.5)\n",
      "Requirement already satisfied: plac<1.2.0,>=0.9.6 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_md==2.2.5) (1.1.3)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_md==2.2.5) (1.0.5)\n",
      "Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_md==2.2.5) (0.8.2)\n",
      "Requirement already satisfied: thinc==7.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_md==2.2.5) (7.4.0)\n",
      "Requirement already satisfied: catalogue<1.1.0,>=0.0.7 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_md==2.2.5) (1.0.0)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_md==2.2.5) (2.0.5)\n",
      "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_md==2.2.5) (1.19.5)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_md==2.2.5) (2.23.0)\n",
      "Requirement already satisfied: blis<0.5.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_md==2.2.5) (0.4.1)\n",
      "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_md==2.2.5) (56.1.0)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_md==2.2.5) (3.0.5)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->en_core_web_md==2.2.5) (4.41.1)\n",
      "Requirement already satisfied: importlib-metadata>=0.20; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from catalogue<1.1.0,>=0.0.7->spacy>=2.2.2->en_core_web_md==2.2.5) (4.0.1)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->en_core_web_md==2.2.5) (1.24.3)\n",
      "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->en_core_web_md==2.2.5) (3.0.4)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->en_core_web_md==2.2.5) (2020.12.5)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->en_core_web_md==2.2.5) (2.10)\n",
      "Requirement already satisfied: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=0.20; python_version < \"3.8\"->catalogue<1.1.0,>=0.0.7->spacy>=2.2.2->en_core_web_md==2.2.5) (3.7.4.3)\n",
      "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=0.20; python_version < \"3.8\"->catalogue<1.1.0,>=0.0.7->spacy>=2.2.2->en_core_web_md==2.2.5) (3.4.1)\n",
      "Building wheels for collected packages: en-core-web-md\n",
      "  Building wheel for en-core-web-md (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for en-core-web-md: filename=en_core_web_md-2.2.5-cp37-none-any.whl size=98051305 sha256=f7db416e14429c0b63eff4a95781170bc574c333625c9ae58ba9afeeed81f63a\n",
      "  Stored in directory: /tmp/pip-ephem-wheel-cache-_vl_9asa/wheels/df/94/ad/f5cf59224cea6b5686ac4fd1ad19c8a07bc026e13c36502d81\n",
      "Successfully built en-core-web-md\n",
      "Installing collected packages: en-core-web-md\n",
      "Successfully installed en-core-web-md-2.2.5\n",
      "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
      "You can now load the model via spacy.load('en_core_web_md')\n"
     ]
    }
   ],
   "source": [
    "# Download the embeddings\n",
    "\n",
    "!python3 -m spacy download en_core_web_md\n",
    "\n",
    "# Load them\n",
    "\n",
    "import en_core_web_md\n",
    "nlp = en_core_web_md.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "m2ie5NjIZ_uW"
   },
   "source": [
    "### Some optionnal information on this model \n",
    "\n",
    "The word embeddings of this model are of size 300 (a pretty standard size) and are trained using [GloVe](https://mlexplained.com/2018/04/29/paper-dissected-glove-global-vectors-for-word-representation-explained/) algorithm. The model you loaded also come with other types of embeddings that may be useful for other NLP tasks (like Part Of speech vectors). \n",
    "\n",
    "There also exist a larger model with more words and models for other languages (see the SpaCy link)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "s7jXkmpINUkl"
   },
   "source": [
    "## Tokens embeddings and similarity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qIxKNZZHflq4"
   },
   "source": [
    "Now that the model is loaded, we can give it a sentence and it will tokenise it and return a list of tokens with a number of attributes.\n",
    "\n",
    "Run the two following cells and try to understand them : "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "l-FvGXNKN9Iq",
    "outputId": "b6e355de-05e9-452b-a273-c5c768a1a8e8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello True 5.586428\n",
      ", True 5.094723\n",
      "I True 6.4231944\n",
      "'m True 5.9417286\n",
      "a True 5.306696\n",
      "data True 7.1505103\n",
      "analyst True 7.489983\n",
      ". True 4.9316354\n",
      "aabbbb False 0.0\n"
     ]
    }
   ],
   "source": [
    "tokens = nlp(\"Hello, I'm a data analyst. aabbbb\")\n",
    "\n",
    "for t in tokens:\n",
    "    print(t.text, t.has_vector, t.vector_norm)\n",
    "\n",
    "# The attribute has_vector for \"aabbbb\" is False, it mean that no vector exist for this word in the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "_hDZxEBlhKoa",
    "outputId": "3835e087-e599-49c4-b86f-166859dd3218"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vector of \"Hello\" : \n",
      " [ 0.25233    0.10176   -0.67485    0.21117    0.43492    0.16542\n",
      "  0.48261   -0.81222    0.041321   0.78502   -0.077857  -0.66324\n",
      "  0.1464    -0.29289   -0.25488    0.019293  -0.20265    0.98232\n",
      "  0.028312  -0.081276  -0.1214     0.13126   -0.17648    0.13556\n",
      " -0.16361   -0.22574    0.055006  -0.20308    0.20718    0.095785\n",
      "  0.22481    0.21537   -0.32982   -0.12241   -0.40031   -0.079381\n",
      " -0.19958   -0.015083  -0.079139  -0.18132    0.20681   -0.36196\n",
      " -0.30744   -0.24422   -0.23113    0.09798    0.1463    -0.062738\n",
      "  0.42934   -0.078038  -0.19627    0.65093   -0.22807   -0.30308\n",
      " -0.12483   -0.17568   -0.14651    0.15361   -0.29518    0.15099\n",
      " -0.51726   -0.033564  -0.23109   -0.7833     0.018029  -0.15719\n",
      "  0.02293    0.49639    0.029225   0.05669    0.14616   -0.19195\n",
      "  0.16244    0.23898    0.36431    0.45263    0.2456     0.23803\n",
      "  0.31399    0.3487    -0.035791   0.56108   -0.25345    0.051964\n",
      " -0.10618   -0.30962    1.0585    -0.42025    0.18216   -0.11256\n",
      "  0.40576    0.11784   -0.19705   -0.075292   0.080723  -0.02782\n",
      " -0.15617   -0.44681   -0.15165    0.1692     0.098255  -0.031894\n",
      "  0.087143   0.26082    0.002706   0.1319     0.34439   -0.37894\n",
      " -0.4114     0.081571  -0.11674   -0.43711    0.011144   0.099353\n",
      "  0.26612    0.40025    0.18895   -0.18438   -0.30355   -0.2725\n",
      "  0.22468   -0.40614    0.15618   -0.16043    0.47147    0.0080203\n",
      "  0.56858    0.21934   -0.11181    0.79925    0.10714   -0.50146\n",
      "  0.063593   0.069465   0.15292   -0.2747    -0.20989    0.20737\n",
      " -0.10681    0.40651   -2.6438    -0.31139   -0.32157   -0.26458\n",
      " -0.35625    0.070013  -0.18838    0.48773   -0.26167   -0.020805\n",
      "  0.17819    0.15758   -0.13752    0.056464   0.30766   -0.066136\n",
      "  0.4748    -0.27335    0.09732   -0.20832    0.0039332  0.346\n",
      " -0.08702   -0.54924   -0.18759   -0.17174    0.060324  -0.13521\n",
      "  0.10419    0.30165    0.05798    0.21872   -0.073594  -0.20423\n",
      " -0.25279   -0.10471   -0.32163    0.12525   -0.31281    0.0097207\n",
      " -0.26777   -0.61121   -0.11089   -0.13652    0.035135  -0.4939\n",
      "  0.084857  -0.15494   -0.063509  -0.23935    0.28272    0.10849\n",
      " -0.3365    -0.60764    0.38576   -0.0095438  0.17499   -0.52723\n",
      "  0.62211    0.19544   -0.48977    0.036582  -0.128     -0.016827\n",
      "  0.25647   -0.31698    0.48257   -0.14184    0.11046   -0.3098\n",
      " -0.63141   -0.37268    0.23183   -0.14268   -0.02341    0.022255\n",
      " -0.044662  -0.16404   -0.25848    0.1629     0.024751   0.23348\n",
      "  0.27933    0.38998   -0.058968   0.11355    0.15673    0.18583\n",
      " -0.19814   -0.48123   -0.035084   0.078458  -0.49833    0.10855\n",
      " -0.20133    0.05292   -0.11583   -0.16009    0.16768    0.42362\n",
      " -0.23106    0.082465   0.24296   -0.16786    0.0080409  0.085947\n",
      "  0.38033    0.072981   0.1633     0.24704   -0.11094    0.15115\n",
      " -0.22068   -0.061944  -0.037091  -0.087923  -0.23181    0.15035\n",
      " -0.19093   -0.19113   -0.11894    0.094908  -0.0043347  0.15362\n",
      " -0.41201   -0.3073     0.18375    0.40206   -0.0034793 -0.10917\n",
      " -0.69522    0.10161   -0.079256   0.40329    0.22285   -0.19374\n",
      " -0.13315    0.073231   0.099832   0.11685   -0.21643   -0.1108\n",
      "  0.10341    0.097286   0.11196   -0.3894    -0.0089363  0.28809\n",
      " -0.10792    0.028811   0.32545    0.26052   -0.038941   0.075204\n",
      "  0.46031   -0.06293    0.21661    0.17869   -0.51917    0.33591  ]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "300"
      ]
     },
     "execution_count": 4,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Vector of \"' + tokens[0].text + '\" : \\n', tokens[0].vector)\n",
    "len(tokens[0].vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8HYFTWYfgsIr"
   },
   "source": [
    "You can also get the similarity between two tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "id": "CVrXPY9xgoWw",
    "outputId": "a027849a-1225-47d5-afd0-a50b4130c7cc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dog cat 0.80168545\n",
      "dog banana 0.24327643\n",
      "cat banana 0.28154364\n"
     ]
    }
   ],
   "source": [
    "tokens = nlp(\"dog cat banana\")\n",
    "\n",
    "for i in range(len(tokens)):\n",
    "    for j in range(i+1, len(tokens)):\n",
    "        print(tokens[i].text, tokens[j].text, tokens[i].similarity(tokens[j]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Bk4vnFT3kdJ8"
   },
   "source": [
    "**Warning** : You may find other pre-trained embeddings that you want to use or even train your owns with another library. All library has different methods, attributes and ways of handling embeddings, read the documentation and examples before using them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NWxsDQZDN93H"
   },
   "source": [
    "# Sentence embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6-J_6NjCmyzH"
   },
   "source": [
    "Now you know how to manipulate word embeddings, congratulation. \n",
    "So you have the sentence that you want to classify, and you have the embedding of each word of this sentence... Now what?\n",
    "\n",
    "Maybe you can concatenate all of these vectors and just give it to the classifier? \n",
    "\n",
    "Problems: \n",
    "\n",
    "- It would give a very very big vector. \n",
    "\n",
    "- It would be EXTREMELY sensible of the orders of the words \n",
    "\n",
    "- You would have to handle sentence having difference size with padding.\n",
    "\n",
    "In practice, state of the art model either train special sentence embeddings for their task or use special sequential neural network (RNN/LSTM). \n",
    "\n",
    "But we won't do that here (phew!). Actually just doing the average of the vectors works surprisingly well. And good news spacy comes with this functionality!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "OHtFgFAkOBpn",
    "outputId": "3ab11dc1-ebcf-45cb-838b-8e480d89e1ac"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "300"
      ]
     },
     "execution_count": 6,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens = nlp(\"Hello, I am a sentence.\")\n",
    "len(tokens.vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IG1IN73AtJKs"
   },
   "source": [
    "You can also get sentences similarity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "VCD70Bf4tSYA",
    "outputId": "e1c017cd-41c9-4483-f3e4-ff30660e79b1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.832282210939598\n",
      "0.7502564114243815\n",
      "0.7618915522647609\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Hello, I am a sentence."
      ]
     },
     "execution_count": 16,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens1 = nlp(\"Hello, I am a sentence.\")\n",
    "tokens2 = nlp(\"Hi, also some sort of phrase!\")\n",
    "tokens3 = nlp(\"This cat is cute.\")\n",
    "\n",
    "print(tokens1.similarity(tokens2))\n",
    "print(tokens1.similarity(tokens3))\n",
    "print(tokens2.similarity(tokens3))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-KU8ZHKYvIYz"
   },
   "source": [
    "Just doing a mere average on untreated sentence actually have one problem: it gives to much weight to stop word or other very frequent and not important words. \n",
    "\n",
    "That is why you should delete the stop words like you did previously.\n",
    "\n",
    "Try to do it now and compute the embeddings for each treated sentences : "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "WRP3vH7SpkdB",
    "outputId": "82531b62-9b06-436d-a868-5851be12f420"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n",
      "0.6751346465775762\n",
      "0.4319715571396021\n",
      "0.48920482878630084\n"
     ]
    }
   ],
   "source": [
    "import string \n",
    "import nltk\n",
    "nltk.download('punkt')\n",
    "nltk.download('stopwords')\n",
    "sent1 = nltk.word_tokenize(\"Hello, I am a sentence.\")\n",
    "sent2 = nltk.word_tokenize(\"Hi, also some sort of phrase!\")\n",
    "sent3 = nltk.word_tokenize(\"This cat is cute.\")\n",
    "tokens1=nlp(' '.join([w.lower() for w in sent1 if w.lower() not in nltk.corpus.stopwords.words(\"english\") and w not in string.punctuation]))\n",
    "tokens2=nlp(' '.join([w.lower() for w in sent2 if w.lower() not in nltk.corpus.stopwords.words(\"english\") and w not in string.punctuation]))\n",
    "tokens3=nlp(' '.join([w.lower() for w in sent3 if w.lower() not in nltk.corpus.stopwords.words(\"english\") and w not in string.punctuation]))\n",
    "\n",
    "print(tokens1.similarity(tokens2))\n",
    "print(tokens1.similarity(tokens3))\n",
    "print(tokens2.similarity(tokens3))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ATk9SisbOBVy"
   },
   "source": [
    "# Sentiment analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QeCfPLpDOPFR"
   },
   "source": [
    "## The dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kGq2dvArzv5O"
   },
   "source": [
    "### Run the code bellow :"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nO6c5rye0eH5"
   },
   "source": [
    "We won't use the twitter dataset that you already know because as strong as embeddings are they aren't great with unknown words/abreviation/emoji and the twitter dataset is full of them.\n",
    "\n",
    "We will instead use a dataset with review from Amazon, Yelp and IMDB. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 406
    },
    "id": "cjs4j_2zOpWW",
    "outputId": "49110635-1cc7-4974-a71b-37c735bef9f5"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>So there is no way for me to plug it in here i...</td>\n",
       "      <td>0</td>\n",
       "      <td>amazon_cells_labelled</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Good case, Excellent value.</td>\n",
       "      <td>1</td>\n",
       "      <td>amazon_cells_labelled</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Great for the jawbone.</td>\n",
       "      <td>1</td>\n",
       "      <td>amazon_cells_labelled</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Tied to charger for conversations lasting more...</td>\n",
       "      <td>0</td>\n",
       "      <td>amazon_cells_labelled</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>The mic is great.</td>\n",
       "      <td>1</td>\n",
       "      <td>amazon_cells_labelled</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2995</th>\n",
       "      <td>I think food should have flavor and texture an...</td>\n",
       "      <td>0</td>\n",
       "      <td>yelp_labelled</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2996</th>\n",
       "      <td>Appetite instantly gone.</td>\n",
       "      <td>0</td>\n",
       "      <td>yelp_labelled</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2997</th>\n",
       "      <td>Overall I was not impressed and would not go b...</td>\n",
       "      <td>0</td>\n",
       "      <td>yelp_labelled</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2998</th>\n",
       "      <td>The whole experience was underwhelming, and I ...</td>\n",
       "      <td>0</td>\n",
       "      <td>yelp_labelled</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2999</th>\n",
       "      <td>Then, as if I hadn't wasted enough of my life ...</td>\n",
       "      <td>0</td>\n",
       "      <td>yelp_labelled</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3000 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               sentence  ...                 source\n",
       "0     So there is no way for me to plug it in here i...  ...  amazon_cells_labelled\n",
       "1                           Good case, Excellent value.  ...  amazon_cells_labelled\n",
       "2                                Great for the jawbone.  ...  amazon_cells_labelled\n",
       "3     Tied to charger for conversations lasting more...  ...  amazon_cells_labelled\n",
       "4                                     The mic is great.  ...  amazon_cells_labelled\n",
       "...                                                 ...  ...                    ...\n",
       "2995  I think food should have flavor and texture an...  ...          yelp_labelled\n",
       "2996                           Appetite instantly gone.  ...          yelp_labelled\n",
       "2997  Overall I was not impressed and would not go b...  ...          yelp_labelled\n",
       "2998  The whole experience was underwhelming, and I ...  ...          yelp_labelled\n",
       "2999  Then, as if I hadn't wasted enough of my life ...  ...          yelp_labelled\n",
       "\n",
       "[3000 rows x 3 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "df_source = pd.read_csv(\"https://raw.githubusercontent.com/CindyAloui/datasets_wcs/master/sentiment_dataset.csv\", usecols=(\"sentence\", \"sentiment\", \"source\"))\n",
    "df_source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 257
    },
    "id": "62avNnxSyFSH",
    "outputId": "9bccf9c0-47a0-4bbd-ed79-ee28b63b91fe"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>source</th>\n",
       "      <th>sentiment</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">amazon_cells_labelled</th>\n",
       "      <th>0</th>\n",
       "      <td>500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">imdb_labelled</th>\n",
       "      <th>0</th>\n",
       "      <td>500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">yelp_labelled</th>\n",
       "      <th>0</th>\n",
       "      <td>500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 sentence\n",
       "source                sentiment          \n",
       "amazon_cells_labelled 0               500\n",
       "                      1               500\n",
       "imdb_labelled         0               500\n",
       "                      1               500\n",
       "yelp_labelled         0               500\n",
       "                      1               500"
      ]
     },
     "execution_count": 21,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_source.groupby(['source', 'sentiment']).count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aZuQ4rBaOSXQ"
   },
   "source": [
    "## Challenge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VL_BaWDO0VAp"
   },
   "source": [
    "Now you have all the elements to train a classifier for sentiment analysis using embeddings! A little reminder of the steps: \n",
    "\n",
    "- First take out the stop words so you won't have to do a weighted average. You can also lemmatize the text is you want but in this case it shouldn't have a big influence.\n",
    "\n",
    "- Then compute the sentence embeddings of the reviews. This is going to be our features.\n",
    "\n",
    "- Do a train test split.\n",
    "\n",
    "- Choose a type of classifier you want to use (for example a Logistic Regression).\n",
    "\n",
    "- Train and evaluate your classifier. \n",
    "\n",
    "You should be able to reach easily an accuracy of 80%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sp0ERRqOnWqn"
   },
   "source": [
    "### Removing stopwords and punctuation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {
    "id": "MaKXyV5oO7Wp"
   },
   "outputs": [],
   "source": [
    "mystr=\"So there is no way for me to plug it in here i...\"\n",
    "### Stemming using snowball stemmer returned a worse performing model\n",
    "def swremover(sent):\n",
    "  wordlist=nltk.word_tokenize(sent)\n",
    "  return ' '.join([w.lower() for w in wordlist if w.lower() not in nltk.corpus.stopwords.words(\"english\") and w not in string.punctuation and w not in [\"...\",\"``\",\"''\"]])\n",
    "swremover(mystr)\n",
    "\n",
    "df_work=df_source.copy()\n",
    "for i,row in df_work.iterrows():\n",
    "  newstring=swremover(df_work.loc[i,'sentence'])\n",
    "  #print(newstring)\n",
    "  df_work.at[i,'sentence']=newstring\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pqcCEn3tneG3"
   },
   "source": [
    "###  Using pretrained BERT model for sentence embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-cMOC45cfRMT",
    "outputId": "8fb440bc-99d3-416d-9dfb-4ca6492ca90a"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.seq_relationship.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.weight']\n",
      "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import transformers as ppb # pytorch transformers\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "model_class, tokenizer_class, pretrained_weights = (ppb.BertModel, ppb.BertTokenizer, 'bert-base-uncased')\n",
    "\n",
    "# Load pretrained model/tokenizer\n",
    "tokenizer = tokenizer_class.from_pretrained(pretrained_weights)\n",
    "model = model_class.from_pretrained(pretrained_weights)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "03EYqZZ_nwZv"
   },
   "source": [
    "### Tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "BB4snqJuMv_2",
    "outputId": "0693640e-0eb3-4879-80d2-e631188dbe6e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3000, 62)"
      ]
     },
     "execution_count": 141,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenized = df_work['sentence'].apply((lambda x: tokenizer.encode(x, add_special_tokens=True)))\n",
    "# Padding and masking the tokenized sentences\n",
    "max_len = 0\n",
    "for i in tokenized.values:\n",
    "    if len(i) > max_len:\n",
    "        max_len = len(i)\n",
    "\n",
    "padded = np.array([i + [0]*(max_len-len(i)) for i in tokenized.values])\n",
    "\n",
    "attention_mask = np.where(padded != 0, 1, 0)\n",
    "attention_mask.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oUIcDBjKodma"
   },
   "source": [
    "### Retrieving the vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {
    "id": "YOfTPI7goh6D"
   },
   "outputs": [],
   "source": [
    "#tokenized\n",
    "import numpy as np\n",
    "input_ids = torch.tensor(padded)  \n",
    "attention_mask = torch.tensor(attention_mask)\n",
    "\n",
    "with torch.no_grad():\n",
    "    last_hidden_states = model(input_ids, attention_mask=attention_mask)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {
    "id": "hoPbTfhU1v9N"
   },
   "outputs": [],
   "source": [
    "# Features\n",
    "features = last_hidden_states[0][:,0,:].numpy()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MHWV0Tm2n2ro"
   },
   "source": [
    "## Logistic regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {
    "id": "cfAJU_QCn6Ag"
   },
   "outputs": [],
   "source": [
    "X_train, X_test,y_train,y_test = train_test_split(features,df_work['sentiment'],\n",
    "                                     test_size = 0.2, \n",
    "                                     random_state = 42, \n",
    "                                     stratify = df_work['sentiment']) #split the dataset into train and test sets in a way that preserves the same proportions of examples in each class as observed in the original dataset\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "LPSfvJq12DoJ",
    "outputId": "fe297fae-c306-49ca-b17c-92aee254dab2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.815"
      ]
     },
     "execution_count": 149,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "lreg = LogisticRegression(max_iter=1000)\n",
    "lreg.fit(X_train, y_train)\n",
    "lreg.score(X_test, y_test)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "aUGIYOqU48lS",
    "outputId": "ae9a8bf0-e03a-4243-ebfa-b9650ec3b4c5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6466666666666666"
      ]
     },
     "execution_count": 148,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "modelDTC = DecisionTreeClassifier()\n",
    "modelDTC.fit(X_train, y_train)\n",
    "\n",
    "modelDTC.predict(X_test)\n",
    "modelDTC.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_SBP9QshOXAq"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "Martin Lukan Word embedding.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}