03_05_generating_custom_word_vectors.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <center>Generating Custom Word Vectors with Gensim</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>Dr. W.J.B. Mattingly</center>\n",
    "\n",
    "<center>Smithsonian Data Science Lab and United States Holocaust Memorial Museum</center>\n",
    "\n",
    "<center>January 2021</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Key Concepts in this Notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1) Gensim library<br>\n",
    "2) How to create word vectors in Gensim<br>\n",
    "3) Topic modeling<br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to Gensim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gensim is a powerful Python library that was originally designed to produce good topic models. **Topic models** are machine learning models that read over an entire corpus and cluster individual documents into clusters of similarity. In order to produce good results, Gensim (and other topic modeling methods) are reliant upon numerical represntations of words. In other words, these methods depend on word vectors. To have accurate results, therefore, Gensim is capable of generating word vectors with relatively minimal code. SpaCy, on the other hand, is an NLP library not capable of generating custom word vectors. While users can inject to words into models, spaCy is not designed to generate word vectors on its own. For this reason, even spaCy's documentation recomends using other libraries, such as Gensim to generate word vectors.\n",
    "\n",
    "In this notebook, we will be going through the process of generating our own word vectors. In order to reduce the time to perform the task at hand, we will use a toy corpus. This process, however, can easily be scaled for a corpus of millions of documents."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the Corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to generate word vectors, we need one thing: a corpus/ Let's create one right now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = \"Tom is cat, while Jerry is a mouse. Tom and Jerry are characters in a cartoon series. Some of the cartoons contain words, but most are silent. Silent cartoons still have music and sound effects.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we can give this corpus to Gensim, however, we need to do a few preprocessing techniques to it.\n",
    "\n",
    "1) First, we need to remove the stopwords from the corpus. Stopwords are words that occur frequently in a corpus, so frequently that they do not necessarily offer much meaning for distant reading and, as a result, throw off machine learning models. Other stopwords are words that occur with high frequency in a langauge as a whole. For our purposes, we will use the following stopwords available from the NLTK (natural language toolkit)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tom cat, jerry mouse. tom jerry characters cartoon series. cartoons contain words, silent. silent cartoons still music sound effects.\n"
     ]
    }
   ],
   "source": [
    "stopwords = [\"i\",\"me\",\"my\",\"myself\",\"we\",\"our\",\"ours\",\"ourselves\",\"you\",\"your\",\"yours\",\"yourself\",\"yourselves\",\n",
    "             \"he\",\"him\",\"his\",\"himself\",\"she\",\"her\",\"hers\",\"herself\",\"it\",\"its\",\"itself\",\"they\",\"them\",\"their\",\n",
    "             \"theirs\",\"themselves\",\"what\",\"which\",\"who\",\"whom\",\"this\",\"that\",\"these\",\"those\",\"am\",\"is\",\"are\",\"was\",\n",
    "             \"were\",\"be\",\"been\",\"being\",\"have\",\"has\",\"had\",\"having\",\"do\",\"does\",\"did\",\"doing\",\"a\",\"an\",\"the\",\"and\",\n",
    "             \"but\",\"if\",\"or\",\"because\",\"as\",\"until\",\"while\",\"of\",\"at\",\"by\",\"for\",\"with\",\"about\",\"against\",\"between\",\n",
    "             \"into\",\"through\",\"during\",\"before\",\"after\",\"above\",\"below\",\"to\",\"from\",\"up\",\"down\",\"in\",\"out\",\"on\",\"off\",\n",
    "             \"over\",\"under\",\"again\",\"further\",\"then\",\"once\",\"here\",\"there\",\"when\",\"where\",\"why\",\"how\",\"all\",\"any\",\"both\",\n",
    "             \"each\",\"few\",\"more\",\"most\",\"other\",\"some\",\"such\",\"no\",\"nor\",\"not\",\"only\",\"own\",\"same\",\"so\",\"than\",\"too\",\"very\",\n",
    "             \"s\",\"t\",\"can\",\"will\",\"just\",\"don\",\"should\",\"now\"\n",
    "            ]\n",
    "corpus = corpus.lower()\n",
    "words = corpus.split()\n",
    "\n",
    "new_corpus = []\n",
    "for word in words:\n",
    "    if word not in stopwords:\n",
    "        new_corpus.append(word)\n",
    "\n",
    "corpus = \" \".join(new_corpus)\n",
    "print (corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2) Second, this corpus should be divided into sentences. In order to do that, I recommend using spaCy's sentence tokenizer.\n",
    "3) While we do this, we should also eliminate the punctuation from the sentences. We can do this with the standard string library from Python.\n",
    "4) Also at this stage, we should lowercase our words (OPTIONAL)\n",
    "5) If we wish to produce a smaller amount of word vectors, we could also consider lemmatizing our words as well (OPTIONAL)\n",
    "6) We need to split the sentence into words and append that list of words to a new object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['tom', 'cat', 'jerry', 'mouse'], ['tom', 'jerry', 'characters', 'cartoon', 'series'], ['cartoons', 'contain', 'words', 'silent'], ['silent', 'cartoons', 'still', 'music', 'sound', 'effects']]\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "import string\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(corpus)\n",
    "\n",
    "sentences = []\n",
    "for sent in doc.sents:\n",
    "    sentence = sent.text.translate(str.maketrans('', '', string.punctuation))\n",
    "    words = sentence.split()\n",
    "    sentences.append(words)\n",
    "print (sentences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating Word Vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this stage, we can start preparing our word vectors. To do this, we will use the function below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n",
      "Made Phrases\n",
      "Made Bigrams\n",
      "Found sentences\n",
      "15\n",
      "Training model now...\n"
     ]
    }
   ],
   "source": [
    "def create_wordvecs(corpus, model_name):\n",
    "    from gensim.models.word2vec import Word2Vec\n",
    "    from gensim.models.phrases import Phrases, Phraser\n",
    "    from collections import defaultdict\n",
    "    \n",
    "    print (len(corpus))\n",
    "    \n",
    "\n",
    "    phrases = Phrases(corpus, min_count=30, progress_per=10000)\n",
    "    print (\"Made Phrases\")\n",
    "    \n",
    "    bigram = Phraser(phrases)\n",
    "    print (\"Made Bigrams\")\n",
    "    \n",
    "    sentences = phrases[corpus]\n",
    "    print (\"Found sentences\")\n",
    "    word_freq = defaultdict(int)\n",
    "\n",
    "    for sent in sentences:\n",
    "        for i in sent:\n",
    "            word_freq[i]+=1\n",
    "\n",
    "    print (len(word_freq))\n",
    "    \n",
    "    print (\"Training model now...\")\n",
    "    w2v_model = Word2Vec(min_count=1,\n",
    "                        window=2,\n",
    "                        size=10,\n",
    "                        sample=6e-5,\n",
    "                        alpha=0.03,\n",
    "                        min_alpha=0.0007,\n",
    "                        negative=20)\n",
    "    w2v_model.build_vocab(sentences, progress_per=10000)\n",
    "    w2v_model.train(sentences, total_examples=w2v_model.corpus_count, epochs=30, report_delay=1)\n",
    "    w2v_model.wv.save_word2vec_format(f\"data/{model_name}.txt\")\n",
    "create_wordvecs(sentences, \"word_vecs\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examining Word Vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can open up our word vectors and examine them. The first line in this text file will be the shape of the word vectors. This should be two integers. The first number (17) is the number of unique words in the vocabulary. The second number (10) are the number of dimensions of each word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15 10\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open (\"data/word_vecs.txt\", \"r\") as f:\n",
    "    data = f.readlines()\n",
    "    print (data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the first word in our word vectors, \"Tom\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tom 0.046630684 -0.019758822 -0.00631089 0.039143123 -0.027235914 -0.013175516 0.027348584 0.0005846504 -0.02984228 -0.016458655\n",
      "\n"
     ]
    }
   ],
   "source": [
    " print (data[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we see two pieces of information. The first is a string and it is the word itself. In this case, \"Tom\". The second bit of data is a series of 10 floats. These are our dimensions for the word. This is the numerical way in which \"Tom\" is understood by the Gensim model. This is precisely the data that spaCy expects to recieve in order to load these vectors into a model. In the next notebook, we will do just that."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try to use the above code to create your own custom word vectors for your own corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Video"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div align=\"center\">\n",
       "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/eZJm7PisZvk\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<div align=\"center\">\n",
    "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/eZJm7PisZvk\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}