notebooks/2-text-processing-corpus.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Processing the Bughunt Corpus\n",
    "This notebook follows the process of taking the manually cleaned Bughunt corpus and creating a frequency distribution of the different bug words.\n",
    "\n",
    "NB: This notebook does not actually process the whole corpus -- that is done by the script `insect-freq-unigram.py`. The examples here are a walk-through and explanation of the code using a single file.\n",
    "\n",
    "We will use the code library called Natural Language Toolkit (NLTK) to provide a lot of text mining functions that are already written. More information on this can be found here: http://www.nltk.org/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Corpus Files\n",
    "\n",
    "We already have the corpus **split into files by decade**. Here is a list of them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('../corpora/bughunt/2-clean-by-decade/bughunt-clean-1800.txt'),\n",
       " PosixPath('../corpora/bughunt/2-clean-by-decade/bughunt-clean-1810.txt'),\n",
       " PosixPath('../corpora/bughunt/2-clean-by-decade/bughunt-clean-1820.txt'),\n",
       " PosixPath('../corpora/bughunt/2-clean-by-decade/bughunt-clean-1830.txt'),\n",
       " PosixPath('../corpora/bughunt/2-clean-by-decade/bughunt-clean-1840.txt'),\n",
       " PosixPath('../corpora/bughunt/2-clean-by-decade/bughunt-clean-1850.txt'),\n",
       " PosixPath('../corpora/bughunt/2-clean-by-decade/bughunt-clean-1860.txt'),\n",
       " PosixPath('../corpora/bughunt/2-clean-by-decade/bughunt-clean-1870.txt'),\n",
       " PosixPath('../corpora/bughunt/2-clean-by-decade/bughunt-clean-1880.txt'),\n",
       " PosixPath('../corpora/bughunt/2-clean-by-decade/bughunt-clean-1890.txt'),\n",
       " PosixPath('../corpora/bughunt/2-clean-by-decade/bughunt-clean-1900.txt'),\n",
       " PosixPath('../corpora/bughunt/2-clean-by-decade/bughunt-clean-1910.txt')]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "from pathlib import Path\n",
    "data_path = Path('..', 'corpora', 'bughunt', '2-clean-by-decade')\n",
    "files = [Path(root, filename) for root, _, files in os.walk(data_path) for filename in files]\n",
    "files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing to Process\n",
    "Before we are ready to process these files, we need to gather together some resources."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bug Words\n",
    "We have our list of **simple bug words** as a text file. Here it is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ant',\n",
       " 'bee',\n",
       " 'beetle',\n",
       " 'butterfly',\n",
       " 'cockroach',\n",
       " 'cricket',\n",
       " 'dragonfly',\n",
       " 'earwig',\n",
       " 'flea',\n",
       " 'fly',\n",
       " 'gnat',\n",
       " 'grasshopper',\n",
       " 'ladybird',\n",
       " 'louse',\n",
       " 'mosquito',\n",
       " 'moth',\n",
       " 'spider',\n",
       " 'termite',\n",
       " 'wasp']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wordlist = Path('..', 'wordlists', 'insect-wordlist.txt')\n",
    "with open(wordlist) as reader:\n",
    "    bug_words = reader.read().splitlines()\n",
    "bug_words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also have a list of the **stems** of bug words. **Stemming** is a form of word normalisation. It means reducing a word to its root, eliminating plurals and other inflections. Stems may not be actual words. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ant',\n",
       " 'bee',\n",
       " 'beetl',\n",
       " 'butterfli',\n",
       " 'cockroach',\n",
       " 'cricket',\n",
       " 'dragonfli',\n",
       " 'earwig',\n",
       " 'flea',\n",
       " 'fli',\n",
       " 'gnat',\n",
       " 'grasshopp',\n",
       " 'ladybird',\n",
       " 'lous',\n",
       " 'mosquito',\n",
       " 'moth',\n",
       " 'spider',\n",
       " 'termit',\n",
       " 'wasp']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stemlist = Path('..', 'wordlists', 'insect-wordstems.txt')\n",
    "with open(stemlist) as reader:\n",
    "    bug_stems = reader.read().splitlines()\n",
    "bug_stems"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see in the list above, the stems 'butterfli', 'dragonfli' and 'fli' are not real words.\n",
    "\n",
    "This contrasts with **lemmatisation** where the reduced word, the **lemma**, is a proper word in the language; in fact, it is the canonical or dictionary form.\n",
    "\n",
    "### English Stopwords\n",
    "We are not interested in common words in English that carry little meaning, such as 'the', 'a' and 'its'. There is no definitive list of stopwords, but a commonly-used list is provided by the Natural Language Toolkit (NLTK)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to ../nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['a',\n",
       " 'about',\n",
       " 'above',\n",
       " 'after',\n",
       " 'again',\n",
       " 'against',\n",
       " 'ain',\n",
       " 'all',\n",
       " 'am',\n",
       " 'an',\n",
       " 'and',\n",
       " 'any',\n",
       " 'are',\n",
       " 'aren',\n",
       " \"aren't\",\n",
       " 'as',\n",
       " 'at',\n",
       " 'be',\n",
       " 'because',\n",
       " 'been']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('stopwords', download_dir=Path('..', 'nltk_data'))\n",
    "from nltk.corpus import stopwords\n",
    "english_stops = set(stopwords.words('english'))\n",
    "sorted(list(english_stops))[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenising the Corpus\n",
    "Tokenising means splitting a text into meaningful elements, such as words, sentences, or symbols.\n",
    "\n",
    "To do this we use a simple facility provided by the NLTK to read in the files and a function to do the tokenising for us. The code example below takes a single corpus file and tokenises it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to ../nltk_data...\n",
      "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['CONTENTS',\n",
       " '.',\n",
       " 'CHAP',\n",
       " '.',\n",
       " 'I',\n",
       " '.',\n",
       " 'A',\n",
       " 'young',\n",
       " 'Bee',\n",
       " ',',\n",
       " 'deceived',\n",
       " 'by',\n",
       " 'fine',\n",
       " 'weather',\n",
       " ',',\n",
       " 'leave*',\n",
       " 'the',\n",
       " 'Hive',\n",
       " 'too',\n",
       " 'early']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download('punkt', download_dir=Path('..', 'nltk_data'))\n",
    "\n",
    "from nltk.corpus.reader import PlaintextCorpusReader\n",
    "reader = PlaintextCorpusReader('.', '')\n",
    "file_1810 = os.path.join(data_path, 'bughunt-clean-1810.txt')\n",
    "text = reader.raw(file_1810)\n",
    "\n",
    "from nltk import word_tokenize\n",
    "tokens = word_tokenize(text)\n",
    "tokens[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a number of problems with these tokens: the capitalisation of the words has been preserved, and some of the tokens have unwanted special characters or comprise single items of punctuation.\n",
    "\n",
    "### Normalising to Lowercase\n",
    "Normalising all words in a corpus to lowercase ensures that the same word in different cases can be recognised as the same word, e.g. we want 'Gnat', 'gnat' and 'GNAT' to be recognised as the same word.\n",
    "\n",
    "However, whether you choose to do this depends on the nature of your corpus and the questions you are investigating. For example, in another case, you may be not want the word 'Conservative' to be conflated with the word 'conservative'.\n",
    "\n",
    "In our case, we will lowercase the whole corpus immediately before tokenising it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['contents',\n",
       " '.',\n",
       " 'chap',\n",
       " '.',\n",
       " 'i.',\n",
       " 'a',\n",
       " 'young',\n",
       " 'bee',\n",
       " ',',\n",
       " 'deceived',\n",
       " 'by',\n",
       " 'fine',\n",
       " 'weather',\n",
       " ',',\n",
       " 'leave*',\n",
       " 'the',\n",
       " 'hive',\n",
       " 'too',\n",
       " 'early',\n",
       " ',']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens = word_tokenize(text.lower())\n",
    "tokens[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing Puctuation\n",
    "Punctuation such as commas, fullstops and apostrophes can complicate processing a corpus. For example, if punctuation is left in, the words \"termite\" and \"termite,\" might be considered to be different words.\n",
    "\n",
    "This is a complicated matter, however, and what you choose to do would vary depending on the nature of your corpus and what questions you wish to ask.\n",
    "\n",
    "It may be appropriate to remove punctuation at different stages of processing. In our case we are going to remove it *after* the text has been tokenised.\n",
    "\n",
    "We will replace *all* punctuation with the empty string ''."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['contents',\n",
       " '',\n",
       " 'chap',\n",
       " '',\n",
       " 'i',\n",
       " 'a',\n",
       " 'young',\n",
       " 'bee',\n",
       " '',\n",
       " 'deceived',\n",
       " 'by',\n",
       " 'fine',\n",
       " 'weather',\n",
       " '',\n",
       " 'leave',\n",
       " 'the',\n",
       " 'hive',\n",
       " 'too',\n",
       " 'early',\n",
       " '']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import string\n",
    "table = str.maketrans('', '', string.punctuation)\n",
    "tokens_nopunct = [token.translate(table) for token in tokens]\n",
    "tokens_nopunct[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing Non-Word Tokens\n",
    "\n",
    "We are still left with some problematic tokens that are not useful words, such as empty tokens `''` and tokens that may be chapter numbers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['', '', '', '', '', '', '', '', '', '']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens_empty = [word for word in tokens_nopunct if not word.isalpha()]\n",
    "tokens_empty[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1', '6', '1', '5', '1', '1', '1', '1', '1', '1']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens_nonwords = [word for word in tokens_nopunct if word.isnumeric()]\n",
    "tokens_nonwords[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can remove both these by filtering for only those words that are alphabetic:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['contents',\n",
       " 'chap',\n",
       " 'i',\n",
       " 'a',\n",
       " 'young',\n",
       " 'bee',\n",
       " 'deceived',\n",
       " 'by',\n",
       " 'fine',\n",
       " 'weather',\n",
       " 'leave',\n",
       " 'the',\n",
       " 'hive',\n",
       " 'too',\n",
       " 'early',\n",
       " 'and',\n",
       " 'contrary',\n",
       " 'to',\n",
       " 'the',\n",
       " 'advice']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words = [word for word in tokens_nopunct if word.isalpha()]\n",
    "words[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing Stopwords\n",
    "We are now ready to remove the stopwords we prepared earlier and thereby create a list of only meaningful words. Before using the stopwords, we will also remove all the punctuation so that it matches the text of the corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['contents',\n",
       " 'chap',\n",
       " 'young',\n",
       " 'bee',\n",
       " 'deceived',\n",
       " 'fine',\n",
       " 'weather',\n",
       " 'leave',\n",
       " 'hive',\n",
       " 'early',\n",
       " 'contrary',\n",
       " 'advice',\n",
       " 'commands',\n",
       " 'mother',\n",
       " 'sufferings',\n",
       " 'close',\n",
       " 'confinement',\n",
       " 'result',\n",
       " 'disobe',\n",
       " 'dience']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "english_stops_nopunct = {stopword.translate(table) for stopword in english_stops}\n",
    "words_nostops = [word for word in words if word not in english_stops_nopunct]\n",
    "words_nostops[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stemming the Tokens\n",
    "Stemming the tokens ensures that plurals and adjectives are reduced to the same stem and can be counted as the same word. For example, 'lice' and 'louse' will be normalised to 'lous', but so too will 'lousy', which may or may not be desirable.\n",
    "\n",
    "To do this we use another facility provided by the NLTK called a **stemmer**. There are many different ways to stems words, but we will use the Porter Stemmer. (The Porter Stemmer is the original stemmer, first created in 1979. It is simple and speedy, but has some important limitations.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['content',\n",
       " 'chap',\n",
       " 'young',\n",
       " 'bee',\n",
       " 'deceiv',\n",
       " 'fine',\n",
       " 'weather',\n",
       " 'leav',\n",
       " 'hive',\n",
       " 'earli',\n",
       " 'contrari',\n",
       " 'advic',\n",
       " 'command',\n",
       " 'mother',\n",
       " 'suffer',\n",
       " 'close',\n",
       " 'confin',\n",
       " 'result',\n",
       " 'disob',\n",
       " 'dienc']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk import PorterStemmer\n",
    "porter = PorterStemmer()\n",
    "stems = [porter.stem(word) for word in words_nostops]\n",
    "stems[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Frequency Distribution\n",
    "At last, we are ready to create a frequency distribution. We will use another NLTK facility called `FreqDist` to count the frequency of each unique word in the corpus, and then create a relative frequency value between `0` and `1`.\n",
    "\n",
    "First, we create a frequency distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.probability import FreqDist\n",
    "freqdist = FreqDist(stems)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the top 20 most frequent words (the numbers are the absolute word count):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('one', 432),\n",
       " ('butterfli', 404),\n",
       " ('bee', 310),\n",
       " ('would', 308),\n",
       " ('friend', 284),\n",
       " ('return', 232),\n",
       " ('littl', 206),\n",
       " ('could', 200),\n",
       " ('wing', 190),\n",
       " ('said', 184),\n",
       " ('much', 182),\n",
       " ('never', 174),\n",
       " ('though', 170),\n",
       " ('time', 170),\n",
       " ('see', 160),\n",
       " ('flower', 160),\n",
       " ('place', 150),\n",
       " ('think', 142),\n",
       " ('joe', 142),\n",
       " ('look', 140)]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "freqdist.most_common(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are not interested in a lot of these words, so the next thing to do is filter out all the words that are not in our list of bugs. Once we have done this we have a dictionary of stems and their relative frequencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ant': 0.00022559104854719364,\n",
       " 'bee': 0.0069933225049630034,\n",
       " 'beetl': 9.023641941887746e-05,\n",
       " 'butterfli': 0.009113878361306624,\n",
       " 'cockroach': 0.0,\n",
       " 'cricket': 0.00022559104854719364,\n",
       " 'dragonfli': 0.0,\n",
       " 'earwig': 0.0,\n",
       " 'flea': 0.0001353546291283162,\n",
       " 'fli': 0.0027522107922757625,\n",
       " 'gnat': 4.511820970943873e-05,\n",
       " 'grasshopp': 4.511820970943873e-05,\n",
       " 'ladybird': 0.0,\n",
       " 'lous': 0.0001353546291283162,\n",
       " 'mosquito': 0.0,\n",
       " 'moth': 0.0002707092582566324,\n",
       " 'spider': 9.023641941887746e-05,\n",
       " 'termit': 0.0,\n",
       " 'wasp': 0.00022559104854719364}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus.reader import WordListCorpusReader\n",
    "insect_words = WordListCorpusReader('.', [stemlist])\n",
    "\n",
    "insect_freq = {word: freqdist.freq(word) for word in insect_words.words()}\n",
    "insect_freq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What's Next\n",
    "In the script `insect-freq-unigram.py` the process above is applied to each of the corpus files in turn, and the results are output as a CSV file `insect-stem-freq-unigram.csv`. We will use this file in the next notebook `3-visualising-data.ipynb` to create some visualisations of the data."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}