gensim-dfr-tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Topic Modeling DFR Data with Gensim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial introduces the [Gensim](https://radimrehurek.com/gensim/index.html) topic modeling library with data obtained from JSTOR's [Data for Research](http://dfr.jstor.org) portal.\n",
    "\n",
    "\n",
    "Assumptions:\n",
    "- You have Python installed and configured so you can install additional packages (If you dont' know how to do this exactly I recommend installing the [Anaconda Python Distribution](https://www.continuum.io/why-anaconda).)\n",
    "- You have or can get some Data from JSTOR (described below).\n",
    "- You know a little somthing about [topic modeling](http://mcburton.net/blog/joy-of-tm/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing all the things"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a small pile of libraries, scripts, and data we need to install before we can start causing trouble with topic models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First things first, get some DATA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "JSTOR offers a \"Data For Research\" service which lets researchers download derived data from their digital collections. By \"derived data\" I mean they provide metadata, wordcounts, and ngram counts from a selection of articles (only 1000 for self-service users). \n",
    "\n",
    "The basic workflow is as follows:\n",
    "1. Use the web based search tool to narrow down a selection of articles. \n",
    "2. Once the set of documents has been determined, submit a *Dataset Request.*\n",
    "3. Specify the **Download Options** for the dataset request.\n",
    "    - For the **Data Type** select \"Citations\" and \"Word Counts\" (you could request the others, but we won't be using those data in this tutorial.\n",
    "    - For the **Output Format** select \"CSV\" (this is a much easier format to work with than XML).\n",
    "    - Give your request a **Job Title** (this is for your own purposes so you can keep track of requests)\n",
    "    - Set the max number of articles to 1000. \n",
    "4. Submit the request and wait for the job status to be \"Completed\" on your [DFR Requests page](http://dfr.jstor.org/fsearch/myrequests)\n",
    "\n",
    "Eventually you'll be able to download a ZIP file containing the following files:\n",
    "\n",
    "```\n",
    ".\n",
    "├── citations.tsv\n",
    "├── MANIFEST.txt\n",
    "├── README.txt\n",
    "└── wordcounts/\n",
    "    ├── wordcounts_10.2307_800826.CSV\n",
    "    ├── wordcounts_10.2307_800827.CSV\n",
    "    └── ...\n",
    "```\n",
    "*Note: if you included bigrams, trigrams, qudgrams, or keyterms you'll see those folders in the directory as well.*\n",
    "\n",
    "The files in the wordcount directory are the ones we are most interested in for this tutorial. These files look like this:\n",
    "\n",
    "```\n",
    "WORDCOUNTS,WEIGHT\n",
    "the,83\n",
    "of,65\n",
    "english,32\n",
    "a,31\n",
    "in,28\n",
    "and,24\n",
    "composition,19\n",
    "school,18\n",
    "for,14\n",
    "high,13\n",
    "v,13\n",
    "i,12\n",
    "...\n",
    "```\n",
    "\n",
    "That is, a column of words and a column of word counts (also called WEIGHT).\n",
    "\n",
    "OK, we have data!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Gensim Topic Modeling library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gensim is main player in this tutorials, it is a text analysis library for python that implements many interesting algorithms for \"distant reading.\" Gensim can do various kinds of [topic modeling](https://radimrehurek.com/gensim/tut2.html), [similarity searchers](https://radimrehurek.com/gensim/tut3.html), [TF-IDF](https://radimrehurek.com/gensim/models/tfidfmodel.html), and even [\"deep learning\" with the word2vec](http://rare-technologies.com/word2vec-tutorial/). Basically, `gensim` is like a swiss army knife for doing machine learning on text. It is worth spending some time reading the [tutorials](https://radimrehurek.com/gensim/tutorial.html), [API documentation](https://radimrehurek.com/gensim/apiref.html), and the [authors's blog](http://rare-technologies.com/blog/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# if Anacoda is on your PATH. If you don't know what that means run this cell.\n",
    "!pip install gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# if Anaconda is not on your PATH\n",
    "!~/anaconda/bin/pip install gensim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A Stopwords list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When doing bag-of-words text processing you need a [stopword](https://en.wikipedia.org/wiki/Stop_words) list so you can clean out all the words that lose their meaning when you slice up the sequences of text.\n",
    "\n",
    "We are going to use the stopword list that is part of the [MALLET topic modeling toolkit.](https://github.com/mimno/Mallet/blob/master/stoplists/en.txt).\n",
    "\n",
    "I have already included the stopword list in the repository for your convenience."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Boilerplate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# You need to put the name \n",
    "DFR_DATA_DIRECTORY = \"2015.10.29.8FrPrj2x\"\n",
    "SINGLE_CORPUS_FILENAME = \"all_documents.txt\"\n",
    "STOPWORD_FILE = \"en.txt\"\n",
    "\n",
    "\n",
    "# don't mess with these\n",
    "WORDCOUNTS = \"data/\" + DFR_DATA_DIRECTORY + \"/wordcounts/*.CSV\"\n",
    "SINGLE_CORPUS_FILE = \"data/\" + DFR_DATA_DIRECTORY + \"/\" + SINGLE_CORPUS_FILENAME"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# import various libraries\n",
    "import gensim\n",
    "import glob\n",
    "import logging\n",
    "\n",
    "\n",
    "logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)\n",
    "logger = logging.getLogger()\n",
    "logger.setLevel(logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to do topic modeling you need to re-shape the data so that the gensim library can work with the data. What this means in practice is combining all the individual CSV files containing wordcounts into a master file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\n",
    "with open(SINGLE_CORPUS_FILE, 'w') as f:\n",
    "    for csv_file in glob.glob(WORDCOUNTS):\n",
    "        with open(csv_file, 'r') as csvfile:\n",
    "            csvfile.readline() # skip the first line\n",
    "            document = \"\"\n",
    "            for line in csvfile:\n",
    "                word, count = line.split(',')\n",
    "                reshaped_document = (word + \" \") * int(count)\n",
    "                document += reshaped_document\n",
    "            f.write(document+\"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:gensim.corpora.dictionary:adding document #0 to Dictionary(0 unique tokens: [])\n",
      "INFO:gensim.corpora.dictionary:built Dictionary(42727 unique tokens: ['uplifter', 'targets', 'vievr', 'motoring', 'sulpment']...) from 1000 documents (total 1978518 corpus positions)\n"
     ]
    }
   ],
   "source": [
    "with open(SINGLE_CORPUS_FILE) as f:\n",
    "    dictionary = gensim.corpora.Dictionary(line.split()[1:] for line in f)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "525"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load stopwords into a python list\n",
    "with open(STOPWORD_FILE) as f:\n",
    "    stopwords = f.read().split('\\n')\n",
    "len(stopwords)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# to remove the stopwords I need to know their ID numbers\n",
    "stop_ids = [dictionary.token2id[word] \n",
    "                for word in stopwords \n",
    "                if word in dictionary.token2id]\n",
    "# use the filter_tokens function to remove the stopwords\n",
    "dictionary.filter_tokens(stop_ids)\n",
    "dictionary.compactify() # run this whenever you remove tokens to clean up gaps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:gensim.corpora.dictionary:discarding 20117 tokens: [('uplifter', 1), ('vievr', 1), ('sulpment', 1), ('jamess', 1), ('pettifogging', 1), ('projectsul', 1), ('whig', 1), ('spicer', 1), ('lukewarm', 1), ('curbed', 1)]...\n",
      "INFO:gensim.corpora.dictionary:keeping 22103 tokens which were in no less than 2 and no more than 1000 (=100.0%) documents\n",
      "INFO:gensim.corpora.dictionary:resulting dictionary: Dictionary(22103 unique tokens: ['targets', 'obligation', 'impersonally', 'egd', 'enlivens']...)\n"
     ]
    }
   ],
   "source": [
    "# filter out words that only appear once\n",
    "#dictionary.filter_extremes(no_below=2, no_above=1, keep_n=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "42220"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(dictionary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "with open(SINGLE_CORPUS_FILE) as f:\n",
    "    corpus = [dictionary.doc2bow(line.split()[1:]) for line in f]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the data in in the right shape we can now finally \"train\" the model and generate a set of topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "NUM_TOPICS = 20\n",
    "NUM_PASSES = 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:gensim.models.ldamodel:using symmetric alpha at 0.05\n",
      "INFO:gensim.models.ldamodel:using serial LDA version on this node\n",
      "INFO:gensim.models.ldamodel:running online LDA training, 20 topics, 4 passes over the supplied corpus of 1000 documents, updating model once every 1000 documents, evaluating perplexity every 1000 documents, iterating 50x with a convergence threshold of 0.001000\n",
      "WARNING:gensim.models.ldamodel:too few updates, training might not converge; consider increasing the number of passes or iterations to improve accuracy\n",
      "INFO:gensim.models.ldamodel:-12.863 per-word bound, 7451.5 perplexity estimate based on a held-out corpus of 1000 documents with 870584 words\n",
      "INFO:gensim.models.ldamodel:PROGRESS: pass 0, at document #1000/1000\n",
      "INFO:gensim.models.ldamodel:topic #5 (0.050): 0.016*english + 0.008*school + 0.007*work + 0.006*literature + 0.006*high + 0.005*teachers + 0.005*teacher + 0.004*time + 0.004*study + 0.004*schools\n",
      "INFO:gensim.models.ldamodel:topic #14 (0.050): 0.012*english + 0.008*school + 0.006*work + 0.005*teachers + 0.005*literature + 0.004*pupils + 0.004*teacher + 0.004*class + 0.004*college + 0.004*student\n",
      "INFO:gensim.models.ldamodel:topic #0 (0.050): 0.022*english + 0.010*school + 0.008*literature + 0.006*high + 0.006*teachers + 0.005*work + 0.004*chicago + 0.004*composition + 0.004*study + 0.004*reading\n",
      "INFO:gensim.models.ldamodel:topic #7 (0.050): 0.015*english + 0.007*work + 0.007*teachers + 0.006*school + 0.005*teaching + 0.004*journal + 0.004*class + 0.004*teacher + 0.004*high + 0.004*good\n",
      "INFO:gensim.models.ldamodel:topic #8 (0.050): 0.019*english + 0.010*school + 0.007*college + 0.006*teachers + 0.006*composition + 0.006*literature + 0.004*university + 0.004*high + 0.004*work + 0.004*study\n",
      "INFO:gensim.models.ldamodel:topic diff=7.035150, rho=1.000000\n",
      "INFO:gensim.models.ldamodel:-9.359 per-word bound, 656.6 perplexity estimate based on a held-out corpus of 1000 documents with 870584 words\n",
      "INFO:gensim.models.ldamodel:PROGRESS: pass 1, at document #1000/1000\n",
      "INFO:gensim.models.ldamodel:topic #13 (0.050): 0.017*english + 0.006*work + 0.005*teachers + 0.004*schools + 0.004*school + 0.004*class + 0.003*teacher + 0.003*time + 0.003*journal + 0.003*university\n",
      "INFO:gensim.models.ldamodel:topic #4 (0.050): 0.017*english + 0.011*school + 0.010*work + 0.006*teachers + 0.005*college + 0.005*schools + 0.004*high + 0.004*class + 0.004*composition + 0.004*student\n",
      "INFO:gensim.models.ldamodel:topic #10 (0.050): 0.011*english + 0.005*work + 0.005*time + 0.004*literature + 0.004*school + 0.004*teacher + 0.003*life + 0.003*teachers + 0.003*made + 0.003*study\n",
      "INFO:gensim.models.ldamodel:topic #11 (0.050): 0.030*english + 0.015*school + 0.012*teachers + 0.009*work + 0.008*high + 0.007*schools + 0.007*committee + 0.005*college + 0.005*teaching + 0.005*association\n",
      "INFO:gensim.models.ldamodel:topic #0 (0.050): 0.022*english + 0.012*school + 0.007*literature + 0.007*chicago + 0.007*university + 0.007*high + 0.006*cents + 0.005*teachers + 0.005*york + 0.004*composition\n",
      "INFO:gensim.models.ldamodel:topic diff=2.610201, rho=0.577350\n",
      "INFO:gensim.models.ldamodel:-8.898 per-word bound, 476.9 perplexity estimate based on a held-out corpus of 1000 documents with 870584 words\n",
      "INFO:gensim.models.ldamodel:PROGRESS: pass 2, at document #1000/1000\n",
      "INFO:gensim.models.ldamodel:topic #16 (0.050): 0.011*english + 0.006*school + 0.006*work + 0.004*literature + 0.004*reading + 0.003*composition + 0.003*teachers + 0.003*present + 0.003*high + 0.003*schools\n",
      "INFO:gensim.models.ldamodel:topic #7 (0.050): 0.012*english + 0.007*work + 0.005*teaching + 0.004*teachers + 0.004*school + 0.004*teacher + 0.004*class + 0.004*literature + 0.004*journal + 0.004*time\n",
      "INFO:gensim.models.ldamodel:topic #19 (0.050): 0.017*english + 0.010*school + 0.009*pp + 0.009*york + 0.008*book + 0.007*teachers + 0.007*university + 0.006*high + 0.005*chicago + 0.005*schools\n",
      "INFO:gensim.models.ldamodel:topic #6 (0.050): 0.008*school + 0.007*english + 0.005*high + 0.004*work + 0.004*thou + 0.003*creon + 0.003*debate + 0.003*schools + 0.003*speaking + 0.003*thee\n",
      "INFO:gensim.models.ldamodel:topic #2 (0.050): 0.016*english + 0.006*teacher + 0.006*study + 0.005*work + 0.005*teachers + 0.005*literature + 0.005*student + 0.005*composition + 0.004*pupils + 0.004*life\n",
      "INFO:gensim.models.ldamodel:topic diff=1.901607, rho=0.500000\n",
      "INFO:gensim.models.ldamodel:-8.708 per-word bound, 418.2 perplexity estimate based on a held-out corpus of 1000 documents with 870584 words\n",
      "INFO:gensim.models.ldamodel:PROGRESS: pass 3, at document #1000/1000\n",
      "INFO:gensim.models.ldamodel:topic #0 (0.050): 0.021*english + 0.012*school + 0.012*university + 0.011*cents + 0.010*chicago + 0.008*high + 0.007*postage + 0.006*single + 0.006*copies + 0.006*literature\n",
      "INFO:gensim.models.ldamodel:topic #10 (0.050): 0.008*english + 0.004*life + 0.004*time + 0.004*work + 0.003*man + 0.003*good + 0.003*illustrations + 0.003*literature + 0.003*school + 0.003*examinations\n",
      "INFO:gensim.models.ldamodel:topic #4 (0.050): 0.013*english + 0.010*school + 0.010*work + 0.005*student + 0.005*speech + 0.005*college + 0.004*class + 0.004*teachers + 0.004*public + 0.004*speaking\n",
      "INFO:gensim.models.ldamodel:topic #9 (0.050): 0.013*english + 0.008*work + 0.008*class + 0.007*teacher + 0.007*composition + 0.007*pupils + 0.006*school + 0.005*time + 0.005*pupil + 0.005*students\n",
      "INFO:gensim.models.ldamodel:topic #17 (0.050): 0.007*english + 0.006*play + 0.005*student + 0.004*class + 0.004*dramatic + 0.004*students + 0.004*plays + 0.004*men + 0.004*literature + 0.003*life\n",
      "INFO:gensim.models.ldamodel:topic diff=1.383266, rho=0.447214\n"
     ]
    }
   ],
   "source": [
    "# train the model, NOTE: THIS STEP TAKES A LONG TIME\n",
    "topic_model = gensim.models.LdaModel(corpus, num_topics=NUM_TOPICS, id2word=dictionary, passes=NUM_PASSES)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "english school university cents chicago high postage single copies literature\n",
      "english literature work teacher teachers teaching school composition students time\n",
      "english study teacher work teachers student literature composition life reading\n",
      "english grammar school speech teaching work study language teacher words\n",
      "english school work student speech college class teachers public speaking\n",
      "english school literature work high teacher class time teachers study\n",
      "school thou english high creon debate thee work thy speaking\n",
      "english work teaching literature teacher teachers class school time journal\n",
      "english literature college school composition teachers schools study reading high\n",
      "english work class teacher composition pupils school time pupil students\n",
      "english life time work man good illustrations literature school examinations\n",
      "english school teachers high work committee schools college council association\n",
      "cd english play school plays en men work women york\n",
      "english bathe page squire schools bird work ship enter business\n",
      "english school student work good teachers time pupils literature journal\n",
      "english literature students work school life college schools study student\n",
      "english school work reading literature present composition thought high teachers\n",
      "english play student class dramatic students plays men literature life\n",
      "pupils english class school voice play reading teacher time read\n",
      "english pp york school book university teachers chicago boston high\n"
     ]
    }
   ],
   "source": [
    "for topic in topic_model.show_topics(num_topics=-1, formatted=False):\n",
    "    print(\" \".join([word[1] for word in topic]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}