lectures/lec02/Indexing1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "<br>\n",
    "## CS 429: Information Retrieval\n",
    "## Lecture 2: Indexing\n",
    "\n",
    "<br>\n",
    "\n",
    "### Dr. Aron Culotta\n",
    "### Illinois Institute of Technology \n",
    "\n",
    "<br><br><br>\n",
    "<br><br><br>\n",
    "\n",
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Indexing Pipeline\n",
    "\n",
    "1. Collect documents\n",
    "2. Tokenize\n",
    "3. Normalize\n",
    "4. Index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Indexing Pipeline\n",
    "\n",
    "document $\\xrightarrow{tokenize}$ (tokens, types) $\\xrightarrow{normalize}$ terms $\\xrightarrow{index}$ inverted index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# 1. Collect documents.\n",
    "document = \"he didn't know where he worked.\"\n",
    "\n",
    "# 2. Tokenize\n",
    "tokens = [\"he\", \"didn't\", \"know\", \"where\", \"he\", \"worked\"] # split; remove punctuation\n",
    "types = [\"he\", \"didn't\", \"know\", \"where\", \"worked\"] # unique tokens.\n",
    "\n",
    "# 3. Normalize\n",
    "# remove common words like 'where'; collapse word forms like worked -> work\n",
    "terms = [\"he\", \"didnt\", \"know\", \"work\"]\n",
    "\n",
    "# 4. Index\n",
    "index = {'he': [0],\n",
    "         'didnt': [0],\n",
    "         'know' : [0],\n",
    "         'work' : [0]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**e·quiv·a·lence class** (/i'kwivələns klas/) *n.*\n",
    "\n",
    "1. A subset whose elements are equivalent according to some relation $\\sim$ .\n",
    "\n",
    "    $S' \\subseteq S \\hspace{.2cm}$ s.t. $\\hspace{.2cm}x_i \\sim x_j \\hspace{.1cm} \\forall x_i, x_j \\in S'$\n",
    "    \n",
    "    E.g., consider the set \\{dog, cat, spider\\} and the relation \"has same number of legs\". Then there are two equivalence classes: \\{dog, cat\\} and \\{spider\\}."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**to·ken** (/tōkən/) *n*.\n",
    "\n",
    "1. A sequence of characters in a document that form a meaningful unit.\n",
    "2. The output of a *tokenizer*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**type** (/tīp/) *n.*\n",
    "\n",
    "1. An equivalence class of *tokens* under the string equality relation.\n",
    "2. By analogy to OO-programming, class:object :: type:token\n",
    "\n",
    "    *e.g., \"to be or not to be\"* $\\xrightarrow{tokenize}$ *tokens={to, be, or, not, to, be}* $\\hspace{.2cm}$  *types={to, be, or, not}*\n",
    "    \n",
    "    The type 'to' is an equivalence class containing the first and fifth tokens of the document.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**term** (/tərm/) *n.*\n",
    "\n",
    "1. An equivalence class of *types* under the relation \"have the same normalized form.\"\n",
    "2. The keys in the inverted index.\n",
    "\n",
    "    e.g., types={John, john, aren't, arent} $\\xrightarrow{normalize}$ terms={john, arent}.\n",
    "\n",
    "    The term \"john\" is an equivalence class containing types {John, john}.\n",
    "    \n",
    "    The term \"arent\" is an equivalence class containing types {arent, aren't}.\n",
    "    \n",
    "    \n",
    "<br><br><br>\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Tokenization\n",
    "\n",
    "**to·ken·i·za·tion** (/ˈtōkən izā-shən/) *n.*\n",
    "\n",
    "1. The process of splitting a document into tokens.\n",
    "\n",
    "    *Simplest approach*: split on whitespace.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['he', \"didn't\", 'know', 'where', 'he', 'worked.']"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "document.split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['what', 'about', 'multiple', 'spaces?']"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'what     about multiple     spaces?'.split()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Tokenization: Compound Nouns\n",
    "\n",
    "- *San Francisco*;  *New York University* vs *York University*\n",
    "- Solved somewhat by *phrase indexing* (next class)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Tokenization: Segmentation\n",
    "\n",
    "- *Lebensversicherungsgesellschaftsangestellter*\n",
    "  - \"life insurance company employee\"\n",
    "- 我不能说中国话\n",
    "- *\\#androidgames*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Statistical classification algorithms can be used to split (Part III of course). \n",
    "- Simpler: index character subsequences (*n-grams*).\n",
    "  - E.g., *\\#androidgames* $\\rightarrow$ {*#andr, andro, ndroi, droid, roidg, ..., games*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Tokenization: Punctuation\n",
    "\n",
    "- Remove all punctuation? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "  - \"didn't\", \"www.google.com\"\n",
    "  - [CAR](https://www.google.com/search?q=CAR) vs [C.A.R](https://www.google.com/search?q=C.A.R.).\n",
    "  - [O'Neill vs ONeill vs O Neill](https://www.google.com/#q=oneill+-o'neill&safe=active)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Tokenization: Regular Expressions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "symbol | meaning\n",
    "------ | -------\n",
    "\\b\t| Word boundary (zero width)\n",
    "\\d\t| Any decimal digit (equivalent to [0-9])\n",
    "\\D\t| Any non-digit character (equivalent to [^0-9])\n",
    "\\s\t| Any whitespace character (equivalent to [ \\t\\n\\r\\f\\v]\n",
    "\\S\t| Any non-whitespace character (equivalent to [^ \\t\\n\\r\\f\\v])\n",
    "\\w\t| Any alphanumeric character (equivalent to [a-zA-Z0-9_])\n",
    "\\W\t| Any non-alphanumeric character (equivalent to [^a-zA-Z0-9_])\n",
    "\\t\t| The tab character\n",
    "\\n\t| The newline character\n",
    "\n",
    "(source: <http://nltk.org/book/ch03.html>)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a', 'b', 'c']"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re  # Regular expression module\n",
    "re.split('x', 'axbxc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function split in module re:\n",
      "\n",
      "split(pattern, string, maxsplit=0, flags=0)\n",
      "    Split the source string by the occurrences of the pattern,\n",
      "    returning a list containing the resulting substrings.  If\n",
      "    capturing parentheses are used in pattern, then the text of all\n",
      "    groups in the pattern are also returned as part of the resulting\n",
      "    list.  If maxsplit is nonzero, at most maxsplit splits occur,\n",
      "    and the remainder of the string is returned as the final element\n",
      "    of the list.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(re.split)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on module re:\n",
      "\n",
      "NAME\n",
      "    re - Support for regular expressions (RE).\n",
      "\n",
      "MODULE REFERENCE\n",
      "    http://docs.python.org/3.5/library/re\n",
      "    \n",
      "    The following documentation is automatically generated from the Python\n",
      "    source files.  It may be incomplete, incorrect or include features that\n",
      "    are considered implementation detail and may vary between Python\n",
      "    implementations.  When in doubt, consult the module reference at the\n",
      "    location listed above.\n",
      "\n",
      "DESCRIPTION\n",
      "    This module provides regular expression matching operations similar to\n",
      "    those found in Perl.  It supports both 8-bit and Unicode strings; both\n",
      "    the pattern and the strings being processed can contain null bytes and\n",
      "    characters outside the US ASCII range.\n",
      "    \n",
      "    Regular expressions can contain both special and ordinary characters.\n",
      "    Most ordinary characters, like \"A\", \"a\", or \"0\", are the simplest\n",
      "    regular expressions; they simply match themselves.  You can\n",
      "    concatenate ordinary characters, so last matches the string 'last'.\n",
      "    \n",
      "    The special characters are:\n",
      "        \".\"      Matches any character except a newline.\n",
      "        \"^\"      Matches the start of the string.\n",
      "        \"$\"      Matches the end of the string or just before the newline at\n",
      "                 the end of the string.\n",
      "        \"*\"      Matches 0 or more (greedy) repetitions of the preceding RE.\n",
      "                 Greedy means that it will match as many repetitions as possible.\n",
      "        \"+\"      Matches 1 or more (greedy) repetitions of the preceding RE.\n",
      "        \"?\"      Matches 0 or 1 (greedy) of the preceding RE.\n",
      "        *?,+?,?? Non-greedy versions of the previous three special characters.\n",
      "        {m,n}    Matches from m to n repetitions of the preceding RE.\n",
      "        {m,n}?   Non-greedy version of the above.\n",
      "        \"\\\\\"     Either escapes special characters or signals a special sequence.\n",
      "        []       Indicates a set of characters.\n",
      "                 A \"^\" as the first character indicates a complementing set.\n",
      "        \"|\"      A|B, creates an RE that will match either A or B.\n",
      "        (...)    Matches the RE inside the parentheses.\n",
      "                 The contents can be retrieved or matched later in the string.\n",
      "        (?aiLmsux) Set the A, I, L, M, S, U, or X flag for the RE (see below).\n",
      "        (?:...)  Non-grouping version of regular parentheses.\n",
      "        (?P<name>...) The substring matched by the group is accessible by name.\n",
      "        (?P=name)     Matches the text matched earlier by the group named name.\n",
      "        (?#...)  A comment; ignored.\n",
      "        (?=...)  Matches if ... matches next, but doesn't consume the string.\n",
      "        (?!...)  Matches if ... doesn't match next.\n",
      "        (?<=...) Matches if preceded by ... (must be fixed length).\n",
      "        (?<!...) Matches if not preceded by ... (must be fixed length).\n",
      "        (?(id/name)yes|no) Matches yes pattern if the group with id/name matched,\n",
      "                           the (optional) no pattern otherwise.\n",
      "    \n",
      "    The special sequences consist of \"\\\\\" and a character from the list\n",
      "    below.  If the ordinary character is not on the list, then the\n",
      "    resulting RE will match the second character.\n",
      "        \\number  Matches the contents of the group of the same number.\n",
      "        \\A       Matches only at the start of the string.\n",
      "        \\Z       Matches only at the end of the string.\n",
      "        \\b       Matches the empty string, but only at the start or end of a word.\n",
      "        \\B       Matches the empty string, but not at the start or end of a word.\n",
      "        \\d       Matches any decimal digit; equivalent to the set [0-9] in\n",
      "                 bytes patterns or string patterns with the ASCII flag.\n",
      "                 In string patterns without the ASCII flag, it will match the whole\n",
      "                 range of Unicode digits.\n",
      "        \\D       Matches any non-digit character; equivalent to [^\\d].\n",
      "        \\s       Matches any whitespace character; equivalent to [ \\t\\n\\r\\f\\v] in\n",
      "                 bytes patterns or string patterns with the ASCII flag.\n",
      "                 In string patterns without the ASCII flag, it will match the whole\n",
      "                 range of Unicode whitespace characters.\n",
      "        \\S       Matches any non-whitespace character; equivalent to [^\\s].\n",
      "        \\w       Matches any alphanumeric character; equivalent to [a-zA-Z0-9_]\n",
      "                 in bytes patterns or string patterns with the ASCII flag.\n",
      "                 In string patterns without the ASCII flag, it will match the\n",
      "                 range of Unicode alphanumeric characters (letters plus digits\n",
      "                 plus underscore).\n",
      "                 With LOCALE, it will match the set [0-9_] plus characters defined\n",
      "                 as letters for the current locale.\n",
      "        \\W       Matches the complement of \\w.\n",
      "        \\\\       Matches a literal backslash.\n",
      "    \n",
      "    This module exports the following functions:\n",
      "        match     Match a regular expression pattern to the beginning of a string.\n",
      "        fullmatch Match a regular expression pattern to all of a string.\n",
      "        search    Search a string for the presence of a pattern.\n",
      "        sub       Substitute occurrences of a pattern found in a string.\n",
      "        subn      Same as sub, but also return the number of substitutions made.\n",
      "        split     Split a string by the occurrences of a pattern.\n",
      "        findall   Find all occurrences of a pattern in a string.\n",
      "        finditer  Return an iterator yielding a match object for each match.\n",
      "        compile   Compile a pattern into a RegexObject.\n",
      "        purge     Clear the regular expression cache.\n",
      "        escape    Backslash all non-alphanumerics in a string.\n",
      "    \n",
      "    Some of the functions in this module takes flags as optional parameters:\n",
      "        A  ASCII       For string patterns, make \\w, \\W, \\b, \\B, \\d, \\D\n",
      "                       match the corresponding ASCII character categories\n",
      "                       (rather than the whole Unicode categories, which is the\n",
      "                       default).\n",
      "                       For bytes patterns, this flag is the only available\n",
      "                       behaviour and needn't be specified.\n",
      "        I  IGNORECASE  Perform case-insensitive matching.\n",
      "        L  LOCALE      Make \\w, \\W, \\b, \\B, dependent on the current locale.\n",
      "        M  MULTILINE   \"^\" matches the beginning of lines (after a newline)\n",
      "                       as well as the string.\n",
      "                       \"$\" matches the end of lines (before a newline) as well\n",
      "                       as the end of the string.\n",
      "        S  DOTALL      \".\" matches any character at all, including the newline.\n",
      "        X  VERBOSE     Ignore whitespace and comments for nicer looking RE's.\n",
      "        U  UNICODE     For compatibility only. Ignored for string patterns (it\n",
      "                       is the default), and forbidden for bytes patterns.\n",
      "    \n",
      "    This module also defines an exception 'error'.\n",
      "\n",
      "CLASSES\n",
      "    builtins.Exception(builtins.BaseException)\n",
      "        sre_constants.error\n",
      "    \n",
      "    class error(builtins.Exception)\n",
      "     |  Common base class for all non-exit exceptions.\n",
      "     |  \n",
      "     |  Method resolution order:\n",
      "     |      error\n",
      "     |      builtins.Exception\n",
      "     |      builtins.BaseException\n",
      "     |      builtins.object\n",
      "     |  \n",
      "     |  Methods defined here:\n",
      "     |  \n",
      "     |  __init__(self, msg, pattern=None, pos=None)\n",
      "     |      Initialize self.  See help(type(self)) for accurate signature.\n",
      "     |  \n",
      "     |  ----------------------------------------------------------------------\n",
      "     |  Data descriptors defined here:\n",
      "     |  \n",
      "     |  __weakref__\n",
      "     |      list of weak references to the object (if defined)\n",
      "     |  \n",
      "     |  ----------------------------------------------------------------------\n",
      "     |  Methods inherited from builtins.Exception:\n",
      "     |  \n",
      "     |  __new__(*args, **kwargs) from builtins.type\n",
      "     |      Create and return a new object.  See help(type) for accurate signature.\n",
      "     |  \n",
      "     |  ----------------------------------------------------------------------\n",
      "     |  Methods inherited from builtins.BaseException:\n",
      "     |  \n",
      "     |  __delattr__(self, name, /)\n",
      "     |      Implement delattr(self, name).\n",
      "     |  \n",
      "     |  __getattribute__(self, name, /)\n",
      "     |      Return getattr(self, name).\n",
      "     |  \n",
      "     |  __reduce__(...)\n",
      "     |      helper for pickle\n",
      "     |  \n",
      "     |  __repr__(self, /)\n",
      "     |      Return repr(self).\n",
      "     |  \n",
      "     |  __setattr__(self, name, value, /)\n",
      "     |      Implement setattr(self, name, value).\n",
      "     |  \n",
      "     |  __setstate__(...)\n",
      "     |  \n",
      "     |  __str__(self, /)\n",
      "     |      Return str(self).\n",
      "     |  \n",
      "     |  with_traceback(...)\n",
      "     |      Exception.with_traceback(tb) --\n",
      "     |      set self.__traceback__ to tb and return self.\n",
      "     |  \n",
      "     |  ----------------------------------------------------------------------\n",
      "     |  Data descriptors inherited from builtins.BaseException:\n",
      "     |  \n",
      "     |  __cause__\n",
      "     |      exception cause\n",
      "     |  \n",
      "     |  __context__\n",
      "     |      exception context\n",
      "     |  \n",
      "     |  __dict__\n",
      "     |  \n",
      "     |  __suppress_context__\n",
      "     |  \n",
      "     |  __traceback__\n",
      "     |  \n",
      "     |  args\n",
      "\n",
      "FUNCTIONS\n",
      "    compile(pattern, flags=0)\n",
      "        Compile a regular expression pattern, returning a pattern object.\n",
      "    \n",
      "    escape(pattern)\n",
      "        Escape all the characters in pattern except ASCII letters, numbers and '_'.\n",
      "    \n",
      "    findall(pattern, string, flags=0)\n",
      "        Return a list of all non-overlapping matches in the string.\n",
      "        \n",
      "        If one or more capturing groups are present in the pattern, return\n",
      "        a list of groups; this will be a list of tuples if the pattern\n",
      "        has more than one group.\n",
      "        \n",
      "        Empty matches are included in the result.\n",
      "    \n",
      "    finditer(pattern, string, flags=0)\n",
      "        Return an iterator over all non-overlapping matches in the\n",
      "        string.  For each match, the iterator returns a match object.\n",
      "        \n",
      "        Empty matches are included in the result.\n",
      "    \n",
      "    fullmatch(pattern, string, flags=0)\n",
      "        Try to apply the pattern to all of the string, returning\n",
      "        a match object, or None if no match was found.\n",
      "    \n",
      "    match(pattern, string, flags=0)\n",
      "        Try to apply the pattern at the start of the string, returning\n",
      "        a match object, or None if no match was found.\n",
      "    \n",
      "    purge()\n",
      "        Clear the regular expression caches\n",
      "    \n",
      "    search(pattern, string, flags=0)\n",
      "        Scan through string looking for a match to the pattern, returning\n",
      "        a match object, or None if no match was found.\n",
      "    \n",
      "    split(pattern, string, maxsplit=0, flags=0)\n",
      "        Split the source string by the occurrences of the pattern,\n",
      "        returning a list containing the resulting substrings.  If\n",
      "        capturing parentheses are used in pattern, then the text of all\n",
      "        groups in the pattern are also returned as part of the resulting\n",
      "        list.  If maxsplit is nonzero, at most maxsplit splits occur,\n",
      "        and the remainder of the string is returned as the final element\n",
      "        of the list.\n",
      "    \n",
      "    sub(pattern, repl, string, count=0, flags=0)\n",
      "        Return the string obtained by replacing the leftmost\n",
      "        non-overlapping occurrences of the pattern in string by the\n",
      "        replacement repl.  repl can be either a string or a callable;\n",
      "        if a string, backslash escapes in it are processed.  If it is\n",
      "        a callable, it's passed the match object and must return\n",
      "        a replacement string to be used.\n",
      "    \n",
      "    subn(pattern, repl, string, count=0, flags=0)\n",
      "        Return a 2-tuple containing (new_string, number).\n",
      "        new_string is the string obtained by replacing the leftmost\n",
      "        non-overlapping occurrences of the pattern in the source\n",
      "        string by the replacement repl.  number is the number of\n",
      "        substitutions that were made. repl can be either a string or a\n",
      "        callable; if a string, backslash escapes in it are processed.\n",
      "        If it is a callable, it's passed the match object and must\n",
      "        return a replacement string to be used.\n",
      "    \n",
      "    template(pattern, flags=0)\n",
      "        Compile a template pattern, returning a pattern object\n",
      "\n",
      "DATA\n",
      "    A = 256\n",
      "    ASCII = 256\n",
      "    DOTALL = 16\n",
      "    I = 2\n",
      "    IGNORECASE = 2\n",
      "    L = 4\n",
      "    LOCALE = 4\n",
      "    M = 8\n",
      "    MULTILINE = 8\n",
      "    S = 16\n",
      "    U = 32\n",
      "    UNICODE = 32\n",
      "    VERBOSE = 64\n",
      "    X = 64\n",
      "    __all__ = ['match', 'fullmatch', 'search', 'sub', 'subn', 'split', 'fi...\n",
      "\n",
      "VERSION\n",
      "    2.2.1\n",
      "\n",
      "FILE\n",
      "    /usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/re.py\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(re)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a', 'b', 'c']"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.split('x+', 'axxxxbxxxxc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a', '', '', '', 'b', '', '', '', 'c']"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.split('x', 'axxxxbxxxxc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['hi', '+there']"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.split('\\+\\+', 'hi+++there')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['wha', 't', '', \"'\", 's', ' ', 'up', '?', '']"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.split('([\\W\\s]|t)', \"what's up?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['A', 'first-class', 'ticket', 'to', 'the', 'U.S.A.', \"isn't\", 'expensive?']"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = \"A first-class ticket to the U.S.A. isn't expensive?\"\n",
    "re.split(' ', text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "How to remove punctuation?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['A',\n",
       " 'first',\n",
       " 'class',\n",
       " 'ticket',\n",
       " 'to',\n",
       " 'the',\n",
       " 'U',\n",
       " 'S',\n",
       " 'A',\n",
       " 'isn',\n",
       " 't',\n",
       " 'expensive',\n",
       " '']"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.split('\\W+', text)           # \\W=not a word character; +=1 or more"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['A',\n",
       " 'first',\n",
       " 'class',\n",
       " 'ticket',\n",
       " 'to',\n",
       " 'the',\n",
       " 'U',\n",
       " 'S',\n",
       " 'A',\n",
       " 'isn',\n",
       " 't',\n",
       " 'expensive']"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall('\\w+', text)         # \\w=a word character [a-zA-Z0-9_]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['A',\n",
       " 'first',\n",
       " '-class',\n",
       " 'ticket',\n",
       " 'to',\n",
       " 'the',\n",
       " 'U',\n",
       " '.S',\n",
       " '.A',\n",
       " '.',\n",
       " 'isn',\n",
       " \"'t\",\n",
       " 'expensive',\n",
       " '?']"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# group punctuation with following letters\n",
    "re.findall('\\w+|\\S\\w*', text)  # \\S=not a space; |=OR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "How to keep hyphenated words and contractions together?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "re.findall(\"\\w+(?:[-']\\w+)*|[-.(]+|\\S\\w*\", text)\n",
    "# (?: specifies what to match, not what to capture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['A',\n",
       " 'first-class',\n",
       " 'ticket',\n",
       " 'to',\n",
       " 'the',\n",
       " 'U.S.A.',\n",
       " \"isn't\",\n",
       " 'expensive',\n",
       " '?']"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"(?:[A-Z]\\.)+|\\w+(?:[-']\\w+)*|[-.(]+|\\S\\w*\", text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Normalization\n",
    "\n",
    "**nor·mal·iz·a·tion** (/ˈnôrməˌlizā-shən/) *n.*\n",
    "\n",
    "1. The process of clustering *types* into *terms*.\n",
    "\n",
    "    Issues include: removing common words, special characters, casing, morphology\n",
    "    \n",
    "<br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Normalization: Stop words\n",
    "\n",
    "- Exclude common words\n",
    "  - *the*, *a*, *be*\n",
    "- Why?\n",
    "\n",
    "<br><br><br>\n",
    "<br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "  - save space (length of postings list is huge!)\n",
    "  - no semantic content (?!)\n",
    "  \n",
    "  <br><br><br>\n",
    "  <br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "*\"[to be or not to be](https://www.google.com/search?q=to+be+or+not+to+be&oq=to+be+or+not+to+be)\"* is all stop words!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Accents/Diacritics\n",
    "\n",
    "- naive vs. naïve\n",
    "- pena (sorrow) vs peña (cliff)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- What will users enter?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Case\n",
    "\n",
    "- Typically, just convert everything to lowercase.\n",
    "- E.g., search Google for [CAT -cat](https://www.google.com/search?q=CAT+-cat)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Stemming / Lemmatizing\n",
    "\n",
    "**mor·phol·o·gy** (/môrˈfäləjē/) *n.*\n",
    "\n",
    "1. (*Linguistics*) The study of the rules governing how words may take different forms in a language.\n",
    "\n",
    "*E.g.* \n",
    "\n",
    "- Pluralization: *dog* $\\xrightarrow{pluralize}$ *dogs* ; *goose* $\\xrightarrow{pluralize}$ *geese*\n",
    "\n",
    "\n",
    "- Tense: *play* $\\xrightarrow{past.tense}$ *played* ; *go* $\\xrightarrow{past.tense}$ *went*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**stem** (/stem/) *v.*\n",
    "\n",
    "1. To normalize based on crude morphology heuristics.\n",
    "\n",
    "    *E.g. remove all \"-s\" and \"-ed\" suffixes*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**lem·ma·tize** (ˈleməˌtīz/) *v.*\n",
    "\n",
    "1. To create equivalence classes of word types using the morphological rules of a language.\n",
    "\n",
    "    *Often relies on **part-of-speech** tagging to select rules*.\n",
    "\n",
    "    *E.g. if * bed * is a noun, then do not remove * -ed *suffix.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Simple stemmer\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'hello'"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = 'hello'\n",
    "s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def stem(word):\n",
    "    for suffix in ['ies', 's', 'ed', 'ing']: # order matters!\n",
    "        if word.endswith(suffix):\n",
    "            return word[:-len(suffix)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**What can go wrong?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Stemming Errors\n",
    "\n",
    "- **over-stemming**: merge types that should not be merged.\n",
    "- **under-stemming**: fail to merge types that should be merged."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ti', 't', 'ti', 'b', 'cit']"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "types = ['tied', 'ties', 'tis', 'bed', 'cities']\n",
    "[stem(w) for w in types]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**How does this affect search?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Porter Stemmer\n",
    "\n",
    "- Very commonly used stemmer with a complex set of heuristics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['tied', 'ties', 'tis', 'bed', 'cities']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['tie', 'tie', 'ti', 'bed', 'citi']"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.stem import PorterStemmer # See nltk.org (`pip install nltk`)\n",
    "# http://tartarus.org/~martin/PorterStemmer/\n",
    "# Original paper: http://web.simmons.edu/~benoit/lis466/PorterStemmingAlgorithm.pdf\n",
    "porter = PorterStemmer()\n",
    "print(types)\n",
    "[porter.stem(x) for x in types]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'citi'"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "porter.stem('city')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['bed',\n",
       " 'kiss',\n",
       " 'tie',\n",
       " 'ti',\n",
       " 'univers',\n",
       " 'univers',\n",
       " 'experi',\n",
       " 'experi',\n",
       " 'past',\n",
       " 'past',\n",
       " 'alumnu',\n",
       " 'alumni',\n",
       " 'adher',\n",
       " 'adhes',\n",
       " 'creat',\n",
       " 'creation']"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "types = ['bed', 'kiss',\n",
    "         'tied', 'tis',\n",
    "         'universal', 'university',\n",
    "         'experiment', 'experience',\n",
    "         'past', 'paste',\n",
    "         'alumnus', 'alumni',\n",
    "         'adhere', 'adhesion',\n",
    "         'create', 'creation']\n",
    "porter_results = [porter.stem(x) for x in types]\n",
    "porter_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# WordNet Lemmatizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "           type\t         porter\t     lemmatizer\n",
      "\n",
      "\n",
      "            bed\t            bed\t            bed\n",
      "           kiss\t           kiss\t           kiss\n",
      "           tied\t            tie\t           tied\n",
      "            tis\t             ti\t             ti\n",
      "      universal\t        univers\t      universal\n",
      "     university\t        univers\t     university\n",
      "     experiment\t         experi\t     experiment\n",
      "     experience\t         experi\t     experience\n",
      "           past\t           past\t           past\n",
      "          paste\t           past\t          paste\n",
      "        alumnus\t         alumnu\t        alumnus\n",
      "         alumni\t         alumni\t        alumnus\n",
      "         adhere\t          adher\t         adhere\n",
      "       adhesion\t          adhes\t       adhesion\n",
      "         create\t          creat\t         create\n",
      "       creation\t       creation\t       creation\n"
     ]
    }
   ],
   "source": [
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "# See description: https://wordnet.princeton.edu/wordnet/man/morphy.7WN.html\n",
    "lemm = WordNetLemmatizer()\n",
    "lemm_results = [lemm.lemmatize(x) for x in types]\n",
    "print('%15s\\t%15s\\t%15s' % ('type', 'porter', 'lemmatizer'))\n",
    "print('\\n')\n",
    "print('\\n'.join(['%15s\\t%15s\\t%15s' % (t[0], t[1], t[2])\n",
    "                 for t in zip(types, porter_results, lemm_results)]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**WordNet Lemmatizer needs part of speech:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "are\n",
      "is\n"
     ]
    }
   ],
   "source": [
    "print(lemm.lemmatize('are'))\n",
    "print(lemm.lemmatize('is'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "be\n",
      "be\n"
     ]
    }
   ],
   "source": [
    "print(lemm.lemmatize('are', 'v'))\n",
    "print(lemm.lemmatize('is', 'v'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# A principled approach?\n",
    "\n",
    "Given the many number of ways to preprocess text, how do we know which one is best?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Approaches:\n",
    "\n",
    "- Assume types that appear in similar contexts can be merged.\n",
    "  - e.g., *universally* and *universal* appear in similar documents, but not *university*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Learn from user behavior\n",
    "  - e.g., users click on very different search results if they search for *universal* vs *university*.\n",
    "  \n",
    "We'll explore both later in the course."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}