lectures/lec07/VSM.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# CS 429: Information Retrieval\n",
    "\n",
    "<br>\n",
    "\n",
    "## Lecture 7: Term weighting and the Vector Space Model\n",
    "\n",
    "<br>\n",
    "\n",
    "### Dr. Aron Culotta\n",
    "### Illinois Institute of Technology \n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Last time:\n",
    "\n",
    "- How do we compress the contents of an index?\n",
    "\n",
    "Today: \n",
    "\n",
    "- From boolean search to ranking\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Assignment 0 notes\n",
    "\n",
    "- <strike>Why is this inefficient?  </strike> &nbsp;&nbsp; <blue> It's not! </blue>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def intersect(list1, list2):\n",
    "    list(set(list1) & set(list2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{2, 3}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def set_intersection(set1, set2):\n",
    "    \"\"\" Return the intersection of two sets;\n",
    "    approximately what `set1 & set2` does.\"\"\"\n",
    "    smaller = set1 if len(set1) < len(set2) else set2\n",
    "    larger = set1 if len(set1) > len(set2) else set2\n",
    "    return set([x for x in smaller if x in larger])\n",
    "\n",
    "set_intersection(set([1,2,3,4,5]), set([2,3,6]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$O(min(|x|, |y|))$\n",
    "for sets $x$ and $y$\n",
    "\n",
    "Compare with $O(|x| + |y|)$ for book's intersection algorithm.\n",
    "\n",
    "<br><br><br><br>\n",
    "Then, why use lists instead of sets?\n",
    "\n",
    "- This simple posting list is rarely used in practice\n",
    "  - Skip lists\n",
    "  - In today's lecture, we'll look at ways to order postings list other than by id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<br><br><br><br><br>\n",
    "Another common error:\n",
    "\n",
    "- `create_index`: no duplicate doc IDs\n",
    "```\n",
    ">>> create_index([['a', 'a']])\n",
    "{'a': [0, 0]}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Why is boolean search bad?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Prone to user error\n",
    "- Have to think of all possible ways of expressing information need\n",
    "- Too many matching results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Why is boolean search good?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Users get exactly what they ask for\n",
    "- Good for integrating with other software"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Problems with boolean search\n",
    "\n",
    "- Too few results: \"error: null ptr exception in line 2341\" $\\rightarrow$ 0 results\n",
    "- Too many results: \"error: null ptr exception\" $\\rightarrow$ 1M results\n",
    "- User must carefully refine the query\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Ranked retrieval\n",
    "\n",
    "- Order search results by **relevance** to query.\n",
    "- Too many results is not a problem: user only looks at top 10-20.\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Ranked retrieval\n",
    "\n",
    "**Problem:** Given a query $q$ and a set of documents $D$, compute a score $s_i \\in [0, 1]$ for each document $d_i \\in D$.\n",
    "\n",
    "- higher scores $\\rightarrow d_i$ is more relevant to $q$.\n",
    "\n",
    "<br><br><br><br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Ranking one-word queries\n",
    "\n",
    "- Search index for \"dog\".\n",
    "\n",
    "- What should the score for a document be?\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Jaccard coefficient\n",
    "\n",
    "$$J(A, B) = \\frac{|A \\cap B|}{|A \\cup B|}$$\n",
    "\n",
    "- $J(A, A) = 1$\n",
    "- $J(A, B) = 0$ if $A \\cap B = 0$\n",
    "\n",
    "- Example: \n",
    "  - *q:* dogs with hats\n",
    "  - $d_1:$ big cats wearing hats\n",
    "  - $d_2:$ I like hats.\n",
    "  - $J(q, d_1)$? \n",
    "  - $J(q, d_2)$?\n",
    "  \n",
    "  \n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$J(q, d_1) = \\frac{|\\{hats\\}|}{|\\{dogs, with, hats, big, cats, wearing\\}} = \\frac{1}{6}$$\n",
    "\n",
    "\n",
    "$$J(q, d_2) = \\frac{|\\{hats\\}|}{|\\{dogs, with, hats, I, like\\}} = \\frac{1}{5}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Issues\n",
    "\n",
    "- **term frequency**:  documents with more occurrences of query term should be ranked higher than documents with few occurrences.\n",
    "  - q=\"dog\", score(\"dog cat dog\") > score(\"dog cat cat\")\n",
    "- **rare terms**: rare terms matter more than common terms\n",
    "  - q=\"the dog\", score(\"dog cat\") > score(\"the cat\")\n",
    "- **length normalization**: how can we make the scores comparable for long and short documents?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Term frequency\n",
    "\n",
    "- Let $tf_{t,d}$ be the frequency of term $t$ in document $d$.\n",
    "- What is the functional relationship between $tf_{t,d}$ and relevance?\n",
    "  - linear, log, something else?\n",
    "  \n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- $\\log$ typically used\n",
    "- The weight of term $t$ in $d$ is:\n",
    "  - $w_{t,d} = 1 + \\log tf_{t,d}$ if $tf_{t,d} > 0$; otherwise $0$.\n",
    "  \n",
    "  \n",
    "- score $s_i = \\sum_{t \\in q \\cap d_i}(1 + \\log tf_{t,d_i})$\n",
    "  - sum of term weights for unique terms in query and document\n",
    "  \n",
    "- If no query terms present? 0 score.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x10a7baac8>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEKCAYAAAAB0GKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFRJJREFUeJzt3X+QZWV95/H3tx1YMyBo3AQqgkPYnUlcVpmmjZiY7elN\n1FCJP2L8xZpdjdRa4G4iY7IustktZlNxDbGSQKqylG6cibIGEsCsGH9ALL2QmIDQ3IFJCzQJVaPi\nimtUIPvDyPR3/zhPz9zpdPc8F+/pe/ryflV1ce65597n+0zT99PnPKefJzITSZKOZWrcBUiSNgcD\nQ5JUxcCQJFUxMCRJVQwMSVIVA0OSVGXLuAsAiAjv7ZWkIWVmbGR7nTnDyEwuu+wyMnNivya5f/Zt\n835Ncv8muW/j0JnAkCR1m4EhSarSqcCYm5sbdwmtmuT+2bfNa5L7N8l9G4cY17Wwo4qIyC7UIUmb\nRUSQT9ZBb0lStxkYkqQqBoYkqYqBIUmqYmBIkqoYGJKkKgaGJKmKgSFJqmJgSJKqGBiSpCoGhiSp\nioEhSarSemBExKURsRAR90TEhyLi+LbblCSNXquBERHbgLcA05n5PJolYc9vs01JUjvaXtP7UeDv\ngBMiYgnYCny55TYlSS1o9QwjM78B/AbwBeAh4JuZ+ak225QktaPVM4yIOBN4O7ANeAS4PiLekJm/\nv/LYPXv2HN6em5tzpSxJGtDr9ej1emOtodUV9yLidcBLMvMt5fG/As7NzJ9fcZwr7knSECZxxb37\ngRdGxFMjIoAfB+5tuU1JUgvaHsO4G/ggMA/cDQTwvjbblCS1o9VLUtVFeElKkoYyiZekJEkTwsCQ\nJFUxMCRJVQwMSVIVA0OSVMXAkCRVMTAkSVUMDElSFQNDklTFwJAkVTEwJElVDAxJUhUDQ5JUpe01\nvSVpwy0tLdHv9wGYnp5masrfjUfBf0VJE6XfX2BmZjezsweZnT3IzMxu+v2FcZc1EVwPQ9LEWFpa\nYmZmN/v3X8GR34eX2LlzN/PzV0zUmYbrYUjSd6Df77O4OMfRH21TLC7uOnyJSk+cgSFJqmJgSJoY\n09PT7NjRA5YG9i6xY8ctTE9Pj6eoCeIYhqSJ0u8vcMEF72VxcRcA27f32LfvIqanzxpzZaM1jjEM\nA0PSxHky3FZrYEiSqniXlCSpswwMSVIVA0OSVMXAkCRVMTAkSVUMDElSFQNDklTFwJAkVTEwJElV\nDAxJUhUDQ5JUxcCQJFUxMCRJVVoPjIg4OSKui4h7I2IhIs5tu01J0uht2YA2rgQ+npmvjYgtwNYN\naHNdT4a58iVp1Fr9pIyIk4B/lpn7ADLz8cx8tM02j6XfX2BmZjezsweZnT3IzMxu+v2FcZYkSZtC\nqwsoRcTZwPuAzwNnA3cCF2fm/11x3IYsoLS0tMTMzG7277+CI1m5xM6du5mfv8IzDUmbxiQuoLQF\nOAf4ncw8B/g/wDtbbnNN/X6fxcU5ju72FIuLuw5fopIkra7tMYwvAV/MzDvL4+uBS1Y7cM+ePYe3\n5+bmmJuba7k0Sdo8er0evV5vrDW0vqZ3RNwCvCUzFyPiMmBrZl6y4hgvSUnSEMZxSWojAuNs4HeB\n44AHgTdn5iMrjtmQwIBm0PuCC97L4uIuALZv77Fv30VMT5+1Ie1L0ihMZGBUFbGBgQHeVitp8zMw\nJElVJvEuKUnShDAwJElVDAxJUhUDQ5JUxcCQJFUxMCRJVQwMSVIVA0OSVMXAkCRVMTAkSVUMDElS\nFQNDklTFwJAkVWl7xb2hOfW4JHVTpz6N+/0FZmZ2Mzt7kNnZg8zM7KbfXxh3WZIkOrQexqFDh1w+\nVZIqjWM9jHUvSUXEFuClwCxwBpDAQeBPgZsy8/FRFdLv91lcnOPok54pFhd30e/3mZmZGVVTkqQn\nYM1f2yPiPwF3Ai8D7gf2AR8o2y8D7oiI/7gRRUqSxm+9M4y7gV9dY+3UfRERwMtHVcj09DQ7dnyA\n/ft/msFLUjt23ML09KtG1Ywk6Qlab2DgtZmZEXHxak9m48aRFTI1xd69F7Jz5262br2BrVtv4Oyz\nL2bv3gsdv5CkDlhz0DsiPg+8GPgEMAccNbiSmV8fWRERh09kvK1Wko5tHIPe6wXG24C3AmcCD3F0\nYGRmnjmyIgYCQ5J0bJ0KjMMHRFyVmW9ttQgDQ5KGMo7AqLnec+LKHRFxdQu1SJI6rCYwzhp8UP42\nwz+KkKQnmfX+DuPSiHgMeF5EPFq+HgMeBj6yYRVKkjqhZgzj3Zl5aatFOIYhSUPp1KB3RJyZmQ+u\n++KKY6qKMDAkaShdC4xrgROAj9JMEfI/aW6tPRV4PvBK4NHM/BffcREGhiQNpVOBARAR/wg4H/hR\nYFvZfRD4M+CaUZxdlHYMDEkaQucCY8OKMDAkaSidm94cICJ+ZpXdjwAHMvOroy9JktRFNXdJfQz4\nYeAzZdccMA98P/Armfkd/xGfZxiSNJxOnmGUY56TmQ8DRMQpwAeBc4FbAf/qW5KeBGr+0vv05bAo\nvlr2fR34djtlSZK6puYMoxcRfwxcVx6/puw7AfhmTSMRMUVza+6XMvMVqx3jtOaS1G01YxgB/AzN\nrbUAnwVuGGbQISLeTjP/1EmrBUZE5M6dv1DW9IYdO3rs3Xsh09NnrTxUkkSHb6st4xYvABL43DB3\nR0XEaTTrgb8L+MW1AgMOMbg0686du5mfv8IzDUlaRSenN4+I1wGfo7kU9Trg9oh4zRBt/BbwDpqw\nqSxlisXFXYcvUUmSxq9mDOOXgR9aPquIiO8BPgVcf6wXRsRPAQ9n5v6ImGPFMq9H2zOwPVdRliQ9\nefR6PXq93lhrqBnDOJCZzx14PAXcPbhvndf+F+BfAo8D3wU8DfhwZr5xxXFekpKkIXRyDCMi3gM8\nD7im7Ho9cE9mXjJUQxG7gF9af9B7FwDbt/fYt+8iB70laQ2dDAyAiHg18KLy8E8z84+GbugYgXHo\n0CFvq5WkSp0NjNaLcGoQSRpKp6YGKcuxrvYpHkBm5kmtVSVJ6pw1AyMzn7aRhUiSus2BAklSFQND\nklTFwJAkVTEwJElVqgMjIm5osxBJUrcNc4ZxZmtVSJI6b93JByPi2cubwHERcXrZJjO/0HJtkqQO\nWfcvvSPiMzR/vBfA84E7OPKHez82siL8S29JGkqnpwaJiH5mTrdShIEhSUPp5AJKkiTBcIFxZWtV\nSJI6z9lqJWkT6tRstRttfn4ecC0MSeqqznwyz84eZHb2IDMzu+n3F8ZdjiRphc5ckjqy9IbreUvS\nsWyau6Qi4s2jLuSIKRYXdx1erlWS1A1P9Ff4/zzSKiRJnbfeEq33rPUUcEo75QAssWPHLUxPv6q9\nJiRJQ1vvLqlTgJ8AvrFifwB/PupCtm5tJsPdvr3H3r0XOX4hSR2zXmD8MXBiZu5f+URE9EZdyK23\nngHA9PSVhoUkdVBn7pLqQh2StFlsmrukJElPPgaGJKmKgSFJqmJgSJKqGBiSpCoGhiSpioEhSapi\nYEiSqhgYkqQqBoYkqYqBIUmqYmBIkqq0GhgRcVpEfDoiFiLiQES8rc32JEntaXW22og4FTg1M/dH\nxInAPPDKzLxvxXHOVitJQ5i42Woz8yvL62lk5t8C9wLParNNSVI7NmwMIyLOAHYCt29Um5Kk0dmQ\nwCiXo64HLi5nGpKkTWa9JVpHIiK20ITF1Zn5kbWO27Nnz+Htubk55ubm2i5NkjaNXq9Hr9cbaw2t\nL9EaER8EvpaZv7jOMQ56S9IQxjHo3fZdUi8CbgUOAFm+/kNmfnLFcQaGJA1h4gKjuggDQ5KGMnG3\n1UqSJoeBIUmqYmBIkqoYGJKkKgaGJKmKgSFJqmJgSJKqGBiSpCoGhiSpioEhSapiYEiSqhgYkqQq\nBoYkqYqBIUmq0vqKe7Xm5+cBmJ6eZmrKHJOkrunMJ/Ps7EFmZw8yM7Obfn9h3OVIklbozAJKzWJ8\nAEvs3Lmb+fkrPNOQpDW4gBIAUywu7qLf74+7EEnSgA4GhiSpizoYGEvs2HEL09PT4y5EkjSgM3dJ\nbd16AwDbt/fYu/cixy8kqWM6M+h95513At5WK0k1xjHo3ZnA6EIdkrRZeJeUJKmzDAxJUhUDQ5JU\nxcCQJFUxMCRJVQwMSVIVA0OSVMXAkCRVMTAkSVUMDElSFQNDklTFwJAkVTEwJElVWg+MiDgvIu6L\niMWIuKTt9iRJ7Wh1evOImAIWgR8HvgzcAZyfmfetOM7pzSVpCJM4vfkLgAcy82Bmfhu4Fnhly21K\nklrQdmA8C/jiwOMvlX2SpE2mM2t679mz5/D23Nwcc3NzY6tFkrqm1+vR6/XGWkPbYxgvBPZk5nnl\n8TuBzMzLVxznGIYkDWESxzDuAP5xRGyLiOOB84EbW25TktSCVi9JZeahiPh54GaacHp/Zt7bZpuS\npHa0ekmquggvSUnSUCbxkpQkaUIYGJKkKgaGJKmKgSFJqmJgSJKqGBiSpCoGhiSpioEhSapiYEiS\nqhgYkqQqBoYkqYqBIUmqYmBIkqoYGJKkKp0KjHEvP9i2Se6ffdu8Jrl/k9y3cTAwNtAk98++bV6T\n3L9J7ts4dCowJEndZWBIkqp0ZonWcdcgSZvNRi/R2onAkCR1n5ekJElVDAxJUpXOBEZEnBcR90XE\nYkRcMu56lkXEaRHx6YhYiIgDEfG2sv8ZEXFzRNwfETdFxMkDr7k0Ih6IiHsj4qUD+8+JiHtKH68Y\n2H98RFxbXvMXEfHsgefeVI6/PyLe2FIfpyLiroi4cQL7dnJEXFfqXYiIcyelf6XWhVLXh0otm7Zv\nEfH+iHg4Iu4Z2DfW/kTEGRFxW3numojYMsK+/XqpfX9E3BARJ3W+b5k59i+a4PorYBtwHLAf+MFx\n11VqOxXYWbZPBO4HfhC4HPj3Zf8lwK+V7X8C9IEtwBmlX8tjRbcDP1S2Pw78RNl+K/Bfy/brgWvL\n9jOAvwZOBp6+vN1CH98O/HfgxvJ4kvr2e8Cby/aW0t6m71/5WXkQOL48/gPgTZu5b8CPAjuBewb2\njbU/5d/1tWX7KuDCEfbtxcBU2f414N1d79vYP5BLsS8EPjHw+J3AJeOua41a/0f5Rt8HnFL2nQrc\nt1rtwCeAc8sxnx/Yfz5wVdn+JHBu2X4K8NWVxwx8U18/4v6cBvwJMMeRwJiUvp0E/PUq+zd9/8oH\nwX3lv1uAGyfh/0uaIBz8UB1rf4D/xZEP9RcCnxxV31Y899PA1V3vW1cuST0L+OLA4y+VfZ0SEWfQ\n/JZwG83/xA8DZOZXgO8th63sy0Nl37No+rVssI+HX5OZh4BHIuK713mvUfot4B1ADuyblL59P/C1\niNgXzSW390XEViagf5n5DeA3gC+U934kMz81CX1b4XvH1Z+IeCbwjcxcGniv7xtRv1a6gOaM4ag6\nB+uhA33rSmB0XkScCFwPXJyZf8vRH7Cs8vg7am6E77V2IxE/BTycmfuP0eam61uxBTgH+J3MPAf4\n3zS/vU3C9+5MmkuJ22h+0E+IiJ9lAvp2DBvdn9b7HBG/DHw7M68Z5duO6JijdCUwHgKePfD4tLKv\nE8pg0PU0p4wfKbsfjohTyvOnAl8t+x8CTh94+XJf1tp/1Gsi4inASZn5ddr/d3kR8IqIeBC4Bvix\niLga+MoE9A2a35q+mJl3lsc30ATIJHzvng98NjO/Xn6j/CPgR5iMvg0aW38y82+AkyNiapX3GomI\n+DngJ4E3DOzubt9Gcd1xBNctn8KRQe/jaQa9nzPuugbq+yDwmyv2XU65zsjqg3HH01wSGRywug14\nAU2yfxw4r+z/NxwZsDqf1Qeslref3lIfd3FkDOPXJ6VvwC3AjrJ9Wfm+bfrvHXA2cAB4aqnp94B/\nu9n7RjPIe6ArP2c0A8PL1/yvAi4aYd/OAxaAZ644rrN9G9uH8Cr/mOfR3IH0APDOcdczUNeLgEM0\nIdYH7iq1fjfwqVLzzYM/MMCl5Zt8L/DSgf0zND/kDwBXDuz/B8Aflv23AWcMPPdzZf8i8MYW+zkY\nGBPTN5oP1jvK9+/D5QdnIvpHM/a0ANwDfIDmDsNN2zfg94EvA9+iGZt5M82H3Nj6Q/OBfXvZ/wfA\ncSPs2wPAQZrPlLsoH/hd7ptTg0iSqnRlDEOS1HEGhiSpioEhSapiYEiSqhgYkqQqBoYkqYqBobGL\nZgryt25ge++JZqr6yzeqTWkS+HcYGrsyqeNHM/O5Q74u8gn8DxwR3wSesfK1EfGUbKbZkLQKzzDU\nBe8Gziwzyl4OEBH/LiI+VxaXuazs2xbNIlsfiIgDwOkR8VhZiOYvy0I750ZELyL+KiJetrKhiPgI\nzbom8xHx2jKT7VURcRtweURsLYvd3BYR8xHxivK6p5ZFZhYi4sPl+XPKc48NvP+rI2Jf2f6HEXF9\nRNxevn647L+stPGZUucvDLz+jRFxd0T0Sz9PjIgHy/xARMTTBh9LG6qN6Rj88muYL/7+GggvAd5b\ntgP4KM0CNNuAxykLyJTnlyhTJ9BM/XETzS9CzwP6a7T36MD2PsqUKOXxu4A3lO2Taaak+C6amWF/\nt+x/LvB3wDmrvN+rgb1l+0PAj5Tt0ylrGdDMafVnNLPpPhP4Gs18amdR1rgoxy3P+fN+4BVl+y3A\ne8b9PfPryfn1hJYblFr2UuAlEXEXTWCcAGynmdf/YGbeMXDstzLz5rJ9APh/mblUzkC2VbZ33Yq2\nXx4R7yiPj6eZ7XMWuBIgMw/EwFKbrD1N9IuB50TE8vMnlvU4AD6WmY8DfxMRDwOnAP8cuC6btS7I\nzG+WY99PM2/UjTRzEP3ryn5JI2VgqIuCZrnK/3bUzohtNGtaDPr2wPYSzeRuZGZWrVHcWPmer87M\nB1a0vVqNywbHQp664phzM3OwxuX3+tbArkMc+Vn8ew1l5p+X9Zd30ayQ9vk1+iG1yjEMdcFjwNMG\nHt8EXBARJwBExPdFxPeU51Z+oK63CMxaz633mpuAtx0+MGJn2bwV+Nmy75/SXPJa9pWI+IGytsCr\nBvbfDFw88F5nH6OeTwOvKSulERHPGDjmapoZT/euU7vUKgNDY5fNQi+fjYh7IuLyzPwTmgWd/qJc\n+rmOZqAahltRbq3ncp1jfhU4rtRyAPiVsv8qmktKC8Ae4M6B11wKfIxmXOLLA/svBp5fBrH/Erhw\nvXrKmcO7gFsiok+zBOuyDwFPB65d4z2k1nlbrfQERMRngF/KzLs2qL3XAC/PzDdtRHvSahzDkJ6Y\nDftNKyJ+m2bRrp/cqDal1XiGIUmq4hiGJKmKgSFJqmJgSJKqGBiSpCoGhiSpioEhSary/wFlB41r\nM9uUpgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x108e3aa20>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%pylab inline\n",
    "tfs = [1, 100, 1000, 10000, 100000, 1000000]\n",
    "xlim((-10000,1.2e6))\n",
    "ylim((-1, 9))\n",
    "plot(tfs, [1 + math.log10(tf) for tf in tfs], 'bo')\n",
    "xlabel('term frequency')\n",
    "ylabel('1 + log(tf)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Document frequency\n",
    "\n",
    "- We want to weight rare terms more than common terms (e.g., *the* versus *excogitate*).\n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **idf weight:**\n",
    "  - Let $N$ be total number of documents\n",
    "  - Let $df_t$ be the total number of documents that term $t$ appears in.\n",
    "  - Define **inverse document freqency** as $idf_t = \\log (\\frac{N}{df_t})$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x10aa1dc88>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEOCAYAAACHE9xHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAE0FJREFUeJzt3X2QXXV9x/H3d42g4UnUUUappDiJD1TNZlFUdLOtbUVU\npM74UK1PaR2gVYk4FB9mSqYzba0dNfxhqU4NCIJPID6MSq2113QcERNuBCOwKE4UKaiVUgRHMPvt\nH+dscrPc3b03e/fu2d++XzN3OPfcc/d8ctn95OR3zp5fZCaSpHKNLHUASdLisuglqXAWvSQVzqKX\npMJZ9JJUOItekgq3aqkDdBMRXvMpSX3KzOi2vrFH9JnJ+eefT2Yum4d5zbzc8y7HzOatHnNpbNFL\nkgbDopekwjW66CcmJpY6Ql/Mu/iWW+bllheWX2bzzi/mG9tZChGRTcwlSU0VEeRyOxkrSRoMi16S\nCmfRS1LhLHpJKpxFL0mFs+glqXAWvSQVzqKXpMJZ9JJUOItekgpn0UtS4Sx6SSrc0Is+It4VEbsj\n4vqIuCwiDhl2BklaSYZa9BFxHPBmYDQzn041leGrh5lBklaaYc8Z+3/A/cBhETEFrAZuH3IGSVpR\nhnpEn5l3Ae8Hfgz8FPjfzPzaMDNI0koz1CP6iDgeeDtwHHA3cEVEvCYzL5+57ZYtW/YtT0xMLLtZ\nZCRpMbVaLVqtVk/bDnWGqYh4JfBHmfnm+vnrgJMy8y0ztnOGKUnqQ5NmmLoZeHZEPCwiAngBcOOQ\nM0jSijLsMfrvApcAO4HvAgF8ZJgZJGmlcXJwSSpAk4ZuJElDZtFLUuEsekkqnEUvSYWz6CWpcBa9\nJBXOopekwln0klQ4i16SCmfRS1LhLHpJKpxFL0mFs+glqXAWvSQVzqKXpMJZ9JJUOItekgpn0UtS\n4Sx6SSqcRS9JhbPoJalwFr0kFc6il6TCWfSSVDiLXpIKZ9FLUuEsekkqnEUvSYWz6CWpcBa9JBXO\nopekwln0klQ4i16SCmfRS1LhLHpJKtzQiz4ijoqIz0TEjRGxOyJOGnYGSVpJVi3BPi8AvpyZr4iI\nVcDqbhvt3LkTgNHRUUZG/IeHJB2syMzh7SziSKCdmU+cZ7tcvfpKANata7Ft2xmMjp4whISStDxF\nBJkZXV8bctE/A/gI8H3gGcAO4OzM/PWM7RKmc02xfv1mdu7c6pG9JM1irqIfdnOuAjYAH8rMDcB9\nwDvnfssIk5Mbabfbi59Okgo07DH624CfZOaO+vkVwHndN92yb2nv3gDWLGowSVpOWq0WrVarp22H\nOnQDEBHfAN6cmZMRcT6wOjPPm7GNQzeS1Ie5hm6W4qqbtwGXRcRDgVuBN3XbaPpk7Nq1LbZtO9OS\nl6SDNPQj+l5ERO7YUY3ueHmlJM2vMVfd9Coisom5JKmpmnTVjSRpyCx6SSqcRS9JhbPoJalwFr0k\nFc6il6TCWfSSVDiLXpIKZ9FLUuEsekkqnEUvSYWz6CWpcBa9JBXOopekwln0klQ4i16SCmfRS1Lh\n+i76iHhYRBy6GGEkSYM37+TgERHA6cBrgOcCAYxExG+BbwGXA59z7j9JaqZ554yNiO3AduCLQDsz\n76/XHwKMAqcBz8vMjQML5ZyxktSXBU0OHhGHTJf7Qrbph0UvSf2Zq+jnHbqZrcAj4kjgXmBkkCUv\nSRqseYt+poh4FnAqkMDFwLHANwcbS5I0KPMO3cz6xogXAY8GDs3Mfx1oKIduJKkvCx2jvzQzXxcR\nZ2fmBYuS8MH7tOglqQ9zFX0v19GPRcTjgE0RcXREPLLzMdiokqRB62WM/l+A/wCOB3ZSXUc/Lev1\nkqSG6nmMPiIuzMyzFjnP9L4cupGkPixojH4pWPSS1J8FXUcfEefM9XpmfuBgg0mSFl8vY/RH1P99\nEvBM4Av185cC1y5GKEnS4PQzRr8deHFm3lM/PwL4UmaODzyUQzeS1JeFXl457bFA560O7q/XSZIa\nrJ9bIFwCXBsRV9XPT6e6BYIkqcF6+c3Y5wDXZGZGxAbg+fVL2zOzvSihHLqRpL4s9BYIFwInAZPA\n1cDVmXnHAgONADuA2zLztC6v544dOwAYHR1lZMQZDyVpLgO5jj4ingy8CHghcBTwn1TF/83M3Ntn\noLcDY8CRsxX96tVXArBuXYtt285gdPSEfnYhSSvKwH9hKiIeDvw+VfE/JzNP7OO9xwIXAX8HnDNb\n0Vd3VwCYYv36zezcudUje0maxUJ/Yepv5nj55/2UfO2DwLlU/yrowQiTkxtpt9uMjY31uStJUi9X\n3dzbZd1hwJ8DjwL+ttedRcSLgTszc1dETHDgDdJm2LJvae/eANb0uhtJKl6r1aLVavW0bV9DN/Uv\nSZ1NVfKfBt6fmT/r4/1/D/wZ8Fvg4VS/dfvZzHz9jO0cupGkPix4jL6+7/w5wGuBjwEXZOZdCwy1\nEXjH7CdjrwBg7doWF110pidjJWkOCx2j/yfg5cBHgKdl5q8GnK+r7dvXADA6eoFH8pK0AL1cRz8F\n/IZquKVz4wAyM48ceCh/YUqS+rKgI/rM9HBakpaxeUs8Iua4Mqb3bSRJS6OXo/VWRJwbEetmvhAR\n6yLiPKA18GSSpIHoZYz+EKqrbV4DnADcQzU+fzjwPeAy4BOZef+sX6TfUI7RS1JfBnYLhPpmZI+u\nn/4iM6cGkK/bfix6SeqDk4NLUuEWdNVNxxfpNkn43cDOzNx1sOEkSYurn9sUXw6cCHyxXvUS4Hqq\nm9B8JjPfN7BQHtFLUl8GdT/67cCp078ZGxGHA18CTqE6qn/qgPJa9JLUp0FNDv4Yqt+QnfYA8NjM\n/PWM9ZKkBulncvDLgG9HxOepLq98CXB5RBwGfH8xwkmSFq7fyytPBE6un34zM3csSiiHbiSpLwO5\n6qb2ADBFdXOzBxYaTJK0+Hoeo4+Is6mGbx5NNV7/8Yh462IFkyQNRj9X3VxPNRH4vfXzw4BvZebT\nBx4qInfsqEaFRkdHvR+9JM1jUFfdBLC34/le5pzzdWHGx/cwPr6HsbHNtNu7F2s3klS8fo7ozwHe\nAFxFVfCnAxdn5gcHHso5YyWpL4O8qdkG9l9181+LdeuDA4seVq++ku3b1zA2NrYYu5OkZW+hc8be\nw4OnEJx+bVGmEpQkDU4vUwkeMYwgs5ti3bpvMDr6J0sbQ5KWqX6vox+a1auvBGDt2hbbtp3p+Lwk\nHaTG3o/eyyslqXdOPCJJhRvUdfSSpGXIopekwln0klQ4i16SCmfRS1LhLHpJKpxFL0mFs+glqXAW\nvSQVzqKXpMJZ9JJUOItekgo31KKPiGMj4usRsTsiboiItw1z/5K0Eg317pURcQxwTGbuiojDgZ3A\nyzLzphnbZWYyNTVFu90GvF2xJM2lMXevzMw7pueZzcxfATcCj++2bbu9m7GxzYyP72F8fA9jY5tp\nt3cPM64kFWHJ7kcfEWuAFvB7del3vpbr17+VXbu2sv/voinWr9/Mzp1bPbKXpBkac0Q/rR62uQI4\ne2bJT5ucnODAeCNMTm7cN5QjSerN0OeMjYhVVCV/aWZ+frbt7r//U8D19bOJ+iFJAmi1WrRarZ62\nHfrQTURcAvwiM8+ZYxuHbiSpD42ZMzYiTga2AzcAWT/enZlXz9gur7vue2za9GEmJzcCsHZti4su\nOpPR0ROGlleSlovGFH2vvLxSkvqzbItektSbxl11I0kaHotekgpn0UtS4Sx6SSqcRS9JhbPoJalw\nFr0kFc6il6TCWfSSVDiLXpIKZ9FLUuEsekkqnEUvSYUb+gxTS8nbHktaiVZM07Xbuxkb28z4+B7G\nx/cwNraZdnv3UseSpEW3Iu5HPzU1xdjYZqcmlFSsFX8/+na7zeTkBAf+cUeYnNy4byhHkkq1Iope\nklayFVH0o6OjrFvXAqY61k6xbt03GB0dXZpQkjQkK2KMHqqTsZs2fZjJyY0ArF3b4qKLzmR09ISB\n7keSloKTg9e8vFJSqSx6SSrcir/qRpJWMotekgpn0UtS4Sx6SSqcRS9JhbPoJalwFr0kFc6il6TC\nWfSSVDiLXpIKZ9FLUuEsekkqnEUvSYUbetFHxCkRcVNETEbEecPevyStNEO9TXFEjACTwAuA24Hv\nAK/OzJtmbOdtiqUhc76G5a1Jtyl+FnBLZu7JzAeATwIvG3IGSTO027sZG9vM+Pgexsf3MDa2mXZ7\n91LH0oAMu+gfD/yk4/lt9TpJS2RqaopNmz7Mrl1bue++l3PffS9n166tbNr0Yaampub/Amo8/20m\nrXDtdpvJyQkOrIMRJic37hvK0fK2asj7+ynwhI7nx9brHmTLli37licmJpiYmFjMXJK0rLRaLVqt\nVk/bDvtk7EOAm6lOxv43cC3wp5l544ztPBkrDcnU1BRjY5vZtWsr+4/qp1i/fjM7d271pOwyMdfJ\n2KEe0Wfm3oh4C/BVqu+oj84seUnDNTIywrZtZ7Bp02YmJzcCsHZti23bzrTkCzHUI/peeUQvDZ+X\nVy5vcx3RW/SSVIAmXUcvSRoyi16SCmfRS1LhLHpJKpxFL0mFs+glqXAWvSQVzqKXpMJZ9JJUOIte\nkgpn0UtS4Sx6SSqcRS9JhWt00fc6e0pTmHfxLbfMyy0vLL/M5p2fRT9A5l18yy3zcssLyy+zeefX\n6KKXJC2cRS9JhWvsDFNLnUGSlptlNZWgJGlwHLqRpMJZ9JJUusxs3AM4BbgJmATOG8L+jgW+DuwG\nbgDeVq8/GvgqcDPwb8BRHe95F3ALcCPwxx3rNwDX19m3dqw/BPhk/Z5vAU/oeO0N9fY3A6/vI/cI\ncB3whabnBY4CPlPvfzdwUpPzdmTYXe/vsnofjckMfBS4E7i+Y92S5gPWANfUr30CWNVD5vfVmXYB\nVwJHNiVzt7wdr70DmAIe2ZS8s34v9/pNP6wHVXn9ADgOeGj9P//Ji7zPY4D19fLh9Qf7ZOAfgb+u\n158HvLdefirQBlbVH/oP2H++49vAM+vlLwMvrJfPAv65Xn4V8MmOH8wfUhXhI6aXe8z9duDj7C/6\nxuYFLgbeVC+vqt/f5LzHAbcCh9TPP0X1g9eYzMDzgPUcWJpLmq/+nF5RL18InNFD5j8ERurl9wL/\n0JTM3fLW648FrgZ+RF30wFOWOu+s38+LUZwLeQDPBr7S8fydDOGofkaGz9XffDcBj63XHQPc1C0T\n8BWqI9RjgO93rH81cGG9fDVwUr38EOBnM7fp+B/3qh4yHgv8OzDB/qJvZF7gSOCHXdY3Mm+93dF1\nvqOpfnC/0MTvCaq/kDpLc0nzAT9nf2k/G7h6vswzXjsduLRJmbvlpfrX6dM4sOgbkbfbo4lj9I8H\nftLx/LZ63VBExBqqv8GvofqBuRMgM+8AHjNLxp/W6x5PlXdaZ/Z978nMvcDdEfHIOb7WfD4InAtk\nx7qm5v1d4BcRcVFEXBcRH4mI1Q3OS2beBbwf+HH9nrsz82tNzlx7zFLli4hHAXdl5lTH13pcj7mn\nbaI64m1s5og4DfhJZt4w46VG5gVPxh4gIg4HrgDOzsxfcWCJ0uX5gnZ30G+MeDFwZ2bumufrNCIv\n1RHxBuBDmbkBuJfq6KeRny9ARBxPNTR2HNUP0mER8VoanHkWw863kO/r9wAPZOYnDvZrHGSenjNH\nxMOBdwPnH3SihWfp+zNuYtH/FHhCx/Nj63WLKiJWUZX8pZn5+Xr1nRHx2Pr1Y4CfdWT8nS4ZZ1t/\nwHsi4iFUJ5x+ycH9eU8GTouIW6lOxvxBRFwK3NHQvLdRHQHtqJ9fSVX8Tf18AU4EvpmZv6yPtK4C\nntvwzCxlvsz8H+CoiBjp8rXmFBFvBE4FXtOxuomZn0g1/v7diPhRvf11EfGY2faxxHkr843tDPtB\nNU41fTL2EKqTsU8Zwn4vAT4wY90/Uo+50f3E1iFUwxKdJ12uAZ5F9bful4FT6vV/yf6TLq+m+0mX\n6eVH9JF7I/vH6N/X1LzAN4B19fL59Wfb2M8XeAbVFVgPq/d1MfBXTctMVTo3NOV7lupE4fRY8oXA\nmT1kPoXq6qZHzdiuEZln5p3x2o+Ao5uUt2vOQRfmIB71//ibqS45eucQ9ncysJfqL5U21SWLpwCP\nBL5WZ/kqHT9sVJdR/YAHX0Y1RlUQtwAXdKw/FPh0vf4aYE3Ha2+s10/Sx+V/9Xs7i76xeamK8zv1\nZ/zZ+hu4sXnr953L/ssrP0Z1FVhjMgOXA7cDv6E6l/AmqlJYsnxUBfftev2ngIf2kPkWYA/Vz911\n1MXXhMzd8s7489zKgy+vXNLPuNvDWyBIUuGaOEYvSRogi16SCmfRS1LhLHpJKpxFL0mFs+glqXAW\nvSQVzqKXpMJZ9FIPIuL8iDinXn5bRHy/vr+Q1HirljqAtAydBbwgM29f6iBSLzyil2YREe+JiJsj\nYjvwpHrdhcDxwFci4uwlDSj1yCN6qYuI2AC8Eng61d0IrwN2ZOZZEXEKMJHV5CRS43lEL3X3fOCq\nzPxNZt5DNZVgp32TP0TEs4aaTOqTRS8t3KlLHUCai0UvdbcdOD0iDo2II4CXdtsoIl4EZEQcOdR0\nUh8co5e6yMx2RHyKatKRO4FrO1/uWH401exT9w4vndQfJx6RFiAi/oJqNqFrM/OBpc4jdWPRS1Lh\nHKOXpMJZ9JJUOItekgpn0UtS4Sx6SSqcRS9JhbPoJalwFr0kFc6il6TC/T/HR208eX/+wgAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10a9ca780>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%pylab inline\n",
    "N = 1e6\n",
    "dfs = [1, 100, 1000, 10000, 100000, 1000000]\n",
    "xlim((-10000,1.5e6))\n",
    "ylim((-1, 9))\n",
    "plot(dfs, [math.log10(1. * N/df) for df in dfs], 'bo')\n",
    "xlabel('df$_t$')\n",
    "ylabel('log(N/df$_t$)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "What effect does idf weight have on one term queries?\n",
    "<br><br><br>\n",
    "<br><br><br><br>  \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **none**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# tf-idf weight\n",
    "\n",
    "- Multiply $tf$ and $idf$\n",
    "\n",
    "\n",
    "- $w_{t,d} = (1 + \\log tf_{t,d}) \\times \\log (\\frac{N}{df_t})$\n",
    "\n",
    "- Perhaps the most common baseline weighting system in IR\n",
    "- Increases with term frequency in document; increases with rarity of term\n",
    "\n",
    "**Final score:**\n",
    "\n",
    "$$s_i = \\sum_{t \\in q \\cap d_i} w_{t, d}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Document representation\n",
    "\n",
    "Previously, each document is a \"bag-of-words\"\n",
    "\n",
    "- \"I saw the man in the hat in the park.\" $\\rightarrow$ `{i: 1, saw: 1, the: 3, man: 1, in: 2, hat: 1, park: 1}`\n",
    "  - order doesn't matter; just term frequency\n",
    "  \n",
    "Now, each term is weighted by *tf-idf*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Document Space\n",
    "\n",
    "Each document $d_i$ is now a vector in $V$-dimensional space, where $V$ is the number of terms.\n",
    "\n",
    "- e.g., assume position0 is \"dog\" and position1 is \"cat\", then two documents could be:\n",
    "  - $d_1 = \\{10.6, 0.5\\}$ (mostly about dogs)\n",
    "  - $d_2 = \\{0.2, 5.6\\}$  (mostly about cats)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x10aa43f60>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEPCAYAAABBUX+lAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAD1hJREFUeJzt3X2sZHddx/H3Z1keSksbQKBKpTzYRW2E3q7lqWE7SAyE\nKmAiBnzgYQ0JUdMuVSyUmF7/kCiG0EZNhFA3QIqSVohgABdspgjyuMzSsi0sCWEpUFYIhQpNsOz9\n+sdMt9vbvbsze++ZmTu/9yu52TPnnjm/78nufuY3v3PO76SqkCQtvi2zLkCSNB0GviQ1wsCXpEYY\n+JLUCANfkhph4EtSIzoP/CRnJLkuya1J9id5etdtSpLub+sU2rga+FBVvSTJVuChU2hTkrRKurzx\nKsnpwKCqntRZI5KksXQ9pPME4HtJdif5QpK3Jzml4zYlScfQdeBvBc4H/qGqzgfuAl7fcZuSpGPo\negz/m8BtVfX50evrgctXb5TECX0kaUJVlUm277SHX1WHgNuSbButei5wyxrbLuTPlVdeOfMaPD6P\nz+NbvJ+TMY2rdC4Brk3yQOBrwKum0KYkaZXOA7+qvghc0HU7kqTj807bjvV6vVmX0CmPb3Pz+NrS\n6XX4YxeR1DzUIUmbRRJqnk7aSpLmh4EvSY0w8CWpEQa+JDViGtfhT2RlZYXBYADA0tISW7b4mSRJ\nG2Gu0nQw2M/27bvYseMgO3YcZPv2XQwG+2ddliQthLm5LPPw4cNs376Lffuu4t7PoRXOO28Xe/de\nZU9fko6yqS/LHAwGHDjQ474lbeHAgYuODPFIkk7e3AS+JKlbcxP4S0tLbNvWB1aOWrvCtm03srS0\nNJuiJGmBzM0YflUxGOxn5863ceDARQCcc06f3btfw9LSuTOuUJLmy8mM4c9V4IOXZUrSOBYi8CVJ\nJ7apr9KRJHXLwJekRhj4ktQIA1+SGmHgS1IjDHxJaoSBL0mNMPAlqREGviQ1wsCXpEYY+JLUCANf\nkhph4EtSI7Z23UCSrwM/ZPhkk7ur6mldtylJur/OA59h0Peq6o4ptCVJWsM0hnQypXYkSccxjSAu\n4KNJPpfk1VNoT5J0DNMY0rmwqm5P8iiGwX9rVX1iCu1Kko7SeeBX1e2jP7+b5P3A04D7Bf7y8vKR\n5V6vR6/X67o0Sdo0+v0+/X5/Xfvo9Jm2SR4KbKmqHyU5FdgD/GVV7Vm1nc+0laQJnMwzbbvu4T8G\neH+SGrV17eqwlyRNR6c9/LGLsIcvSRM5mR6+l0tKUiMMfElqhIEvSY0w8CWpEQa+JDXCwJekRhj4\nktQIA1+SGmHgS1IjDHxJaoSBL0mNMPAlqREGviQ1wsCXpEYY+JLUCANfkhph4EtSIwx8SWqEgS9J\njTDwJakRBr4kNcLAl6RGGPiS1AgDX5IaYeBLUiMMfElqhIEvSY0w8CWpEQa+JDViKoGfZEuSLyT5\nwDTakyTd37R6+JcCt0ypLUnSMXQe+EnOAl4AvKPrtiRJa5tGD/+twOuAmkJbkqQ1bO1y50kuBg5V\n1b4kPSBrbbu8vHxkudfr0ev1uixNkjaVfr9Pv99f1z5S1V3HO8mbgN8HfgqcAjwMeF9VvXzVdtVl\nHZK0aJJQVWt2oo/5nmkFbZKLgD+tqhce43cGviRN4GQC3+vwJakRU+vhH7cIe/iSNBF7+JKkNRn4\nktQIA1+SGmHgS1IjDHxJaoSBL0mNMPAlqREGviQ1wsCXpEYY+JLUCANfkhph4EtSIwx8SWqEgS9J\njTDwJakRBr4kNcLAl6RGGPiS1AgDX5IaYeBLUiMMfElqhIEvSY0w8CWpEQa+JDXCwJekRhj4ktQI\nA1+SGjFW4Ce5cJx1kqT5NW4P/+/GXHcfSR6c5DNJBkn2J3nTZOVJkjbK1uP9MskzgWcBj0py2VG/\nOh14wIl2XlU/SfKcqroryQOATya5sKo+ua6qJUkTO27gAw8CThtt97Cj1t8J/PY4DVTVXaPFBzP8\nRnHHhDVKkjZAqurEGyVnV9XBk2og2QLsBZ4E/GNV/fkxtqlx6pAkDSWhqjLJe07Uw7/HXUn+FjgX\neMg9K6vq1070xqpaAZaSnA7sSXJRVd24ervl5eUjy71ej16vN2ZpkrT4+v0+/X5/XfsYt4e/B3gv\n8GfAa4BXAN+tqssnaiz5C+CuqnrLqvX28CVpAifTwx/3Kp1HVtU1wN1VdWNV7QRO2LtP8jNJzhgt\nnwL8OrBvkgIlSRtj3CGdu0d/3p7kYuDbwCPGeN/PAu9MEoYfLu+uqv+cvExJ0nqNO6TzG8B/AT/P\n8Pr704HlqvrghhThkI4kTaTLIZ2XMPxw+FJVPYfh0MxvTVqgJGl2xg38p1TVD+55UVXfB5a6KUmS\n1IVxA39Lkoff8yLJIxh//F+SNAfGDe23AJ9Kct3o9UuAv+qmJElSF8Y6aQuQ5Je591LMG6rqlg0r\nwpO2kjSRkzlpO3bgd8nAl6TJdHmVjiRpkzPwJakRBr4kNcLAl6RGGPiS1AgDX5IaYeBLUiMMfElq\nhIEvSY0w8CWpEQa+JDXCwJekRhj4ktQIA1+SGmHgS1IjDHxJaoSBL0mNMPAlqREGviQ1wsCXpEYY\n+JLUCANfkhrRaeAnOSvJDUn2J7k5ySVdtidJWluqqrudJ2cCZ1bVviSnAXuBF1XVl1dtV13WIUmL\nJglVlUne02kPv6q+U1X7Rss/Am4FHttlm5KkY5vaGH6SxwPnAZ+ZVpuSpHttnUYjo+Gc64FLRz39\n+1leXj6y3Ov16PV60yhNkjaFfr9Pv99f1z46HcMHSLIV+Hfgw1V19RrbOIYvSRM4mTH8aQT+u4Dv\nVdVlx9nGwJekCcxd4Ce5EPg4cDNQo58rquojq7Yz8CVpAnMX+GMXYeBL0kTm7rJMSdL8MPAlqREG\nviQ1wsCXpEYY+JLUCANfkhph4EtSIwx8SWqEgS9JjTDwJakRBr4kNcLAl6RGGPiS1AgDX5IaYeBL\nUiMMfElqhIEvSY0w8CWpEQa+JDXCwJekRhj4ktQIA1+SGmHgS1IjDHxJaoSBL0mNMPAlqREGviQ1\nwsCXpEZ0GvhJrklyKMlNXbYjSTqxrnv4u4HnddyGJGkMnQZ+VX0CuKPLNiRJ43EMX5IaYeBLUiO2\nzrqAeywvLx9Z7vV69Hq9mdUiSfOm3+/T7/fXtY9U1cZUs1YDyeOBD1bVrxxnm+q6DklaJEmoqkzy\nnq4vy3wP8N/AtiTfSPKqLtuTJK2t8x7+WEXYw5ekicxdD1+SND8MfElqhIEvSY0w8CWpEQa+JDXC\nwJekRhj4ktQIA1+SGmHgS1IjDHxJasTczJYpSfNiZWWFwWAAwNLSElu2LEbfeDGOQpI2yGCwn+3b\nd7Fjx0F27DjI9u27GAz2z7qsDeHkaZI0srKywvbtu9i37yru7Q+vcN55u9i796q56uk7eZokrcNg\nMODAgR73jcYtHDhw0ZEhns3MwJekRhj4kjSytLTEtm19YOWotSts23YjS0tLsylqAzmGL0lHGQz2\ns3Pn2zhw4CIAzjmnz+7dr2Fp6dwZV3ZfJzOGb+BL0iqb4bJMA1+SGuFVOpKkNRn4ktQIA1+SGmHg\nS1IjDHxJaoSBL0mNMPAlqREGviQ1wsCXpEZ0HvhJnp/ky0kOJLm86/YkScfWaeAn2QL8PfA84Fzg\nZUl+scs2502/3591CZ3y+DY3j68tXffwnwZ8taoOVtXdwL8AL+q4zbmy6P/gPL7NzeNrS9eB/1jg\ntqNef3O0TpI0ZZ60laRGdDo9cpJnAMtV9fzR69cDVVV/s2o750aWpAnN1Xz4SR4AfAV4LnA78Fng\nZVV1a2eNSpKOaWuXO6+qw0n+BNjDcPjoGsNekmZjLp54JUnq3kxP2i7yTVlJzkpyQ5L9SW5Ocsms\na9poSbYk+UKSD8y6li4kOSPJdUluHf09Pn3WNW2UJG8YHdNNSa5N8qBZ17QeSa5JcijJTUete3iS\nPUm+kuQ/kpwxyxrXY43je/Po3+a+JP+a5PQT7Wdmgd/ATVk/BS6rqnOBZwJ/vGDHB3ApcMusi+jQ\n1cCHquqXgKcCCzEcmeRs4NXAUlU9heHQ7ktnW9W67WaYJUd7PfCxqnoycAPwhqlXtXGOdXx7gHOr\n6jzgq4xxfLPs4S/0TVlV9Z2q2jda/hHDsFiYexCSnAW8AHjHrGvpwqi39Oyq2g1QVT+tqjtnXNZG\nuRP4P+DUJFuBhwLfnm1J61NVnwDuWLX6RcA7R8vvBF481aI20LGOr6o+VlUro5efBs460X5mGfjN\n3JSV5PHAecBnZlvJhnor8DpgUU8CPQH4XpLdo2Grtyc5ZdZFbYSqugN4C/AN4FvAD6rqY7OtqhOP\nrqpDMOyAAY+ecT1d2gl8+EQbeeNVx5KcBlwPXDrq6W96SS4GDo2+wWT0s2i2AucD/1BV5wN3MRwi\n2PSSPBF4LXA28HPAaUl+d7ZVTcVCdk6SvBG4u6rec6JtZxn43wIed9Trs0brFsbo6/L1wLur6t9m\nXc8GuhB4YZKvAf8MPCfJu2Zc00b7JnBbVX1+9Pp6hh8Ai+BXgU9W1fer6jDwPuBZM66pC4eSPAYg\nyZnA/8y4ng2X5JUMh1bH+sCeZeB/DviFJGePrhB4KbBoV3v8E3BLVV0960I2UlVdUVWPq6onMvx7\nu6GqXj7rujbSaCjgtiTbRquey+KcoP4K8IwkD0kShse2CCekV3/b/ADwytHyK4DN3um6z/EleT7D\nYdUXVtVPxtlBpzdeHc+i35SV5ELg94CbkwwYfp28oqo+MtvKNIFLgGuTPBD4GvCqGdezIarqi6Nv\nZHuBw8AAePtsq1qfJO8BesAjk3wDuBL4a+C6JDuBg8DvzK7C9Vnj+K4AHgR8dPi5zaer6o+Oux9v\nvJKkNnjSVpIaYeBLUiMMfElqhIEvSY0w8CWpEQa+JDXCwJdGklyZ5LJZ1yF1xcCXpEYY+GpakjeO\nHpDxceDJo3VPTfKpox4sccZo/QVJvjiaPfPNSW6eafHShAx8NSvJ+Qxvt38KcDFwAcO5St4FvG70\nYIkvMbyNHYZzI716NHvmYRZ09kUtLgNfLXs28P6q+klV/S/DybVOBc4YPXAChg/O2DHq5Z9WVZ8d\nrT/hVLTSvDHwpXudaF7/RZz3Xw0x8NWyjwMvTvLgJA8DfhP4MXDHaLZTgD8AbqyqHwJ3JrlgtH6z\nPwNWDZrZ9MjSrFXVIMl7gZuAQ8BnGY7LvwJ42+iRhkdPi/yHwDuSHAZuBH44/aqlk+f0yNKYkpxa\nVT8eLV8OnFlVr51xWdLY7OFL47s4yRsY/r/5Ovc+TUnaFOzhS1IjPGkrSY0w8CWpEQa+JDXCwJek\nRhj4ktQIA1+SGvH/ymRJDaYPwt0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10a865ac8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot([10.6, 0.2], [.5, 5.6], 'bo')\n",
    "xlabel('dog')\n",
    "ylabel('cat')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<br><br><br><br>\n",
    "\n",
    "# First idea: Euclidean distance\n",
    "\n",
    "- Score a query by the distance between the query and document in vector space.\n",
    "\n",
    "$ dist(a, b) = \\sqrt{\\sum_i (a_i - b_i)^2} $\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Problem with Euclidean distance\n",
    "\n",
    "![length](files/length.png)\n",
    "\n",
    "$q$ is similar to $d_2$, but because $d_2$ is longer, Euclidean distance is large\n",
    "\n",
    "- Another way of thinking about it:\n",
    "  - Let $d_i^2$ be $d_i$ appended to itself. \n",
    "  - $dist(d_i, d_i^2)$ should be small, but it is not"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Cosine similarity\n",
    "\n",
    "**Idea:** score query by angle between query vector and document vector\n",
    "\n",
    "![cosine](files/cosine.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Cosine similarity\n",
    "\n",
    "$$sim(a, b) = \\frac{a \\cdot b}{||a||\\hbox{ } ||b||}$$\n",
    "\n",
    "- $a \\cdot b$ is dot product: $\\sum_i a_i \\times b_i$\n",
    "\n",
    "\n",
    "- $||a||$ is norm: $\\sqrt{\\sum_i a_i^2}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise\n",
    "\n",
    "Given three documents:\n",
    "\n",
    "- $d_1$: `['the', 'dog', 'barked']`\n",
    "- $d_2$: `['the', 'dog', 'jumped']`\n",
    "- $d_3$: `['a', 'cat', 'jumped']`\n",
    "\n",
    "\n",
    "1. Compute tf-idf vectors for $d_1, d_2$\n",
    "2. Compute the cosine similarity between $d_1$ and $d_2$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- $w_{t,d} = (1 + \\log tf_{t,d}) \\times \\log (\\frac{N}{df_t})$\n",
    "\n",
    "$sim(a, b) = \\frac{a \\cdot b}{||a||\\hbox{ } ||b||}$\n",
    "\n",
    "- $a \\cdot b$ is dot product: $\\sum_i a_i \\times b_i$\n",
    "\n",
    "- $||a||$ is norm: $\\sqrt{\\sum_i a_i^2}$\n",
    "\n",
    "<br><br><br><br><br>\n",
    "<br><br><br><br><br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, create term frequency matrix:\n",
    "\n",
    "| doc  | a | barked | cat | dog | jumped | the |\n",
    "|------|---|--------|-----|-----|--------|-----|\n",
    "|$d_1$ | 0 |  1     | 0   | 1   |   0    |  1  |\n",
    "|$d_2$ | 0 |  0     | 0   | 1   |   1    |  1  |\n",
    "|$d_3$ | 1 |  0     | 1   | 0   |   1    |  0  |\n",
    "\n",
    "\n",
    "$N$ = 3\n",
    "\n",
    "**tf-idf matrix:**\n",
    "\n",
    "| doc  | a | barked | cat | dog | jumped | the |\n",
    "|------|---|--------|-----|-----|--------|-----|\n",
    "|$d_1$ | 0 |  log(3)     | 0   | log(3/2)   |   0    |  log(3/2)  |\n",
    "|$d_2$ | 0 |  0     | 0   | log(3/2)  |   log(3)    |  log(3/2)  |\n",
    "|$d_3$ | log(3) |  0     | log(3)   | 0   |   log(3/2)    |  0  |\n",
    "\n",
    "\n",
    "** cosine**$(d_1, d_2)$:\n",
    "\n",
    "$$  cos(d1, d2) = (d1 * d2) / (||d1||*||d2||)$$\n",
    "\n",
    "$$d1 * d2 = (0 + 0 + 0 + log(3/2) * log(3/2) + 0 + log(3/2) * log(3/2))$$\n",
    "\n",
    "$$||d1|| = sqrt(log(3)^2 + log(3/2)^2 + log(3/2)^2)$$\n",
    "\n",
    "$$  ||d2|| = sqrt(log(3)^2 + log(3)^2 + log(3/2)^2)$$\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}