lectures/lec09/Evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# CS 429: Information Retrieval\n",
    "\n",
    "<br>\n",
    "\n",
    "## Lecture 9: Evaluation\n",
    "\n",
    "<br>\n",
    "\n",
    "### Dr. Aron Culotta\n",
    "### Illinois Institute of Technology \n",
    "\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Last time\n",
    "\n",
    "- Scalable cosine similarity ranking\n",
    "\n",
    "Today\n",
    "\n",
    "- Is our search engine any good?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Evaluation\n",
    "\n",
    "- Your first day at Bing, your boss says \"make our search engine better than Google's\"\n",
    "\n",
    "\n",
    "- How do you know when you're done?\n",
    "- Maybe it's already better?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# How to measure goodness?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "1. Relevant results\n",
    "2. Speed (indexing and search)\n",
    "3. Query capabilities\n",
    "4. Pleasing user interface\n",
    "5. User is *happy*?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# How to tell when the user is happy?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- They come back.\n",
    "- They come back frequently.\n",
    "- They take surveys and tell you they like you.\n",
    "\n",
    "\n",
    "- We'll settle for measuring **relevance**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# What is relevance?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "A document is **relevant** if it addresses the user's information need."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "What is **information need**?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- The user's information need is the explicit definition of what they are looking for.\n",
    "- The query is only a proxy for that need.\n",
    "\n",
    "\n",
    "- E.g. the **information need** may be \"find a flight from Chicago to Detroit leaving tomorrow\"\n",
    "   - The **query** may be \"ORD to DTW\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# How relevant are the results of your search engine?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- An **Information Retrieval Benchmark** consists of: \n",
    "  1. A documents collection.\n",
    "  2. A set of queries (and corresponding information needs).\n",
    "  3. A set of **relevance judgements** for each query-document pair (usually binary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Document collection:**\n",
    "\n",
    "- Which documents?\n",
    "- How many?\n",
    "\n",
    "**Queries:**\n",
    "\n",
    "- Which queries?\n",
    "- How many?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Generally, want a diverse set of queries that people actually use, over a diverse set of documents that people actually read."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Relevance judgements\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Who is making these judgements?\n",
    "- How many do we need?\n",
    "- How do we know they are right?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Suppose we have 10M documents and 1K queries:\n",
    "  - 10B relevance judgements??"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Crowd-sourcing can help, but not solve.\n",
    "- Either consider small datasets (<10K docs)\n",
    "- Or only judge a subset\n",
    "  - E.g., only consider those documents in top $K$ by some reasonable baseline system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Are the humans right?\n",
    "\n",
    "- How do we know we can trust the human relevance judgements?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Make multiple people label the same document.\n",
    "- **Inter-annotator agreement:** How often do two humans agree on the label?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- E.g., consider two humans labeling 100 documents:\n",
    "\n",
    "<table>\n",
    "<tr><td> </td> <td> </td> <td colspan=2> **Person 1** </td> </tr>\n",
    "<tr><td> </td> <td> </td> <td> Relevant </td> <td> Not Relevant </td> </tr>\n",
    "<tr><td rowspan=2> **Person 2** </td> <td> Relevant </td> <td> 50 </td> <td> 20 </td> </tr>\n",
    "<tr>                              <td> Not Relevant </td> <td> 10 </td> <td> 20 </td> </tr>\n",
    "</table>\n",
    "\n",
    "\n",
    "- Simple **agreement**: fraction of documents with matching labels. $\\frac{70}{100} = 70\\%$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- But, how much agreement would we expect by chance?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Person 1 says Relevant $60\\%$ of the time.\n",
    "- Person 2 says Relevant $70\\%$ of the time.\n",
    "- Chance that they both say relevant at the same time? $60\\% \\times 70\\% = 42\\%$.\n",
    "\n",
    "\n",
    "- Person 1 says Not Relevant $40\\%$ of the time.\n",
    "- Person 2 says Not Relevant $30\\%$ of the time.\n",
    "- Chance that they both say not relevant at the same time? $40\\% \\times 30\\% = 12\\%$.\n",
    "\n",
    "\n",
    "- Chance that they agree on any document (both say yes or both say no): $42\\% + 12\\% = 54\\%$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "** Cohen's Kappa ** $\\kappa$\n",
    "\n",
    "- Percent agreement beyond that expected by chance\n",
    "\n",
    "$ \\kappa = \\frac{P(A) - P(E)}{1 - P(E)}$\n",
    "\n",
    "- $P(A)$ = simple agreement proportion\n",
    "- $P(E)$ = agreement proportion expected by chance\n",
    "\n",
    "\n",
    "E.g., $\\kappa = \\frac{.7 - .54}{1 - .54} = .3478$\n",
    "\n",
    "- $k=0$ if no better than chance, $k=1$ if perfect agreement"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Assuming humans are right...\n",
    "\n",
    "Given document collection, a query, and relevance judgements, what score do we give to our search engine?\n",
    "\n",
    "- Assume search engine returns $K$ results\n",
    "\n",
    "\n",
    "<table>\n",
    "<tr> <td>                   </td>  <td> **Relevant**  </td>  <td> **Nonrelevant** </td> </tr>\n",
    "<tr> <td> **Retrieved**     </td>  <td> true pos (**tp**)      </td>  <td> false pos (**fp**)   </td> </tr>\n",
    "<tr> <td> **Not Retrieved** </td>  <td> false neg (**fn**)     </td>  <td> true neg (**tn**)    </td> </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Example:\n",
    "\n",
    "\n",
    "```\n",
    "Rank    DocID    Relevant\n",
    "----    -----    --------\n",
    "1       123      Y\n",
    "2       456      N\n",
    "3       789      N\n",
    "4       321      Y\n",
    "5       654      N\n",
    "```\n",
    "\n",
    "Assume there are 10 relevant documents total and 100 documents in the collection."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<table>\n",
    "<tr> <td>                   </td>  <td> **Relevant**  </td>  <td> **Nonrelevant** </td> </tr>\n",
    "<tr> <td> **Retrieved**     </td>  <td> 2 (**tp**)      </td>  <td> 3 (**fp**)   </td> </tr>\n",
    "<tr> <td> **Not Retrieved** </td>  <td> 8 (**fn**)     </td>  <td> 87 (**tn**)    </td> </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Accuracy:** $\\frac{tp + tn}{tp + fn + fp + tn}$\n",
    "\n",
    "e.g., $\\frac{2 + 87}{2 + 8 + 3 + 87} = 89\\%$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Why is this a mostly useless measure?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Very few documents are relevant, so returning 0 documents has nearly perfect true negatives.\n",
    "\n",
    "<table>\n",
    "<tr> <td>                   </td>  <td> **Relevant**  </td>  <td> **Nonrelevant** </td> </tr>\n",
    "<tr> <td> **Retrieved**     </td>  <td> 0 (**tp**)      </td>  <td> 0 (**fp**)   </td> </tr>\n",
    "<tr> <td> **Not Retrieved** </td>  <td> 10 (**fn**)     </td>  <td> 90 (**tn**)    </td> </tr>\n",
    "</table>\n",
    "\n",
    "**Accuracy:** $\\frac{tp + tn}{tp + fn + fp + tn}$\n",
    "\n",
    "e.g., $\\frac{0 + 90}{0 + 10 + 0 + 90} = 90\\%$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Precision and Recall\n",
    "\n",
    "\n",
    "<table>\n",
    "<tr> <td>                   </td>  <td> **Relevant**  </td>  <td> **Nonrelevant** </td> </tr>\n",
    "<tr> <td> **Retrieved**     </td>  <td> true pos (**tp**)      </td>  <td> false pos (**fp**)   </td> </tr>\n",
    "<tr> <td> **Not Retrieved** </td>  <td> false neg (**fn**)     </td>  <td> true neg (**tn**)    </td> </tr>\n",
    "</table>\n",
    "\n",
    "- **Precision:** $P = \\frac{tp}{tp + fp}$,  The fraction of *returned* documents that are *relevant*.\n",
    "\n",
    "\n",
    "- **Recall:** &nbsp;&nbsp;&nbsp; $R = \\frac{tp}{tp + fn}$,  The fraction of *relevant* documents that are *returned*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<table>\n",
    "<tr> <td>                   </td>  <td> **Relevant**  </td>  <td> **Nonrelevant** </td> </tr>\n",
    "<tr> <td> **Retrieved**     </td>  <td> 2 (**tp**)      </td>  <td> 3 (**fp**)   </td> </tr>\n",
    "<tr> <td> **Not Retrieved** </td>  <td> 8 (**fn**)     </td>  <td> 87 (**tn**)    </td> </tr>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$P = \\frac{tp}{tp + fp} = \\frac{2}{2 + 3} = 40\\%$\n",
    "\n",
    "$R = \\frac{tp}{tp + fn} = \\frac{2}{2 + 8} = 20\\%$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "<table>\n",
    "<tr> <td>                   </td>  <td> **Relevant**  </td>  <td> **Nonrelevant** </td> </tr>\n",
    "<tr> <td> **Retrieved**     </td>  <td> 0 (**tp**)      </td>  <td> 0 (**fp**)   </td> </tr>\n",
    "<tr> <td> **Not Retrieved** </td>  <td> 10 (**fn**)     </td>  <td> 90 (**tn**)    </td> </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$P = \\frac{tp}{tp + fp} = \\frac{0}{0} = NaN \\rightarrow 0\\%$\n",
    "\n",
    "$R = \\frac{tp}{tp + fn} = \\frac{0}{10} = 0\\%$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "How to combine precision and recall?\n",
    "\n",
    "**Harmonic mean** of $P$ and $R$ is $\\frac{1}{\\frac{1}{P} + \\frac{1}{R}}$\n",
    "\n",
    "**Weighted harmonic mean:**  $\\frac{1}{\\alpha\\frac{1}{P} + (1-\\alpha)\\frac{1}{R}}$\n",
    "\n",
    "- Greater $\\alpha \\rightarrow$ precision is more important than recall\n",
    "\n",
    "**F1:** Setting $\\alpha=\\frac{1}{2}$ leads to:  $\\frac{2\\cdot P \\cdot R}{P + R}$\n",
    "\n",
    "- **F1** commonly used; weights precision and recall equally\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Why didn't we just use the arithmetic mean? ($\\frac{P + R}{2}$)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x112b4d5c0>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEPCAYAAABY9lNGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8VFX+//HXJ7SVKkWKCEhREBtFIoJCENDAUhWWoiAg\nioDuoq5i/RL97drWuoq6KCKoiCKCvYAQVGqoohBAQAhdglRBMDm/PyYJIQwwkNy5M8n7+XjkwdyZ\nM3c+cyeZD/d8zjnXnHOIiIjkFON3ACIiEpmUIEREJCglCBERCUoJQkREglKCEBGRoJQgREQkKM8T\nhJnFm1myma0ys+FBHi9vZl+Y2RIzW2Zm/byOSURETs68nAdhZjHAKqA1sBlIAno655KztRkB/MU5\nd7+ZVQBWApWcc396FpiIiJyU12cQscBq59x659xhYALQOUebrUCpjNulgFQlBxER/xX2eP9VgZRs\n2xsJJI3sXgO+MbPNQEmgh8cxiYhICCKhSH0/sNQ5dzbQEBhpZiV9jklEpMDz+gxiE1A92/Y5Gfdl\n1xz4N4Bzbo2ZrQPqAQuyNzIzLRolInIanHN2Os/z+gwiCahjZjXMrCjQE/g4R5sVQBsAM6sEnA+s\nDbYz55x+nGPEiBG+xxApPzoWOhY6Fif+yQ1PzyCcc2lmdjvwNYFkNNo5t8LMBgUedqOAx4ExZrYU\nMOBe59xOL+MSEZGT87qLCefcl0DdHPf9L9vtHUBHr+MQEZFTEwlFajlFcXFxfocQMXQsjtCxOELH\nIm94OlEuL5mZi5ZYRUQihZnhTrNI7XkXk9fOPfdc1q9f73cYcgI1atTgl19+8TsMETlFUX8GkZEd\nfYhIQqXPSMQ/uTmDUA1CRESCUoIQEZGglCBERCQoJYgIMn78eOLj44/7+MyZM6lWrVoYIzpa+/bt\neeutt3x7fREJLxWpI1hMTAw///wztWrVAgIJok+fPmzYsMHz137kkUdYs2YN48aNy/W+8vNnJBLp\nVKTOB9LS0o65z+y0PlMRKeAOHoSxY+Hyy3O3HyUIjz355JPUqVOH0qVLc9FFFzFlyhQAxo4dy5VX\nXsldd91FhQoVeOSRRxg7dixXXXUVAC1btsQ5xyWXXELp0qWZOHEiEFiw8Nlnn6VSpUpUrVqVN998\nM+u1+vfvz9ChQ2nfvj2lSpWiRYsWbN26lWHDhlG2bFnq16/P0qVLs9pv2bKFbt26UbFiRWrXrs2L\nL74IwFdffcVjjz3Ge++9R6lSpWjYsCEArVq14o033sh6/muvvUb9+vWz3tuSJUs8PZYicmLr1sG9\n90K1ajBhAjz8cO72pwThsTp16jBr1iz27NnDiBEj6NOnD9u2bQNg3rx51KlTh+3bt/Pggw8CR84a\nZs6cCcCyZcvYs2cP3bt3B2Dr1q3s3buXzZs38/rrrzN06FB2796d9XoTJ07kscceIzU1lSJFitC0\naVOaNGnCzp07uf7667nzzjuBQKLp2LEjDRs2ZMuWLXzzzTe88MILTJ06lWuvvZYHHniAHj16sHfv\nXhYvXnzM+5o4cSKPPvoob7/9Nnv27OHjjz+mfPny3h1IEQkqPR0+/xw6dIAmTQLbc+fCF18E7suN\nApEgzHL/c7quv/56KlWqBED37t2pU6cO8+fPB6Bq1aoMGTKEmJgYihUrFvT5OfvuixYtysMPP0yh\nQoVo164dJUuWZOXKlVmPd+3alQYNGlC0aFG6du1KiRIluOGGGzAzevTokfW//Pnz57Njxw4efPBB\nChUqxLnnnsvAgQOZMGFCSO9r9OjR3HvvvTRq1AiAWrVq+VpAFyloUlPh6afhvPPg//4Prr8eNmwI\n3Fe7dt68RtQvtREKP+uj48aN47nnnstaamL//v3s2LGDmJiY0/pCLV++PDExR/J68eLF2bdvX9Z2\nZjICOOOMM47Zzmy7YcMGNm3aRLly5YBAIkpPT6dFixYhxZGSkkLtvPotFJGQLVgAI0fClCnQqROM\nHw+xsbn7j+zxFIgE4ZcNGzZw6623MmPGDK644goAGjZsmHVW4GcRulq1atSqVeuos4/sThZbtWrV\nWLNmjRehiUgOBw/Ce+/Byy/Dtm0weDCsXg0VKnj7ugWii8kv+/fvJyYmhgoVKpCens6YMWP48ccf\nQ35+5cqVWbs26MX1TltmcoqNjaVUqVI89dRTHDx4kLS0NH766ScWLAhc6bVSpUr88ssvxx2eOnDg\nQJ5++mkWLVoEwJo1a8Iy/FakIAlWdF6zBoYP9z45gBKEpy644ALuvvtumjZtSuXKlfnpp5+48sor\nQ35+QkICffv2pVy5cnzwwQdB25zqWUhm+5iYGD799FOWLFlCzZo1qVixIrfccgt79uwBAvUS5xzl\ny5fnsssuO+a1unXrxoMPPkjv3r0pXbo0Xbt25bfffjulWETkWOnpRwrMwYrOhQqFLxZNlBPP6TMS\nObnUVBgzBl55BcqWhaFDoUcPKF48d/st0NeDEBGJZuEsOp8qzxOEmcUDzxPozhrtnHsyx+P/BG4A\nHFAEuACo4Jzb5XVsIiJ+8KvofKo87WIysxhgFdAa2AwkAT2dc8nHad8BGOacaxPkMXUxRSl9RiIB\n69bBq68GupIaNw50I7Vr521dIZLXYooFVjvn1jvnDgMTgM4naN8LeNfjmEREwiZn0TktDWbP9qfo\nfKq87mKqCqRk295IIGkcw8zOAOKBoR7HJCLiuWBF5/ffz33ROZwiqUjdEfhetQcRiWbZi84dO0ZW\n0flUeZ0gNgHVs22fk3FfMD05SfdSQkJC1u24uDji4uJyF52ISB6IpKJzYmIiiYmJebIvr4vUhYCV\nBIrUW4D5QC/n3Ioc7coAa4FznHMHjrMvFamjlD4jya/8KDqfqogtUjvn0oDbga+Bn4AJzrkVZjbI\nzG7N1rQL8NXxkkM0W7VqFQ0bNqRMmTK89NJLfocjIrkUzUXnU6WZ1B4bOHAgZcqU4ZlnniExMZFH\nH32URYsWUa5cuTxfZylSRfpnJBIKr2Y6ey1izyAE1q9fz4UXXghAiRIluPnmm3n66ad9jkpEQrVg\nAfTvD3XqwLJlgaJzUlLgvkhPDrmlBOGh1q1bM2PGDIYOHUrp0qUpW7YsN9xwAzVr1vQ7NBE5gezX\ndO7WDerVCxSdM++LxhFJpyOShrnmO9988w2tWrWiT58+DBgwwO9wROQk1q0LdCGNGQOXXRZYXjvS\nis7hVCAShD2S+3TvRqgPXSQ/Sk+Hr74KzF2YOxf69Qv8qwsmFpAEoS93EckpP8x0DubA4QMs3baU\nhZsXsmjLolztq0AkCBGRTJG8vPap+v3w7yzZuoSFmxeycEsgIfy882fqVahH4yqNaVK1CW/wxmnv\nXwkijJxzHDp0iEOHDpGens4ff/xBTEwMRYoU8Ts0kXwtkmY6n659h/YdkwzW/raW+mfVp3GVxjSr\n1ow7Yu/goooXUaxwsaznDWbwab+mEoTHsl+m89tvv6VVq1ZZ9xUvXpyWLVsyffp0v8ITydeitei8\n94+9LN66+Khk8MuuX7io4kU0rtKYFjVaMKzpMC6qeBFFCxX1LA5NlBPP6TOScApWdB48OHKLzrsP\n7j4mGaTsSeHiihfTuEpjGlVpROOzG3PhWRdSpNCp9zbkZqKcEoR4Tp+RhEM0zHTedXAXi7YsykoG\nC7csZMveLVxS6ZKjksEFFS44rWQQjBJElLyHgkqfkXgpZ9F56NDAGkl+F513Hth5TDLYtm8bDSo3\nOCoZ1KtQj8Ix3vX2K0FEyXsoqPQZSV7LLDqPHAnbtwe6kG6+2b+is3OOlakrmZ0ym1kbZjF742w2\n7tlIw8oNA4mgSmMan92YuuXrUigmvAUQJYgoeQ8FlT4jySs5i85+La994PABFmxewKyUWcxOmc3s\nlNmULFqSZtWa0bxac5pVa8bFlS729MwgVEoQUfIeCip9RpIb6enw5ZeBIaqZRefbbgssnhcuW/dt\nDZwZpMxmVsoslm1fRv2z6mclg2bVmnFO6XPCF9ApUIKIkvdQUOkzktORveh85pmBs4WePb0vOqel\np/HTrz9lJYNZG2ax6+Aurqh2Bc3OaUbz6s1pcnYTShQt4W0geSQ3CcL/859cqlGjxlFzDSTy1KhR\nw+8QJIqE+5rOe//Yy/xN87O6i+ZunEvFEhVpVq0ZLaq34P4r76dehXrEWMFb/DrqzyBEJPoFKzoP\nGABnnZW3r+OcI2VPylHdRStTV9KgcgOaV2tO82rNuaLaFVQsUTFvX9hHBbqLSUSiV86i85Ah0L59\n3hWdD6cdZum2pUd1Fx1KO0Tz6s2z6geNqzQ+ammK/EYJQkSiRmbReeRImDcPbropcMaQF0XnA4cP\n8P2G75m5fiazUmaxYPMCapSpkZUMmldvTu2ytQtUt3REJwgziweeJ3D1utHOuSeDtIkDngOKAL86\n51oFaaMEIRLFUlPhjTfg1Vfzruiclp7G4q2LmbZ2GlPXTmXexnlcWvlSWp3biubVmtP0nKaUPaNs\n3r2JKBSxCcLMYoBVQGtgM5AE9HTOJWdrUwaYDVzjnNtkZhWcczuC7EsJQiQKJSUFhqhOnnxkpnNu\nis5rf1vL1DVTmbZuGtPXTadSiUq0qdWGtrXa0vLclpQuVjpv30CUi+RRTLHAaufcegAzmwB0BpKz\ntekNTHLObQIIlhxEJLrkLDrfdltgee3TKTqn/p7K9HXTs84SDvx5gDa12tDhvA48d+1zETv/ID/w\nOkFUBVKybW8kkDSyOx8oYmYzgJLAf51zb3kcl4h4IHvRuXHjwPLap1p0PvjnQWZtmMXUtVOZtnYa\nq1JXcVWNq2hbqy13XH4HF551YYGqIfgpEuZBFAYaAVcDJYA5ZjbHOfdzzoYJCQlZt+Pi4oiLiwtT\niCJyPMGKznPmhF50TnfpLNm6hGlrpzFt7TTmbJzDxRUvpk2tNjx77bM0Paepp9c8yG8SExNJTEzM\nk315XYNoCiQ45+Iztu8DXPZCtZkNB/7inHskY/t14Avn3KQc+1INQiSC5Kbo/MuuX46qI5Q/ozxt\na7WlTa02xJ0bR5m/lPH+DRQQkVykLgSsJFCk3gLMB3o551Zka1MPeBGIB4oB84AezrnlOfalBCES\nAXIWnYcMgcsvP3HReeeBncxYNyOrjrD30F7a1GpDm5ptaFOrDdXKVAvfGyhgIrZI7ZxLM7Pbga85\nMsx1hZkNCjzsRjnnks3sK+AHIA0YlTM5iIi/DhyA998Pvej8x59/MDtldlYdIXlHMs2rN6dtrbYM\nbjKYiypeVCCXrog2mignIseVs+g8dOjxi84/7/yZj5I/4uu1XzM7ZTYXnnVh1vDTpuc0zdezlSNZ\nxHYx5SUlCJHwCHWms3OOxVsXM3nFZCYnTyb1QCqdzu9EfJ144s6NK/AT1CKFEoSI5Fpm0Tn7NZ1z\nFp3/TP+T7zd8z+QVk5mycgpFCxWla72udK3XlcvPuVzdRhEoYmsQIhL5chadx48/uuh84PABpq2d\nxuTkyXyy6hOql6lOl7pd+Kz3Z5qTkM8pQYgUQCcrOu86uIvPVn3G5OTJTF07lYaVG9K1XldGtBxB\njTN1fY+CQl1MIgXIiYrOm/du5qPkj5icPJm5G+cSd24cXet1pcP5HTirRB5fmEHCRjUIETmuExWd\nV6WuYkryFCYnT2bljpW0P689Xep1Ib5OPCWLlvQ7dMkDShAicoxgRecePRzJuxcxOTkw8ui3A7/R\nuW5nul7Qlbhz47SkRT6kBCEiWXIWnW+97U8OVf6Oj1ZOYcrKKRQrVCww8uiCrsRWjdXIo3xOCUKk\ngMu5vPbNgw5wbuupTN8ymU9XfUqNMjXoUq8LXet1pf5Z9TXyqABRghApoLIXnS+J3cWl3T/llzMm\n8826aTSq0oiu9brSpV4Xqpep7neo4hMlCJECJHvReW7SIZr1/4T9573Jgl9n0qpmq6yRRxWKV/A7\nVIkAmignUgBkLzoXq7GEqh3GENNyPHsrXkj/Bv35uP67GnkkeUoJQiTCZRadJ32RSt3u71Dk9jH8\nHrOTZpfexKgG86hVtpbfIUo+pS4mkQiUWXR+6eU/WV/kKypdO4YNhafRoe5f6d+gP1fXvFqjjyQk\nqkGI5BOZRefXJidT9uox7K7xFnUqVmdAg/70uKgHZ/7lTL9DlCijBCESxTKLzs+/upvZu9+jdIsx\npJVaT7+GfejXoB8XnHWB3yFKFFOCEIlCqakw+o10np8yg4P1x3Cw2qdcU6c1t1w2gGvrXEvhGJUI\nJfeUIESiyIIF8MT/1vHpxjcp3HgsZ5c7kyHN+nPjJTdoaKrkOSUIkQh38CCMe3c/T3w8ic2VxlC4\nyo/0vrg3Q5r1p0HlBn6HJ/lYRCcIM4sHngdigNHOuSdzPN4S+AhYm3HXh865fwXZjxKERJ21ax0P\njZrNh2vHkFZ3Eg3KNefetv3pVK+DrtEsYRGxE+XMLAZ4CWgNbAaSzOwj51xyjqbfOuc6eRmLSLik\np8M7n2zi35+OY3WJNzmzdAx/v6E/d169nCqlqvgdnkjIvK6CxQKrnXPrAcxsAtAZyJkgtHKYRL1N\n2w5y35iPmbR2DIfOmseVdbvzapextKx9uRbHk6jkdYKoCqRk295IIGnkdIWZLQE2Afc455Z7HJdI\nnnnnm8X86/PRrCwygUquAcPb9eeff51EiaLF/Q5NJFciYRzdQqC6c+53M2sHTAHOD9YwISEh63Zc\nXBxxcXHhiE/kGAcOOB568yteW/EkvxdbQ6tyA3mn10Ia1dL1msVfiYmJJCYm5sm+PC1Sm1lTIME5\nF5+xfR/gchaqczxnHdDYObczx/0qUovvVq05zF2j3+fLvU9xxhmOW+rfy2O9e/CXokX8Dk0kqIgt\nUgNJQB0zqwFsAXoCvbI3MLNKzrltGbdjCSStncfsScQn6enw0ef7eejD0SSXfZaqxWvyv25PMKBF\nvGoLkq95miCcc2lmdjvwNUeGua4ws0GBh90ooJuZDQYOAweAHl7GJBKq1FR48Y1feWHOS+yr9wqX\n1mvB9Oveo2Wdy/0OTSQsNFFOJIfMmc6f7HgGLh5P+xrdeaLz3dStELQ0JhLRIrmLSSQqZC6v/Z+3\nF7Pu7Keg1lQGd7qV+1otp3LJyn6HJ+ILnUFIgbZuHbz8imPU1OkUiXsSzlrO8JZ3MuiyWyhdrLTf\n4YnkWkQvtZFXlCAkr6Snw1dfwYsj/+S7HR9S4pqnKFF2Pw/F3csNl9xA0UJF/Q5RJM8oQYiEIDUV\nxoyBkaMOkHbxmxxs+DS1K1Xm/hbD6XB+B12hTfIlJQiRE1iwAEaOhA+/2EnNv71MSpWXaH5uLMOb\nD6d59eZ+hyfiKRWpRXLILDq//DJs2reBc3s9R8ywsTS8oDPjm02n/ln1/Q5RJOIpQUi+knlN5zFj\noO5VP1Lixqf4/fdPadpwABOa/sA5pc/xO0SRqKEuJol6mdd0fvllmDPX0fbm79ha50lW7lnE32P/\nzuAmgznzL2f6HaaIL1SDkAIps+j8yitQ5sx0mvX/iPlFn2TXHzv5Z7N/0vfSvvyl8F/8DlPEV0oQ\nUqBkFp2nTIH2Hf+gRqe3mLT1P5QpVobhzYfTpV4XCsUU8jtMkYigBCH5Xvai87ZtcOttfxIT+z/+\nu/jfXFr5UoY3H07LGi21eJ5IDhrFJPlW9qLzZZfBww9DkbrTuHvqMCptrMRnvT+jYZWGfocpki+F\nnCDMrBlwbvbnOOfGeRCTFHDZi85z50K/fjBnDli5Ndz99d388PkPPHPNM3Sp10VnDCIeCilBmNlb\nQG1gCZCWcbcDlCAkz2QvOp95JgwdCu+/D2mF9vLYd4/x2oevcfcVdzOh2wQVn0XCINQziMuA+ioC\niBeyF507doTx4yE2FhzpvLX0LR6Y/gBtarXhh8E/cHaps/0OV6TACDVB/AhUJnBVOJFcy1l0HjwY\nVq2Cs84KPD5341z+8eU/APjwbx9y+Tm6SI9IuIWaICoAy81sPvBH5p3OuU6eRCX5VrCic7t2UChj\nVOqmPZu4/5v7+WbdNzzR+gluuOQGLaIn4pNQE0SCl0FI/pa5vPbIkUcXnevUOdLm4J8HeXbOszwz\n5xkGNR5E8tBkShUr5VvMIhJignDOzfQ6EMl/shedy5Y9UnQuXvxIG+cck5Mn88+v/0mDyg1IuiWJ\nWmVr+Re0iGQJdRRTU+BF4AKgKFAI2O+cO+klt8wsHngeiAFGO+eePE67JsBsoIdz7sPQwpdIlL3o\n3KnTkaJzzhGpy7YtY9hXw9i+fzuvdXyN1rVa+xOwiAQVahfTS0BPYCKBEU19gZNewd3MYjKe2xrY\nDCSZ2UfOueQg7Z4Avgo9dIkkwYrOq1dDhQrHtk39PZX/m/F/TFw+kYS4BG5tfCuFYzRnUyTShFz9\nc879DBRyzqU558YA8SE8LRZY7Zxb75w7DEwAOgdpdwfwAbA91HgkMqxbB/feC9WqwYQJgaLzmjUw\nfPixyeFw2mFenPciF4y8gBiLIfn2ZIY0GaLkIBKhQv3L/N3MigJLzOwpAsNdQ0kuVYGUbNsbCSSN\nLGZ2NtDFOdfKzI56TCJTKEXnnKaumcqwr4ZRpWQVpt80nYsqXhS2eEXk9ISaIPoQSAi3A3cC1YDr\n8yiG54Hh2baPu3ZCQkJC1u24uDji4uLyKAQJxfFmOmcvOuf0886fufvru/lx+488e82zdKrbSctj\niHgoMTGRxMTEPNlXyKu5mtkZQHXn3MqQdx4obic45+Iztu8DXPZCtZmtzbxJYL7FfuBW59zHOfal\nidw+yVl0HjIkeNE5u71/7OXf3/2b1xe9zj3N7mFY02EUK1wsfEGLCBCG1VzNrCPwNIERTDXNrAHw\naAgT5ZKAOmZWg0C3VE+gV/YGzrmsMY1mNgb4JGdykPA7laJzdukunXFLx/HANw9wbZ1rWTZ4GVVK\nVQlP0CKSp05lolwskAjgnFtiZjVP9iTnXJqZ3Q58zZFhrivMbFDgYTcq51NCDVy8cbKZzicyJ2UO\nf//y7xSOKcyUnlOIraqSkkg0CzVBHHbO7c7RdxzSl7lz7kugbo77/nectgNCjEfy0PGW1z5R0Tm7\nTXs2cd839zFj3QyeaPMEvS/ureUxRPKBUBPET2bWGyhkZucBfycwqU2i2OkUnbM7cPgAz855lufm\nPsdtl91G8u3JlCxa0tugRSRsQk0QdwAPEliobzyBCW3/z6ugxFvHW177VAYXTUmewp1f3UnjKo1J\nuiWJmmVP2uMoIlEm1ARRP+OncMZPZ6ATcIlHcUkeO9ny2qH6/fDv/OOLfzBz/UxGdxrN1TWv9iZg\nEfFdqAniHeCfBK4Lke5dOJLXchadH3oI2rcPreic0/Jfl9Pjgx5cUukSFt66UKutiuRzoSaIHc65\nTzyNRPJMbovOwby55E3umXoPT7R+ggENB2iym0gBENJEOTNrS2AOwzSOvmBQ2FZd1US5kwtWdO7Z\nM/SiczD7Du1j6OdDSdqUxPvd39cSGSJRxvOJckA/AkNVC3Oki8kBWpY7AmQWnSdPPvHy2qfqh20/\n0OODHjQ9pylJtyRRomiJvAlYRKJCqAniMudc3ZM3k3DJLDqPHAnbtx+Z6XyqRedgnHO8tug1Hpz+\nIM9c8wx9L+2b+52KSNQJNUHMNrP6zrnlnkYjJxVspvPpFp2D2fPHHgZ9Ooiftv/Ed/2/o16Fenmz\nYxGJOqFOd21KYKnvlWb2g5ktM7MfvAxMjkhPh88/h7/+FZo0gbS0QNH5iy8C8xjyKjks2rKIxqMa\nU6ZYGeYNnKfkIFLAhXoGEcrFgSSPBSs6T5yYu6JzMM45RiaN5JGZj/BiuxfpeVHPvH0BEYlKISUI\n59x6rwORI7wqOgez6+Aubv74Zn7Z9Qtzbp5DnXK5GAsrIvmKrvUYIXIWnW+7Le+Kzsczf9N8en7Q\nkw7nd2D8deN1vQYROUrIFwzyW36dB5Gz6DxkSN4WnYNxzvHc3Od44vsneLXDq1x3wXXevZiI+Coc\n8yAkD2XOdB45EubNg5tuyv1M51Cl/p5Kv4/6sX3/duYNnKdF9kTkuJQgwig1Fd54A1591dui8/HM\n2jCL3h/2pnv97kz62ySKFioanhcWkaikBBEGSUmBdZHCUXQOJt2l89Ssp3hu7nO83vF1OtbtGJ4X\nFpGopgThkQMHAhff8WKm86nYvn87fSf3Zd+hfSy4ZQHVylQLbwAiErV0Xcg8tm4d3HsvVK8OEyYE\nZjqvWQPDh4c/Ocz8ZSaN/teIRlUaMeOmGUoOInJKPE8QZhZvZslmtsrMhgd5vJOZLTWzxWa2wMyi\n7go0OWc6p6d7M9M5VGnpaTw681F6TurJ6E6jeaz1YxQpVCS8QYhI1PN0mKuZxQCrgNbAZiAJ6Omc\nS87Wprhz7veM2xcDk51zx4znicRhrsGKzrldXju3tu7byg0f3kC6S+ed697h7FJn+xeMiPguN8Nc\nvT6DiAVWO+fWO+cOAxMIXK40S2ZyyFAS2OFxTLmWlAT9+0Pt2rBsWaDovGABDBjgb3KYtnYajf7X\niCurXcm0PtOUHEQkV7wuUlcFUrJtbySQNI5iZl2Ax4HKwLUex3RaIqXoHMyf6X+SkJjAmCVjePu6\nt3WdaBHJExExisk5NwWYYmZXAm8RuDjRMRISErJux8XFERcX53ls2Wc6N26c98tr59bGPRvpPak3\nxQoXY9Gti6hUspLfIYmIjxITE0lMTMyTfXldg2gKJDjn4jO27wOcc+7JEzxnDRDrnEvNcX/YahA5\nZzr36xdYGykcM51PxeerP2fARwO4I/YO7r/qfmJMg9JE5GiRvNRGElDHzGoAWwhc17pX9gZmVts5\ntybjdiOAnMkhXDKLzq+8AmXLhn+mc6gOpx3mwekP8u6P7zKx+0SuqnGV3yGJSD7kaYJwzqWZ2e3A\n1wQK4qOdcyvMbFDgYTcKuN7M+gKHgP1ADy9jCibnTOd33w3vTOdTsWnPJrpN7Ea5M8qxeNBiKhSv\n4HdIIpJPFdjVXIMVnQcMiIyi8/Gk7E6h1dhW9GvQjweuekBdSiJyUrnpYipwCcKP5bXzwobdG2g1\nthVDmwze3MQeAAAOMklEQVTlrivu8jscEYkSkTwPIiJE2kznU7V+13ri3ozjjtg7lBxEJGwiYpir\nV6Kl6Hwiv+z6hVZjW3Fn0zv5++V/9zscESlA8mWCiKai84ms+20dV4+7mruvuJvbY2/3OxwRKWDy\nTYKI5JnOp2Ptb2u5euzV3NPsHobGDvU7HBEpgKK+SB2tRecTWbNzDVePu5r7mt/H4CaD/Q5HRKJY\ngStSR3vR+UR+3vkzrca24oErH1ByEBFfRVUXU34oOp/I6tTVtB7XmodbPMwtjW/xOxwRKeCiKkHU\nrh3dRecTWbljJW3eakNCywRubnSz3+GIiERXDWL7dhe1RecTSd6RTJtxbXi01aMMaDjA73BEJB/R\nTOootuLXFbR5qw3/vvrf9GvQz+9wRCSfieTVXOUElv+6nLZvteXx1o/T99K+focjInIUJQif/LT9\nJ9q+1Zan2j7FjZfc6Hc4IiLHiMphrtHux+0/0vattvyn7X+UHEQkYukMIsyWbVvGNW9fw7PXPEuv\ni3ud/AkiIj5RggijpVuXEv9OPM9f+zw9Lgr7dZFERE6JEkSYLNm6hPi34/lvu//ytwv/5nc4IiIn\npQQRBou3LKbdO+14qf1LdKvfze9wRERCogThsUVbFtH+nfa8/NeXue6C6/wOR0QkZJ6PYjKzeDNL\nNrNVZjY8yOO9zWxpxs/3Znax1zGFy8LNC2n3Tjte7fCqkoOIRB1PZ1KbWQywCmgNbAaSgJ7OueRs\nbZoCK5xzu80sHkhwzjUNsq+omkmdtCmJDu92YFSHUXSu19nvcESkgIrk5b5jgdXOufXOucPABOCo\nb0vn3Fzn3O6MzblAVY9j8tz8TfPp8G4HXu/4upKDiEQtrxNEVSAl2/ZGTpwABgJfeBqRx+ZtnEeH\n8R0Y3Wk0Het29DscEZHTFjFFajNrBfQHrjxem4SEhKzbcXFxxMXFeR7XqZiTMofOEzrzZpc3aX9e\ne7/DEZECKDExkcTExDzZl9c1iKYEagrxGdv3Ac4592SOdpcAk4B459ya4+wromsQs1Nm02VCF8Z1\nHUd8nXi/wxERASK7BpEE1DGzGmZWFOgJfJy9gZlVJ5Ac+hwvOUS6WRtm0WVCF97q+paSg4jkG552\nMTnn0szsduBrAslotHNuhZkNCjzsRgEPA+WAl83MgMPOuVgv48pL363/juvfv563r3uba2pf43c4\nIiJ5RhcMyoVv139Lt/e7Mf768bSp1cbvcEREjqELBvlg5i8z6T6xO+9e/y6ta7X2OxwRkTynBHEa\nZqybQY8PevBet/doVbOV3+GIiHhCCeIUfbP2G3pO6snE7hOJOzfO73BERDyjBHEKpq2dRq9JvZj0\nt0m0qNHC73BERDylInWI1uxcQ9PRTfnwbx9yVY2rfItDRORURPI8iHzhcNphen/Ym4dbPKzkICIF\nhhJECEYkjqD8GeW5I/YOv0MREQkb1SBOYsa6Gby55E2W3LaEwDw+EZGCQWcQJ5D6eyp9p/RlTOcx\nVCxR0e9wRETCSkXq43DOcd3711HrzFo8c+0zYXtdEZG8pJnUHhi1cBTrd61nwvUT/A5FRMQXShBB\nLP91OQ/NeIjv+39PscLF/A5HRMQXqkHkcPDPg/Sa1IvHWz9O3Qp1/Q5HRMQ3qkHkMOzLYWzcs5GJ\n3Sdq1JKIRD3VIPLI56s/Z3LyZJYM0pBWEREliAxb923l5o9v5r1u71H2jLJ+hyMi4jvVIIB0l85N\nU25iYMOBWoRPRCSDEgTwwtwX2PvHXkbEjfA7FBGRiOF5gjCzeDNLNrNVZjY8yON1zWy2mR00s7u8\njienxVsW89j3j/HOde9QOEY9biIimTz9RjSzGOAloDWwGUgys4+cc8nZmqUCdwBdvIwlmP2H9tNr\nUi9eiH+BmmVrhvvlRUQimtdnELHAaufceufcYWAC0Dl7A+fcDufcQuBPj2M5xp1f3Uls1Vh6X9w7\n3C8tIhLxvO5TqQqkZNveSCBp+O6D5R8wfd10Fg9a7HcoIiIRqUB2uqfsTmHIZ0P4tPenlCpWyu9w\nREQiktcJYhNQPdv2ORn3nZaEhISs23FxccTFxZ3yPtLS07hx8o3c2TTQvSQikp8kJiaSmJiYJ/vy\ndKkNMysErCRQpN4CzAd6OedWBGk7AtjnnAu6tnZeLbXxr2//xfR105naZyqFYgrlen8iIpEsYpfa\ncM6lmdntwNcECuKjnXMrzGxQ4GE3yswqAQuAUkC6mf0DqO+c25fX8cxJmcOL819k4a0LlRxERE6i\nwCzWt/vgbhr+ryHPXvssXeqFfUStiIgvcnMGUSAShHOOGyffSOmipXmlwyt5HJmISOSK2C6mSPH2\nD2+zeMtiFty6wO9QRESiRr5PEGt2ruGur+9iWp9pFC9S3O9wRESiRr5erO9w2mF6f9ibh1s8zKWV\nL/U7HBGRqJKvE8SIxBGUP6M8d8Te4XcoIiJRJ992Mc1YN4M3l7zJktt0dTgRkdORL88gUn9Ppe+U\nvozpPIaKJSr6HY6ISFTKd8NcnXNc9/511DqzFs9cG3RStohIgaFhrtmMWjiK9bvWM+H6CX6HIiIS\n1fJVglj+63IemvEQ3/f/nmKFi/kdjohIVMs3NYiDfx6k16RePN76cepWqOt3OCIiUS/f1CCGfTmM\njXs2MrH7RI1aEhHJUOBrEJ+v/pzJyZNZMkhDWkVE8krUJ4it+7Zy88c381639yh7Rlm/wxERyTei\nugaR7tK5acpNDGw4kBY1WvgdjohIvhLVCeKFuS+w94+9jIgb4XcoIiL5TtR2MS3espjHvn+M+QPn\nUzgmat+GiEjEisoziP2H9tNrUi9eiH+BmmVr+h2OiEi+FJXDXG/95Fb+SPuDsV3G+hyViEhky80w\nV8/PIMws3sySzWyVmQ0/Tpv/mtlqM1tiZg1OtL8Pln/A9HXTeandS94ELCIigMcJwsxigJeAa4EL\ngV5mVi9Hm3ZAbefcecAg4NXj7S9ldwpDPhvC+OvHU6pYKQ8jj2yJiYl+hxAxdCyO0LE4Qscib3h9\nBhELrHbOrXfOHQYmAJ1ztOkMjANwzs0DyphZpWA7u3Hyjdx1xV3EVo31MuaIp1/+I3QsjtCxOELH\nIm94nSCqAinZtjdm3HeiNpuCtAGgkBXinmb35GmAIiISXFSNYhrXdRyFYgr5HYaISIHg6SgmM2sK\nJDjn4jO27wOcc+7JbG1eBWY4597L2E4GWjrntuXYV3QMtxIRiTCRulhfElDHzGoAW4CeQK8cbT4G\nhgLvZSSUXTmTA5z+GxQRkdPjaYJwzqWZ2e3A1wS6s0Y751aY2aDAw26Uc+5zM2tvZj8D+4H+XsYk\nIiKhiZqJciIiEl4RV6TO64l10exkx8LMepvZ0oyf783sYj/iDIdQfi8y2jUxs8Nmdl044wunEP9G\n4sxssZn9aGYzwh1juITwN1LezL7I+K5YZmb9fAjTc2Y22sy2mdkPJ2hz6t+bzrmI+SGQsH4GagBF\ngCVAvRxt2gGfZdy+HJjrd9w+HoumQJmM2/EF+Vhka/cN8Clwnd9x+/h7UQb4CaiasV3B77h9PBYj\ngMczjwOQChT2O3YPjsWVQAPgh+M8flrfm5F2BpGnE+ui3EmPhXNurnNud8bmXI4zfyQfCOX3AuAO\n4ANgeziDC7NQjkVvYJJzbhOAc25HmGMMl1COxVYgc9mFUkCqc+7PMMYYFs6574HfTtDktL43Iy1B\n5OnEuigXyrHIbiDwhacR+eekx8LMzga6OOdeAfLziLdQfi/OB8qZ2QwzSzKzPmGLLrxCORavARea\n2WZgKfCPMMUWaU7re1MXUsgHzKwVgdFfV/odi4+eB7L3QefnJHEyhYFGwNVACWCOmc1xzv3sb1i+\nuB9Y6pxrZWa1galmdolzbp/fgUWDSEsQm4Dq2bbPybgvZ5tqJ2mTH4RyLDCzS4BRQLxz7kSnmNEs\nlGNxGTDBzIxAX3M7MzvsnPs4TDGGSyjHYiOwwzl3EDhoZt8ClxLor89PQjkWzYF/Azjn1pjZOqAe\nsCAsEUaO0/rejLQupqyJdWZWlMDEupx/4B8DfSFrpnbQiXX5wEmPhZlVByYBfZxza3yIMVxOeiyc\nc7UyfmoSqEMMyYfJAUL7G/kIuNLMCplZcQJFyRVhjjMcQjkWK4A2ABl97ucDa8MaZfgYxz9zPq3v\nzYg6g3CaWJcllGMBPAyUA17O+J/zYedcvlvqNsRjcdRTwh5kmIT4N5JsZl8BPwBpwCjn3HIfw/ZE\niL8XjwNjzGwpgS/Pe51zO/2L2htmNh6IA8qb2QYCo7eKksvvTU2UExGRoCKti0lERCKEEoSIiASl\nBCEiIkEpQYiISFBKECIiEpQShIiIBKUEIRIGGZO5lmXcbmlmn/gdk8jJKEGInETGJMS84I5zWyQi\nKUGI5JDxv/1kMxub8b/+PmY228wWmNl7GctXZF6caFbGBVjmmlmJjOd+m9F2QcayBiJRKaKW2hCJ\nIHWAPgTW7fkQaO2cO2Bm9wJ3mdmTBK4/0N05t8jMSgIHgG1AG+fcITOrA7wLNPHnLYjkjhKESHDr\nnXNJZvZXoD4wK6OrqQgwB6gLbHbOLQLIXD46Y9G4lzIu6ZgGnOdL9CJ5QAlCJLj9Gf8a8LVz7obs\nD5rZRQRfOfNOYKtz7hIzK0TgrEIkKqkGIRJc5pf/XKB5xsVmMLPiZnYesBKobGaNM+4vmZEQygBb\nMp7bFygU3rBF8o4ShEhwDrKu59wPeDdjyejZQN2MayD3INCdtITAktPFgJeBfma2mMC1B/YH2bdI\nVNBy3yIiEpTOIEREJCglCBERCUoJQkREglKCEBGRoJQgREQkKCUIEREJSglCRESCUoIQEZGg/j8M\naaVaZJ0+TQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x111199c50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%pylab inline\n",
    "\n",
    "# Compare arithmetic mean of precision and recall to F1 for fixed precision of 50%.\n",
    "\n",
    "def arith_mean(p, r):\n",
    "    return (p + r) / 2.0\n",
    "\n",
    "def f1(p, r):\n",
    "    return (2.0 * p * r) / (p + r)\n",
    "\n",
    "p = .5\n",
    "r = [0, .1, .2, .3, .4, .5, .6, .7, .8, .9, 1.0]\n",
    "\n",
    "xlabel('recall')\n",
    "ylabel('mean')\n",
    "plot(r, [arith_mean(p, ri) for ri in r], label='arithmetic')\n",
    "plot(r, [f1(p, ri) for ri in r], label='f1')\n",
    "legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Precision-Recall Curves\n",
    "\n",
    "- Depending on use-case, precision or recall may be more important\n",
    "  - Lawyer who cannot miss a single \"smoking-gun\" document needs 100% recall.\n",
    "  - Average web user doesn't want **all** documents with the song lyrics she's searching for.\n",
    "- Precision-recall curves displays results varying the number of returned documents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "```\n",
    "Rank    DocID    Relevant\n",
    "----    -----    --------\n",
    "1       123      Y\n",
    "2       456      N\n",
    "3       789      N\n",
    "4       321      Y\n",
    "5       654      N\n",
    "```\n",
    "\n",
    "Assume there are 10 relevant documents total and 100 documents in the collection.\n",
    "\n",
    "$P = \\frac{tp}{tp + fp} = \\frac{2}{2 + 3} = 40\\%$\n",
    "\n",
    "$R = \\frac{tp}{tp + fn} = \\frac{2}{2 + 8} = 20\\%$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "```\n",
    "Rank    DocID    Relevant\n",
    "----    -----    --------\n",
    "1       123      Y\n",
    "2       456      N\n",
    "3       789      N\n",
    "4       321      Y\n",
    "5       654      N\n",
    "--\n",
    "6       987      Y\n",
    "7       135      N\n",
    "8       246      N\n",
    "9       357      N\n",
    "10      468      N\n",
    "```\n",
    "\n",
    "<table>\n",
    "<tr> <td>                   </td>  <td> **Relevant**  </td>  <td> **Nonrelevant** </td> </tr>\n",
    "<tr> <td> **Retrieved**     </td>  <td> <del>2</del> &nbsp; 3 (**tp**)      </td>  <td> <del>3</del> &nbsp;&nbsp; 7 (**fp**)   </td> </tr>\n",
    "<tr> <td> **Not Retrieved** </td>  <td> <del>8</del> &nbsp; 7 (**fn**)     </td>  <td> <del>87</del> &nbsp; 83 (**tn**)    </td> </tr>\n",
    "</table>\n",
    "\n",
    "$P = \\frac{tp}{tp + fp} = \\frac{3}{3 + 7} = 30\\%$\n",
    "\n",
    "$R = \\frac{tp}{tp + fn} = \\frac{3}{3 + 7} = 30\\%$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 1.1)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAErtJREFUeJzt3X+w5XV93/Hna0VMEUKitnQCkUR+SGKkLj93KsVrcOKi\niTg0iWBiR0madRpMqJkRnGmGm3Q6DdOZDpMQf6wFpklMMBOTVFudkHE8ragIrHfx164LaldANCUa\nY8C2K/edP84X9nBzdz/n7L3f82N9PmbucL7nfL7f89oPd3nxPd/z/X5TVUiSdDhbZh1AkjT/LAtJ\nUpNlIUlqsiwkSU2WhSSpybKQJDUdM+sA40rid3wl6QhUVTa6jYXas6iquf+5/vrrZ57BnOZc1Izm\n3PyfzbJQZSFJmg3LQpLUZFlssqWlpVlHGIs5N9ci5FyEjGDOeZXN/EyrT0lqUbJK0rxIQn23HeCW\nJM2GZSFJarIsJElNloUkqcmykCQ1WRaSpCbLQpLUZFlIkposC0lSU69lkeTmJF9L8qnDjPntJPcl\n2Z3kRX3mkSQdmb73LG4FXn6oF5NcCpxWVWcAO4B39JxHknQEei2LqroD+MZhhlwG/F439hPAiUlO\n6jOTJGlysz5mcTLwwMjyQ91zkqQ5MuuykCQtgFnfg/sh4AdHlk/pnlvX8vLyk4+Xlpa+664nL0kt\ng8GAwWCw6dvt/X4WSX4IeH9VvXCd114B/HJVvTLJNuDGqtp2iO14PwtJmtBm3c+i1z2LJH8ILAHP\nTvJl4HrgWKCqamdVfSDJK5LcDzwKvKHPPJKkI+Od8iTpKOad8iRJU2NZSJKaLAtJUpNlIUlqsiwk\nSU2WhSSpybKQJDVZFpKkJstCktRkWUiSmiwLSVKTZSFJarIsJElNloUkqcmykCQ1WRaSpCbLQpLU\nZFlIkposC0lSk2UhSWqyLCRJTZaFJKnJspAkNVkWkqQmy0KS1GRZSJKaLAtJUpNlIUlqsiwkSU2W\nhSSpybKQJDVZFpKkpt7LIsn2JHuT7Ety7TqvPzvJB5PsTvLpJK/vO5MkaTKpqv42nmwB9gGXAF8B\n7gauqKq9I2OuB76nqt6a5DnA54GTquo7a7ZVfWaVpKNREqoqG91O33sWFwD3VdX+qjoA3AZctmbM\nV4ETuscnAH+9tigkSbN1TM/bPxl4YGT5QYYFMupdwIeSfAU4HnhNz5kkSRPquyzG8Vbg3qp6aZLT\ngL9McnZV/d3agcvLy08+XlpaYmlpaWohJWkRDAYDBoPBpm+372MW24DlqtreLV8HVFXdMDLmA8B/\nqKqPdssfAq6tqnvWbMtjFpI0oUU5ZnE3cHqSU5McC1wBvG/NmD3AywCSnAScCXyx51ySpAn0+jFU\nVT2e5GrgdobFdHNV7UmyY/hy7QT+I3BrknuBAG+pqq/3mUuSNJleP4baTH4MJUmTW5SPoSRJRwHL\nQpLUZFlIkposC0lSk2UhSWqyLCRJTZaFJKnJspAkNVkWkqQmy0KS1GRZSJKaLAtJUpNlIUlqsiwk\nSU2WhSSpybKQJDVZFpKkJstCktRkWUiSmiwLSVKTZSFJarIsJElNloUkqcmykCQ1WRaSpCbLQpLU\ndMw4g5I8A/iXwA+NrlNVv9lPLEnSPBmrLID/BnwT2AX8v/7iSJLm0bhlcUpVbe81iSRpbo17zOJj\nSV7YaxJJ0txKVbUHJZ8DTge+xPBjqABVVWf3G+8pGWqcrJKkg5JQVdnodsb9GOrSjb6RJGlxjfUx\nVFXtB74P+Knu5/u655qSbE+yN8m+JNceYsxSkpUkn0ny4XHDS5KmY6yySPKrwLuBf9L9/EGSN42x\n3hbgJuDlwAuAK5OctWbMicDvAj9ZVT8G/MxEfwJJUu/G/RjqF4ALq+pRgCQ3AB8Hfqex3gXAfU/s\nhSS5DbgM2Dsy5rXAe6vqIYCqemT8+JKkaRj321ABHh9Zfrx7ruVk4IGR5Qe750adCTwryYeT3J3k\ndWNmkiRNybh7FrcCn0jyZ93yq4GbNzHDOcCPA88EPp7k41V1/9qBy8vLTz5eWlpiaWlpkyJI0tFh\nMBgwGAw2fbtjfXUWIMk5wEXd4keqamWMdbYBy0+c0JfkOoZfub1hZMy1wPdU1W90y/8F+GBVvXfN\ntvzqrCRNaLO+OnvYskjyvVX1t0metd7rVfX1w248eRrweeAS4GHgLuDKqtozMuYshsc+tgPPAD4B\nvKaqPrdmW5aFJE1oWudZ/CHwkwyvCTX6X+p0y8873MpV9XiSq4HbGR4fubmq9iTZMXy5dlbV3iR/\nAXyK4bGQnWuLQpI0W2N/DDVr7llI0uQ2a89i3PMsXpzkmd3jn0/yn5M8d6NvLklaDON+dfbtwGNJ\n/hnwa8AXgN/vLZUkaa6MWxbf6T4Dugy4qap+Fzihv1iSpHky7nkW30ryVuDngYu7y3g8vb9YkqR5\nMu6exWsYXpr8F6rqq8ApwH/qLZUkaa74bShJOopN5TyLJHdU1UVJvsU651lU1fduNIAkaf65ZyFJ\nR7Fpn2exLckJI8snJLlwo28uSVoM496DewU454n/te++DXVPVZ3Tc77RDO5ZSNKEprpnwbBUnvwv\ndVWtMv7XbiVJC27csvhikl9J8vTu51eBL/YZTJI0P8YtizcC/xx4iOHd7i4EfqmvUJKk+eK3oSTp\nKDbtb0OdmeRDST7TLZ+d5N9t9M0lSYth3I+h3gW8FTgAUFWfAq7oK5Qkab6MWxbHVdVda577zmaH\nkSTNp3HL4pEkp9Fd8iPJTzO8p7Yk6bvAuCflPQ/YyfAbUd8AvgT8XFXt7zfeUzJ4gFuSJjSVCwl2\nb7QFOK+qXtbdWnVLVX1ro28sSVoc4+5Z3FNV500hz+EyuGchSRParD2Lccvit4BHgPcAjz7xfFV9\nfaMBxmVZSNLkpl0WX+Kp97MAoKqet9EA47IsJGly0y6LfwT8G+AihqXxEeAdVfXtjQYYl2UhSZOb\ndln8MfC3wLu7p14LnFhVP7vRAOOyLCRpctMui89V1Y+2nuuTZSFJk5v2/Sw+mWTbyJtfCNyz0TeX\nJC2Gcfcs9gDPB77cPfVc4PMML/lRVXV2bwkPZnDPQpImNLWT8jrbN/pGkqTF5f0sJOkoNu1jFpKk\n72K9l0WS7Un2JtmX5NrDjDs/yYEkl/edSZI0mV7LorsI4U3Ay4EXAFcmOesQ434L+Is+80ibbXV1\nlV27drFr1y5WV1dnHUfqTd97FhcA91XV/qo6ANwGXLbOuDcBfwL8Vc95pE2zsvJZzj33Gi6+eD8X\nX7yfc8+9hpWVz846ltSLvsviZOCBkeUHu+eelOQHgFdX1duBDR+EkaZhdXWVq656J7t338hjj13O\nY49dzu7dN3LVVe90D0NHpXk4wH0jMHosw8LQ3FtZWWHfviWe+ldoC/v2vYSVlZUZpZL6M+55Fkfq\nIYYn8D3hlO65UecBtyUJ8Bzg0iQHqup9aze2vLz85OOlpSWWlpY2O68kLbTBYMBgMNj07fZ6nkWS\npzE80/sShvfsvgu4sqr2HGL8rcD7q+pP13nN8yw0N1ZXVzn33GvYvftGDu5drPKiF13Drl03smXL\nPOy0S9M/g/uIVNXjSa4Gbmf4N+rmqtqTZMfw5dq5dpU+80ibZcuWLdxyyw6uuuoa9u17CQBnnDHg\nllveaFHoqOQZ3NIGrK6uPnmMYuvWrRaF5s5UL1E+DywLSZqcl/uQJE2NZSFJarIsJElNloUkqcmy\nkCQ1WRaSpCbLQpLUZFlIkposC0lSk2UhSWqyLCRJTX3fz0KSxuJFGeeb/zYkzZz3M59/XnVW0kx5\nI6l+edVZSUcF72e+GCwLSVKTZSFpprZu3cqZZw6A1ZFnVznzzP/J1q1bZxNK/4DHLCTN3MrKZ7nq\nqnc+5X7mt976RrZufcGMky0+b6sq6ajiV2f7YVlIkpr8NpQkaWosC0lSk2UhSWqyLCRJTZaFJKnJ\nspAkNVkWkqQmy0KS1GRZSJKaLAtJUpNlIUlq6r0skmxPsjfJviTXrvP6a5Pc2/3ckeSFfWeSJE2m\n1wsJJtkC7AMuAb4C3A1cUVV7R8ZsA/ZU1TeTbAeWq2rbOtvyQoKSNKFFuZDgBcB9VbW/qg4AtwGX\njQ6oqjur6pvd4p3AyT1nkiRNqO+yOBl4YGT5QQ5fBr8IfLDXRJKkiR0z6wBPSPJS4A3ARYcas7y8\n/OTjpaUllpaWes8lSYtkMBgwGAw2fbt9H7PYxvAYxPZu+TqgquqGNePOBt4LbK+qLxxiWx6zkKQJ\nLcoxi7uB05OcmuRY4ArgfaMDkjyXYVG87lBFIUmarV4/hqqqx5NcDdzOsJhurqo9SXYMX66dwK8D\nzwLeliTAgaq6oM9ckqTJeA9uSTqKLcrHUJKko4BlIUlqsiwkSU2WhSSpybKQJDVZFpKkJstCktRk\nWUiSmiwLSVKTZSFJarIsJElNloUkqcmykCQ1WRaSpCbLQpLUZFlIkposC0lSk2UhSWqyLCRJTZaF\nJKnJspAkNVkWkqQmy0KS1GRZSJKaLAtJUpNlIUlqsiwkSU2WhSSpybKQJDVZFpKkJstCktTUe1kk\n2Z5kb5J9Sa49xJjfTnJfkt1JXtR3JknSZHotiyRbgJuAlwMvAK5MctaaMZcCp1XVGcAO4B19Zurb\nYDCYdYSxmHNzLULORcgI5pxXfe9ZXADcV1X7q+oAcBtw2ZoxlwG/B1BVnwBOTHJSz7l6syi/QObc\nXIuQcxEygjnnVd9lcTLwwMjyg91zhxvz0DpjJEkz5AFuSVJTqqq/jSfbgOWq2t4tXwdUVd0wMuYd\nwIer6j3d8l7gJVX1tTXb6i+oJB3Fqiob3cYxmxHkMO4GTk9yKvAwcAVw5Zox7wN+GXhPVy5/s7Yo\nYHP+sJKkI9NrWVTV40muBm5n+JHXzVW1J8mO4cu1s6o+kOQVSe4HHgXe0GcmSdLkev0YSpJ0dJiL\nA9wbOXFvnHVnlHHryPP/O8m9SVaS3NVXxnFyJnl+ko8l+b9J3jzJunOUc57m87VdlnuT3JHk7HHX\nnaOc8zSfrxrJck+SHx933TnKOZX5HHc+kpyf5ECSyydd9ymqaqY/DAvrfuBU4OnAbuCsNWMuBf5H\n9/hC4M5x1511xm75i8D3z8lcPgc4F/j3wJsnWXcecs7hfG4DTuweb5/27+ZGc87hfB438viFwP1z\nOp/r5pzWfI47H924DwH/Hbh8I3M5D3sWGzlxb5x1Z50RIExnL66Zs6oeqapdwHcmXXdOcsJ8zeed\nVfXNbvFODp4jNG/zeaicMF/z+djI4vHAI+OuOyc5YTrzOe58vAn4E+CvjmDdp5iHsjiSE/eeGDPO\nurPKOHpyYQF/meTuJP+6h3yHyjDJfExrLjfjveZ1Pn8R+OARrrsRG8kJczafSV6dZA/wAeBXJll3\nDnLCdOazmTHJDwCvrqq3MyywsdddT99fne3Lon2N9sVV9XCSf8zwl2hPVd0x61ALbO7mM8lLGX6T\n76JZ5mg5RM65ms+q+nPgz5P8C+D3gefPKsvhjOS8iKfmnJf5vBHYtGM787Bn8RDw3JHlU7rn1o75\nwXXGjLPurDNSVQ93//w/wJ8x3A3sw0bmY1pzueH3mrf57A4W7wReVVXfmGTdOcg5d/M5kusjwDFJ\nnj3puhs0ac47OJhzWvM5TsbzgNuSfAn4aeBtSV415rr/UJ8HYcY8UPM0Dh5sOZbhwZYfWTPmFRw8\neLyNgwcRm+vOQcbjgOO7x88EPgr8xKzmcmTs9cCvHcm6M845V/PZ/aW7D9h2pH/GGeect/k8beTx\nOcAX5nQ+D5VzKvM56XwAt3LwAPcRzeWmT/QR/sG3A5/vfpmv657bAfzSyJibuj/gvcA5h1t3njIC\nP9z9y1gBPt1nxnFyAicx/Lzyb4CvA18e+eWeylxuJOcczue7gL8GPtllumvav5sbyTmH8/kW4DNd\nzo8A58/pfK7Ned6057OVcc3YW+jK4kjn0pPyJElN83DMQpI05ywLSVKTZSFJarIsJElNloUkqcmy\nkCQ1WRbSFCQ5Ncmnu8cvSfL+WWeSJmFZSA1JNutaZHWIx9LcsyykNbq9gL1J/mu3N/C67kZM9yR5\nT5LjunHnJ/lohje7ujPJM7t1/1c39p4M7ysvLbxFveqs1LfTgdcxvJHNnwKXVNW3k7wFeHOSGxje\nB+BnquqTSY4Hvg18DXhZVf3/JKcDfwScP5s/grR5LAtpffur6u4krwR+FPho93HU04GPM7wc9Veq\n6pMAVfV3AEmOBW7K8Na/jwNnzCS9tMksC2l9j3b/DHB7Vf3c6ItJfoz176vyb4GvVtXZSZ7GcG9D\nWnges5DW90QR3Am8OMlpAEmOS3IGwyt2/tMk53bPH9+Vw4nAw926/4rh5aClhWdZSOsrGN4LHHg9\n8EdJ7gU+Bjy/hvcufg3Dj5x2A7cDzwDeBrw+yQpwJgf3UKSF5iXKJUlN7llIkposC0lSk2UhSWqy\nLCRJTZaFJKnJspAkNVkWkqQmy0KS1PT3P7WjkHDexHsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x111171e80>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "xlabel('recall')\n",
    "ylabel('precision')\n",
    "plot([.2, .3], [.4, .3], 'bo')\n",
    "xlim((0, .4))\n",
    "ylim((0, 1.1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Precision and recall at each rank:\n",
    "\n",
    "```\n",
    "Rank    DocID    Relevant   P      R\n",
    "----    -----    --------   ----   ----\n",
    "1       123      Y          1.0    0.1\n",
    "2       456      N          0.5    0.1\n",
    "3       789      N          0.33   0.1  \n",
    "4       321      Y          0.5    0.2\n",
    "5       654      N          0.4    0.2\n",
    "6       987      Y          0.5    0.3\n",
    "7       135      N          0.43   0.3\n",
    "8       246      N          0.375  0.3\n",
    "9       357      N          0.33   0.3\n",
    "10      468      N          0.3    0.3\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 1.1)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFHJJREFUeJzt3X+QJGV9x/H3d0FMTghGTUgJigqcJkZkWYGrSI5BrHgY\nIxQxEUxMyWk8kxC9aJVgVVJskkolVKpSF4OKGA4TkwhW0EQTrWBZThQFOdY9VLxjwR8nvzTB3wGT\nIPPNH9Nwc5u9e2Z2pmd69t6vqq2bnnm657vPzu1nu5+nuyMzkSTpQGYmXYAkqfkMC0lSkWEhSSoy\nLCRJRYaFJKnIsJAkFR066QL6FRHO8ZWkVcjMGHYbU7VnkZmN/7r00ksnXoN1Wue01mido/8alakK\nC0nSZBgWkqQiw2LEWq3WpEvoi3WO1jTUOQ01gnU2VYzymFadIiKnpVZJaoqIIA+2AW5J0mQYFpKk\nIsNCklRkWEiSigwLSVKRYSFJKjIsJElFhoUkqciwkCQV1RoWEXFVRHwjIj53gDZvjYg7ImJnRJxU\nZz2SpNWpe8/iauBF+3sxIs4GjsvME4AtwBU113PQ63Q6LCwssLCwQKfTmXQ5kqZErWGRmTcA3z5A\nk3OAv63afgY4MiKOqrOmg9ni4m3MzW1l48Y9bNy4h7m5rSwu3jbpsiRNgUmPWRwN3NWzfE/1nEas\n0+mwefM72blzGw8+eB4PPngeO3duY/Pmd7qHIalo0mGhMVlcXGRpqcW+P/IZlpbOYHFxcUJVSZoW\nk74H9z3AU3qWj6meW9H8/Pyjj1ut1kF3PXlJKmm327Tb7ZFvt/b7WUTE04APZeZzVnjtxcDvZOYv\nRsQGYFtmbtjPdryfxRA6nQ5zc1vZuXMbe/cuOpx00lYWFrYxM+NOprQWjep+FrXuWUTEPwAt4IkR\n8TXgUuAwIDPzysz8cES8OCLuBB4ALqyznoPZzMwM27dvYfPmrSwtnQHACSe02b79dQaFpCLvlHeQ\n6XQ6j45RzM7OGhTSGjeqPQvDQpLWMG+rKkkaG8NCklRkWEiSigwLSVKRYSFJKjIsJElFhoUkqciw\nkCQVGRaSpCLDQpJUZFhIkooMC0lSkWEhSSoyLCRJRYaFJKnIsJAkFRkWkqQiw0KSVGRYSJKKDAtJ\nUpFhIUkqMiwkSUWGhSSpyLCQJBUZFpKkIsNCklRkWEiSigwLSVKRYSFJKjIsJElFhoUkqaj2sIiI\nTRGxOyKWIuLiFV5/YkR8JCJ2RsTnI+JVddckSRpMZGZ9G4+YAZaAs4B7gR3A+Zm5u6fNpcCPZOZb\nIuJJwO3AUZn5w2XbyjprlaS1KCLIzBh2O3XvWZwK3JGZezLzIeAa4Jxlbb4OHFE9PgL45vKgkCRN\n1qE1b/9o4K6e5bvpBkivdwEfi4h7gcOBl9dckyRpQHWHRT/eAtyamWdGxHHARyPixMz8r+UN5+fn\nH33carVotVpjK1KSpkG73abdbo98u3WPWWwA5jNzU7V8CZCZeVlPmw8Df5KZn6qWPwZcnJm3LNuW\nYxaSNKBpGbPYARwfEcdGxGHA+cAHl7XZBbwQICKOAtYDX665LknSAGo9DJWZD0fERcD1dIPpqszc\nFRFbui/nlcCfAldHxK1AAG/OzG/VWZckaTC1HoYaJQ9DSdLgpuUwlCRpDTAsJElFhoUkqciwkCQV\nGRaSpCLDQpJUZFhIkooMC0lSkWEhSSoyLCRJRYaFJKnIsJAkFRkWkqQiw0KSVGRYSJKKDAtJUpFh\nIUkqMiwkSUWGhSSpyLCQJBUZFpKkIsNCklRkWEiSigwLSVKRYSFJKjIsJElFh/bTKCIeC/wy8LTe\ndTLzj+opS5LUJH2FBfDPwHeBBeB/6itHktRE/YbFMZm5qdZKJEmN1e+Yxacj4jm1ViJJaqzIzHKj\niC8CxwNfoXsYKoDMzBPrLW+fGrKfWiVJe0UEmRnDbqffw1BnD/tGkqTp1ddhqMzcAzwe+KXq6/HV\nc0URsSkidkfEUkRcvJ82rYhYjIgvRMTH+y1ekjQefYVFRLwB+HvgJ6uvv4uI3+1jvRngcuBFwLOB\nCyLiWcvaHAm8DXhJZv4s8CsDfQeSpNr1exjq1cBpmfkAQERcBtwI/FVhvVOBOx7ZC4mIa4BzgN09\nbV4BXJeZ9wBk5v39ly9JGod+Z0MF8HDP8sPVcyVHA3f1LN9dPddrPfCEiPh4ROyIiFf2WZMkaUz6\n3bO4GvhMRHygWj4XuGqENZwMvAB4HHBjRNyYmXcubzg/P//o41arRavVGlEJkrQ2tNtt2u32yLfb\n19RZgIg4GTi9WvxkZi72sc4GYP6RE/oi4hK6U24v62lzMfAjmfmH1fJfAx/JzOuWbcups5I0oFFN\nnT1gWETEj2Xm9yLiCSu9npnfOuDGIw4BbgfOAu4DbgYuyMxdPW2eRXfsYxPwWOAzwMsz84vLtmVY\nSNKAxnWexT8AL6F7Taje39RRLT/jQCtn5sMRcRFwPd3xkasyc1dEbOm+nFdm5u6I+Dfgc3THQq5c\nHhSSpMnq+zDUpLlnIUmDG9WeRb/nWTw/Ih5XPf71iPiLiHjqsG8uSZoO/U6dfQfwYEQ8F3gT8CXg\nPbVVJUlqlH7D4ofVMaBzgMsz823AEfWVJUlqkn7Ps/h+RLwF+HVgY3UZj8fUV5YkqUn63bN4Od1L\nk786M78OHAP8eW1VSZIaxdlQkrSGjeU8i4i4ITNPj4jvs8J5Fpn5Y8MWIElqPvcsJGkNG/d5Fhsi\n4oie5SMi4rRh31ySNB36vQf3InDyI3/aV7OhbsnMk2uur7cG9ywkaUBj3bOgGyqP/qbOzA79T7uV\nJE25fsPiyxHx+oh4TPX1BuDLdRamenQ6HRYWFlhYWKDT6Uy6nKlnf46Ofdls/YbF64CfA+6he7e7\n04DX1lWU6rG4eBtzc1vZuHEPGzfuYW5uK4uLt026rKllf46Ofdl8zoY6SHQ6HebmtrJz5zb2/o3Q\n4aSTtrKwsI2ZmX7/bhDYn6NkX9Zr3LOh1kfExyLiC9XyiRHx+8O+ucZncXGRpaUW+/7IZ1haOoPF\nxeJND7WM/Tk69uV06Dey3wW8BXgIIDM/B5xfV1GSpGbpNyzWZebNy5774aiLUX1mZ2dZv74N9A4c\ndli//t+ZnZ2dTFFTzP4cHftyOvQ7/fX+iDiO6pIfEfEyuvfU1pSYmZlh+/YtbN68laWlMwA44YQ2\n27e/zmPCq2B/jo59OR36PSnvGcCVdGdEfRv4CvBrmbmn3vL2qcEB7hHodDqPHgeenZ31P+OQ7M/R\nsS/rMaoB7mJYVGdrvywz31fdWnUmM78/7BsPyrCQpMGNLSyqN7slM5837JsNw7CQpMGNOyz+DLgf\nuBZ44JHnM/NbwxbQL8NCkgY37rD4CvvezwKAzHzGsAX0y7CQpMGNOyx+FPht4HS6ofFJ4IrM/MGw\nBfTLsJCkwY07LN4HfA/4++qpVwBHZuavDltAvwwLSRrcuMPii5n5M6Xn6mRYSNLgxn0/i89GxIae\nNz8NuGXYN5ekR3iJ8mbrNyzmgE9HxFcj4qvAjcApEfH5iPhcbdVJOih4ifLm6/cw1LEHen0cZ3J7\nGEpam7xEeb1GdRiqr2tDjfOyHpIOLqVLlM/NzU2oMvUysiVJRbWHRURsiojdEbEUERcfoN0pEfFQ\nRJxXd02SmsNLlE+HWm+rWl2EcAk4C7gX2AGcn5m7V2j3UeAHwPbMfP8K23LMQo3jlVJHY3HxNjZv\nfuc+lyi/+urXMTv77AlXNv3Gep7FqjfenW57aWaeXS1fAmRmXras3RuA/wVOAf7FsNA02PsLrgXA\n+vVttm/f4i+4VTJ46zHWAe4hHA3c1bN8N3Bqb4OIeDJwbmaeGRH7vCY1VafTYfPmd+4zg2fnznPZ\nvNkZPKs1MzPjYHaDNeETvQ3oHcsYOgGlupVm8EhrTd17FvcAT+1ZPqZ6rtfzgGsiIoAnAWdHxEOZ\n+cHlG5ufn3/0cavVotVqjbpeSZpq7Xabdrs98u3WPWZxCHA73QHu+4CbgQsyc9d+2l8NfMgxCzWd\nJ5KNnmMW9Rj3taFWJTMfBi4CrgduA67JzF0RsSUiXrvSKnXWI43KzMwM27dv4aSTtrJu3XWsW3cd\nz33uG9i+fYu/5FbBy300X617FqPknoWayL+Gh+deWr2mYursKBkW0tq0sLDAxo17ePDBfc/HXbfu\nOj7xiac5Q2pIU3EYSpK0NhgWkibKy31MBw9DSZo4L/dRH8csJK0pThaoh2EhSSqalmtDqWH8601N\n5Wez2fxpHEQ88UlN5Wez+TwMdZDwxCc1lZ/NenmehQbiVVLVVH42p4NhIUkqMiwOEp74pKbyszkd\nHLM4iHjik5pqcfE2LrzwCm6//SkArF//Nd797t/yszkCTp3VwGZnn83Cwrae6Yl/6eChGubpAETc\nVWincXPPQtJEORuqXs6GkrQmOBtqOhgWkqQiw0LSRDkbajo4ZiFp4pypVx+vOitpTfFCgvUwLCRJ\nRc6GkiSNjWEhSSoyLCRJRYaFJKnIsJAkFRkWkqQiw0KSVGRYSJKKDAtJUpFhIUkqMiwkSUW1h0VE\nbIqI3RGxFBEXr/D6KyLi1urrhoh4Tt01SZIGU+uFBCNiBlgCzgLuBXYA52fm7p42G4BdmfndiNgE\nzGfmhhW25YUEJWlA03IhwVOBOzJzT2Y+BFwDnNPbIDNvyszvVos3AUfXXJMkaUB1h8XRwF09y3dz\n4DB4DfCRWiuSJA3s0EkX8IiIOBO4EDh9f23m5+cffdxqtWi1WrXXJUnTpN1u0263R77duscsNtAd\ng9hULV8CZGZetqzdicB1wKbM/NJ+tuWYhSQNaFrGLHYAx0fEsRFxGHA+8MHeBhHxVLpB8cr9BYUk\nabJqPQyVmQ9HxEXA9XSD6arM3BURW7ov55XAHwBPAN4eEQE8lJmn1lmXJGkw3oNbktawaTkMJUla\nAwwLSVKRYSFJKjIsJElFhoUkqciwkCQVGRaSpCLDQpJUZFhIkooMC0lSkWEhSSoyLCRJRYaFJKnI\nsJAkFRkWkqQiw0KSVGRYSJKKDAtJUpFhIUkqMiwkSUWGhSSpyLCQJBUZFpKkIsNCklRkWEiSigwL\nSVKRYSFJKjIsJElFhoUkqciwkCQVGRaSpKLawyIiNkXE7ohYioiL99PmrRFxR0TsjIiT6q5JkjSY\nWsMiImaAy4EXAc8GLoiIZy1rczZwXGaeAGwBrqizprq12+1Jl9AX6xytaahzGmoE62yquvcsTgXu\nyMw9mfkQcA1wzrI25wB/C5CZnwGOjIijaq6rNtPyAbLO0ZqGOqehRrDOpqo7LI4G7upZvrt67kBt\n7lmhjSRpghzgliQVRWbWt/GIDcB8Zm6qli8BMjMv62lzBfDxzLy2Wt4NnJGZ31i2rfoKlaQ1LDNj\n2G0cOopCDmAHcHxEHAvcB5wPXLCszQeB3wGurcLlO8uDAkbzzUqSVqfWsMjMhyPiIuB6uoe8rsrM\nXRGxpftyXpmZH46IF0fEncADwIV11iRJGlyth6EkSWtDIwa4hzlxr591J1TjbM/zX42IWyNiMSJu\nrqvGfuqMiGdGxKcj4r8j4o2DrNugOpvUn6+oark1Im6IiBP7XbdBdTapP1/aU8stEfGCftdtUJ1j\n6c9++yMiTomIhyLivEHX3UdmTvSLbmDdCRwLPAbYCTxrWZuzgX+tHp8G3NTvupOusVr+MvDjDenL\nJwFzwB8Dbxxk3SbU2cD+3AAcWT3eNO7P5rB1NrA/1/U8fg5wZ0P7c8U6x9Wf/fZH1e5jwL8A5w3T\nl03YsxjmxL1+1p10jQDBePbiinVm5v2ZuQD8cNB1G1InNKs/b8rM71aLN7H3HKGm9ef+6oRm9eeD\nPYuHA/f3u25D6oTx9Ge//fG7wD8C/7GKdffRhLBYzYl7j7TpZ91J1dh7cmECH42IHRHxmzXUt78a\nBumPcfXlKN6rqf35GuAjq1x3GMPUCQ3rz4g4NyJ2AR8GXj/Iug2oE8bTn8UaI+LJwLmZ+Q66Adb3\nuiupe+psXaZtGu3zM/O+iPgJuh+iXZl5w6SLmmKN68+IOJPuTL7TJ1lHyX7qbFR/ZuY/Af8UET8P\nvAd45qRqOZCeOk9n3zqb0p/bgJGN7TRhz+Ie4Kk9y8dUzy1v85QV2vSz7qRrJDPvq/79T+ADdHcD\n6zBMf4yrL4d+r6b1ZzVYfCXw0sz89iDrNqDOxvVnT12fBA6NiCcOuu6QBq3zBvbWOa7+7KfG5wHX\nRMRXgJcBb4+Il/a57v9X5yBMnwM1h7B3sOUwuoMtP72szYvZO3i8gb2DiMV1G1DjOuDw6vHjgE8B\nvzCpvuxpeynwptWsO+E6G9Wf1X+6O4ANq/0eJ1xn0/rzuJ7HJwNfamh/7q/OsfTnoP0BXM3eAe5V\n9eXIO3qV3/gm4Pbqw3xJ9dwW4LU9bS6vvsFbgZMPtG6TagSeXv0wFoHP11ljP3UCR9E9Xvkd4FvA\n13o+3GPpy2HqbGB/vgv4JvDZqqabx/3ZHKbOBvbnm4EvVHV+Ejilof25vM7njbs/SzUua7udKixW\n25eelCdJKmrCmIUkqeEMC0lSkWEhSSoyLCRJRYaFJKnIsJAkFRkW0hhExLER8fnq8RkR8aFJ1yQN\nwrCQCiJiVNciy/08lhrPsJCWqfYCdkfE31R7A6+sbsR0S0RcGxHrqnanRMSnonuzq5si4nHVup+o\n2t4S3fvKS1NvWq86K9XteOCVdG9k837grMz8QUS8GXhjRFxG9z4Av5KZn42Iw4EfAN8AXpiZ/xsR\nxwPvBU6ZzLcgjY5hIa1sT2buiIhfBH4G+FR1OOoxwI10L0d9b2Z+FiAz/wsgIg4DLo/urX8fBk6Y\nSPXSiBkW0soeqP4N4PrM/LXeFyPiZ1n5viq/B3w9M0+MiEPo7m1IU88xC2lljwTBTcDzI+I4gIhY\nFxEn0L1i509FxFz1/OFVOBwJ3Fet+xt0LwctTT3DQlpZQvde4MCrgPdGxK3Ap4FnZvfexS+ne8hp\nJ3A98Fjg7cCrImIRWM/ePRRpqnmJcklSkXsWkqQiw0KSVGRYSJKKDAtJUpFhIUkqMiwkSUWGhSSp\nyLCQJBX9H3eGtQxAcobpAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x112be7da0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# compute precision/recall at each sublist of size 1 to 10\n",
    "xlabel('recall')\n",
    "ylabel('precision')\n",
    "precisions = [1, .5, .33, .5, .4, .5, .43, .375, .33, .3]\n",
    "recalls = [.1, .1, .1, .2, .2, .3, .3, .3, .3, .3]\n",
    "plot(recalls, precisions, 'bo')\n",
    "xlim((0, .4))\n",
    "ylim((0, 1.1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 0.5, 0.5, 0.5, 0.5, 0.5, 0.43, 0.375, 0.33, 0.3]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(0, 1.1)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFORJREFUeJzt3XuQnfV93/H3Zy1jg7mktls6AZvEXEx9AckCrKkprEOm\nFk5qPIQY4UQdIKnlNiRWndbgmXZQb9MwnekwKXEwjiAXYoQnJClO7Qmpx9saGwzIK2xjiQVfZMDY\nrWPHF8CNpP32j3MEh7Wk56z2POey+37NnNF5zvk9z/nub3f12d/zey6pKiRJOpSpURcgSRp/hoUk\nqZFhIUlqZFhIkhoZFpKkRoaFJKnRqlEX0K8kHuMrSYehqrLUbUzUyKKqxv5x7bXXjrwG67TOSa3R\nOgf/GJSJCgtJ0mgYFpKkRobFgE1PT4+6hL5Y52BNQp2TUCNY57jKIPdptSlJTUqtkjQuklArbYJb\nkjQahoUkqZFhIUlqZFhIkhoZFpKkRoaFJKmRYSFJamRYSJIaGRaSpEathkWSrUm+leTzh2jz20ke\nSbIjyeo265EkHZ62Rxa3AG852JtJLgROrqpTgU3AjS3Xs+LNz8+zfft2tm/fzvz8/KjLkTQhWg2L\nqrob+O4hmlwE/GG37WeB45Ic32ZNK9ns7EOsXbuZ887bzXnn7Wbt2s3Mzj406rIkTYBRz1mcADzW\ns/xE9zUN2Pz8PFde+UF27Liep5++mKefvpgdO67nyis/6AhDUqNRh4WGZHZ2lrm5aZ7/LZ9ibu58\nZmdnR1SVpEkx6ntwPwG8omf5xO5rB7Rly5Znn09PT6+468lLUpOZmRlmZmYGvt3W72eR5KeAj1bV\n6w/w3luBX6uqn0uyDri+qtYdZDvez2IJ5ufnWbt2Mzt2XM9zo4t5Vq/ezPbt1zM15SBTWo4GdT+L\nVkcWST4MTAMvS/J14FrgCKCq6qaq+liStyZ5FHgKuKLNelayqakpbr55E1deuZm5ufN5+mk488wZ\nbr753QaFpEbeKW+FmZ+fZ3Z2lrPOgn371hgU0jI3qJGFYbFCJWB3Ssuft1WVJA2NYSFJamRYSJIa\nGRaSpEaGhSSpkWEhSWpkWEiSGhkWkqRGhoUkqZFhIUlqZFhIkhoZFpKkRoaFJKmRYSFJamRYSJIa\nGRaSpEaGhSSpkWEhSWpkWEiSGhkWkqRGhoUkqZFhIUlqZFhIkhoZFpKkRoaFJKmRYSFJamRYSJIa\nGRaSpEaGhSSpkWEhSWpkWEiSGrUeFknWJ9mVZC7J1Qd4/2VJPp5kR5IvJLm87ZokSYuTqmpv48kU\nMAdcAHwDuB/YUFW7etpcC7y4qt6f5OXAw8DxVbV3wbaqzVpXmgTsTmn5S0JVZanbaXtkcQ7wSFXt\nrqo9wDbgogVtvgkc031+DPDXC4NCkjRaq1re/gnAYz3Lj9MJkF4fAj6R5BvA0cClLdckSVqktsOi\nH+8HHqyqNyc5GfirJGdU1Q8XNtyyZcuzz6enp5menh5akZI0CWZmZpiZmRn4dtues1gHbKmq9d3l\na4Cqqut62nwM+E9V9enu8ieAq6vqgQXbcs5igJyzkFaGSZmzuB84JclJSY4ANgB3LmizE/hZgCTH\nA6cBX2m5LknSIrS6G6qq9iW5CriLTjBtraqdSTZ13q6bgP8M3JLkQSDA+6rqO23WJUlanFZ3Qw2S\nu6EGy91Q0sowKbuhJEnLgGEhSWpkWEiSGhkWkqRGhoUkqZFhIUlqZFhIkhoZFpKkRoaFJKmRYSFJ\namRYSJIaGRaSpEaGhSSpkWEhSWpkWEiSGhkWkqRGhoUkqZFhIUlqZFhIkhoZFpKkRoaFJKmRYSFJ\namRYSJIaGRaSpEaGhSSpkWEhSWq0qp9GSV4E/ALwU73rVNW/b6csSdI46SssgP8OfA/YDvy/9sqR\nJI2jfsPixKpa32olkqSx1e+cxWeSvL7VSiRJYytV1dwo+RJwCvBVOruhAlRVndFuec+rofqpVf1J\nwO6Ulr8kVFWWup1+d0NduNQPkiRNrr52Q1XVbuAngH/SffxE97VGSdYn2ZVkLsnVB2kznWQ2yReT\nfLLf4iVJw9FXWCR5D/DHwN/rPm5N8ut9rDcF3AC8BXgtcFmS0xe0OQ74HeDnq+p1wC8u6iuQJLWu\n391QvwK8saqeAkhyHXAP8N8a1jsHeGT/KCTJNuAiYFdPm3cCd1TVEwBV9e3+y5ckDUO/R0MF2Nez\nvK/7WpMTgMd6lh/vvtbrNOClST6Z5P4kG/usSZI0JP2OLG4BPpvkz7rLbwe2DrCGNwA/A7wEuCfJ\nPVX16MKGW7Zsefb59PQ009PTAypBkpaHmZkZZmZmBr7dvg6dBUjyBuDc7uKnqmq2j3XWAVv2n9CX\n5Bo6h9xe19PmauDFVfXvusu/B3y8qu5YsC0PnR0gD52VVoZBHTp7yLBIcmxVfT/JSw/0flV955Ab\nT14APAxcADwJ3AdcVlU7e9qcTmfuYz3wIuCzwKVV9aUF2zIsBsiwkFaGYZ1n8WHg5+lcE6r3v5Z0\nl191qJWral+Sq4C76MyPbK2qnUk2dd6um6pqV5K/BD5PZy7kpoVBIUkarb53Q42aI4vBcmQhrQyD\nGln0e57Fm5K8pPv8l5P81ySvXOqHS5ImQ7+Hzv4u8HSSM4HfBL4M/FFrVUmSxkq/YbG3uw/oIuCG\nqvod4Jj2ypIkjZN+z7P4QZL3A78MnNe9jMcL2ytLkjRO+h1ZXErn0uS/UlXfBE4E/ktrVUmSxopH\nQ61QHg0lrQxDOc8iyd1VdW6SH3CA8yyq6tilFiBJGn+OLFYoRxbSyjDs8yzWJTmmZ/mYJG9c6odL\nkibDYs6z+GHP8lPd1yRJK0Df97Po3QdUVfP0f9itJGnC9RsWX0nyG0le2H28B/hKm4WpHXv37uXW\nW28FbmXv3r2jLmfi7e/PW2+1P5dqfn6e7du3s337dubn50ddjhboNyzeDfxD4Ak6d7t7I/CutopS\nO2677U6OPfYdbNx4BHAExx77Dm677c5RlzWxevtz40b7cylmZx9i7drNnHfebs47bzdr125mdvah\nUZelHh4NtULs3buXY499B8888yc89zfCPEceeQnf//5HWLXKvYqLYX8Ozvz8PGvXbmbHjuvp7cvV\nqzezffv1TE31+zetDmTYR0OdluQTSb7YXT4jyb9Z6odreLZt28Yzz2zg+d/yKZ555lK2bds2qrIm\nlv05OLOzs8zNTbOwL+fmzmd2tvGGnBqSfv/8+RDwr4EPAlTV55N8GPiPbRWm4dm4sfPQYNifWo76\nHd8dVVX3LXjN2bwJsmHDBo48chvQO3E4z5FH3s6ePRuowsciHnv22J+Deuzbt4bVq2d+rC9PO+1/\nsWbNmiH8dqgf/YbFt5OcDJ1LfiS5hM49tTUhVq1axdatl3PkkZcAtwO38+IX/wJbt17u/vXDYH8O\nztTUFDffvInVqzdz1FF3AHdw5pnv4eabNzlfMUb6muBO8irgJjpHRH0X+CrwS1W1u93ynleDE9wD\nsHfv3mf3qW/YsMH/2JbI/hyc+fl5ZmdnOeuszmjDoBiMQU1wN4ZF994Vl1TVR7q3Vp2qqh8s9YMX\ny7CQVgavWzZYQwuL7oc9UFVnLfXDlsKwkFYGw2KwhnroLPA/k/yrJK9I8tL9j6V+uCRpMvQ7svgq\n8GMNq+pVbRR1kBocWUgrgCOLwRrKzY96vAb4F8C5dELjU8CNS/1wSdJk6Hdk8RHg+8Afd196J3Bc\nVb2jxdoW1uDIQloBHFkM1rBHFq+rqtf0LH8yyZeW+uGSpMnQ7wT355Ks27/QvUveA+2UJGkl2n+J\ncvAS5eOo37BYC3wmydeSfA24Bzg7yReSfL616iStCL2XKAcvUT6O+p2zOOlQ7w/jTG7nLKTlyUuU\nt2uocxbDvKyHpJWl6RLla9euHVFl6mVkS5IatR4WSdYn2ZVkLsnVh2h3dpI9SS5uuyZJ42PNmjWc\ndtoMXqJ8vLUaFt2LEN4AvAV4LXBZktMP0u63gL9ssx5J48dLlE+GVu/B3T3c9tqqurC7fA1QVXXd\ngnbvAf4WOBv4i6r60wNsywluaRnzEuXtGPZJeYfrBOCxnuXHgXN6GyT5SeDtVfXmJM97T9LKMTU1\n9exktjkxfsbhW3I90DuXseQElCQNVtsjiyeAV/Ysn9h9rddZwLYkAV4OXJhkT1XduXBjW7Zsefb5\n9PQ009PTg65XkibazMwMMzMzA99u23MWLwAeBi6gc8/u+4DLqmrnQdrfAnzUOQtp5XHOoh3DvvnR\nYamqfcBVwF3AQ8C2qtqZZFOSdx1olTbrkTSevNzH+Gt1ZDFIjiyk5cnLfbRrIkYWktSk6XIfGg+G\nhSSpkWEhaaS83MdkMCwkjZSX+5gMTnBLGgseOtuOQU1wGxaSxkoC/qoPzqRcG0qS+rJ/ZNF57shi\n3PjdkDRynpQ3/twNJWmkPCmvXZ6UJ2lZ8KS8yWBYSJIaGRaSRsqT8iaDcxaSRm529iGuuOJGHn74\nFfzoR3DGGV/n93//n7NmzWtHXdrE89BZScvQTwOQPNbQTsPmyELSSHk0VLs8GkrSsuDRUJPBsJAk\nNTIsJI2UR0NNBucsJI3c7OxDXHnlB5mbOx+AU0+d4ZZb3u3RUAPgVWclLSu9FxJcs8YLCQ6KYSFJ\nauTRUJKkoTEsJEmNDAtJUiPDQpLUyLCQJDUyLCRJjQwLSVIjw0KS1MiwkCQ1MiwkSY0MC0lSo9bD\nIsn6JLuSzCW5+gDvvzPJg93H3Ule33ZNkqTFafVCgkmmgDngAuAbwP3Ahqra1dNmHbCzqr6XZD2w\nparWHWBbXkhQkhZpUi4keA7wSFXtrqo9wDbgot4GVXVvVX2vu3gvcELLNUmSFqntsDgBeKxn+XEO\nHQa/Cny81YokSYu2atQF7JfkzcAVwLkHa7Nly5Znn09PTzM9Pd16XZI0SWZmZpiZmRn4dtues1hH\nZw5ifXf5GqCq6roF7c4A7gDWV9WXD7It5ywkaZEmZc7ifuCUJCclOQLYANzZ2yDJK+kExcaDBYUk\nabRa3Q1VVfuSXAXcRSeYtlbVziSbOm/XTcC/BV4KfCBJgD1VdU6bdUmSFsd7cEvSMjYpu6EkScuA\nYSFJamRYSJIaGRaSpEaGhSSpkWEhSWpkWEiSGhkWkqRGhoUkqZFhIUlqZFhIkhoZFpKkRoaFJKmR\nYSFJamRYSJIaGRaSpEaGhSSpkWEhSWpkWEiSGhkWkqRGhoUkqZFhIUlqZFhIkhoZFpKkRoaFJKmR\nYSFJamRYSJIaGRaSpEaGhSSpkWEhSWpkWEiSGrUeFknWJ9mVZC7J1Qdp89tJHkmyI8nqtmuSJC1O\nq2GRZAq4AXgL8FrgsiSnL2hzIXByVZ0KbAJubLOmts3MzIy6hL5Y52BNQp2TUCNY57hqe2RxDvBI\nVe2uqj3ANuCiBW0uAv4QoKo+CxyX5PiW62rNpPwAWedgTUKdk1AjWOe4ajssTgAe61l+vPvaodo8\ncYA2kqQRcoJbktQoVdXexpN1wJaqWt9dvgaoqrqup82NwCer6vbu8i7g/Kr61oJttVeoJC1jVZWl\nbmPVIAo5hPuBU5KcBDwJbAAuW9DmTuDXgNu74fI3C4MCBvPFSpIOT6thUVX7klwF3EVnl9fWqtqZ\nZFPn7bqpqj6W5K1JHgWeAq5osyZJ0uK1uhtKkrQ8jMUE91JO3Otn3RHVuKbn9a8leTDJbJL72qqx\nnzqTvDrJZ5L8KMl7F7PuGNU5Tv35zm4tDya5O8kZ/a47RnWOU3++raeWB5L8TL/rjlGdQ+nPfvsj\nydlJ9iS5eLHrPk9VjfRBJ7AeBU4CXgjsAE5f0OZC4H90n78RuLffdUddY3f5K8DfGZO+fDmwFvgP\nwHsXs+441DmG/bkOOK77fP2wfzaXWucY9udRPc9fDzw6pv15wDqH1Z/99ke33SeAvwAuXkpfjsPI\nYikn7vWz7qhrBAjDGcU11llV366q7cDexa47JnXCePXnvVX1ve7ivTx3jtC49efB6oTx6s+nexaP\nBr7d77pjUicMpz/77Y9fB/4E+D+Hse7zjENYHM6Je/vb9LPuqGrsPbmwgL9Kcn+Sf9ZCfQerYTH9\nMay+HMRnjWt//irw8cNcdymWUieMWX8meXuSncDHgN9YzLpjUCcMpz8ba0zyk8Dbq+p36QRY3+se\nSNuHzrZl0g6jfVNVPZnk79L5IdpZVXePuqgJNnb9meTNdI7kO3eUdTQ5SJ1j1Z9V9efAnyf5R8Af\nAa8eVS2H0lPnuTy/znHpz+uBgc3tjMPI4gnglT3LJ3ZfW9jmFQdo08+6o66Rqnqy++//Bf6MzjCw\nDUvpj2H15ZI/a9z6sztZfBPwtqr67mLWHYM6x64/e+r6FLAqycsWu+4SLbbOu3muzmH1Zz81ngVs\nS/JV4BLgA0ne1ue6P67NSZg+J2pewHOTLUfQmWz5BwvavJXnJo/X8dwkYuO6Y1DjUcDR3ecvAT4N\n/ONR9WVP22uB3zycdUdc51j1Z/eX7hFg3eF+jSOuc9z68+Se528Avjym/XmwOofSn4vtD+AWnpvg\nPqy+HHhHH+YXvh54uPvDfE33tU3Au3ra3ND9Ah8E3nCodcepRuCnu9+MWeALbdbYT53A8XT2V/4N\n8B3g6z0/3EPpy6XUOYb9+SHgr4HPdWu6b9g/m0upcwz7833AF7t1fgo4e0z7c2GdZw27P5tqXND2\nZrphcbh96Ul5kqRG4zBnIUkac4aFJKmRYSFJamRYSJIaGRaSpEaGhSSpkWEhDUGSk5J8ofv8/CQf\nHXVN0mIYFlKDJIO6Flkd5Lk09gwLaYHuKGBXkj/ojgY2dm/E9ECS25Mc1W13dpJPp3Ozq3uTvKS7\n7v/utn0gnfvKSxNvUq86K7XtFGAjnRvZ/ClwQVU9k+R9wHuTXEfnPgC/WFWfS3I08AzwLeBnq+pv\nk5wC3AacPZovQRocw0I6sN1VdX+SnwNeA3y6uzvqhcA9dC5H/Y2q+hxAVf0QIMkRwA3p3Pp3H3Dq\nSKqXBsywkA7sqe6/Ae6qql/qfTPJ6zjwfVX+JfDNqjojyQvojDakieechXRg+4PgXuBNSU4GSHJU\nklPpXLHz7ydZ23396G44HAc82V33n9K5HLQ08QwL6cAKOvcCBy4HbkvyIPAZ4NXVuXfxpXR2Oe0A\n7gJeBHwAuDzJLHAaz41QpInmJcolSY0cWUiSGhkWkqRGhoUkqZFhIUlqZFhIkhoZFpKkRoaFJKmR\nYSFJavT/AbVsLdHBnhslAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x112d65320>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Interpolated precision: max of precisions to right of value\n",
    "xlabel('recall')\n",
    "ylabel('precision')\n",
    "interpolated_pre = [max(precisions[i:]) for i in range(len(precisions))]\n",
    "print(interpolated_pre)\n",
    "step(recalls, interpolated_pre, 'bo')\n",
    "xlim((0, .4))\n",
    "ylim((0, 1.1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Mean Average Precision (MAP)\n",
    "\n",
    "- Average the precision values for top $k$ documents, considering only those elements where a relevant document is found.\n",
    "\n",
    "|Rank|    DocID  |    Relevant |   P       |   R   |\n",
    "|----|    -----  |    -------- |  ----     |  ---- |\n",
    "| 1  |       123 |     Y       |   **1.0** |   0.1 |\n",
    "| 2  |       456 |     N       |   0.5     |   0.1 |\n",
    "| 3  |       789 |     N       |   0.33    |   0.1 |  \n",
    "| 4  |       321 |     Y       |   **0.5** |   0.2 |\n",
    "| 5  |       654 |     N       |   0.4     |   0.2 |\n",
    "| 6  |       987 |     Y       |   **0.5** |   0.3 |\n",
    "| 7  |       135 |     N       |   0.43    |   0.3 |\n",
    "| 8  |       246 |     N       |   0.375   |   0.3 |\n",
    "| 9  |       357 |     N       |   0.33    |   0.3 |\n",
    "| 10 |       468 |     N       |   0.3     |   0.3 |\n",
    "\n",
    "If relevant document not returned, assume 0 precision for those. So, if there are 10 relevant documents, MAP is:\n",
    "\n",
    "$MAP=\\frac{1.0 + 0.5 + 0.5 + 0 + \\ldots + 0}{10} = .2$\n",
    "\n",
    "If there are 5 relevant documents, MAP is:\n",
    "\n",
    "$MAP=\\frac{1.0 + 0.5 + 0.5 + 0 + \\ldots + 0}{5} = .4$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# R-Precision\n",
    "\n",
    "Precision considering the top $R$ documents, where $R$ is the number of relevant documents.\n",
    "\n",
    "In our example, precision \"at 10\" ($P$@$10$) is $0.3$\n",
    "\n",
    "Perfect system has $R$-precision of 1.0 (e.g., top $R$ results are all relevant)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Labeling data is hard...\n",
    "\n",
    "Up to now, we've assumed a human has annotated many documents by relevance.\n",
    "\n",
    "Can we get this data with less effort?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "For query $q$, the system returns\n",
    "\n",
    "```\n",
    "Rank    DocID\n",
    "----    -----\n",
    "1       123\n",
    "2       456\n",
    "3       789\n",
    "```\n",
    "\n",
    "Click-through data:\n",
    "\n",
    "```\n",
    "DocID   Clicks\n",
    "-----   ------\n",
    "123     1,000\n",
    "456     500\n",
    "789     100\n",
    "```\n",
    "\n",
    "Does this mean that users think document 123 is more relevant than document 456?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Positional bias\n",
    "\n",
    "Users are *a-priori* more likely to click a higher ranked document.\n",
    "\n",
    "How can we compensate for this?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Pairwise preferences:**\n",
    "\n",
    "```\n",
    "DocID   Clicks\n",
    "-----   ------\n",
    "123     1,000\n",
    "456     500\n",
    "789     5,000\n",
    "```\n",
    "\n",
    "Can probably conclude that 789 > 456 and 789 > 123\n",
    "  - otherwise, user would have clicked the earlier link\n",
    "\n",
    "How can we compute a score from this?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Let $L^*$ be the set of **true** pairwise preferences $\\{d_i > d_j\\}$\n",
    "\n",
    "Let $L$ be the ranking produced by our system: $d_{123} > d_{456} > d_{789} \\ldots$\n",
    "\n",
    "Let $r_L(i, j)$ be 1 if ranking $L$ ranks $i$ before $j$.\n",
    "\n",
    "**Kendall tau distance** is the number of pairwise disagreements: \n",
    "\n",
    "$\\tau(L, L^*) = {\\Big|}{\\big \\{}(i,j) $ s.t. $r_L(i, j) \\ne r_{L^*}(i, j){\\big \\}}{\\Big|}$\n",
    "\n",
    "If perfectly matched: $\\tau=0$; if perfectly mismatched: $\\tau=\\Big| L^* \\Big|$\n",
    "\n",
    "E.g., \n",
    "\n",
    "- true list &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$=[2, 1, 3]$\n",
    "- predicted list $=$ $[1,2,3]$\n",
    "\n",
    "- $L^*= (2>1,$ &nbsp; $ 2>3,$ &nbsp; $ 1>3)$\n",
    "- $L$ &nbsp;&nbsp; $= (1>2,$ &nbsp; $ 1>3,$ &nbsp; $ 2>3)$\n",
    "- $\\tau(L, L^*) = \\Big|(1,2)\\Big| = 1$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Interleaving rankings\n",
    "\n",
    "Given two ranking systems A and B, which is better?\n",
    "\n",
    "Interleave rankings from each, showing half with A first, half with B first.\n",
    "\n",
    "Group 1: $A_1$, $B_1$, $A_2$, $B_2 \\ldots$\n",
    "\n",
    "Group 2: $B_1$, $A_1$, $B_2$, $A_2 \\ldots$\n",
    "\n",
    "E.g., if $A_1$ is clicked more than $B_1$ in Group 2, then A might be better than B."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# A/B Testing\n",
    "\n",
    "A method of measuring the impact of a system parameter.\n",
    "\n",
    "- E.g., Should I stem? What should the size of the champion list be? How should I tokenize?\n",
    "\n",
    "\n",
    "<br><br>\n",
    "\n",
    "To do A/B testing:\n",
    "\n",
    "\n",
    "1. Create a search engine with millions of users.\n",
    "2. Divert a small sample (1%) to the *experimental* search engine with that feature modified.\n",
    "3. Measure difference in user happiness between groups."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Happiness:\n",
    "  - proportion of time first result is clicked\n",
    "  - proportion of time top $k$ result is clicked  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}