lectures/lec25/Near Duplicate Detection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Here is an example of using shingling to efficiently estimate whether two documents are near duplicates. **\n",
    "\n",
    "See [Chapter 19](http://nlp.stanford.edu/IR-book/pdf/19web.pdf) of your book for more details.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "doc1 = 'a rose is a rose is a rose'.split()    # Gertrude Stein\n",
    "doc2 = 'a rose is a rose is an onion'.split()  # Ernest Hemmingway"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'rose is a rose', 'a rose is a', 'is a rose is'}\n",
      "{'rose is a rose', 'a rose is an', 'rose is an onion', 'a rose is a', 'is a rose is'}\n"
     ]
    }
   ],
   "source": [
    "# Represent each document as a *set* of ngrams.\n",
    "\n",
    "def shingle(doc, n):\n",
    "    return set(' '.join(doc[i:i+n]) for i in range(len(doc) - n + 1))\n",
    "\n",
    "print(shingle(doc1, 4))\n",
    "print(shingle(doc2, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Similarity of two documents $\\propto$ number of overlapping shingles. **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5"
      ]
     },
     "execution_count": 193,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Jaccard similarity: size of intersection over size of union.\n",
    "def jaccard(set1, set2):\n",
    "    return len(set1 & set2) / len(set1 | set2)\n",
    "\n",
    "jaccard(set([1,2,3]), set([2,3,4]))\n",
    "# |{2,3}| / |{1,2,3,4}|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6"
      ]
     },
     "execution_count": 194,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jaccard(shingle(doc1, 4), shingle(doc2, 4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'a rose is a', 'is a rose is', 'rose is a rose'}"
      ]
     },
     "execution_count": 195,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "shingle(doc1, 4) & shingle(doc2, 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'a rose is a',\n",
       " 'a rose is an',\n",
       " 'is a rose is',\n",
       " 'rose is a rose',\n",
       " 'rose is an onion'}"
      ]
     },
     "execution_count": 196,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "shingle(doc1, 4) | shingle(doc2, 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**How can we compute Jaccard for all pairs of documents?**\n",
    "\n",
    "$O(n^2)$, where $n \\approx 1B$ web pages\n",
    "\n",
    "To compute Jaccard for each pair of documents, we need to compute intersection of all their shingles: $O(k)$, where $k$ is number of tokens in a document.\n",
    "\n",
    "So, total time is  \n",
    "$O((Kn)^2)$, where $K$ is typical document length.\n",
    "\n",
    "Can we get by without comparing all shingles?  \n",
    "i.e., can we reduce the size of $K$?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**MinHash Trick**\n",
    "\n",
    "Idea: used a small, fixed set of values to summarize a document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8686785072249457076"
      ]
     },
     "execution_count": 197,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Hash each string to a 64-bit number.\n",
    "hash('rose')\n",
    "# Small chance of collision."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{-8757600265101086734, 1747002848098801166, 1935487005774968538}"
      ]
     },
     "execution_count": 198,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now, for each doc, we convert its set of shingles to a set of hash values, one per ngram.\n",
    "def hash_doc(doc):\n",
    "    return set(hash(s) for s in shingle(doc, 4))\n",
    "hash_doc(doc1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6"
      ]
     },
     "execution_count": 199,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jaccard(hash_doc(doc1), hash_doc(doc2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**We've saved space (string->int), but not much time (still same number of comparisons)**\n",
    "\n",
    "**Idea:** Use a small, fixed number of random hashes to represent each document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'bicyn'"
      ]
     },
     "execution_count": 200,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Generate a random string of fixed size.\n",
    "import random\n",
    "import string\n",
    "\n",
    "def random_string(size, seed):\n",
    "    random.seed(seed)\n",
    "    return ''.join(random.choice(string.ascii_lowercase)\n",
    "                   for _ in range(size))\n",
    "\n",
    "random_string(5, 123)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'bicyn'"
      ]
     },
     "execution_count": 201,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_string(5, 123)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'udaxi'"
      ]
     },
     "execution_count": 202,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_string(5, 42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{-7455963190061466951, -5961080585398496229, 2527587970878870217},\n",
       " {-3587527669377022179, 750713077338955787, 1588185390600274052}]"
      ]
     },
     "execution_count": 176,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now, prepend a random string to each ngram prior to hashing.\n",
    "# This results in M different sets of hash values.\n",
    "def hash_doc_random(doc, seeds):\n",
    "    sets = []\n",
    "    for seed in seeds:\n",
    "        prefix = random_string(5, seed)\n",
    "        sets.append(set([hash(prefix + s) for s in shingle(doc, 4)]))\n",
    "    return sets\n",
    "\n",
    "# E.g., for M=2\n",
    "hash_doc_random(doc1, [123, 456])\n",
    "# Here, row i is the output of the ith hashing function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Now, summarize these M different sets by taking the minimum value from each (somewhat arbitrary). **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2746317213,\n",
       " 8697354961,\n",
       " 1181241943,\n",
       " 958682846,\n",
       " 3163119785,\n",
       " 8963334018,\n",
       " 1812140441,\n",
       " 127978094,\n",
       " 939042955,\n",
       " 8703905715,\n",
       " 9443935785,\n",
       " 6635473142,\n",
       " 5241752544,\n",
       " 6825844140,\n",
       " 3460967357,\n",
       " 7293453178,\n",
       " 5756332150,\n",
       " 667779376,\n",
       " 1445662585,\n",
       " 4693307665]"
      ]
     },
     "execution_count": 203,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# M=20: Create 20 random seeds for 20 different hash functions.\n",
    "# (In practice, we would use a larger value of M.)\n",
    "random.seed(42)\n",
    "seeds = [random.randint(0, 1e10) for _ in range(20)]\n",
    "seeds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{-8339764253898357490,\n",
       " -8331239421977272583,\n",
       " -7941353375205134921,\n",
       " -7471063588538754369,\n",
       " -7329917667602004428,\n",
       " -7122975656568478326,\n",
       " -7105912375894461766,\n",
       " -6822862262083637827,\n",
       " -6612366677129883046,\n",
       " -5643788513242149421,\n",
       " -5423508810653492103,\n",
       " -5020890668431292256,\n",
       " -2680650638414010812,\n",
       " -2519516617808959957,\n",
       " -434632871186134070,\n",
       " 426531534575266359,\n",
       " 1667459092347916612,\n",
       " 2263830750172537746,\n",
       " 2982601731111424032,\n",
       " 5484842312009809327}"
      ]
     },
     "execution_count": 204,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Prepend a random string to each ngram prior to hashing.\n",
    "# This results in K different sets of hash values.\n",
    "def min_hash(doc, seeds):\n",
    "    result = set()\n",
    "    for seed in seeds:\n",
    "        prefix = random_string(5, seed)\n",
    "        result.add(min([hash(prefix + s) for s in shingle(doc, 4)]))\n",
    "    return result\n",
    "\n",
    "min_hash(doc1, seeds)\n",
    "# Output is of length M (=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3333333333333333"
      ]
     },
     "execution_count": 205,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now, we compute Jaccard of this reduced representation.\n",
    "jaccard(min_hash(doc1, seeds), min_hash(doc2, seeds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Of course, in this example, we've *increased* the values to compare, because the documents are so short. In practice, want M << K**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.08108108108108109"
      ]
     },
     "execution_count": 206,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Should be able to distinguish between, e.g.:\n",
    "doc3 = \"a rose is a flower is a plant \".split()\n",
    "jaccard(min_hash(doc1, seeds), min_hash(doc3, seeds))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.08108108108108109"
      ]
     },
     "execution_count": 207,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jaccard(min_hash(doc2, seeds), min_hash(doc3, seeds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Quality of Jaccard estimate will improve with length of documents (k) and number of random hash functions (M)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** So, to compare two documents, rather than $O(2k)$ time to compare shingles, requires only $O(2M)$ time to compare min_hashed shingles.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**...but, we still need to do this in the worst case for all pairs of documents, $O(n^2)$**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Ways to reduce $O(n^2)$ comparisons:**\n",
    "\n",
    "- Take a hash of the hashes:  \n",
    "  $H_1 = hash(h_1, h_2, h_3, h_4)$, $H_2 = hash(h_5, h_6, h_7, h_8) \\ldots$\n",
    "- Only consider documents that match in at least one \"hash of hashes\""
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}