CH3/templates/1_TextsClusteringExample.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Пример: кластеризация текстов"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Выборка"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "logging.basicConfig()\n",
    "\n",
    "from sklearn.datasets import fetch_20newsgroups\n",
    "\n",
    "train_all = fetch_20newsgroups(subset='train')\n",
    "print train_all.target_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "simple_dataset = fetch_20newsgroups(\n",
    "    subset='train', \n",
    "    categories=['comp.sys.mac.hardware', 'soc.religion.christian', 'rec.sport.hockey'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "From: erik@cheshire.oxy.edu (Erik Adams)\n",
      "Subject: HELP!!  My Macintosh \"luggable\" has lines on its screen!\n",
      "Organization: Occidental College, Los Angeles, CA 90041 USA.\n",
      "Distribution: comp\n",
      "Lines: 20\n",
      "\n",
      "Okay, I don't use it very much, but I would like for it to keep working\n",
      "correctly, at least as long as Apple continues to make System software\n",
      "that will run on it, if slowly :-)\n",
      "\n",
      "Here is the problem:  When the screen is tilted too far back, vertical\n",
      "lines appear on the screen.  They are every 10 pixels or so, and seem\n",
      "to be affected somewhat by opening windows and pulling down menus.\n",
      "It looks to a semi-technical person like there is a loose connection\n",
      "between the screen and the rest of the computer.\n",
      "\n",
      "I am open to suggestions that do not involve buying a new computer,\n",
      "or taking this one to the shop.  I would also like to not have\n",
      "to buy one of Larry Pina's books.  I like Larry, but I'm not sure\n",
      "I feel strongly enough about the computer to buy a service manual\n",
      "for it.\n",
      "\n",
      "On a related note:  what does the monitor connector connect to?\n",
      "\n",
      "Erik\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print simple_dataset.data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 1, ..., 0, 1, 2])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "simple_dataset.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "From: dlecoint@garnet.acns.fsu.edu (Darius_Lecointe)\n",
      "Subject: Re: Sabbath Admissions 5of5\n",
      "Organization: Florida State University\n",
      "Lines: 21\n",
      "\n",
      "I find it interesting that cls never answered any of the questions posed. \n",
      "Then he goes on the make statements which make me shudder.  He has\n",
      "established a two-tiered God.  One set of rules for the Jews (his people)\n",
      "and another set for the saved Gentiles (his people).  Why would God\n",
      "discriminate?  Does the Jew who accepts Jesus now have to live under the\n",
      "Gentile rules.\n",
      "\n",
      "God has one set of rules for all his people.  Paul was never against the\n",
      "law.  In fact he says repeatedly that faith establishes rather that annuls\n",
      "the law.  Paul's point is germane to both Jews and Greeks.  The Law can\n",
      "never be used as an instrument of salvation.  And please do not combine\n",
      "the ceremonial and moral laws in one.\n",
      "\n",
      "In Matt 5:14-19 Christ plainly says what He came to do and you say He was\n",
      "only saying that for the Jews's benefit.  Your Christ must be a\n",
      "politician, speaking from both sides of His mouth.  As Paul said, \"I have\n",
      "not so learned Christ.\"  Forget all the theology, just do what Jesus says.\n",
      " Your excuses will not hold up in a court of law on earth, far less in\n",
      "God's judgement hall.\n",
      "\n",
      "Darius\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print simple_dataset.data[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "From: scialdone@nssdca.gsfc.nasa.gov (John Scialdone)\n",
      "Subject: CUT Vukota and Pilon!!!\n",
      "News-Software: VAX/VMS VNEWS 1.41    \n",
      "Organization: NASA - Goddard Space Flight Center\n",
      "Lines: 32\n",
      "\n",
      "I have been to all 3 Isles/Caps tilts at the Crap Centre this year, all Isles\n",
      "wins and there is no justification for Vukota and Pilon to play for the Isles.\n",
      "Vukota is absolutely the worst puck handler in the world!! He couldn't hit a\n",
      "bull in the ass with a banjo!! Al must remember a few years back when Mick \n",
      "scored 3 goals in one period against the Caps in a 5-3 Isles win. I was there\n",
      "and was astonished as was the rest of the crowd. Wake-up Al!!! Years later he's\n",
      "gotten worse. He's a cheap shot artist and always ends up getting\n",
      "stupid/senseless penalties. I think he would make a good police officier!!!\n",
      "\n",
      "As for Pilon, he can't carry the puck out to center ice by himself. He either\n",
      "makes a bad pass resulting in a turnover, or he attempts to bring the puck \n",
      "towards the neutral zone and skates right into an opposing skater. He can't\n",
      "stay on his skates with most forwards or centers. He either falls down or \n",
      "committs a penalty. Call up somebody from Capital District AL!!!!!\n",
      "\n",
      "As far as the playoffs, the Isles are as difficult to figure out as the Caps.\n",
      "Two good teams with talent but so inconsistent. They should meet in the first\n",
      "round. The Isles seem to play up to the level of their competition so they\n",
      "should play well against Jersey tonite. It'll probably be another tight 1-goal\n",
      "game as the last 20 games hve been for the Isles. I wish when the get a lead\n",
      "they could continue to pour it on instead of settling back into a defensive\n",
      "shell and letting the opposition get back in the game. Al MUST understand he\n",
      "can't do with this team what he did with the 80-83 Isles. maybe Al should got\n",
      "to. Where is Bobby Nystrom?? Clark Gilles?? John Tonelli?? These are the kind\n",
      "of young minds we need behing the bench!!    FIRE AL!!!!\n",
      "\n",
      "John Scialdone\n",
      "SCIALDONE@NSSDCA.GSFC.NASA.GOV\n",
      "\n",
      "**********When your ship comes in, first man takes the Sail********************\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print simple_dataset.data[-2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1777\n"
     ]
    }
   ],
   "source": [
    "print len(simple_dataset.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Признаки"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
    "\n",
    "vectorizer = TfidfVectorizer(max_df=500, min_df=10)\n",
    "matrix = vectorizer.fit_transform(simple_dataset.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'matrix' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-1-6bfc972aad28>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[1;32mprint\u001b[0m \u001b[0mmatrix\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m: name 'matrix' is not defined"
     ]
    }
   ],
   "source": [
    "print matrix.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Аггломеративная кластеризация (neighbour joining)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.cluster.hierarchical import AgglomerativeClustering\n",
    "\n",
    "model = AgglomerativeClustering(n_clusters=3, affinity='cosine', linkage='complete')\n",
    "preds = model.fit_predict(matrix.toarray())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 1]\n"
     ]
    }
   ],
   "source": [
    "print list(preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 877)\t0.111806579628\n",
      "  (0, 880)\t0.11851815596\n",
      "  (0, 2243)\t0.0925317447412\n",
      "  (0, 1136)\t0.0500506718781\n",
      "  (0, 2358)\t0.0840797530073\n",
      "  (0, 2820)\t0.108042909343\n",
      "  (0, 2108)\t0.1235763517\n",
      "  (0, 3035)\t0.101544778998\n",
      "  (0, 1229)\t0.0736045063897\n",
      "  (0, 3262)\t0.130212979504\n",
      "  (0, 1366)\t0.0836339497399\n",
      "  (0, 3305)\t0.0678317940862\n",
      "  (0, 551)\t0.0978140030819\n",
      "  (0, 1961)\t0.228859994985\n",
      "  (0, 619)\t0.170473343334\n",
      "  (0, 294)\t0.0517710627222\n",
      "  (0, 3322)\t0.0953890204835\n",
      "  (0, 2325)\t0.054192064708\n",
      "  (0, 620)\t0.111806579628\n",
      "  (0, 1831)\t0.125052646182\n",
      "  (0, 3285)\t0.112645815931\n",
      "  (0, 2422)\t0.0942723610696\n",
      "  (0, 301)\t0.0556325669605\n",
      "  (0, 855)\t0.201625540883\n",
      "  (0, 2868)\t0.0859620897079\n",
      "  :\t:\n",
      "  (0, 900)\t0.1235763517\n",
      "  (0, 354)\t0.0694743738355\n",
      "  (0, 2048)\t0.0748844895603\n",
      "  (0, 1991)\t0.0711302768586\n",
      "  (0, 926)\t0.117420593185\n",
      "  (0, 3722)\t0.0950102887763\n",
      "  (0, 1905)\t0.0817353647899\n",
      "  (0, 2273)\t0.0579600218377\n",
      "  (0, 3600)\t0.0573701527992\n",
      "  (0, 3555)\t0.0622908798771\n",
      "  (0, 2413)\t0.102588780336\n",
      "  (0, 69)\t0.0685826705273\n",
      "  (0, 838)\t0.101544778998\n",
      "  (0, 1121)\t0.0644281622922\n",
      "  (0, 3553)\t0.0734668995949\n",
      "  (0, 625)\t0.0520717390913\n",
      "  (0, 319)\t0.100549353806\n",
      "  (0, 2056)\t0.0961672074361\n",
      "  (0, 810)\t0.0809427637022\n",
      "  (0, 2991)\t0.389561083053\n",
      "  (0, 1846)\t0.0674142631248\n",
      "  (0, 2082)\t0.0965671776392\n",
      "  (0, 1659)\t0.0722667093676\n",
      "  (0, 239)\t0.0950102887763\n",
      "  (0, 1246)\t0.390638938513\n"
     ]
    }
   ],
   "source": [
    "print matrix[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'00',\n",
       " u'000',\n",
       " u'01',\n",
       " u'02',\n",
       " u'03',\n",
       " u'030',\n",
       " u'0358',\n",
       " u'04',\n",
       " u'040',\n",
       " u'05',\n",
       " u'06',\n",
       " u'07',\n",
       " u'08',\n",
       " u'09',\n",
       " u'10',\n",
       " u'100',\n",
       " u'1000',\n",
       " u'101',\n",
       " u'102',\n",
       " u'104',\n",
       " u'105',\n",
       " u'106',\n",
       " u'109',\n",
       " u'11',\n",
       " u'110',\n",
       " u'112',\n",
       " u'113',\n",
       " u'119',\n",
       " u'12',\n",
       " u'120',\n",
       " u'126',\n",
       " u'127',\n",
       " u'128',\n",
       " u'129',\n",
       " u'13',\n",
       " u'132',\n",
       " u'133',\n",
       " u'14',\n",
       " u'140',\n",
       " u'15',\n",
       " u'150',\n",
       " u'152',\n",
       " u'16',\n",
       " u'160',\n",
       " u'17',\n",
       " u'170',\n",
       " u'175',\n",
       " u'18',\n",
       " u'180',\n",
       " u'19',\n",
       " u'1987',\n",
       " u'1988',\n",
       " u'1989',\n",
       " u'199',\n",
       " u'1990',\n",
       " u'1991',\n",
       " u'1992',\n",
       " u'1993',\n",
       " u'1993apr14',\n",
       " u'1993apr15',\n",
       " u'1993apr16',\n",
       " u'1993apr18',\n",
       " u'1993apr19',\n",
       " u'1993apr20',\n",
       " u'1993apr5',\n",
       " u'1993apr6',\n",
       " u'1d17',\n",
       " u'1d20',\n",
       " u'1st',\n",
       " u'20',\n",
       " u'200',\n",
       " u'2000',\n",
       " u'203',\n",
       " u'21',\n",
       " u'210',\n",
       " u'22',\n",
       " u'23',\n",
       " u'230',\n",
       " u'24',\n",
       " u'240',\n",
       " u'241',\n",
       " u'25',\n",
       " u'253',\n",
       " u'256',\n",
       " u'25mhz',\n",
       " u'26',\n",
       " u'27',\n",
       " u'28',\n",
       " u'286',\n",
       " u'29',\n",
       " u'2nd',\n",
       " u'30',\n",
       " u'300',\n",
       " u'30602',\n",
       " u'31',\n",
       " u'32',\n",
       " u'33',\n",
       " u'34',\n",
       " u'35',\n",
       " u'36',\n",
       " u'37',\n",
       " u'38',\n",
       " u'386',\n",
       " u'39',\n",
       " u'3rd',\n",
       " u'40',\n",
       " u'400',\n",
       " u'403',\n",
       " u'408',\n",
       " u'41',\n",
       " u'42',\n",
       " u'43',\n",
       " u'44',\n",
       " u'45',\n",
       " u'46',\n",
       " u'47',\n",
       " u'48',\n",
       " u'486',\n",
       " u'49',\n",
       " u'4mb',\n",
       " u'4th',\n",
       " u'50',\n",
       " u'500',\n",
       " u'51',\n",
       " u'512',\n",
       " u'512k',\n",
       " u'52',\n",
       " u'53',\n",
       " u'54',\n",
       " u'542',\n",
       " u'55',\n",
       " u'56',\n",
       " u'57',\n",
       " u'58',\n",
       " u'59',\n",
       " u'5of5',\n",
       " u'5th',\n",
       " u'60',\n",
       " u'600',\n",
       " u'61',\n",
       " u'610',\n",
       " u'62',\n",
       " u'63',\n",
       " u'64',\n",
       " u'65',\n",
       " u'650',\n",
       " u'66',\n",
       " u'67',\n",
       " u'68',\n",
       " u'68030',\n",
       " u'68040',\n",
       " u'680x0',\n",
       " u'69',\n",
       " u'6th',\n",
       " u'70',\n",
       " u'700',\n",
       " u'706',\n",
       " u'71',\n",
       " u'72',\n",
       " u'73',\n",
       " u'74',\n",
       " u'7415',\n",
       " u'75',\n",
       " u'76',\n",
       " u'77',\n",
       " u'78',\n",
       " u'79',\n",
       " u'7th',\n",
       " u'80',\n",
       " u'800',\n",
       " u'80ns',\n",
       " u'81',\n",
       " u'82',\n",
       " u'83',\n",
       " u'84',\n",
       " u'85',\n",
       " u'86',\n",
       " u'87',\n",
       " u'88',\n",
       " u'89',\n",
       " u'90',\n",
       " u'900',\n",
       " u'91',\n",
       " u'92',\n",
       " u'93',\n",
       " u'94',\n",
       " u'95',\n",
       " u'950',\n",
       " u'96',\n",
       " u'97',\n",
       " u'9760',\n",
       " u'98',\n",
       " u'99',\n",
       " u'__',\n",
       " u'___',\n",
       " u'____',\n",
       " u'aa888',\n",
       " u'aaron',\n",
       " u'ab',\n",
       " u'abc',\n",
       " u'ability',\n",
       " u'able',\n",
       " u'abo',\n",
       " u'above',\n",
       " u'abraham',\n",
       " u'absence',\n",
       " u'absolute',\n",
       " u'absolutely',\n",
       " u'absolutes',\n",
       " u'ac',\n",
       " u'academic',\n",
       " u'acc',\n",
       " u'accelerator',\n",
       " u'accept',\n",
       " u'acceptable',\n",
       " u'accepted',\n",
       " u'accepting',\n",
       " u'access',\n",
       " u'according',\n",
       " u'account',\n",
       " u'accounts',\n",
       " u'accurate',\n",
       " u'achieve',\n",
       " u'achkar',\n",
       " u'acknowledge',\n",
       " u'acns',\n",
       " u'acquired',\n",
       " u'across',\n",
       " u'acs',\n",
       " u'acsu',\n",
       " u'act',\n",
       " u'action',\n",
       " u'actions',\n",
       " u'active',\n",
       " u'acts',\n",
       " u'actual',\n",
       " u'actually',\n",
       " u'ad',\n",
       " u'adam',\n",
       " u'adams',\n",
       " u'adapter',\n",
       " u'adb',\n",
       " u'add',\n",
       " u'added',\n",
       " u'adding',\n",
       " u'addition',\n",
       " u'additional',\n",
       " u'address',\n",
       " u'addresses',\n",
       " u'adirondack',\n",
       " u'admissions',\n",
       " u'admit',\n",
       " u'adopt',\n",
       " u'advance',\n",
       " u'advanced',\n",
       " u'advantage',\n",
       " u'advice',\n",
       " u'affected',\n",
       " u'afford',\n",
       " u'afraid',\n",
       " u'africa',\n",
       " u'after',\n",
       " u'afternoon',\n",
       " u'again',\n",
       " u'against',\n",
       " u'age',\n",
       " u'agnostic',\n",
       " u'ago',\n",
       " u'agree',\n",
       " u'agreed',\n",
       " u'agreement',\n",
       " u'ah',\n",
       " u'ahead',\n",
       " u'ahl',\n",
       " u'ai',\n",
       " u'aids',\n",
       " u'air',\n",
       " u'aisun3',\n",
       " u'al',\n",
       " u'alan',\n",
       " u'alberta',\n",
       " u'alchemy',\n",
       " u'alexander',\n",
       " u'alive',\n",
       " u'allen',\n",
       " u'allow',\n",
       " u'allowed',\n",
       " u'allows',\n",
       " u'almost',\n",
       " u'alone',\n",
       " u'along',\n",
       " u'alot',\n",
       " u'alpha',\n",
       " u'already',\n",
       " u'also',\n",
       " u'alt',\n",
       " u'alternative',\n",
       " u'although',\n",
       " u'altogether',\n",
       " u'alvin',\n",
       " u'always',\n",
       " u'am',\n",
       " u'am2x',\n",
       " u'amateur',\n",
       " u'amazing',\n",
       " u'ambiguous',\n",
       " u'america',\n",
       " u'american',\n",
       " u'americans',\n",
       " u'among',\n",
       " u'amount',\n",
       " u'amour',\n",
       " u'analysis',\n",
       " u'ancient',\n",
       " u'anderson',\n",
       " u'andersson',\n",
       " u'andrew',\n",
       " u'andreychuk',\n",
       " u'andy',\n",
       " u'angeles',\n",
       " u'angels',\n",
       " u'anger',\n",
       " u'angry',\n",
       " u'animals',\n",
       " u'ann',\n",
       " u'anna',\n",
       " u'announce',\n",
       " u'announced',\n",
       " u'announcer',\n",
       " u'announcers',\n",
       " u'annoying',\n",
       " u'another',\n",
       " u'answer',\n",
       " u'answered',\n",
       " u'answering',\n",
       " u'answers',\n",
       " u'antonio',\n",
       " u'anybody',\n",
       " u'anymore',\n",
       " u'anyone',\n",
       " u'anything',\n",
       " u'anytime',\n",
       " u'anyway',\n",
       " u'anyways',\n",
       " u'anywhere',\n",
       " u'apart',\n",
       " u'apologize',\n",
       " u'apostle',\n",
       " u'apostles',\n",
       " u'apparent',\n",
       " u'apparently',\n",
       " u'appear',\n",
       " u'appeared',\n",
       " u'appears',\n",
       " u'apple',\n",
       " u'applelink',\n",
       " u'appletalk',\n",
       " u'application',\n",
       " u'applications',\n",
       " u'applied',\n",
       " u'applies',\n",
       " u'apply',\n",
       " u'appointed',\n",
       " u'appreciate',\n",
       " u'appreciated',\n",
       " u'approach',\n",
       " u'appropriate',\n",
       " u'approved',\n",
       " u'apr',\n",
       " u'april',\n",
       " u'aps',\n",
       " u'aquinas',\n",
       " u'arbor',\n",
       " u'area',\n",
       " u'areas',\n",
       " u'aren',\n",
       " u'arena',\n",
       " u'argue',\n",
       " u'arguing',\n",
       " u'argument',\n",
       " u'arguments',\n",
       " u'army',\n",
       " u'around',\n",
       " u'arrogance',\n",
       " u'arrogant',\n",
       " u'art',\n",
       " u'articles',\n",
       " u'artificial',\n",
       " u'arts',\n",
       " u'ashley',\n",
       " u'aside',\n",
       " u'ask',\n",
       " u'asked',\n",
       " u'asking',\n",
       " u'asks',\n",
       " u'aspect',\n",
       " u'aspects',\n",
       " u'ass',\n",
       " u'assertion',\n",
       " u'assist',\n",
       " u'assistant',\n",
       " u'assists',\n",
       " u'associate',\n",
       " u'associated',\n",
       " u'association',\n",
       " u'assume',\n",
       " u'assumed',\n",
       " u'assuming',\n",
       " u'assumption',\n",
       " u'astray',\n",
       " u'atheism',\n",
       " u'atheist',\n",
       " u'atheists',\n",
       " u'athena',\n",
       " u'athens',\n",
       " u'athos',\n",
       " u'atlanta',\n",
       " u'att',\n",
       " u'attack',\n",
       " u'attempt',\n",
       " u'attempted',\n",
       " u'attempting',\n",
       " u'attempts',\n",
       " u'attendance',\n",
       " u'attention',\n",
       " u'atterlep',\n",
       " u'attitude',\n",
       " u'au',\n",
       " u'audience',\n",
       " u'austin',\n",
       " u'australia',\n",
       " u'author',\n",
       " u'authorities',\n",
       " u'authority',\n",
       " u'authors',\n",
       " u'auto',\n",
       " u'available',\n",
       " u'ave',\n",
       " u'avenue',\n",
       " u'average',\n",
       " u'avoid',\n",
       " u'awarded',\n",
       " u'awards',\n",
       " u'aware',\n",
       " u'away',\n",
       " u'awesome',\n",
       " u'awful',\n",
       " u'axelsson',\n",
       " u'back',\n",
       " u'background',\n",
       " u'backup',\n",
       " u'bad',\n",
       " u'baker',\n",
       " u'ball',\n",
       " u'ballentine',\n",
       " u'baltimore',\n",
       " u'band',\n",
       " u'baptism',\n",
       " u'barrasso',\n",
       " u'base',\n",
       " u'baseball',\n",
       " u'based',\n",
       " u'basic',\n",
       " u'basically',\n",
       " u'basis',\n",
       " u'basketball',\n",
       " u'battery',\n",
       " u'battle',\n",
       " u'bay',\n",
       " u'bbs',\n",
       " u'bc',\n",
       " u'bear',\n",
       " u'bears',\n",
       " u'beat',\n",
       " u'beautiful',\n",
       " u'became',\n",
       " u'because',\n",
       " u'become',\n",
       " u'becomes',\n",
       " u'becoming',\n",
       " u'been',\n",
       " u'before',\n",
       " u'beg',\n",
       " u'began',\n",
       " u'begin',\n",
       " u'beginning',\n",
       " u'begins',\n",
       " u'behalf',\n",
       " u'behavior',\n",
       " u'behaviour',\n",
       " u'behind',\n",
       " u'being',\n",
       " u'beings',\n",
       " u'belfour',\n",
       " u'belief',\n",
       " u'beliefs',\n",
       " u'believe',\n",
       " u'believed',\n",
       " u'believers',\n",
       " u'believes',\n",
       " u'believing',\n",
       " u'bell',\n",
       " u'bellows',\n",
       " u'belong',\n",
       " u'below',\n",
       " u'bench',\n",
       " u'benefit',\n",
       " u'beranek',\n",
       " u'berkeley',\n",
       " u'bernoulli',\n",
       " u'beside',\n",
       " u'besides',\n",
       " u'best',\n",
       " u'bet',\n",
       " u'better',\n",
       " u'between',\n",
       " u'beyond',\n",
       " u'bgsu',\n",
       " u'bgu',\n",
       " u'biased',\n",
       " u'bible',\n",
       " u'biblical',\n",
       " u'big',\n",
       " u'bigger',\n",
       " u'biggest',\n",
       " u'bill',\n",
       " u'birth',\n",
       " u'bishop',\n",
       " u'bit',\n",
       " u'bitnet',\n",
       " u'bits',\n",
       " u'black',\n",
       " u'blackhawks',\n",
       " u'blame',\n",
       " u'bless',\n",
       " u'blessed',\n",
       " u'blind',\n",
       " u'blindly',\n",
       " u'block',\n",
       " u'blood',\n",
       " u'blow',\n",
       " u'blue',\n",
       " u'blues',\n",
       " u'bnr',\n",
       " u'board',\n",
       " u'boarding',\n",
       " u'boards',\n",
       " u'bob',\n",
       " u'bobby',\n",
       " u'body',\n",
       " u'book',\n",
       " u'books',\n",
       " u'bookstore',\n",
       " u'boot',\n",
       " u'born',\n",
       " u'bos',\n",
       " u'boss',\n",
       " u'boston',\n",
       " u'both',\n",
       " u'bother',\n",
       " u'bothers',\n",
       " u'bottom',\n",
       " u'bought',\n",
       " u'boulder',\n",
       " u'bound',\n",
       " u'boundary',\n",
       " u'bourque',\n",
       " u'bowling',\n",
       " u'box',\n",
       " u'boxes',\n",
       " u'boy',\n",
       " u'brad',\n",
       " u'bradley',\n",
       " u'brain',\n",
       " u'branch',\n",
       " u'brand',\n",
       " u'braves',\n",
       " u'break',\n",
       " u'breaker',\n",
       " u'breaking',\n",
       " u'brent',\n",
       " u'breton',\n",
       " u'bri',\n",
       " u'brian',\n",
       " u'brief',\n",
       " u'brind',\n",
       " u'bring',\n",
       " u'brings',\n",
       " u'british',\n",
       " u'broadcast',\n",
       " u'broke',\n",
       " u'broken',\n",
       " u'brother',\n",
       " u'brothers',\n",
       " u'brought',\n",
       " u'brown',\n",
       " u'bruce',\n",
       " u'bruins',\n",
       " u'brunswick',\n",
       " u'bryan',\n",
       " u'btw',\n",
       " u'bu',\n",
       " u'buddhism',\n",
       " u'buddhist',\n",
       " u'buf',\n",
       " u'buffalo',\n",
       " u'bug',\n",
       " u'build',\n",
       " u'building',\n",
       " u'built',\n",
       " u'bunch',\n",
       " u'bure',\n",
       " u'burnaby',\n",
       " u'burned',\n",
       " u'burns',\n",
       " u'bus',\n",
       " u'business',\n",
       " u'butt',\n",
       " u'button',\n",
       " u'buy',\n",
       " u'buying',\n",
       " u'byler',\n",
       " u'bytes',\n",
       " u'c610',\n",
       " u'c650',\n",
       " u'ca',\n",
       " u'cable',\n",
       " u'cables',\n",
       " u'cache',\n",
       " u'cadkey',\n",
       " u'cage',\n",
       " u'cal',\n",
       " u'calder',\n",
       " u'calgary',\n",
       " u'california',\n",
       " u'call',\n",
       " u'called',\n",
       " u'calling',\n",
       " u'calls',\n",
       " u'calvin',\n",
       " u'cam',\n",
       " u'came',\n",
       " u'camp',\n",
       " u'campbell',\n",
       " u'campus',\n",
       " u'canada',\n",
       " u'canadian',\n",
       " u'canadians',\n",
       " u'canadiens',\n",
       " u'cannot',\n",
       " u'canon',\n",
       " u'canucks',\n",
       " u'capability',\n",
       " u'capable',\n",
       " u'capacity',\n",
       " u'cape',\n",
       " u'capital',\n",
       " u'capitals',\n",
       " u'caps',\n",
       " u'captain',\n",
       " u'captains',\n",
       " u'car',\n",
       " u'caralv',\n",
       " u'card',\n",
       " u'cardinal',\n",
       " u'cards',\n",
       " u'care',\n",
       " u'career',\n",
       " u'careful',\n",
       " u'carefully',\n",
       " u'cares',\n",
       " u'carleton',\n",
       " u'carnegie',\n",
       " u'carol',\n",
       " u'carolina',\n",
       " u'carpenter',\n",
       " u'carried',\n",
       " u'carry',\n",
       " u'carrying',\n",
       " u'carson',\n",
       " u'cartridge',\n",
       " u'case',\n",
       " u'cases',\n",
       " u'cassels',\n",
       " u'cast',\n",
       " u'category',\n",
       " u'catholic',\n",
       " u'catholics',\n",
       " u'caught',\n",
       " u'cause',\n",
       " u'caused',\n",
       " u'causes',\n",
       " u'cb',\n",
       " u'cbc',\n",
       " u'cbnewsh',\n",
       " u'cbnewsk',\n",
       " u'cc',\n",
       " u'ccu',\n",
       " u'cd',\n",
       " u'cd300',\n",
       " u'cec1',\n",
       " u'celebrate',\n",
       " u'cent',\n",
       " u'center',\n",
       " u'central',\n",
       " u'centre',\n",
       " u'centris',\n",
       " u'centuries',\n",
       " u'century',\n",
       " u'ceremonial',\n",
       " u'ceremony',\n",
       " u'certain',\n",
       " u'certainly',\n",
       " u'certainty',\n",
       " u'cf',\n",
       " u'cgsvax',\n",
       " u'chain',\n",
       " u'champions',\n",
       " u'championship',\n",
       " u'championships',\n",
       " u'champs',\n",
       " u'chance',\n",
       " u'chances',\n",
       " u'change',\n",
       " u'changed',\n",
       " u'changes',\n",
       " u'changing',\n",
       " u'channel',\n",
       " u'chapter',\n",
       " u'character',\n",
       " u'charge',\n",
       " u'charles',\n",
       " u'charlie',\n",
       " u'cheap',\n",
       " u'cheaper',\n",
       " u'check',\n",
       " u'checked',\n",
       " u'checking',\n",
       " u'cheers',\n",
       " u'chelios',\n",
       " u'chem',\n",
       " u'chemistry',\n",
       " u'cherry',\n",
       " u'chhabra',\n",
       " u'chi',\n",
       " u'chicago',\n",
       " u'chief',\n",
       " u'child',\n",
       " u'children',\n",
       " u'chip',\n",
       " u'chips',\n",
       " u'choice',\n",
       " u'choices',\n",
       " u'choose',\n",
       " u'choosing',\n",
       " u'chose',\n",
       " u'chosen',\n",
       " u'chris',\n",
       " u'christ',\n",
       " u'christian',\n",
       " u'christianity',\n",
       " u'christians',\n",
       " u'christopher',\n",
       " u'chuck',\n",
       " u'church',\n",
       " u'churches',\n",
       " u'ciccarelli',\n",
       " u'circle',\n",
       " u'circumstances',\n",
       " u'cis',\n",
       " u'cited',\n",
       " u'cities',\n",
       " u'city',\n",
       " u'civil',\n",
       " u'claim',\n",
       " u'claimed',\n",
       " u'claiming',\n",
       " u'claims',\n",
       " u'claremont',\n",
       " u'clark',\n",
       " u'clarke',\n",
       " u'clarkson',\n",
       " u'class',\n",
       " u'classic',\n",
       " u'classy',\n",
       " u'claude',\n",
       " u'clean',\n",
       " u'clear',\n",
       " u'clearly',\n",
       " u'clement',\n",
       " u'cleveland',\n",
       " u'clh',\n",
       " u'clinton',\n",
       " u'clock',\n",
       " u'close',\n",
       " u'closed',\n",
       " u'closely',\n",
       " u'closer',\n",
       " u'closest',\n",
       " u'club',\n",
       " u'cmu',\n",
       " u'co',\n",
       " u'coach',\n",
       " u'coaches',\n",
       " u'coaching',\n",
       " u'code',\n",
       " u'coffey',\n",
       " u'cohen',\n",
       " u'col',\n",
       " u'collection',\n",
       " u'college',\n",
       " u'collingridge',\n",
       " u'color',\n",
       " u'colorado',\n",
       " u'colors',\n",
       " u'colostate',\n",
       " u'colour',\n",
       " u'columbia',\n",
       " u'com',\n",
       " u'combination',\n",
       " u'combined',\n",
       " u'come',\n",
       " u'comes',\n",
       " u'comics',\n",
       " u'coming',\n",
       " u'command',\n",
       " u'commandments',\n",
       " u'commands',\n",
       " u'comment',\n",
       " u'commentary',\n",
       " u'comments',\n",
       " u'commitment',\n",
       " u'committed',\n",
       " u'common',\n",
       " u'communication',\n",
       " u'communications',\n",
       " u'communion',\n",
       " u'community',\n",
       " u'comp',\n",
       " u'companies',\n",
       " u'company',\n",
       " u'comparable',\n",
       " u'comparative',\n",
       " u'compare',\n",
       " u'compared',\n",
       " u'comparing',\n",
       " u'comparison',\n",
       " u'compatible',\n",
       " u'complain',\n",
       " u'complete',\n",
       " u'completed',\n",
       " u'completely',\n",
       " u'complex',\n",
       " u'complicated',\n",
       " u'compuserve',\n",
       " u'computer',\n",
       " u'computers',\n",
       " u'computing',\n",
       " u'con',\n",
       " u'concept',\n",
       " u'conception',\n",
       " u'concern',\n",
       " u'concerned',\n",
       " u'concerning',\n",
       " u'conclude',\n",
       " u'conclusion',\n",
       " u'conclusions',\n",
       " u'condemned',\n",
       " u'conditions',\n",
       " u'conditt',\n",
       " u'conference',\n",
       " u'configuration',\n",
       " u'conflict',\n",
       " u'confused',\n",
       " u'confusing',\n",
       " u'confusion',\n",
       " u'congregation',\n",
       " u'connect',\n",
       " u'connected',\n",
       " u'connection',\n",
       " u'connector',\n",
       " u'consecutive',\n",
       " u'consensus',\n",
       " u'conservative',\n",
       " u'consider',\n",
       " u'considered',\n",
       " u'considering',\n",
       " u'consistent',\n",
       " u'consistently',\n",
       " u'constant',\n",
       " u'constantly',\n",
       " u'consultant',\n",
       " u'contact',\n",
       " u'contained',\n",
       " u'contains',\n",
       " u'content',\n",
       " u'context',\n",
       " u'continent',\n",
       " u'continue',\n",
       " u'continued',\n",
       " u'continues',\n",
       " u'contract',\n",
       " u'contradict',\n",
       " u'contradiction',\n",
       " u'contradictory',\n",
       " u'contrary',\n",
       " u'contribution',\n",
       " u'control',\n",
       " u'conversion',\n",
       " u'convert',\n",
       " u'converted',\n",
       " u'convictions',\n",
       " u'convince',\n",
       " u'cook',\n",
       " u'cool',\n",
       " u'coos',\n",
       " u'copies',\n",
       " u'coprocessor',\n",
       " u'copy',\n",
       " u'cor',\n",
       " u'cordially',\n",
       " u'corinthians',\n",
       " u'corner',\n",
       " u'corp',\n",
       " u'corporation',\n",
       " u'correct',\n",
       " u'correctly',\n",
       " u'cost',\n",
       " u'costs',\n",
       " u'cote',\n",
       " u'could',\n",
       " u'couldn',\n",
       " u'council',\n",
       " u'count',\n",
       " u'counted',\n",
       " u'countries',\n",
       " u'country',\n",
       " u'couple',\n",
       " u'course',\n",
       " u'court',\n",
       " u'courtnall',\n",
       " u'cover',\n",
       " u'coverage',\n",
       " u'covered',\n",
       " u'covington',\n",
       " u'cpu',\n",
       " u'craft',\n",
       " u'craig',\n",
       " u'crap',\n",
       " u'crashes',\n",
       " u'craven',\n",
       " u'crazy',\n",
       " u'create',\n",
       " u'created',\n",
       " u'creation',\n",
       " u'creator',\n",
       " u'credit',\n",
       " u'critical',\n",
       " u'criticism',\n",
       " u'cross',\n",
       " u'crowd',\n",
       " u'cs',\n",
       " u'csd',\n",
       " u'cso',\n",
       " u'cu',\n",
       " u'cullen',\n",
       " u'cult',\n",
       " u'cultural',\n",
       " u'culture',\n",
       " u'cunixb',\n",
       " u'cunixc',\n",
       " u'cup',\n",
       " u'cups',\n",
       " u'curious',\n",
       " u'current',\n",
       " u'currently',\n",
       " u'curtis',\n",
       " u'cut',\n",
       " u'cwis',\n",
       " u'cwru',\n",
       " u'czech',\n",
       " u'd88',\n",
       " u'dahlen',\n",
       " u'daily',\n",
       " u'dal',\n",
       " u'dale',\n",
       " u'dalhousie',\n",
       " u'dallas',\n",
       " u'damn',\n",
       " u'damphousse',\n",
       " u'dan',\n",
       " u'danger',\n",
       " u'dangerous',\n",
       " u'daniel',\n",
       " u'dare',\n",
       " u'darius',\n",
       " u'darius_lecointe',\n",
       " u'darren',\n",
       " u'dartmouth',\n",
       " u'daryl',\n",
       " ...]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorizer.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'connect'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorizer.get_feature_names()[877]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'From: erik@cheshire.oxy.edu (Erik Adams)\\nSubject: HELP!!  My Macintosh \"luggable\" has lines on its screen!\\nOrganization: Occidental College, Los Angeles, CA 90041 USA.\\nDistribution: comp\\nLines: 20\\n\\nOkay, I don\\'t use it very much, but I would like for it to keep working\\ncorrectly, at least as long as Apple continues to make System software\\nthat will run on it, if slowly :-)\\n\\nHere is the problem:  When the screen is tilted too far back, vertical\\nlines appear on the screen.  They are every 10 pixels or so, and seem\\nto be affected somewhat by opening windows and pulling down menus.\\nIt looks to a semi-technical person like there is a loose connection\\nbetween the screen and the rest of the computer.\\n\\nI am open to suggestions that do not involve buying a new computer,\\nor taking this one to the shop.  I would also like to not have\\nto buy one of Larry Pina\\'s books.  I like Larry, but I\\'m not sure\\nI feel strongly enough about the computer to buy a service manual\\nfor it.\\n\\nOn a related note:  what does the monitor connector connect to?\\n\\nErik\\n\\n'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "simple_dataset.data[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## KMeans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 0 2 ..., 0 2 1]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.cluster import KMeans\n",
    "\n",
    "model = KMeans(n_clusters=3, random_state=1)\n",
    "preds = model.fit_predict(matrix.toarray())\n",
    "print preds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 0 1 ..., 0 1 2]\n"
     ]
    }
   ],
   "source": [
    "print simple_dataset.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0483961733258\n"
     ]
    }
   ],
   "source": [
    "mapping = {2 : 1, 1: 2, 0: 0}\n",
    "mapped_preds = [mapping[pred] for pred in preds]\n",
    "print float(sum(mapped_preds != simple_dataset.target)) / len(simple_dataset.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.985360318588\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.cross_validation import cross_val_score\n",
    "clf = LogisticRegression()\n",
    "print cross_val_score(clf, matrix, simple_dataset.target).mean()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Более сложная выборка"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dataset = fetch_20newsgroups(\n",
    "    subset='train', \n",
    "    categories=['comp.sys.mac.hardware', 'comp.os.ms-windows.misc', 'comp.graphics'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 1 2 ..., 0 2 0]\n",
      "[2 1 1 ..., 2 0 2]\n"
     ]
    }
   ],
   "source": [
    "matrix = vectorizer.fit_transform(dataset.data)\n",
    "model = KMeans(n_clusters=3, random_state=42)\n",
    "preds = model.fit_predict(matrix.toarray())\n",
    "print preds\n",
    "print dataset.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.261266400456\n"
     ]
    }
   ],
   "source": [
    "mapping = {2 : 0, 1: 1, 0: 2}\n",
    "mapped_preds = [mapping[pred] for pred in preds]\n",
    "print float(sum(mapped_preds != dataset.target)) / len(dataset.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.917279226713\n"
     ]
    }
   ],
   "source": [
    "clf = LogisticRegression()\n",
    "print cross_val_score(clf, matrix, dataset.target).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SVD + KMeans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 2 1 ..., 0 1 0]\n",
      "[2 1 1 ..., 2 0 2]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.decomposition import TruncatedSVD\n",
    "\n",
    "model = KMeans(n_clusters=3, random_state=42)\n",
    "svd = TruncatedSVD(n_components=1000, random_state=123)\n",
    "features = svd.fit_transform(matrix)\n",
    "preds = model.fit_predict(features)\n",
    "print preds\n",
    "print dataset.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.206503137479\n"
     ]
    }
   ],
   "source": [
    "mapping = {0 : 2, 1: 0, 2: 1}\n",
    "mapped_preds = [mapping[pred] for pred in preds]\n",
    "print float(sum(mapped_preds != dataset.target)) / len(dataset.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2 0 1 ..., 2 1 2]\n",
      "[2 1 1 ..., 2 0 2]\n"
     ]
    }
   ],
   "source": [
    "model = KMeans(n_clusters=3, random_state=42)\n",
    "svd = TruncatedSVD(n_components=200, random_state=123)\n",
    "features = svd.fit_transform(matrix)\n",
    "preds = model.fit_predict(features)\n",
    "print preds\n",
    "print dataset.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.900741585853\n",
      "0.674272675414\n",
      "0.705647461495\n",
      "0.893896177981\n",
      "0.205362236167\n",
      "0.620079863092\n"
     ]
    }
   ],
   "source": [
    "import itertools\n",
    "def validate_with_mappings(preds, target, dataset):\n",
    "    permutations = itertools.permutations([0, 1, 2])\n",
    "    for a, b, c in permutations:\n",
    "        mapping = {2 : a, 1: b, 0: c}\n",
    "        mapped_preds = [mapping[pred] for pred in preds]\n",
    "        print float(sum(mapped_preds != target)) / len(target)\n",
    "        \n",
    "validate_with_mappings(preds, dataset.target, dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2 1 0 ..., 2 0 2]\n",
      "[2 1 1 ..., 2 0 2]\n",
      "0.713063320023\n",
      "0.845407872219\n",
      "0.889332572732\n",
      "0.70051340559\n",
      "0.586423274387\n",
      "0.265259555048\n"
     ]
    }
   ],
   "source": [
    "model = KMeans(n_clusters=3, random_state=42)\n",
    "svd = TruncatedSVD(n_components=200, random_state=321)\n",
    "features = svd.fit_transform(matrix)\n",
    "preds = model.fit_predict(features)\n",
    "print preds\n",
    "print dataset.target\n",
    "validate_with_mappings(preds, dataset.target, dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Итоги"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Получили интерпретируемый результат на обеих выборках\n",
    "1. Реальность, однако, намного более жестока\n",
    "1. Попробовали использовать AgglomerativeClustering и KMeans"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}