2018/day24_recommender_system.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Book Recommender System"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "%matplotlib inline \n",
    "sns.set()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Data\n",
    "\n",
    "Book-Crossings is a book ratings dataset compiled by Cai-Nicolas Ziegler. It contains 1.1 million ratings of 270,000 books by 90,000 users. The ratings are on a scale from 1 to 10.\n",
    "\n",
    "The data consists of three tables: ratings, books info, and users info."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "b'Skipping line 6452: expected 8 fields, saw 9\\nSkipping line 43667: expected 8 fields, saw 10\\nSkipping line 51751: expected 8 fields, saw 9\\n'\n",
      "b'Skipping line 92038: expected 8 fields, saw 9\\nSkipping line 104319: expected 8 fields, saw 9\\nSkipping line 121768: expected 8 fields, saw 9\\n'\n",
      "b'Skipping line 144058: expected 8 fields, saw 9\\nSkipping line 150789: expected 8 fields, saw 9\\nSkipping line 157128: expected 8 fields, saw 9\\nSkipping line 180189: expected 8 fields, saw 9\\nSkipping line 185738: expected 8 fields, saw 9\\n'\n",
      "b'Skipping line 209388: expected 8 fields, saw 9\\nSkipping line 220626: expected 8 fields, saw 9\\nSkipping line 227933: expected 8 fields, saw 11\\nSkipping line 228957: expected 8 fields, saw 10\\nSkipping line 245933: expected 8 fields, saw 9\\nSkipping line 251296: expected 8 fields, saw 9\\nSkipping line 259941: expected 8 fields, saw 9\\nSkipping line 261529: expected 8 fields, saw 9\\n'\n",
      "/Users/adhiman/anaconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py:2728: DtypeWarning: Columns (3) have mixed types. Specify dtype option on import or set low_memory=False.\n",
      "  interactivity=interactivity, compiler=compiler, result=result)\n"
     ]
    }
   ],
   "source": [
    "books = pd.read_csv('data/BX-Books.csv', sep=';', error_bad_lines=False, encoding=\"latin-1\")\n",
    "books.columns = ['ISBN', 'bookTitle', 'bookAuthor', 'yearOfPublication', 'publisher', 'imageUrlS', \n",
    "                 'imageUrlM', 'imageUrlL']\n",
    "\n",
    "users = pd.read_csv('data/BX-Users.csv', sep=';', error_bad_lines=False, encoding=\"latin-1\")\n",
    "users.columns = ['userID', 'Location', 'Age']\n",
    "\n",
    "ratings = pd.read_csv('data/BX-Book-Ratings.csv', sep=';', error_bad_lines=False, encoding=\"latin-1\")\n",
    "ratings.columns = ['userID', 'ISBN', 'bookRating']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ratings Data\n",
    "The ratings data set provides a list of ratings that users have given to books. It includes 1,149,780 records and 3 fields: userID, ISBN, and rating."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1149780, 3)\n",
      "['userID', 'ISBN', 'bookRating']\n"
     ]
    }
   ],
   "source": [
    "print(ratings.shape)\n",
    "print(list(ratings.columns))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userID</th>\n",
       "      <th>ISBN</th>\n",
       "      <th>bookRating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>276725</td>\n",
       "      <td>034545104X</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>276726</td>\n",
       "      <td>0155061224</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>276727</td>\n",
       "      <td>0446520802</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>276729</td>\n",
       "      <td>052165615X</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>276729</td>\n",
       "      <td>0521795028</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userID        ISBN  bookRating\n",
       "0  276725  034545104X           0\n",
       "1  276726  0155061224           5\n",
       "2  276727  0446520802           0\n",
       "3  276729  052165615X           3\n",
       "4  276729  0521795028           6"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAEhCAYAAAC6Hk0fAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAGy1JREFUeJzt3X2UXVWZ5/FvVYoQoxWsXpaO9IiIyNMz2iIBVkCBBATTgIpvy8Fe2gZkXGpsRUUFm5eArY0I2DIGsUEMOr6MYis0DhA1iBF5EQGFkX4iotKj3WPAxBQGxCQ1f5xTUhSpqpvse26l6n4/a2Vx77777GfvIrm/Oufce07P8PAwkiSV6J3qCUiSpj/DRJJUzDCRJBUzTCRJxQwTSVIxw0SSVKxvqicgtVtEDAN3AZuBYWAusAF4a2beOsm2JwCzM/PCiHgL8OTMPLsNc9od+BlwZ93UCzwI/GNmfrnucxZwT2Z+doJxTgd+lJlXbOW1P21f/wwGM/P+bZjj/sCbMvMtEbEfcHJmvqbV7dXdDBPNVIeOfiONiJOA/wEcOMl2B1EFEZl5UZvn9FBmvmDUnJ4JfDsiNmfmVzPz9BbGOAz4ydZeaHH7iTwX+M/1WLcCBolaZphoxouIPmA34Lf186cBnwKeBvwn4JfAa4EXAS8HjoiIh4BB4CmZ+faI+AWwAnhxPdZnM/O0eryTgTcBQ8B3gVdk5u6TzSszf1nvabwX+GpErADuysxzI+JM4JXAI8ADwBLgVcB+wEcjYjNwDPBnwLOBq+r13JWZ59YlPlTvbfQCp2bmVRGxBHhNZr60nvsSqtB4K3AWsEtEfAa4DPhEZj4vInYBlgMvoNrTuxr4QGZuioiHgbOBlwBPB87JzE9OtnbNPJ4z0Ux1XUT8OCJ+Dayp246r/3sscGNmHgjsAWwE3pCZXwOuBD6Wmcu3MuaTMvNg4IXASRHxrIhYTPVGvz+wL9C/jfP8EfCXoxsi4hnAicD+mbkfsBJYUM/pVuC99VwB5mbmczPz/VsZ+97MnA+8HrgsIgbHm0Rm/htwOrA6M48b8/IFVIH2l1RhtjdwUv3azsD9mflCqlD6WETMaXHtmkEME81Uh2bm84GXUp0zuS4zfwOQmR8Hvh8R7wYuBJ4HPKmFMa+ot/8V8BuqvYKjgK9k5vrMHKb6DX5bDFOF2Wi/ogqZ2yLiXOCOzPz6ONt/b4KxL6rnexfVobHJDvGN50iqvZThzPxDPe6Ro14fOX9zG1W4PHE762gaM0w0o2XmbcC7gBX1SXAi4iNUh3TWAv9E9Zt/TwvDPTTq8XC9zaYx227exinuz6Mn5UfmvAVYSLXH8wDVb/vnjLP9gxOMPXouvcAfeXTeI2a3MMfeervRz3ca9fyhet4jfVr5WWqGMUw042XmF4FbgI/VTYupPkX1Oao9jCOAWfVrm3jsG+VkvgG8uj6vANW5k5aunhoRewGnAeeNad+b6kMAd2fmP9Tz3n875rekHm8+sCdwM1WAPi8i5kTETjz2JPt4Y18LvD0ieiJiZ+DNwDdbnIO6hCfg1S3eDvy4PsdxFnBuRHyQ6rf171G92UJ1cvn8iGhp0MxcFREXAzdGxEbg//D4w1YjnhARd9SPtwAPA6dk5jfGjPmjiPgycGtEPEj1m/876pevBP4hIlrZo9gjIm6nCrdjM/O3EbESuB74V+DfgeuA59f9bwLOiIh/pjpPMuIdVJ+Eu5NqT+Ya4EMt1FcX6fES9NL2q7+P8cLMvKB+/m6qk+X/bWpnJnWWeyZSmTXA+yPizVR7APdRHQaSuop7JpKkYp6AlyQVM0wkScUME0lSMcNEklTMMJEkFTNMJEnFDBNJUjHDRJJUzDCRJBUzTCRJxQwTSVIxw0SSVMwwkSQVM0wkScW68n4ma9cOFV13f2BgLuvWjXczveZ0W92prO2au6N2t9UtrT042N8z3mvumWyHvr5Zk3ey7rSu7Zq7o3a31W2ytmEiSSpmmEiSihkmkqRihokkqZhhIkkqZphIkooZJpKkYoaJJKmYYSJJKtaVl1OZzPFnr9rubS89+bA2zkSSpofGwiQilgBL6qdzgBcAi4CPA5uAlZl5ZkT0AhcCewN/AE7IzHsi4oCSvk2tS5L0eI0d5srMFZm5KDMXAT8E3gFcBPw1cBCwICLmA68A5mTmgcDJwHn1EKV9JUkd0vg5k4jYD3gu8CVg58z8WWYOA9cCL6YKgGsAMvMmYL+ImNeGvpKkDunEOZMPAGcC84ANo9qHgD3q9t+Nat/cpr7jGhiY29iVMwcH+xsZt1Pj72h1p7K2a+6O2t1Wt6najYZJRDwZ+IvMvK7egxi9gn5gPTB3THsvVTiU9h1Xk/cRWLt2qLGxBwf7Gx1/R6s7lbVdc3fU7ra6pbUnCqGmD3MdAnwLIDM3AI9ExLMjogdYDKwGbgCOAqhPpN/Zpr6SpA5p+jBXAPeOev4W4PPALKpPXd0cET8AjoiI7wM9wHHt6NvwuiRJozQaJpn50THPbwIOGNO2hSoMxm5b1FeS1Dl+A16SVMwwkSQVM0wkScUME0lSMcNEklTMMJEkFTNMJEnFDBNJUjHDRJJUzDCRJBUzTCRJxQwTSVIxw0SSVMwwkSQVM0wkScUME0lSMcNEklTMMJEkFTNMJEnFDBNJUjHDRJJUrK/JwSPiFODlwGzgQuB6YAUwDNwFLM3MLRFxBnA0sAk4MTNviYg9S/s2uTZJ0qMa2zOJiEXAC4EXAQuBZwDnA6dm5sFAD3BMRMyvX18AHAssr4co6tvUuiRJj9fkYa7FwJ3A14B/Aa4C9qXaOwG4GjgcOAhYmZnDmXkf0BcRg23oK0nqkCYPcz0FeCbwUuBZwJVAb2YO168PAbsA84AHRm030t5T2HdcAwNz6eubtZ3LmtjgYH8j43Zq/B2t7lTWds3dUbvb6jZVu8kweQD418x8BMiIeJjqUNeIfmA9sKF+PLZ9S2Hfca1bt3GbFrIt1q4damzswcH+Rsff0epOZW3X3B21u61uae2JQqjJw1zfA/4qInoiYlfgicC363MpAEcCq4EbgMUR0RsRu1HtvdwP3F7YV5LUIY3tmWTmVRFxCHALVWgtBX4OXBwRs4G7gcszc3NErAZuHNUP4D0lfZtalyTp8Rr9aHBmvm8rzQu30m8ZsGxM25rSvpKkzvBLi5KkYoaJJKmYYSJJKmaYSJKKGSaSpGKGiSSpmGEiSSpmmEiSihkmkqRihokkqZhhIkkqZphIkooZJpKkYoaJJKmYYSJJKmaYSJKKGSaSpGKGiSSpmGEiSSpmmEiSihkmkqRifU0OHhG3A7+rn/4c+BTwcWATsDIzz4yIXuBCYG/gD8AJmXlPRBxQ0rfJdUmSHquxMImIOQCZuWhU2x3Aq4F7gW9ExHxgd2BOZh5Yh8J5wDHARSV9M/O2ptYmSXqsJvdM9gbmRsTKus4yYOfM/BlARFwLvBh4OnANQGbeFBH7RcS8NvQ1TCSpQ5oMk43AucAlwHOAq4H1o14fAvYA5vHooTCAzXXbhsK+4xoYmEtf36xtWErrBgf7Gxm3U+PvaHWnsrZr7o7a3Va3qdpNhska4J7MHAbWRMTvgD8b9Xo/VbjMrR+P6KUKh/7CvuNat27jNi1kW6xdO9TY2IOD/Y2Ov6PVncrarrk7andb3dLaE4VQk5/mOp7qnAYRsStVEPw+Ip4dET3AYmA1cANwVN3vAODOzNwAPFLYV5LUIU3umXwaWBER3wOGqcJlC/B5YBbVp65ujogfAEdExPeBHuC4evu3lPRtcF2SpDEaC5PMfAT46628dMCYfluowmDs9jeV9JUkdY5fWpQkFTNMJEnFDBNJUjHDRJJUzDCRJBUzTCRJxQwTSVIxw0SSVMwwkSQVM0wkScUME0lSMcNEklTMMJEkFTNMJEnFDBNJUjHDRJJUzDCRJBUzTCRJxVoKk4h47lbavE2uJAmY5B7wEfEiYBZwSUS8CegZtd1FwF7NTk+SNB1MGCbAEcBC4OnAWaPaNwGfampSkqTpZcIwycxlABHxhsz83LYOHhFPBX5IFUqbgBXAMHAXsDQzt0TEGcDR9esnZuYtEbFnad9tnaskafu1egL+uxHx0Yj4dERcOvJnog0iYieqvZeH6qbzgVMz82Cqw2XHRMR8qj2fBcCxwPJ29G1xTZKkNmk1TL5M9Ua9Grh+1J+JnEt1XuXX9fN9R21zNXA4cBCwMjOHM/M+oC8iBtvQV5LUQZOdMxmxU2ae1OqgEbEEWJuZ10bEKXVzT2YO14+HgF2AecADozYdaS/tO6GBgbn09c1qdTnbZHCwv5FxOzX+jlZ3Kmu75u6o3W11m6rdaph8LyJeBlybmY+00P94YDgiDgdeAHwWeOqo1/uB9cCG+vHY9i2FfSe0bt3GFpawfdauHWps7MHB/kbH39HqTmVt19wdtbutbmntiUKo1cNcrwGuAB6OiC31n83jdc7MQzJzYWYuAu4A/ga4OiIW1V2OpDpkdgOwOCJ6I2I3oDcz7wduL+wrSeqglvZMMnPXNtR6D3BxRMwG7gYuz8zNEbEauJEq2Ja2o28b5ipJ2gYthUlEnL619sw8a2vtY/osGvV04VZeXwYsG9O2prSvJKlzWj3M1TPqz2zg5cDTmpqUJGl6afUw15mjn0fEB4GVjcxIkjTtbO9Vg58E7NbOiUiSpq9Wz5n8nOpyJVAF0ABwTlOTkiRNL61+z2TRqMfDwPrM3ND+6UiSpqNWD3PdBxwFnAdcACyJCG+sJUkCWt8zOQd4DnAp1Se6jgOeDbyzoXlJkqaRVsPkJcA+I5d2j4hvAHc2NitJ0rTS6qGqPmCnMc/HvZyKJKm7tLpn8nnguoj4Yv38dcAXmpmSJGm6mTRMImIAuBi4DXgxcBjwj9tz50VJ0sw04WGuiNgH+Amwb2Zek5nvBa4Fzo6I53digpKkHd9k50zOBV6XmdeMNGTmB6juV3J+kxOTJE0fk4XJQGZ+Z2xjZl4LPKWRGUmSpp3JwmSnrX05sW6b3cyUJEnTzWRhcj1wxlbaTwVubf90JEnT0WSf5joF+N8R8Uaq2+8+DMwHfkN1TxNJkiYOk8wciohDgEOBfYAtwPLM9D7rkqQ/mfR7Jpk5DKyq/0iS9Dhe+VeSVMwwkSQVa/XaXNssImZRXYYlqC4KeRzV5etXUN1g6y5gaWZuiYgzgKOBTcCJmXlLROxZ2reptUmSHqvJPZOXAWTmi4DTqb4xfz5wamYeTBUsx0TEfGAhsAA4Flheb1/Ut8F1SZLGaCxMMvPrwJvrp88E/h+wL9V3VwCuBg4HDgJWZuZwZt4H9EXEYBv6SpI6pLHDXACZuSkiLgNeCbwGeGn96TCAIWAXYB7wwKjNRtp7CvuOa2BgLn19s7Z7XRMZHOxvZNxOjb+j1Z3K2q65O2p3W92majcaJgCZ+caIeD9wM/CEUS/1A+uBDfXjse1bCvuOa926jdu2iG2wdu1QY2MPDvY3Ov6OVncqa7vm7qjdbXVLa08UQo0d5oqIN0TEKfXTjVRv+LdGxKK67UhgNXADsDgieiNiN6A3M+8Hbi/sK0nqkCb3TP4Z+ExEfJfqlr8nAncDF0fE7Prx5Zm5OSJWAzdShdvSevv3lPRtcF2SpDEaC5PM/D3w2q28tHArfZcBy8a0rSntK0nqDL+0KEkqZphIkooZJpKkYoaJJKmYYSJJKmaYSJKKGSaSpGKGiSSpmGEiSSpmmEiSihkmkqRihokkqZhhIkkqZphIkooZJpKkYoaJJKmYYSJJKmaYSJKKGSaSpGKGiSSpmGEiSSrW18SgEbETcCmwO7Az8PfAT4AVwDBwF7A0M7dExBnA0cAm4MTMvCUi9izt28S6JElb19SeyeuBBzLzYOBI4BPA+cCpdVsPcExEzAcWAguAY4Hl9fZFfRtakyRpHE2FyVeA00Y93wTsC1xfP78aOBw4CFiZmcOZeR/QFxGDbegrSeqgRg5zZeaDABHRD1wOnAqcm5nDdZchYBdgHvDAqE1H2nsK+05oYGAufX2ztmNlkxsc7G9k3E6Nv6PVncrarrk7andb3aZqNxImABHxDOBrwIWZ+YWIOGfUy/3AemBD/Xhs+5bCvhNat25j6wvZRmvXDjU29uBgf6Pj72h1p7K2a+6O2t1Wt7T2RCHUyGGuiHgasBJ4f2ZeWjffHhGL6sdHAquBG4DFEdEbEbsBvZl5fxv6SpI6qKk9kw8AA8BpETFy7uSdwAURMRu4G7g8MzdHxGrgRqpgW1r3fQ9w8fb2bWhNkqRxNHXO5J1U4THWwq30XQYsG9O2prSvJKlz/NKiJKmYYSJJKmaYSJKKGSaSpGKGiSSpmGEiSSpmmEiSihkmkqRihokkqZhhIkkqZphIkooZJpKkYoaJJKmYYSJJKmaYSJKKGSaSpGKGiSSpmGEiSSpmmEiSihkmkqRihokkqVhfk4NHxALgI5m5KCL2BFYAw8BdwNLM3BIRZwBHA5uAEzPzlnb0bXJdkqTHamzPJCLeB1wCzKmbzgdOzcyDgR7gmIiYDywEFgDHAsvb0bepNUmStq7JPZOfAa8CPlc/3xe4vn58NfASIIGVmTkM3BcRfREx2Ia+X2twXZKmiaWr3le0/fLDzmnTTGa+xsIkM78aEbuPauqpgwBgCNgFmAc8MKrPSHtp3wkNDMylr2/Wti2oRYOD/Y2M26nxd7S6U1nbNXdP7fE0OaeZ9rNu9JzJGKPPY/QD64EN9eOx7aV9J7Ru3cZtmfc2Wbt2qLGxBwf7Gx1/R6s7lbVdc/fUnkhTc5quP+uJQqiTn+a6PSIW1Y+PBFYDNwCLI6I3InYDejPz/jb0lSR1UCf3TN4DXBwRs4G7gcszc3NErAZupAq2pe3o27EVSZKAhsMkM38BHFA/XkP1aayxfZYBy8a0FfeVJHWOX1qUJBXr5GEuSVLDPnn2d4q2f+vJi7ZrO8NEUuOOP3vVdm976cmHtXEmaoqHuSRJxdwzkaQGrDlhyfivTbLtXpesaOdUOsI9E0lSMcNEklTMMJEkFTNMJEnFPAEvddgHfvDT7d72w/s/p40zkdrHPRNJUjHDRJJUzDCRJBXznInURbysiZrinokkqZhhIkkqZphIkooZJpKkYoaJJKmYYSJJKmaYSJKKzYjvmUREL3AhsDfwB+CEzLxnamelHd19t581/muTbLvbPqe3dzLSNDcjwgR4BTAnMw+MiAOA84BjpnhOasEnz/7Odm/71pMXtW0eksrMlDA5CLgGIDNvioj9png+22XpqvcVbb/8sHO2e9tuu8WopPbqGR4enuo5FIuIS4CvZubV9fP7gD0yc9PUzkySusNMOQG/Aegf9bzXIJGkzpkpYXIDcBRAfc7kzqmdjiR1l5lyzuRrwBER8X2gBzhuiucjSV1lRpwzkSRNrZlymEuSNIUME0lSMcNEklTMMGlRfcmWrhURO09BzSdExOwpqPvUTtes6/ZGxJ9Pxd+1iHhKRPR0qNa8TtSZTETMjogndLhmR37GU8ET8BOIiD2A84H9gE1U4Xsn8K7MnOyL4dNSRLwM+ATwR+DvMvN/1e2rMrPRm4BHxLOAjwH/AVwOXAJsBt6ZmVc1WHevMU2fBf4GoOn/zxHx6cx8U0QsAD4PPED1nanjM/OmBuseBzwDuAr4AvAwMBd4W2Z+q6m6de2NwN9m5qebrLOVunsBHwYeAS6g+v/cB5wy8ve8obrPBpYD/wXYFfghcC/w7sz8j6bqdtpM+WhwUy6h+ot280hD/T2WzwAvmrJZNevvgH2oPmL9lYiYk5mX1c+b9hngDGB3qjDZi+pN7mqqN72mfAvYCPyaap0BfAoYBhoNUOBZ9X8/BByZmT+NiF2BLwILG6z7NmARcCXw8sxcU9e9gurn0aQfAftExCrgzMy8vuF6Iy4GPgjsQvX3aW9gPdV6GwsTqiB5R/0zPgA4Gvg68On6ceMi4hjgcKq1rwdWA5dnZtv2JgyTic0ZHSTwp2t/daR4RFwHjD281AMMZ+YLGyr7SGb+tq5/DLCqvjxNJ3Zh++o3lusj4tDM/E09j6avZrAfcBHwycz8ZkRcl5mHNlxzrM2Z+VOAzPx1Bw51/TEzfx8RQ1S/JY/U7cT/54cy8+31NfROiYjlVG/o92bmBQ3W7cvMb9WHmj6cmb8CiIg/NlgTYJeRPdz6/ePDmXlaRAw0XBeA+ufbS/VL2RDVnu+RwGLghHbVMUwm9qOIuJTqIpK/o/qfcBTw4w7VP5nqt6lXUh1m64RfRMT5wGmZORQRrwKuBZ7cgdpZX2ftzZm5BCAiTqY67NVc0czfRMRrgXMjYv8ma23FkyPih8ATI+JNVIe6zgN+2XDdKyPiCuAu4KqIuBb4K2BVw3Wh3svNzFuBV0fELsAhVHuETfpFRHyJ6n3vwYj4ENW/639vuO69EXER1Zv5S4E76n9Xv2+47ojnZebYvdwrI+KGdhYxTCb2NqrL2x8EzKO6BthVVN+4b1xm3hwRnwOen5kdqQkcD7yeek8kM/8tIg4FTulA7f8OvCwzt4xq+79Ux7cbVV/L7cSIWEIHP5iSmfPrDzfsTXWobQvVeblGzydk5tkRsZDqt9P7gKcCF2TmN5qsW1sxZi6/A/6l/tOkN1L9MrgGeBB4F9XP/PiG6x5H9Xf7JcAtwKXA/sCxDdcd0RsRB2fm6pGGiDiE6rxo23gCXpJmsPoDAOcD+1LtFW4BbgNOGjm02g6GiSSpmIe5JGkGG+eDPAC084M8hokkzWwd+SCPh7kkaYaLiPcC9zT5QR7DRJJUrKuvNyVJag/DRJJUzBPwUgMiYneqL8f9pG7qpfri62WZecYE2/3pUi4RcUdmvqDpuUrtYJhIzfn16DCoL6T404j4UmbePc42i0YeGCSaTgwTqXOeTvUN5KGIuBh4HvA0qmu9vQ74CEBE3JyZCyJiODN7ImIZ8OfAc4BnApdk5ociYieqC1QeBPyK6hI4H8zM73R2WZJhIjVp14i4A5gDPAX4AdVn/fegujrzgfXVgVcBR2XmOyLibzNzwVbGej5wMNUFN39WXwn2DcATgb8AdqO6ppc0JTwBLzVn5DDXfwU+R/Xv7ZuZ+V3gwohYCnycao/jSZOMdV1mPlJflv+3VPelOAL4fGYOZ+YvgW83tRBpMoaJ1LD6KsjvpTpUdVJEvJzqUvMbqW4I9l0mv/nYw6MeD9f9N+O/Ye0g/IsodUB9ifuTgNOo7mnx5cz8DNVd7w4FZtVdN0dEq4efvwUcGxE99cn9RXTmJmbS4xgmUodk5jXAjVSHtV4XEXcCXwFu4NHb915BdVO2OS0M+U9Ud867E7iM6oZaD7V73lIrvJyKNE1FxNFAT2ZeVd+t8HZgv5HbLkudZJhI01REPIvqxP7IyftzM/N/TuGU1MUME0lSMc+ZSJKKGSaSpGKGiSSpmGEiSSpmmEiSihkmkqRi/x9W0pxLHoYCQwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a210fd9e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.rc('font', size=15)\n",
    "ratings.bookRating.value_counts(sort=False).plot(kind='bar')\n",
    "\n",
    "plt.title('Rating Distribution\\n')\n",
    "plt.xlabel('Rating')\n",
    "plt.ylabel('Count');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Books dataset\n",
    "This dataset provides books details. It includes 271360 records and 8 fields: ISBN, book title, book author, publisher and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(271360, 8)\n",
      "['ISBN', 'bookTitle', 'bookAuthor', 'yearOfPublication', 'publisher', 'imageUrlS', 'imageUrlM', 'imageUrlL']\n"
     ]
    }
   ],
   "source": [
    "print(books.shape)\n",
    "print(list(books.columns))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ISBN</th>\n",
       "      <th>bookTitle</th>\n",
       "      <th>bookAuthor</th>\n",
       "      <th>yearOfPublication</th>\n",
       "      <th>publisher</th>\n",
       "      <th>imageUrlS</th>\n",
       "      <th>imageUrlM</th>\n",
       "      <th>imageUrlL</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0195153448</td>\n",
       "      <td>Classical Mythology</td>\n",
       "      <td>Mark P. O. Morford</td>\n",
       "      <td>2002</td>\n",
       "      <td>Oxford University Press</td>\n",
       "      <td>http://images.amazon.com/images/P/0195153448.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0195153448.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0195153448.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0002005018</td>\n",
       "      <td>Clara Callan</td>\n",
       "      <td>Richard Bruce Wright</td>\n",
       "      <td>2001</td>\n",
       "      <td>HarperFlamingo Canada</td>\n",
       "      <td>http://images.amazon.com/images/P/0002005018.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0002005018.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0002005018.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0060973129</td>\n",
       "      <td>Decision in Normandy</td>\n",
       "      <td>Carlo D'Este</td>\n",
       "      <td>1991</td>\n",
       "      <td>HarperPerennial</td>\n",
       "      <td>http://images.amazon.com/images/P/0060973129.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0060973129.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0060973129.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0374157065</td>\n",
       "      <td>Flu: The Story of the Great Influenza Pandemic...</td>\n",
       "      <td>Gina Bari Kolata</td>\n",
       "      <td>1999</td>\n",
       "      <td>Farrar Straus Giroux</td>\n",
       "      <td>http://images.amazon.com/images/P/0374157065.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0374157065.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0374157065.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0393045218</td>\n",
       "      <td>The Mummies of Urumchi</td>\n",
       "      <td>E. J. W. Barber</td>\n",
       "      <td>1999</td>\n",
       "      <td>W. W. Norton &amp;amp; Company</td>\n",
       "      <td>http://images.amazon.com/images/P/0393045218.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0393045218.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0393045218.0...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         ISBN                                          bookTitle  \\\n",
       "0  0195153448                                Classical Mythology   \n",
       "1  0002005018                                       Clara Callan   \n",
       "2  0060973129                               Decision in Normandy   \n",
       "3  0374157065  Flu: The Story of the Great Influenza Pandemic...   \n",
       "4  0393045218                             The Mummies of Urumchi   \n",
       "\n",
       "             bookAuthor yearOfPublication                   publisher  \\\n",
       "0    Mark P. O. Morford              2002     Oxford University Press   \n",
       "1  Richard Bruce Wright              2001       HarperFlamingo Canada   \n",
       "2          Carlo D'Este              1991             HarperPerennial   \n",
       "3      Gina Bari Kolata              1999        Farrar Straus Giroux   \n",
       "4       E. J. W. Barber              1999  W. W. Norton &amp; Company   \n",
       "\n",
       "                                           imageUrlS  \\\n",
       "0  http://images.amazon.com/images/P/0195153448.0...   \n",
       "1  http://images.amazon.com/images/P/0002005018.0...   \n",
       "2  http://images.amazon.com/images/P/0060973129.0...   \n",
       "3  http://images.amazon.com/images/P/0374157065.0...   \n",
       "4  http://images.amazon.com/images/P/0393045218.0...   \n",
       "\n",
       "                                           imageUrlM  \\\n",
       "0  http://images.amazon.com/images/P/0195153448.0...   \n",
       "1  http://images.amazon.com/images/P/0002005018.0...   \n",
       "2  http://images.amazon.com/images/P/0060973129.0...   \n",
       "3  http://images.amazon.com/images/P/0374157065.0...   \n",
       "4  http://images.amazon.com/images/P/0393045218.0...   \n",
       "\n",
       "                                           imageUrlL  \n",
       "0  http://images.amazon.com/images/P/0195153448.0...  \n",
       "1  http://images.amazon.com/images/P/0002005018.0...  \n",
       "2  http://images.amazon.com/images/P/0060973129.0...  \n",
       "3  http://images.amazon.com/images/P/0374157065.0...  \n",
       "4  http://images.amazon.com/images/P/0393045218.0...  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "books.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Users dataset\n",
    "This dataset provides the user demographic information. It includes 278858 records and 3 fields: user id, location and age."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(278858, 3)\n",
      "['userID', 'Location', 'Age']\n"
     ]
    }
   ],
   "source": [
    "print(users.shape)\n",
    "print(list(users.columns))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userID</th>\n",
       "      <th>Location</th>\n",
       "      <th>Age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>nyc, new york, usa</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>stockton, california, usa</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>moscow, yukon territory, russia</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>porto, v.n.gaia, portugal</td>\n",
       "      <td>17.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>farnborough, hants, united kingdom</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userID                            Location   Age\n",
       "0       1                  nyc, new york, usa   NaN\n",
       "1       2           stockton, california, usa  18.0\n",
       "2       3     moscow, yukon territory, russia   NaN\n",
       "3       4           porto, v.n.gaia, portugal  17.0\n",
       "4       5  farnborough, hants, united kingdom   NaN"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEfCAYAAAC9CZqZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAFfhJREFUeJzt3X+0XWV95/H3TW4iQ7lhrnrpklHKsuh3Va1AghNUIGkLjQExVm1L/QnRoq44A20cQRYWtK1Fh2CLxsGJxYBL+gMsdbAG0hbEgChqk6mM+GWgaGYWa8ZLJjHRaDXJmT/2vs3h9ube555k33PvOe/XWlns/ZznOft5zj2cz9nPPnvvgVarhSRJJeZ1uwOSpLnD0JAkFTM0JEnFDA1JUjFDQ5JUzNCQJBUb7HYHpCZFxAJgO7AtM1cewee9EPgT4PG6aD7wHeDqzPxGXecLwLsz81uTPM9m4PWZ+eQEj30BeDdwHPCxzHzRNPv4NmBhZn48It4B/NvMvGY6zyGNZ2io170G2AacFhG/kJkPH8Hn3pKZrxxbiYizgTsj4rTM/G5mnlvwHOcc6oGx9hFxXIf9OwN4qH6uGzp8DukpDA31uncCfw48BlwCvAMgIi4H3grsAb4EvDozT4yIhcCHgGVUew9bgf+Ymbun2lBm/l1E3F5v8/KI+A7wOuDbwKeA5wEHgG8Abwf+tG56T0ScC2wBvgq8GLgC+EjdHuCYiLgNOAnYBVycmY9ExEbgocy8th7XRqqgeAx4FXBORPwIGAGemZnviogXAh8DngG0gHWZeXNELAf+EPgn4EXAAuDtmXn/lK+y+obHNNSzIuIFwEuBW4GbgDdHxDMiYgVwIfASYAkw1NbscmAfsCQzTwaeAKYzpfPfgV8cV/ZrwFBmnlJvE+C5mXlRvfxLmfm/6uWHMvMXMvP2cc/xHOC6+jluAT49WSfq9v8N+Ehmrh8rj4jBuvyjmfliYCXwwYh4aV1lKVWInEoVdB8sGrX6hqGhXvZO4POZuSMzv0Z1/OFi4Fzg1szclZktYH1bm1cCq4CtEbENeDXwgmlsswXsHVd2H/DCiPgiVSj9cWY+eoj2Ww5R/o+Z+eV6eSPVdNux0+jXmOcDR2XmXwFk5hPAZ4FX1I9/NzO31cv/ADy9g22ohxka6kkR8TPAm4AzIuI79VTRs4B3Ue1JDLRV39+2PB+4JDNPqb/V/3sOThGVeAnwzfaCzHycalrpj4BFwN9FxPmHaP+DQ5TvH7feAn5a/7d9LAun6N/8uk27eVRTUQA/GreNAaQ2hoZ61RuAHcDxmXliZp4IPBc4huqYwmvbvqm/lYMfpHcB74qIhRExD9hA9WE/pfq4xHnAfx1X/k6qqZ7NmXlZvY3F9cP7OfiBPZmTI+KUevntwH2ZuRcYBU6rt3M81bGYMfsmeO5vAz+NiNe0tXkt8LclY5Q8EK5e9U6qYwD/8g09M3dFxPXA71CFwQMRsRf4HxycUvp94FqqA+DzqX55tfYQ2ziznsKCKnSeAFZk5v8ZV+9mYDnwrXp724Hr68duBe4d+xCfxMPAVRHxXOB7wFvq8o8Cn4mIpPrJ791tbTYB10XEvxRk5k8j4tXA9RFxNdVnwAcy8576QLg0qQEvja5+ExGnAS/LzOvr9d8Flmbmb3a3Z9Ls556G+tEjwGURcTHVHsJ2qgPkkqbgnoYkqZgHwiVJxQwNSVIxQ0OSVMzQkCQVMzQkScUMDUlSMUNDklTM0JAkFTM0JEnFDA1JUjFDQ5JUzNCQJBUzNCRJxQwNSVKxnr+fxujono6v/T48fDQ7d+6dumIPccz9wTH3vsMd78jI0IT3h3dPYxKDg/O73YUZ55j7g2PufU2N19CQJBUzNCRJxQwNSVIxQ0OSVMzQkCQVMzQkScUMDUlSMUNDklTM0JAkFev5y4hoes5f+7mubPfGy3+5K9uVND3uaUiSihkakqRijU5PRcRW4Pv16uPAJ4A/AfYBmzPz/RExD/g4cDLwz8DbMvPRiDi9tG6TY5AkHdRYaETEUQCZubytbBvwWuCfgL+JiMXAicBRmfnSOijWAauAG6ZRV5I0A5rc0zgZODoiNtfbuRp4WmY+BhARdwG/AjwLuBMgM78SEadFxKLSug32X5I0TpOhsRe4Fvgk8DxgE7Cr7fE9wHOBRRycwgLYX5ftLqkbEYOZue9QnRgePvqwris/MjLUcVuV6/br3O3td4Nj7n1NjLfJ0HgEeDQzW8AjEfF94Oltjw9RhcjR9fKYeVSBMVRSd7LAAA73zlWMju7puL3KdfN17se/s2PufYc73kMFTpO/nlpNdcyBiDie6gP/hxHx8xExAKwAtgD3A+fW9U4HvpmZu4GflNRtsP+SpHGa3NP4U2BjRNwHtKhC5ADwGWA+1S+ivhoRXwPOiYgvAwPARXX7d0yjriRpBjQWGpn5E+D1Ezx0+rh6B6gCYnz7r5TWlSTNDE/ukyQVMzQkScUMDUlSMUNDklTM0JAkFTM0JEnFDA1JUjFDQ5JUzNCQJBUzNCRJxQwNSVIxQ0OSVMzQkCQVMzQkScUMDUlSMUNDklTM0JAkFTM0JEnFDA1JUjFDQ5JUzNCQJBUzNCRJxQwNSVIxQ0OSVMzQkCQVMzQkScUMDUlSscFud0ACWH3N3V3b9h3rVnVt29Jc456GJKmYoSFJKmZoSJKKGRqSpGKNHgiPiOOAbwDnAPuAjUALeAhYk5kHIuIq4Lz68Usz88GIOKm0bpP9lyQ9VWN7GhGxAPgE8KO66Drgysw8ExgAVkXEYmAZsBS4AFjfQV1J0gxpcnrqWuAG4Il6fQlwb728CTgbOAPYnJmtzNwODEbEyDTrSpJmSCPTUxFxITCamXdFxHvr4oHMbNXLe4BjgUXAjramY+XTqTs6WV+Gh49mcHB+x2MZGRnquK3mjn78Ozvm3tfEeJs6prEaaEXE2cApwM3AcW2PDwG7gN318vjyA9OoO6mdO/d20P3KyMgQo6N7Om6vuaPf/s79+N7utzEf7ngPFTiNTE9l5lmZuSwzlwPbgDcDmyJieV1lJbAFuB9YERHzIuIEYF5mPglsnUZdSdIMmcnLiKwFNkTEQuBh4LbM3B8RW4AHqAJsTQd1JUkzZKDVak1daw4bHd3T8QD7bXcWunsNqG65Y92qvvs79+N7u9/GfASmpwYmKvfkPklSMUNDklTM0JAkFTM0JEnFDA1JUjFDQ5JUzNCQJBUzNCRJxQwNSVIxQ0OSVMzQkCQVMzQkScUMDUlSMUNDklTM0JAkFTM0JEnFDA1JUjFDQ5JUzNCQJBUzNCRJxQwNSVIxQ0OSVMzQkCQVMzQkScUMDUlSMUNDklTM0JAkFTM0JEnFDA1JUjFDQ5JUzNCQJBUzNCRJxQabeuKImA9sAALYD1wEDAAbgRbwELAmMw9ExFXAecA+4NLMfDAiTiqt29QYJElP1eSexvkAmfly4PeA6+p/V2bmmVQBsioiFgPLgKXABcD6uv106kqSZkBRaETECycoO32yNpn518DF9erPAf8XWALcW5dtAs4GzgA2Z2YrM7cDgxExMs26kqQZMOn0VES8HJgPfDIi3kr1jX+s3Q3A8ydrn5n7IuIm4NeA1wGvzMxW/fAe4FhgEbCjrdlY+cA06o4eqg/Dw0czODh/sm5OamRkqOO2mjv68e/smHtfE+Od6pjGOVTTQc8CPtBWvg/4RMkGMvMtEXEZ8FXg37Q9NATsAnbXy+PLD0yj7iHt3Lm3pJsTGhkZYnR0T8ftNXf029+5H9/b/Tbmwx3voQJn0tDIzKsBIuJNmfnp6WwwIt4EPDsz/wjYSxUCX4+I5Zn5RWAlcA/wKPDhiLgWeDYwLzOfjIitpXWn0y9JUudKfz31pYj4z8DTOThFRWaunqTNXwGfiogvAQuAS4GHgQ0RsbBevi0z90fEFuABqmMsa+r2a6dRV5I0A0pD4y+BLfW/1hR1AcjMHwK/McFDyyaoezVw9biyR0rrSpJmRmloLMjMdzfaE0nSrFd6nsZ9EXF+PVUkSepTpXsarwPeBRARY2WtzOz8t6ySpDmnKDQy8/imOyJJmv2KQiMifm+i8sz8wETlkqTeVDo9NdC2vAB4BdXJepI6tPqau7vdBfWwO9atauR5S6en3t++HhG/D2xupEeSpFmr06vcHgOccCQ7Ikma/UqPaTzOwZP65gHDwIeb6pQkaXYqPaaxvG25BezKzN1HvjvSzDt/7ee63QVpziidntoOnAusA64HLowIbxUrSX2mdE/jw8DzgBupfkl1EfDzwCUN9UuSNAuVhsavAqdm5gGAiPgb4JuN9UqSNCuVTjENUp2f0b6+/8h3R5I0m5XuaXwGuCci/qxe/y3glma6JEmaraYMjYgYBjYA/wD8CvDLwB9P905+kqS5b9LpqYg4FfgWsCQz78zM/wTcBVwTES+eiQ5KkmaPqY5pXAv8VmbeOVaQmVcAq4HrmuyYJGn2mSo0hjPzi+MLM/Mu4JmN9EiSNGtNFRoLJjqJry7zLn6S1GemCo17gasmKL8S+PqR744kaTab6tdT7wW+EBFvAbYBPwYWA98DXtVw3yRJs8ykoZGZeyLiLOCXgFOBA8D6zNwyE52TJM0uU56nkZkt4O76nySpj3mlWklSMUNDklTM0JAkFTM0JEnFDA1JUjFDQ5JUzNCQJBUzNCRJxUrv3DctEbEAuBE4EXga8AdU9+XYCLSAh4A1mXkgIq4CzgP2AZdm5oMRcVJp3Sb6L0maWFN7Gm8EdmTmmcBK4GNU99+4si4bAFZFxGJgGbAUuABYX7efTl1J0gxpKjRuBd7Xtr4PWEJ11VyATcDZwBnA5sxsZeZ2YDAiRqZZV5I0QxqZnsrMHwBExBBwG9Wl1K+tr2MFsAc4FlgE7GhrOlY+MI26o5P1ZXj4aAYH53c8lpGRoY7bSlI3NfH51UhoAETEc4DbgY9n5i0R8eG2h4eAXcDuenl8+YFp1J3Uzp17O+o/VC/46OiejttLUjcdzufXoQKnkempiPhZYDNwWWbeWBdvjYjl9fJKYAtwP7AiIuZFxAnAvMx8cpp1JUkzpKk9jSuAYeB9ETF2bOMS4PqIWAg8DNyWmfsjYgvwAFWAranrrgU2FNaVJM2QgVarNXWtOWx0dE/HA+zH6anV13jbFKkX3LFu1eFOTw1MVN7YMQ11zg9uSbOVZ4RLkooZGpKkYoaGJKmYoSFJKmZoSJKKGRqSpGKGhiSpmKEhSSpmaEiSihkakqRihoYkqZihIUkqZmhIkooZGpKkYoaGJKmYoSFJKmZoSJKKGRqSpGKGhiSpmKEhSSpmaEiSihkakqRihoYkqZihIUkqZmhIkooZGpKkYoaGJKmYoSFJKmZoSJKKGRqSpGKGhiSpmKEhSSo22OSTR8RS4EOZuTwiTgI2Ai3gIWBNZh6IiKuA84B9wKWZ+eB06jbZf0nSUzW2pxER7wE+CRxVF10HXJmZZwIDwKqIWAwsA5YCFwDrO6grSZohTU5PPQa8pm19CXBvvbwJOBs4A9icma3M3A4MRsTINOtKkmZIY9NTmfnZiDixrWggM1v18h7gWGARsKOtzlj5dOqOTtaP4eGjGRyc3+kwGBkZ6ritJHVTE59fjR7TGOdA2/IQsAvYXS+PL59O3Unt3Lm3w+5WL/jo6J6O20tSNx3O59ehAmcmfz21NSKW18srgS3A/cCKiJgXEScA8zLzyWnWlSTNkJnc01gLbIiIhcDDwG2ZuT8itgAPUAXYmg7qSpJmyECr1Zq61hw2Orqn4wF2a3pq9TV3z/g2JfWWO9atOtzpqYGJyj25T5JUzNCQJBUzNCRJxQwNSVIxQ0OSVMzQkCQVMzQkScUMDUlSMUNDklTM0JAkFTM0JEnFDA1JUjFDQ5JUzNCQJBUzNCRJxQwNSVIxQ0OSVMzQkCQVMzQkScUMDUlSMUNDklTM0JAkFTM0JEnFDA1JUjFDQ5JUzNCQJBUzNCRJxQwNSVIxQ0OSVMzQkCQVMzQkScUGu92B2ez8tZ/rdhckaVaZc6EREfOAjwMnA/8MvC0zH+1urySpP8zF6alXA0dl5kuBy4F1Xe6PJPWNuRgaZwB3AmTmV4DTutsdSeofc256ClgEfL9tfX9EDGbmvokqj4wMDXS6oTvWreq0qSR13cjI0BF/zrm4p7EbaH8l5h0qMCRJR9ZcDI37gXMBIuJ04Jvd7Y4k9Y+5OD11O3BORHwZGAAu6nJ/JKlvDLRarW73QZI0R8zF6SlJUpcYGpKkYnPxmEbj+uWs84hYANwInAg8DfgD4FvARqAFPASsycwDXepiIyLiOOAbwDnAPnp/vO8FXgUspHpf30sPj7l+X99E9b7eD/w2Pfx3joilwIcyc3lEnMQE44yIq4DzqF6HSzPzwU63557GxPrlrPM3Ajsy80xgJfAx4DrgyrpsAOipk1XqD5RPAD+qi3p9vMuBlwEvB5YBz6HHx0z168rBzHwZ8AHgD+nRMUfEe4BPAkfVRf9qnBGxmOpvvxS4AFh/ONs0NCbWL2ed3wq8r219H7CE6psowCbg7JnuVMOuBW4AnqjXe328K6h+ln47cAfweXp/zI8Ag/WMwSLgp/TumB8DXtO2PtE4zwA2Z2YrM7dTvTYjnW7Q0JjYhGedd6szTcnMH2TmnogYAm4DrgQGMnPsJ3V7gGO71sEjLCIuBEYz86624p4db+2ZVF96fh14B/AZqhNie3nMP6Camvo2sAG4nh79O2fmZ6lCccxE4xz/eXZY4zc0JtY3Z51HxHOAe4BPZ+YtQPs87xCwqysda8ZqqnN8vgicAtwMHNf2eK+NF2AHcFdm/iQzE/gxT/3A6MUx/w7VmJ9PdVzyJqrjOWN6ccxjJvr/d/zn2WGN39CYWF+cdR4RPwtsBi7LzBvr4q31PDhUxzm2dKNvTcjMszJzWWYuB7YBbwY29ep4a/cBr4iIgYg4HvgZ4O97fMw7OfjN+v8BC+jh9/U4E43zfmBFRMyLiBOovgQ/2ekGem7K5Qjpl7POrwCGgfdFxNixjUuA6yNiIfAw1bRVL1sLbOjV8Wbm5yPiLOBBqi+Ja4DH6eExAx8BboyILVR7GFcAX6e3xzzmX72fM3N//Vo8wMH3QMc8I1ySVMzpKUlSMUNDklTM0JAkFTM0JEnFDA1JUjFDQ2pQRLwoIloR8dpu90U6EgwNqVmrqa7x9fZud0Q6EjxPQ2pIfUXd/w2cCXwZWJqZj9Vn7H6U6gKRDwAvaLus9X8BngHsBf5DZm7tSuelQ3BPQ2rOecB3M/MR4K+Bi+sg+TTwhsw8ladebO4m4D2ZuRi4GPjzme6wNBVDQ2rORcCf1ct/Ua+fCnwvM/+xLr8RICKOAV4CfCoitgG3AMdExDNmtsvS5Lz2lNSA+u6AK4ElEXEJ1TXMhuuyib6szQd+nJmntD3Hs6kuuCfNGu5pSM14E/D3mfnszDwxM3+O6g5yK4DhiPjFut7rgVZmfh/4nxHxRoCIOAf4Ujc6Lk3GPQ2pGRdSXV213XrgPcCvAjdHxAEgOXjr2TcAN9S38PwJ8JttN9SRZgV/PSXNoPoWpNcA78/MH0bE7wL/LjPXdrlrUhGnp6QZlJkHqI5TfK0+4H0W8MHu9koq556GJKmYexqSpGKGhiSpmKEhSSpmaEiSihkakqRihoYkqdj/B7AHH72ZPTO5AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a21166b38>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "users.Age.hist(bins=[0, 10, 20, 30 , 40, 50, 100])\n",
    "plt.title('Age Distribution\\n')\n",
    "plt.xlabel('Age')\n",
    "plt.ylabel('Count');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The most active users are among 20-30s.\n",
    "\n",
    "## Recommendations based on rating counts\n",
    "\n",
    "Let's find out which books are in the top 5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bookRating</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ISBN</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0971880107</th>\n",
       "      <td>2502</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0316666343</th>\n",
       "      <td>1295</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0385504209</th>\n",
       "      <td>883</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0060928336</th>\n",
       "      <td>732</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0312195516</th>\n",
       "      <td>723</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            bookRating\n",
       "ISBN                  \n",
       "0971880107        2502\n",
       "0316666343        1295\n",
       "0385504209         883\n",
       "0060928336         732\n",
       "0312195516         723"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rating_count = pd.DataFrame(ratings.groupby('ISBN')\n",
    "                            ['bookRating'].count())\n",
    "rating_count.sort_values('bookRating', ascending=False).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ISBN</th>\n",
       "      <th>bookTitle</th>\n",
       "      <th>bookAuthor</th>\n",
       "      <th>yearOfPublication</th>\n",
       "      <th>publisher</th>\n",
       "      <th>imageUrlS</th>\n",
       "      <th>imageUrlM</th>\n",
       "      <th>imageUrlL</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0971880107</td>\n",
       "      <td>Wild Animus</td>\n",
       "      <td>Rich Shapero</td>\n",
       "      <td>2004</td>\n",
       "      <td>Too Far</td>\n",
       "      <td>http://images.amazon.com/images/P/0971880107.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0971880107.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0971880107.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0316666343</td>\n",
       "      <td>The Lovely Bones: A Novel</td>\n",
       "      <td>Alice Sebold</td>\n",
       "      <td>2002</td>\n",
       "      <td>Little, Brown</td>\n",
       "      <td>http://images.amazon.com/images/P/0316666343.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0316666343.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0316666343.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0385504209</td>\n",
       "      <td>The Da Vinci Code</td>\n",
       "      <td>Dan Brown</td>\n",
       "      <td>2003</td>\n",
       "      <td>Doubleday</td>\n",
       "      <td>http://images.amazon.com/images/P/0385504209.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0385504209.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0385504209.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0060928336</td>\n",
       "      <td>Divine Secrets of the Ya-Ya Sisterhood: A Novel</td>\n",
       "      <td>Rebecca Wells</td>\n",
       "      <td>1997</td>\n",
       "      <td>Perennial</td>\n",
       "      <td>http://images.amazon.com/images/P/0060928336.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0060928336.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0060928336.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0312195516</td>\n",
       "      <td>The Red Tent (Bestselling Backlist)</td>\n",
       "      <td>Anita Diamant</td>\n",
       "      <td>1998</td>\n",
       "      <td>Picador USA</td>\n",
       "      <td>http://images.amazon.com/images/P/0312195516.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0312195516.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0312195516.0...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         ISBN                                        bookTitle     bookAuthor  \\\n",
       "0  0971880107                                      Wild Animus   Rich Shapero   \n",
       "1  0316666343                        The Lovely Bones: A Novel   Alice Sebold   \n",
       "2  0385504209                                The Da Vinci Code      Dan Brown   \n",
       "3  0060928336  Divine Secrets of the Ya-Ya Sisterhood: A Novel  Rebecca Wells   \n",
       "4  0312195516              The Red Tent (Bestselling Backlist)  Anita Diamant   \n",
       "\n",
       "  yearOfPublication      publisher  \\\n",
       "0              2004        Too Far   \n",
       "1              2002  Little, Brown   \n",
       "2              2003      Doubleday   \n",
       "3              1997      Perennial   \n",
       "4              1998    Picador USA   \n",
       "\n",
       "                                           imageUrlS  \\\n",
       "0  http://images.amazon.com/images/P/0971880107.0...   \n",
       "1  http://images.amazon.com/images/P/0316666343.0...   \n",
       "2  http://images.amazon.com/images/P/0385504209.0...   \n",
       "3  http://images.amazon.com/images/P/0060928336.0...   \n",
       "4  http://images.amazon.com/images/P/0312195516.0...   \n",
       "\n",
       "                                           imageUrlM  \\\n",
       "0  http://images.amazon.com/images/P/0971880107.0...   \n",
       "1  http://images.amazon.com/images/P/0316666343.0...   \n",
       "2  http://images.amazon.com/images/P/0385504209.0...   \n",
       "3  http://images.amazon.com/images/P/0060928336.0...   \n",
       "4  http://images.amazon.com/images/P/0312195516.0...   \n",
       "\n",
       "                                           imageUrlL  \n",
       "0  http://images.amazon.com/images/P/0971880107.0...  \n",
       "1  http://images.amazon.com/images/P/0316666343.0...  \n",
       "2  http://images.amazon.com/images/P/0385504209.0...  \n",
       "3  http://images.amazon.com/images/P/0060928336.0...  \n",
       "4  http://images.amazon.com/images/P/0312195516.0...  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_rated_books = pd.DataFrame(['0971880107', '0316666343', \n",
    "                                 '0385504209', '0060928336', '0312195516'], \n",
    "                                index=np.arange(5), columns = ['ISBN'])\n",
    "most_rated_books_summary = pd.merge(most_rated_books, books, on='ISBN')\n",
    "most_rated_books_summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Something in common among these five most rated books - they are fictions or novels. The recommender suggests that novels and fictions are popular and likely receive more ratings. And if someone likes \"Wild Animus\", probably we should recommend him(her) \"The Lovely Bones: A Novel\n",
    "\n",
    "## Recommendations based on correlations\n",
    "Find out the average rating and the number of ratings each book received."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bookRating</th>\n",
       "      <th>ratingCount</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ISBN</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0971880107</th>\n",
       "      <td>1.019584</td>\n",
       "      <td>2502</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0316666343</th>\n",
       "      <td>4.468726</td>\n",
       "      <td>1295</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0385504209</th>\n",
       "      <td>4.652322</td>\n",
       "      <td>883</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0060928336</th>\n",
       "      <td>3.448087</td>\n",
       "      <td>732</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0312195516</th>\n",
       "      <td>4.334716</td>\n",
       "      <td>723</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            bookRating  ratingCount\n",
       "ISBN                               \n",
       "0971880107    1.019584         2502\n",
       "0316666343    4.468726         1295\n",
       "0385504209    4.652322          883\n",
       "0060928336    3.448087          732\n",
       "0312195516    4.334716          723"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "average_rating = pd.DataFrame(ratings.groupby('ISBN')['bookRating'].mean())\n",
    "average_rating['ratingCount'] = pd.DataFrame(ratings.groupby('ISBN')['bookRating'].count())\n",
    "average_rating.sort_values('ratingCount', ascending=False).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observation:**\n",
    "In this dataet, the book that received the most ratings is not highly rated at all. So if we were set to use recommendations based on rating counts, we would definitely make mistaks here.\n",
    "\n",
    "To ensure statistical significance, users with less than 200 ratings, and books with less than 100 ratings are excluded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "counts1 = ratings['userID'].value_counts()\n",
    "ratings = ratings[ratings['userID'].isin(counts1[counts1 >= 200].index)]\n",
    "counts = ratings['bookRating'].value_counts()\n",
    "ratings = ratings[ratings['bookRating'].isin(counts[counts >= 100].index)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Rating matrix**\n",
    "\n",
    "Convert the table to a 2D matrix. The matrix will be sparse because not every user rate every book."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(905, 207699)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>ISBN</th>\n",
       "      <th>0330299891</th>\n",
       "      <th>0375404120</th>\n",
       "      <th>0586045007</th>\n",
       "      <th>9022906116</th>\n",
       "      <th>9032803328</th>\n",
       "      <th>9044922564</th>\n",
       "      <th>9044922572</th>\n",
       "      <th>9044922718</th>\n",
       "      <th>9044923161</th>\n",
       "      <th>904492401X</th>\n",
       "      <th>...</th>\n",
       "      <th>UNGRANDHOMMED</th>\n",
       "      <th>X000000000</th>\n",
       "      <th>YOUTELLEM,AND</th>\n",
       "      <th>ZR903CX0003</th>\n",
       "      <th>\\0432534220\\\"\"</th>\n",
       "      <th>\\2842053052\\\"\"</th>\n",
       "      <th>b00005wz75</th>\n",
       "      <th>cn108465</th>\n",
       "      <th>cn113107</th>\n",
       "      <th>Ô½crosoft</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>userID</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>254</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2276</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2766</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2977</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3363</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 207699 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "ISBN     0330299891   0375404120   0586045007   9022906116   9032803328  \\\n",
       "userID                                                                    \n",
       "254             NaN          NaN          NaN          NaN          NaN   \n",
       "2276            NaN          NaN          NaN          NaN          NaN   \n",
       "2766            NaN          NaN          NaN          NaN          NaN   \n",
       "2977            NaN          NaN          NaN          NaN          NaN   \n",
       "3363            NaN          NaN          NaN          NaN          NaN   \n",
       "\n",
       "ISBN     9044922564   9044922572   9044922718   9044923161   904492401X  \\\n",
       "userID                                                                    \n",
       "254             NaN          NaN          NaN          NaN          NaN   \n",
       "2276            NaN          NaN          NaN          NaN          NaN   \n",
       "2766            NaN          NaN          NaN          NaN          NaN   \n",
       "2977            NaN          NaN          NaN          NaN          NaN   \n",
       "3363            NaN          NaN          NaN          NaN          NaN   \n",
       "\n",
       "ISBN      ...      UNGRANDHOMMED  X000000000  YOUTELLEM,AND  ZR903CX0003  \\\n",
       "userID    ...                                                              \n",
       "254       ...                NaN         NaN            NaN          NaN   \n",
       "2276      ...                NaN         NaN            NaN          NaN   \n",
       "2766      ...                NaN         NaN            NaN          NaN   \n",
       "2977      ...                NaN         NaN            NaN          NaN   \n",
       "3363      ...                NaN         NaN            NaN          NaN   \n",
       "\n",
       "ISBN    \\0432534220\\\"\"  \\2842053052\\\"\"  b00005wz75  cn108465  cn113107  \\\n",
       "userID                                                                   \n",
       "254                NaN             NaN         NaN       NaN       NaN   \n",
       "2276               NaN             NaN         NaN       NaN       NaN   \n",
       "2766               NaN             NaN         NaN       NaN       NaN   \n",
       "2977               NaN             NaN         NaN       NaN       NaN   \n",
       "3363               NaN             NaN         NaN       NaN       NaN   \n",
       "\n",
       "ISBN    Ô½crosoft  \n",
       "userID             \n",
       "254           NaN  \n",
       "2276          NaN  \n",
       "2766          NaN  \n",
       "2977          NaN  \n",
       "3363          NaN  \n",
       "\n",
       "[5 rows x 207699 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings_pivot = ratings.pivot(index='userID', columns='ISBN').bookRating\n",
    "userID = ratings_pivot.index\n",
    "ISBN = ratings_pivot.columns\n",
    "print(ratings_pivot.shape)\n",
    "ratings_pivot.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's find out which books are correlated with the 2nd most rated book \"The Lovely Bones: A Novel\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/adhiman/anaconda3/lib/python3.6/site-packages/numpy/lib/function_base.py:3175: RuntimeWarning: Degrees of freedom <= 0 for slice\n",
      "  c = cov(x, y, rowvar)\n",
      "/Users/adhiman/anaconda3/lib/python3.6/site-packages/numpy/lib/function_base.py:3109: RuntimeWarning: divide by zero encountered in double_scalars\n",
      "  c *= 1. / np.float64(fact)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pearsonR</th>\n",
       "      <th>ratingCount</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ISBN</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0316666343</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1295</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0312291639</th>\n",
       "      <td>0.471872</td>\n",
       "      <td>354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0316601950</th>\n",
       "      <td>0.434248</td>\n",
       "      <td>568</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0446610038</th>\n",
       "      <td>0.429712</td>\n",
       "      <td>391</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0446672211</th>\n",
       "      <td>0.421478</td>\n",
       "      <td>585</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0385265700</th>\n",
       "      <td>0.351635</td>\n",
       "      <td>319</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0345342968</th>\n",
       "      <td>0.316922</td>\n",
       "      <td>321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0060930535</th>\n",
       "      <td>0.309860</td>\n",
       "      <td>494</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0375707972</th>\n",
       "      <td>0.308145</td>\n",
       "      <td>354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0684872153</th>\n",
       "      <td>0.272480</td>\n",
       "      <td>326</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            pearsonR  ratingCount\n",
       "ISBN                             \n",
       "0316666343  1.000000         1295\n",
       "0312291639  0.471872          354\n",
       "0316601950  0.434248          568\n",
       "0446610038  0.429712          391\n",
       "0446672211  0.421478          585\n",
       "0385265700  0.351635          319\n",
       "0345342968  0.316922          321\n",
       "0060930535  0.309860          494\n",
       "0375707972  0.308145          354\n",
       "0684872153  0.272480          326"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bones_ratings = ratings_pivot['0316666343']\n",
    "similar_to_bones = ratings_pivot.corrwith(bones_ratings)\n",
    "corr_bones = pd.DataFrame(similar_to_bones, columns=['pearsonR'])\n",
    "corr_bones.dropna(inplace=True)\n",
    "corr_summary = corr_bones.join(average_rating['ratingCount'])\n",
    "corr_summary[corr_summary['ratingCount']>=300].sort_values('pearsonR', ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ISBN</th>\n",
       "      <th>bookTitle</th>\n",
       "      <th>bookAuthor</th>\n",
       "      <th>yearOfPublication</th>\n",
       "      <th>publisher</th>\n",
       "      <th>imageUrlS</th>\n",
       "      <th>imageUrlM</th>\n",
       "      <th>imageUrlL</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0312291639</td>\n",
       "      <td>The Nanny Diaries: A Novel</td>\n",
       "      <td>Emma McLaughlin</td>\n",
       "      <td>2003</td>\n",
       "      <td>St. Martin's Griffin</td>\n",
       "      <td>http://images.amazon.com/images/P/0312291639.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0312291639.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0312291639.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0316601950</td>\n",
       "      <td>The Pilot's Wife : A Novel</td>\n",
       "      <td>Anita Shreve</td>\n",
       "      <td>1999</td>\n",
       "      <td>Back Bay Books</td>\n",
       "      <td>http://images.amazon.com/images/P/0316601950.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0316601950.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0316601950.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0446610038</td>\n",
       "      <td>1st to Die: A Novel</td>\n",
       "      <td>James Patterson</td>\n",
       "      <td>2002</td>\n",
       "      <td>Warner Vision</td>\n",
       "      <td>http://images.amazon.com/images/P/0446610038.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0446610038.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0446610038.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0446672211</td>\n",
       "      <td>Where the Heart Is (Oprah's Book Club (Paperba...</td>\n",
       "      <td>Billie Letts</td>\n",
       "      <td>1998</td>\n",
       "      <td>Warner Books</td>\n",
       "      <td>http://images.amazon.com/images/P/0446672211.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0446672211.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0446672211.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0385265700</td>\n",
       "      <td>The Book of Ruth (Oprah's Book Club (Paperback))</td>\n",
       "      <td>Jane Hamilton</td>\n",
       "      <td>1990</td>\n",
       "      <td>Anchor</td>\n",
       "      <td>http://images.amazon.com/images/P/0385265700.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0385265700.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0385265700.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0345342968</td>\n",
       "      <td>Fahrenheit 451</td>\n",
       "      <td>RAY BRADBURY</td>\n",
       "      <td>1987</td>\n",
       "      <td>Del Rey</td>\n",
       "      <td>http://images.amazon.com/images/P/0345342968.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0345342968.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0345342968.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0060930535</td>\n",
       "      <td>The Poisonwood Bible: A Novel</td>\n",
       "      <td>Barbara Kingsolver</td>\n",
       "      <td>1999</td>\n",
       "      <td>Perennial</td>\n",
       "      <td>http://images.amazon.com/images/P/0060930535.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0060930535.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0060930535.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0375707972</td>\n",
       "      <td>The Reader</td>\n",
       "      <td>Bernhard Schlink</td>\n",
       "      <td>1999</td>\n",
       "      <td>Vintage Books USA</td>\n",
       "      <td>http://images.amazon.com/images/P/0375707972.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0375707972.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0375707972.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0684872153</td>\n",
       "      <td>Angela's Ashes (MMP) : A Memoir</td>\n",
       "      <td>Frank McCourt</td>\n",
       "      <td>1999</td>\n",
       "      <td>Scribner</td>\n",
       "      <td>http://images.amazon.com/images/P/0684872153.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0684872153.0...</td>\n",
       "      <td>http://images.amazon.com/images/P/0684872153.0...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         ISBN                                          bookTitle  \\\n",
       "0  0312291639                         The Nanny Diaries: A Novel   \n",
       "1  0316601950                         The Pilot's Wife : A Novel   \n",
       "2  0446610038                                1st to Die: A Novel   \n",
       "3  0446672211  Where the Heart Is (Oprah's Book Club (Paperba...   \n",
       "4  0385265700   The Book of Ruth (Oprah's Book Club (Paperback))   \n",
       "5  0345342968                                     Fahrenheit 451   \n",
       "6  0060930535                      The Poisonwood Bible: A Novel   \n",
       "7  0375707972                                         The Reader   \n",
       "8  0684872153                    Angela's Ashes (MMP) : A Memoir   \n",
       "\n",
       "           bookAuthor yearOfPublication             publisher  \\\n",
       "0     Emma McLaughlin              2003  St. Martin's Griffin   \n",
       "1        Anita Shreve              1999        Back Bay Books   \n",
       "2     James Patterson              2002         Warner Vision   \n",
       "3        Billie Letts              1998          Warner Books   \n",
       "4       Jane Hamilton              1990                Anchor   \n",
       "5        RAY BRADBURY              1987               Del Rey   \n",
       "6  Barbara Kingsolver              1999             Perennial   \n",
       "7    Bernhard Schlink              1999     Vintage Books USA   \n",
       "8       Frank McCourt              1999              Scribner   \n",
       "\n",
       "                                           imageUrlS  \\\n",
       "0  http://images.amazon.com/images/P/0312291639.0...   \n",
       "1  http://images.amazon.com/images/P/0316601950.0...   \n",
       "2  http://images.amazon.com/images/P/0446610038.0...   \n",
       "3  http://images.amazon.com/images/P/0446672211.0...   \n",
       "4  http://images.amazon.com/images/P/0385265700.0...   \n",
       "5  http://images.amazon.com/images/P/0345342968.0...   \n",
       "6  http://images.amazon.com/images/P/0060930535.0...   \n",
       "7  http://images.amazon.com/images/P/0375707972.0...   \n",
       "8  http://images.amazon.com/images/P/0684872153.0...   \n",
       "\n",
       "                                           imageUrlM  \\\n",
       "0  http://images.amazon.com/images/P/0312291639.0...   \n",
       "1  http://images.amazon.com/images/P/0316601950.0...   \n",
       "2  http://images.amazon.com/images/P/0446610038.0...   \n",
       "3  http://images.amazon.com/images/P/0446672211.0...   \n",
       "4  http://images.amazon.com/images/P/0385265700.0...   \n",
       "5  http://images.amazon.com/images/P/0345342968.0...   \n",
       "6  http://images.amazon.com/images/P/0060930535.0...   \n",
       "7  http://images.amazon.com/images/P/0375707972.0...   \n",
       "8  http://images.amazon.com/images/P/0684872153.0...   \n",
       "\n",
       "                                           imageUrlL  \n",
       "0  http://images.amazon.com/images/P/0312291639.0...  \n",
       "1  http://images.amazon.com/images/P/0316601950.0...  \n",
       "2  http://images.amazon.com/images/P/0446610038.0...  \n",
       "3  http://images.amazon.com/images/P/0446672211.0...  \n",
       "4  http://images.amazon.com/images/P/0385265700.0...  \n",
       "5  http://images.amazon.com/images/P/0345342968.0...  \n",
       "6  http://images.amazon.com/images/P/0060930535.0...  \n",
       "7  http://images.amazon.com/images/P/0375707972.0...  \n",
       "8  http://images.amazon.com/images/P/0684872153.0...  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "books_corr_to_bones = pd.DataFrame(['0312291639', '0316601950', '0446610038', '0446672211', '0385265700', '0345342968', '0060930535', '0375707972', '0684872153'], \n",
    "                                  index=np.arange(9), columns=['ISBN'])\n",
    "corr_books = pd.merge(books_corr_to_bones, books, on='ISBN')\n",
    "corr_books"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's select three books to examine from the above highly correlated list \"The Nanny Diaries: A Novel\", \"The Pilot's Wife: A Novel\" and \"Where the heart is\".\n",
    "\n",
    "\"The Nanny Diaries\" satirizes upper class Manhattan society as seen through the eyes of their children's caregivers.\n",
    "\n",
    "Written by the same author of \"The Lovely Bones\", \"The Pilot's Wife\" is the third novel in Shreve's informal trilogy to be set in a large beach house on the New Hampshire coast that used to be a conventis.\n",
    "\n",
    "\"Where the Heart Is\" dramatizes in detail the tribulations of lower-income and foster children in the United States.\n",
    "\n",
    "These three books sound right to me to be highly correlated with \"The Lovely Bones\". Seems our correlation recommender system is working.\n",
    "\n",
    "## Collaborative Filtering Using k-Nearest Neighbors (kNN)\n",
    "\n",
    "kNN is a machine learning algorithm to find clusters of similar users based on common book ratings, and make predictions using the average rating of top-k nearest neighbors. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from scipy.sparse import csr_matrix\n",
    "import sklearn\n",
    "from sklearn.decomposition import TruncatedSVD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userID</th>\n",
       "      <th>ISBN</th>\n",
       "      <th>bookRating</th>\n",
       "      <th>bookTitle</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>277427</td>\n",
       "      <td>002542730X</td>\n",
       "      <td>10</td>\n",
       "      <td>Politically Correct Bedtime Stories: Modern Ta...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3363</td>\n",
       "      <td>002542730X</td>\n",
       "      <td>0</td>\n",
       "      <td>Politically Correct Bedtime Stories: Modern Ta...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11676</td>\n",
       "      <td>002542730X</td>\n",
       "      <td>6</td>\n",
       "      <td>Politically Correct Bedtime Stories: Modern Ta...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>12538</td>\n",
       "      <td>002542730X</td>\n",
       "      <td>10</td>\n",
       "      <td>Politically Correct Bedtime Stories: Modern Ta...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>13552</td>\n",
       "      <td>002542730X</td>\n",
       "      <td>0</td>\n",
       "      <td>Politically Correct Bedtime Stories: Modern Ta...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userID        ISBN  bookRating  \\\n",
       "0  277427  002542730X          10   \n",
       "1    3363  002542730X           0   \n",
       "2   11676  002542730X           6   \n",
       "3   12538  002542730X          10   \n",
       "4   13552  002542730X           0   \n",
       "\n",
       "                                           bookTitle  \n",
       "0  Politically Correct Bedtime Stories: Modern Ta...  \n",
       "1  Politically Correct Bedtime Stories: Modern Ta...  \n",
       "2  Politically Correct Bedtime Stories: Modern Ta...  \n",
       "3  Politically Correct Bedtime Stories: Modern Ta...  \n",
       "4  Politically Correct Bedtime Stories: Modern Ta...  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "combine_book_rating = pd.merge(ratings, books, on='ISBN')\n",
    "columns = ['yearOfPublication', 'publisher', 'bookAuthor', 'imageUrlS', 'imageUrlM', 'imageUrlL']\n",
    "combine_book_rating = combine_book_rating.drop(columns, axis=1)\n",
    "combine_book_rating.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filter to only popular books\n",
    "Remove rows where book title is missing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "combine_book_rating = combine_book_rating.dropna(axis = 0, subset = ['bookTitle'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bookTitle</th>\n",
       "      <th>totalRatingCount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A Light in the Storm: The Civil War Diary of ...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Always Have Popsicles</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Apple Magic (The Collector's series)</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Beyond IBM: Leadership Marketing and Finance ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Clifford Visita El Hospital (Clifford El Gran...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           bookTitle  totalRatingCount\n",
       "0   A Light in the Storm: The Civil War Diary of ...                 2\n",
       "1                              Always Have Popsicles                 1\n",
       "2               Apple Magic (The Collector's series)                 1\n",
       "3   Beyond IBM: Leadership Marketing and Finance ...                 1\n",
       "4   Clifford Visita El Hospital (Clifford El Gran...                 1"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "book_ratingCount = (combine_book_rating.\n",
    "     groupby(by = ['bookTitle'])['bookRating'].\n",
    "     count().\n",
    "     reset_index().\n",
    "     rename(columns = {'bookRating': 'totalRatingCount'})\n",
    "     [['bookTitle', 'totalRatingCount']]\n",
    "    )\n",
    "book_ratingCount.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can merge the total rating count data into the rating data, giving us exactly what we need to filter out the lesser known books."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userID</th>\n",
       "      <th>ISBN</th>\n",
       "      <th>bookRating</th>\n",
       "      <th>bookTitle</th>\n",
       "      <th>totalRatingCount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>277427</td>\n",
       "      <td>002542730X</td>\n",
       "      <td>10</td>\n",
       "      <td>Politically Correct Bedtime Stories: Modern Ta...</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3363</td>\n",
       "      <td>002542730X</td>\n",
       "      <td>0</td>\n",
       "      <td>Politically Correct Bedtime Stories: Modern Ta...</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11676</td>\n",
       "      <td>002542730X</td>\n",
       "      <td>6</td>\n",
       "      <td>Politically Correct Bedtime Stories: Modern Ta...</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>12538</td>\n",
       "      <td>002542730X</td>\n",
       "      <td>10</td>\n",
       "      <td>Politically Correct Bedtime Stories: Modern Ta...</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>13552</td>\n",
       "      <td>002542730X</td>\n",
       "      <td>0</td>\n",
       "      <td>Politically Correct Bedtime Stories: Modern Ta...</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userID        ISBN  bookRating  \\\n",
       "0  277427  002542730X          10   \n",
       "1    3363  002542730X           0   \n",
       "2   11676  002542730X           6   \n",
       "3   12538  002542730X          10   \n",
       "4   13552  002542730X           0   \n",
       "\n",
       "                                           bookTitle  totalRatingCount  \n",
       "0  Politically Correct Bedtime Stories: Modern Ta...                82  \n",
       "1  Politically Correct Bedtime Stories: Modern Ta...                82  \n",
       "2  Politically Correct Bedtime Stories: Modern Ta...                82  \n",
       "3  Politically Correct Bedtime Stories: Modern Ta...                82  \n",
       "4  Politically Correct Bedtime Stories: Modern Ta...                82  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rating_with_totalRatingCount = combine_book_rating.merge(book_ratingCount, left_on = 'bookTitle', right_on = 'bookTitle', how = 'left')\n",
    "rating_with_totalRatingCount.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "count   160576.000\n",
      "mean         3.044\n",
      "std          7.428\n",
      "min          1.000\n",
      "25%          1.000\n",
      "50%          1.000\n",
      "75%          2.000\n",
      "max        365.000\n",
      "Name: totalRatingCount, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "pd.set_option('display.float_format', lambda x: '%.3f' % x)\n",
    "print(book_ratingCount['totalRatingCount'].describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The median book has only been rated one time. Let’s take a look at the top of the distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.900    5.000\n",
      "0.910    6.000\n",
      "0.920    7.000\n",
      "0.930    7.000\n",
      "0.940    8.000\n",
      "0.950   10.000\n",
      "0.960   11.000\n",
      "0.970   14.000\n",
      "0.980   19.000\n",
      "0.990   31.000\n",
      "Name: totalRatingCount, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "print(book_ratingCount['totalRatingCount'].quantile(np.arange(.9, 1, .01)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we have so many books in our data, we’ll limit it to the top 1%, this will give us 2713 different books."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userID</th>\n",
       "      <th>ISBN</th>\n",
       "      <th>bookRating</th>\n",
       "      <th>bookTitle</th>\n",
       "      <th>totalRatingCount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>277427</td>\n",
       "      <td>002542730X</td>\n",
       "      <td>10</td>\n",
       "      <td>Politically Correct Bedtime Stories: Modern Ta...</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3363</td>\n",
       "      <td>002542730X</td>\n",
       "      <td>0</td>\n",
       "      <td>Politically Correct Bedtime Stories: Modern Ta...</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11676</td>\n",
       "      <td>002542730X</td>\n",
       "      <td>6</td>\n",
       "      <td>Politically Correct Bedtime Stories: Modern Ta...</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>12538</td>\n",
       "      <td>002542730X</td>\n",
       "      <td>10</td>\n",
       "      <td>Politically Correct Bedtime Stories: Modern Ta...</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>13552</td>\n",
       "      <td>002542730X</td>\n",
       "      <td>0</td>\n",
       "      <td>Politically Correct Bedtime Stories: Modern Ta...</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userID        ISBN  bookRating  \\\n",
       "0  277427  002542730X          10   \n",
       "1    3363  002542730X           0   \n",
       "2   11676  002542730X           6   \n",
       "3   12538  002542730X          10   \n",
       "4   13552  002542730X           0   \n",
       "\n",
       "                                           bookTitle  totalRatingCount  \n",
       "0  Politically Correct Bedtime Stories: Modern Ta...                82  \n",
       "1  Politically Correct Bedtime Stories: Modern Ta...                82  \n",
       "2  Politically Correct Bedtime Stories: Modern Ta...                82  \n",
       "3  Politically Correct Bedtime Stories: Modern Ta...                82  \n",
       "4  Politically Correct Bedtime Stories: Modern Ta...                82  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "popularity_threshold = 20\n",
    "rating_popular_book = rating_with_totalRatingCount.query('totalRatingCount >= @popularity_threshold')\n",
    "rating_popular_book.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Filtering to US users only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(120880, 6)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userID</th>\n",
       "      <th>ISBN</th>\n",
       "      <th>bookRating</th>\n",
       "      <th>bookTitle</th>\n",
       "      <th>totalRatingCount</th>\n",
       "      <th>Location</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>277427</td>\n",
       "      <td>002542730X</td>\n",
       "      <td>10</td>\n",
       "      <td>Politically Correct Bedtime Stories: Modern Ta...</td>\n",
       "      <td>82</td>\n",
       "      <td>gilbert, arizona, usa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3363</td>\n",
       "      <td>002542730X</td>\n",
       "      <td>0</td>\n",
       "      <td>Politically Correct Bedtime Stories: Modern Ta...</td>\n",
       "      <td>82</td>\n",
       "      <td>knoxville, tennessee, usa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>12538</td>\n",
       "      <td>002542730X</td>\n",
       "      <td>10</td>\n",
       "      <td>Politically Correct Bedtime Stories: Modern Ta...</td>\n",
       "      <td>82</td>\n",
       "      <td>byron, minnesota, usa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>13552</td>\n",
       "      <td>002542730X</td>\n",
       "      <td>0</td>\n",
       "      <td>Politically Correct Bedtime Stories: Modern Ta...</td>\n",
       "      <td>82</td>\n",
       "      <td>cordova, tennessee, usa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>16795</td>\n",
       "      <td>002542730X</td>\n",
       "      <td>0</td>\n",
       "      <td>Politically Correct Bedtime Stories: Modern Ta...</td>\n",
       "      <td>82</td>\n",
       "      <td>mechanicsville, maryland, usa</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userID        ISBN  bookRating  \\\n",
       "0  277427  002542730X          10   \n",
       "1    3363  002542730X           0   \n",
       "3   12538  002542730X          10   \n",
       "4   13552  002542730X           0   \n",
       "5   16795  002542730X           0   \n",
       "\n",
       "                                           bookTitle  totalRatingCount  \\\n",
       "0  Politically Correct Bedtime Stories: Modern Ta...                82   \n",
       "1  Politically Correct Bedtime Stories: Modern Ta...                82   \n",
       "3  Politically Correct Bedtime Stories: Modern Ta...                82   \n",
       "4  Politically Correct Bedtime Stories: Modern Ta...                82   \n",
       "5  Politically Correct Bedtime Stories: Modern Ta...                82   \n",
       "\n",
       "                        Location  \n",
       "0          gilbert, arizona, usa  \n",
       "1      knoxville, tennessee, usa  \n",
       "3          byron, minnesota, usa  \n",
       "4        cordova, tennessee, usa  \n",
       "5  mechanicsville, maryland, usa  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "combined = rating_popular_book.merge(users, left_on = 'userID', right_on = 'userID', how = 'left')\n",
    "\n",
    "us_canada_user_rating = combined[combined['Location'].str.contains(\"usa|canada\")]\n",
    "us_canada_user_rating=us_canada_user_rating.drop('Age', axis=1)\n",
    "\n",
    "print(us_canada_user_rating.shape)\n",
    "us_canada_user_rating.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial dataframe shape (120880, 6)\n",
      "New dataframe shape (118214, 6)\n",
      "Removed 2666 rows\n"
     ]
    }
   ],
   "source": [
    "if not us_canada_user_rating[us_canada_user_rating.duplicated(['userID', 'bookTitle'])].empty:\n",
    "    initial_rows = us_canada_user_rating.shape[0]\n",
    "\n",
    "    print('Initial dataframe shape {0}'.format(us_canada_user_rating.shape))\n",
    "    us_canada_user_rating = us_canada_user_rating.drop_duplicates(['userID', 'bookTitle'])\n",
    "    current_rows = us_canada_user_rating.shape[0]\n",
    "    print('New dataframe shape {0}'.format(us_canada_user_rating.shape))\n",
    "    print('Removed {0} rows'.format(initial_rows - current_rows))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "us_canada_user_rating_pivot = us_canada_user_rating.pivot(index = 'bookTitle', columns = 'userID', values = 'bookRating').fillna(0)\n",
    "us_canada_user_rating_matrix = csr_matrix(us_canada_user_rating_pivot.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementing kNN\n",
    "We convert our table to a 2D matrix, and fill the missing values with zeros (since we will calculate distances between rating vectors). We then transform the values(ratings) of the matrix dataframe into a scipy sparse matrix for more efficient calculations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NearestNeighbors(algorithm='brute', leaf_size=30, metric='cosine',\n",
       "         metric_params=None, n_jobs=1, n_neighbors=5, p=2, radius=1.0)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.neighbors import NearestNeighbors\n",
    "\n",
    "model_knn = NearestNeighbors(metric = 'cosine', algorithm = 'brute')\n",
    "model_knn.fit(us_canada_user_rating_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recommendations for Wild Justice:\n",
      "\n",
      "1: Star Flight, with distance of 0.15200169599491198:\n",
      "2: Lost &amp; Found (Red Dress Ink), with distance of 0.3646214561787957:\n",
      "3: After the Fire, with distance of 0.40482186977570034:\n",
      "4: Tell Me No Secrets, with distance of 0.484223062563475:\n",
      "5: Girls' Poker Night, with distance of 0.4858255032709391:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/adhiman/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:2: FutureWarning: reshape is deprecated and will raise in a subsequent release. Please use .values.reshape(...) instead\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "query_index = np.random.choice(us_canada_user_rating_pivot.shape[0])\n",
    "distances, indices = model_knn.kneighbors(us_canada_user_rating_pivot.iloc[query_index, :].reshape(1, -1), n_neighbors = 6)\n",
    "\n",
    "for i in range(0, len(distances.flatten())):\n",
    "    if i == 0:\n",
    "        print('Recommendations for {0}:\\n'.format(us_canada_user_rating_pivot.index[query_index]))\n",
    "    else:\n",
    "        print('{0}: {1}, with distance of {2}:'.format(i, us_canada_user_rating_pivot.index[indices.flatten()[i]], distances.flatten()[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Collaborative Filtering Using Matrix Factorization\n",
    "\n",
    "We use singular value decomposition (SVD) — one of the Matrix Factorization models for identifying latent factors.\n",
    "\n",
    "Similar with kNN, we convert our USA Canada user rating table into a 2D matrix (called a utility matrix here) and fill the missing values with zeros."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "us_canada_user_rating_pivot2 = us_canada_user_rating.pivot(index = 'userID', columns = 'bookTitle', values = 'bookRating').fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>bookTitle</th>\n",
       "      <th>10 Lb. Penalty</th>\n",
       "      <th>101 Dalmatians</th>\n",
       "      <th>16 Lighthouse Road</th>\n",
       "      <th>1984</th>\n",
       "      <th>1st to Die: A Novel</th>\n",
       "      <th>2010: Odyssey Two</th>\n",
       "      <th>204 Rosewood Lane</th>\n",
       "      <th>2061: Odyssey Three</th>\n",
       "      <th>24 Hours</th>\n",
       "      <th>2nd Chance</th>\n",
       "      <th>...</th>\n",
       "      <th>You'll Never Eat Lunch in This Town Again</th>\n",
       "      <th>Young Wives</th>\n",
       "      <th>Your Erroneous Zones</th>\n",
       "      <th>Zen and the Art of Motorcycle Maintenance: An Inquiry into Values</th>\n",
       "      <th>Zia</th>\n",
       "      <th>Zlata's Diary: A Child's Life in Sarajevo</th>\n",
       "      <th>Zombies of the Gene Pool</th>\n",
       "      <th>Zoya</th>\n",
       "      <th>\\O\\\" Is for Outlaw\"</th>\n",
       "      <th>\\Surely You're Joking, Mr. Feynman!\\\": Adventures of a Curious Character\"</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>userID</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>254</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>9.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2276</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>10.000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>10.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2766</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2977</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3363</th>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 3166 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "bookTitle  10 Lb. Penalty  101 Dalmatians  16 Lighthouse Road  1984  \\\n",
       "userID                                                                \n",
       "254                 0.000           0.000               0.000 9.000   \n",
       "2276                0.000           0.000               0.000 0.000   \n",
       "2766                0.000           0.000               0.000 0.000   \n",
       "2977                0.000           0.000               0.000 0.000   \n",
       "3363                0.000           0.000               0.000 0.000   \n",
       "\n",
       "bookTitle  1st to Die: A Novel  2010: Odyssey Two  204 Rosewood Lane  \\\n",
       "userID                                                                 \n",
       "254                      0.000              0.000              0.000   \n",
       "2276                     0.000              0.000              0.000   \n",
       "2766                     0.000              0.000              0.000   \n",
       "2977                     0.000              0.000              0.000   \n",
       "3363                     0.000              0.000              0.000   \n",
       "\n",
       "bookTitle  2061: Odyssey Three  24 Hours  2nd Chance  \\\n",
       "userID                                                 \n",
       "254                      0.000     0.000       0.000   \n",
       "2276                     0.000     0.000      10.000   \n",
       "2766                     0.000     0.000       0.000   \n",
       "2977                     0.000     0.000       0.000   \n",
       "3363                     0.000     0.000       0.000   \n",
       "\n",
       "bookTitle                                    ...                                      \\\n",
       "userID                                       ...                                       \n",
       "254                                          ...                                       \n",
       "2276                                         ...                                       \n",
       "2766                                         ...                                       \n",
       "2977                                         ...                                       \n",
       "3363                                         ...                                       \n",
       "\n",
       "bookTitle  You'll Never Eat Lunch in This Town Again  Young Wives  \\\n",
       "userID                                                              \n",
       "254                                            0.000        0.000   \n",
       "2276                                           0.000        0.000   \n",
       "2766                                           0.000        0.000   \n",
       "2977                                           0.000        0.000   \n",
       "3363                                           0.000        0.000   \n",
       "\n",
       "bookTitle  Your Erroneous Zones  \\\n",
       "userID                            \n",
       "254                       0.000   \n",
       "2276                      0.000   \n",
       "2766                      0.000   \n",
       "2977                      0.000   \n",
       "3363                      0.000   \n",
       "\n",
       "bookTitle  Zen and the Art of Motorcycle Maintenance: An Inquiry into Values  \\\n",
       "userID                                                                         \n",
       "254                                                    0.000                   \n",
       "2276                                                   0.000                   \n",
       "2766                                                   0.000                   \n",
       "2977                                                   0.000                   \n",
       "3363                                                   0.000                   \n",
       "\n",
       "bookTitle   Zia  Zlata's Diary: A Child's Life in Sarajevo  \\\n",
       "userID                                                       \n",
       "254       0.000                                      0.000   \n",
       "2276      0.000                                      0.000   \n",
       "2766      0.000                                      0.000   \n",
       "2977      0.000                                      0.000   \n",
       "3363      0.000                                      0.000   \n",
       "\n",
       "bookTitle  Zombies of the Gene Pool  Zoya  \\O\\\" Is for Outlaw\"  \\\n",
       "userID                                                           \n",
       "254                           0.000 0.000                0.000   \n",
       "2276                         10.000 0.000                0.000   \n",
       "2766                          0.000 0.000                0.000   \n",
       "2977                          0.000 0.000                0.000   \n",
       "3363                          0.000 0.000                0.000   \n",
       "\n",
       "bookTitle  \\Surely You're Joking, Mr. Feynman!\\\": Adventures of a Curious Character\"  \n",
       "userID                                                                                \n",
       "254                                                    0.000                          \n",
       "2276                                                   0.000                          \n",
       "2766                                                   0.000                          \n",
       "2977                                                   0.000                          \n",
       "3363                                                   0.000                          \n",
       "\n",
       "[5 rows x 3166 columns]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "us_canada_user_rating_pivot2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(734, 3166)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "us_canada_user_rating_pivot2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3166, 734)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = us_canada_user_rating_pivot2.values.T\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3166, 12)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sklearn\n",
    "from sklearn.decomposition import TruncatedSVD\n",
    "\n",
    "SVD = TruncatedSVD(n_components=12, random_state=17)\n",
    "matrix = SVD.fit_transform(X)\n",
    "matrix.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3166, 3166)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import  warnings\n",
    "warnings.filterwarnings(\"ignore\",category =RuntimeWarning)\n",
    "corr = np.corrcoef(matrix)\n",
    "corr.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2492\n"
     ]
    }
   ],
   "source": [
    "us_canada_book_title = us_canada_user_rating_pivot2.columns\n",
    "us_canada_book_list = list(us_canada_book_title)\n",
    "coffey_hands = us_canada_book_list.index(\"The Green Mile: Coffey's Hands (Green Mile Series)\")\n",
    "print(coffey_hands)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "corr_coffey_hands  = corr[coffey_hands]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['K Is for Killer (Grafton, Sue. Kinsey Millhone Mysteries.)',\n",
       " 'Love and Marriage',\n",
       " 'The Girl Who Loved Tom Gordon : A Novel',\n",
       " 'The Green Mile: Night Journey (Green Mile Series)',\n",
       " 'The Green Mile: The Mouse on the Mile (Green Mile Series)',\n",
       " 'The Two Dead Girls (Green Mile Series)']"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(us_canada_book_title[(corr_coffey_hands<1.0) & (corr_coffey_hands>0.9)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}