03-user-based-similarity.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# User Based Similarity\n",
    "\n",
    "In this notebook we will compute movie similarity between a pair of movies based on the number of users that liked both. We will use the following correlation measure:\n",
    "\n",
    "$$\n",
    "S_{1,2} = \\displaystyle\\frac{n_{12}}{\\sqrt{n_1n_2}}\n",
    "$$\n",
    "\n",
    "- $n_1 = $ number of users that liked movie 1.\n",
    "- $n_2 = $ number of users that liked movie 2.\n",
    "- $n_{12} = $ number of users that liked both movie 1 and 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import vaex\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find the number of users that liked each pair of movies we will need to compute a cartesian join with the ratings dataset. The `pandas` library is not able to perform this task. It will require lots of RAM and will take several minutes. So, we decided to try the `vaex` library. It allows to split data into chuncks and perform operations chunck by chunk. It is very efficient!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(17130115, 2)\n",
      "(15967, 2)\n",
      "CPU times: user 10.3 s, sys: 3.86 s, total: 14.2 s\n",
      "Wall time: 3.75 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "ratings = vaex.read_csv('data/movie_ratings.csv', convert=True, chunk_size=1_000_000)\n",
    "ratings = ratings[ratings['rating'] >= 3.5].extract()\n",
    "ratings.drop(columns=['rating'], inplace=True)\n",
    "\n",
    "movies = ratings.groupby(by='id', agg={'num_likes': vaex.agg.nunique('user')})\n",
    "\n",
    "print(ratings.shape)\n",
    "print(movies.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To reduce time and memory consumption even more we will randomly select a subset of 500 movies. They will be used to evaluate the embedding features. We will use the number of likes as the probability to select a movie."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(4599307, 2)\n"
     ]
    }
   ],
   "source": [
    "NUM_EVENTS = 500\n",
    "\n",
    "prob = movies['num_likes'].values / movies['num_likes'].sum()\n",
    "\n",
    "np.random.seed(0)\n",
    "ids = np.random.choice(movies['id'].values, size=NUM_EVENTS, p=prob)\n",
    "\n",
    "ratings = ratings[ratings['id'].isin(ids)].extract()\n",
    "\n",
    "print(ratings.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have to join movies on user so that we will identify each user that liked each pair of movies. Then, we will aggregate by movie pairs and compute the `n_{12}` counts. Using `pandas` took several minutes and resulted in memory overflown. But `vaex` easily solved the problem!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(238593639, 3)\n",
      "CPU times: user 7.47 s, sys: 12.6 s, total: 20.1 s\n",
      "Wall time: 10.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "likes_left = ratings.copy()\n",
    "likes_left.rename('id', 'id_left')\n",
    "\n",
    "likes_right = ratings.copy()\n",
    "likes_right.rename('id', 'id_right')\n",
    "\n",
    "movie_sim = likes_left.join(likes_right, on='user', allow_duplication=True)\n",
    "\n",
    "print(movie_sim.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Such a huge dataset!!! Let's remove the self relations (movie with itself)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(233994332, 3)\n",
      "CPU times: user 6.42 s, sys: 1.76 s, total: 8.19 s\n",
      "Wall time: 3.39 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "movie_sim = movie_sim[movie_sim['id_left'] != movie_sim['id_right']].extract()\n",
    "print(movie_sim.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need the information about number of likes for each movie from the pairs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 9.4 s, sys: 156 ms, total: 9.55 s\n",
      "Wall time: 793 ms\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th>#                            </th><th style=\"text-align: right;\">  user</th><th style=\"text-align: right;\">  id_left</th><th style=\"text-align: right;\">  id_right</th><th style=\"text-align: right;\">  num_likes_left</th><th style=\"text-align: right;\">  num_likes_right</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td><i style='opacity: 0.6'>0</i></td><td style=\"text-align: right;\">     2</td><td style=\"text-align: right;\">     2352</td><td style=\"text-align: right;\">      2108</td><td style=\"text-align: right;\">            5069</td><td style=\"text-align: right;\">             4787</td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>1</i></td><td style=\"text-align: right;\">     4</td><td style=\"text-align: right;\">       34</td><td style=\"text-align: right;\">         2</td><td style=\"text-align: right;\">           21916</td><td style=\"text-align: right;\">            12345</td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>2</i></td><td style=\"text-align: right;\">     4</td><td style=\"text-align: right;\">       50</td><td style=\"text-align: right;\">         2</td><td style=\"text-align: right;\">           55297</td><td style=\"text-align: right;\">            12345</td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>3</i></td><td style=\"text-align: right;\">     4</td><td style=\"text-align: right;\">       62</td><td style=\"text-align: right;\">         2</td><td style=\"text-align: right;\">           14857</td><td style=\"text-align: right;\">            12345</td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>4</i></td><td style=\"text-align: right;\">     4</td><td style=\"text-align: right;\">      110</td><td style=\"text-align: right;\">         2</td><td style=\"text-align: right;\">           53582</td><td style=\"text-align: right;\">            12345</td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>5</i></td><td style=\"text-align: right;\">     4</td><td style=\"text-align: right;\">      145</td><td style=\"text-align: right;\">         2</td><td style=\"text-align: right;\">            7504</td><td style=\"text-align: right;\">            12345</td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>6</i></td><td style=\"text-align: right;\">     4</td><td style=\"text-align: right;\">      150</td><td style=\"text-align: right;\">         2</td><td style=\"text-align: right;\">           42984</td><td style=\"text-align: right;\">            12345</td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>7</i></td><td style=\"text-align: right;\">     4</td><td style=\"text-align: right;\">      161</td><td style=\"text-align: right;\">         2</td><td style=\"text-align: right;\">           16972</td><td style=\"text-align: right;\">            12345</td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>8</i></td><td style=\"text-align: right;\">     4</td><td style=\"text-align: right;\">      165</td><td style=\"text-align: right;\">         2</td><td style=\"text-align: right;\">           22340</td><td style=\"text-align: right;\">            12345</td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>9</i></td><td style=\"text-align: right;\">     4</td><td style=\"text-align: right;\">      223</td><td style=\"text-align: right;\">         2</td><td style=\"text-align: right;\">           19729</td><td style=\"text-align: right;\">            12345</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "  #    user    id_left    id_right    num_likes_left    num_likes_right\n",
       "  0       2       2352        2108              5069               4787\n",
       "  1       4         34           2             21916              12345\n",
       "  2       4         50           2             55297              12345\n",
       "  3       4         62           2             14857              12345\n",
       "  4       4        110           2             53582              12345\n",
       "  5       4        145           2              7504              12345\n",
       "  6       4        150           2             42984              12345\n",
       "  7       4        161           2             16972              12345\n",
       "  8       4        165           2             22340              12345\n",
       "  9       4        223           2             19729              12345"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "movie_sim = movie_sim.join(movies, left_on='id_left', right_on='id')\n",
    "movie_sim.drop(columns=['id'], inplace=True)\n",
    "movie_sim.rename('num_likes', 'num_likes_left')\n",
    "\n",
    "movie_sim = movie_sim.join(movies, left_on='id_right', right_on='id')\n",
    "movie_sim.drop(columns=['id'], inplace=True)\n",
    "movie_sim.rename('num_likes', 'num_likes_right')\n",
    "\n",
    "movie_sim.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can compute the movie similarity (user correlation):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 38.6 s, sys: 188 ms, total: 38.8 s\n",
      "Wall time: 1.92 s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th>#                            </th><th style=\"text-align: right;\">  id_left</th><th style=\"text-align: right;\">  id_right</th><th style=\"text-align: right;\">  num_likes_both</th><th style=\"text-align: right;\">  num_likes_left</th><th style=\"text-align: right;\">  num_likes_right</th><th style=\"text-align: right;\">  similarity</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td><i style='opacity: 0.6'>0</i></td><td style=\"text-align: right;\">    32587</td><td style=\"text-align: right;\">     45499</td><td style=\"text-align: right;\">            2811</td><td style=\"text-align: right;\">           18106</td><td style=\"text-align: right;\">             6252</td><td style=\"text-align: right;\">   0.264204 </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>1</i></td><td style=\"text-align: right;\">    32587</td><td style=\"text-align: right;\">     48982</td><td style=\"text-align: right;\">             434</td><td style=\"text-align: right;\">           18106</td><td style=\"text-align: right;\">             1028</td><td style=\"text-align: right;\">   0.100596 </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>2</i></td><td style=\"text-align: right;\">    32587</td><td style=\"text-align: right;\">     50872</td><td style=\"text-align: right;\">            4735</td><td style=\"text-align: right;\">           18106</td><td style=\"text-align: right;\">            15446</td><td style=\"text-align: right;\">   0.28314  </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>3</i></td><td style=\"text-align: right;\">    32587</td><td style=\"text-align: right;\">     52722</td><td style=\"text-align: right;\">            1297</td><td style=\"text-align: right;\">           18106</td><td style=\"text-align: right;\">             3253</td><td style=\"text-align: right;\">   0.169    </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>4</i></td><td style=\"text-align: right;\">    32587</td><td style=\"text-align: right;\">     58559</td><td style=\"text-align: right;\">           10553</td><td style=\"text-align: right;\">           18106</td><td style=\"text-align: right;\">            38909</td><td style=\"text-align: right;\">   0.397594 </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>5</i></td><td style=\"text-align: right;\">    32587</td><td style=\"text-align: right;\">     60069</td><td style=\"text-align: right;\">            7051</td><td style=\"text-align: right;\">           18106</td><td style=\"text-align: right;\">            23500</td><td style=\"text-align: right;\">   0.341826 </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>6</i></td><td style=\"text-align: right;\">    32587</td><td style=\"text-align: right;\">       110</td><td style=\"text-align: right;\">            8772</td><td style=\"text-align: right;\">           18106</td><td style=\"text-align: right;\">            53582</td><td style=\"text-align: right;\">   0.281629 </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>7</i></td><td style=\"text-align: right;\">    32587</td><td style=\"text-align: right;\">       112</td><td style=\"text-align: right;\">             804</td><td style=\"text-align: right;\">           18106</td><td style=\"text-align: right;\">             6558</td><td style=\"text-align: right;\">   0.0737835</td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>8</i></td><td style=\"text-align: right;\">    32587</td><td style=\"text-align: right;\">       145</td><td style=\"text-align: right;\">            1815</td><td style=\"text-align: right;\">           18106</td><td style=\"text-align: right;\">             7504</td><td style=\"text-align: right;\">   0.155711 </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>9</i></td><td style=\"text-align: right;\">    32587</td><td style=\"text-align: right;\">       165</td><td style=\"text-align: right;\">            4482</td><td style=\"text-align: right;\">           18106</td><td style=\"text-align: right;\">            22340</td><td style=\"text-align: right;\">   0.222853 </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "  #    id_left    id_right    num_likes_both    num_likes_left    num_likes_right    similarity\n",
       "  0      32587       45499              2811             18106               6252     0.264204\n",
       "  1      32587       48982               434             18106               1028     0.100596\n",
       "  2      32587       50872              4735             18106              15446     0.28314\n",
       "  3      32587       52722              1297             18106               3253     0.169\n",
       "  4      32587       58559             10553             18106              38909     0.397594\n",
       "  5      32587       60069              7051             18106              23500     0.341826\n",
       "  6      32587         110              8772             18106              53582     0.281629\n",
       "  7      32587         112               804             18106               6558     0.0737835\n",
       "  8      32587         145              1815             18106               7504     0.155711\n",
       "  9      32587         165              4482             18106              22340     0.222853"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "agg_funs = {'num_likes_both':vaex.agg.count('user'), 'num_likes_left':vaex.agg.max('num_likes_left'), 'num_likes_right':vaex.agg.max('num_likes_right')}\n",
    "movie_sim = movie_sim.groupby(by=['id_left','id_right'], agg=agg_funs)\n",
    "movie_sim['similarity'] = movie_sim['num_likes_both'] / np.exp(0.5*np.log(movie_sim['num_likes_left']) + 0.5*np.log(movie_sim['num_likes_right']))\n",
    "movie_sim.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f7ae8104990>]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAZBklEQVR4nO3df/RcdX3n8edL8Adi00IJGBLYYJuuJ1BFSRF17dJFT6mtBruo0UWisieWtf7o2W6F3XPE6nJKu/2luNJyqAKtiKxYkyqUKi5td8XgF02LRG2zhkLgW4jCMeh2EfC9f8xNmXy/k2S+yXdm7sw8H+fM+d753M+d+XzOKK987v3cz01VIUlS2zxp1A2QJKkXA0qS1EoGlCSplQwoSVIrGVCSpFY6dNQNGLajjjqqVq5cOepmSJIat99++7eqaunc8qkLqJUrVzIzMzPqZkiSGkn+oVe5p/gkSa1kQEmSWsmAkiS1kgElSWolA0qS1EoGlCSplQwoSVIrGVCSpFaauht1h+GazXezccu9PfetPXk5r3/B8UNukSSNH0dQA7Bxy71snd01r3zr7K69BpckaU8DG0El+TDwC8ADVXVSU3Yk8HFgJXAX8JqqeqjZdyFwHvA48PaquqkpPwW4EjgMuAF4R1VVkqcCVwOnAN8GXltVdw2qPwu1etkSPv6WF+5R9to/vHVErZGk8TPIEdSVwJlzyi4Abq6qVcDNzXuSrAbWASc2x3woySHNMZcBG4BVzWv3Z54HPFRVPw78HvCbA+uJJGnoBhZQVfVXwINzitcCVzXbVwFndZVfW1WPVNV2YBtwapJlwJKqurWqis6I6awen/UJ4IwkGURfJEnDN+xrUMdU1SxA8/fopnw5cE9XvR1N2fJme275HsdU1WPAd4Af7fWlSTYkmUkys3PnzkXqiiRpkNoySaLXyKf2Ub6vY+YXVl1eVWuqas3SpfMeOSJJaqFhB9T9zWk7mr8PNOU7gOO66q0A7mvKV/Qo3+OYJIcCP8z8U4qSpDE17IDaBKxvttcDG7vK1yV5apIT6EyGuK05DfhwktOa60vnzjlm92edDXy+uU4lSZoAg5xm/jHgdOCoJDuAi4BLgOuSnAfcDbwaoKruTHIdsBV4DHhrVT3efNT5PDHN/MbmBfBHwB8n2UZn5LRuUH2RJA3fwAKqql63l11n7KX+xcDFPcpngJN6lP8/moAbJ1tnd827H8rVJSRpPpc6GqK1Jy+fV7Z7xQkDSpL2ZEAN0etfcPy8IHJ1CUnqrS3TzCVJ2oMBJUlqJQNKktRKBpQkqZWcJHGQej2ccOvsLlYvWzKiFknSZHAEdZB6PZxw9bIlPaeUS5L65whqEfR6OKEk6eAYUC3g6hKSNJ8BNWKuLiFJvRlQI+bqEpLUm5MkJEmtZEBJklrJgJIktZIBJUlqJQNKktRKzuIbE72WVALvl5I0uRxBjYleSyptnd3VM7QkaRI4gmqpuatL7F6AtntJJe+XkjTJDKgW6rW6hAvQSpo2BlQL9VpdQpKmjQE15lxoVtKkMqDGmAvNSppkBtQYc6FZSZPMaeaSpFYyoCRJrWRASZJayYCSJLWSASVJaiUDSpLUSgaUJKmVDChJUiuNJKCS/EqSO5N8NcnHkjwtyZFJPpvk75u/R3TVvzDJtiTfSPKzXeWnJLmj2feBJBlFfyRJi2/oAZVkOfB2YE1VnQQcAqwDLgBurqpVwM3Ne5KsbvafCJwJfCjJIc3HXQZsAFY1rzOH2BVJ0gCN6hTfocBhSQ4Fng7cB6wFrmr2XwWc1WyvBa6tqkeqajuwDTg1yTJgSVXdWlUFXN11jCRpzA09oKrqXuC3gbuBWeA7VfUXwDFVNdvUmQWObg5ZDtzT9RE7mrLlzfbc8nmSbEgyk2Rm586di9kdSdKAjOIU3xF0RkUnAMcChyc5Z1+H9CirfZTPL6y6vKrWVNWapUuXLrTJkqQRGMUpvpcC26tqZ1U9CnwSeBFwf3PajubvA039HcBxXcevoHNKcEezPbdckjQBRhFQdwOnJXl6M+vuDOBrwCZgfVNnPbCx2d4ErEvy1CQn0JkMcVtzGvDhJKc1n3Nu1zGSpDE39OdBVdXmJJ8Avgw8BnwFuBx4BnBdkvPohNirm/p3JrkO2NrUf2tVPd583PnAlcBhwI3Na+r5lF1Jk2AkDyysqouAi+YUP0JnNNWr/sXAxT3KZ4CTFr2BY8yn7EqaFD5Rd8L4lF1Jk8KljiRJrWRASZJayVN8U8KJE5LGjQE1BZw4IWkcGVBTwIkTksaR16AkSa1kQEmSWsmAkiS1kgElSWolA0qS1EoGlCSplQwoSVIreR/UFHN1CUltZkBNKVeXkNR2BtSUcnUJSW3nNShJUisZUJKkVjKgJEmtZEBJklrJgJIktZIBJUlqJQNKktRK3gelPbi6hKS2MKD0z1xdQlKbGFD6Z64uIalNvAYlSWolA0qS1EoGlCSplQwoSVIrGVCSpFYyoCRJrWRASZJaaST3QSX5EeAK4CSggDcD3wA+DqwE7gJeU1UPNfUvBM4DHgfeXlU3NeWnAFcChwE3AO+oqhpeT6aDq0tIGoVRjaDeD/x5VT0beC7wNeAC4OaqWgXc3LwnyWpgHXAicCbwoSSHNJ9zGbABWNW8zhxmJ6bB2pOXs3rZkj3Kts7uYuOWe0fUIknTYugjqCRLgJ8G3ghQVd8Hvp9kLXB6U+0q4BbgXcBa4NqqegTYnmQbcGqSu4AlVXVr87lXA2cBNw6pK1PB1SUkjcooRlDPAnYCH0nylSRXJDkcOKaqZgGav0c39ZcD93Qdv6MpW95szy2fJ8mGJDNJZnbu3Lm4vZEkDcSCAirJ4V2n1w7UocDzgcuq6nnA92hO5+3ta3uU1T7K5xdWXV5Va6pqzdKlSxfaXknSCOwzoJI8Kcnrk3wmyQPA14HZJHcm+W9JVh3Ad+4AdlTV5ub9J+gE1v1JljXfuwx4oKv+cV3HrwDua8pX9CiXJE2A/Y2g/ifwY8CFwDOr6riqOhp4CfBF4JIk5yzkC6vqH4F7kvzLpugMYCuwCVjflK0HNjbbm4B1SZ6a5AQ6kyFua04DPpzktCQBzu06RpI05vY3SeKlVfXo3MKqehC4Hrg+yZMP4HvfBnw0yVOAbwJvohOW1yU5D7gbeHXzXXcmuY5OiD0GvLWqHm8+53yemGZ+I06QkKSJsc+AqqpHk7y+qq5Jsq6qru1VZ6FfWlVbgDU9dp2xl/oXAxf3KJ+hcy+VJGnC9DNJYnmS17Dn9R5JkgZqnyOoJBcBTwN+A/itJO+uqvcOpWUt9Ot/didb79u1R9nW2V3zbmSdBq4uIWnQ9jmCqqpfBx4EzgEenOZw2pvVy5aw9uSet19NLFeXkDQM/awkcV9VXZvkdQNvTctd9IoTR92EVnB1CUnDsL/7oJ5RVR8FqKqP7a3OIBomSZpu+5sksTHJ7yT56WY5IgCSPCvJeUluwgVaJUkDsL9p5mckeTnwFuDFSY6gcy/SN+g83mJ9c+OtJEmLar/XoKrqBjphJEnS0PS1WGySm/spkyRpsezvPqinAU8HjmpO7+1eQXwJcOyA2yZJmmL7O8X3FuCddMLodp4IqF3Afx9csyRJ025/kyTeD7w/yduq6tIhtUljqtfqEuAKE5IOTF+PfK+qS5O8CFjZfUxVXT2gdmnM7G01ja2znaWhDChJC9VXQCX5YzrPhdoC7H7URQEGlIDeq0uAK0xIOnB9BRSdR2Osrqqej1SXJGmx9RtQXwWeCcwOsC2aUK58LulA9BtQRwFbk9wGPLK7sKpeOZBWaWL0ujbldSlJ/eg3oN4zyEZocrnyuaQD1e8svr8cdEMkSerW7yy+h+nM2gN4CvBk4HtVNX2PkpUkDUW/I6gf6n6f5Czg1EE0SJIk6HOx2Lmq6lPAv1ncpkiS9IR+T/H9YtfbJ9G5L8p7oiRJA9PvLL5XdG0/BtwFrF301kiS1Oj3GtSbBt0QTRdv3pW0P/0+sHBFkj9N8kCS+5Ncn2TFoBunybT25OWsXrbnBNCts7vYuOXeEbVIUhv1e4rvI8A1wKub9+c0ZS8bRKM02bx5V1I/+p3Ft7SqPlJVjzWvK4GlA2yXJGnK9RtQ30pyTpJDmtc5wLcH2TBJ0nTrN6DeDLwG+Ec6K5qfDThxQpI0MP1eg3ofsL6qHgJIciTw23SCS5KkRdfvCOo5u8MJoKoeBJ43mCZJktT/COpJSY6YM4Lq91ipL94bJalbvyOo3wG+kOR9Sd4LfAH4rYP54mayxVeSfLp5f2SSzyb5++bvEV11L0yyLck3kvxsV/kpSe5o9n0gSQ6mTRod742SNFe/K0lcnWSGzgKxAX6xqrYe5He/A/gasPu/ShcAN1fVJUkuaN6/K8lqYB1wInAs8LkkP1FVjwOXARuALwI3AGcCNx5kuzQC3hslaa6+VzOvqq1V9cGquvRgw6lZheLngSu6itcCVzXbVwFndZVfW1WPVNV2YBtwapJlwJKqurWqCri66xhJ0pg7oMdtLILfB34N+EFX2TFVNQvQ/D26KV8O3NNVb0dTtrzZnls+T5INSWaSzOzcuXNROiBJGqyhB1SSXwAeqKrb+z2kR1nto3x+YdXlVbWmqtYsXeoCGJI0DkYxE+/FwCuTvBx4GrAkyZ8A9ydZVlWzzem7B5r6O4Djuo5fAdzXlK/oUS5JmgBDH0FV1YVVtaKqVtKZ/PD5qjoH2ASsb6qtBzY225uAdUmemuQEYBVwW3Ma8OEkpzWz987tOkaSNObadC/TJcB1Sc4D7qZZOb2q7kxyHbCVzsMS39rM4AM4H7gSOIzO7D1n8E0Y742SptdIA6qqbgFuaba/DZyxl3oXAxf3KJ8BThpcCzVKa0+eP+dl6+wuAANKmgJtGkFJe/DeKGm6jWqauSRJ+2RASZJayYCSJLWSASVJaiUDSpLUSgaUJKmVnGausePNu9J0MKA0Vrx5V5oeBpTGijfvStPDa1CSpFYyoCRJrWRASZJayYCSJLWSkyQ0EZx6Lk0eA0pjr9fU883bH2Tz9gfZuOXeeXUNLWk8GFAae72mnl+z+e554eT9UtJ4MaA0kbxfShp/TpKQJLWSASVJaiUDSpLUSgaUJKmVDChJUis5i09TxRt6pfFhQGlq+CwpabwYUJoa3hsljRevQUmSWskRlKae16WkdjKgNNW8LiW1lwGlqeZ1Kam9vAYlSWolR1BSD16XkkbPgJLm8AGIUjsMPaCSHAdcDTwT+AFweVW9P8mRwMeBlcBdwGuq6qHmmAuB84DHgbdX1U1N+SnAlcBhwA3AO6qqhtkfTR4fgCi1wyhGUI8B/7Gqvpzkh4Dbk3wWeCNwc1VdkuQC4ALgXUlWA+uAE4Fjgc8l+Ymqehy4DNgAfJFOQJ0J3Dj0HmniOZlCGr6hT5Koqtmq+nKz/TDwNWA5sBa4qql2FXBWs70WuLaqHqmq7cA24NQky4AlVXVrM2q6uusYSdKYG+k1qCQrgecBm4FjqmoWOiGW5Oim2nI6I6TddjRljzbbc8t7fc8GOiMtjj/e0zFaPE6mkAZnZNPMkzwDuB54Z1Xt2lfVHmW1j/L5hVWXV9WaqlqzdOnShTdW6mHtyctZvWzJHmVbZ3fNu1Yl6cCMZASV5Ml0wumjVfXJpvj+JMua0dMy4IGmfAdwXNfhK4D7mvIVPcqlofC6lDRYo5jFF+CPgK9V1e927doErAcuaf5u7Cq/Jsnv0pkksQq4raoeT/JwktPonCI8F7h0SN2Q9qrXaT/w1J+0UKMYQb0YeANwR5ItTdl/phNM1yU5D7gbeDVAVd2Z5DpgK50ZgG9tZvABnM8T08xvxBl8GrFe91CBU9KlA5Fpu21ozZo1NTMzM+pmaMrsHlF9/C0vHHFLpPZJcntVrZlb7lp8kqRWMqAkSa3kWnzSkHjPlLQwBpQ0BD4YUVo4A0oaAu+ZkhbOa1CSpFZyBCWNkNelpL0zoKQR8bqUtG/eqCu1yGv/8Fa2zu6atwitoypNsr3dqOsISmoRR1XSExxBSS3nqEqTzhGUNKYcVWlaOYKSxpCjKk0SR1DSBHFUpWlgQEljyJUpNA1cSUKS1EqOoKQJsrfHzc/ltSqNAwNKmhB7e9z8XF6r0rgwoKQJ0eu6VC9eq9K48BqUJKmVDChJUit5ik+aQv1OpujFCRYaFgNKmjL9TqboxQkWGiYDSpoy/U6m6MUJFhomA0rSgvgUYA2LASWpb71OD27e/iCbtz/Ixi33zqtraOlguJq5pINyzea754XT5u0PAvCCE46cV9/g0lx7W83cgJK06HqFFuw7uOYyyKaHj9uQNDR7m4ixt+Cay9mCAkdQklqo3wcy7i3wHH2NF0dQksZGv5Mxep0ydPQ1ORxBSRoL/Y6W+h19qT0cQUkaa/3eYLyQqfAHw8AbvLEfQSU5E3g/cAhwRVVdsq/6jqCk6dPv5Ix+ORtxcU3kNPMkhwB/B7wM2AF8CXhdVW3d2zEGlKSD1W/gLSTIDtY4B+GknuI7FdhWVd8ESHItsBbYa0BJ0sHq93TjYo/c9mYQpzD7tfrYJVz0ihMH8tnjHlDLgXu63u8AXjC3UpINwAaA448fz39hSBo/B7Mw70IMKwiHbdwDKj3K5p2zrKrLgcuhc4pv0I2SpGEaVhAO27g/UXcHcFzX+xXAfSNqiyRpEY17QH0JWJXkhCRPAdYBm0bcJknSIhjrU3xV9ViSXwZuojPN/MNVdeeImyVJWgRjHVAAVXUDcMOo2yFJWlzjfopPkjShDChJUisZUJKkVjKgJEmtZEBJklpprBeLPRBJdgL/cICHHwV8axGbMy7s93Sx39OlDf3+F1W1dG7h1AXUwUgy02vF3Ulnv6eL/Z4ube63p/gkSa1kQEmSWsmAWpjLR92AEbHf08V+T5fW9ttrUJKkVnIEJUlqJQNKktRKBlQPSc5M8o0k25Jc0GN/knyg2f+3SZ4/inYutj76/ewktyZ5JMmvjqKNg9BHv/9d8zv/bZIvJHnuKNq52Pro99qmz1uSzCT5V6No52LbX7+76v1UkseTnD3M9g1CH7/16Um+0/zWW5K8exTtnKeqfHW96DxX6v8AzwKeAvwNsHpOnZcDN9J55PxpwOZRt3tI/T4a+CngYuBXR93mIfb7RcARzfbPTdHv/QyeuE79HODro273MPrdVe/zdB7lc/ao2z2E3/p04NOjbuvclyOo+U4FtlXVN6vq+8C1wNo5ddYCV1fHF4EfSbJs2A1dZPvtd1U9UFVfAh4dRQMHpJ9+f6GqHmrefhFYMeQ2DkI//f5uNf/1Ag4HJmFGVT///wZ4G3A98MAwGzcg/fa5dQyo+ZYD93S939GULbTOuJnEPvVjof0+j87oedz11e8kr0rydeAzwJuH1LZB2m+/kywHXgX8wRDbNUj9/m/8hUn+JsmNSU4cTtP2zYCaLz3K5v7LsZ8642YS+9SPvvud5GfoBNS7Btqi4eir31X1p1X1bOAs4H2DbtQQ9NPv3wfeVVWPD745Q9FPn79MZz285wKXAp8adKP6YUDNtwM4ruv9CuC+A6gzbiaxT/3oq99JngNcAaytqm8PqW2DtKDfu6r+CvixJEcNumED1k+/1wDXJrkLOBv4UJKzhtK6wdhvn6tqV1V9t9m+AXhyG35rA2q+LwGrkpyQ5CnAOmDTnDqbgHOb2XynAd+pqtlhN3SR9dPvSbTffic5Hvgk8Iaq+rsRtHEQ+un3jydJs/18OhfYxz2c99vvqjqhqlZW1UrgE8B/qKpPDb2li6ef3/qZXb/1qXSyYeS/9aGjbkDbVNVjSX4ZuInO7JcPV9WdSX6p2f8HdGb2vBzYBvxf4E2jau9i6affSZ4JzABLgB8keSed2UC7RtXug9Xn7/1u4Efp/Esa4LFq6erP/eqz3/+Wzj/EHgX+CXht16SJsdRnvydKn30+Gzg/yWN0fut1bfitXepIktRKnuKTJLWSASVJaiUDSpLUSgaUJKmVDChJUisZUNKQJbkiyeoF1F+T5APN9huTfHCB39d9/OlJXrSwFkuj4X1Q0pBV1b9fYP0ZOvefLViSQ+ccfzrwXeALB/J50jA5gpIGKMnhST7TLML51SSvTXJLkjXN/u8m+c0ktyf5XJJTm/3fTPLKps7pST7d47NfkWRzkq80xx7TlL8nyeVJ/gK4evfxSVYCvwT8SvPMn5ck2Z7kyc1xS5Lctfu9NGoGlDRYZwL3VdVzq+ok4M/n7D8cuKWqTgEeBv4r8DI6q2m/dz+f/b+A06rqeXQeofBrXftOobNu4Ot3F1TVXXRW6P69qjq5qv4auAX4+abKOuD6qpqkx6lojBlQ0mDdAby0GSW9pKq+M2f/93kitO4A/rIJiDuAlfv57BXATUnuAP4T0P2IhE1V9U99tO8Knliq603AR/o4RhoKA0oaoGZx2VPoBM5v9HiU9qNda579AHikOe4H7P8a8aXAB6vqJ4G3AE/r2ve9Ptv3v4GVSf41cEhVfbWf46RhcJKENEBJjgUerKo/SfJd4I2L+PE/DNzbbK/v85iH6Sz22+1q4GNMxvOeNEEcQUmD9ZPAbUm2AP+FzjWmxfIe4H8k+WvgW30e82fAq3ZPkmjKPgocQSekpNZwNXNpyiU5m86EijeMui1SN0/xSVMsyaXAz9F5vpnUKo6gJEmt5DUoSVIrGVCSpFYyoCRJrWRASZJayYCSJLXS/weAP4xcesxAjQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "movie_sim.plot1d(movie_sim.similarity, limits='99.9%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To reduce data dimension, let's take the top 30 correlated movie for each one of the 500 selected movies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(171544, 6)\n",
      "(12480, 6)\n"
     ]
    }
   ],
   "source": [
    "movie_sim_pd = movie_sim.to_pandas_df()\n",
    "movie_sim_pd.sort_values(['id_left','similarity'], ascending=[True,False], inplace=True)\n",
    "movie_sim_pd = movie_sim_pd.groupby('id_left').head(30)\n",
    "\n",
    "print(movie_sim.shape)\n",
    "print(movie_sim_pd.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not os.path.exists('output'):\n",
    "    os.makedirs('output')\n",
    "\n",
    "movie_sim_pd.to_csv('output/movie_similarity.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's check to highest correlated movies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id_left</th>\n",
       "      <th>title_left</th>\n",
       "      <th>title_right</th>\n",
       "      <th>similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6500</th>\n",
       "      <td>4993</td>\n",
       "      <td>The Lord of the Rings: The Fellowship of the Ring</td>\n",
       "      <td>The Lord of the Rings: The Return of the King</td>\n",
       "      <td>0.816243</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4414</th>\n",
       "      <td>7153</td>\n",
       "      <td>The Lord of the Rings: The Return of the King</td>\n",
       "      <td>The Lord of the Rings: The Fellowship of the Ring</td>\n",
       "      <td>0.816243</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3227</th>\n",
       "      <td>260</td>\n",
       "      <td>Star Wars</td>\n",
       "      <td>The Empire Strikes Back</td>\n",
       "      <td>0.753279</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3393</th>\n",
       "      <td>1196</td>\n",
       "      <td>The Empire Strikes Back</td>\n",
       "      <td>Star Wars</td>\n",
       "      <td>0.753279</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5055</th>\n",
       "      <td>1196</td>\n",
       "      <td>The Empire Strikes Back</td>\n",
       "      <td>Return of the Jedi</td>\n",
       "      <td>0.722074</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3261</th>\n",
       "      <td>1210</td>\n",
       "      <td>Return of the Jedi</td>\n",
       "      <td>The Empire Strikes Back</td>\n",
       "      <td>0.722074</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3398</th>\n",
       "      <td>1210</td>\n",
       "      <td>Return of the Jedi</td>\n",
       "      <td>Star Wars</td>\n",
       "      <td>0.702372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5031</th>\n",
       "      <td>260</td>\n",
       "      <td>Star Wars</td>\n",
       "      <td>Return of the Jedi</td>\n",
       "      <td>0.702372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7354</th>\n",
       "      <td>5418</td>\n",
       "      <td>The Bourne Identity</td>\n",
       "      <td>The Bourne Supremacy</td>\n",
       "      <td>0.692932</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7190</th>\n",
       "      <td>8665</td>\n",
       "      <td>The Bourne Supremacy</td>\n",
       "      <td>The Bourne Identity</td>\n",
       "      <td>0.692932</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1060</th>\n",
       "      <td>6377</td>\n",
       "      <td>Finding Nemo</td>\n",
       "      <td>Monsters, Inc.</td>\n",
       "      <td>0.625467</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1509</th>\n",
       "      <td>4886</td>\n",
       "      <td>Monsters, Inc.</td>\n",
       "      <td>Finding Nemo</td>\n",
       "      <td>0.625467</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>334</th>\n",
       "      <td>2012</td>\n",
       "      <td>Back to the Future Part III</td>\n",
       "      <td>Back to the Future Part II</td>\n",
       "      <td>0.618704</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>778</th>\n",
       "      <td>2011</td>\n",
       "      <td>Back to the Future Part II</td>\n",
       "      <td>Back to the Future Part III</td>\n",
       "      <td>0.618704</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>214</th>\n",
       "      <td>1240</td>\n",
       "      <td>The Terminator</td>\n",
       "      <td>Terminator 2: Judgment Day</td>\n",
       "      <td>0.616285</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2422</th>\n",
       "      <td>589</td>\n",
       "      <td>Terminator 2: Judgment Day</td>\n",
       "      <td>The Terminator</td>\n",
       "      <td>0.616285</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1828</th>\n",
       "      <td>593</td>\n",
       "      <td>The Silence of the Lambs</td>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>0.613286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2078</th>\n",
       "      <td>296</td>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>The Silence of the Lambs</td>\n",
       "      <td>0.613286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3605</th>\n",
       "      <td>2571</td>\n",
       "      <td>The Matrix</td>\n",
       "      <td>Fight Club</td>\n",
       "      <td>0.607790</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3859</th>\n",
       "      <td>2959</td>\n",
       "      <td>Fight Club</td>\n",
       "      <td>The Matrix</td>\n",
       "      <td>0.607790</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4368</th>\n",
       "      <td>2571</td>\n",
       "      <td>The Matrix</td>\n",
       "      <td>The Lord of the Rings: The Fellowship of the Ring</td>\n",
       "      <td>0.606074</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3885</th>\n",
       "      <td>4993</td>\n",
       "      <td>The Lord of the Rings: The Fellowship of the Ring</td>\n",
       "      <td>The Matrix</td>\n",
       "      <td>0.606074</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2450</th>\n",
       "      <td>1214</td>\n",
       "      <td>Alien</td>\n",
       "      <td>The Terminator</td>\n",
       "      <td>0.591627</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4259</th>\n",
       "      <td>1240</td>\n",
       "      <td>The Terminator</td>\n",
       "      <td>Alien</td>\n",
       "      <td>0.591627</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6489</th>\n",
       "      <td>2571</td>\n",
       "      <td>The Matrix</td>\n",
       "      <td>The Lord of the Rings: The Return of the King</td>\n",
       "      <td>0.590578</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3897</th>\n",
       "      <td>7153</td>\n",
       "      <td>The Lord of the Rings: The Return of the King</td>\n",
       "      <td>The Matrix</td>\n",
       "      <td>0.590578</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1811</th>\n",
       "      <td>318</td>\n",
       "      <td>The Shawshank Redemption</td>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>0.590023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2243</th>\n",
       "      <td>296</td>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>The Shawshank Redemption</td>\n",
       "      <td>0.590023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1793</th>\n",
       "      <td>50</td>\n",
       "      <td>The Usual Suspects</td>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>0.586244</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1906</th>\n",
       "      <td>296</td>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>The Usual Suspects</td>\n",
       "      <td>0.586244</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id_left                                         title_left  \\\n",
       "6500     4993  The Lord of the Rings: The Fellowship of the Ring   \n",
       "4414     7153      The Lord of the Rings: The Return of the King   \n",
       "3227      260                                          Star Wars   \n",
       "3393     1196                            The Empire Strikes Back   \n",
       "5055     1196                            The Empire Strikes Back   \n",
       "3261     1210                                 Return of the Jedi   \n",
       "3398     1210                                 Return of the Jedi   \n",
       "5031      260                                          Star Wars   \n",
       "7354     5418                                The Bourne Identity   \n",
       "7190     8665                               The Bourne Supremacy   \n",
       "1060     6377                                       Finding Nemo   \n",
       "1509     4886                                     Monsters, Inc.   \n",
       "334      2012                        Back to the Future Part III   \n",
       "778      2011                         Back to the Future Part II   \n",
       "214      1240                                     The Terminator   \n",
       "2422      589                         Terminator 2: Judgment Day   \n",
       "1828      593                           The Silence of the Lambs   \n",
       "2078      296                                       Pulp Fiction   \n",
       "3605     2571                                         The Matrix   \n",
       "3859     2959                                         Fight Club   \n",
       "4368     2571                                         The Matrix   \n",
       "3885     4993  The Lord of the Rings: The Fellowship of the Ring   \n",
       "2450     1214                                              Alien   \n",
       "4259     1240                                     The Terminator   \n",
       "6489     2571                                         The Matrix   \n",
       "3897     7153      The Lord of the Rings: The Return of the King   \n",
       "1811      318                           The Shawshank Redemption   \n",
       "2243      296                                       Pulp Fiction   \n",
       "1793       50                                 The Usual Suspects   \n",
       "1906      296                                       Pulp Fiction   \n",
       "\n",
       "                                            title_right  similarity  \n",
       "6500      The Lord of the Rings: The Return of the King    0.816243  \n",
       "4414  The Lord of the Rings: The Fellowship of the Ring    0.816243  \n",
       "3227                            The Empire Strikes Back    0.753279  \n",
       "3393                                          Star Wars    0.753279  \n",
       "5055                                 Return of the Jedi    0.722074  \n",
       "3261                            The Empire Strikes Back    0.722074  \n",
       "3398                                          Star Wars    0.702372  \n",
       "5031                                 Return of the Jedi    0.702372  \n",
       "7354                               The Bourne Supremacy    0.692932  \n",
       "7190                                The Bourne Identity    0.692932  \n",
       "1060                                     Monsters, Inc.    0.625467  \n",
       "1509                                       Finding Nemo    0.625467  \n",
       "334                          Back to the Future Part II    0.618704  \n",
       "778                         Back to the Future Part III    0.618704  \n",
       "214                          Terminator 2: Judgment Day    0.616285  \n",
       "2422                                     The Terminator    0.616285  \n",
       "1828                                       Pulp Fiction    0.613286  \n",
       "2078                           The Silence of the Lambs    0.613286  \n",
       "3605                                         Fight Club    0.607790  \n",
       "3859                                         The Matrix    0.607790  \n",
       "4368  The Lord of the Rings: The Fellowship of the Ring    0.606074  \n",
       "3885                                         The Matrix    0.606074  \n",
       "2450                                     The Terminator    0.591627  \n",
       "4259                                              Alien    0.591627  \n",
       "6489      The Lord of the Rings: The Return of the King    0.590578  \n",
       "3897                                         The Matrix    0.590578  \n",
       "1811                                       Pulp Fiction    0.590023  \n",
       "2243                           The Shawshank Redemption    0.590023  \n",
       "1793                                       Pulp Fiction    0.586244  \n",
       "1906                                 The Usual Suspects    0.586244  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "movies_info = pd.read_csv('data/movie_info.csv', usecols=['id','title'])\n",
    "\n",
    "high_sim = pd.merge(movie_sim_pd, movies_info, left_on='id_left', right_on='id')\n",
    "high_sim.drop(columns=['id'], inplace=True)\n",
    "high_sim.rename(columns={'title':'title_left'}, inplace=True)\n",
    "\n",
    "high_sim = pd.merge(high_sim, movies_info, left_on='id_right', right_on='id')\n",
    "high_sim.drop(columns=['id'], inplace=True)\n",
    "high_sim.rename(columns={'title':'title_right'}, inplace=True)\n",
    "\n",
    "display(high_sim[['id_left','title_left','title_right','similarity']].sort_values('similarity', ascending=False)\n",
    "                                                                     .head(30))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's test some movies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title_left</th>\n",
       "      <th>title_right</th>\n",
       "      <th>similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6500</th>\n",
       "      <td>The Lord of the Rings: The Fellowship of the Ring</td>\n",
       "      <td>The Lord of the Rings: The Return of the King</td>\n",
       "      <td>0.816243</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3885</th>\n",
       "      <td>The Lord of the Rings: The Fellowship of the Ring</td>\n",
       "      <td>The Matrix</td>\n",
       "      <td>0.606074</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3628</th>\n",
       "      <td>The Lord of the Rings: The Fellowship of the Ring</td>\n",
       "      <td>Fight Club</td>\n",
       "      <td>0.541024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>721</th>\n",
       "      <td>The Lord of the Rings: The Fellowship of the Ring</td>\n",
       "      <td>Shrek</td>\n",
       "      <td>0.530268</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3337</th>\n",
       "      <td>The Lord of the Rings: The Fellowship of the Ring</td>\n",
       "      <td>The Empire Strikes Back</td>\n",
       "      <td>0.525910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3441</th>\n",
       "      <td>The Lord of the Rings: The Fellowship of the Ring</td>\n",
       "      <td>Star Wars</td>\n",
       "      <td>0.502728</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7708</th>\n",
       "      <td>The Lord of the Rings: The Fellowship of the Ring</td>\n",
       "      <td>The Dark Knight</td>\n",
       "      <td>0.502115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1511</th>\n",
       "      <td>The Lord of the Rings: The Fellowship of the Ring</td>\n",
       "      <td>Finding Nemo</td>\n",
       "      <td>0.491545</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1050</th>\n",
       "      <td>The Lord of the Rings: The Fellowship of the Ring</td>\n",
       "      <td>Monsters, Inc.</td>\n",
       "      <td>0.485182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7162</th>\n",
       "      <td>The Lord of the Rings: The Fellowship of the Ring</td>\n",
       "      <td>The Bourne Identity</td>\n",
       "      <td>0.471722</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             title_left  \\\n",
       "6500  The Lord of the Rings: The Fellowship of the Ring   \n",
       "3885  The Lord of the Rings: The Fellowship of the Ring   \n",
       "3628  The Lord of the Rings: The Fellowship of the Ring   \n",
       "721   The Lord of the Rings: The Fellowship of the Ring   \n",
       "3337  The Lord of the Rings: The Fellowship of the Ring   \n",
       "3441  The Lord of the Rings: The Fellowship of the Ring   \n",
       "7708  The Lord of the Rings: The Fellowship of the Ring   \n",
       "1511  The Lord of the Rings: The Fellowship of the Ring   \n",
       "1050  The Lord of the Rings: The Fellowship of the Ring   \n",
       "7162  The Lord of the Rings: The Fellowship of the Ring   \n",
       "\n",
       "                                        title_right  similarity  \n",
       "6500  The Lord of the Rings: The Return of the King    0.816243  \n",
       "3885                                     The Matrix    0.606074  \n",
       "3628                                     Fight Club    0.541024  \n",
       "721                                           Shrek    0.530268  \n",
       "3337                        The Empire Strikes Back    0.525910  \n",
       "3441                                      Star Wars    0.502728  \n",
       "7708                                The Dark Knight    0.502115  \n",
       "1511                                   Finding Nemo    0.491545  \n",
       "1050                                 Monsters, Inc.    0.485182  \n",
       "7162                            The Bourne Identity    0.471722  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(high_sim.loc[high_sim['id_left']==4993, ['title_left','title_right','similarity']]\n",
    "                .sort_values('similarity', ascending=False)\n",
    "                .head(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title_left</th>\n",
       "      <th>title_right</th>\n",
       "      <th>similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3227</th>\n",
       "      <td>Star Wars</td>\n",
       "      <td>The Empire Strikes Back</td>\n",
       "      <td>0.753279</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5031</th>\n",
       "      <td>Star Wars</td>\n",
       "      <td>Return of the Jedi</td>\n",
       "      <td>0.702372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3798</th>\n",
       "      <td>Star Wars</td>\n",
       "      <td>The Matrix</td>\n",
       "      <td>0.558130</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1340</th>\n",
       "      <td>Star Wars</td>\n",
       "      <td>Indiana Jones and the Last Crusade</td>\n",
       "      <td>0.530317</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>376</th>\n",
       "      <td>Star Wars</td>\n",
       "      <td>Back to the Future</td>\n",
       "      <td>0.507713</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4341</th>\n",
       "      <td>Star Wars</td>\n",
       "      <td>The Lord of the Rings: The Fellowship of the Ring</td>\n",
       "      <td>0.502728</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2412</th>\n",
       "      <td>Star Wars</td>\n",
       "      <td>The Terminator</td>\n",
       "      <td>0.496144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4222</th>\n",
       "      <td>Star Wars</td>\n",
       "      <td>Alien</td>\n",
       "      <td>0.487180</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6474</th>\n",
       "      <td>Star Wars</td>\n",
       "      <td>The Lord of the Rings: The Return of the King</td>\n",
       "      <td>0.483217</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2742</th>\n",
       "      <td>Star Wars</td>\n",
       "      <td>Monty Python and the Holy Grail</td>\n",
       "      <td>0.471676</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     title_left                                        title_right  similarity\n",
       "3227  Star Wars                            The Empire Strikes Back    0.753279\n",
       "5031  Star Wars                                 Return of the Jedi    0.702372\n",
       "3798  Star Wars                                         The Matrix    0.558130\n",
       "1340  Star Wars                 Indiana Jones and the Last Crusade    0.530317\n",
       "376   Star Wars                                 Back to the Future    0.507713\n",
       "4341  Star Wars  The Lord of the Rings: The Fellowship of the Ring    0.502728\n",
       "2412  Star Wars                                     The Terminator    0.496144\n",
       "4222  Star Wars                                              Alien    0.487180\n",
       "6474  Star Wars      The Lord of the Rings: The Return of the King    0.483217\n",
       "2742  Star Wars                    Monty Python and the Holy Grail    0.471676"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(high_sim.loc[high_sim['id_left']==260, ['title_left','title_right','similarity']]\n",
    "                .sort_values('similarity', ascending=False)\n",
    "                .head(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title_left</th>\n",
       "      <th>title_right</th>\n",
       "      <th>similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2078</th>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>The Silence of the Lambs</td>\n",
       "      <td>0.613286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2243</th>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>The Shawshank Redemption</td>\n",
       "      <td>0.590023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1906</th>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>The Usual Suspects</td>\n",
       "      <td>0.586244</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3573</th>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>Fight Club</td>\n",
       "      <td>0.557995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3458</th>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>Reservoir Dogs</td>\n",
       "      <td>0.557944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1981</th>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>Fargo</td>\n",
       "      <td>0.518520</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3688</th>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>American Beauty</td>\n",
       "      <td>0.517062</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3799</th>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>The Matrix</td>\n",
       "      <td>0.509230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2301</th>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>Schindler's List</td>\n",
       "      <td>0.493973</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>180</th>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>Terminator 2: Judgment Day</td>\n",
       "      <td>0.482289</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        title_left                 title_right  similarity\n",
       "2078  Pulp Fiction    The Silence of the Lambs    0.613286\n",
       "2243  Pulp Fiction    The Shawshank Redemption    0.590023\n",
       "1906  Pulp Fiction          The Usual Suspects    0.586244\n",
       "3573  Pulp Fiction                  Fight Club    0.557995\n",
       "3458  Pulp Fiction              Reservoir Dogs    0.557944\n",
       "1981  Pulp Fiction                       Fargo    0.518520\n",
       "3688  Pulp Fiction             American Beauty    0.517062\n",
       "3799  Pulp Fiction                  The Matrix    0.509230\n",
       "2301  Pulp Fiction            Schindler's List    0.493973\n",
       "180   Pulp Fiction  Terminator 2: Judgment Day    0.482289"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(high_sim.loc[high_sim['id_left']==296, ['title_left','title_right','similarity']]\n",
    "                .sort_values('similarity', ascending=False)\n",
    "                .head(10))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}