ods a3 copied notebook from yorko.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "<img src=\"../../img/ods_stickers.jpg\" />\n",
    "    \n",
    "## [mlcourse.ai](https://mlcourse.ai) – Open Machine Learning Course \n",
    "Authors: [Yury Kashnitsky](https://yorko.github.io) (@yorko), Yury Isakov. Edited by Anna Tarelina (@feuerengel), Aditya Soni (@ecdrid) and Kolchenko Sergey (@KolchenkoSergey). This material is subject to the terms and conditions of the [Creative Commons CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) license. Free use is permitted for any non-commercial purpose."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <center> Assignment #3. Fall 2019\n",
    "## <center> Part 1. User Identification with Logistic Regression <br>(beating baselines in the \"Alice\" competition)\n",
    "\n",
    "    \n",
    "Today we are going to practice working with sparse matrices, training Logistic Regression models, and doing feature engineering. We will reproduce a couple of baselines in the  Kaggle Inclass competition [\"Catch Me If You Can: Intruder Detection through Webpage Session Tracking\"](https://www.kaggle.com/c/catch-me-if-you-can-intruder-detection-through-webpage-session-tracking2) (a.k.a. \"Alice\"). More credits will be given for beating stronger baselines. \n",
    "\n",
    "Prior to working on the assignment, you'd better check out the corresponding course material:\n",
    " 1. [Classification, Decision Trees and k Nearest Neighbors](https://nbviewer.jupyter.org/github/Yorko/mlcourse_open/blob/master/jupyter_english/topic03_decision_trees_kNN/topic3_decision_trees_kNN.ipynb?flush_cache=true), the same as an interactive web-based [Kaggle Kernel](https://www.kaggle.com/kashnitsky/topic-3-decision-trees-and-knn) (basics of machine learning are covered here)\n",
    " 2. Linear classification and regression in 5 parts: \n",
    "    - [ordinary least squares](https://www.kaggle.com/kashnitsky/topic-4-linear-models-part-1-ols)\n",
    "    - [linear classification](https://www.kaggle.com/kashnitsky/topic-4-linear-models-part-2-classification)\n",
    "    - [regularization](https://www.kaggle.com/kashnitsky/topic-4-linear-models-part-3-regularization)\n",
    "    - [logistic regression: pros and cons](https://www.kaggle.com/kashnitsky/topic-4-linear-models-part-4-more-of-logit)\n",
    "    - [validation](https://www.kaggle.com/kashnitsky/topic-4-linear-models-part-5-validation)\n",
    " 3. You can also practice with demo assignments, which are simpler and already shared with solutions: \n",
    "    - \"Sarcasm detection with logistic regression\": [assignment](https://www.kaggle.com/kashnitsky/a4-demo-sarcasm-detection-with-logit) + [solution](https://www.kaggle.com/kashnitsky/a4-demo-sarcasm-detection-with-logit-solution)\n",
    "    - \"Linear regression as optimization\": [assignment](https://www.kaggle.com/kashnitsky/a4-demo-linear-regression-as-optimization) (solution cannot be officially shared)\n",
    "    - \"Exploring OLS, Lasso and Random Forest in a regression task\": [assignment](https://www.kaggle.com/kashnitsky/a6-demo-linear-models-and-rf-for-regression) + [solution](https://www.kaggle.com/kashnitsky/a6-demo-regression-solution)\n",
    " 4. Alice baseline with logistic regression and \"bag of sites\", [Kernel](https://www.kaggle.com/kashnitsky/alice-logistic-regression-baseline)\n",
    " 5. Correct time-aware cross-validation scheme, more features, and hyperparameter optimization, [Kernel](https://www.kaggle.com/kashnitsky/correct-time-aware-cross-validation-scheme)\n",
    " 6. **\"Model validation in a competition\"**, [Kernel](https://www.kaggle.com/kashnitsky/model-validation-in-a-competition) - this one reproduces a solution with **0.95055** Public LB ROC AUC and gives a lot of hints how to proceed with this competition\n",
    " 7. Other [Kernels](https://www.kaggle.com/c/catch-me-if-you-can-intruder-detection-through-webpage-session-tracking2/kernels?sortBy=voteCount&group=everyone&pageSize=20&competitionId=7173) in this competition. You can share yours as well, but not high-performing ones (Public LB ROC AUC shall be < 0.95055). Please don't spoil the competitive spirit. \n",
    " 8. If that's still not enough, watch two videos on logistic regression: [mlcourse.ai/lectures](https://mlcourse.ai/lectures)\n",
    "\n",
    "**Your task:**\n",
    " 1. \"Follow me\". Complete the missing code and submit your answers via [the google form](https://docs.google.com/forms/d/1a2PrdKsc7gV2fO7bSwO-uGVEWAY6CIxx-9YV5RNnDIs). Use **the same email and name** as in A1 and A2 (for newcomers: remember your email and use it for all forms during the course). 10 credits max. for this part\n",
    " 2. \"Freeride\". Come up with good features to beat the baselines \"A3 baseline (10 credits)\" (**0.95343** Public LB ROC-AUC, press \"Load more\" in the bottom of the [Leaderboard](https://www.kaggle.com/c/catch-me-if-you-can-intruder-detection-through-webpage-session-tracking2/leaderboard) to actually see it) and \"A3 strong baseline (20 credits)\" (**0.95965** Public LB ROC-AUC). As names suggest, you'll get 10 more credits for beating the first one, and 10 more (20 in total) for beating the second one. You need to name your [team](https://www.kaggle.com/c/catch-me-if-you-can-intruder-detection-through-webpage-session-tracking2/team) (out of 1 person) in full accordance with the [course rating](https://docs.google.com/spreadsheets/d/15e1K0tg5ponA5R6YQkZfihrShTDLAKf5qeKaoVCiuhQ/) (for newcomers: you need to name your team with your real full name). You can think of it as a part of the assignment.\n",
    " 3. If you've beaten \"A3 baseline (10 credits)\" or performed better, you need to upload your solution as described in [course roadmap](https://mlcourse.ai/roadmap) (\"Kaggle Inclass Competition Alice\" -> Rules). For all baselines that you see on Public Leaderboard, it's OK to beat them on Public LB as well. But 10 winners will be defined according to the private LB, which will be revealed by @yorko on October 27. \n",
    " \n",
    "### <center> Deadline for A3: 2019 October 27, 20:59 GMT+1 (London time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Subpart 1. Follow me"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='../../img/followme_alice.png' width=50%>\n",
    "\n",
    "*image credit [@muradosmann](https://www.instagram.com/muradosmann/?hl=en)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import libraries and set desired options\n",
    "import pickle\n",
    "from pathlib2 import Path\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from scipy.sparse import csr_matrix, hstack\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from matplotlib import pyplot as plt\n",
    "import seaborn as sns\n",
    "sns.set()\n",
    "%config InlineBackend.figure_format = 'retina'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Problem description\n",
    "\n",
    "In this competition, we'll analyze the sequence of websites consequently visited by a particular person and try to predict whether this person is Alice or someone else. As a metric we will use [ROC AUC](https://en.wikipedia.org/wiki/Receiver_operating_characteristic)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Data Downloading and Transformation\n",
    "Go to the competition [page](https://inclass.kaggle.com/c/catch-me-if-you-can-intruder-detection-through-webpage-session-tracking2) and download the data.\n",
    "\n",
    "First, read the training and test sets. Then we'll explore the data in hand and do a couple of simple exercises. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>site1</th>\n",
       "      <th>time1</th>\n",
       "      <th>site2</th>\n",
       "      <th>time2</th>\n",
       "      <th>site3</th>\n",
       "      <th>time3</th>\n",
       "      <th>site4</th>\n",
       "      <th>time4</th>\n",
       "      <th>site5</th>\n",
       "      <th>time5</th>\n",
       "      <th>...</th>\n",
       "      <th>time6</th>\n",
       "      <th>site7</th>\n",
       "      <th>time7</th>\n",
       "      <th>site8</th>\n",
       "      <th>time8</th>\n",
       "      <th>site9</th>\n",
       "      <th>time9</th>\n",
       "      <th>site10</th>\n",
       "      <th>time10</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>session_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>21669</th>\n",
       "      <td>56</td>\n",
       "      <td>2013-01-12 08:05:57</td>\n",
       "      <td>55.0</td>\n",
       "      <td>2013-01-12 08:05:57</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>...</td>\n",
       "      <td>NaT</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54843</th>\n",
       "      <td>56</td>\n",
       "      <td>2013-01-12 08:37:23</td>\n",
       "      <td>55.0</td>\n",
       "      <td>2013-01-12 08:37:23</td>\n",
       "      <td>56.0</td>\n",
       "      <td>2013-01-12 09:07:07</td>\n",
       "      <td>55.0</td>\n",
       "      <td>2013-01-12 09:07:09</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>...</td>\n",
       "      <td>NaT</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77292</th>\n",
       "      <td>946</td>\n",
       "      <td>2013-01-12 08:50:13</td>\n",
       "      <td>946.0</td>\n",
       "      <td>2013-01-12 08:50:14</td>\n",
       "      <td>951.0</td>\n",
       "      <td>2013-01-12 08:50:15</td>\n",
       "      <td>946.0</td>\n",
       "      <td>2013-01-12 08:50:15</td>\n",
       "      <td>946.0</td>\n",
       "      <td>2013-01-12 08:50:16</td>\n",
       "      <td>...</td>\n",
       "      <td>2013-01-12 08:50:16</td>\n",
       "      <td>948.0</td>\n",
       "      <td>2013-01-12 08:50:16</td>\n",
       "      <td>784.0</td>\n",
       "      <td>2013-01-12 08:50:16</td>\n",
       "      <td>949.0</td>\n",
       "      <td>2013-01-12 08:50:17</td>\n",
       "      <td>946.0</td>\n",
       "      <td>2013-01-12 08:50:17</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>114021</th>\n",
       "      <td>945</td>\n",
       "      <td>2013-01-12 08:50:17</td>\n",
       "      <td>948.0</td>\n",
       "      <td>2013-01-12 08:50:17</td>\n",
       "      <td>949.0</td>\n",
       "      <td>2013-01-12 08:50:18</td>\n",
       "      <td>948.0</td>\n",
       "      <td>2013-01-12 08:50:18</td>\n",
       "      <td>945.0</td>\n",
       "      <td>2013-01-12 08:50:18</td>\n",
       "      <td>...</td>\n",
       "      <td>2013-01-12 08:50:18</td>\n",
       "      <td>947.0</td>\n",
       "      <td>2013-01-12 08:50:19</td>\n",
       "      <td>945.0</td>\n",
       "      <td>2013-01-12 08:50:19</td>\n",
       "      <td>946.0</td>\n",
       "      <td>2013-01-12 08:50:19</td>\n",
       "      <td>946.0</td>\n",
       "      <td>2013-01-12 08:50:20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146670</th>\n",
       "      <td>947</td>\n",
       "      <td>2013-01-12 08:50:20</td>\n",
       "      <td>950.0</td>\n",
       "      <td>2013-01-12 08:50:20</td>\n",
       "      <td>948.0</td>\n",
       "      <td>2013-01-12 08:50:20</td>\n",
       "      <td>947.0</td>\n",
       "      <td>2013-01-12 08:50:21</td>\n",
       "      <td>950.0</td>\n",
       "      <td>2013-01-12 08:50:21</td>\n",
       "      <td>...</td>\n",
       "      <td>2013-01-12 08:50:21</td>\n",
       "      <td>946.0</td>\n",
       "      <td>2013-01-12 08:50:21</td>\n",
       "      <td>951.0</td>\n",
       "      <td>2013-01-12 08:50:22</td>\n",
       "      <td>946.0</td>\n",
       "      <td>2013-01-12 08:50:22</td>\n",
       "      <td>947.0</td>\n",
       "      <td>2013-01-12 08:50:22</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            site1               time1  site2               time2  site3  \\\n",
       "session_id                                                                \n",
       "21669          56 2013-01-12 08:05:57   55.0 2013-01-12 08:05:57    NaN   \n",
       "54843          56 2013-01-12 08:37:23   55.0 2013-01-12 08:37:23   56.0   \n",
       "77292         946 2013-01-12 08:50:13  946.0 2013-01-12 08:50:14  951.0   \n",
       "114021        945 2013-01-12 08:50:17  948.0 2013-01-12 08:50:17  949.0   \n",
       "146670        947 2013-01-12 08:50:20  950.0 2013-01-12 08:50:20  948.0   \n",
       "\n",
       "                         time3  site4               time4  site5  \\\n",
       "session_id                                                         \n",
       "21669                      NaT    NaN                 NaT    NaN   \n",
       "54843      2013-01-12 09:07:07   55.0 2013-01-12 09:07:09    NaN   \n",
       "77292      2013-01-12 08:50:15  946.0 2013-01-12 08:50:15  946.0   \n",
       "114021     2013-01-12 08:50:18  948.0 2013-01-12 08:50:18  945.0   \n",
       "146670     2013-01-12 08:50:20  947.0 2013-01-12 08:50:21  950.0   \n",
       "\n",
       "                         time5  ...               time6  site7  \\\n",
       "session_id                      ...                              \n",
       "21669                      NaT  ...                 NaT    NaN   \n",
       "54843                      NaT  ...                 NaT    NaN   \n",
       "77292      2013-01-12 08:50:16  ... 2013-01-12 08:50:16  948.0   \n",
       "114021     2013-01-12 08:50:18  ... 2013-01-12 08:50:18  947.0   \n",
       "146670     2013-01-12 08:50:21  ... 2013-01-12 08:50:21  946.0   \n",
       "\n",
       "                         time7  site8               time8  site9  \\\n",
       "session_id                                                         \n",
       "21669                      NaT    NaN                 NaT    NaN   \n",
       "54843                      NaT    NaN                 NaT    NaN   \n",
       "77292      2013-01-12 08:50:16  784.0 2013-01-12 08:50:16  949.0   \n",
       "114021     2013-01-12 08:50:19  945.0 2013-01-12 08:50:19  946.0   \n",
       "146670     2013-01-12 08:50:21  951.0 2013-01-12 08:50:22  946.0   \n",
       "\n",
       "                         time9 site10              time10 target  \n",
       "session_id                                                        \n",
       "21669                      NaT    NaN                 NaT      0  \n",
       "54843                      NaT    NaN                 NaT      0  \n",
       "77292      2013-01-12 08:50:17  946.0 2013-01-12 08:50:17      0  \n",
       "114021     2013-01-12 08:50:19  946.0 2013-01-12 08:50:20      0  \n",
       "146670     2013-01-12 08:50:22  947.0 2013-01-12 08:50:22      0  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read the training and test data sets, change paths if needed\n",
    "PATH_TO_DATA = Path('../../data/alice/')\n",
    "\n",
    "times = ['time%s' % i for i in range(1, 11)]\n",
    "train_df = pd.read_csv(PATH_TO_DATA / 'train_sessions.csv',\n",
    "                       index_col='session_id', parse_dates=times)\n",
    "test_df = pd.read_csv(PATH_TO_DATA / 'test_sessions.csv',\n",
    "                      index_col='session_id', parse_dates=times)\n",
    "\n",
    "# Sort the data by time\n",
    "train_df = train_df.sort_values(by='time1')\n",
    "\n",
    "# Look at the first rows of the training set\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training data set contains the following features:\n",
    "\n",
    "- **site1** – id of the first visited website in the session\n",
    "- **time1** – visiting time for the first website in the session\n",
    "- ...\n",
    "- **site10** – id of the tenth visited website in the session\n",
    "- **time10** – visiting time for the tenth website in the session\n",
    "- **target** – target variable, 1 for Alice's sessions, and 0 for the other users' sessions\n",
    "    \n",
    "User sessions are chosen in the way that they are shorter than 30 min. long and contain no more than 10 websites. I.e. a session is considered over either if a user has visited 10 websites or if a session has lasted over 30 minutes.\n",
    "\n",
    "There are some empty values in the table, it means that some sessions contain less than ten websites. Replace empty values with 0 and change columns types to integer. Also load the websites dictionary and check how it looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Websites total: 48371\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>site</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>25075</th>\n",
       "      <td>www.abmecatronique.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13997</th>\n",
       "      <td>groups.live.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42436</th>\n",
       "      <td>majeureliguefootball.wordpress.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30911</th>\n",
       "      <td>cdt46.media.tourinsoft.eu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8104</th>\n",
       "      <td>www.hdwallpapers.eu</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     site\n",
       "25075              www.abmecatronique.com\n",
       "13997                     groups.live.com\n",
       "42436  majeureliguefootball.wordpress.com\n",
       "30911           cdt46.media.tourinsoft.eu\n",
       "8104                  www.hdwallpapers.eu"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Change site1, ..., site10 columns type to integer and fill NA-values with zeros\n",
    "sites = ['site%s' % i for i in range(1, 11)]\n",
    "train_df[sites] = train_df[sites].fillna(0).astype(np.uint16)\n",
    "test_df[sites] = test_df[sites].fillna(0).astype(np.uint16)\n",
    "\n",
    "# Load websites dictionary\n",
    "with open(PATH_TO_DATA / 'site_dic.pkl', \"rb\") as input_file:\n",
    "    site_dict = pickle.load(input_file)\n",
    "\n",
    "# Create dataframe for the dictionary\n",
    "sites_dict = pd.DataFrame(list(site_dict.keys()), index=list(site_dict.values()), \n",
    "                          columns=['site'])\n",
    "print(u'Websites total:', sites_dict.shape[0])\n",
    "sites_dict.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Brief Exploratory Data Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we start training models, we have to perform Exploratory Data Analysis ([EDA](https://en.wikipedia.org/wiki/Exploratory_data_analysis)). Today, we are going to perform a shorter version, but we will use other techniques as we move forward. Let's check which websites in the training data set are the most visited. As you can see, they are Google services and a bioinformatics website (a website with 'zero'-index is our missed values, just ignore it):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "21     123776\n",
      "0      122730\n",
      "23      87619\n",
      "782     77055\n",
      "22      58258\n",
      "dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>site</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>www.google.fr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>www.google.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>782</th>\n",
       "      <td>annotathon.org</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>apis.google.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                site\n",
       "21     www.google.fr\n",
       "23    www.google.com\n",
       "782   annotathon.org\n",
       "22   apis.google.com"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Top websites in the training data set\n",
    "top_sites = pd.Series(train_df[sites].values.flatten()\n",
    "                     ).value_counts().sort_values(ascending=False).head(5)\n",
    "print(top_sites)\n",
    "sites_dict.loc[top_sites.drop(0).index]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1. What kind of websites does Alice visit the most?\n",
    "\n",
    "*For discussions, please stick to [ODS Slack](https://opendatascience.slack.com/), channel #mlcourse_ai_news, pinned thread __#a3_part1_fall2019__*\n",
    "\n",
    "- videohostings\n",
    "- social networks\n",
    "- torrent trackers\n",
    "- news"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us look at the timestamps and try to characterize sessions as timeframes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>target</th>\n",
       "      <th>min</th>\n",
       "      <th>max</th>\n",
       "      <th>seconds</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>session_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>21669</th>\n",
       "      <td>0</td>\n",
       "      <td>2013-01-12 08:05:57</td>\n",
       "      <td>2013-01-12 08:05:57</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54843</th>\n",
       "      <td>0</td>\n",
       "      <td>2013-01-12 08:37:23</td>\n",
       "      <td>2013-01-12 09:07:09</td>\n",
       "      <td>1786.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77292</th>\n",
       "      <td>0</td>\n",
       "      <td>2013-01-12 08:50:13</td>\n",
       "      <td>2013-01-12 08:50:17</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>114021</th>\n",
       "      <td>0</td>\n",
       "      <td>2013-01-12 08:50:17</td>\n",
       "      <td>2013-01-12 08:50:20</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146670</th>\n",
       "      <td>0</td>\n",
       "      <td>2013-01-12 08:50:20</td>\n",
       "      <td>2013-01-12 08:50:22</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            target                 min                 max  seconds\n",
       "session_id                                                         \n",
       "21669            0 2013-01-12 08:05:57 2013-01-12 08:05:57      0.0\n",
       "54843            0 2013-01-12 08:37:23 2013-01-12 09:07:09   1786.0\n",
       "77292            0 2013-01-12 08:50:13 2013-01-12 08:50:17      4.0\n",
       "114021           0 2013-01-12 08:50:17 2013-01-12 08:50:20      3.0\n",
       "146670           0 2013-01-12 08:50:20 2013-01-12 08:50:22      2.0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a separate dataframe where we will work with timestamps\n",
    "time_df = pd.DataFrame(index=train_df.index)\n",
    "time_df['target'] = train_df['target']\n",
    "\n",
    "# Find sessions' starting and ending\n",
    "time_df['min'] = train_df[times].min(axis=1)\n",
    "time_df['max'] = train_df[times].max(axis=1)\n",
    "\n",
    "# Calculate sessions' duration in seconds\n",
    "time_df['seconds'] = (time_df['max'] - time_df['min']) / np.timedelta64(1, 's')\n",
    "\n",
    "time_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to perform the next task, generate descriptive statistics as you did in the first assignment.\n",
    "\n",
    "*In the next question, we are using the notion of \"approximately the same\". To be strict, let's define it: $a$ is approximately the same as $b$ ($a \\approx b $) if their difference is less than or equal to 5% of the maximum between $a$ and $b$, i.e. $a \\approx b \\leftrightarrow \\frac{|a-b|}{max(a,b)} \\leq 0.05$.*\n",
    "\n",
    "##### 2. Select all correct statements:\n",
    "\n",
    "*For discussions, please stick to [ODS Slack](https://opendatascience.slack.com/), channel #mlcourse_ai_news, pinned thread __#a3_part1_fall2019__*\n",
    "\n",
    "- on average, Alice's session is shorter than that of other users\n",
    "- more than 1% of all sessions in the dataset belong to Alice\n",
    "- minimum and maximum durations of Alice's and other users' sessions are approximately the same\n",
    "- standard deviation of Alice's sessions duration is approximately the same as for non-Alice's sessions\n",
    "- less than a quarter of Alice's sessions are greater than or equal to 40 seconds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to train our first model, we need to prepare the data. First of all, exclude the target variable from the training set. Now both training and test sets have the same number of columns, therefore aggregate them into one dataframe.  Thus, all transformations will be performed simultaneously on both training and test data sets. \n",
    "\n",
    "On the one hand, it leads to the fact that both data sets have one feature space (you don't have to worry that you forgot to transform a feature in some data sets). On the other hand, processing time will increase. \n",
    "For the enormously large sets it might turn out that it is impossible to transform both data sets simultaneously (and sometimes you have to split your transformations into several stages only for train/test data set).\n",
    "In our case, with this particular data set, we are going to perform all the transformations for the whole united dataframe at once, and before training the model or making predictions we will just take its appropriate part."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Our target variable\n",
    "y_train = train_df['target']\n",
    "\n",
    "# United dataframe of the initial data \n",
    "full_df = pd.concat([train_df.drop('target', axis=1), test_df])\n",
    "\n",
    "# Index to split the training and test data sets\n",
    "idx_split = train_df.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the very basic model, we will use only the visited websites in the session (but we will not take into account timestamp features). The point behind this data selection is: *Alice has her favorite sites, and the more often you see these sites in the session, the higher probability that this is Alice's session, and vice versa.*\n",
    "\n",
    "Let us prepare the data, we will take only features `site1, site2, ... , site10` from the whole dataframe. Keep in mind that the missing values are replaced with zero. Here is how the first rows of the dataframe look like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>site1</th>\n",
       "      <th>site2</th>\n",
       "      <th>site3</th>\n",
       "      <th>site4</th>\n",
       "      <th>site5</th>\n",
       "      <th>site6</th>\n",
       "      <th>site7</th>\n",
       "      <th>site8</th>\n",
       "      <th>site9</th>\n",
       "      <th>site10</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>session_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>21669</th>\n",
       "      <td>56</td>\n",
       "      <td>55</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54843</th>\n",
       "      <td>56</td>\n",
       "      <td>55</td>\n",
       "      <td>56</td>\n",
       "      <td>55</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77292</th>\n",
       "      <td>946</td>\n",
       "      <td>946</td>\n",
       "      <td>951</td>\n",
       "      <td>946</td>\n",
       "      <td>946</td>\n",
       "      <td>945</td>\n",
       "      <td>948</td>\n",
       "      <td>784</td>\n",
       "      <td>949</td>\n",
       "      <td>946</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>114021</th>\n",
       "      <td>945</td>\n",
       "      <td>948</td>\n",
       "      <td>949</td>\n",
       "      <td>948</td>\n",
       "      <td>945</td>\n",
       "      <td>946</td>\n",
       "      <td>947</td>\n",
       "      <td>945</td>\n",
       "      <td>946</td>\n",
       "      <td>946</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146670</th>\n",
       "      <td>947</td>\n",
       "      <td>950</td>\n",
       "      <td>948</td>\n",
       "      <td>947</td>\n",
       "      <td>950</td>\n",
       "      <td>952</td>\n",
       "      <td>946</td>\n",
       "      <td>951</td>\n",
       "      <td>946</td>\n",
       "      <td>947</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            site1  site2  site3  site4  site5  site6  site7  site8  site9  \\\n",
       "session_id                                                                  \n",
       "21669          56     55      0      0      0      0      0      0      0   \n",
       "54843          56     55     56     55      0      0      0      0      0   \n",
       "77292         946    946    951    946    946    945    948    784    949   \n",
       "114021        945    948    949    948    945    946    947    945    946   \n",
       "146670        947    950    948    947    950    952    946    951    946   \n",
       "\n",
       "            site10  \n",
       "session_id          \n",
       "21669            0  \n",
       "54843            0  \n",
       "77292          946  \n",
       "114021         946  \n",
       "146670         947  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Dataframe with indices of visited websites in session\n",
    "full_sites = full_df[sites]\n",
    "full_sites.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sessions are sequences of website indices, and data in this representation is useless for machine learning method (just think, what happens if we switched all ids of all websites). \n",
    "\n",
    "According to our hypothesis (Alice has favorite websites), we need to transform this dataframe so each website has a corresponding feature (column) and its value is equal to number of this website visits in the session. It can be done in two lines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sequence of indices\n",
    "sites_flatten = full_sites.values.flatten()\n",
    "\n",
    "# and the matrix we are looking for \n",
    "# (make sure you understand which of the `csr_matrix` constructors is used here)\n",
    "# a further toy example will help you with it\n",
    "full_sites_sparse = csr_matrix(([1] * sites_flatten.shape[0],\n",
    "                                sites_flatten,\n",
    "                                range(0, sites_flatten.shape[0]  + 10, 10)))[:, 1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(336358, 48371)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "full_sites_sparse.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you understand what just happened here, then you can skip the next passage (perhaps, you can handle logistic regression too?), If not, then let us figure it out.\n",
    "\n",
    "### Important detour #1: Sparse Matrices\n",
    "\n",
    "Let us estimate how much memory it will require to store our data in the example above. Our united dataframe contains 336 thousand samples of 48 thousand integer features in each. It's easy to calculate the required amount of memory, roughly:\n",
    "\n",
    "$$336\\ K * 48\\ K * 8\\ bytes \\approx 16* 10^9 * 8\\ bytes = 130\\ GB,$$\n",
    "\n",
    "(that's the [exact](http://www.wolframalpha.com/input/?i=336358*48371*8+bytes) value). Obviously, ordinary mortals have no such volumes (strictly speaking, Python may allow you to create such a matrix, but it will not be easy to do anything with it). The interesting fact is that most of the elements of our matrix are zeros. If we count non-zero elements, then it will be about 1.8 million, i.е. slightly more than 10% of all matrix elements. Such a matrix, where most elements are zeros, is called sparse, and the ratio between the number of zero elements and the total number of elements is called the sparsity of the matrix.\n",
    "\n",
    "For the work with such matrices you can use `scipy.sparse` library, check [documentation](https://docs.scipy.org/doc/scipy-0.18.1/reference/sparse.html) to understand what possible types of sparse matrices are, how to work with them and in which cases their usage is most effective. You can learn how they are arranged, for example, in Wikipedia [article](https://en.wikipedia.org/wiki/Sparse_matrix).\n",
    "Note, that a sparse matrix contains only non-zero elements, and you can get the allocated memory size like this (significant memory savings are obvious):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1866898 elements * 8 bytes = 14935184 bytes\n",
      "sparse_matrix_size = 14935184 bytes\n"
     ]
    }
   ],
   "source": [
    "# How much memory does a sparse matrix occupy?\n",
    "print('{0} elements * {1} bytes = {2} bytes'.format(full_sites_sparse.count_nonzero(), 8, \n",
    "                                                    full_sites_sparse.count_nonzero() * 8))\n",
    "# Or just like this:\n",
    "print('sparse_matrix_size = {0} bytes'.format(full_sites_sparse.data.nbytes))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us explore how the matrix with the websites has been formed using a mini example. Suppose we have the following table with user sessions:\n",
    "\n",
    "| id | site1 | site2 | site3 |\n",
    "|---|---|---|---|\n",
    "| 1 | 1 | 0 | 0 |\n",
    "| 2 | 1 | 3 | 1 |\n",
    "| 3 | 2 | 3 | 4 |\n",
    "\n",
    "There are 3 sessions, and no more than 3 websites in each. Users visited four different sites in total (there are numbers from 1 to 4 in the table cells). And let us assume that the mapping is:\n",
    "\n",
    " 1. vk.com\n",
    " 2. habrahabr.ru \n",
    " 3. yandex.ru\n",
    " 4. ods.ai\n",
    "\n",
    "If the user has visited less than 3 websites during the session, the last few values will be zero. We want to convert the original dataframe in a way that each session has a corresponding row which shows the number of visits to each particular site. I.e. we want to transform the previous table into the following form:\n",
    "\n",
    "| id | vk.com | habrahabr.ru | yandex.ru | ods.ai |\n",
    "|---|---|---|---|---|\n",
    "| 1 | 1 | 0 | 0 | 0 |\n",
    "| 2 | 2 | 0 | 1 | 0 |\n",
    "| 3 | 0 | 1 | 1 | 1 |\n",
    "\n",
    "\n",
    "To do this, use the constructor: `csr_matrix ((data, indices, indptr))` and create a frequency table (see examples, code and comments on the links above to see how it works). Here we set all the parameters explicitly for greater clarity:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[2, 1, 0, 0, 0],\n",
       "        [0, 2, 0, 1, 0],\n",
       "        [0, 0, 1, 1, 1]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# data, create the list of ones, length of which equal to the number of elements in the initial dataframe (9)\n",
    "# By summing the number of ones in the cell, we get the frequency,\n",
    "# number of visits to a particular site per session\n",
    "data = [1] * 9\n",
    "\n",
    "# To do this, you need to correctly distribute the ones in cells\n",
    "# Indices - website ids, i.e. columns of a new matrix. We will sum ones up grouping them by sessions (ids)\n",
    "indices = [1, 0, 0, 1, 3, 1, 2, 3, 4]\n",
    "\n",
    "# Indices for the division into rows (sessions)\n",
    "# For example, line 0 is the elements between the indices [0; 3) - the rightmost value is not included\n",
    "# Line 1 is the elements between the indices [3; 6)\n",
    "# Line 2 is the elements between the indices [6; 9) \n",
    "indptr = [0, 3, 6, 9]\n",
    "\n",
    "# Aggregate these three variables into a tuple and compose a matrix\n",
    "# To display this matrix on the screen transform it into the usual \"dense\" matrix\n",
    "csr_matrix((data, indices, indptr)).todense()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you might have noticed, there are not four columns in the resulting matrix (corresponding to number of different websites) but five. A zero column has been added, which indicates if the session was shorter (in our mini example we took sessions of three). This column is excessive and should be removed from the dataframe (do that yourself).\n",
    "\n",
    "##### 3. What is the sparsity of the matrix in our small example?\n",
    "\n",
    "*For discussions, please stick to [ODS Slack](https://opendatascience.slack.com/), channel #mlcourse_ai_news, pinned thread __#a3_part1_fall2019__*\n",
    "\n",
    "- 42%\n",
    "- 47%\n",
    "- 50%\n",
    "- 53%\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another benefit of using sparse matrices is that there are special implementations of both matrix operations and machine learning algorithms for them, which sometimes allows to significantly accelerate operations due to the data structure peculiarities. This applies to logistic regression as well. Now everything is ready to build our first model.\n",
    "\n",
    "### 3. Training the first model\n",
    "\n",
    "So, we have an algorithm and data for it. Let us build our first model, using [logistic regression](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) implementation from ` Sklearn` with default parameters. We will use the first 90% of the data for training (the training data set is sorted by time), and the remaining 10% for validation. Let's write a simple function that returns the quality of the model and then train our first classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_auc_lr_valid(X, y, C=1.0, seed=17, ratio = 0.9):\n",
    "    # Split the data into the training and validation sets\n",
    "    idx = int(round(X.shape[0] * ratio))\n",
    "    # Classifier training\n",
    "    lr = LogisticRegression(C=C, random_state=seed, solver='liblinear').fit(X[:idx, :], y[:idx])\n",
    "    # Prediction for validation set\n",
    "    y_pred = lr.predict_proba(X[idx:, :])[:, 1]\n",
    "    # Calculate the quality\n",
    "    score = roc_auc_score(y[idx:], y_pred)\n",
    "    \n",
    "    return score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9195241058359782\n",
      "CPU times: user 12.5 s, sys: 130 ms, total: 12.6 s\n",
      "Wall time: 3.2 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Select the training set from the united dataframe (where we have the answers)\n",
    "X_train = full_sites_sparse[:idx_split, :]\n",
    "\n",
    "# Calculate metric on the validation set\n",
    "print(get_auc_lr_valid(X_train, y_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first model demonstrated the quality  of 0.92 on the validation set. Let's take it as the first baseline and starting point. To make a prediction on the test data set **we need to train the model again on the entire training data set** (until this moment, our model used only part of the data for training), which will increase its generalizing ability:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function for writing predictions to a file\n",
    "def write_to_submission_file(predicted_labels, out_file,\n",
    "                             target='target', index_label=\"session_id\"):\n",
    "    predicted_df = pd.DataFrame(predicted_labels,\n",
    "                                index = np.arange(1, predicted_labels.shape[0] + 1),\n",
    "                                columns=[target])\n",
    "    predicted_df.to_csv(out_file, index_label=index_label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the model on the whole training data set\n",
    "# Use random_state=17 for repeatability\n",
    "# Parameter C=1 by default, but here we set it explicitly\n",
    "lr = LogisticRegression(C=1.0, random_state=17, solver='liblinear').fit(X_train, y_train)\n",
    "\n",
    "# Make a prediction for test data set\n",
    "X_test = full_sites_sparse[idx_split:,:]\n",
    "y_test = lr.predict_proba(X_test)[:, 1]\n",
    "\n",
    "# Write it to the file which could be submitted\n",
    "write_to_submission_file(y_test, 'baseline_1.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you follow these steps and upload the answer to the competition [page](https://inclass.kaggle.com/c/catch-me-if-you-can-intruder-detection-through-webpage-session-tracking2), you will get `ROC AUC = 0.90812` on the public leaderboard (\"A3 baseline 1\").\n",
    "\n",
    "### 4. Model Improvement: Feature Engineering\n",
    "\n",
    "Now we are going to try to improve the quality of our model by adding new features to the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a feature that will be a number in YYYYMM format from the date when the session was held, for example 201407 -- year 2014 and 7th month. Thus, we will take into account the monthly [linear trend](http://people.duke.edu/~rnau/411trend.htm) for the entire period of the data provided."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dataframe for new features\n",
    "full_new_feat = pd.DataFrame(index=full_df.index)\n",
    "\n",
    "# Add start_month feature\n",
    "full_new_feat['start_month'] = full_df['time1'].apply(lambda ts: \n",
    "                                                      100 * ts.year + ts.month).astype('float64')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 4. Plot the graph of the number of Alice sessions versus the new feature, start_month. Choose the correct statement:\n",
    "\n",
    "*For discussions, please stick to [ODS Slack](https://opendatascience.slack.com/), channel #mlcourse_ai_news, pinned thread __#a3_part1_fall2019__*\n",
    "\n",
    "- Alice wasn't online at all for the entire period\n",
    "- From the beginning of 2013 to mid-2014, the number of Alice's sessions per month decreased\n",
    "- The number of Alice's sessions per month is generally constant for the entire period\n",
    "- From the beginning of 2013 to mid-2014, the number of Alice's sessions per month increased\n",
    "\n",
    "*Hint: the graph will be more explicit if you treat `start_month` as a categorical ordinal variable*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this way, we have an illustration and thoughts about the usefulness of the new feature, add it to the training sample and check the quality of the new model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7508354860175162\n"
     ]
    }
   ],
   "source": [
    "# Add the new feature to the sparse matrix\n",
    "tmp = full_new_feat[['start_month']].values\n",
    "X_train = csr_matrix(hstack([full_sites_sparse[:idx_split,:], tmp[:idx_split,:]]))\n",
    "\n",
    "# Compute the metric on the validation set\n",
    "print(get_auc_lr_valid(X_train, y_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The quality of the model has decreased significantly. We added a feature that definitely seemed useful to us, but its usage only worsened the model. Why did it happen?\n",
    "\n",
    "### Important detour #2: is it necessary to scale features?\n",
    "\n",
    "Here we give an intuitive reasoning (a rigorous mathematical justification for one or another aspect in linear models you can easily find on the internet). Consider the features more closely: those of them that correspond to the number of visits to a particular web-site per session vary from 0 to 10. The feature `start_month` has a completely different range: from 201301 to 201412, this means the contribution of this variable is significantly greater than the others. It would seem that problem can be avoided if we put less weight in a linear combination of attributes in this case, but in our case logistic regression with regularization is used (by default, this parameter is `C = 1`), which penalizes the model the stronger the greater its weights are. Therefore, for linear methods with regularization, it is recommended to convert features to the same scale (you can read more about the regularization, for example, [here](https://habrahabr.ru/company/ods/blog/322076/)).\n",
    "\n",
    "One way to do this is standardization: for each observation you need to subtract the average value of the feature and divide this difference by the standard deviation:\n",
    "\n",
    "$$ x^{*}_{i} = \\dfrac{x_{i} - \\mu_x}{\\sigma_x}$$\n",
    "\n",
    "The following practical tips can be given:\n",
    "- It is recommended to scale features if they have essentially different ranges or different units of measurement (for example, the country's population is indicated in units, and the country's GNP in trillions)\n",
    "- Scale features if you do not have a reason/expert opinion to give a greater weight to any of them\n",
    "- Scaling can be excessive if the ranges of some of your features differ from each other, but they are in the same system of units (for example, the proportion of middle-aged people and people over 80 among the entire population)\n",
    "- If you want to get an interpreted model, then build a model without regularization and scaling (most likely, its quality will be worse)\n",
    "- Binary features (which take only values of 0 or 1) are usually left without conversion, (but)\n",
    "- If the quality of the model is crucial, try different options and select one where the quality is better\n",
    "\n",
    "Getting back to `start_month`, let us rescale the new feature and train the model again. This time the quality has increased:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9196984641972088\n"
     ]
    }
   ],
   "source": [
    "# Add the new standardized feature to the sparse matrix\n",
    "tmp = StandardScaler().fit_transform(full_new_feat[['start_month']])\n",
    "X_train = csr_matrix(hstack([full_sites_sparse[:idx_split,:], tmp[:idx_split,:]]))\n",
    "\n",
    "# Compute metric on the validation set\n",
    "print(get_auc_lr_valid(X_train, y_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 5. Add to the training set a new feature \"n_unique_sites\" – the number of the unique web-sites in a session. Calculate how the quality on the validation set has changed\n",
    "\n",
    "*For discussions, please stick to [ODS Slack](https://opendatascience.slack.com/), channel #mlcourse_ai_news, pinned thread __#a3_part1_fall2019__*\n",
    "\n",
    "- It has decreased. It is better not to add a new feature.\n",
    "- It has not changed\n",
    "- It has decreased. The new feature should be scaled.\n",
    "- I am confused, and I do not know if it's necessary to scale a new feature.\n",
    "\n",
    "*Tips: use the nunique() function from `pandas`. Do not forget to include the start_month in the set. Will you scale a new feature? Why?*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, the new feature has slightly decreased the quality, so we will not use it. Nevertheless, do not rush to throw features out because they haven't performed well. They can be useful in a combination with other features (for example, when a new feature is a ratio or a product of two others).\n",
    "\n",
    "#####  6. Add two new features: start_hour and morning. Calculate the metric. Which of these features gives an improvement?\n",
    "\n",
    "The `start_hour` feature is the hour at which the session started (from 0 to 23), and the binary feature `morning` is equal to 1 if the session started in the morning and 0 if the session started later (we assume that morning means `start_hour` is equal to 11 or less).\n",
    "\n",
    "Will you scale the new features? Make your assumptions and test them in practice.\n",
    "\n",
    "*For discussions, please stick to [ODS Slack](https://opendatascience.slack.com/), channel #mlcourse_ai_news, pinned thread __#a3_part1_fall2019__*\n",
    "\n",
    "- None of the features gave an improvement :(\n",
    "- `start_hour` feature gave an improvement, and `morning` did not\n",
    "- `morning` feature gave an improvement, and `start_hour` did not\n",
    "- Both features gave an improvement\n",
    "\n",
    "*Tip: find suitable functions for working with time series data in [documentation](http://pandas.pydata.org/pandas-docs/stable/api.html). Do not forget to include the `start_month` feature.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Regularization and Parameter Tuning\n",
    "\n",
    "We have introduced features that improve the quality of our model in comparison with the first baseline. Can we do even better? After we have changed the training and test sets, it almost always makes sense to search for the optimal hyperparameters - the parameters of the model that do not change during training.\n",
    "\n",
    "For example, in week 3, you learned that, in decision trees, the depth of the tree is a hyperparameter, but the feature by which splitting occurs and its threshold is not. \n",
    "\n",
    "In the logistic regression that we use, the weights of each feature are changing, and we find their optimal values during training; meanwhile, the regularization parameter remains constant. This is the hyperparameter that we are going to optimize now.\n",
    "\n",
    "Calculate the quality on a validation set with a regularization parameter, which is equal to 1 by default:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compose the training set\n",
    "tmp_scaled = StandardScaler().fit_transform(full_new_feat[['start_month', \n",
    "                                                           'start_hour', \n",
    "                                                           'morning']])\n",
    "X_train = csr_matrix(hstack([full_sites_sparse[:idx_split,:], \n",
    "                             tmp_scaled[:idx_split,:]]))\n",
    "\n",
    "# Capture the quality with default parameters\n",
    "score_C_1 = get_auc_lr_valid(X_train, y_train)\n",
    "print(score_C_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will try to beat this result by optimizing the regularization parameter. We will take a list of possible values of C and calculate the quality metric on the validation set for each of C-values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List of possible C-values\n",
    "Cs = np.logspace(-3, 1, 10)\n",
    "\n",
    "# You code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the graph of the quality metric (AUC-ROC) versus the value of the regularization parameter. The value of quality metric corresponding to the default value of C=1 is represented by a horizontal dotted line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 7. What is the value of parameter C (if rounded to 2 decimals) that corresponds to the highest model quality?\n",
    "\n",
    "*For discussions, please stick to [ODS Slack](https://opendatascience.slack.com/), channel #mlcourse_ai_news, pinned thread __#a3_part1_fall2019__*\n",
    "\n",
    "- 0.17\n",
    "- 0.46\n",
    "- 1.29\n",
    "- 3.14"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the last task in this assignment: train the model using the optimal regularization parameter you found (do not round up to two digits like in the last question). If you do everything correctly and submit your solution, you should see `ROC AUC = 0.92784` on the public leaderboard (\"A3 baseline 2\"):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare the training and test data\n",
    "tmp_scaled = StandardScaler().fit_transform(full_new_feat[['start_month', 'start_hour', \n",
    "                                                           'morning']])\n",
    "X_train = csr_matrix(hstack([full_sites_sparse[:idx_split,:], \n",
    "                             tmp_scaled[:idx_split,:]]))\n",
    "X_test = csr_matrix(hstack([full_sites_sparse[idx_split:,:], \n",
    "                            tmp_scaled[idx_split:,:]]))\n",
    "\n",
    "# Train the model on the whole training data set using optimal regularization parameter\n",
    "lr = LogisticRegression(C=C, random_state=17, solver='liblinear').fit(X_train, y_train)\n",
    "\n",
    "# Make a prediction for the test set\n",
    "y_test = lr.predict_proba(X_test)[:, 1]\n",
    "\n",
    "# Write it to the submission file\n",
    "write_to_submission_file(y_test, 'baseline_2.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part of the assignment, you have learned how to use sparse matrices, train logistic regression models, create new features and selected the best ones, learned why you need to scale features, and how to select hyperparameters. That's a lot!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Subpart 2. Freeride"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='../../img/snowboard.jpg' width=70%>\n",
    "\n",
    "*Yorko in Sheregesh, the best place in Russia for snowboarding and skiing.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part, you'll need to beat the 2 more baselines mentioned in the beginning of this assignment. No more step-by-step instructions. But it'll be very helpful for you to study the Kernel \"[Model validation in a competition](https://www.kaggle.com/kashnitsky/model-validation-in-a-competition)\".\n",
    "\n",
    "Here are a few tips for finding new features: think about what you can come up with using existing features, try multiplying or dividing two of them, justify or decline your hypotheses with plots, extract useful information from time series data (time1 ... time10), do not hesitate to convert an existing feature (for example, take a logarithm), etc. Checkout other [Kernels](https://www.kaggle.com/c/catch-me-if-you-can-intruder-detection-through-webpage-session-tracking2/kernels). We encourage you to try new ideas and models - it's fun!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}