Engagement_Score_Prediction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "39a0684c",
   "metadata": {
    "papermill": {
     "duration": 0.029985,
     "end_time": "2022-02-12T13:46:28.534163",
     "exception": false,
     "start_time": "2022-02-12T13:46:28.504178",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Engagement Score Prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de67ce5b",
   "metadata": {
    "papermill": {
     "duration": 0.028323,
     "end_time": "2022-02-12T13:46:28.591003",
     "exception": false,
     "start_time": "2022-02-12T13:46:28.562680",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Problem Statement\n",
    "\n",
    "ABC is an online content sharing platform that enables users to create, upload and share the content in the form of videos. It includes videos from different genres like entertainment, education, sports, technology and so on. The maximum duration of video is 10 minutes.\n",
    "\n",
    "Users can like, comment and share the videos on the platform. \n",
    "\n",
    "Based on the user’s interaction with the videos, engagement score is assigned to the video with respect to each user. Engagement score defines how engaging the content of the video is. \n",
    "\n",
    "Understanding the engagement score of the video improves the user’s interaction with the platform. It defines the type of content that is appealing to the user and engages the larger audience.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1133288",
   "metadata": {
    "papermill": {
     "duration": 0.043557,
     "end_time": "2022-02-12T13:46:28.685007",
     "exception": false,
     "start_time": "2022-02-12T13:46:28.641450",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Objective\n",
    "The main objective of the problem is to develop the machine learning approach to predict the engagement score of the video on the user level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f0be7ee6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:28.754422Z",
     "iopub.status.busy": "2022-02-12T13:46:28.753639Z",
     "iopub.status.idle": "2022-02-12T13:46:29.663845Z",
     "shell.execute_reply": "2022-02-12T13:46:29.663252Z",
     "shell.execute_reply.started": "2022-02-12T10:00:02.613933Z"
    },
    "papermill": {
     "duration": 0.950214,
     "end_time": "2022-02-12T13:46:29.664003",
     "exception": false,
     "start_time": "2022-02-12T13:46:28.713789",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# import libraries\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# warnings\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c24fa4ef",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:29.728028Z",
     "iopub.status.busy": "2022-02-12T13:46:29.727255Z",
     "iopub.status.idle": "2022-02-12T13:46:29.971666Z",
     "shell.execute_reply": "2022-02-12T13:46:29.971167Z",
     "shell.execute_reply.started": "2022-02-12T10:00:45.424048Z"
    },
    "papermill": {
     "duration": 0.277606,
     "end_time": "2022-02-12T13:46:29.971788",
     "exception": false,
     "start_time": "2022-02-12T13:46:29.694182",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# import machine learning libraries\n",
    "from sklearn.preprocessing import LabelEncoder, StandardScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.metrics import r2_score\n",
    "\n",
    "# import linear regression\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "# import xgboost\n",
    "from xgboost import XGBRegressor\n",
    "\n",
    "# import gridsearchcv for cross-validation\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import RandomizedSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "127355d7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:30.035318Z",
     "iopub.status.busy": "2022-02-12T13:46:30.034484Z",
     "iopub.status.idle": "2022-02-12T13:46:30.207368Z",
     "shell.execute_reply": "2022-02-12T13:46:30.208149Z",
     "shell.execute_reply.started": "2022-02-12T10:00:10.583041Z"
    },
    "papermill": {
     "duration": 0.206475,
     "end_time": "2022-02-12T13:46:30.208387",
     "exception": false,
     "start_time": "2022-02-12T13:46:30.001912",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>row_id</th>\n",
       "      <th>user_id</th>\n",
       "      <th>category_id</th>\n",
       "      <th>video_id</th>\n",
       "      <th>age</th>\n",
       "      <th>gender</th>\n",
       "      <th>profession</th>\n",
       "      <th>followers</th>\n",
       "      <th>views</th>\n",
       "      <th>engagement_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>19990</td>\n",
       "      <td>37</td>\n",
       "      <td>128</td>\n",
       "      <td>24</td>\n",
       "      <td>Male</td>\n",
       "      <td>Student</td>\n",
       "      <td>180</td>\n",
       "      <td>1000</td>\n",
       "      <td>4.33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>5304</td>\n",
       "      <td>32</td>\n",
       "      <td>132</td>\n",
       "      <td>14</td>\n",
       "      <td>Female</td>\n",
       "      <td>Student</td>\n",
       "      <td>330</td>\n",
       "      <td>714</td>\n",
       "      <td>1.79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1840</td>\n",
       "      <td>12</td>\n",
       "      <td>24</td>\n",
       "      <td>19</td>\n",
       "      <td>Male</td>\n",
       "      <td>Student</td>\n",
       "      <td>180</td>\n",
       "      <td>138</td>\n",
       "      <td>4.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>12597</td>\n",
       "      <td>23</td>\n",
       "      <td>112</td>\n",
       "      <td>19</td>\n",
       "      <td>Male</td>\n",
       "      <td>Student</td>\n",
       "      <td>220</td>\n",
       "      <td>613</td>\n",
       "      <td>3.77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>13626</td>\n",
       "      <td>23</td>\n",
       "      <td>112</td>\n",
       "      <td>27</td>\n",
       "      <td>Male</td>\n",
       "      <td>Working Professional</td>\n",
       "      <td>220</td>\n",
       "      <td>613</td>\n",
       "      <td>3.13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   row_id  user_id  category_id  video_id  age  gender            profession  \\\n",
       "0       1    19990           37       128   24    Male               Student   \n",
       "1       2     5304           32       132   14  Female               Student   \n",
       "2       3     1840           12        24   19    Male               Student   \n",
       "3       4    12597           23       112   19    Male               Student   \n",
       "4       5    13626           23       112   27    Male  Working Professional   \n",
       "\n",
       "   followers  views  engagement_score  \n",
       "0        180   1000              4.33  \n",
       "1        330    714              1.79  \n",
       "2        180    138              4.35  \n",
       "3        220    613              3.77  \n",
       "4        220    613              3.13  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import data\n",
    "train_data = pd.read_csv('data/train_0OECtn8.csv')\n",
    "train_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e0e881af",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:30.330366Z",
     "iopub.status.busy": "2022-02-12T13:46:30.329461Z",
     "iopub.status.idle": "2022-02-12T13:46:30.333197Z",
     "shell.execute_reply": "2022-02-12T13:46:30.331121Z",
     "shell.execute_reply.started": "2022-02-12T10:00:12.520164Z"
    },
    "papermill": {
     "duration": 0.067495,
     "end_time": "2022-02-12T13:46:30.333350",
     "exception": false,
     "start_time": "2022-02-12T13:46:30.265855",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(89197, 10)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# shape \n",
    "train_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e6b8246c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:30.439431Z",
     "iopub.status.busy": "2022-02-12T13:46:30.438679Z",
     "iopub.status.idle": "2022-02-12T13:46:30.465091Z",
     "shell.execute_reply": "2022-02-12T13:46:30.466030Z",
     "shell.execute_reply.started": "2022-02-12T10:00:12.836612Z"
    },
    "papermill": {
     "duration": 0.085368,
     "end_time": "2022-02-12T13:46:30.466213",
     "exception": false,
     "start_time": "2022-02-12T13:46:30.380845",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "row_id              0\n",
       "user_id             0\n",
       "category_id         0\n",
       "video_id            0\n",
       "age                 0\n",
       "gender              0\n",
       "profession          0\n",
       "followers           0\n",
       "views               0\n",
       "engagement_score    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# lets check for null values\n",
    "train_data.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "262e6806",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:30.569326Z",
     "iopub.status.busy": "2022-02-12T13:46:30.565670Z",
     "iopub.status.idle": "2022-02-12T13:46:30.605651Z",
     "shell.execute_reply": "2022-02-12T13:46:30.606579Z",
     "shell.execute_reply.started": "2022-02-12T10:00:14.624152Z"
    },
    "papermill": {
     "duration": 0.092626,
     "end_time": "2022-02-12T13:46:30.606760",
     "exception": false,
     "start_time": "2022-02-12T13:46:30.514134",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "row_id              89197\n",
       "user_id             27734\n",
       "category_id            47\n",
       "video_id              175\n",
       "age                    58\n",
       "gender                  2\n",
       "profession              3\n",
       "followers              17\n",
       "views                  43\n",
       "engagement_score      229\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# unique values in a column\n",
    "train_data.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b017f55e",
   "metadata": {
    "papermill": {
     "duration": 0.047632,
     "end_time": "2022-02-12T13:46:30.706037",
     "exception": false,
     "start_time": "2022-02-12T13:46:30.658405",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## data preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "35816153",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:30.802596Z",
     "iopub.status.busy": "2022-02-12T13:46:30.801479Z",
     "iopub.status.idle": "2022-02-12T13:46:30.804679Z",
     "shell.execute_reply": "2022-02-12T13:46:30.805115Z",
     "shell.execute_reply.started": "2022-02-12T10:00:16.270631Z"
    },
    "papermill": {
     "duration": 0.051516,
     "end_time": "2022-02-12T13:46:30.805249",
     "exception": false,
     "start_time": "2022-02-12T13:46:30.753733",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>row_id</th>\n",
       "      <th>user_id</th>\n",
       "      <th>category_id</th>\n",
       "      <th>video_id</th>\n",
       "      <th>age</th>\n",
       "      <th>gender</th>\n",
       "      <th>profession</th>\n",
       "      <th>followers</th>\n",
       "      <th>views</th>\n",
       "      <th>engagement_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>19990</td>\n",
       "      <td>37</td>\n",
       "      <td>128</td>\n",
       "      <td>24</td>\n",
       "      <td>Male</td>\n",
       "      <td>Student</td>\n",
       "      <td>180</td>\n",
       "      <td>1000</td>\n",
       "      <td>4.33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>5304</td>\n",
       "      <td>32</td>\n",
       "      <td>132</td>\n",
       "      <td>14</td>\n",
       "      <td>Female</td>\n",
       "      <td>Student</td>\n",
       "      <td>330</td>\n",
       "      <td>714</td>\n",
       "      <td>1.79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1840</td>\n",
       "      <td>12</td>\n",
       "      <td>24</td>\n",
       "      <td>19</td>\n",
       "      <td>Male</td>\n",
       "      <td>Student</td>\n",
       "      <td>180</td>\n",
       "      <td>138</td>\n",
       "      <td>4.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>12597</td>\n",
       "      <td>23</td>\n",
       "      <td>112</td>\n",
       "      <td>19</td>\n",
       "      <td>Male</td>\n",
       "      <td>Student</td>\n",
       "      <td>220</td>\n",
       "      <td>613</td>\n",
       "      <td>3.77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>13626</td>\n",
       "      <td>23</td>\n",
       "      <td>112</td>\n",
       "      <td>27</td>\n",
       "      <td>Male</td>\n",
       "      <td>Working Professional</td>\n",
       "      <td>220</td>\n",
       "      <td>613</td>\n",
       "      <td>3.13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   row_id  user_id  category_id  video_id  age  gender            profession  \\\n",
       "0       1    19990           37       128   24    Male               Student   \n",
       "1       2     5304           32       132   14  Female               Student   \n",
       "2       3     1840           12        24   19    Male               Student   \n",
       "3       4    12597           23       112   19    Male               Student   \n",
       "4       5    13626           23       112   27    Male  Working Professional   \n",
       "\n",
       "   followers  views  engagement_score  \n",
       "0        180   1000              4.33  \n",
       "1        330    714              1.79  \n",
       "2        180    138              4.35  \n",
       "3        220    613              3.77  \n",
       "4        220    613              3.13  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ad420100",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:30.875487Z",
     "iopub.status.busy": "2022-02-12T13:46:30.868459Z",
     "iopub.status.idle": "2022-02-12T13:46:30.877934Z",
     "shell.execute_reply": "2022-02-12T13:46:30.877462Z",
     "shell.execute_reply.started": "2022-02-12T10:00:18.046804Z"
    },
    "papermill": {
     "duration": 0.042749,
     "end_time": "2022-02-12T13:46:30.878047",
     "exception": false,
     "start_time": "2022-02-12T13:46:30.835298",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# remove unwanted features/columns to assign X and y\n",
    "X = train_data.drop(['row_id', 'user_id','category_id','video_id', 'engagement_score'], axis=1)\n",
    "y = train_data['engagement_score']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2ba38425",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:30.961326Z",
     "iopub.status.busy": "2022-02-12T13:46:30.950586Z",
     "iopub.status.idle": "2022-02-12T13:46:31.001657Z",
     "shell.execute_reply": "2022-02-12T13:46:31.002327Z",
     "shell.execute_reply.started": "2022-02-12T10:00:19.065357Z"
    },
    "papermill": {
     "duration": 0.09443,
     "end_time": "2022-02-12T13:46:31.002553",
     "exception": false,
     "start_time": "2022-02-12T13:46:30.908123",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>gender</th>\n",
       "      <th>profession</th>\n",
       "      <th>followers</th>\n",
       "      <th>views</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>24</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>180</td>\n",
       "      <td>1000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>14</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>330</td>\n",
       "      <td>714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>19</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>180</td>\n",
       "      <td>138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>19</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>220</td>\n",
       "      <td>613</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>27</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>220</td>\n",
       "      <td>613</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age  gender  profession  followers  views\n",
       "0   24       1           1        180   1000\n",
       "1   14       0           1        330    714\n",
       "2   19       1           1        180    138\n",
       "3   19       1           1        220    613\n",
       "4   27       1           2        220    613"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# label encoder for catagorical features\n",
    "encoder = LabelEncoder()\n",
    "\n",
    "cat_cols = ['gender', 'profession']\n",
    "\n",
    "for col in cat_cols:\n",
    "    X[col] = encoder.fit_transform(X[col])\n",
    "\n",
    "X.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d85f5e8c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:31.124015Z",
     "iopub.status.busy": "2022-02-12T13:46:31.123186Z",
     "iopub.status.idle": "2022-02-12T13:46:31.128940Z",
     "shell.execute_reply": "2022-02-12T13:46:31.129953Z",
     "shell.execute_reply.started": "2022-02-12T10:00:23.540937Z"
    },
    "papermill": {
     "duration": 0.070177,
     "end_time": "2022-02-12T13:46:31.130153",
     "exception": false,
     "start_time": "2022-02-12T13:46:31.059976",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "age           58\n",
       "gender         2\n",
       "profession     3\n",
       "followers     17\n",
       "views         43\n",
       "dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "532febd6",
   "metadata": {
    "papermill": {
     "duration": 0.048175,
     "end_time": "2022-02-12T13:46:31.226649",
     "exception": false,
     "start_time": "2022-02-12T13:46:31.178474",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Train Test Split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d2fb23a3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:31.336464Z",
     "iopub.status.busy": "2022-02-12T13:46:31.335787Z",
     "iopub.status.idle": "2022-02-12T13:46:31.351573Z",
     "shell.execute_reply": "2022-02-12T13:46:31.352542Z",
     "shell.execute_reply.started": "2022-02-12T10:00:25.425663Z"
    },
    "papermill": {
     "duration": 0.076896,
     "end_time": "2022-02-12T13:46:31.352728",
     "exception": false,
     "start_time": "2022-02-12T13:46:31.275832",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "24106fe6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:31.462327Z",
     "iopub.status.busy": "2022-02-12T13:46:31.461465Z",
     "iopub.status.idle": "2022-02-12T13:46:31.465081Z",
     "shell.execute_reply": "2022-02-12T13:46:31.463046Z",
     "shell.execute_reply.started": "2022-02-12T10:00:26.778613Z"
    },
    "papermill": {
     "duration": 0.060285,
     "end_time": "2022-02-12T13:46:31.465233",
     "exception": false,
     "start_time": "2022-02-12T13:46:31.404948",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(62437, 5)\n",
      "(62437,)\n",
      "(26760, 5)\n",
      "(26760,)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape)\n",
    "print(y_train.shape)\n",
    "print(X_test.shape)\n",
    "print(y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61b348f0",
   "metadata": {
    "papermill": {
     "duration": 0.051295,
     "end_time": "2022-02-12T13:46:31.566413",
     "exception": false,
     "start_time": "2022-02-12T13:46:31.515118",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Data Standardization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "41c11b6d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:31.677681Z",
     "iopub.status.busy": "2022-02-12T13:46:31.676933Z",
     "iopub.status.idle": "2022-02-12T13:46:31.704747Z",
     "shell.execute_reply": "2022-02-12T13:46:31.705840Z",
     "shell.execute_reply.started": "2022-02-12T10:00:28.886223Z"
    },
    "papermill": {
     "duration": 0.088807,
     "end_time": "2022-02-12T13:46:31.706049",
     "exception": false,
     "start_time": "2022-02-12T13:46:31.617242",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# # MinMaxScaler\n",
    "# scaler = MinMaxScaler()\n",
    "\n",
    "# Standarscaler\n",
    "scaler = StandardScaler()\n",
    "\n",
    "# all columns to all_cols\n",
    "all_cols = X_train.columns\n",
    "\n",
    "# fit scaler\n",
    "X_train[all_cols] = scaler.fit_transform(X_train[all_cols])\n",
    "X_test[all_cols] = scaler.transform(X_test[all_cols])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "785923a9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:31.825571Z",
     "iopub.status.busy": "2022-02-12T13:46:31.824800Z",
     "iopub.status.idle": "2022-02-12T13:46:31.831752Z",
     "shell.execute_reply": "2022-02-12T13:46:31.832779Z",
     "shell.execute_reply.started": "2022-02-12T10:00:30.216536Z"
    },
    "papermill": {
     "duration": 0.072795,
     "end_time": "2022-02-12T13:46:31.832984",
     "exception": false,
     "start_time": "2022-02-12T13:46:31.760189",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>gender</th>\n",
       "      <th>profession</th>\n",
       "      <th>followers</th>\n",
       "      <th>views</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6159</th>\n",
       "      <td>0.350986</td>\n",
       "      <td>0.839918</td>\n",
       "      <td>-1.280995</td>\n",
       "      <td>-0.055484</td>\n",
       "      <td>-1.204521</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45351</th>\n",
       "      <td>1.690816</td>\n",
       "      <td>-1.190592</td>\n",
       "      <td>-1.280995</td>\n",
       "      <td>-0.272351</td>\n",
       "      <td>-1.022034</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19559</th>\n",
       "      <td>0.350986</td>\n",
       "      <td>0.839918</td>\n",
       "      <td>-1.280995</td>\n",
       "      <td>-0.272351</td>\n",
       "      <td>-0.694301</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83744</th>\n",
       "      <td>0.909248</td>\n",
       "      <td>0.839918</td>\n",
       "      <td>-1.280995</td>\n",
       "      <td>1.679457</td>\n",
       "      <td>0.784218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23113</th>\n",
       "      <td>0.797596</td>\n",
       "      <td>-1.190592</td>\n",
       "      <td>-1.280995</td>\n",
       "      <td>-0.489219</td>\n",
       "      <td>-0.500641</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            age    gender  profession  followers     views\n",
       "6159   0.350986  0.839918   -1.280995  -0.055484 -1.204521\n",
       "45351  1.690816 -1.190592   -1.280995  -0.272351 -1.022034\n",
       "19559  0.350986  0.839918   -1.280995  -0.272351 -0.694301\n",
       "83744  0.909248  0.839918   -1.280995   1.679457  0.784218\n",
       "23113  0.797596 -1.190592   -1.280995  -0.489219 -0.500641"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "31b22aa0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:31.945893Z",
     "iopub.status.busy": "2022-02-12T13:46:31.945106Z",
     "iopub.status.idle": "2022-02-12T13:46:31.995782Z",
     "shell.execute_reply": "2022-02-12T13:46:31.996430Z",
     "shell.execute_reply.started": "2022-02-12T10:00:50.826276Z"
    },
    "papermill": {
     "duration": 0.110011,
     "end_time": "2022-02-12T13:46:31.996643",
     "exception": false,
     "start_time": "2022-02-12T13:46:31.886632",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>gender</th>\n",
       "      <th>profession</th>\n",
       "      <th>followers</th>\n",
       "      <th>views</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>6.243700e+04</td>\n",
       "      <td>6.243700e+04</td>\n",
       "      <td>6.243700e+04</td>\n",
       "      <td>6.243700e+04</td>\n",
       "      <td>6.243700e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1.884330e-15</td>\n",
       "      <td>1.474584e-16</td>\n",
       "      <td>-1.244541e-15</td>\n",
       "      <td>5.011927e-16</td>\n",
       "      <td>1.021175e-15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.000008e+00</td>\n",
       "      <td>1.000008e+00</td>\n",
       "      <td>1.000008e+00</td>\n",
       "      <td>1.000008e+00</td>\n",
       "      <td>1.000008e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-1.658760e+00</td>\n",
       "      <td>-1.190592e+00</td>\n",
       "      <td>-1.280995e+00</td>\n",
       "      <td>-2.007292e+00</td>\n",
       "      <td>-1.763155e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>-7.655395e-01</td>\n",
       "      <td>-1.190592e+00</td>\n",
       "      <td>-1.280995e+00</td>\n",
       "      <td>-4.892191e-01</td>\n",
       "      <td>-1.022034e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>-2.072770e-01</td>\n",
       "      <td>8.399184e-01</td>\n",
       "      <td>1.470418e-01</td>\n",
       "      <td>-2.723514e-01</td>\n",
       "      <td>-1.356668e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>7.975955e-01</td>\n",
       "      <td>8.399184e-01</td>\n",
       "      <td>1.470418e-01</td>\n",
       "      <td>5.951191e-01</td>\n",
       "      <td>7.842179e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>4.817086e+00</td>\n",
       "      <td>8.399184e-01</td>\n",
       "      <td>1.575079e+00</td>\n",
       "      <td>2.330060e+00</td>\n",
       "      <td>1.849348e+00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                age        gender    profession     followers         views\n",
       "count  6.243700e+04  6.243700e+04  6.243700e+04  6.243700e+04  6.243700e+04\n",
       "mean   1.884330e-15  1.474584e-16 -1.244541e-15  5.011927e-16  1.021175e-15\n",
       "std    1.000008e+00  1.000008e+00  1.000008e+00  1.000008e+00  1.000008e+00\n",
       "min   -1.658760e+00 -1.190592e+00 -1.280995e+00 -2.007292e+00 -1.763155e+00\n",
       "25%   -7.655395e-01 -1.190592e+00 -1.280995e+00 -4.892191e-01 -1.022034e+00\n",
       "50%   -2.072770e-01  8.399184e-01  1.470418e-01 -2.723514e-01 -1.356668e-01\n",
       "75%    7.975955e-01  8.399184e-01  1.470418e-01  5.951191e-01  7.842179e-01\n",
       "max    4.817086e+00  8.399184e-01  1.575079e+00  2.330060e+00  1.849348e+00"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5b4b09e",
   "metadata": {
    "papermill": {
     "duration": 0.052495,
     "end_time": "2022-02-12T13:46:32.103846",
     "exception": false,
     "start_time": "2022-02-12T13:46:32.051351",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Linear Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "2cfa2700",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:32.187498Z",
     "iopub.status.busy": "2022-02-12T13:46:32.186586Z",
     "iopub.status.idle": "2022-02-12T13:46:32.208399Z",
     "shell.execute_reply": "2022-02-12T13:46:32.208770Z",
     "shell.execute_reply.started": "2022-02-12T10:00:52.999798Z"
    },
    "papermill": {
     "duration": 0.059645,
     "end_time": "2022-02-12T13:46:32.208937",
     "exception": false,
     "start_time": "2022-02-12T13:46:32.149292",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression()"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "le_regg = LinearRegression()\n",
    "\n",
    "le_regg.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "e2db1e09",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:32.283258Z",
     "iopub.status.busy": "2022-02-12T13:46:32.279551Z",
     "iopub.status.idle": "2022-02-12T13:46:32.287107Z",
     "shell.execute_reply": "2022-02-12T13:46:32.287962Z",
     "shell.execute_reply.started": "2022-02-12T10:00:54.416601Z"
    },
    "papermill": {
     "duration": 0.045669,
     "end_time": "2022-02-12T13:46:32.288187",
     "exception": false,
     "start_time": "2022-02-12T13:46:32.242518",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "y_train_pred = le_regg.predict(X_train)\n",
    "y_test_pred = le_regg.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "9bded113",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:32.421724Z",
     "iopub.status.busy": "2022-02-12T13:46:32.420783Z",
     "iopub.status.idle": "2022-02-12T13:46:32.426640Z",
     "shell.execute_reply": "2022-02-12T13:46:32.426209Z",
     "shell.execute_reply.started": "2022-02-12T10:01:07.395376Z"
    },
    "papermill": {
     "duration": 0.079601,
     "end_time": "2022-02-12T13:46:32.426755",
     "exception": false,
     "start_time": "2022-02-12T13:46:32.347154",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train score: 0.22873778028867142\n",
      "Test Score: 0.23078575916618294\n"
     ]
    }
   ],
   "source": [
    "print('Train score:', r2_score(y_train, y_train_pred))\n",
    "print('Test Score:',r2_score(y_test, y_test_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40e7b046",
   "metadata": {
    "papermill": {
     "duration": 0.033484,
     "end_time": "2022-02-12T13:46:32.493393",
     "exception": false,
     "start_time": "2022-02-12T13:46:32.459909",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## XGBoost Regressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "ce185a97",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:32.563376Z",
     "iopub.status.busy": "2022-02-12T13:46:32.562548Z",
     "iopub.status.idle": "2022-02-12T13:46:42.610196Z",
     "shell.execute_reply": "2022-02-12T13:46:42.609509Z",
     "shell.execute_reply.started": "2022-02-12T13:45:15.086752Z"
    },
    "papermill": {
     "duration": 10.083471,
     "end_time": "2022-02-12T13:46:42.610331",
     "exception": false,
     "start_time": "2022-02-12T13:46:32.526860",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
       "             colsample_bynode=1, colsample_bytree=1, enable_categorical=False,\n",
       "             gamma=0, gpu_id=-1, importance_type=None,\n",
       "             interaction_constraints='', learning_rate=0.01, max_delta_step=0,\n",
       "             max_depth=8, min_child_weight=1, missing=nan,\n",
       "             monotone_constraints='()', n_estimators=800, n_jobs=8,\n",
       "             num_parallel_tree=1, predictor='auto', random_state=0, reg_alpha=0,\n",
       "             reg_lambda=1, scale_pos_weight=1, subsample=1, tree_method='exact',\n",
       "             validate_parameters=1, verbosity=None)"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb_reg = XGBRegressor(n_estimators = 800, max_depth = 8, learning_rate=0.01)\n",
    "# {n_estimators = 500, max_depth = 12, learning_rate=0.01} - test score: 0.342808574108327\n",
    "# {n_estimators = 800, max_depth = 8, learning_rate=0.01} - test score: 0.3528540644392151\n",
    "\n",
    "xgb_reg.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "ad15c03b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:42.681628Z",
     "iopub.status.busy": "2022-02-12T13:46:42.680891Z",
     "iopub.status.idle": "2022-02-12T13:46:44.394731Z",
     "shell.execute_reply": "2022-02-12T13:46:44.395685Z",
     "shell.execute_reply.started": "2022-02-12T13:45:21.689361Z"
    },
    "papermill": {
     "duration": 1.751736,
     "end_time": "2022-02-12T13:46:44.395883",
     "exception": false,
     "start_time": "2022-02-12T13:46:42.644147",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "y_train_pred = xgb_reg.predict(X_train)\n",
    "y_test_pred = xgb_reg.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "5d73f87f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:44.470168Z",
     "iopub.status.busy": "2022-02-12T13:46:44.469240Z",
     "iopub.status.idle": "2022-02-12T13:46:44.474588Z",
     "shell.execute_reply": "2022-02-12T13:46:44.474990Z",
     "shell.execute_reply.started": "2022-02-12T13:45:23.425582Z"
    },
    "papermill": {
     "duration": 0.043448,
     "end_time": "2022-02-12T13:46:44.475130",
     "exception": false,
     "start_time": "2022-02-12T13:46:44.431682",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train score: 0.38667672834521194\n",
      "Test Score: 0.3528540644392151\n"
     ]
    }
   ],
   "source": [
    "print('Train score:', r2_score(y_train, y_train_pred))\n",
    "print('Test Score:',r2_score(y_test, y_test_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1859d93f",
   "metadata": {
    "papermill": {
     "duration": 0.033565,
     "end_time": "2022-02-12T13:46:44.542505",
     "exception": false,
     "start_time": "2022-02-12T13:46:44.508940",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Hyper parameter tuning with XGBoost & RandomizedSearchCV - 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "77a5daef",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:44.614026Z",
     "iopub.status.busy": "2022-02-12T13:46:44.613235Z",
     "iopub.status.idle": "2022-02-12T13:46:44.615767Z",
     "shell.execute_reply": "2022-02-12T13:46:44.615369Z",
     "shell.execute_reply.started": "2022-02-12T10:02:19.163968Z"
    },
    "papermill": {
     "duration": 0.039869,
     "end_time": "2022-02-12T13:46:44.615905",
     "exception": false,
     "start_time": "2022-02-12T13:46:44.576036",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# import gridsearchcv\n",
    "from sklearn.model_selection import RandomizedSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "5bd15106",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:44.686536Z",
     "iopub.status.busy": "2022-02-12T13:46:44.685741Z",
     "iopub.status.idle": "2022-02-12T13:46:44.689935Z",
     "shell.execute_reply": "2022-02-12T13:46:44.689508Z",
     "shell.execute_reply.started": "2022-02-12T13:38:52.851529Z"
    },
    "papermill": {
     "duration": 0.040258,
     "end_time": "2022-02-12T13:46:44.690058",
     "exception": false,
     "start_time": "2022-02-12T13:46:44.649800",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "xgb_reg_cv = XGBRegressor(random_state=42)\n",
    "\n",
    "params = {\n",
    "    'learning_rate': [0.005, 0.01, 0.02, 0.03, 0.04, 0.05],\n",
    "    'max_depth': [3, 5, 8, 10, 12],\n",
    "    'n_estimators': [100, 250, 400, 500, 700, 800]\n",
    "}\n",
    "\n",
    "# # grid search\n",
    "# grid_model = GridSearchCV(estimator=xgb_reg_cv, param_grid=params, n_jobs=-1, cv=4,\n",
    "#                           verbose=2, return_train_score=True, scoring=['r2'])\n",
    "\n",
    "# randomized search\n",
    "random_model = RandomizedSearchCV(estimator=xgb_reg_cv, param_distributions=params, \n",
    "                                  n_jobs=-1, cv=4,\n",
    "                          verbose=2, return_train_score=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "1bf13f26",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:46:44.765575Z",
     "iopub.status.busy": "2022-02-12T13:46:44.764735Z",
     "iopub.status.idle": "2022-02-12T13:46:44.767067Z",
     "shell.execute_reply": "2022-02-12T13:46:44.766525Z",
     "shell.execute_reply.started": "2022-02-12T13:38:53.894218Z"
    },
    "papermill": {
     "duration": 0.042813,
     "end_time": "2022-02-12T13:46:44.767177",
     "exception": false,
     "start_time": "2022-02-12T13:46:44.724364",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 4 folds for each of 10 candidates, totalling 40 fits\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomizedSearchCV(cv=4,\n",
       "                   estimator=XGBRegressor(base_score=None, booster=None,\n",
       "                                          colsample_bylevel=None,\n",
       "                                          colsample_bynode=None,\n",
       "                                          colsample_bytree=None,\n",
       "                                          enable_categorical=False, gamma=None,\n",
       "                                          gpu_id=None, importance_type=None,\n",
       "                                          interaction_constraints=None,\n",
       "                                          learning_rate=None,\n",
       "                                          max_delta_step=None, max_depth=None,\n",
       "                                          min_child_weight=None, missing=nan,\n",
       "                                          monotone_constraints=...\n",
       "                                          num_parallel_tree=None,\n",
       "                                          predictor=None, random_state=42,\n",
       "                                          reg_alpha=None, reg_lambda=None,\n",
       "                                          scale_pos_weight=None, subsample=None,\n",
       "                                          tree_method=None,\n",
       "                                          validate_parameters=None,\n",
       "                                          verbosity=None),\n",
       "                   n_jobs=-1,\n",
       "                   param_distributions={'learning_rate': [0.005, 0.01, 0.02,\n",
       "                                                          0.03, 0.04, 0.05],\n",
       "                                        'max_depth': [3, 5, 8, 10, 12],\n",
       "                                        'n_estimators': [100, 250, 400, 500,\n",
       "                                                         700, 800]},\n",
       "                   return_train_score=True, verbose=2)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# # fit grid model\n",
    "# grid_model.fit(X_train, y_train)\n",
    "\n",
    "# fit random model\n",
    "random_model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "e70df31a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'n_estimators': 700, 'max_depth': 5, 'learning_rate': 0.05}"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_model.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "9a149f35",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-12T13:37:41.953836Z",
     "iopub.status.busy": "2022-02-12T13:37:41.953282Z",
     "iopub.status.idle": "2022-02-12T13:37:41.957871Z",
     "shell.execute_reply": "2022-02-12T13:37:41.956978Z",
     "shell.execute_reply.started": "2022-02-12T13:37:41.953802Z"
    },
    "papermill": {
     "duration": 0.034076,
     "end_time": "2022-02-12T13:46:44.836135",
     "exception": false,
     "start_time": "2022-02-12T13:46:44.802059",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
       "             colsample_bynode=1, colsample_bytree=1, enable_categorical=False,\n",
       "             gamma=0, gpu_id=-1, importance_type=None,\n",
       "             interaction_constraints='', learning_rate=0.05, max_delta_step=0,\n",
       "             max_depth=5, min_child_weight=1, missing=nan,\n",
       "             monotone_constraints='()', n_estimators=700, n_jobs=8,\n",
       "             num_parallel_tree=1, predictor='auto', random_state=42,\n",
       "             reg_alpha=0, reg_lambda=1, scale_pos_weight=1, subsample=1,\n",
       "             tree_method='exact', validate_parameters=1, verbosity=None)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_model_xgb = random_model.best_estimator_\n",
    "best_model_xgb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "2bc431bd",
   "metadata": {
    "papermill": {
     "duration": 0.033045,
     "end_time": "2022-02-12T13:46:44.969368",
     "exception": false,
     "start_time": "2022-02-12T13:46:44.936323",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train score: 0.3752077290138349\n",
      "Test Score: 0.35364993096676844\n"
     ]
    }
   ],
   "source": [
    "# prediction\n",
    "y_train_pred = best_model_xgb.predict(X_train)\n",
    "y_test_pred = best_model_xgb.predict(X_test)\n",
    "\n",
    "# evaluation\n",
    "print('Train score:', r2_score(y_train, y_train_pred))\n",
    "print('Test Score:',r2_score(y_test, y_test_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51e938b2",
   "metadata": {},
   "source": [
    "Model with the above parameters present in `best_model_xgb` has better test score.\n",
    "\n",
    "Looks better than the previous 1st model of XGB, so lets proceed in predicting scores for test data set."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "592b0344",
   "metadata": {},
   "source": [
    "## RandomForest Regessor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "b133eb44",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import libraries\n",
    "from sklearn.ensemble import RandomForestRegressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "f9220610",
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_reg = RandomForestRegressor(random_state=42)\n",
    "\n",
    "params = {\n",
    "    'max_depth': [4, 5, 8, 10, 12],\n",
    "    'min_samples_leaf': [50, 100, 150, 200],\n",
    "    'n_estimators': [300, 500, 600, 800, 900]\n",
    "}\n",
    "\n",
    "random_model_rf = RandomizedSearchCV(estimator=rf_reg, param_distributions=params, \n",
    "                                  n_jobs=-1, cv=4,\n",
    "                          verbose=2, return_train_score=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "e3a2faef",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 4 folds for each of 10 candidates, totalling 40 fits\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomizedSearchCV(cv=4, estimator=RandomForestRegressor(random_state=42),\n",
       "                   n_jobs=-1,\n",
       "                   param_distributions={'max_depth': [4, 5, 8, 10, 12],\n",
       "                                        'min_samples_leaf': [50, 100, 150, 200],\n",
       "                                        'n_estimators': [300, 500, 600, 800,\n",
       "                                                         900]},\n",
       "                   return_train_score=True, verbose=2)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fit random model rf\n",
    "random_model_rf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "6dd55ec6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_estimators': 600, 'min_samples_leaf': 50, 'max_depth': 10}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomForestRegressor(max_depth=10, min_samples_leaf=50, n_estimators=600,\n",
       "                      random_state=42)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(random_model_rf.best_params_)\n",
    "best_model = random_model_rf.best_estimator_\n",
    "best_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "23899e73",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train score: 0.3507882359070732\n",
      "Test Score: 0.3416063796796587\n"
     ]
    }
   ],
   "source": [
    "# prediction\n",
    "y_train_pred = best_model.predict(X_train)\n",
    "y_test_pred = best_model.predict(X_test)\n",
    "\n",
    "# evaluation\n",
    "print('Train score:', r2_score(y_train, y_train_pred))\n",
    "print('Test Score:',r2_score(y_test, y_test_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e469b6bc",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "df69ec3d",
   "metadata": {},
   "source": [
    "## DecisionTree Regressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "9cfe9646",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import libraries\n",
    "from sklearn.tree import DecisionTreeRegressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "ccac12ca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeRegressor(random_state=42)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt_reg = DecisionTreeRegressor(random_state=42)\n",
    "\n",
    "# fit base DecisionTree model\n",
    "dt_reg.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "8aef78d0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train score: 0.43858813521702833\n",
      "Test Score: 0.29816839125241024\n"
     ]
    }
   ],
   "source": [
    "# prediction\n",
    "y_train_pred = dt_reg.predict(X_train)\n",
    "y_test_pred = dt_reg.predict(X_test)\n",
    "\n",
    "# evaluation\n",
    "print('Train score:', r2_score(y_train, y_train_pred))\n",
    "print('Test Score:',r2_score(y_test, y_test_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86f64193",
   "metadata": {},
   "source": [
    "### Hyperparameter tuning with GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "8eb38b13",
   "metadata": {},
   "outputs": [],
   "source": [
    "dt_reg = DecisionTreeRegressor(random_state=42)\n",
    "\n",
    "params = {\n",
    "    \"criterion\": [\"mse\", \"mae\"],\n",
    "    \"min_samples_split\": [20, 40, 50, 100],\n",
    "    \"max_depth\": [2, 6, 8, 12, 15],\n",
    "    \"min_samples_leaf\": [20, 40, 100],\n",
    "}\n",
    "\n",
    "grid_model_dt = GridSearchCV(estimator=dt_reg, param_grid=params,\n",
    "                         cv=4, n_jobs=-1, verbose=2, return_train_score=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "3bb42471",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 4 folds for each of 120 candidates, totalling 480 fits\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=4, estimator=DecisionTreeRegressor(random_state=42), n_jobs=-1,\n",
       "             param_grid={'criterion': ['mse', 'mae'],\n",
       "                         'max_depth': [2, 6, 8, 12, 15],\n",
       "                         'min_samples_leaf': [20, 40, 100],\n",
       "                         'min_samples_split': [20, 40, 50, 100]},\n",
       "             return_train_score=True, verbose=2)"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fit grid_model\n",
    "grid_model_dt.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "54c24cf6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'criterion': 'mse', 'max_depth': 12, 'min_samples_leaf': 20, 'min_samples_split': 100}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "DecisionTreeRegressor(max_depth=12, min_samples_leaf=20, min_samples_split=100,\n",
       "                      random_state=42)"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(grid_model_dt.best_params_)\n",
    "best_model_dt = grid_model_dt.best_estimator_\n",
    "best_model_dt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "e7e3a60e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train score: 0.36354033994012336\n",
      "Test Score: 0.3436947450183483\n"
     ]
    }
   ],
   "source": [
    "# prediction\n",
    "y_train_pred = best_model_dt.predict(X_train)\n",
    "y_test_pred = best_model_dt.predict(X_test)\n",
    "\n",
    "# evaluation\n",
    "print('Train score:', r2_score(y_train, y_train_pred))\n",
    "print('Test Score:',r2_score(y_test, y_test_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c163526c",
   "metadata": {},
   "source": [
    "### Till now **`best_model_xgb`** has given highest test score of **0.35364993096676844**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d66dae24",
   "metadata": {},
   "source": [
    "Lets predict score for Test data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab7dc253",
   "metadata": {
    "papermill": {
     "duration": 0.033427,
     "end_time": "2022-02-12T13:46:45.103511",
     "exception": false,
     "start_time": "2022-02-12T13:46:45.070084",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Test Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee7abcb9",
   "metadata": {},
   "source": [
    "**import test data and do preprocessing techniques as done for train data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "f41c7491",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of Test data: (11121, 9)\n",
      "Columns present in Test data: Index(['row_id', 'user_id', 'category_id', 'video_id', 'age', 'gender',\n",
      "       'profession', 'followers', 'views'],\n",
      "      dtype='object')\n",
      "[4.1037755 3.7482507 2.7046568 3.8991432 2.6361735 3.9714184 3.7615764\n",
      " 3.8025105 2.6055307 4.040778 ]\n"
     ]
    }
   ],
   "source": [
    "# import data\n",
    "test_data = pd.read_csv('data/test_1zqHu22.csv')\n",
    "\n",
    "# info of data\n",
    "print(\"Shape of Test data:\", test_data.shape)\n",
    "print(\"Columns present in Test data:\",test_data.columns)\n",
    "\n",
    "# preprocess the data\n",
    "new_test_data = test_data.drop(['row_id', 'user_id','category_id','video_id'], axis=1)\n",
    "\n",
    "# encoding categorical columns\n",
    "cat_cols = ['gender', 'profession']\n",
    "for col in cat_cols:\n",
    "    new_test_data[col] = encoder.fit_transform(new_test_data[col])\n",
    "\n",
    "# standardize the data\n",
    "all_test_cols = new_test_data.columns\n",
    "new_test_data[all_test_cols] = scaler.transform(new_test_data[all_test_cols])\n",
    "\n",
    "# predict the engagement score from `best_model_xgb`\n",
    "target = best_model_xgb.predict(new_test_data)\n",
    "\n",
    "print(target[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "5174236d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>row_id</th>\n",
       "      <th>engagement_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>89198</td>\n",
       "      <td>4.103776</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>89199</td>\n",
       "      <td>3.748251</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>89200</td>\n",
       "      <td>2.704657</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>89201</td>\n",
       "      <td>3.899143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>89202</td>\n",
       "      <td>2.636173</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   row_id  engagement_score\n",
       "0   89198          4.103776\n",
       "1   89199          3.748251\n",
       "2   89200          2.704657\n",
       "3   89201          3.899143\n",
       "4   89202          2.636173"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# submission dataframe\n",
    "submission_df = pd.DataFrame({'row_id': test_data.row_id.values,\n",
    "                            'engagement_score': target})\n",
    "\n",
    "submission_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "b87f7892",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File saved successfully!!\n"
     ]
    }
   ],
   "source": [
    "# save to sumission_csv\n",
    "# submission_df.to_csv('submission.csv', index=False)\n",
    "# print(\"File saved successfully!!\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 25.976334,
   "end_time": "2022-02-12T13:46:46.248069",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2022-02-12T13:46:20.271735",
   "version": "2.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}