Bayesian_optimization_case_study.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WNK7vbHo-KYU"
   },
   "source": [
    "## Bayesian methods of hyperparameter optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BlFdvPwF-KYW"
   },
   "source": [
    "In addition to the random search and the grid search methods for selecting optimal hyperparameters, we can use Bayesian methods of probabilities to select the optimal hyperparameters for an algorithm.\n",
    "\n",
    "In this case study, we will be using the BayesianOptimization library to perform hyperparmater tuning. This library has very good documentation which you can find here: https://github.com/fmfn/BayesianOptimization\n",
    "\n",
    "You will need to install the Bayesian optimization module. Running a cell with an exclamation point in the beginning of the command will run it as a shell command — please do this to install this module from our notebook in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Pssx080d-Ulf"
   },
   "outputs": [],
   "source": [
    "#! pip install bayesian-optimization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-22T16:39:09.312682Z",
     "start_time": "2019-04-22T16:39:09.309208Z"
    },
    "_kg_hide-input": true,
    "colab": {},
    "colab_type": "code",
    "id": "l9nfFTyj-KYY"
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import lightgbm\n",
    "from bayes_opt import BayesianOptimization\n",
    "from catboost import CatBoostClassifier, cv, Pool"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "id": "D16Dquw1AAK0",
    "outputId": "44167587-f22e-4bf5-a816-e2bcfdc6c4ee"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['.DS_Store',\n",
       " '.ipynb_checkpoints',\n",
       " 'Bayesian_optimization_case_study.ipynb',\n",
       " 'flight_delays_test.csv.zip',\n",
       " 'flight_delays_train.csv.zip']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "os.listdir()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-22T14:48:15.929012Z",
     "start_time": "2019-04-22T14:48:15.926574Z"
    },
    "colab_type": "text",
    "id": "AkBt3yds-KYu"
   },
   "source": [
    "## How does Bayesian optimization work?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "E1kyBCUs-KYv"
   },
   "source": [
    "Bayesian optimization works by constructing a posterior distribution of functions (Gaussian process) that best describes the function you want to optimize. As the number of observations grows, the posterior distribution improves, and the algorithm becomes more certain of which regions in parameter space are worth exploring and which are not, as seen in the picture below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gAdHF72R-KYw"
   },
   "source": [
    "<img src=\"https://github.com/fmfn/BayesianOptimization/blob/master/examples/bo_example.png?raw=true\" />\n",
    "As you iterate over and over, the algorithm balances its needs of exploration and exploitation while taking into account what it knows about the target function. At each step, a Gaussian Process is fitted to the known samples (points previously explored), and the posterior distribution, combined with an exploration strategy (such as UCB — aka Upper Confidence Bound), or EI (Expected Improvement). This process is used to determine the next point that should be explored (see the gif below).\n",
    "<img src=\"https://github.com/fmfn/BayesianOptimization/raw/master/examples/bayesian_optimization.gif\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RTP8KUlLoYzu"
   },
   "source": [
    "## Let's look at a simple example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "crpPqKdC-KYx"
   },
   "source": [
    "The first step is to create an optimizer. It uses two items:\n",
    "* function to optimize\n",
    "* bounds of parameters\n",
    "\n",
    "The function is the procedure that counts metrics of our model quality. The important thing is that our optimization will maximize the value on function. Smaller metrics are best. Hint: don't forget to use negative metric values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "e09ciF8gpTfr"
   },
   "source": [
    "Here we define our simple function we want to optimize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ofwvnfEwo5mG"
   },
   "outputs": [],
   "source": [
    "def simple_func(a, b):\n",
    "    return a + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XCGsdciCpeI3"
   },
   "source": [
    "Now, we define our bounds of the parameters to optimize, within the Bayesian optimizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4jLYW2qnpOFr"
   },
   "outputs": [],
   "source": [
    "optimizer = BayesianOptimization(\n",
    "    simple_func,\n",
    "    {'a': (1, 3),\n",
    "    'b': (4, 7)})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dg6LdYx8pq2T"
   },
   "source": [
    "These are the main parameters of this function:\n",
    "\n",
    "* **n_iter:** This is how many steps of Bayesian optimization you want to perform. The more steps, the more likely you are to find a good maximum.\n",
    "\n",
    "* **init_points:** This is how many steps of random exploration you want to perform. Random exploration can help by diversifying the exploration space."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "i-GKMJ1uqMYv"
   },
   "source": [
    "Let's run an example where we use the optimizer to find the best values to maximize the target value for a and b given the inputs of 3 and 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 153
    },
    "colab_type": "code",
    "id": "Oy44Ro7wqNat",
    "outputId": "9cc64d54-b1e6-46d1-dc29-4c0039a1c72d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|   iter    |  target   |     a     |     b     |\n",
      "-------------------------------------------------\n",
      "| \u001b[0m 1       \u001b[0m | \u001b[0m 8.284   \u001b[0m | \u001b[0m 2.674   \u001b[0m | \u001b[0m 5.61    \u001b[0m |\n",
      "| \u001b[0m 2       \u001b[0m | \u001b[0m 6.607   \u001b[0m | \u001b[0m 1.004   \u001b[0m | \u001b[0m 5.603   \u001b[0m |\n",
      "| \u001b[95m 3       \u001b[0m | \u001b[95m 9.151   \u001b[0m | \u001b[95m 2.999   \u001b[0m | \u001b[95m 6.152   \u001b[0m |\n",
      "| \u001b[95m 4       \u001b[0m | \u001b[95m 10.0    \u001b[0m | \u001b[95m 3.0     \u001b[0m | \u001b[95m 7.0     \u001b[0m |\n",
      "| \u001b[0m 5       \u001b[0m | \u001b[0m 9.171   \u001b[0m | \u001b[0m 2.171   \u001b[0m | \u001b[0m 7.0     \u001b[0m |\n",
      "=================================================\n"
     ]
    }
   ],
   "source": [
    "optimizer.maximize(3,2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tyKFMF2Hq2Sx"
   },
   "source": [
    "Great, now let's print the best parameters and the associated maximized target."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "_H6DixyfscV_",
    "outputId": "fd0c35d7-e30d-4d30-9ab2-12c0fa837971"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'a': 3.0, 'b': 7.0}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "10.0"
      ]
     },
     "execution_count": 49,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(optimizer.max['params']);optimizer.max['target']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tQ1T1V6Mspi4"
   },
   "source": [
    "## Test it on real data using the Light GBM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "y_oGwREZkm4h"
   },
   "source": [
    "The dataset we will be working with is the famous flight departures dataset. Our modeling goal will be to predict if a flight departure is going to be delayed by 15 minutes based on the other attributes in our dataset. As part of this modeling exercise, we will use Bayesian hyperparameter optimization to identify the best parameters for our model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "abYSagjQANDZ"
   },
   "source": [
    "**<font color='teal'> You can load the zipped csv files just as you would regular csv files using Pandas read_csv. In the next cell load the train and test data into two seperate dataframes. </font>**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EWKBApVuAeJe"
   },
   "outputs": [],
   "source": [
    "train_df = pd.read_csv('flight_delays_train.csv.zip')\n",
    "test_df = pd.read_csv('flight_delays_test.csv.zip')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OapNcT9Eikis"
   },
   "source": [
    "**<font color='teal'> Print the top five rows of the train dataframe and review the columns in the data. </font>**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "__4cXZ8iiYaC",
    "outputId": "8718ad4b-8955-486c-9ae8-1dee6aa6c2fb"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Month</th>\n",
       "      <th>DayofMonth</th>\n",
       "      <th>DayOfWeek</th>\n",
       "      <th>DepTime</th>\n",
       "      <th>UniqueCarrier</th>\n",
       "      <th>Origin</th>\n",
       "      <th>Dest</th>\n",
       "      <th>Distance</th>\n",
       "      <th>dep_delayed_15min</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>c-8</td>\n",
       "      <td>c-21</td>\n",
       "      <td>c-7</td>\n",
       "      <td>1934</td>\n",
       "      <td>AA</td>\n",
       "      <td>ATL</td>\n",
       "      <td>DFW</td>\n",
       "      <td>732</td>\n",
       "      <td>N</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>c-4</td>\n",
       "      <td>c-20</td>\n",
       "      <td>c-3</td>\n",
       "      <td>1548</td>\n",
       "      <td>US</td>\n",
       "      <td>PIT</td>\n",
       "      <td>MCO</td>\n",
       "      <td>834</td>\n",
       "      <td>N</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>c-9</td>\n",
       "      <td>c-2</td>\n",
       "      <td>c-5</td>\n",
       "      <td>1422</td>\n",
       "      <td>XE</td>\n",
       "      <td>RDU</td>\n",
       "      <td>CLE</td>\n",
       "      <td>416</td>\n",
       "      <td>N</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>c-11</td>\n",
       "      <td>c-25</td>\n",
       "      <td>c-6</td>\n",
       "      <td>1015</td>\n",
       "      <td>OO</td>\n",
       "      <td>DEN</td>\n",
       "      <td>MEM</td>\n",
       "      <td>872</td>\n",
       "      <td>N</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>c-10</td>\n",
       "      <td>c-7</td>\n",
       "      <td>c-6</td>\n",
       "      <td>1828</td>\n",
       "      <td>WN</td>\n",
       "      <td>MDW</td>\n",
       "      <td>OMA</td>\n",
       "      <td>423</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Month DayofMonth DayOfWeek  DepTime UniqueCarrier Origin Dest  Distance  \\\n",
       "0   c-8       c-21       c-7     1934            AA    ATL  DFW       732   \n",
       "1   c-4       c-20       c-3     1548            US    PIT  MCO       834   \n",
       "2   c-9        c-2       c-5     1422            XE    RDU  CLE       416   \n",
       "3  c-11       c-25       c-6     1015            OO    DEN  MEM       872   \n",
       "4  c-10        c-7       c-6     1828            WN    MDW  OMA       423   \n",
       "\n",
       "  dep_delayed_15min  \n",
       "0                 N  \n",
       "1                 N  \n",
       "2                 N  \n",
       "3                 N  \n",
       "4                 Y  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UxGBsPQhffgd"
   },
   "source": [
    "**<font color='teal'> Use the describe function to review the numeric columns in the train dataframe. </font>**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 297
    },
    "colab_type": "code",
    "id": "_bRRKG3DAtae",
    "outputId": "7cfb9975-ec97-422c-abbd-98923a0b7aec"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DepTime</th>\n",
       "      <th>Distance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>100000.000000</td>\n",
       "      <td>100000.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1341.523880</td>\n",
       "      <td>729.39716</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>476.378445</td>\n",
       "      <td>574.61686</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>30.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>931.000000</td>\n",
       "      <td>317.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>1330.000000</td>\n",
       "      <td>575.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1733.000000</td>\n",
       "      <td>957.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>2534.000000</td>\n",
       "      <td>4962.00000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             DepTime      Distance\n",
       "count  100000.000000  100000.00000\n",
       "mean     1341.523880     729.39716\n",
       "std       476.378445     574.61686\n",
       "min         1.000000      30.00000\n",
       "25%       931.000000     317.00000\n",
       "50%      1330.000000     575.00000\n",
       "75%      1733.000000     957.00000\n",
       "max      2534.000000    4962.00000"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "i6k-_fI5Aiyh"
   },
   "source": [
    "Notice, `DepTime` is the departure time in a numeric representation in 2400 hours. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gtZS4-hrlQah"
   },
   "source": [
    " **<font color='teal'>The response variable is 'dep_delayed_15min' which is a categorical column, so we need to map the Y for yes and N for no values to 1 and 0. Run the code in the next cell to do this.</font>**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-22T15:38:42.677690Z",
     "start_time": "2019-04-22T15:38:42.481963Z"
    },
    "colab": {},
    "colab_type": "code",
    "id": "yRlOTbnW-KYc"
   },
   "outputs": [],
   "source": [
    "#train_df = train_df[train_df.DepTime <= 2400].copy()\n",
    "y_train = train_df['dep_delayed_15min'].map({'Y': 1, 'N': 0}).values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "z3WPkFQO9uo9"
   },
   "source": [
    "## Feature Engineering\n",
    "Use these defined functions to create additional features for the model. Run the cell to add the functions to your workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cXqsqz5W9t3r"
   },
   "outputs": [],
   "source": [
    "def label_enc(df_column):\n",
    "    df_column = LabelEncoder().fit_transform(df_column)\n",
    "    return df_column\n",
    "\n",
    "def make_harmonic_features_sin(value, period=2400):\n",
    "    value *= 2 * np.pi / period \n",
    "    return np.sin(value)\n",
    "\n",
    "def make_harmonic_features_cos(value, period=2400):\n",
    "    value *= 2 * np.pi / period \n",
    "    return np.cos(value)\n",
    "\n",
    "def feature_eng(df):\n",
    "    df['flight'] = df['Origin']+df['Dest']\n",
    "    df['Month'] = df.Month.map(lambda x: x.split('-')[-1]).astype('int32')\n",
    "    df['DayofMonth'] = df.DayofMonth.map(lambda x: x.split('-')[-1]).astype('uint8')\n",
    "    df['begin_of_month'] = (df['DayofMonth'] < 10).astype('uint8')\n",
    "    df['midddle_of_month'] = ((df['DayofMonth'] >= 10)&(df['DayofMonth'] < 20)).astype('uint8')\n",
    "    df['end_of_month'] = (df['DayofMonth'] >= 20).astype('uint8')\n",
    "    df['DayOfWeek'] = df.DayOfWeek.map(lambda x: x.split('-')[-1]).astype('uint8')\n",
    "    df['hour'] = df.DepTime.map(lambda x: x/100).astype('int32')\n",
    "    df['morning'] = df['hour'].map(lambda x: 1 if (x <= 11)& (x >= 7) else 0).astype('uint8')\n",
    "    df['day'] = df['hour'].map(lambda x: 1 if (x >= 12) & (x <= 18) else 0).astype('uint8')\n",
    "    df['evening'] = df['hour'].map(lambda x: 1 if (x >= 19) & (x <= 23) else 0).astype('uint8')\n",
    "    df['night'] = df['hour'].map(lambda x: 1 if (x >= 0) & (x <= 6) else 0).astype('int32')\n",
    "    df['winter'] = df['Month'].map(lambda x: x in [12, 1, 2]).astype('int32')\n",
    "    df['spring'] = df['Month'].map(lambda x: x in [3, 4, 5]).astype('int32')\n",
    "    df['summer'] = df['Month'].map(lambda x: x in [6, 7, 8]).astype('int32')\n",
    "    df['autumn'] = df['Month'].map(lambda x: x in [9, 10, 11]).astype('int32')\n",
    "    df['holiday'] = (df['DayOfWeek'] >= 5).astype(int) \n",
    "    df['weekday'] = (df['DayOfWeek'] < 5).astype(int)\n",
    "    df['airport_dest_per_month'] = df.groupby(['Dest', 'Month'])['Dest'].transform('count')\n",
    "    df['airport_origin_per_month'] = df.groupby(['Origin', 'Month'])['Origin'].transform('count')\n",
    "    df['airport_dest_count'] = df.groupby(['Dest'])['Dest'].transform('count')\n",
    "    df['airport_origin_count'] = df.groupby(['Origin'])['Origin'].transform('count')\n",
    "    df['carrier_count'] = df.groupby(['UniqueCarrier'])['Dest'].transform('count')\n",
    "    df['carrier_count_per month'] = df.groupby(['UniqueCarrier', 'Month'])['Dest'].transform('count')\n",
    "    df['deptime_cos'] = df['DepTime'].map(make_harmonic_features_cos)\n",
    "    df['deptime_sin'] = df['DepTime'].map(make_harmonic_features_sin)\n",
    "    df['flightUC'] = df['flight']+df['UniqueCarrier']\n",
    "    df['DestUC'] = df['Dest']+df['UniqueCarrier']\n",
    "    df['OriginUC'] = df['Origin']+df['UniqueCarrier']\n",
    "    return df.drop('DepTime', axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-BYbxXpU-FGE"
   },
   "source": [
    "Concatenate the training and testing dataframes.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Cj6bfSNw_RAf"
   },
   "outputs": [],
   "source": [
    "full_df = pd.concat([train_df.drop('dep_delayed_15min', axis=1), test_df])\n",
    "full_df = feature_eng(full_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GSO8JbfM_W-F"
   },
   "source": [
    "Apply the earlier defined feature engineering functions to the full dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "x6RfAINftjwi"
   },
   "outputs": [],
   "source": [
    "for column in ['UniqueCarrier', 'Origin', 'Dest','flight',  'flightUC', 'DestUC', 'OriginUC']:\n",
    "    full_df[column] = label_enc(full_df[column])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IJAw1RGB_ZuM"
   },
   "source": [
    "\n",
    "Split the new full dataframe into X_train and X_test. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "15cPtQU5tjfz"
   },
   "outputs": [],
   "source": [
    "X_train = full_df[:train_df.shape[0]]\n",
    "X_test = full_df[train_df.shape[0]:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "umfAw-9JErLV"
   },
   "source": [
    "Create a list of the categorical features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-22T14:31:58.412296Z",
     "start_time": "2019-04-22T14:31:58.409088Z"
    },
    "colab": {},
    "colab_type": "code",
    "id": "5ibeVyNb-KZI"
   },
   "outputs": [],
   "source": [
    "categorical_features = ['Month',  'DayOfWeek', 'UniqueCarrier', 'Origin', 'Dest','flight',  'flightUC', 'DestUC', 'OriginUC']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NzMIsMPIETVk"
   },
   "source": [
    "Let's build a light GBM model to test the bayesian optimizer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-22T15:18:04.466965Z",
     "start_time": "2019-04-22T15:18:04.457992Z"
    },
    "colab_type": "text",
    "id": "2hfm1i5G-KZH"
   },
   "source": [
    "### [LightGBM](https://lightgbm.readthedocs.io/en/latest/) is a gradient boosting framework that uses tree-based learning algorithms. It is designed to be distributed and efficient with the following advantages:\n",
    "\n",
    "* Faster training speed and higher efficiency.\n",
    "* Lower memory usage.\n",
    "* Better accuracy.\n",
    "* Support of parallel and GPU learning.\n",
    "* Capable of handling large-scale data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jf-3F2Wg-KZL"
   },
   "source": [
    "First, we define the function we want to maximize and that will count cross-validation metrics of lightGBM for our parameters.\n",
    "\n",
    "Some params such as num_leaves, max_depth, min_child_samples, min_data_in_leaf should be integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-22T15:40:14.034265Z",
     "start_time": "2019-04-22T15:40:14.027868Z"
    },
    "colab": {},
    "colab_type": "code",
    "id": "LyUJBhGX-KZM"
   },
   "outputs": [],
   "source": [
    "def lgb_eval(num_leaves,max_depth,lambda_l2,lambda_l1,min_child_samples, min_data_in_leaf):\n",
    "    params = {\n",
    "        \"objective\" : \"binary\",\n",
    "        \"metric\" : \"auc\", \n",
    "        'is_unbalance': True,\n",
    "        \"num_leaves\" : int(num_leaves),\n",
    "        \"max_depth\" : int(max_depth),\n",
    "        \"lambda_l2\" : lambda_l2,\n",
    "        \"lambda_l1\" : lambda_l1,\n",
    "        \"num_threads\" : 20,\n",
    "        \"min_child_samples\" : int(min_child_samples),\n",
    "        'min_data_in_leaf': int(min_data_in_leaf),\n",
    "        \"learning_rate\" : 0.03,\n",
    "        \"subsample_freq\" : 5,\n",
    "        \"bagging_seed\" : 42,\n",
    "        \"verbosity\" : -1\n",
    "    }\n",
    "    lgtrain = lightgbm.Dataset(X_train, y_train,categorical_feature=categorical_features)\n",
    "    cv_result = lightgbm.cv(params,\n",
    "                       lgtrain,\n",
    "                       1000,\n",
    "                       early_stopping_rounds=100,\n",
    "                       stratified=True,\n",
    "                       nfold=3)\n",
    "    return cv_result['auc-mean'][-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FJwqBhdeF11Q"
   },
   "source": [
    "Apply the Bayesian optimizer to the function we created in the previous step to identify the best hyperparameters. We will run 10 iterations and set init_points = 2.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-22T15:48:04.682447Z",
     "start_time": "2019-04-22T15:40:14.641634Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "JheCOkUE-KZP",
    "outputId": "8f37ee51-885d-44e4-cdcd-ceb7abd58b61"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|   iter    |  target   | lambda_l1 | lambda_l2 | max_depth | min_ch... | min_da... | num_le... |\n",
      "-------------------------------------------------------------------------------------------------\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=266, min_child_samples=6671 will be ignored. Current value: min_data_in_leaf=266\n",
      "| \u001b[0m 1       \u001b[0m | \u001b[0m 0.7202  \u001b[0m | \u001b[0m 0.02296 \u001b[0m | \u001b[0m 0.02289 \u001b[0m | \u001b[0m 57.87   \u001b[0m | \u001b[0m 6.671e+0\u001b[0m | \u001b[0m 266.2   \u001b[0m | \u001b[0m 3.024e+0\u001b[0m |\n",
      "| \u001b[95m 2       \u001b[0m | \u001b[95m 0.7419  \u001b[0m | \u001b[95m 0.01685 \u001b[0m | \u001b[95m 0.008582\u001b[0m | \u001b[95m 8.326   \u001b[0m | \u001b[95m 945.2   \u001b[0m | \u001b[95m 1.143e+0\u001b[0m | \u001b[95m 1.317e+0\u001b[0m |\n",
      "| \u001b[0m 3       \u001b[0m | \u001b[0m 0.7279  \u001b[0m | \u001b[0m 0.02419 \u001b[0m | \u001b[0m 0.04197 \u001b[0m | \u001b[0m 24.57   \u001b[0m | \u001b[0m 4.923e+0\u001b[0m | \u001b[0m 866.0   \u001b[0m | \u001b[0m 3.085e+0\u001b[0m |\n",
      "| \u001b[95m 4       \u001b[0m | \u001b[95m 0.7432  \u001b[0m | \u001b[95m 0.0407  \u001b[0m | \u001b[95m 0.004093\u001b[0m | \u001b[95m 58.11   \u001b[0m | \u001b[95m 227.8   \u001b[0m | \u001b[95m 1.881e+0\u001b[0m | \u001b[95m 3.911e+0\u001b[0m |\n",
      "| \u001b[0m 5       \u001b[0m | \u001b[0m 0.7222  \u001b[0m | \u001b[0m 0.009338\u001b[0m | \u001b[0m 0.03271 \u001b[0m | \u001b[0m 51.68   \u001b[0m | \u001b[0m 187.2   \u001b[0m | \u001b[0m 131.8   \u001b[0m | \u001b[0m 3.616e+0\u001b[0m |\n",
      "| \u001b[0m 6       \u001b[0m | \u001b[0m 0.7225  \u001b[0m | \u001b[0m 0.01155 \u001b[0m | \u001b[0m 0.008904\u001b[0m | \u001b[0m 15.99   \u001b[0m | \u001b[0m 5.99e+03\u001b[0m | \u001b[0m 524.6   \u001b[0m | \u001b[0m 3.602e+0\u001b[0m |\n",
      "| \u001b[95m 7       \u001b[0m | \u001b[95m 0.7438  \u001b[0m | \u001b[95m 0.01099 \u001b[0m | \u001b[95m 0.01539 \u001b[0m | \u001b[95m 14.85   \u001b[0m | \u001b[95m 562.5   \u001b[0m | \u001b[95m 1.993e+0\u001b[0m | \u001b[95m 2.552e+0\u001b[0m |\n",
      "| \u001b[0m 8       \u001b[0m | \u001b[0m 0.7429  \u001b[0m | \u001b[0m 0.04717 \u001b[0m | \u001b[0m 0.01175 \u001b[0m | \u001b[0m 37.31   \u001b[0m | \u001b[0m 3.685e+0\u001b[0m | \u001b[0m 1.59e+03\u001b[0m | \u001b[0m 3.454e+0\u001b[0m |\n",
      "| \u001b[0m 9       \u001b[0m | \u001b[0m 0.7425  \u001b[0m | \u001b[0m 0.01966 \u001b[0m | \u001b[0m 0.04689 \u001b[0m | \u001b[0m 52.59   \u001b[0m | \u001b[0m 7.23e+03\u001b[0m | \u001b[0m 1.233e+0\u001b[0m | \u001b[0m 1.416e+0\u001b[0m |\n",
      "| \u001b[0m 10      \u001b[0m | \u001b[0m 0.7434  \u001b[0m | \u001b[0m 0.006001\u001b[0m | \u001b[0m 0.01215 \u001b[0m | \u001b[0m 47.52   \u001b[0m | \u001b[0m 238.6   \u001b[0m | \u001b[0m 1.885e+0\u001b[0m | \u001b[0m 3.91e+03\u001b[0m |\n",
      "| \u001b[0m 11      \u001b[0m | \u001b[0m 0.7435  \u001b[0m | \u001b[0m 0.006872\u001b[0m | \u001b[0m 0.009372\u001b[0m | \u001b[0m 31.61   \u001b[0m | \u001b[0m 1.935e+0\u001b[0m | \u001b[0m 1.998e+0\u001b[0m | \u001b[0m 3.899e+0\u001b[0m |\n",
      "| \u001b[0m 12      \u001b[0m | \u001b[0m 0.7437  \u001b[0m | \u001b[0m 0.03627 \u001b[0m | \u001b[0m 0.02906 \u001b[0m | \u001b[0m 49.52   \u001b[0m | \u001b[0m 2.413e+0\u001b[0m | \u001b[0m 1.973e+0\u001b[0m | \u001b[0m 2.213e+0\u001b[0m |\n",
      "=================================================================================================\n"
     ]
    }
   ],
   "source": [
    "lgbBO = BayesianOptimization(lgb_eval, {'num_leaves': (25, 4000),\n",
    "                                                'max_depth': (5, 63),\n",
    "                                                'lambda_l2': (0.0, 0.05),\n",
    "                                                'lambda_l1': (0.0, 0.05),\n",
    "                                                'min_child_samples': (50, 10000),\n",
    "                                                'min_data_in_leaf': (100, 2000)\n",
    "                                                })\n",
    "\n",
    "lgbBO.maximize(n_iter=10, init_points=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rdkxhhST-KZS"
   },
   "source": [
    " **<font color='teal'> Print the best result by using the '.max' function.</font>**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-22T15:49:01.513767Z",
     "start_time": "2019-04-22T15:49:01.509392Z"
    },
    "colab": {},
    "colab_type": "code",
    "id": "oc8z6mfy-KZS"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'target': 0.7438198244178351,\n",
       " 'params': {'lambda_l1': 0.010985715583999883,\n",
       "  'lambda_l2': 0.015393774670654043,\n",
       "  'max_depth': 14.847018336687166,\n",
       "  'min_child_samples': 562.4944814981243,\n",
       "  'min_data_in_leaf': 1993.400363185335,\n",
       "  'num_leaves': 2551.5048049165457}}"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lgbBO.max"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-22T15:50:29.049881Z",
     "start_time": "2019-04-22T15:50:29.045908Z"
    },
    "colab_type": "text",
    "id": "J5LAydKC-KZW"
   },
   "source": [
    "Review the process at each step by using the '.res' attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-22T15:51:01.001688Z",
     "start_time": "2019-04-22T15:51:00.997484Z"
    },
    "colab": {},
    "colab_type": "code",
    "id": "X1ttZmrI-KZX"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'target': 0.7225360904836782,\n",
       " 'params': {'lambda_l1': 0.011551607884949322,\n",
       "  'lambda_l2': 0.00890388713349934,\n",
       "  'max_depth': 15.993298544504599,\n",
       "  'min_child_samples': 5990.378002354822,\n",
       "  'min_data_in_leaf': 524.571411514294,\n",
       "  'num_leaves': 3601.732443383757}}"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lgbBO.res[5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "name": "Bayesian_optimization_exercise.ipynb",
   "provenance": []
  },
  "deepnote_execution_queue": [],
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}