Grid_Search_CV_and_Random_Search_CV.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "#**Grid Search CV and Random Search CV**"
      ],
      "metadata": {
        "id": "cLE08q0e7YIm"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Taking Boston house price dataset to check accuracy of Random Forest Regression model and tuning hyperparameters-number of estimators and max depth of the tree to find the best value. \n",
        "\n",
        "First load boston data and split into train and test sets."
      ],
      "metadata": {
        "id": "lNGWQnJl3oU_"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "RZw_nWbY3FWN"
      },
      "outputs": [],
      "source": [
        "import numpy as np \n",
        "import pandas as pd\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "data_url = \"http://lib.stat.cmu.edu/datasets/boston\"\n",
        "raw_df = pd.read_csv(data_url, sep=\"\\s+\", skiprows=22, header=None)\n",
        "data = np.hstack([raw_df.values[::2, :], raw_df.values[1::2, :2]])\n",
        "target = raw_df.values[1::2, 2]"
      ],
      "metadata": {
        "id": "gn0LpYh54gxs"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#splitting data into train and test\n",
        "from sklearn.model_selection import train_test_split\n",
        "x_train, x_test, y_train, y_test = train_test_split(data,target,test_size=0.3)"
      ],
      "metadata": {
        "id": "rUQ68fJc4p9_"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Apply Random Forest Regressor model with n_estimators of 5 and max_depth of 3"
      ],
      "metadata": {
        "id": "2hqeAX5q405T"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn import ensemble\n",
        "dt=ensemble.RandomForestRegressor(n_estimators=5,max_depth=3)\n",
        "dt.fit(x_train,y_train)\n",
        "print('training score: ', dt.score(x_train,y_train))\n",
        "print('test score: ',dt.score(x_test,y_test))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "X7Sw14UZ3Z9h",
        "outputId": "42a2b048-fdd4-41cf-a9f5-b5f35c31d29c"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "training score:  0.8764723428120347\n",
            "test score:  0.6901009379432874\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's try with cross validation and see:"
      ],
      "metadata": {
        "id": "9lctULak49N2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.model_selection import cross_val_score\n",
        "scores1 = cross_val_score(ensemble.RandomForestRegressor(n_estimators=5,max_depth=3),x_train,y_train,cv=10)\n",
        "np.average(scores1)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ckcPpRtA4Hps",
        "outputId": "e8462b92-69a0-4cba-a1a7-5f7d2c767e14"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.8068719541667427"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The *Cross Validation* alone is not giving us something good. Let's combine it with *GridSearchCV*"
      ],
      "metadata": {
        "id": "UtCL7YcL6Po_"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "For this, a list of dictionaries of the list of hyperparameter values is passed. Then model is evaluated on every combination of each list of values to find the best one.\n",
        "\n"
      ],
      "metadata": {
        "id": "aze-d0lU5w1V"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ensemble.RandomForestRegressor().get_params()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qEYG1b-qDEpA",
        "outputId": "2d80b8c3-cbd4-414e-fdac-59567510342d"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'bootstrap': True,\n",
              " 'ccp_alpha': 0.0,\n",
              " 'criterion': 'squared_error',\n",
              " 'max_depth': None,\n",
              " 'max_features': 'auto',\n",
              " 'max_leaf_nodes': None,\n",
              " 'max_samples': None,\n",
              " 'min_impurity_decrease': 0.0,\n",
              " 'min_samples_leaf': 1,\n",
              " 'min_samples_split': 2,\n",
              " 'min_weight_fraction_leaf': 0.0,\n",
              " 'n_estimators': 100,\n",
              " 'n_jobs': None,\n",
              " 'oob_score': False,\n",
              " 'random_state': None,\n",
              " 'verbose': 0,\n",
              " 'warm_start': False}"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model=ensemble.RandomForestRegressor()\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "parameters=[{'n_estimators':[20,30,40,60,100], 'max_depth':  \n",
        "             [5,10,15,20]},\n",
        "             {'n_estimators':[20,30,40,60,100], 'max_depth': \n",
        "             [5,10,15,20],'max_features':[2,5,8]}]\n",
        "             \n",
        "grid_search = GridSearchCV(estimator=model,\n",
        "                           param_grid=parameters,\n",
        "                           cv=10,\n",
        "                           n_jobs=-1)\n",
        "                           \n",
        "grid = grid_search.fit(x_train,y_train)\n",
        "grid.best_score_"
      ],
      "metadata": {
        "id": "kSzCJ7k84LYg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "grid.best_params_"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZBE9y15W4Ur5",
        "outputId": "0f351404-0956-4786-e097-be2b73d9356b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'max_depth': 10, 'max_features': 5, 'n_estimators': 100}"
            ]
          },
          "metadata": {},
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "zeL6-mDq6hcA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#trying the best hyperparameters suggested by GridSearchCV\n",
        "#{'max_depth': 10, 'max_features': 5, 'n_estimators': 100}\n",
        "\n",
        "from sklearn import ensemble\n",
        "dt_1=ensemble.RandomForestRegressor(n_estimators=100, max_depth=10, max_features = 5)\n",
        "dt_1.fit(x_train,y_train)\n",
        "print('training score: ', dt_1.score(x_train,y_train))\n",
        "print('test score: ',dt_1.score(x_test,y_test))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O6PdXl9VEnGh",
        "outputId": "180d8a1d-22ad-40f6-b1ca-ec69e922dbd8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "training score:  0.9765502544034178\n",
            "test score:  0.8901441683241353\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Random Search CV"
      ],
      "metadata": {
        "id": "55DodKDT69fP"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In Random search CV, the best combination can not be identified because all the combinations are not tested. But the advantage is we can test a broad range of values for hyperparameters within the same computation time as grid search CV.\n",
        "\n",
        "Below is the implementation of Random search for the above example of Boston Housing prices dataset."
      ],
      "metadata": {
        "id": "a3L_6y317Ftj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.model_selection import RandomizedSearchCV\n",
        "model=ensemble.RandomForestRegressor()\n",
        "param_grid=[{'n_estimators':[20,30,40,60,100], 'max_depth':[5,10,15,20] \n",
        "          },{'n_estimators':[20,30,40,60,100], 'max_depth':[5,10,15,20]\n",
        "             ,'max_features':[2,5,8]}]\n",
        "\n",
        "rnd_search = RandomizedSearchCV(model, param_grid, cv=10, \n",
        "                                          return_train_score=True)\n",
        "rnd_search.fit(x_train,y_train)\n",
        "rnd_search.best_score_"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yaEIZJc96-gX",
        "outputId": "53524c6e-0cea-4ed6-f360-24947d63de79"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.8235312365882501"
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "rnd_search.best_params_"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NgHa2fAH7IMg",
        "outputId": "3163e75c-7d02-40a4-e6db-894127380695"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'max_depth': 10, 'max_features': 8, 'n_estimators': 20}"
            ]
          },
          "metadata": {},
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#trying the best hyperparameters suggested by RandomizedSearchCV\n",
        "\n",
        "#{'max_depth': 10, 'max_features': 8, 'n_estimators': 20}\n",
        "\n",
        "\n",
        "\n",
        "from sklearn import ensemble\n",
        "dt=ensemble.RandomForestRegressor(n_estimators=20, max_depth=10, max_features = 8)\n",
        "dt.fit(x_train,y_train)\n",
        "print('training score: ', dt.score(x_train,y_train))\n",
        "print('test score: ',dt.score(x_test,y_test))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0MSM0ZV_FTRe",
        "outputId": "6780b1ba-d006-4db6-b184-675fea6a32c7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "training score:  0.9644230305537048\n",
            "test score:  0.881733229588773\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "BL7tW1TR7PUE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#**Save and Load Machine Learning Models**"
      ],
      "metadata": {
        "id": "YycHz0obRHOE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In Machine Learning, when we finally arrive at the model that we are satisfied with, we can save and reuse it later.\n",
        "\n",
        "In this session, we will see the two(2) most common ways of saving our model and reloading it for future predictions:\n",
        "\n",
        "1. Finalize Your Model with pickle\n",
        "2. Finalize Your Model with Joblib"
      ],
      "metadata": {
        "id": "lO8egex2RRcK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Finalize Your Model with pickle"
      ],
      "metadata": {
        "id": "vMKaRXXKR-_s"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Pickle is the standard way of serializing objects in Python. You can use the pickle1 operation to serialize your machine learning algorithms and save the serialized format to a file. Later you can load this file to deserialize your model and use it to make new predictions. The example below demonstrates how you can save your model to file and load it to make predictions on the unseen test set. We will save the best model from our previous tutorial(see code above)."
      ],
      "metadata": {
        "id": "bLR9CX_-SKYc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pickle\n",
        "\n",
        "\n",
        "pickle.dump(dt_1, open('model.pkl', 'wb'))\n",
        "\n",
        "\n",
        "\n",
        "# some time later...\n",
        "# load the model from disk\n",
        "pickled_model = pickle.load(open('model.pkl', 'rb'))"
      ],
      "metadata": {
        "id": "aotIzOCcRI-f"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Running the example saves the model to finalized model.pkl in your local working directory. Load the saved model and evaluating it provides an estimate of accuracy of the model on unseen data."
      ],
      "metadata": {
        "id": "cM0rCY2SUggw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pickled_model.predict(x_test) "
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bT3TMPqaWGhx",
        "outputId": "c7cc01e0-f200-4edd-bb6b-b508e5165380"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([26.83962126,  8.61232381, 20.51751326, 28.19453292, 23.94116815,\n",
              "       25.2110988 , 34.93170079, 20.39077568, 16.02745032, 16.89315219,\n",
              "       21.85855031, 19.75804735, 22.33963641, 14.32117489, 15.36852813,\n",
              "       21.3690862 , 15.02309333, 20.38319334, 19.3126351 , 23.22964672,\n",
              "       23.56209709, 21.1691938 , 41.71069231, 24.2908043 , 20.38991156,\n",
              "       23.16937334, 44.898     , 16.36586787, 20.49836588, 20.66797121,\n",
              "       25.87777379, 20.77371312, 20.55072132, 21.87943995, 17.85003591,\n",
              "       46.557     , 26.16849376, 19.88904843, 13.01537154, 19.76115808,\n",
              "       14.46104524, 23.88767507, 10.21754719, 15.71266115, 24.62951764,\n",
              "       18.65706899, 27.71905513, 27.29179945, 20.29504793, 20.62034342,\n",
              "       23.52518336, 20.56068088, 24.05032143, 21.01796634, 13.11094538,\n",
              "       20.85114141, 27.7025    , 34.13029419, 14.11621995, 21.98597677,\n",
              "       15.77697325, 24.57999068, 29.2788869 , 31.66714921,  9.13535909,\n",
              "       23.38985045, 21.22257574, 32.2963918 , 11.0103697 , 20.96616667,\n",
              "       23.1521037 ,  8.93989603, 19.71276917, 13.90613193, 15.72998077,\n",
              "       20.57043471, 24.21147754, 27.86623745, 27.34596872, 21.51369914,\n",
              "       24.59840198, 20.4131829 , 21.68673797, 18.52448047, 17.90301768,\n",
              "       31.21857522, 22.81107194, 20.76403126,  9.68087381, 41.96621855,\n",
              "       22.44428972, 36.36546481, 32.77375794, 17.07153875, 17.79924762,\n",
              "       20.65112536, 20.48090116, 21.8399203 , 24.23859168, 14.51238932,\n",
              "       14.25263947, 22.99871866, 43.22283333, 12.74237381, 43.976     ,\n",
              "       15.19585317, 21.65761881, 23.77183467, 19.74641422, 23.10860792,\n",
              "       23.89652258, 20.12379973, 10.46683839, 11.26714524, 36.50522832,\n",
              "       19.17503769, 26.86041204, 40.43556571, 15.04821388, 21.56151805,\n",
              "       12.1918856 , 24.93043533, 19.95758979, 19.28896516, 15.84463107,\n",
              "       22.63984544, 27.29522975, 16.46377767, 26.27016697, 23.86294409,\n",
              "       47.525     , 19.31743791, 13.84800826, 21.18584881, 12.42373745,\n",
              "       30.63073208, 23.78769763, 18.86772498, 32.97432659, 15.67505555,\n",
              "       21.51417816, 32.57396641, 20.62676879, 33.5572812 , 22.5515797 ,\n",
              "       11.57066256,  9.79238052, 20.51161508, 34.36230783, 19.61517702,\n",
              "       32.64564708, 14.90156114])"
            ]
          },
          "metadata": {},
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Finalize Your Model with Joblib"
      ],
      "metadata": {
        "id": "Ya_ACWa-UjH0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install joblib"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pgRO5vEJWh6A",
        "outputId": "89972ba6-33c9-4960-c61c-36c730c341d5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (1.1.0)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import joblib\n",
        "\n",
        "\n",
        "joblib.dump(dt_1 , 'joblib_model')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0vRnfZUcS7ba",
        "outputId": "02a8b275-9974-461f-90f6-e48cd641ab8e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['joblib_model']"
            ]
          },
          "metadata": {},
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "jobLib = joblib.load('joblib_model')\n",
        "\n",
        "\n",
        "\n",
        "jobLib.predict(x_test)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6Vl6SkI0Wcy4",
        "outputId": "619b06f0-db34-4bac-9114-c63927561a98"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([26.83962126,  8.61232381, 20.51751326, 28.19453292, 23.94116815,\n",
              "       25.2110988 , 34.93170079, 20.39077568, 16.02745032, 16.89315219,\n",
              "       21.85855031, 19.75804735, 22.33963641, 14.32117489, 15.36852813,\n",
              "       21.3690862 , 15.02309333, 20.38319334, 19.3126351 , 23.22964672,\n",
              "       23.56209709, 21.1691938 , 41.71069231, 24.2908043 , 20.38991156,\n",
              "       23.16937334, 44.898     , 16.36586787, 20.49836588, 20.66797121,\n",
              "       25.87777379, 20.77371312, 20.55072132, 21.87943995, 17.85003591,\n",
              "       46.557     , 26.16849376, 19.88904843, 13.01537154, 19.76115808,\n",
              "       14.46104524, 23.88767507, 10.21754719, 15.71266115, 24.62951764,\n",
              "       18.65706899, 27.71905513, 27.29179945, 20.29504793, 20.62034342,\n",
              "       23.52518336, 20.56068088, 24.05032143, 21.01796634, 13.11094538,\n",
              "       20.85114141, 27.7025    , 34.13029419, 14.11621995, 21.98597677,\n",
              "       15.77697325, 24.57999068, 29.2788869 , 31.66714921,  9.13535909,\n",
              "       23.38985045, 21.22257574, 32.2963918 , 11.0103697 , 20.96616667,\n",
              "       23.1521037 ,  8.93989603, 19.71276917, 13.90613193, 15.72998077,\n",
              "       20.57043471, 24.21147754, 27.86623745, 27.34596872, 21.51369914,\n",
              "       24.59840198, 20.4131829 , 21.68673797, 18.52448047, 17.90301768,\n",
              "       31.21857522, 22.81107194, 20.76403126,  9.68087381, 41.96621855,\n",
              "       22.44428972, 36.36546481, 32.77375794, 17.07153875, 17.79924762,\n",
              "       20.65112536, 20.48090116, 21.8399203 , 24.23859168, 14.51238932,\n",
              "       14.25263947, 22.99871866, 43.22283333, 12.74237381, 43.976     ,\n",
              "       15.19585317, 21.65761881, 23.77183467, 19.74641422, 23.10860792,\n",
              "       23.89652258, 20.12379973, 10.46683839, 11.26714524, 36.50522832,\n",
              "       19.17503769, 26.86041204, 40.43556571, 15.04821388, 21.56151805,\n",
              "       12.1918856 , 24.93043533, 19.95758979, 19.28896516, 15.84463107,\n",
              "       22.63984544, 27.29522975, 16.46377767, 26.27016697, 23.86294409,\n",
              "       47.525     , 19.31743791, 13.84800826, 21.18584881, 12.42373745,\n",
              "       30.63073208, 23.78769763, 18.86772498, 32.97432659, 15.67505555,\n",
              "       21.51417816, 32.57396641, 20.62676879, 33.5572812 , 22.5515797 ,\n",
              "       11.57066256,  9.79238052, 20.51161508, 34.36230783, 19.61517702,\n",
              "       32.64564708, 14.90156114])"
            ]
          },
          "metadata": {},
          "execution_count": 50
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "xxRgMTdBXI8G"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}