1_28_2020_Supervised_learning_with_Sklearn.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "1-28-2020-Supervised_learning_with_Sklearn.ipynb",
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/hussain0048/Machine-Learning/blob/master/1_28_2020_Supervised_learning_with_Sklearn.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A4e9rm4b2PSL"
      },
      "source": [
        "## **Data Loading**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ufeXJwlA2Vzz",
        "outputId": "d4a6b204-a413-4c57-8733-359edc8be5d3"
      },
      "source": [
        "\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9B9THni-hHqx"
      },
      "source": [
        "# **1-Classification**\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n309L9RARmN6"
      },
      "source": [
        "##**1.1- Supervised Learning**\n",
        "\n",
        "\n",
        "**2. What is machine learning?**\n",
        "\n",
        "But what is machine learning? Machine learning is the science and art of giving computers the ability to learn to make decisions from data without being explicitly programmed. For example, your computer can learn to predict whether an email is spam or not spam given its content and sender. Another example: your computer can learn to cluster, say, Wikipedia entries, into different categories based on the words they contain. It could then assign any new Wikipedia article to one of the existing clusters. Notice that, in the first example, we are trying to predict a particular class label, that is, spam or not spam. In the second example, there is no such label. When there are labels present, we call it supervised learning. When there are no labels present, we call it unsupervised learning.\n",
        "\n",
        "**3. Unsupervised learning?**\n",
        "\n",
        "Unsupervised learning, in essence, is the machine learning task of uncovering hidden patterns and structures from unlabeled data. For example, a business may wish to group its customers into distinct categories based on their purchasing behavior without knowing in advance what these categories maybe. This is known as clustering, one branch of unsupervised learning.\n",
        "\n",
        "**4. Reinforcement learning**\n",
        "\n",
        "There is also reinforcement learning, in which machines or software agents interact with an environment. Reinforcement agents are able to automatically figure out how to optimize their behavior given a system of rewards and punishments. Reinforcement learning draws inspiration from behavioral psychology and has applications in many fields, such as, economics, genetics, as well as game playing. In 2016, reinforcement learning was used to train Google DeepMind's AlphaGo, which was the first computer program to beat the world champion in Go.\n",
        "\n",
        "**5. Supervised learning**\n",
        "\n",
        "But let's come back to supervised learning, which will be the focus of this course. In supervised learning, we have several data points or samples, described using predictor variables or features and a target variable. Our data is commonly represented in a table structure such as the one you see here, in which there is a row for each data point and a column for each feature. Here, we see the iris dataset: each row represents measurements of a different flower and each column is a particular kind of measurement, like the width and length of a certain part of the flower. The aim of supervised learning is to build a model that is able to predict the target variable, here the particular species of a flower, given the predictor variables, here the physical measurements. If the target variable consists of categories, like 'click' or 'no click', 'spam' or 'not spam', or different species of flowers, we call the learning task classification. Alternatively, if the target is a continuously varying variable, for example, the price of a house, it is a regression task. In this chapter, we will focus on classification. In the following, on regression.\n",
        "\n",
        "![](\n",
        "https://drive.google.com/uc?export=view&id=16Hdc-xwQKPyLSbBdgfkYgassyu2368Vw)\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SaYbgGcieCG3"
      },
      "source": [
        "**6. Naming conventions**\n",
        "\n",
        "A note on naming conventions: out in the wild, you will find that what we call a feature, others may call a predictor variable or independent variable, and what we call the target variable, others may call dependent variable or response variable."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w8tmZ9v4eT5e"
      },
      "source": [
        "**7. Supervised learning**\n",
        "\n",
        "The goal of supervised learning is frequently to either automate a time-consuming or expensive manual task, such as a doctor's diagnosis, or to make predictions about the future, say whether a customer will click on an add or not. For supervised learning, you need labeled data and there are many ways to get it: you can get historical data, which already has labels that you are interested in; you can perform experiments to get labeled data, such as A/B-testing to see how many clicks you get; or you can also crowdsourced labeling data which, like reCAPTCHA does for text recognition. In any case, the goal is to learn from data for which the right output is known, so that we can make predictions on new data for which we don't know the output."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-GOYgcZce0BE"
      },
      "source": [
        "**8. Supervised learning in Python**\n",
        "\n",
        "There are many ways to perform supervised learning in Python. In this course, we will use scikit-learn, or sklearn, one of the most popular and user-friendly machine learning libraries for Python. It also integrates very well with the SciPy stack, including libraries such as NumPy. There are a number of other ML libraries out there, such as TensorFlow and keras, which are well worth checking out once you got the basics down."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AA4oiwXKgsjc"
      },
      "source": [
        "## **1.2 Exploratory data analysis**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y3Ku3f2LmkWX"
      },
      "source": [
        "**1-The Iris dataset**\n",
        "\n",
        "![](\n",
        "https://drive.google.com/uc?export=view&id=1WTn1t7HRbkZbQRPcAQ4EzO1w630qkfXK\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5b7P9TOonweD"
      },
      "source": [
        "**The Iris dataset in scikit-learn**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D_axHWIwoPxU"
      },
      "source": [
        "from sklearn import datasets\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "plt.style.use('ggplot')\n",
        "iris = datasets.load_iris()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IEZ5LGUIpWX5",
        "outputId": "8d47aa8e-06af-4598-b5ee-a0766446ce23"
      },
      "source": [
        "iris = datasets.load_iris() \n",
        "In [7]: type(iris) "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "sklearn.utils.Bunch"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EbfpuVMJphJL",
        "outputId": "feb17aba-7024-496d-d43c-bfe8948a6e0a"
      },
      "source": [
        "print(iris.keys()) "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "dict_keys(['data', 'target', 'target_names', 'DESCR', 'feature_names', 'filename'])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0d-Q5AH6olQz"
      },
      "source": [
        "type(iris.data), type(iris.target)\n",
        "#Out(numpy.ndarray, numpy.ndarray)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MhOhuLw0pxkV",
        "outputId": "b809c8ea-c3d4-4b58-de04-86f626c438e0"
      },
      "source": [
        "iris.data.shape "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(150, 4)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9Kwx8Oy1p2WC",
        "outputId": "f3325fd6-c5dd-4ab0-f54a-0f14acf8a053"
      },
      "source": [
        "iris.target_names "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array(['setosa', 'versicolor', 'virginica'], dtype='<U10')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sM6YCCFRp_IK"
      },
      "source": [
        "**Exploratory data analysis (EDA)**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hgbWSURiqLva"
      },
      "source": [
        "X = iris.data\n",
        "y = iris.target"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kIxeM1SNqRJh"
      },
      "source": [
        "df = pd.DataFrame(X, columns=iris.feature_names) "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pt5b8j-RqfOi",
        "outputId": "57503f51-70d0-43f9-ba45-bed31d46d7fc"
      },
      "source": [
        "print(df.head()) "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "   sepal length (cm)  sepal width (cm)  petal length (cm)  petal width (cm)\n",
            "0                5.1               3.5                1.4               0.2\n",
            "1                4.9               3.0                1.4               0.2\n",
            "2                4.7               3.2                1.3               0.2\n",
            "3                4.6               3.1                1.5               0.2\n",
            "4                5.0               3.6                1.4               0.2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m_g0O6PbqqoZ"
      },
      "source": [
        "**Visual EDA**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XAH2NelJqvlY"
      },
      "source": [
        "_ = pd.scatter_matrix(df, c = y, figsize = [8, 8], s=150, marker = 'D')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ghF69fYuszdd"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1UqiYyKsAmnrMPIG6Ahr6esFgPdy0RTlT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TRVuJVHLtIr2"
      },
      "source": [
        "### **1.2.1 Numerical EDA**\n",
        "In this chapter, you'll be working with a dataset obtained from the [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/Congressional+Voting+Records) UCI Machine Learning Repository consisting of votes made by US House of Representatives Congressmen. Your goal will be to predict their party affiliation ('Democrat' or 'Republican') based on how they voted on certain key issues. Here, it's worth noting that we have preprocessed this dataset to deal with missing values. This is so that your focus can be directed towards understanding how to train and evaluate supervised learning models. Once you have mastered these fundamentals, you will be introduced to preprocessing techniques in Chapter 4 and have the chance to apply them there yourself - including on this very same dataset!\n",
        "\n",
        "Before thinking about what supervised learning models you can apply to this, however, you need to perform Exploratory data analysis (EDA) in order to understand the structure of the data. For a refresher on the importance of EDA, check out the first two chapters of Statistical Thinking in Python (Part 1).\n",
        "\n",
        "Get started with your EDA now by exploring this voting records dataset numerically. It has been pre-loaded for you into a DataFrame called df. Use pandas' .head(), .info(), and .describe() methods in the IPython Shell to explore the DataFrame, and select the statement below that is not true.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_OFC7ycmf5Xq"
      },
      "source": [
        "df.head()\n",
        "df.info()\n",
        "df.described()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GVHGTmNqgRvy"
      },
      "source": [
        "### **1.2.2 Visual EDA**\n",
        "\n",
        "The Numerical EDA you did in the previous exercise gave you some very important information, such as the names and data types of the columns, and the dimensions of the DataFrame. Following this with some visual EDA will give you an even better understanding of the data. In the video, Hugo used the scatter_matrix() function on the Iris data for this purpose. However, you may have noticed in the previous exercise that all the features in this dataset are binary; that is, they are either 0 or 1. So a different type of plot would be more useful here, such as Seaborn's countplot.\n",
        "\n",
        "Given on the right is a countplot of the 'education' bill, generated from the following code:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-EWSIcZngnxP"
      },
      "source": [
        "plt.figure()\n",
        "sns.countplot(x='education', hue='party', data=df, palette='RdBu')\n",
        "plt.xticks([0,1], ['No', 'Yes'])\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WUnp7fm6iIsd"
      },
      "source": [
        "![](\n",
        "https://drive.google.com/uc?export=view&id=1ev2xHhcqCVsrZ4nipIrjG47DstY3OFLj)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uzOxK_UhhGs5"
      },
      "source": [
        "In sns.countplot(), we specify the x-axis data to be 'education', and hue to be 'party'. Recall that 'party' is also our target variable. So the resulting plot shows the difference in voting behavior between the two parties for the 'education' bill, with each party colored differently. We manually specified the color to be 'RdBu', as the Republican party has been traditionally associated with red, and the Democratic party with blue.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hcOo2vVLiPTa"
      },
      "source": [
        "It seems like Democrats voted resoundingly against this bill, compared to Republicans. This is the kind of information that our machine learning model will seek to learn when we try to predict party affiliation solely based on voting behavior. An expert in U.S politics may be able to predict this without machine learning, but probably not instantaneously - and certainly not if we are dealing with hundreds of samples!\n",
        "\n",
        "In the IPython Shell, explore the voting behavior further by generating countplots for the 'satellite' and 'missile' bills, and answer the following question: Of these two bills, for which ones do Democrats vote resoundingly in favor of, compared to Republicans? Be sure to begin your plotting statements for each figure with plt.figure() so that a new figure will be set up. Otherwise, your plots will be overlayed onto the same figure."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ACNfrD0IjkwO"
      },
      "source": [
        "plt.figure()\n",
        "sns.countplot(x='satellite', hue='party', data=df, palette='RdBu')\n",
        "plt.xticks([0,1], ['No', 'Yes'])\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xytbzq6nj7g1"
      },
      "source": [
        "## **1.3 The classification challenge**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kqa3HUwPkQ5m"
      },
      "source": [
        "**1. The classification challenge**\n",
        "\n",
        "We have a set of labeled data and we want to build a classifier that takes unlabeled data as input and outputs a label. So how do we construct this classifier? We first need choose a type of classifier and it needs to learn from the already labeled data. For this reason, we call the already labeled data the training data. So let's build our first classifier!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mrr65SKpkVDn"
      },
      "source": [
        "**2. k-Nearest Neighbors**\n",
        "\n",
        "We'll choose a simple algorithm called K-nearest neighbors. The basic idea of K-nearest neighbors, or KNN, is to predict the label of any data point by looking at the K, for example, 3, closest labeled data points and getting them to vote on what label the unlabeled point should have.\n",
        "\n",
        "**3. k-Nearest Neighbors**\n",
        "\n",
        "In this image, there's an example of KNN in two dimensions: how do you classify the data point in the middle?\n",
        "\n",
        "![](https://drive.google.com/uc?export=view&id=1en3RyJ-TTung7hGabN1jgjUvfM5YOBK3)\n",
        "\n",
        "**5. k-Nearest Neighbors**\n",
        "\n",
        "you would classify it as red and, if k equals 5, as green.\n",
        "\n",
        "**6. k-Nearest Neighbors**\n",
        "\n",
        "and, if k equals 5,\n",
        "\n",
        "7. **k-Nearest Neighbors**\n",
        "\n",
        "as green."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zBW83jiAmaE-"
      },
      "source": [
        "**8. k-NN: Intuition**\n",
        "\n",
        "To get a bit of intuition for KNN, let's check out a scatter plot of two dimensions of the iris dataset, petal length and petal width. The following holds for higher dimensions, however, we'll show the 2D case for illustrative purposes.\n",
        "\n",
        "![](\n",
        "https://drive.google.com/uc?export=view&id=1iMg24DZBcSGlONoyYTu_Vm0vfYFd7K-d)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4GZtZMa6sXB5"
      },
      "source": [
        "**10. k-NN: Intuition**\n",
        "\n",
        "Any new data point here will be predicted 'setosa',\n",
        "\n",
        "**11. k-NN: Intuition**\n",
        "\n",
        "any new data point here will be predicted 'virginica',\n",
        "\n",
        "**12**. **k-NN: Intuition**\n",
        "\n",
        "and any new data point here will be predicted 'versicolor'."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fobO_NR3si2r"
      },
      "source": [
        "**13. Scikit-learn fit and predict**\n",
        "\n",
        "All machine learning models in scikit-learn are implemented as python classes. These classes serve two purposes: they implement the algorithms for learning a model, and predicting, while also storing all the information that is learned from the data. Training a model on the data is also called fitting the model to the data. In scikit-learn, we use the fit method to do this. Similarly, the predict method is what we use to predict the label of an, unlabeled data point."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9AAavWQgtBFz"
      },
      "source": [
        "**14. Using scikit-learn to fit a classifier**\n",
        "\n",
        "Now we're going to fit our very first classifier using scikit-learn! To do so, we first need to import it. To this end, we import KNeighborsClassifier from sklearn dot neighbors. We then instantiate our KNeighborsClassifier, set the number of neighbors equal to 6, and assign it to the variable knn. Then we can fit this classifier to our training set, the labeled data. To do so, we apply the method fit to the classifier and pass it two arguments: the features as a NumPy array and the labels, or target, as a NumPy array. The scikit-learn API requires firstly that you have the data as a NumPy array or pandas DataFrame. It also requires that the features take on continuous values, such as the price of a house, as opposed to categories, such as 'male' or 'female'. It also requires that there are no missing values in the data. All datasets that we'll work with now satisfy these final two properties. Later in the course, you'll see how to deal with categorical features and missing data. In particular, the scikit-learn API requires that the features are in an array where each column is a feature and each row a different observation or data point. Looking at the shape of iris data, we see that there are 150 observations of four features. Similarly, the target needs to be a single column with the same number of observations as the feature data. We see in this case there are indeed also 150 labels. Also check out what is returned when we fit the classifier: it returns the classifier itself and modifies it to fit it to the data. Now that we have fit our classifier, lets use it to predict on some unlabeled data!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7CCkJDultmDR"
      },
      "source": [
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "knn = KNeighborsClassifier(n_neighbors=6)\n",
        "knn.fit(iris['data'], iris['target'])\n",
        "KNeighborsClassifier(algorithm='auto', leaf_size=30,\n",
        "metric='minkowski',metric_params=None, n_jobs=1,\n",
        "n_neighbors=6, p=2,weights='uniform')\n",
        "iris['data'].shape\n",
        "#(150, 4)\n",
        "iris['target'].shape\n",
        "#(150,)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TZgx8B2It8dB"
      },
      "source": [
        "**15. Predicting on unlabeled data**\n",
        "\n",
        "Here we have set of observations, X new. We use the predict method on the classifier and pass it the data. Once again, the API requires that we pass the data as a NumPy array with features in columns and observations in rows; checking the shape of X new, we see that it has three rows and four columns, that is, three observations and four features. Then we would expect calling knn dot predict of X new to return a three-by-one array with a prediction for each observation or row in X new. And indeed it does! It predicts one, which corresponds to 'versicolor' for the first two observations and 0, which corresponds to 'setosa' for the third."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jDAuwkG3uT9p"
      },
      "source": [
        "prediction = knn.predict(X_new)\n",
        "X_new.shape\n",
        "#(3, 4)\n",
        "print('Prediction {}’.format(prediction))\n",
        "#Prediction: [1 1 0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ENrKTSWPeQnZ"
      },
      "source": [
        "### **1.3.1 k-Nearest Neighbors: Fit**\n",
        "Having explored the Congressional voting records dataset, it is time now to build your first classifier. In this exercise, you will fit a k-Nearest Neighbors classifier to the voting dataset, which has once again been pre-loaded for you into a DataFrame df.\n",
        "\n",
        "In the video, Hugo discussed the importance of ensuring your data adheres to the format required by the scikit-learn API. The features need to be in an array where each column is a feature and each row a different observation or data point - in this case, a Congressman's voting record. The target needs to be a single column with the same number of observations as the feature data. We have done this for you in this exercise. Notice we named the feature array X and response variable y: This is in accordance with the common scikit-learn practice.\n",
        "\n",
        "Your job is to create an instance of a k-NN classifier with 6 neighbors (by specifying the n_neighbors parameter) and then fit it to the data. The data has been pre-loaded into a DataFrame called df.\n",
        "\n",
        "**Instructions**\n",
        "\n",
        "- Import KNeighborsClassifier from sklearn.neighbors.\n",
        "- Create arrays X and y for the features and the target variable. Here this has been done for you. Note the use of .drop() to drop the target variable 'party' from the feature array X as well as the use of the .values attribute to ensure X and y are NumPy arrays. Without using .values, X and y are a DataFrame and Series respectively; the scikit-learn API will accept them in this form also as long as they are of the right shape.\n",
        "- Instantiate a KNeighborsClassifier called knn with 6 neighbors by specifying the n_neighbors parameter.\n",
        "- Fit the classifier to the data using the .fit() method.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2GZZ5vLqe3ph"
      },
      "source": [
        "**Hint**\n",
        "\n",
        "- To import thing from pkg, use the command from pkg import thing.\n",
        "- To instantiate a KNeighborsClassifier with 6 neighbors, use the function KNeighborsClassifier() and specify the keyword argument n_neighbors=6.\n",
        "- Use the .fit() method on knn with X and y passed in as the arguments."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3iQwHhwzkNpV"
      },
      "source": [
        "# Import KNeighborsClassifier from sklearn.neighbors\n",
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "# Create arrays for the features and the response variable\n",
        "y = df['party'].values\n",
        "X = df.drop('party', axis=1).values\n",
        "# Create a k-NN classifier with 6 neighbors\n",
        "knn = KNeighborsClassifier(n_neighbors=6)\n",
        "# Fit the classifier to the data\n",
        "knn.fit(X,y)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VGhTbl_BkUyk"
      },
      "source": [
        "Excellent! Now that your k-NN classifier with 6 neighbors has been fit to the data, it can be used to predict the labels of new data points."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "74fwCO3akETV"
      },
      "source": [
        "### **1.3.2 k-Nearest Neighbors: Predict**\n",
        "\n",
        "Having fit a k-NN classifier, you can now use it to predict the label of a new data point. However, there is no unlabeled data available since all of it was used to fit the model! You can still use the .predict() method on the X that was used to fit the model, but it is not a good indicator of the model's ability to generalize to new, unseen data.\n",
        "\n",
        "In the next video, Hugo will discuss a solution to this problem. For now, a random unlabeled data point has been generated and is available to you as X_new. You will use your classifier to predict the label for this new data point, as well as on the training data X that the model has already seen. Using .predict() on X_new will generate 1 prediction, while using it on X will generate 435 predictions: 1 for each sample.\n",
        "\n",
        "The DataFrame has been pre-loaded as df. This time, you will create the feature array X and target variable array y yourself.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "foGCoaK3kkZn"
      },
      "source": [
        "**Instruction**\n",
        "\n",
        "- Create arrays for the features and the target variable from df. As a reminder, the target variable is 'party'.\n",
        "- Instantiate a KNeighborsClassifier with 6 neighbors.\n",
        "- Fit the classifier to the data.\n",
        "- Predict the labels of the training data, X.\n",
        "- Predict the label of the new data point X_new."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f9vZqaWzm9ON"
      },
      "source": [
        "# Import KNeighborsClassifier from sklearn.neighbors\n",
        "from sklearn.neighbors import KNeighborsClassifier \n",
        "# Create arrays for the features and the response variable\n",
        "Y = df['party'].values\n",
        "X = df.drop('party',axis=1).values\n",
        "# Create a k-NN classifier with 6 neighbors: knn\n",
        "knn = KNeighborsClassifier(n_neighbors=6)\n",
        "# Fit the classifier to the data\n",
        "knn.fit(X,Y)\n",
        "# Predict the labels for the training data X\n",
        "y_pred = knn.predict(X)\n",
        "# Predict and print the label for the new data point X_new\n",
        "new_prediction = knn.predict(X_new)\n",
        "print(\"Prediction: {}\".format(new_prediction))\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pEk3MM1bneba"
      },
      "source": [
        "Great work! Did your model predict 'democrat' or 'republican'? How sure can you be of its predictions? In other words, how can you measure its performance? This is what you will learn in the next video."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QNpcM-Kmnleo"
      },
      "source": [
        "## **1.4-Measuring model performance**\n",
        "\n",
        "**1. Measuring model performance**\n",
        "\n",
        "Now that we know how to fit a classifier and use it to predict the labels of previously unseen data, we need to figure out how to measure its performance. That is, we need a metric.\n",
        "\n",
        "**2. Measuring model performance**\n",
        "\n",
        "In classification problems, accuracy is a commonly-used metric. The accuracy of a classifier is defined as the number of correct predictions divided by the total number of data points. This begs the question though: which data do we use to compute accuracy? What we are really interested in is how well our model will perform on new data, that is, samples that the algorithm has never seen before.\n",
        "\n",
        "**3. Measuring model performance**\n",
        "\n",
        "Well, you could compute the accuracy on the data you used to fit the classifier. However, as this data was used to train it, the classifier's performance will not be indicative of how well it can generalize to unseen data. For this reason, it is common practice to split your data into two sets, a training set and a test set. You train or fit the classifier on the training set. Then you make predictions on the labeled test set and compare these predictions with the known labels. You then compute the accuracy of your predictions.\n",
        "\n",
        "**4. Train/test split**\n",
        "\n",
        "To do this, we first import train test split from sklearn dot model selection. We then use the train test split function to randomly split our data. The first argument will be the feature data, the second the targets or labels. The test size keyword argument specifies what proportion of the original data is used for the test set. Lastly, the random state kwarg sets a seed for the random number generator that splits the data into train and test. Setting the seed with the same argument later will allow you to reproduce the exact split and your downstream results. train test split returns four arrays: the training data, the test data, the training labels, and the test labels. We unpack these into four variables: X train, X test, y train, and y test, respectively. By default, train test split splits the data into 75% training data and 25% test data, which is a good rule of thumb. We specify the size of the test size using the keyword argument test size, which we do here to set it to 30%. It is also best practice to perform your split so that the split reflects the the labels on your data. That is, you want the labels to be distributed in train and test sets as they are in the original dataset. To achieve this, we use the keyword argument stratify equals y, where y the list or array containing the labels. We then instantiate our K-nearest neighbors classifier, fit it to the training data using the fit method, make our predictions on the test data and store the results as y pred. Printing them shows that the predictions take on three values, as expected. To check out the accuracy of our model, we use the score method of the model and pass it X test and y test. See here that the accuracy of our K-nearest neighbors model is approximately 95%, which is pretty good for an out-of-the-box model!\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xf2MI0WgrRAn"
      },
      "source": [
        "In [1]: from sklearn.model_selection import train_test_split\n",
        "In [2]: X_train, X_test, y_train, y_test =\n",
        " ...: train_test_split(X, y, test_size=0.3,\n",
        " ...: random_state=21, stratify=y)\n",
        "In [3]: knn = KNeighborsClassifier(n_neighbors=8)\n",
        "In [4]: knn.fit(X_train, y_train)\n",
        "In [5]: y_pred = knn.predict(X_test)\n",
        "In [6]: print(\"Test set predictions:\\n {}\".format(y_pred))\n",
        "Test set predictions:\n",
        " [2 1 2 2 1 0 1 0 0 1 0 2 0 2 2 0 0 0 1 0 2 2 2 0 1 1 1 0 0\n",
        " 1 2 2 0 0 2 2 1 1 2 1 1 0 2 1]\n",
        "In [7]: knn.score(X_test, y_test)\n",
        "Out[7]: 0.9555555555555556"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qXXVboCVrqZ_"
      },
      "source": [
        "**5. Model complexity**\n",
        "\n",
        "Recall that we recently discussed the concept of a decision boundary. Here, we visualize a decision boundary for several, increasing values of K in a KNN model. Note that, as K increases, the decision boundary gets smoother and less curvy. Therefore, we consider it to be a less complex model than those with a lower K. Generally, complex models run the risk of being sensitive to noise in the specific data that you have, rather than reflecting general trends in the data. This is know as overfitting.\n",
        "\n",
        "1 Source: Andreas Müller & Sarah Guido, Introduction to Machine Learning with Python"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f5prd8pCthFF"
      },
      "source": [
        "![](\n",
        "https://drive.google.com/uc?export=view&id=1dYN4OVLuOuVlm_Nk6aB5UJF5oOrCAHgu)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3GZGec3mHGyv"
      },
      "source": [
        "**6. Model complexity and over/underfitting**\n",
        "\n",
        "If you increase K even more and make the model even simpler, then the model will perform less well on both test and training sets, as indicated in this schematic figure, known as a model complexity curve.\n",
        "\n",
        "**7. Model complexity and over/underfitting**\n",
        "\n",
        "This is called underfitting.\n",
        "\n",
        "8-**Model complexity and over/underfitting**\n",
        "\n",
        "We can see that there is a sweet spot in the middle that gives us the best performance on the test set."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-FIduBl4IkQE"
      },
      "source": [
        "![](\n",
        "https://drive.google.com/uc?export=view&id=1PpCtAvU7BRZ9jVv4u68bNYqM56wwu7DU)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ycz4-NxmKIGS"
      },
      "source": [
        "### **1.4.1-The digits recognition dataset**\n",
        "\n",
        "Up until now, you have been performing binary classification, since the target variable had two possible outcomes. Hugo, however, got to perform multi-class classification in the videos, where the target variable could take on three possible outcomes. Why does he get to have all the fun?! In the following exercises, you'll be working with the [MNIST digits](http://yann.lecun.com/exdb/mnist/) recognition dataset, which has 10 classes, the digits 0 through 9! A reduced version of the MNIST dataset is one of scikit-learn's included datasets, and that is the one we will use in this exercise.\n",
        "\n",
        "Each sample in this scikit-learn dataset is an 8x8 image representing a handwritten digit. Each pixel is represented by an integer in the range 0 to 16, indicating varying levels of black. Recall that scikit-learn's built-in datasets are of type Bunch, which are dictionary-like objects. Helpfully for the MNIST dataset, scikit-learn provides an 'images' key in addition to the 'data' and 'target' keys that you have seen with the Iris data. Because it is a 2D array of the images corresponding to each sample, this 'images' key is useful for visualizing the images, as you'll see in this exercise (for more on plotting 2D arrays, see Chapter 2 of DataCamp's course on Data Visualization with Python). On the other hand, the 'data' key contains the feature array - that is, the images as a flattened array of 64 pixels.\n",
        "\n",
        "Notice that you can access the keys of these Bunch objects in two different ways: By using the . notation, as in digits.images, or the [] notation, as in digits['images'].\n",
        "\n",
        "For more on the MNIST data, check out this exercise in Part 1 of DataCamp's Importing Data in Python course. There, the full version of the MNIST dataset is used, in which the images are 28x28. It is a famous dataset in machine learning and computer vision, and frequently used as a benchmark to evaluate the performance of a new model.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OzHI9pqGLp2z"
      },
      "source": [
        "**Instruction**\n",
        "- Import datasets from sklearn and matplotlib.pyplot as plt.\n",
        "- Load the digits dataset using the .load_digits() method on datasets.\n",
        "- Print the keys and DESCR of digits.\n",
        "- Print the shape of images and data keys using the . notation.\n",
        "- Display the 1011th image using plt.imshow(). This has been done for you, so - "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ni-INgDJPruZ"
      },
      "source": [
        "# Import necessary modules\n",
        "from sklearn import datasets\n",
        "import matplotlib.pyplot as plt\n",
        "# Load the digits dataset: digits\n",
        "digits = datasets.load_digits()\n",
        "# Print the keys and DESCR of the dataset\n",
        "print(digits.keys())\n",
        "print(digits.DESCR)\n",
        "# Print the shape of the images and data keys\n",
        "print(digits.images.shape)\n",
        "print(digits.data.shape)\n",
        "# Display digit 1010\n",
        "plt.imshow(digits.images[1010], cmap=plt.cm.gray_r, interpolation='nearest')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HzNkgyrCPuf6"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1fxZvtyQK9aDEz5p5r6v507HSg9AOCH07)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r9B6SKRPQAoL"
      },
      "source": [
        "Good job! It looks like the image in question corresponds to the digit '5'. Now, can you build a classifier that can make this prediction not only for this image, but for all the other ones in the dataset? You'll do so in the next exercise!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3X9x-ul1QU7R"
      },
      "source": [
        "### **1.4.2 Train/Test Split + Fit/Predict/Accuracy**\n",
        "\n",
        "Now that you have learned about the importance of splitting your data into training and test sets, it's time to practice doing this on the digits dataset! After creating arrays for the features and target variable, you will split them into training and test sets, fit a k-NN classifier to the training data, and then compute its accuracy using the .score() method.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Neb9FScKQlLe"
      },
      "source": [
        "**Instruction** \n",
        "\n",
        "- Import KNeighborsClassifier from sklearn.neighbors and train_test_split from sklearn.model_selection.\n",
        "- Create an array for the features using digits.data and an array for the target using digits.target.\n",
        "- Create stratified training and test sets using 0.2 for the size of the test set. Use a random state of 42. Stratify the split according to the labels so that they are distributed in the training and test sets as they are in the original dataset.\n",
        "- Create a k-NN classifier with 7 neighbors and fit it to the training data.\n",
        "Compute and print the accuracy of the classifier's predictions using the .score() method."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xJ2lDOruSYsW"
      },
      "source": [
        "# Import necessary modules\n",
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn import datasets\n",
        "# Create feature and target arrays\n",
        "X = digits.data\n",
        "y = digits.target\n",
        "# Split into training and test set\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state=42, stratify=y)\n",
        "# Create a k-NN classifier with 7 neighbors: knn\n",
        "knn = KNeighborsClassifier(n_neighbors=7)\n",
        "# Fit the classifier to the training data\n",
        "knn.fit(X_train,y_train)\n",
        "# Print the accuracy\n",
        "print(knn.score(X_test, y_test))\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XmNElNccSbbo"
      },
      "source": [
        "- Print the accuracy\n",
        "- print(knn.score(X_test, y_test))\n",
        "- 0.9833333333333333"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "neaNhYnRStbf"
      },
      "source": [
        "Excellent work! Incredibly, this out of the box k-NN classifier with 7 neighbors has learned from the training data and predicted the labels of the images in the test set with 98% accuracy, and it did so in less than a second! This is one illustration of how incredibly useful machine learning techniques can be."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H9N65LgZSv-U"
      },
      "source": [
        "### **1.4.3 Overfitting and underfitting**\n",
        "\n",
        "Remember the model complexity curve that Hugo showed in the video? You will now construct such a curve for the digits dataset! In this exercise, you will compute and plot the training and testing accuracy scores for a variety of different neighbor values. By observing how the accuracy scores differ for the training and testing sets with different values of k, you will develop your intuition for overfitting and underfitting.\n",
        "\n",
        "The training and testing sets are available to you in the workspace as X_train, X_test, y_train, y_test. In addition, KNeighborsClassifier has been imported from sklearn.neighbors.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "puY6eFcyS_Qc"
      },
      "source": [
        "**Insruction**\n",
        "\n",
        "- Inside the for loop:\n",
        "  - Setup a k-NN classifier with the number of neighbors equal to k.\n",
        "- Fit the classifier with k neighbors to the training data.\n",
        "- Compute accuracy scores the training set and test set separately using the .- score() method and assign the results to the train_accuracy and test_accuracy arrays respectively."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0lA9Q7x_sesw"
      },
      "source": [
        "# Setup arrays to store train and test accuracies\n",
        "neighbors = np.arange(1, 9)\n",
        "train_accuracy = np.empty(len(neighbors))\n",
        "test_accuracy = np.empty(len(neighbors))\n",
        "# Loop over different values of k\n",
        "for i, k in enumerate(neighbors):\n",
        "    # Setup a k-NN Classifier with k neighbors: knn\n",
        "    knn = KNeighborsClassifier(n_neighbors=k)\n",
        "    # Fit the classifier to the training data\n",
        "    knn.fit(X_train,y_train)\n",
        "    #Compute accuracy on the training set\n",
        "    train_accuracy[i] = knn.score(X_train, y_train)\n",
        "    #Compute accuracy on the testing set\n",
        "    test_accuracy[i] = knn.score(X_test, y_test)\n",
        "# Generate plot\n",
        "plt.title('k-NN: Varying Number of Neighbors')\n",
        "plt.plot(neighbors, test_accuracy, label = 'Testing Accuracy')\n",
        "plt.plot(neighbors, train_accuracy, label = 'Training Accuracy')\n",
        "plt.legend()\n",
        "plt.xlabel('Number of Neighbors')\n",
        "plt.ylabel('Accuracy')\n",
        "plt.show()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jAGVZvf4sjLU"
      },
      "source": [
        "![](\n",
        "https://drive.google.com/uc?export=view&id=1YWwLXp6yvdcpm4cn4TKPAPTVkGsxUTnL\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rU6Pe1PftCrK"
      },
      "source": [
        "Great work! It looks like the test accuracy is highest when using 3 and 5 neighbors. Using 8 neighbors or more seems to result in a simple model that underfits the data. Now that you've grasped the fundamentals of classification, you will learn about regression in the next chapter!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xj769rRxVaE6"
      },
      "source": [
        "# **2-Regression**\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IyVCKSxCmjK6"
      },
      "source": [
        "## **2.1 Introduction to regression**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ujHIBC9JVpL6"
      },
      "source": [
        "\n",
        "**1. Introduction to regression**\n",
        "\n",
        "Congrats on making it through that introduction to supervised learning and classification. Now, we're going to check out the other type of supervised learning problem: regression. In regression tasks, the target value is a continuously varying variable, such as a country's GDP or the price of a house.\n",
        "\n",
        "**2. Boston housing data**\n",
        "\n",
        "Our first regression task will be using the Boston housing dataset! Let's check out the data. First, we load it from a comma-separated values file, also known as a csv file, using pandas' read csv function. See the DataCamp course on importing data for more information on file formats and loading your data. Note that you can also load this data from scikit-learn's built-in datasets. We then view the head of the data frame using the head method. The documentation tells us the feature 'CRIM' is per capita crime rate, 'NX' is nitric oxides concentration, and 'RM' average number of rooms per dwelling, for example. The target variable, 'MEDV', is the median value of owner occupied homes in thousands of dollars.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_bXNLBV6Wzay"
      },
      "source": [
        " boston = pd.read_csv('boston.csv')\n",
        "print(boston.head())\n",
        "\n",
        "CRIM ZN INDUS CHAS NX RM AGE DIS RAD TAX \\\n",
        "0 0.00632 18.0 2.31 0 0.538 6.575 65.2 4.0900 1 296.0\n",
        "1 0.02731 0.0 7.07 0 0.469 6.421 78.9 4.9671 2 242.0\n",
        "2 0.02729 0.0 7.07 0 0.469 7.185 61.1 4.9671 2 242.0\n",
        "3 0.03237 0.0 2.18 0 0.458 6.998 45.8 6.0622 3 222.0\n",
        "4 0.06905 0.0 2.18 0 0.458 7.147 54.2 6.0622 3 222.0\n",
        " PTRATIO B LSTAT MEDV\n",
        "0 15.3 396.90 4.98 24.0\n",
        "1 17.8 396.90 9.14 21.6 "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hD1nRpj8XF2h"
      },
      "source": [
        "**3. Creating feature and target arrays**\n",
        "\n",
        "Now, given data as such, recall that scikit-learn wants 'features' and target' values in distinct arrays, X and y,. Thus, we split our DataFrame: in the first line here, we drop the target; in the second, we keep only the target. Using the values attributes returns the NumPy arrays that we will use."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lmbyGoxBXLoH"
      },
      "source": [
        " X = boston.drop('MEDV', axis=1).values\n",
        " y = boston['MEDV'].values"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rp526EPKcsL6"
      },
      "source": [
        "**4. Predicting house value from a single feature**\n",
        "\n",
        "As a first task, let's try to predict the price from a single feature: the average number of rooms in a block. To do this, we slice out the number of rooms column of the DataFrame X, which is the fifth column into the variable X rooms. Checking the type of X rooms and y, we see that both are NumPy arrays. To turn them into NumPy arrays of the desired shape, we apply the reshape method to keep the first dimension, but add another dimension of size one to X."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OSRY9EQRdERj"
      },
      "source": [
        "X_rooms = X[:,5]\n",
        "type(X_rooms), type(y)\n",
        "(numpy.ndarray, numpy.ndarray)\n",
        "y = y.reshape(-1, 1)\n",
        "X_rooms = X_rooms.reshape(-1, 1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aupjha5JdVbw"
      },
      "source": [
        "**5. Plotting house value vs. number of rooms**\n",
        "\n",
        "Now, let's plot house value as a function of number of rooms using matplotlib's plt dot scatter. We'll also label our axes using x label and y label.\n",
        "\n",
        "**6. Plotting house value vs. number of rooms**\n",
        "\n",
        "We can immediately see that, as one might expect, more rooms lead to higher prices."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mKVJHBZ5dejq"
      },
      "source": [
        "plt.scatter(X_rooms, y)\n",
        "plt.ylabel('Value of house /1000 ($)')\n",
        "plt.xlabel('Number of rooms')\n",
        "plt.show();"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qzW9Jkuoltms"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1jqCvIr04_b0bXGcSfc0hWeSK1C2uwBos)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PQrS8Nsqu81Y"
      },
      "source": [
        "**7. Fitting a regression model**\n",
        "\n",
        "It's time to fit a regression model to our data. We're going to use a model called linear regression, which we'll explain in the next video. But first, I'm going to show you how to fit it and to plot its predictions. We import numpy as np, linear model from sklearn, and instantiate LinearRegression as regr. We then fit the regression to the data using regr dot fit and passing in the data, the number of rooms, and the target variable, the house price, as we did with the classification problems. After this, we want to check out the regressors predictions over the range of the data. We can achieve that by using np linspace between the maximum and minimum number of rooms and make a prediction for this data.\n",
        "\n",
        "**8. Fitting a regression model**\n",
        "\n",
        "Plotting this line with the scatter plot results in the figure you see here."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H3NFaX_TvFI-"
      },
      "source": [
        "import numpy as np\n",
        "from sklearn import linear_model\n",
        "reg = linear_model.LinearRegression()\n",
        "reg.fit(X_rooms, y)\n",
        "prediction_space = np.linspace(min(X_rooms),\n",
        "max(X_rooms)).reshape(-1, 1)\n",
        "plt.scatter(X_rooms, y, color='blue')\n",
        "plt.plot(prediction_space, reg.predict(prediction_space),\n",
        " ...: color='black', linewidth=3)\n",
        "plt.show()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_sB1lUDxvtiM"
      },
      "source": [
        "![](\n",
        "https://drive.google.com/uc?export=view&id=1BDWAvjpKGo3iaNwy0ctajpVdQ2TiKdg2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lKcVRy4IacI6"
      },
      "source": [
        "### **2.1.1 Importing data for supervised learning**\n",
        "\n",
        "In this chapter, you will work with [Gapminder](https://www.gapminder.org/data/) data that we have consolidated into one CSV file available in the workspace as 'gapminder.csv'. Specifically, your goal will be to use this data to predict the life expectancy in a given country based on features such as the country's GDP, fertility rate, and population. As in Chapter 1, the dataset has been preprocessed.\n",
        "\n",
        "Since the target variable here is quantitative, this is a regression problem. To begin, you will fit a linear regression with just one feature: 'fertility', which is the average number of children a woman in a given country gives birth to. In later exercises, you will use all the features to build regression models.\n",
        "\n",
        "Before that, however, you need to import the data and get it into the form needed by scikit-learn. This involves creating feature and target variable arrays. Furthermore, since you are going to use only one feature to begin with, you need to do some reshaping using NumPy's .reshape() method. Don't worry too much about this reshaping right now, but it is something you will have to do occasionally when working with scikit-learn so it is useful to practice."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RctzEfYThTEx"
      },
      "source": [
        "**instruction**\n",
        "\n",
        "- Import numpy and pandas as their standard aliases.\n",
        "- Read the file 'gapminder.csv' into a DataFrame df using the read_csv() function.\n",
        "- Create array X for the 'fertility' feature and array y for the 'life' target variable.\n",
        "- Reshape the arrays by using the .reshape() method and passing in -1 and 1."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n83hDUknhdwi"
      },
      "source": [
        "**Hint**\n",
        "\n",
        "- Use import x as y to import x as the alias y.\n",
        "- To read in the CSV file 'gapminder.csv', pass it in as an argument to pd.read_csv().\n",
        "- To create the arrays, first select the column using df['column_name'] and then access its .values attribute.\n",
        "- Use .reshape(-1, 1) on X and y to reshape them."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VVyccSJui90a"
      },
      "source": [
        "# Import numpy and pandas\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "# Read the CSV file into a DataFrame: df\n",
        "df = pd.read_csv('gapminder.csv')\n",
        "# Create arrays for features and target variable\n",
        "y = df['life'].values\n",
        "X = df['fertility'].values\n",
        "# Print the dimensions of X and y before reshaping\n",
        "print(\"Dimensions of y before reshaping: {}\".format(y.shape))\n",
        "print(\"Dimensions of X before reshaping: {}\".format(X.shape))\n",
        "# Reshape X and y\n",
        "y = y.reshape(1,-1)\n",
        "X = X.reshape(1,-1)\n",
        "# Print the dimensions of X and y after reshaping\n",
        "print(\"Dimensions of y after reshaping: {}\".format(y.shape))\n",
        "print(\"Dimensions of X after reshaping: {}\".format(X.shape))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SJiqlJpajmkH"
      },
      "source": [
        "Great work! Notice the differences in shape before and after applying the .reshape() method. Getting the feature and target variable arrays into the right format for scikit-learn is an important precursor to model building."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D4GK_RS8mZ2k"
      },
      "source": [
        "### **2.1.2 Exploring the Gapminder data**\n",
        "\n",
        "As always, it is important to explore your data before building models. On the right, we have constructed a heatmap showing the correlation between the different features of the Gapminder dataset, which has been pre-loaded into a DataFrame as df and is available for exploration in the IPython Shell. Cells that are in green show positive correlation, while cells that are in red show negative correlation. Take a moment to explore this: Which features are positively correlated with life, and which ones are negatively correlated? Does this match your intuition?\n",
        "\n",
        "Then, in the IPython Shell, explore the DataFrame using pandas methods such as .info(), .describe(), .head().\n",
        "\n",
        "In case you are curious, the heatmap was generated using Seaborn's heatmap function and the following line of code, where df.corr() computes the pairwise correlation between columns:\n",
        "\n",
        "sns.heatmap(df.corr(), square=True, cmap='RdYlGn')\n",
        "\n",
        "Once you have a feel for the data, consider the statements below and select the one that is not true. After this, Hugo will explain the mechanics of linear regression in the next video and you will be on your way building regression models!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JoxxOGUFaBVs"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1xvQRRgKKUOe-Tz9ZhC0Z-PpZJuenUZNQ)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7iMkoWmvaM5E"
      },
      "source": [
        "**Possible Answers**\n",
        "\n",
        "\n",
        "- The DataFrame has 139 samples (or rows) and 9 columns.\n",
        "- life and fertility are negatively correlated.\n",
        "- The mean of life is 69.602878.\n",
        "- **fertility is of type int64.**\n",
        "- GDP and life are positively correlated."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EgsDpwE-a1L3"
      },
      "source": [
        "Good job! As seen by using df.info(), fertility, along with all the other columns, is of type float64, not int64"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wn0fRa8Fci_w"
      },
      "source": [
        "## **2.2- The basics of linear regression**\n",
        "\n",
        "1.**The basics of linear regression**\n",
        "\n",
        "Now, how does linear regression actually work?\n",
        "\n",
        "**2. Regression mechanics**\n",
        "\n",
        "We want to fit a line to the data and a line in two dimensions is always of the form y = ax + b, where y is the target, x is the single feature, and a and b are the parameters of the model that we want to learn. So the question of fitting is reduced to: how do we choose a and b? A common method is to define an error function for any given line and then to choose the line that minimizes the error function. Such an error function is also called a loss or a cost function.\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Import libraries\n",
        "import numpy as np\n",
        "from sklearn.linear_model import LinearRegression\n",
        "# Prepare input data\n",
        "# X represents independent variables\n",
        "X = np.array([[1, 1], [1, 2], [1, 3], [2, 1], [2, 2], [2, 3]])\n",
        "# Regression equation: y = 1 * x_0 + 2 * x_1 + 3\n",
        "# y represents dependant variable\n",
        "y = np.dot(X, np.array([1, 2])) + 3\n",
        "# array([ 6,  8, 10,  7,  9, 11])\n",
        "reg = LinearRegression().fit(X, y)\n",
        "reg.score(X, y)\n",
        "# Regression coefficients\n",
        "reg.coef_\n",
        "# array([1., 2.])\n",
        "reg.intercept_\n",
        "# 2.999999999999999\n",
        "reg.predict(np.array([[4, 4]]))\n",
        "# array([15.])\n",
        "reg.predict(np.array([[6, 7]]))\n",
        "array([23.])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 208
        },
        "id": "srOLWnpMEypJ",
        "outputId": "35c9a3f1-9460-445c-9190-34f52ba248c5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-1-f1274b00c467>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     19\u001b[0m \u001b[0;31m# array([15.])\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     20\u001b[0m \u001b[0mreg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m6\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m7\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 21\u001b[0;31m \u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m23.\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m: name 'array' is not defined"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_z_h2SAsdNXp"
      },
      "source": [
        "**3. The loss function**\n",
        "\n",
        "What will our loss function be? Intuitively, we want the line to be as close to the\n",
        "\n",
        "**4. The loss function**\n",
        "\n",
        "actual data points as possible. For this reason, we wish to minimize the vertical distance between the fit and the data. So for each data point,"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eG44e7UJfGas"
      },
      "source": [
        "**5. The loss function**\n",
        "\n",
        "we calculate the vertical distance between it and the line. This distance is called a residual.\n",
        "\n",
        "**6. The loss function**\n",
        "\n",
        "Now, we could try to minimize the sum of the residuals,\n",
        "\n",
        "**7. The loss function**\n",
        "\n",
        "but then a large positive residual would cancel out\n",
        "\n",
        "**8. The loss function**\n",
        "\n",
        "a large negative residual. For this reason we minimize the sum of the squares of the residuals! This will be our loss function and using this loss function is commonly called ordinary least squares, or OLS for short. Note that this is the same as minimizing the mean squared error of the predictions on the training set. See our statistics curriculum for more detail. When you call fit on a linear regression model in scikit-learn, it performs this OLS under the hood."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ol2Fg2tte3Hr"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=15dcv0DfIVn8qDtFCAvj6Aaxrr267PDxo)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lIG8wdbPbIy0"
      },
      "source": [
        "### **2.2.1 Fit & predict for regression**\n",
        "\n",
        "Now, you will fit a linear regression and predict life expectancy using just one feature. You saw Andy do this earlier using the 'RM' feature of the Boston housing dataset. In this exercise, you will use the 'fertility' feature of the Gapminder dataset. Since the goal is to predict life expectancy, the target variable here is 'life'. The array for the target variable has been pre-loaded as y and the array for 'fertility' has been pre-loaded as X_fertility.\n",
        "\n",
        "A scatter plot with 'fertility' on the x-axis and 'life' on the y-axis has been generated. As you can see, there is a strongly negative correlation, so a linear regression should be able to capture this trend. Your job is to fit a linear regression and then predict the life expectancy, overlaying these predicted values on the plot to generate a regression line. You will also compute and print the  score using sckit-learn's .score() method."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Juk_945pcAGI"
      },
      "source": [
        "**Instruction**\n",
        "\n",
        "- Import LinearRegression from sklearn.linear_model.\n",
        "- Create a LinearRegression regressor called reg.\n",
        "- Set up the prediction space to range from the minimum to the maximum of X_fertility. This has been done for you.\n",
        "- Fit the regressor to the data (X_fertility and y) and  compute its predictions using the .predict() method and the prediction_space array.\n",
        "- Compute and print the  score using the .score() method.\n",
        "- Overlay the plot with your linear regression line. This has been done for you, so hit 'Submit Answer' to see the result!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ktSdO0V9cc_e"
      },
      "source": [
        "**Hint**\n",
        "\n",
        "- You can import x from y using the command from y import x.\n",
        "- Use the function LinearRegression() to create the regressor.\n",
        "- Use the .fit() method on reg with X_fertility and y as arguments to fit the model.\n",
        "- Use the .predict() method on reg with prediction_space as the argument to compute the predictions.\n",
        "- Use the .score() method with X_fertility and y as arguments to compute the  score."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z4oDGz_KeTgu"
      },
      "source": [
        "# Import LinearRegression\n",
        "from  sklearn.linear_model import LinearRegression \n",
        "\n",
        "# Create the regressor: reg\n",
        "reg = LinearRegression()\n",
        "\n",
        "# Create the prediction space\n",
        "prediction_space = np.linspace(min(X_fertility), max(X_fertility)).reshape(-1,1)\n",
        "\n",
        "# Fit the model to the data\n",
        "reg.fit(X_fertility,y)\n",
        "\n",
        "# Compute predictions over the prediction space: y_pred\n",
        "y_pred = reg.predict(prediction_space)\n",
        "\n",
        "# Print R^2 \n",
        "print(reg.score(X_fertility, y))\n",
        "\n",
        "# Plot regression line\n",
        "plt.plot(prediction_space, y_pred, color='black', linewidth=3)\n",
        "plt.show()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ia78yQqteY88"
      },
      "source": [
        "0.6192442167740035\n",
        "![](https://drive.google.com/uc?export=view&id=17iAekZUxgGDNbyrLGGkTR1h7Cps-LKHm)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3AVQHUT0f8ku"
      },
      "source": [
        "### **2.2.2 Train/test split for regression**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4slCY06SgOMz"
      },
      "source": [
        "As you learned in Chapter 1, train and test sets are vital to ensure that your supervised learning model is able to generalize well to new data. This was true for classification models, and is equally true for linear regression models.\n",
        "\n",
        "In this exercise, you will split the Gapminder dataset into training and testing sets, and then fit and predict a linear regression over all features. In addition to computing the  score, you will also compute the Root Mean Squared Error (RMSE), which is another commonly used metric to evaluate regression models. The feature array X and target variable array y have been pre-loaded for you from the DataFrame df."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dOK-gUTzgV9d"
      },
      "source": [
        "**Instruction**\n",
        "\n",
        "- Import LinearRegression from sklearn.linear_model, mean_squared_error from sklearn.metrics, and train_test_split from sklearn.model_selection.\n",
        "- Using X and y, create training and test sets such that 30% is used for testing and 70% for training. Use a random state of 42.\n",
        "- Create a linear regression regressor called reg_all, fit it to the training set, and evaluate it on the test set.\n",
        "- Compute and print the  score using the .score() method on the test set.\n",
        "- Compute and print the RMSE. To do this, first compute the Mean Squared Error using the mean_squared_error() function with the arguments y_test and y_pred, and then take its square root using np.sqrt()."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GHUotWmGjxWq"
      },
      "source": [
        "# Import necessary modules\n",
        "from sklearn.linear_model import LinearRegression\n",
        "from sklearn.metrics import mean_squared_error\n",
        "from sklearn.model_selection import train_test_split\n",
        "# Create training and test sets\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state=42)\n",
        "# Create the regressor: reg_all\n",
        "reg_all = LinearRegression()\n",
        "# Fit the regressor to the training data\n",
        "reg_all.fit(X_train,y_train)\n",
        "# Predict on the test data: y_pred\n",
        "y_pred = reg_all.predict(X_test)\n",
        "# Compute and print R^2 and RMSE\n",
        "print(\"R^2: {}\".format(reg_all.score(X_test, y_test)))\n",
        "rmse = np.sqrt(mean_squared_error(y_test,y_pred))\n",
        "print(\"Root Mean Squared Error: {}\".format(rmse))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EwsWul3lkI2k"
      },
      "source": [
        "Compute and print R^2 and RMSE\n",
        "print(\"R^2: {}\".format(reg_all.score(X_test, y_test))) rmse = np.sqrt(mean_squared_error(y_test,y_pred)) print(\"Root Mean Squared Error: {}\".format(rmse))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SxQKUDXrpiwa"
      },
      "source": [
        "Excellent! Using all features has improved the model score. This makes sense, as the model has more information to learn from. However, there is one potential pitfall to this process. Can you spot it? You'll learn about this as well how to better validate your models in the next video!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iOhh6F5mpyNe"
      },
      "source": [
        "## 2.3 **Cross-validation**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PzCwAt-7rzC_"
      },
      "source": [
        "**1. Cross-validation**\n",
        "\n",
        "Great work on those regression challenges! You are now also becoming more acquainted with train test split and computing model performance metrics on your test set. Can you spot a potential pitfall of this process? Well, let's think about it for a bit:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PsEdi0uCsARN"
      },
      "source": [
        "**2. Cross-validation motivation**\n",
        "\n",
        "if you're computing R squared on your test set, the R squared returned is dependent on the way that you split up the data! The data points in the test set may have some peculiarities that mean the R squared computed on it is not representative of the model's ability to generalize to unseen data. To combat this dependence on what is essentially an arbitrary split, we use a technique called cross-validation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tA3e16c9saqm"
      },
      "source": [
        "**3. Cross-validation basics**\n",
        "\n",
        "We begin by splitting the dataset into five groups or folds.\n",
        "\n",
        "**4. Cross-validation basics**\n",
        "\n",
        "Then we hold out the first fold as a test set,\n",
        "\n",
        "**5. Cross-validation basics **\n",
        "\n",
        "fit our model on the remaining four folds,\n",
        "\n",
        "**6. Cross-validation basics**\n",
        "\n",
        "predict on the test set, and\n",
        "\n",
        "**7. Cross-validation basics**\n",
        "\n",
        "compute the metric of interest.\n",
        "\n",
        "**8. Cross-validation basics**\n",
        "\n",
        "Next, we hold out the\n",
        "\n",
        "**9. Cross-validation basics**\n",
        "\n",
        "second fold as our test set,\n",
        "\n",
        "**10. Cross-validation basics**\n",
        "\n",
        "fit on the remaining data,\n",
        "\n",
        "**11. Cross-validation basics**\n",
        "\n",
        "predict on the test set, and\n",
        "\n",
        "**12. Cross-validation basics **\n",
        "\n",
        "compute the metric of interest. Then similarly\n",
        "\n",
        "**13. Cross-validation basics**\n",
        "\n",
        "with the third,\n",
        "\n",
        "**14. Cross-validation basics**\n",
        "\n",
        "fourth, and\n",
        "\n",
        "**15. Cross-validation basics**\n",
        "\n",
        "fifth fold.\n",
        "\n",
        "**16. Cross-validation basics**\n",
        "\n",
        "As a result we get five values of R squared from which we can compute statistics of interest, such as the mean and median and 95% confidence intervals."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HwkqCBzbuAWX"
      },
      "source": [
        "![](\n",
        "https://drive.google.com/uc?export=view&id=1b8bAOYtidKp9F497K6Hif40ZpOnIPU1k)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4ivf-xPhui5N"
      },
      "source": [
        "**17. Cross-validation and model performance**\n",
        "\n",
        "As we split the dataset into five folds, we call this process 5-fold cross validation. If you use 10 folds, it is called 10-fold cross validation. More generally, if you use k folds, it is called k-fold cross validation or k-fold CV. There is, however, a trade-off as using more folds is more computationally expensive. This is because you are fittings and predicting more times. This method avoids the problem of your metric of choice being dependent on the train test split.\n",
        "\n",
        "**18. Cross-validation in scikit-learn**\n",
        "\n",
        "To perform k-fold CV in scikit-learn, we first import cross val score from sklearn dot model selection. As always, we instantiate our model, in this case, a regressor. We then call cross val score with the regressor, the feature data, and the target data as the first three positional arguments. We also specify the number of folds with the keyword argument, cv. This returns an array of cross-validation scores, which we assign to cv results. The length of the array is the number of folds utilized. Note that the score reported is R squared, as this is the default score for linear regression. . We print the scores here. We can also, for example, compute the mean, which we also do."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6s12P0SZu6bu"
      },
      "source": [
        "In [1]: from sklearn.model_selection import cross_val_score\n",
        "In [2]: reg = linear_model.LinearRegression()\n",
        "In [3]: cv_results = cross_val_score(reg, X, y, cv=5)\n",
        "In [4]: print(cv_results)\n",
        "[ 0.63919994 0.71386698 0.58702344 0.07923081 -0.25294154]\n",
        "In [5]: np.mean(cv_results)\n",
        "Out[5]: 0.35327592439587058"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "from sklearn.model_selection import KFold\n",
        "\n",
        "X = np.array([[1, 2], [3, 4], [1, 2], [3, 4]])\n",
        "y = np.array([1, 2, 3, 4])\n",
        "kf = KFold(n_splits=2)\n",
        "\n",
        "for train_index, test_index in kf.split(X):\n",
        "  print(\"TRAIN:\", train_index, \"TEST:\", test_index)\n",
        "  X_train, X_test = X[train_index], X[test_index]\n",
        "  y_train, y_test = y[train_index], y[test_index]"
      ],
      "metadata": {
        "id": "FeRF_K_rk8Hb",
        "outputId": "e84f6d81-44f5-4c52-a108-2d056aef4798",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TRAIN: [2 3] TEST: [0 1]\n",
            "TRAIN: [0 1] TEST: [2 3]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A9vB4ZfEkOpI"
      },
      "source": [
        "### **2.3.1- 5-fold cross-validation**\n",
        "\n",
        "Cross-validation is a vital step in evaluating a model. It maximizes the amount of data that is used to train the model, as during the course of training, the model is not only trained, but also tested on all of the available data.\n",
        "\n",
        "In this exercise, you will practice 5-fold cross validation on the Gapminder data. By default, scikit-learn's cross_val_score() function uses  as the metric of choice for regression. Since you are performing 5-fold cross-validation, the function will return 5 scores. Your job is to compute these 5 scores and then take their average.\n",
        "\n",
        "The DataFrame has been loaded as df and split into the feature/target variable arrays X and y. The modules pandas and numpy have been imported as pd and np, respectively.\n",
        "\n",
        "**Instruction**\n",
        "\n",
        "- Import LinearRegression from sklearn.linear_model and cross_val_score from sklearn.model_selection.\n",
        "- Create a linear regression regressor called reg.\n",
        "- Use the cross_val_score() function to perform 5-fold cross-validation on X and y.\n",
        "- Compute and print the average cross-validation score. You can use NumPy's mean() function to compute the average."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kLF5ZeepqSr1"
      },
      "source": [
        "# Import the necessary modules\n",
        "from sklearn.linear_model import LinearRegression\n",
        "from sklearn.model_selection import cross_val_score\n",
        "# Create a linear regression object: reg\n",
        "reg = LinearRegression()\n",
        "# Compute 5-fold cross-validation scores: cv_scores\n",
        "cv_scores = cross_val_score(reg,X,y,cv=3)\n",
        "# Print the 5-fold cross-validation scores\n",
        "print(np.mean(cv_scores))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qebutKhhqVl_"
      },
      "source": [
        "0.8718712782622108\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "os8eNEE3q3Lw"
      },
      "source": [
        "### **2.3.2 K-Fold CV comparison**\n",
        "\n",
        "Cross validation is essential but do not forget that the more folds you use, the more computationally expensive cross-validation becomes. In this exercise, you will explore this for yourself. Your job is to perform 3-fold cross-validation and then 10-fold cross-validation on the Gapminder dataset.\n",
        "\n",
        "In the IPython Shell, you can use %timeit to see how long each 3-fold CV takes compared to 10-fold CV by executing the following cv=3 and cv=10:\n",
        "\n",
        "%timeit cross_val_score(reg, X, y, cv = ____)\n",
        "pandas and numpy are available in the workspace as pd and np. The DataFrame has been loaded as df and the feature/target variable arrays X and y have been created.\n",
        "\n",
        "- Import LinearRegression from sklearn.linear_model and cross_val_score from sklearn.model_selection.\n",
        "- Create a linear regression regressor called reg.\n",
        "- Perform 3-fold CV and then 10-fold CV. Compare the resulting mean scores."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WLgRWO13vxTc"
      },
      "source": [
        "# Import necessary modules\n",
        "from sklearn.linear_model import LinearRegression\n",
        "from sklearn.model_selection import cross_val_score\n",
        "\n",
        "# Create a linear regression object: reg\n",
        "reg = LinearRegression()\n",
        "\n",
        "# Perform 3-fold CV\n",
        "%timeit cvscores_3 = cross_val_score(reg,X,y,cv=3)\n",
        "print(np.mean(cvscores_3))\n",
        "\n",
        "# Perform 10-fold CV\n",
        "%timeit cvscores_10 = cross_val_score(reg,X,y,cv=10)\n",
        "print(np.mean(cvscores_10))\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kBXg-ibBwe1A"
      },
      "source": [
        "Perform 10-fold CV\n",
        "- %timeit cvscores_10 = cross_val_score(reg,X,y,cv=10)\n",
        "- print(np.mean(cvscores_10))\n",
        "- 100 loops, best of 3: 10.9 ms per loop\n",
        "- 0.8718712782622108\n",
        "- 10 loops, best of 3: 34.2 ms per loop\n",
        "- 0.8436128620131201"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nmflJFJX4EfR"
      },
      "source": [
        "## **2.4-Regularized regression**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9UaemC3d4PoB"
      },
      "source": [
        "**1. Regularized regression**\n",
        "\n",
        "Recall that\n",
        "\n",
        "**2. Why regularize?**\n",
        "\n",
        "what fitting a linear regression does is minimize a loss function to choose a coefficient ai for each feature variable. If we allow these coefficients or parameters to be super large, we can get overfitting. It isn't so easy to see in two dimensions, but when you have loads and loads of features, that is, if your data sit in a high-dimensional space with large coefficients, it gets easy to predict nearly anything. For this reason, it is common practice to alter the loss function so that it penalizes for large coefficients. This is called regularization. The first type of regularized regression that we'll look at is called ridge regression"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2A1UvF4q5hEK"
      },
      "source": [
        "**3. Ridge regression**\n",
        "\n",
        "in which our loss function is the standard OLS loss function plus the squared value of each coefficient multiplied by some constant alpha. Thus, when minimizing the loss function to fit to our data, models are penalized for coefficients with a large magnitude: large positive and large negative coefficients, that is. Note that alpha is a parameter we need to choose in order to fit and predict. Essentially, we can select the alpha for which our model performs best. Picking alpha for ridge regression is similar to picking k in KNN. This is called hyperparameter tuning and we'll see much more of this soon. This alpha, which you may also see called lambda in the wild, can be thought of as a parameter that controls model complexity. Notice that when alpha is equal to zero, we get back OLS. Large coefficients in this case are not penalized and the overfitting problem is not accounted for. A very high alpha means that large coefficients are significantly penalized, which can lead to a model that is too simple and ends up underfitting the data. The method of performing ridge regression with scikit-learn mirrors the other models that we have seen."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rlstCJap8kT5"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=16TIM4nHfHU9Rc2Gfa3hRtfuRUjR5Z7II)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kvy_XAFHOYeU"
      },
      "source": [
        "**4. Ridge regression in scikit-learn**\n",
        "\n",
        "We import Ridge from sklearn dot linear model, we split our data into test and train, fit on the training, and predict on the test. Note that we set alpha using the keyword argument alpha. Also notice the argument normalize: setting this equal to True ensures that all our variables are on the same scale and we will cover this in more depth later. There is another type of regularized regression called lasso regression,"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FJqfJ67HO7X4"
      },
      "source": [
        "In [1]: from sklearn.linear_model import Ridge\n",
        "In [2]: X_train, X_test, y_train, y_test = train_test_split(X, y,\n",
        " ...: test_size = 0.3, random_state=42)\n",
        "In [3]: ridge = Ridge(alpha=0.1, normalize=True)\n",
        "In [4]: ridge.fit(X_train, y_train)\n",
        "In [5]: ridge_pred = ridge.predict(X_test)\n",
        "In [6]: ridge.score(X_test, y_test)\n",
        "Out[6]: 0.69969382751273179"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Import library\n",
        "from sklearn.linear_model import Ridge\n",
        "# Building ridge regression model with hyperparameter alpha = 0.1\n",
        "clf = Ridge(alpha=0.1)\n",
        "# Prepare input data\n",
        "X = np.array([[1, 1], [1, 2], [1, 3], [2, 1], [2, 2], [2, 3]])\n",
        "y = [ 6,  8, 10,  7,  9, 11]\n",
        "clf.fit(X, y)\n",
        "clf.coef_\n",
        "# array([0.9375, 1.95121951])\n",
        "clf.intercept_\n",
        "# 3.1913109756097553\n",
        "clf.predict(np.array([[4, 4]]))\n",
        "# array([14.74618902])\n",
        "clf.predict(np.array([[6, 7]]))\n",
        "# array([22.47484756])"
      ],
      "metadata": {
        "id": "zCS1C7h6ch2K"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CFffkIs0O9bZ"
      },
      "source": [
        "**5. Lasso regression**\n",
        "\n",
        "in which our loss function is the standard OLS loss function plus the absolute value of each coefficient multiplied by some constant alpha.\n",
        "\n",
        "**6. Lasso regression in scikit-learn**\n",
        "\n",
        "The method of performing lasso regression in scikit-learn mirrors ridge regression, as you can see here."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1VRg2l4nROZX"
      },
      "source": [
        "In [1]: from sklearn.linear_model import Lasso\n",
        "In [2]: X_train, X_test, y_train, y_test = train_test_split(X, y,\n",
        " ...: test_size = 0.3, random_state=42)\n",
        "In [3]: lasso = Lasso(alpha=0.1, normalize=True)\n",
        "In [4]: lasso.fit(X_train, y_train)\n",
        "In [5]: lasso_pred = lasso.predict(X_test)\n",
        "In [6]: lasso.score(X_test, y_test)\n",
        "Out[6]: 0.59502295353285506"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Import library\n",
        "from sklearn import linear_model\n",
        "# Building lasso regression model with hyperparameter alpha = 0.1\n",
        "clf = linear_model.Lasso(alpha=0.1)\n",
        "# Prepare input data\n",
        "X = np.array([[1, 1], [1, 2], [1, 3], [2, 1], [2, 2], [2, 3]])\n",
        "y = [ 6,  8, 10,  7,  9, 11]\n",
        "clf.fit([[0,0], [1, 1], [2, 2]], [0, 1, 2])\n",
        "# Regression coefficients\n",
        "clf.coef_\n",
        "# array([0.6 , 1.85])\n",
        "clf.intercept_\n",
        "# 3.8999999999999995\n",
        "clf.predict(np.array([[4, 4]]))\n",
        "# array([13.7])\n",
        "clf.predict(np.array([[6, 7]]))\n",
        "# array([20.45])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TMzhTfNnUZY2",
        "outputId": "62b23e8b-5c9a-4e6c-f297-a774b78e44d7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([5.25])"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UeFWmLubRQBT"
      },
      "source": [
        "**7. Lasso regression for feature selection**\n",
        "\n",
        "One of the really cool aspects of lasso regression is that it can be used to select important features of a dataset. This is because it tends to shrink the coefficients of less important features to be exactly zero. The features whose coefficients are not shrunk to zero are 'selected' by the LASSO algorithm. Let's check this out in practice."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gmvgs1ScRmZ6"
      },
      "source": [
        "**8. Lasso for feature selection in scikit-learn**\n",
        "\n",
        "We import Lasso as before and store the feature names in the variable names. We then instantiate our regressor, fit it to the data as always. Then we can extract the coef attribute and store in lasso coef. Plotting the coefficients as a function of feature name yields this figure\n",
        "\n",
        "**9. Lasso for feature selection in scikit-learn**\n",
        "\n",
        "and you can see directly that the most important predictor for our target variable, housing price, is number of rooms! This is not surprising and is a great sanity check. This type of feature selection is very important for machine learning in an industry or business setting because it allows you, as a Data Scientist, to communicate important results to non-technical colleagues. And bosses! The power of reporting important features from a linear model cannot be overestimated. It is also valuable in research science, in order identify which factors are important predictors for various physical phenomena."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dgi9WFu6xU-b"
      },
      "source": [
        "In [1]: from sklearn.linear_model import Lasso\n",
        "In [2]: names = boston.drop('MEDV', axis=1).columns\n",
        "In [3]: lasso = Lasso(alpha=0.1)\n",
        "In [4]: lasso_coef = lasso.fit(X, y).coef_\n",
        "In [5]: _ = plt.plot(range(len(names)), lasso_coef)\n",
        "In [6]: _ = plt.xticks(range(len(names)), names, rotation=60)\n",
        "In [7]: _ = plt.ylabel('Coefficients')\n",
        "In [8]: plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y8slb0Iyzx2l"
      },
      "source": [
        "### **2.4.1 Regularization I: Lasso**\n",
        "\n",
        "In the video, you saw how Lasso selected out the 'RM' feature as being the most important for predicting Boston house prices, while shrinking the coefficients of certain other features to 0. Its ability to perform feature selection in this way becomes even more useful when you are dealing with data involving thousands of features.\n",
        "\n",
        "In this exercise, you will fit a lasso regression to the Gapminder data you have been working with and plot the coefficients. Just as with the Boston data, you will find that the coefficients of some features are shrunk to 0, with only the most important ones remaining.\n",
        "\n",
        "The feature and target variable arrays have been pre-loaded as X and y."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xD7Mk25U0EP9"
      },
      "source": [
        "**instruction**\n",
        "\n",
        "- Import Lasso from sklearn.linear_model.\n",
        "- Instantiate a Lasso regressor with an alpha of 0.4 and specify normalize=True.\n",
        "- Fit the regressor to the data and compute the coefficients using the coef_ attribute.\n",
        "- Plot the coefficients on the y-axis and column names on the x-axis. This has been done for you, so hit 'Submit Answer' to view the plot!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PMbGcQ9k0SXx"
      },
      "source": [
        "# Import Lasso\n",
        "from sklearn.linear_model import Lasso\n",
        "# Instantiate a lasso regressor: lasso\n",
        "lasso = Lasso(alpha=0.4, normalize=True)\n",
        "# Fit the regressor to the data\n",
        "lasso.fit(X,y)\n",
        "# Compute and print the coefficients\n",
        "lasso_coef = lasso.coef_\n",
        "print(lasso_coef)\n",
        "# Plot the coefficients\n",
        "plt.plot(range(len(df_columns)), lasso_coef)\n",
        "plt.xticks(range(len(df_columns)), df_columns.values, rotation=60)\n",
        "plt.margins(0.02)\n",
        "plt.show()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CIOncXn61bBV"
      },
      "source": [
        "Great work! According to the lasso algorithm, it seems like 'child_mortality' is the most important feature when predicting life expectancy.\n",
        "\n",
        "![](https://drive.google.com/uc?export=view&id=1cJ2kH4MZNsfdv5_5v9CdgPjGebOJtpaQ)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "otLCQ6K3PeHz"
      },
      "source": [
        "### **2.4.2- Regularization II: Ridge**\n",
        "\n",
        "Lasso is great for feature selection, but when building regression models, Ridge regression should be your first choice.\n",
        "\n",
        "Recall that lasso performs regularization by adding to the loss function a penalty term of the absolute value of each coefficient multiplied by some alpha. This is also known as  regularization because the regularization term is the  norm of the coefficients. This is not the only way to regularize, however.\n",
        "\n",
        "If instead you took the sum of the squared values of the coefficients multiplied by some alpha - like in Ridge regression - you would be computing the  norm. In this exercise, you will practice fitting ridge regression models over a range of different alphas, and plot cross-validated  scores for each, using this function that we have defined for you, which plots the  score as well as standard error for each alpha:\n",
        "\n",
        "def display_plot(cv_scores, cv_scores_std):\n",
        "    fig = plt.figure()\n",
        "    ax = fig.add_subplot(1,1,1)\n",
        "    ax.plot(alpha_space, cv_scores)\n",
        "\n",
        "    std_error = cv_scores_std / np.sqrt(10)\n",
        "\n",
        "    ax.fill_between(alpha_space, cv_scores + std_error, cv_scores - std_error, alpha=0.2)\n",
        "    ax.set_ylabel('CV Score +/- Std Error')\n",
        "    ax.set_xlabel('Alpha')\n",
        "    ax.axhline(np.max(cv_scores), linestyle='--', color='.5')\n",
        "    ax.set_xlim([alpha_space[0], alpha_space[-1]])\n",
        "    ax.set_xscale('log')\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nQcuqs19PwMO"
      },
      "source": [
        "Don't worry about the specifics of the above function works. The motivation behind this exercise is for you to see how the  score varies with different alphas, and to understand the importance of selecting the right value for alpha. You'll learn how to tune alpha in the next chapter."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RGbF3ZNBP1j0"
      },
      "source": [
        "**Instruction**\n",
        "\n",
        "- Instantiate a Ridge regressor and specify normalize=True.\n",
        "- Inside the for loop:\n",
        "-  Specify the alpha value for the regressor to use.\n",
        "-  Perform 10-fold cross-validation on the regressor with the specified alpha. The data is available in the arrays X and y.\n",
        "-  Append the average and the standard deviation of the computed cross-validated scores. NumPy has been pre-imported for you as np.\n",
        "- Use the display_plot() function to visualize the scores and standard deviations.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UGQ6bmSnR2OF"
      },
      "source": [
        "# Import necessary modules\n",
        "from sklearn.linear_model import Ridge\n",
        "from sklearn.model_selection import cross_val_score\n",
        "\n",
        "# Setup the array of alphas and lists to store scores\n",
        "alpha_space = np.logspace(-4, 0, 50)\n",
        "ridge_scores = []\n",
        "ridge_scores_std = []\n",
        "\n",
        "# Create a ridge regressor: ridge\n",
        "ridge = Ridge(normalize=True)\n",
        "\n",
        "# Compute scores over range of alphas\n",
        "for alpha in alpha_space:\n",
        "\n",
        "    # Specify the alpha value to use: ridge.alpha\n",
        "    ridge.alpha = alpha\n",
        "    \n",
        "    # Perform 10-fold CV: ridge_cv_scores\n",
        "    ridge_cv_scores = cross_val_score(ridge, X, y, cv=10)\n",
        "    \n",
        "    # Append the mean of ridge_cv_scores to ridge_scores\n",
        "    ridge_scores.append(np.mean(ridge_cv_scores))\n",
        "    \n",
        "    # Append the std of ridge_cv_scores to ridge_scores_std\n",
        "    ridge_scores_std.append(np.std(ridge_cv_scores))\n",
        "\n",
        "# Display the plot\n",
        "display_plot(ridge_scores, ridge_scores_std)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6ZxsvuqvR3wi"
      },
      "source": [
        "Great work! Notice how the cross-validation scores change with different alphas. Which alpha should you pick? How can you fine-tune your model? You'll learn all about this in the next chapter!\n",
        "![](https://drive.google.com/uc?export=view&id=1nrq1hdEXwbwcTpIq5Ycn9yEWxZM1Ht7X)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **2.5-The Elastic Net Regression Algorithm**"
      ],
      "metadata": {
        "id": "kdF6tZSBcK6q"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Import library\n",
        "from sklearn.linear_model import ElasticNet\n",
        "# Building elastic net regression model with hyperparameter alpha = 0.1\n",
        "regr = ElasticNet(alpha=0.1)\n",
        "# Prepare input data\n",
        "X = np.array([[1, 1], [1, 2], [1, 3], [2, 1], [2, 2], [2, 3]])\n",
        "y = [ 6,  8, 10,  7,  9, 11]\n",
        "regr.fit(X, y)\n",
        "regr.coef_\n",
        "# array([0.66666667, 1.79069767])\n",
        "regr.intercept_)\n",
        "# 3.9186046511627914\n",
        "regr.predict([[4, 4]])\n",
        "# array([13.74806202])\n",
        "regr.predict([[0, 0]])\n",
        "# array([20.45348837])"
      ],
      "metadata": {
        "id": "C4qOdwKecBN1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ih3D_CAyTBg9"
      },
      "source": [
        "# **3-Fine-tuning your model**\n",
        "Having trained your model, your next task is to evaluate its performance. In this chapter, you will learn about some of the other metrics available in scikit-learn that will allow you to assess your model's performance in a more nuanced manner. Next, learn to optimize your classification and regression models using hyperparameter tuning.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0MOZg_x0TT9h"
      },
      "source": [
        "## **3.1 How good is your model?**\n",
        "\n",
        "**1. How good is your model?**\n",
        "\n",
        "In classification,\n",
        "\n",
        "**2. Classification metrics**\n",
        "\n",
        "we've seen that you can use accuracy, the fraction of correctly classified samples, to measure model performance. However, accuracy is not always a useful metric.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EY7N8wBDsZSE"
      },
      "source": [
        "**3. Class imbalance example: Emails**\n",
        "\n",
        "Consider a spam classification problem in which 99% of emails are real and only 1% are spam. I could build a model that classifies all emails as real; this model would be correct 99% of the time and thus have an accuracy of 99%, which sounds great. However, this naive classifier does a horrible job of predicting spam: it never predicts spam at all, so it completely fails at its original purpose. The situation when one class is more frequent is called class imbalance because the class of real emails contains way more instances than the class of spam. This is a very common situation in practice and requires a more nuanced metric to assess the performance of our model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lfW1UQV2s3zl"
      },
      "source": [
        "**4. Diagnosing classification predictions**\n",
        "\n",
        "Given a binary classifier, such as our spam email example, we can draw up a 2-by-2 matrix that summarizes predictive performance called a confusion matrix:\n",
        "\n",
        "**5. Diagnosing classification predictions**\n",
        "\n",
        "across the top are the predicted labels,"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vt8-xU7Oty7C"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1nlKnpg8O_6XXcNwhiUMGthkNFPmhPIYq)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3vmasbdtuN45"
      },
      "source": [
        "**6. Diagnosing classification predictions**\n",
        "\n",
        "down the side the actual labels,\n",
        "\n",
        "**7. Diagnosing classification predictions**\n",
        "\n",
        "such as you see here. Given any model, we can fill in the confusion matrix according to its predictions.\n",
        "\n",
        "**8. Diagnosing classification predictions**\n",
        "\n",
        "In the top left square, we have the number of spam emails correctly labeled;\n",
        "\n",
        "**9. Diagnosing classification predictions**\n",
        "\n",
        "in the bottom right square, we have the number of real emails correctly labeled;\n",
        "\n",
        "**10. Diagnosing classification predictions**\n",
        "\n",
        "in the top right, the number of spam emails incorrectly labeled;\n",
        "\n",
        "**11. Diagnosing classification predictions**\n",
        "\n",
        "and in the bottom left, the number of real emails incorrectly labeled. Note that correctly labeled spam emails\n",
        "\n",
        "**12. Diagnosing classification predictions**\n",
        "\n",
        "are referred to as true positives and\n",
        "\n",
        "**13. Diagnosing classification predictions**\n",
        "\n",
        "correctly labeled real emails as true negatives.\n",
        "\n",
        "**14. Diagnosing classification predictions**\n",
        "\n",
        "While incorrectly labeled spam will be referred to as false negatives and\n",
        "\n",
        "15. **Diagnosing classification predictions**\n",
        " \n",
        "incorrectly labeled real emails as false positives."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xk9NMjS_WIrX"
      },
      "source": [
        "**6. Diagnosing classification predictions**\n",
        "\n",
        "Usually, the \"class of interest\" is called the positive class. As we are trying to detect spam, this makes spam the positive class. Which class you call positive is really up to you. So why do we care about the confusion matrix? First, notice that you can retrieve accuracy from the confusion matrix: it's the sum of the diagonal divided by the total sum of the matrix.\n",
        "\n",
        "**17. Metrics from the confusion matrix**\n",
        "\n",
        "There are several other important metrics you can easily calculate from the confusion matrix. Precision, which is the number of true positives divided by the total number of true positives and false positives. It is also called the positive predictive value or PPV. In our case, this is the number of correctly labeled spam emails divided by the total number of emails classified as spam. Recall, which is the number of true positives divided by the total number of true positives and false negatives. This is also called sensitivity, hit rate, or true positive rate. The F1-score is defined as two times the product of the precision and recall divided by the sum of the precision and recall, in other words, it's the harmonic mean of precision and recall. To put it in plain language, high precision means that our classifier had a low false positive rate, that is, not many real emails were predicted as being spam. Intuitively, high recall means that our classifier predicted most positive or spam emails correctly."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G8G2Qwi1WRlr"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1gfxitY4zQpgraBU_N2nyH0CViEIuE98U)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zr9h2GBOXBsV"
      },
      "source": [
        "**18. Confusion matrix in scikit-learn**\n",
        "\n",
        "Let's now compute the confusion matrix, along with the metrics for the classifier we trained on the voting dataset in the exercise. We import classification report and confusion matrix from sklearn dot metrics. As always, we instantiate our classifier, split the data into train and test, fit the training data, and predict the labels of the test set.\n",
        "\n",
        "**19. Confusion matrix in scikit-learn**\n",
        "\n",
        "To compute the confusion matrix, we pass the test set labels and the predicted labels to the function confusion matrix. To compute the resulting metrics, we pass the same arguments to classification report, which outputs a string containing all the relevant metrics, which you can see here. For all metrics in scikit-learn, the first argument is always the true label and the prediction is always the second argument."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RuV5PLkTX70S"
      },
      "source": [
        "In [1]: from sklearn.metrics import classification_report\n",
        "In [2]: from sklearn.metrics import confusion_matrix\n",
        "In [3]: knn = KNeighborsClassifier(n_neighbors=8)\n",
        "In [4]: X_train, X_test, y_train, y_test = train_test_split(X, y,\n",
        " ...: test_size=0.4, random_state=42)\n",
        "In [5]: knn.fit(X_train, y_train)\n",
        "In [6]: y_pred = knn.predict(X_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZnLtMydgX9D3"
      },
      "source": [
        "In [7]: print(confusion_matrix(y_test, y_pred))\n",
        "[[52 7]\n",
        " [ 3 112]]\n",
        "In [8]: print(classification_report(y_test, y_pred))\n",
        " precision recall f1-score support\n",
        " 0 0.95 0.88 0.91 59\n",
        " 1 0.94 0.97 0.96 115\n",
        "avg / total 0.94 0.94 0.94 174"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U6o3Z6AN4gbC"
      },
      "source": [
        "### **3.1.1 Metrics for classification**\n",
        "\n",
        "In Chapter 1, you evaluated the performance of your k-NN classifier based on its accuracy. However, as Andy discussed, accuracy is not always an informative metric. In this exercise, you will dive more deeply into evaluating the performance of binary classifiers by computing a confusion matrix and generating a classification report.\n",
        "\n",
        "You may have noticed in the video that the classification report consisted of three rows, and an additional support column. The support gives the number of samples of the true response that lie in that class - so in the video example, the support was the number of Republicans or Democrats in the test set on which the classification report was computed. The precision, recall, and f1-score columns, then, gave the respective metrics for that particular class.\n",
        "\n",
        "Here, you'll work with the PIMA Indians dataset obtained from the UCI Machine Learning Repository. The goal is to predict whether or not a given female patient will contract diabetes\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0Wl5vp51CNrA"
      },
      "source": [
        "**instruction**\n",
        "\n",
        "- Import classification_report and confusion_matrix from sklearn.metrics.\n",
        "- Create training and testing sets with 40% of the data used for testing. Use a random state of 42.\n",
        "- Instantiate a k-NN classifier with 6 neighbors, fit it to the training data, and predict the labels of the test set.\n",
        "- Compute and print the confusion matrix and classification report using the confusion_matrix() and classification_report() functions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "99vyYlnhFjMK"
      },
      "source": [
        "# Import necessary modules\n",
        "from sklearn.metrics import classification_report\n",
        "from sklearn.metrics import confusion_matrix\n",
        "\n",
        "# Create training and test set\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4,random_state=42)\n",
        "\n",
        "# Instantiate a k-NN classifier: knn\n",
        "knn = KNeighborsClassifier(n_neighbors=6)\n",
        "\n",
        "# Fit the classifier to the training data\n",
        "knn.fit(X_train, y_train)\n",
        "\n",
        "# Predict the labels of the test data: y_pred\n",
        "y_pred = knn.predict(X_test)\n",
        "\n",
        "# Generate the confusion matrix and classification report\n",
        "print(confusion_matrix(y_test, y_pred))\n",
        "print(classification_report(y_test, y_pred))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fxtU8iJNFktu"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=11vh8y-tGo7n2ZjtHqjfYCC1mxVY-5gNF)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2NC-1uxYGfJd"
      },
      "source": [
        "Excellent work! By analyzing the confusion matrix and classification report, you can get a much better understanding of your classifier's performance."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QrY9j032a1yC"
      },
      "source": [
        "## **3.2 Logistic regression and the ROC curve**\n",
        "\n",
        "Logistic regression and the ROC curve\n",
        "It's time to introduce another model to your classification arsenal: logistic regression. Despite its name, logistic regression is used in classification problems, not regression problems. We won't go into the mathematical details here, see our stats courses for that, but we will provide an intuition towards how logistic regression or log reg works for binary classification, that is, when we have two possible labels for the target variable.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7sFpd7hNbiko"
      },
      "source": [
        "**2. Logistic regression for binary classification**\n",
        "\n",
        "Given one feature, log reg will output a probability, p, with respect to the target variable. If p is greater than 0 (point) 5, we label the data as '1'; if p is less than 0 (point) 5, we label it '0'."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p0YNIDKRbp7-"
      },
      "source": [
        "**3. Linear decision boundary**\n",
        "\n",
        "Note that log reg produces a linear decision boundary. Using logistic regression in scikit-learn"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RY1PE7SmmO2n"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=17j-UfyVncDMTjG03DzHThC0jASmD4_2i\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NnFFkTcjmrTK"
      },
      "source": [
        "**4. Logistic regression in scikit-learn**\n",
        "\n",
        "follows exactly the same formula that you now know so well: perform the necessary imports, instantiate the classifier, split your data into training and test sets, fit the model on your training data, and predict on your test set. Here we have used the voting dataset that you worked with earlier in the course."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xN3Vxy4Vmwmq"
      },
      "source": [
        "In [1]: from sklearn.linear_model import LogisticRegression\n",
        "In [2]: from sklearn.model_selection import train_test_split\n",
        "In [3]: logreg = LogisticRegression()\n",
        "In [4]: X_train, X_test, y_train, y_test = train_test_split(X, y,\n",
        " ...: test_size=0.4, random_state=42)\n",
        "In [5]: logreg.fit(X_train, y_train)\n",
        "In [6]: y_pred = logreg.predict(X_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-_BCfzo4nBLw"
      },
      "source": [
        "**5. Probability thresholds**\n",
        "\n",
        "Notice that in defining logistic regression, we have specified a threshold of 0 (point) 5 for the probability, a threshold that defines our model. Note that this is not particular for log reg but also could be used for KNN, for example. Now, what happens as we vary this threshold?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yesfJkKCrN2u"
      },
      "source": [
        "**6. The ROC curve**\n",
        "\n",
        "In particular, what happens to the true positive and false positive rates as we vary the threshold?\n",
        "\n",
        "**7. The ROC curve**\n",
        "\n",
        "When the threshold equals zero, the model predicts '1' for all the data, which means the true positive rate is equal to the false positive rate\n",
        "\n",
        "**8. The ROC curve**\n",
        "\n",
        "is equal to one. When the threshold\n",
        "\n",
        "**9. The ROC curve**\n",
        "\n",
        "equals '1', the model predicts '0' for all data, which means that both true and false positive rates\n",
        "\n",
        "**10. The ROC curve**\n",
        "\n",
        "are 0. If we\n",
        "\n",
        "**11. The ROC curve**\n",
        "\n",
        "vary the threshold between these two extremes, we get a series of different false positive and true positive rates.\n",
        "\n",
        "**12. The ROC curve**\n",
        "\n",
        "The set of points we get when trying all possible thresholds is called the receiver operating characteristic curve or ROC curve."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eK0odl8TrexE"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1SBYB7TpC1gda-pUktjtRr2xW6FjgsV7c\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EHHuDLNbwSkt"
      },
      "source": [
        "**13. Plotting the ROC curve**\n",
        "\n",
        "To plot the ROC curve, we import roc curve from sklearn dot metrics; we then call the function roc curve; the first argument is given by the actual labels, the second by the predicted probabilities. A word on this in a second. We unpack the result into three variables: false positive rate, FPR; true positive rate, TPR; and the thresholds. We can then plot the FPR and TPR using pyplot's plot function to produce a figure such as this.\n",
        "\n",
        "**14. Plotting the ROC curve**\n",
        "\n",
        "We used the predicted probabilities of the model assigning a value of '1' to the observation in question. This is because to compute the ROC we do not merely want the predictions on the test set, but we want the probability that our log reg model outputs before using a threshold to predict the label. To do this we apply the method predict proba to the model and pass it the test data. predict proba returns an array with two columns: each column contains the probabilities for the respective target values. We choose the second column, the one with index 1, that is, the probabilities of the predicted labels being '1'."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1hXuMBYfwedQ"
      },
      "source": [
        "In [1]: from sklearn.metrics import roc_curve\n",
        "In [2]: y_pred_prob = logreg.predict_proba(X_test)[:,1]\n",
        "In [3]: fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob)\n",
        "In [4]: plt.plot([0, 1], [0, 1], 'k--')\n",
        "In [5]: plt.plot(fpr, tpr, label='Logistic Regression')\n",
        "In [6]: plt.xlabel('False Positive Rate’)\n",
        "In [7]: plt.ylabel('True Positive Rate')\n",
        "In [8]: plt.title('Logistic Regression ROC Curve')\n",
        "In [9]: plt.show();"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lYrrKY9Mwfvg"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1UdvH-85QXAKmzWdOzrvbVmOr9IECMadD)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cjhx9M59TkDz"
      },
      "source": [
        "### **3.2.1 Building a logistic regression model**\n",
        "\n",
        "Time to build your first logistic regression model! As Hugo showed in the video, scikit-learn makes it very easy to try different models, since the Train-Test-Split/Instantiate/Fit/Predict paradigm applies to all classifiers and regressors - which are known in scikit-learn as 'estimators'. You'll see this now for yourself as you train a logistic regression model on exactly the same data as in the previous exercise. Will it outperform k-NN? There's only one way to find out!\n",
        "\n",
        "The feature and target variable arrays X and y have been pre-loaded, and train_test_split has been imported for you from sklearn.model_selection.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D_gfOahRT23b"
      },
      "source": [
        "**Instruction**\n",
        "\n",
        "- Import:\n",
        "LogisticRegression from sklearn.linear_model.\n",
        "confusion_matrix and classification_report from sklearn.metrics.\n",
        "- Create training and test sets with 40% (or 0.4) of the data used for testing. Use a random state of 42. This has been done for you.\n",
        "- Instantiate a LogisticRegression classifier called logreg.\n",
        "Fit the classifier to the training data and predict the labels of the test set.\n",
        "- Compute and print the confusion matrix and classification report. This has been done for you, so hit 'Submit Answer' to see how logistic regression compares to k-NN!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RJ8ytKwjWdqX"
      },
      "source": [
        "# Import the necessary modules\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.metrics import confusion_matrix, classification_report\n",
        "\n",
        "# Create training and test sets\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.4, random_state=42)\n",
        "\n",
        "# Create the classifier: logreg\n",
        "logreg = LogisticRegression()\n",
        "\n",
        "# Fit the classifier to the training data\n",
        "logreg.fit(X_train, y_train\n",
        "\n",
        "# Predict the labels of the test set: y_pred\n",
        "y_pred = logreg.predict(X_test)\n",
        "# Compute and print the confusion matrix and classification report\n",
        "print(confusion_matrix(y_test, y_pred))\n",
        "print(classification_report(y_test, y_pred))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PdQ1NqG9Wrjr"
      },
      "source": [
        "### **3.2.2 Plotting an ROC curve**\n",
        "\n",
        "Great job in the previous exercise - you now have a new addition to your toolbox of classifiers!\n",
        "\n",
        "Classification reports and confusion matrices are great methods to quantitatively evaluate model performance, while ROC curves provide a way to visually evaluate models. As Hugo demonstrated in the video, most classifiers in scikit-learn have a .predict_proba() method which returns the probability of a given sample being in a particular class. Having built a logistic regression model, you'll now evaluate its performance by plotting an ROC curve. In doing so, you'll make use of the .predict_proba() method and become familiar with its functionality.\n",
        "\n",
        "Here, you'll continue working with the PIMA Indians diabetes dataset. The classifier has already been fit to the training data and is available as logreg.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FynRJ8XEW_lx"
      },
      "source": [
        "**Instruction**\n",
        "\n",
        "- Import roc_curve from sklearn.metrics.\n",
        "- Using the logreg classifier, which has been fit to the training data, compute the predicted probabilities of the labels of the test set X_test. Save the result as y_pred_prob.\n",
        "- Use the roc_curve() function with y_test and y_pred_prob and unpack the result into the variables fpr, tpr, and thresholds.\n",
        "- Plot the ROC curve with fpr on the x-axis and tpr on the y-axis."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pBl6HGm6X8cN"
      },
      "source": [
        "# Import necessary modules\n",
        "from sklearn.metrics import roc_curve\n",
        "# Compute predicted probabilities: y_pred_prob\n",
        "y_pred_prob = logreg.predict_proba(X_test)[:,1]\n",
        "\n",
        "# Generate ROC curve values: fpr, tpr, thresholds\n",
        "fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob)\n",
        "# Plot ROC curve\n",
        "plt.plot([0, 1], [0, 1], 'k--')\n",
        "plt.plot(fpr, tpr)\n",
        "plt.xlabel('False Positive Rate')\n",
        "plt.ylabel('True Positive Rate')\n",
        "plt.title('ROC Curve')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UJ_wzSwwYwvW"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1PT7ktkqS0eX4saM2U9CCyykOiJGlxroP\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "adf0BUGeX_pU"
      },
      "source": [
        "Excellent! This ROC curve provides a nice visual way to assess your classifier's performance."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_z0Uxax9ZGc7"
      },
      "source": [
        "### **3.2.3 Precision-recall Curve**\n",
        "\n",
        "When looking at your ROC curve, you may have noticed that the y-axis (True positive rate) is also known as recall. Indeed, in addition to the ROC curve, there are other ways to visually evaluate model performance. One such way is the precision-recall curve, which is generated by plotting the precision and recall for different thresholds. As a reminder, precision and recall are defined as:\n",
        "\n",
        "On the right, a precision-recall curve has been generated for the diabetes dataset. The classification report and confusion matrix are displayed in the IPython Shell.\n",
        "\n",
        "Study the precision-recall curve and then consider the statements given below. Choose the one statement that is not true. Note that here, the class is positive (1) if the individual has diabetes.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gQBsQZ4-ZyUE"
      },
      "source": [
        "# Import necessary modules\n",
        "from sklearn.metrics import roc_curve\n",
        "# Compute predicted probabilities: y_pred_prob\n",
        "y_pred_prob = logreg.predict_proba(X_test)[:,1]\n",
        "# Generate ROC curve values: fpr, tpr, thresholds\n",
        "fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob)\n",
        "\n",
        "# Plot ROC curve\n",
        "plt.plot([0, 1], [0, 1], 'k--')\n",
        "plt.plot(fpr, tpr)\n",
        "plt.xlabel('False Positive Rate')\n",
        "plt.ylabel('True Positive Rate')\n",
        "plt.title('ROC Curve')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "78x_sLc8Z1l0"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1xD_vW4JGzTXQeuNg1HijWG71kosXy8HS)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fdRpm0Lvar0Y"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1qEhN3cI2QRwQ5-sFxvfPCGyAx3nb76pT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZEGB4ZH-bJaX"
      },
      "source": [
        "**Possible Answers**\n",
        "\n",
        "- A recall of 1 corresponds to a classifier with a low threshold in which all females who contract diabetes were correctly classified as such, at the expense of many misclassifications of those who did not have diabetes.\n",
        "\n",
        "- Precision is undefined for a classifier which makes no positive predictions, that is, classifies everyone as not having diabetes.\n",
        "\n",
        "\n",
        "- When the threshold is very close to 1, precision is also 1, because the classifier is absolutely certain about its predictions.\n",
        "\n",
        "- **Precision and recall take true negatives into consideration.**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IvaYtqjjbwIE"
      },
      "source": [
        "Great work! True negatives do not appear at all in the definitions of precision and recall."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QAbl8L21fjnf"
      },
      "source": [
        "## **3.3 Area under the ROC curve**\n",
        "\n",
        "**1. Area under the ROC curve**\n",
        "\n",
        "Now the question is: given the ROC curve, can we extract a metric of interest?\n",
        "\n",
        "**2. Area under the ROC curve (AUC)**\n",
        "\n",
        "Consider the following: the larger the area under the ROC curve, the better our model is! The way to think about this is the following: if we had a model which produced an ROC curve that had a single point at 1,0, the upper left corner, representing a true positive rate of one and a false positive rate of zero, this would be a great model! For this reason, the area under the ROC, commonly denoted as AUC, is another popular metric for classification models.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qv0V5eqZfJ_z"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=16NExDrHAbis_r8Z_RpYJ18aK_CjInBMF)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NdK90Kqqfd3o"
      },
      "source": [
        "**6. AUC in scikit-learn**\n",
        "\n",
        "To compute the AUC, we import roc auc score from sklearn dot metrics. We instantiate our classifier, split our data into train and test sets, and fit the model to the training set. To compute the AUC, we first compute the predicted probabilities as above and then pass the true labels and the predicted probabilities to roc auc score. We can also compute the AUC using cross-validation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CL__xcpkgy8W"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1uT89wr6rdcsEqHhLgvmzkllA20TDxRXa)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Jwd95-rg_h9"
      },
      "source": [
        "**7. AUC using cross-validation**\n",
        "\n",
        "To do so, we import and use the function cross val score as before, passing it the estimator, the features, and the target. We then additionally pass it the keyword argument scoring equals \"roc auc\" and print the AUC list as you can see here."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s3kAA3dthT-r"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1WN1HoAwU3AXCTpw6V5W3S7Vqt6lZdObl)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CFMv0K9-18zw"
      },
      "source": [
        "### **3.3.1 AUC computation**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z9XRsqeu3nEt"
      },
      "source": [
        "import pandas as pd"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9DZrnY2X2L91"
      },
      "source": [
        "dia = pd.read_csv('/content/drive/MyDrive/Datasets/Supervised_learning_with_Sklearn/diabetes.csv')\n",
        "dia.head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0wo_j5Gv3ty_"
      },
      "source": [
        "# ['pregnancies', 'glucose', 'diastolic', 'triceps', 'insulin', 'bmi', 'dpf', 'age']\n",
        "\n",
        "y = dia['diabetes']\n",
        "\n",
        "X = dia.drop(['diabetes'], axis=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hbw8A1PJ3y-P"
      },
      "source": [
        "# Import necessary modules\n",
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import classification_report\n",
        "from sklearn.metrics import confusion_matrix"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k048zzBp33ZP"
      },
      "source": [
        "# Create training and test set\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42, test_size=.4)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V6M33hJk38ew"
      },
      "source": [
        "# Instantiate a k-NN classifier: knn\n",
        "knn = KNeighborsClassifier(n_neighbors=6)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ROSWwh2q4AB9",
        "outputId": "3bf93dd6-1f5f-4868-c6b3-7e8aa8528bc7"
      },
      "source": [
        "# Fit the classifier to the training data\n",
        "knn.fit(X_train, y_train)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
              "                     metric_params=None, n_jobs=None, n_neighbors=6, p=2,\n",
              "                     weights='uniform')"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QAP7PZQi4EzV"
      },
      "source": [
        "# Predict the labels of the test data: y_pred\n",
        "y_pred = knn.predict(X_test)\n",
        "\n",
        "# Generate the confusion matrix and classification report\n",
        "print(confusion_matrix(y_test, y_pred))\n",
        "print(classification_report(y_test, y_pred))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aQs5slma9VPY",
        "outputId": "9bb41641-a114-484d-d431-6db8d209ad9d"
      },
      "source": [
        "# Import the necessary modules\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.metrics import classification_report, confusion_matrix\n",
        "\n",
        "# Create training and test sets\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.4, random_state=42)\n",
        "\n",
        "# Create the classifier: logreg\n",
        "logreg = LogisticRegression()\n",
        "\n",
        "# Fit the classifier to the training data\n",
        "logreg.fit(X_train, y_train)\n",
        "\n",
        "# Predict the labels of the test set: y_pred\n",
        "y_pred = logreg.predict(X_test)\n",
        "\n",
        "# Compute and print the confusion matrix and classification report\n",
        "print(confusion_matrix(y_test, y_pred))\n",
        "print(classification_report(y_test, y_pred))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[170  36]\n",
            " [ 36  66]]\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.83      0.83      0.83       206\n",
            "           1       0.65      0.65      0.65       102\n",
            "\n",
            "    accuracy                           0.77       308\n",
            "   macro avg       0.74      0.74      0.74       308\n",
            "weighted avg       0.77      0.77      0.77       308\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
            "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
            "\n",
            "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
            "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
            "Please also refer to the documentation for alternative solver options:\n",
            "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
            "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TebOW5nb4HcM"
      },
      "source": [
        "# Import necessary modules\n",
        "from sklearn.metrics import roc_auc_score\n",
        "from sklearn.model_selection import cross_val_score\n",
        "\n",
        "# Compute predicted probabilities: y_pred_prob\n",
        "y_pred_prob = logreg.predict_proba(X_test)[:,1]\n",
        "\n",
        "# Compute and print AUC score\n",
        "print(\"AUC: {}\".format(roc_auc_score(y_test,y_pred_prob)))\n",
        "\n",
        "# Compute cross-validated AUC scores: cv_auc\n",
        "cv_auc = cross_val_score(logreg, X, y, cv=5, scoring='roc_auc')\n",
        "\n",
        "# Print list of AUC scores\n",
        "print(\"AUC scores computed using 5-fold cross-validation: {}\".format(cv_auc))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dgx6nzey-5KJ"
      },
      "source": [
        "## 3.4 **Hyperparameter tuning**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "10Fc_6C9UkU5"
      },
      "source": [
        "**1. Hyperparameter tuning**\n",
        "\n",
        "Welcome back. Now that you're developing a feel for judging how well your models are performing, it is time to supercharge them.\n",
        "\n",
        "**2. Hyperparameter tuning**\n",
        "\n",
        "We have seen that when fitting a linear regression, what we are really doing is choosing parameters for the model that fit the data the best. We also saw that we had to choose a value for the alpha in ridge and lasso regression before fitting it. Analogously, before fitting and predicting K-nearest neighbors, we need to choose n neighbors. Such parameters, ones that need to be specified before fitting a model, are called hyperparameters. In other words, these are parameters that cannot be explicitly learned by fitting the model. Herein lies a fundamental key for building a successful model:\n",
        "\n",
        "**3. Choosing the correct hyperparameter**\n",
        "\n",
        "choosing the correct hyperparameter. The basic idea is to try a whole bunch of different values, fit all of them separately, see how well each performs, and choose the best one! This is called hyperparameter tuning and doing so in this fashion is the current standard. There may be a better way, however, and if you figure it out, I'd be surprised if it wouldn't make you famous. Now, when fitting different values of a hyperparameter, it is essential to use cross-validation as using train test split alone would risk overfitting the hyperparameter to the test set. We'll see in the next video that, even after tuning our hyperparameters using cross-validation, we'll want to have already split off a test set in order to report how well our model can be expected to perform on a dataset that it has never seen before.\n",
        "\n",
        "**4. Grid search cross-validation**\n",
        "\n",
        "The basic idea is as follows: we choose a grid of possible values we want to try for the hyperparameter or hyperparameters. For example, if we had two hyperparameters, C and alpha, the grid of values to test could look like this.\n",
        "\n",
        "**5. Grid search cross-validation**\n",
        "\n",
        "We then perform k-fold cross-validation for each point in the grid, that is, for each choice of hyperparameter or combination of hyperparameters.\n",
        "\n",
        "**6. Grid search cross-validation**\n",
        "\n",
        "We then choose for our model the choice of hyperparameters that performed the best! This is called a grid search and in scikit-learn we implement it using the class GridSearchCV.\n",
        "\n",
        "**7. GridSearchCV in scikit-learn**\n",
        "\n",
        "We import GridSearchCV from sklearn dot model selection. We then specify the hyperparameter as a dictionary in which the keys are the hyperparameter names, such as n neighbors in KNN or alpha in lasso regression. See the documentation of each model for the name of its hyperparameters. The values in the grid dictionary are lists containing the values we wish to tune the relevant hyperparameter or hyperparameters over. If we specify multiple parameters, all possible combinations will be tried. As always, we instantiate our classifier. We then use GridSearchCV and pass it our model, the grid we wish to tune over and the number of folds that we wish to use. This returns a GridSearch object that you can then fit to the data and this fit performs the actual grid search inplace. We can then apply the attributes best params and best score, respectively, to retrieve the hyperparameters that perform the best along with the mean cross-validation score over that fold.\n",
        "\n",
        "![](https://drive.google.com/uc?export=view&id=1ukiFn7TDo5ZfLo2jE4uq5KUz0tMGWQNX)\n",
        "\n",
        "**8. Let's practice!**\n",
        "\n",
        "Now, it's your turn to practice your new grid search cross-validation chops. You'll also learn about RandomizedSearchCV, which is similar to GridSearch, except that it is able to jump around the grid. Happy grid searching!!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DD06XOWy-AE_"
      },
      "source": [
        "###**3.4.1 Hyperparameter tuning with GridSearchCV**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cg6hF3mPlqZj"
      },
      "source": [
        "\n",
        "\n",
        "Hugo demonstrated how to tune the n_neighbors parameter of the KNeighborsClassifier() using GridSearchCV on the voting dataset. You will now practice this yourself, but by using logistic regression on the diabetes dataset instead!\n",
        "\n",
        "Like the alpha parameter of lasso and ridge regularization that you saw earlier, logistic regression also has a regularization parameter: .  controls the inverse of the regularization strength, and this is what you will tune in this exercise. A large  can lead to an overfit model, while a small  can lead to an underfit model.\n",
        "\n",
        "The hyperparameter space for  has been setup for you. Your job is to use GridSearchCV and logistic regression to find the optimal  in this hyperparameter space. The feature array is available as X and target variable array is available as y.\n",
        "\n",
        "You may be wondering why you aren't asked to split the data into training and test sets. Good observation! Here, we want you to focus on the process of setting up the hyperparameter grid and performing grid-search cross-validation. In practice, you will indeed want to hold out a portion of your data for evaluation purposes, and you will learn all about this in the next video!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GzZG8x9kl0fR"
      },
      "source": [
        "**Instruction** "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jc2O44zJl6XZ"
      },
      "source": [
        "- Import LogisticRegression from sklearn.linear_model and GridSearchCV from sklearn.model_selection.\n",
        "- Setup the hyperparameter grid by using c_space as the grid of values to tune  over.\n",
        "- Instantiate a logistic regression classifier called logreg.\n",
        "- Use GridSearchCV with 5-fold cross-validation to tune :\n",
        "- Inside GridSearchCV(), specify the classifier, parameter grid, and number of folds to use.\n",
        "- Use the .fit() method on the GridSearchCV object to fit it to the data X and y.\n",
        "- Print the best parameter and best score obtained from GridSearchCV by accessing the best_params_ and best_score_ attributes of logreg_cv."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8vkeLTEJnrmp"
      },
      "source": [
        "**Hints**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2EhVtg46-bvm"
      },
      "source": [
        "# Import necessary modules\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "import numpy as np\n",
        "\n",
        "# Setup the hyperparameter grid\n",
        "c_space = np.logspace(-5, 8, 15)\n",
        "param_grid = {'C': c_space}\n",
        "\n",
        "# Instantiate a logistic regression classifier: logreg\n",
        "logreg = LogisticRegression()\n",
        "\n",
        "# Instantiate the GridSearchCV object: logreg_cv\n",
        "logreg_cv = GridSearchCV(logreg, param_grid, cv=5)\n",
        "\n",
        "# Fit it to the data\n",
        "logreg_cv.fit(X,y)\n",
        "\n",
        "# Print the tuned parameters and score\n",
        "print(\"Tuned Logistic Regression Parameters: {}\".format(logreg_cv.best_params_)) \n",
        "print(\"Best score is {}\".format(logreg_cv.best_score_))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_xI3WlJ-Q8qB"
      },
      "source": [
        "### **3.4.2-Hyperparameter tuning with RandomizedSearchCV**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yez6hTpNRF-J"
      },
      "source": [
        "GridSearchCV can be computationally expensive, especially if you are searching over a large hyperparameter space and dealing with multiple hyperparameters. A solution to this is to use RandomizedSearchCV, in which not all hyperparameter values are tried out. Instead, a fixed number of hyperparameter settings is sampled from specified probability distributions. You'll practice using RandomizedSearchCV in this exercise and see how this works.\n",
        "\n",
        "Here, you'll also be introduced to a new model: the Decision Tree. Don't worry about the specifics of how this model works. Just like k-NN, linear regression, and logistic regression, decision trees in scikit-learn have .fit() and .predict() methods that you can use in exactly the same way as before. Decision trees have many parameters that can be tuned, such as max_features, max_depth, and min_samples_leaf: This makes it an ideal use case for RandomizedSearchCV.\n",
        "\n",
        "As before, the feature array X and target variable array y of the diabetes dataset have been pre-loaded. The hyperparameter settings have been specified for you. Your goal is to use RandomizedSearchCV to find the optimal hyperparameters. Go for it!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4bMmTl71RMDY"
      },
      "source": [
        "**Instruction**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XkunjQ23RVWJ"
      },
      "source": [
        "- Import DecisionTreeClassifier from sklearn.tree and RandomizedSearchCV from sklearn.model_selection.\n",
        "- Specify the parameters and distributions to sample from. This has been done for you.\n",
        "- Instantiate a DecisionTreeClassifier.\n",
        "- Use RandomizedSearchCV with 5-fold cross-validation to tune the hyperparameters:\n",
        "- Inside RandomizedSearchCV(), specify the classifier, parameter distribution, and number of folds to use.\n",
        "- Use the .fit() method on the RandomizedSearchCV object to fit it to the data X and y.\n",
        "- Print the best parameter and best score obtained from RandomizedSearchCV by accessing the best_params_ and best_score_ attributes of tree_cv"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yh_WCJxoRnLj"
      },
      "source": [
        "**Hints**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tybmRA9yRrhY"
      },
      "source": [
        "- Use the command from y import x to import x from y.\n",
        "- Use DecisionTreeClassifier() to instantiate a decision tree classifier.\n",
        "- Inside RandomizedSearchCV(), pass in the classifier tree, parameter - - distribution param_dist, and specify the number of folds to use. Then use the .- fit() method on this with X and y passed in as arguments.\n",
        "- Use the following format to access an attribute of an object: object_name.attribute_name. For example: tree_cv.best_score_."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c-NnavxpbDP8"
      },
      "source": [
        "![](\n",
        "https://drive.google.com/uc?export=view&id=14CEUdCyyZZY8nvCm_Hyz27T-CxN1gapu)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u1W3zkG7YGxH"
      },
      "source": [
        "## 3.5 **Hold-out set for final evaluation**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eiz6j3L8YVtk"
      },
      "source": [
        "**1. Hold-out set for final evaluation**\n",
        "\n",
        "Congrats on making it through those exercises. I hope that you had some serious fun with GridSearchCV and RandomizedSearchCV. After using K-fold cross-validation to tune my model's hyperparameters,\n",
        "\n",
        "**2. Hold-out set reasoning**\n",
        "\n",
        "I may want to report how well my model can be expected to perform on a dataset that it has never seen before, given my scoring function of choice. So, I want to use my model to predict on some labeled data, compare my prediction to the actual labels, and compute the scoring function. However, if I have used all of my data for cross-validation, estimating my model performance on any of it may not provide an accurate picture of how it will perform on unseen data. For this reason, it is important to split all of my data at the very beginning into a training set and hold-out set, then perform cross-validation on the training set to tune my model's hyperparameters. After this, I can select the best hyperparameters and use the hold-out set, which has not been used at all, to test how well the model can be expected to perform on a dataset that it has never seen before.\n",
        "\n",
        "**3. Let's practice!**\n",
        "\n",
        "You already have all the tools to perform this technique. Your old friend train test split and your new pal GridSearchCV will be particularly handy. Have a crack at it in the interactive exercises and we'll see you in the next chapter!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X4ToC80ZaLbi"
      },
      "source": [
        "### **3.5.1- Hold out set reasoning**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HcPjnDo9aVar"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1RDZRBUGJruIssmmfQawXherfChSoke5y)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_SAx2iYGnsBe"
      },
      "source": [
        "### **3.5.2-Hold-out set in practice I: Classification**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "87rUrVlpn8HN"
      },
      "source": [
        "You will now practice evaluating a model with tuned hyperparameters on a hold-out set. The feature array and target variable array from the diabetes dataset have been pre-loaded as X and y.\n",
        "\n",
        "In addition to , logistic regression has a 'penalty' hyperparameter which specifies whether to use 'l1' or 'l2' regularization. Your job in this exercise is to create a hold-out set, tune the 'C' and 'penalty' hyperparameters of a logistic regression classifier using GridSearchCV on the training set."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H0SZT9N3oKLT"
      },
      "source": [
        "**Instruction**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5wXsAuryoQ3T"
      },
      "source": [
        "- Create the hyperparameter grid:\n",
        "  - Use the array c_space as the grid of values for 'C'.\n",
        "  - For 'penalty', specify a list consisting of 'l1' and 'l2'.\n",
        "- Instantiate a logistic regression classifier.\n",
        "- Create training and test sets. Use a test_size of 0.4 and random_state of 42. - In practice, the test set here will function as the hold-out set.\n",
        "- Tune the hyperparameters on the training set using GridSearchCV with 5-folds. - This involves first instantiating the GridSearchCV object with the correct parameters and then fitting it to the training data.\n",
        "- Print the best parameter and best score obtained from GridSearchCV by accessing the best_params_ and best_score_ attributes of logreg_cv."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PHQkZPbvouAA"
      },
      "source": [
        "**Hints**\n",
        "\n",
        "- In the param_grid dictionary, use 'C' and c_space as the first key:value pair, and 'penalty' and ['l1', 'l2'] as the second.\n",
        "- Use LogisticRegression() to instantiate a logistic regression classifier.\n",
        "- Use train_test_split() to create training and test sets. Pass in the arguments X and y, and specify the keyword arguments test_size=0.4 and random_state=42.\n",
        "- Inside GridSearchCV(), pass in the classifier logreg, parameter grid - param_grid, and specify the number of folds to use. Then use the .fit() method on this with X_train and y_train passed in as arguments.\n",
        "- Use the following format to access an attribute of an object: object_name.attribute_name. For example: logreg_cv.best_score_."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uA-XXIzfpc0m"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1fhioqFHxyoJISY7xIAZKT_LkBhlFHVVi)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pao17I1yqow_"
      },
      "source": [
        "### **3.5.3 Hold-out set in practice II: Regression**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JlBIveuAqz1X"
      },
      "source": [
        "Remember lasso and ridge regression from the previous chapter? Lasso used the  penalty to regularize, while ridge used the  penalty. There is another type of regularized regression known as the elastic net. In elastic net regularization, the penalty term is a linear combination of the  and  penalties:\n",
        "\n",
        "\n",
        "In scikit-learn, this term is represented by the 'l1_ratio' parameter: An 'l1_ratio' of 1 corresponds to an  penalty, and anything lower is a combination of  and .\n",
        "\n",
        "In this exercise, you will GridSearchCV to tune the 'l1_ratio' of an elastic net model trained on the Gapminder data. As in the previous exercise, use a hold-out set to evaluate your model's performance."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "--U6yJ-_q1_P"
      },
      "source": [
        "**Instruction**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5BkrREBvq-ja"
      },
      "source": [
        "Import the following modules:\n",
        "- ElasticNet from sklearn.linear_model.\n",
        "- mean_squared_error from sklearn.metrics.\n",
        "- GridSearchCV and train_test_split from sklearn.model_selection."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xev44dcVrE0i"
      },
      "source": [
        "- Create training and test sets, with 40% of the data used for the test set. Use a random state of 42.\n",
        "- Specify the hyperparameter grid for 'l1_ratio' using l1_space as the grid of values to search over.\n",
        "- Instantiate the ElasticNet regressor.\n",
        "- Use GridSearchCV with 5-fold cross-validation to tune 'l1_ratio' on the - training data X_train and y_train. This involves first instantiating the - GridSearchCV object with the correct parameters and then fitting it to the training data.\n",
        "- Predict on the test set and compute the  and mean squared error."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h1jgsgp9rQHe"
      },
      "source": [
        "**Hindts**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MuzAkZXqrTDl"
      },
      "source": [
        "- Use the command from y import x to import x from y.\n",
        "- Use train_test_split() to create training and test sets. Pass in the arguments X and y, and specify the keyword arguments test_size=0.4 and random_state=42.\n",
        "- In the param_grid dictionary, use l1_space as the grid of values to tune 'l1_ratio' over.\n",
        "- Use ElasticNet() to instantiate the regressor.\n",
        "- Inside GridSearchCV(), pass in the regressor elastic_net, parameter grid - param_grid, and specify the number of folds to use. Then use the .fit() method - on this with X_train and y_train passed in as arguments.\n",
        "- To compute r2, use the .score() method with X_test and y_test as arguments. To compute mse, use the mean_squared_error() function with y_test and y_pred as arguments."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YumB76Xgr_cS"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1vbl2IzRcUBDd2EAYlpupH06Bg1kgjw6l)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HlLtamGjSpv_"
      },
      "source": [
        "# **4-Preprocessing and pipelines**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t_oq8i7ycTcf"
      },
      "source": [
        "## 4.2 **Preprocessing data**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bHaLkyvicY0I"
      },
      "source": [
        "**1. Preprocessing data**\n",
        "\n",
        "Welcome to the final chapter of this introductory course on supervised learning with scikit-learn! You have learnt how to implement both classification and regression models, how to measure model performance, and how to tune hyperparameters in order to improve performance. However, all the data that you have used so far has been relatively nice and in a format that allows you to plug and play into scikit-learn from the get-go. With real-world data, this will rarely be the case, and instead you will have to preprocess your data before you can build models. In this chapter, you will learn all about this vital preprocessing step.\n",
        "\n",
        "**2. Dealing with categorical features**\n",
        "\n",
        "Say you are dealing with a dataset that has categorical features, such as 'red' or 'blue', or 'male' or 'female'. As these are not numerical values, the scikit-learn API will not accept them and you will have to preprocess these features into the correct format. Our goal is to convert these features so that they are numerical. The way we achieve this by splitting the feature into a number of binary features called **'dummy variables'**, one for each category: **'0'** means the observation was not that category, while **'1'** means it was.\n",
        "\n",
        "**3. Dummy variables**\n",
        "\n",
        "For example, say we are dealing with a car dataset that has a 'origin' feature with three different possible values: 'US', 'Asia', and 'Europe'.\n",
        "\n",
        "**4. Dummy variables**\n",
        "\n",
        "We create binary features for each of the origins, as each car is made in exactly one country, each row in the dataset will have a one in exactly one of the three columns and zeros in the other two. Notice that in this case, if a car is not from the US and not from Asia, then implicitly, it is from Europe. That means that we do not actually need three separate features, but only two, so we can\n",
        "\n",
        "![](https://drive.google.com/uc?export=view&id=1s3tUqOsntCgPXzIg_s_6gSdXlKrUyowj)\n",
        "\n",
        "**5. Dummy variables**\n",
        "\n",
        "delete the 'Europe' column. If we do not do this, we are duplicating information, which might be an issue for some models.\n",
        "\n",
        "\n",
        "**6. Dealing with categorical features in Python**\n",
        "\n",
        "There are several ways to create dummy variables in Python. In scikit-learn, we can use **OneHotEncoder**. Or we can use pandas' **get dummies function**. Here, we will use get dummies.\n",
        "\n",
        "**7. Automobile dataset**\n",
        "\n",
        "The target variable here is miles per gallon or mpg. Remember that there is one categorical feature, origin, with three possible values: 'US', 'Asia', and 'Europe'.\n",
        "\n",
        "![](https://drive.google.com/uc?export=view&id=1asM8M4rSJS2xbpWPaWwGj01P1wgOjTVN)\n",
        "\n",
        "**8. EDA w/ categorical feature**\n",
        "\n",
        "Here is a box plot showing how mpg varies by origin. Let's encode this feature using dummy variables.\n",
        "\n",
        "![](\n",
        "https://drive.google.com/uc?export=view&id=1ooj0yUFBen9CtFjvgV43Og5A1S4qX0xE)\n",
        "\n",
        "**9. Encoding dummy variables**\n",
        "\n",
        "We import pandas, read in the DataFrame, and then apply the get dummies function. Notice, how pandas creates three new binary features. In the third row, origin USA and origin Europe have zeroes, while origin Asia has a one, indicating that the car is of Asian origin. But if origin USA and origin Europe are zero, then we already know that the car is Asian!\n",
        "\n",
        "![](https://drive.google.com/uc?export=view&id=1RxLqB5YalrjNMtLeVuv92GjOT7KV1GoM)\n",
        "\n",
        "**10. Encoding dummy variables**\n",
        "\n",
        "So, we drop the origin Asia column. Alternatively, we could have passed the \"drop first\" option to get dummies. Notice that the new column names have the following structure: original column name, underscore, value name. Once we have created our dummy variables, we can fit models as before.\n",
        "\n",
        "![](https://drive.google.com/uc?export=view&id=1v9m5wdKYY5K5OQC9l5yWaqTVKlLW4heh)\n",
        "\n",
        "\n",
        "**11. Linear regression with dummy variables**\n",
        "\n",
        "Here, for example, we fit the ridge regression model to the data and compute its R-squared.\n",
        "\n",
        "![](https://drive.google.com/uc?export=view&id=1hd0DqZD6QtvWDY4MEokcvnDBb8Wct8sZ)\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4AkfQ5AWbFOT"
      },
      "source": [
        "### 4.2.1 **Exploring categorical features**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ijX3dU-TbNF0"
      },
      "source": [
        "The Gapminder dataset that you worked with in previous chapters also contained a categorical 'Region' feature, which we dropped in previous exercises since you did not have the tools to deal with it. Now however, you do, so we have added it back in!\n",
        "\n",
        "Your job in this exercise is to explore this feature. Boxplots are particularly useful for visualizing categorical features such as this."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TTxv1dA6bUHs"
      },
      "source": [
        "**Instruction**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UHJP2ffzbel8"
      },
      "source": [
        "- Import pandas as pd.\n",
        "- Read the CSV file 'gapminder.csv' into a DataFrame called df.\n",
        "- Use pandas to create a boxplot showing the variation of life expectancy ('life') by region ('Region'). To do so, pass the column names in to df.boxplot() (in that order)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f0nZ4cmPblzt"
      },
      "source": [
        "**Hints**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8xGjpotRbu39"
      },
      "source": [
        "- Use the command import x as y to import x as the alias y.\n",
        "- Use the pandas' read_csv() function with 'gapminder.csv' as argument to read it into a DataFrame.\n",
        "- To create the boxplot, pass 'life' and 'Region' in as arguments to df.boxplot()."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bmKXKU4DcYBX"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1Tg40UYOpxwG-F025yth22CDA-lbf0ZK-\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L0OSr37dcscO"
      },
      "source": [
        "### 4.2.2. **Create dummy variables** "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N582xmHLdFcC"
      },
      "source": [
        "As Andy discussed in the video, scikit-learn does not accept non-numerical features. You saw in the previous exercise that the 'Region' feature contains very useful information that can predict life expectancy. For example, Sub-Saharan Africa has a lower life expectancy compared to Europe and Central Asia. Therefore, if you are trying to predict life expectancy, it would be preferable to retain the 'Region' feature. To do this, you need to binarize it by creating dummy variables, which is what you will do in this exercise."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bc4KdoOIgjXu"
      },
      "source": [
        "**Instruction** "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H2o522d3ghsI"
      },
      "source": [
        "- Use the pandas get_dummies() function to create dummy variables from the df - DataFrame. Store the result as df_region.\n",
        "- Print the columns of df_region. This has been done for you.\n",
        "- Use the get_dummies() function again, this time specifying drop_first=True to drop the unneeded dummy variable (in this case, 'Region_America').\n",
        "- Hit 'Submit Answer to print the new columns of df_region and take note of how one column was dropped!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Orr5Yyy-gzft"
      },
      "source": [
        "**Hints**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CGA7h2TMg7Ov"
      },
      "source": [
        "- Use pd.get_dummies() with df as argument to create the dummy variables.\n",
        "- To automatically drop one dummy variable, use pd.get_dummies() with df as before, but specify the additional keyword argument drop_first=True."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "![](\n",
        "https://drive.google.com/uc?export=view&id=1KicEzT5vdZQPNMjhAdSnQbsZtGCTmuZ8)"
      ],
      "metadata": {
        "id": "eEIti6gINKJa"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z-v1LFHer7XE"
      },
      "source": [
        "### 4.2.3 **Regression with categorical features**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T311pJuEsJEB"
      },
      "source": [
        "Having created the dummy variables from the 'Region' feature, you can build regression models as you did before. Here, you'll use ridge regression to perform 5-fold cross-validation.\n",
        "\n",
        "The feature array X and target variable array y have been pre-loaded."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SKpH1xfZsKo5"
      },
      "source": [
        "**Instruction**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XN00bqDAsP2I"
      },
      "source": [
        "- Import Ridge from sklearn.linear_model and cross_val_score from sklearn.model_selection.\n",
        "- Instantiate a ridge regressor called ridge with alpha=0.5 and normalize=True.\n",
        "- Perform 5-fold cross-validation on X and y using the cross_val_score() function.\n",
        "Print the cross-validated scores."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nOe-7O5_sjvw"
      },
      "source": [
        "**Hints**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pbheXqIhscys"
      },
      "source": [
        "- Use the command from y import x to import x from y.\n",
        "- Inside Ridge(), specify the keyword arguments alpha=0.5 and normalize=True to instantiate the ridge regressor.\n",
        "- Inside cross_val_score(), specify the regressor ridge, the feature array X, - target variable y, and number of folds to use.\n",
        "- Pass ridge_cv as an argument to the provided print() function to print the cross-validated scores."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1n4Iq72zWNV9s7wOPj5fjdOEZzU7xPUee)"
      ],
      "metadata": {
        "id": "4EE4R07NN3XX"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **4.3 Handling missing data**\n",
        "\n",
        "**1. Handling missing data**\n",
        "\n",
        "We say that data is missing when there is no value for a given feature in a particular row. This can occur in the real-world for many reasons: there may have been no observation, there may have been a transcription error, or the data may have been corrupted. Whatever the case, we, as data scientists, need to deal with it.\n",
        "\n",
        "**2. PIMA Indians dataset**\n",
        "\n",
        "Let's now load the PIMA Indians dataset. It doesn't look like it has any missing values as, according to df dot info, all features have 768 non-null entries. However, missing values can be encoded in a number of different ways, such as by zeroes, or question marks, or negative ones.\n",
        "\n",
        "![](https://drive.google.com/uc?export=view&id=1mrcjpgQGDieISqLJ7y9XuUYglrxtmuMS)\n",
        "\n",
        "**3. PIMA Indians dataset**\n",
        "\n",
        "Checking out df dot head, it looks as though there are observations where insulin is zero. And triceps, which is the thickness of the skin, is zero. These are not possible and, as we have no indication of the real values, the data is, for all intents and purposes, missing.\n",
        "\n",
        "![](https://drive.google.com/uc?export=view&id=155vpEVCRY1-eGeLYsZilMxKBxsli64f4)\n",
        "\n",
        "\n",
        "**4. Dropping missing data**\n",
        "\n",
        "Before we go any further, let's make all these entries 'NaN' using the replace method on the relevant columns. So, how do we deal with missing data? One way is to drop all rows containing missing data.\n",
        "![](https://drive.google.com/uc?export=view&id=1v9m5wdKYY5K5OQC9l5yWaqTVKlLW4heh)\n",
        "\n",
        "**5. Dropping missing data**\n",
        "\n",
        "We can do so using the pandas DataFrame method dropna. Checking out the shape of the resulting data frame, though, we see that we now have only approximately half the rows left! We've lost half of our data and this is unacceptable. If only a few rows contain missing values, then it's not so bad, but generally we need a more robust method. It is generally an equally bad idea to remove columns that contain NaNs.\n",
        "\n"
      ],
      "metadata": {
        "id": "oBnxMl8xN_Wp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        " df = df.dropna()\n",
        "df.shape\n",
        "(393, 9)"
      ],
      "metadata": {
        "id": "Jc1Q4eRVSY-y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**6. Imputing missing data**\n",
        "\n",
        "Another option is to impute missing data. All imputing means is to make an educated guess as to what the missing values could be. A common strategy is, in any given column with missing values, to compute the mean of all the non-missing entries and to replace all missing values with the mean. Let's try this now on our dataset. We import Imputer from sklearn dot preprocessing and instantiate an instance of the Imputer: imp. The keyword argument missing values here specifies that missing values are represented by NaN; strategy specifies that we will use the mean as described above; axis equals 0 specifies that we will impute along columns, a '1' would mean rows. Now, we can fit this imputer to our data using the fit method and then transform our data using the transform method! Due to their ability to transform our data as such, imputers are known as transformers, and any model that can transform data this way, using the transform method, is called a transformer. After transforming the data, we could then fit our supervised learning model to it, but is there a way to do both at once?"
      ],
      "metadata": {
        "id": "obhO-5RMSkSh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.preprocessing import Imputer\n",
        "imp = Imputer(missing_values='NaN', strategy='mean', axis=0)\n",
        "imp.fit(X)\n",
        "X = imp.transform(X)"
      ],
      "metadata": {
        "id": "MwggJi9XTBKp"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**7. Imputing within a pipeline**\n",
        "\n",
        "There sure is! We can use the scikit-learn pipeline object. We import Pipeline from sklearn dot pipeline and Imputer from sklearn dot preprocessing. We also instantiate a log reg model. We then build the Pipeline object! We construct a list of steps in the pipeline, where each step is a 2-tuple containing the name you wish to give the relevant step and the estimator. We then pass this list to the Pipeline constructor. We can split our data into training and test sets and"
      ],
      "metadata": {
        "id": "-OpVLjarTNPa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "In [1]: from sklearn.pipeline import Pipeline\n",
        "In [2]: from sklearn.preprocessing import Imputer\n",
        "In [3]: imp = Imputer(missing_values='NaN', strategy='mean', axis=0)\n",
        "In [4]: logreg = LogisticRegression()\n",
        "In [5]: steps = [('imputation', imp),\n",
        " ...: ('logistic_regression', logreg)]\n",
        "In [6]: pipeline = Pipeline(steps)\n",
        "In [7]: X_train, X_test, y_train, y_test = train_test_split(X, y,\n",
        " ...: test_size=0.3, random_state=42) "
      ],
      "metadata": {
        "id": "jqvNgcaNTeEi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**8. Imputing within a pipeline**\n",
        "\n",
        "fit the pipeline to the training set and then predict on the test set, as with any other model. For good measure here, we compute accuracy. Note that, in a pipeline, each step but the last must be a transformer and the last must be an estimator, such as, a classifier or a regressor."
      ],
      "metadata": {
        "id": "tDu1N0tiTj20"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "In [8]: pipeline.fit(X_train, y_train)\n",
        "In [9]: y_pred = pipeline.predict(X_test)\n",
        "In [10]: pipeline.score(X_test, y_test)\n",
        "Out[10]: 0.75324675324675328"
      ],
      "metadata": {
        "id": "oEeifS1pTpEg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **4.3.1 Dropping missing data**\n"
      ],
      "metadata": {
        "id": "8aF8OFGSVQml"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The voting dataset from Chapter 1 contained a bunch of missing values that we dealt with for you behind the scenes. Now, it's time for you to take care of these yourself!\n",
        "\n",
        "The unprocessed dataset has been loaded into a DataFrame df. Explore it in the IPython Shell with the .head() method. You will see that there are certain data points labeled with a '?'. These denote missing values. As you saw in the video, different datasets encode missing values in different ways. Sometimes it may be a '9999', other times a 0 - real-world data can be very messy! If you're lucky, the missing values will already be encoded as NaN. We use NaN because it is an efficient and simplified way of internally representing missing data, and it lets us take advantage of pandas methods such as .dropna() and .fillna(), as well as scikit-learn's Imputation transformer Imputer().\n",
        "\n",
        "In this exercise, your job is to convert the '?'s to NaNs, and then drop the rows that contain them from the DataFrame."
      ],
      "metadata": {
        "id": "-l2OP-0bVWb_"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Instruction** "
      ],
      "metadata": {
        "id": "IppZBWekVeYE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- Explore the DataFrame df in the IPython Shell. Notice how the missing value is represented.\n",
        "- Convert all '?' data points to np.nan.\n",
        "- Count the total number of NaNs using the .isnull() and .sum() methods. This has been done for you.\n",
        "- Drop the rows with missing values from df using .dropna().\n",
        "- Hit 'Submit Answer' to see how many rows were lost by dropping the missing values."
      ],
      "metadata": {
        "id": "UC2wiLTLVlEs"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Hints**"
      ],
      "metadata": {
        "id": "g5YWJBWMVsUn"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- Use df.head() to explore df in the IPython Shell and notice how '?' is used to represent missing values.\n",
        "- Access all the entries that are '?' using DataFrame slicing methods and convert these to np.nan.\n",
        "- Use the .dropna() method on df to drop all rows with missing values."
      ],
      "metadata": {
        "id": "7nCWWexOVvUP"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1tSn8nrf2G_Shnv9Nuxrh9B9l06t_8KGD)"
      ],
      "metadata": {
        "id": "qjvQXT8ZWYnx"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **4.3.2 Imputing missing data in a ML Pipeline I**\n",
        "\n",
        "As you've come to appreciate, there are many steps to building a model, from creating training and test sets, to fitting a classifier or regressor, to tuning its parameters, to evaluating its performance on new data. Imputation can be seen as the first step of this machine learning process, the entirety of which can be viewed within the context of a pipeline. Scikit-learn provides a pipeline constructor that allows you to piece together these steps into one process and thereby simplify your workflow.\n",
        "\n",
        "You'll now practice setting up a pipeline with two steps: the imputation step, followed by the instantiation of a classifier. You've seen three classifiers in this course so far: k-NN, logistic regression, and the decision tree. You will now be introduced to a fourth one - the Support Vector Machine, or SVM. For now, do not worry about how it works under the hood. It works exactly as you would expect of the scikit-learn estimators that you have worked with previously, in that it has the same .fit() and .predict() methods as before.\n"
      ],
      "metadata": {
        "id": "s_5J3uOkWe7t"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Instruction**"
      ],
      "metadata": {
        "id": "Q7ld9ntiXPeC"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- import Imputer from sklearn.preprocessing and SVC from sklearn.svm. SVC stands for Support Vector Classification, which is a type of SVM.\n",
        "- Setup the Imputation transformer to impute missing data (represented as 'NaN') with the 'most_frequent' value in the column (axis=0).\n",
        "- Instantiate a SVC classifier. Store the result in clf.\n",
        "- Create the steps of the pipeline by creating a list of tuples:\n",
        "-  The first tuple should consist of the imputation step, using imp.\n",
        "-- The second should consist of the classifier."
      ],
      "metadata": {
        "id": "6dc9xrXXXYq4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**hints**"
      ],
      "metadata": {
        "id": "N-jYy9CCXn_G"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- Use the command from y import x to import x from y.\n",
        "- Inside Imputer(), pass in 'NaN', 'most_frequent' and 0 for - missing_values, strategy, and axis, respectively.\n",
        "- Use SVC() to instantiate the SVC classifier.\n",
        "- The two tuples needed are ('imputation', imp), and ('SVM', clf)."
      ],
      "metadata": {
        "id": "locBiF2JXtuG"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1VfHfxz5fgqlBRjqvFxSDf6Rx4HoQHceo)"
      ],
      "metadata": {
        "id": "d_1UlhyLYQqZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **4.3.3-Imputing missing data in a ML Pipeline II**\n"
      ],
      "metadata": {
        "id": "cQOqE3j4YjLQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Having setup the steps of the pipeline in the previous exercise, you will now use it on the voting dataset to classify a Congressman's party affiliation. What makes pipelines so incredibly useful is the simple interface that they provide. You can use the .fit() and .predict() methods on pipelines just as you did with your classifiers and regressors!\n",
        "\n",
        "Practice this for yourself now and generate a classification report of your predictions. The steps of the pipeline have been set up for you, and the feature array X and target variable array y have been pre-loaded. Additionally, train_test_split and classification_report have been imported from sklearn.model_selection and sklearn.metrics respectively."
      ],
      "metadata": {
        "id": "pRDWxQHBYrtG"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**instruction**"
      ],
      "metadata": {
        "id": "_CRVG6aIYwmU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- Import the following modules:\n",
        "- Imputer from sklearn.preprocessing and Pipeline from sklearn.pipeline.\n",
        "SVC from sklearn.svm.\n",
        "- Create the pipeline using Pipeline() and steps.\n",
        "- Create training and test sets. Use 30% of the data for testing and a random state of 42.\n",
        "- Fit the pipeline to the training set and predict the labels of the test set.\n",
        "- Compute the classification report."
      ],
      "metadata": {
        "id": "LtlBqPZgY1yt"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**hints**"
      ],
      "metadata": {
        "id": "qbQjKAZbZARF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- Use the command from y import x to import x from y.\n",
        "- To create the pipeline, pass in steps to Pipeline().\n",
        "- Use the train_test_split() function to create the training and test sets. Pass in X and y, and specify the keyword arguments test_size=0.3 and random_state=42.\n",
        "- Use the .fit() method on pipeline with X_train, y_train as arguments. Then, use the .predict() method with X_test as argument to predict the labels.\n",
        "- To generate the classification report, pass y_test and y_pred as arguments to classification_report()"
      ],
      "metadata": {
        "id": "zcQ3htzNZC47"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1VXeWy9RAXTszCND4U5r5lQqc6QSkSYI2)"
      ],
      "metadata": {
        "id": "SHvWr111ZpWO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Compute metrics\n",
        "print(____(___,____))"
      ],
      "metadata": {
        "id": "Qc56lGMDaAwV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **4.4 Centering and scaling**\n"
      ],
      "metadata": {
        "id": "40dwISn5aQtE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**. Centering and scaling**\n",
        "\n",
        "Great work on imputing data and building machine learning pipelines using scikit-learn! Data imputation is one of several important preprocessing steps for machine learning. In this video, will cover another: centering and scaling your data."
      ],
      "metadata": {
        "id": "ikBbbDTFVJ61"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**2. Why scale your data?**\n",
        "\n",
        "To motivate this, let's use df dot describe to check out the ranges of the feature variables in the red wine quality dataset. The features are chemical properties such as acidity, pH, and alcohol content. The target value is good or bad, encoded as '1' and '0', respectively. We see that the ranges vary widely: 'density' varies from (point) 99 to to 1 and 'total sulfur dioxide' from 6 to 289!"
      ],
      "metadata": {
        "id": "P7ID31x6VWOx"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1SqLj5oVX0UTrIG1QC3jC3xYu-qCRCP1r)"
      ],
      "metadata": {
        "id": "1OB1gcANWfAp"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**3. Why scale your data?**\n",
        "\n",
        "Many machine learning models use some form of distance to inform them so if you have features on far larger scales, they can unduly influence your model. For example, K-nearest neighbors uses distance explicitly when making predictions. For this reason, we actually want features to be on a similar scale. To achieve this, we do what is called normalizing or scaling and centering."
      ],
      "metadata": {
        "id": "Go5dVZ13WzRB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**4. Ways to normalize your data**\n",
        "\n",
        "There are several ways to normalize your data: given any column, you can subtract the mean and divide by the variance so that all features are centred around zero and have variance one. This is called standardization. You can also subtract the minimum and divide by the range of the data so the normalized dataset has minimum zero and maximum one. You can also normalize so that data ranges from -1 to 1 instead. In this video, I'll show you to to perform standardization. See the scikit-learn docs for how to implement the other approaches."
      ],
      "metadata": {
        "id": "obVRq0emXHSR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**5. Scaling in scikit-learn**\n",
        "\n",
        "To scale our features, we import scale from sklearn dot preprocessing. We then pass the feature data to scale and this returns our scaled data. Looking at the mean and standard deviation of the columns of both the original and scaled data verifies this."
      ],
      "metadata": {
        "id": "0ppiwhnkXPv-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "In [2]: from sklearn.preprocessing import scale\n",
        "In [3]: X_scaled = scale(X)\n",
        "In [4]: np.mean(X), np.std(X)\n",
        "Out[4]: (8.13421922452, 16.7265339794)\n",
        "In [5]: np.mean(X_scaled), np.std(X_scaled)\n",
        "Out[5]: (2.54662653149e-15, 1.0)"
      ],
      "metadata": {
        "id": "ztbW8ojEXO8-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**6. Scaling in a pipeline**\n",
        "\n",
        "We can also put a scalar in a pipeline object! To do so, we import StandardScaler from sklearn dot reprocessing and build a pipeline object as we did earlier; here we'll use a K-nearest neighbors algorithm. We then split our wine quality dataset in training and test sets, fit the pipeline to our training set, and predict on our test set. Computing the accuracy yields (point) 956, whereas performing KNN without scaling resulted in an accuracy of (point) 928. Scaling did improve our model performance!"
      ],
      "metadata": {
        "id": "xtNwL_v3Xuy5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "In [6]: from sklearn.preprocessing import StandardScaler\n",
        "In [7]: steps = [('scaler', StandardScaler()),\n",
        " ...: ('knn', KNeighborsClassifier())]\n",
        "In [8]: pipeline = Pipeline(steps)\n",
        "In [9]: X_train, X_test, y_train, y_test = train_test_split(X, y,\n",
        " ...: test_size=0.2, random_state=21)\n",
        "In [10]: knn_scaled = pipeline.fit(X_train, y_train)\n",
        "In [11]: y_pred = pipeline.predict(X_test)\n",
        "In [12]: accuracy_score(y_test, y_pred)\n",
        "Out[12]: 0.956\n",
        "In [13]: knn_unscaled = KNeighborsClassifier().fit(X_train, y_train)\n",
        "In [14]: knn_unscaled.score(X_test, y_"
      ],
      "metadata": {
        "id": "F2bEHkN2YBPm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**7. CV and scaling in a pipeline**\n",
        "\n",
        "Let's also take a look at how we can use cross-validation with a supervised learning pipeline. We first build our pipeline. We then specify our hyperparameter space by creating a dictionary: the keys are the pipeline step name followed by a double underscore, followed by the hyperparameter name; the corresponding value is a list or an array of the values to try for that particular hyperparameter. In this case, we are tuning only the n neighbors in the KNN model. As always, we split our data into cross-validation and hold-out sets. We then perform a GridSearch over the parameters in the pipeline by instantiating the GridSearchCV object and fitting it to our training data. The predict method will call predict on the estimator with the best found parameters and we do this on the hold-out set."
      ],
      "metadata": {
        "id": "nnCTsNUhYCtg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "In [14]: steps = [('scaler', StandardScaler()),\n",
        " ...: (('knn', KNeighborsClassifier())]\n",
        "In [15]: pipeline = Pipeline(steps)\n",
        "In [16]: parameters = {knn__n_neighbors=np.arange(1, 50)}\n",
        "In [17]: X_train, X_test, y_train, y_test = train_test_split(X, y,\n",
        " ...: test_size=0.2, random_state=21)\n",
        "In [18]: cv = GridSearchCV(pipeline, param_grid=parameters)\n",
        "In [19]: cv.fit(X_train, y_train)\n",
        "In [20]: y_pred = cv.predict(X_test)"
      ],
      "metadata": {
        "id": "Ov1ELUPqYRm9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**8. Scaling and CV in a pipeline**\n",
        "\n",
        "We also print the best parameters chosen by our gridsearch, along with the accuracy and classification report of the predictions on the hold-out set."
      ],
      "metadata": {
        "id": "MXKJqQZlYqoB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "In [21]: print(cv.best_params_)\n",
        "{'knn__n_neighbors': 41}\n",
        "In [22]: print(cv.score(X_test, y_test))\n",
        "0.956\n",
        "In [23]: print(classification_report(y_test, y_pred))\n",
        " precision recall f1-score support"
      ],
      "metadata": {
        "id": "YQphDPKzYys1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **4.4.1- Centering and scaling your data**\n"
      ],
      "metadata": {
        "id": "ZyxUiZK5ZHa3"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In the video, Hugo demonstrated how significantly the performance of a model can improve if the features are scaled. Note that this is not always the case: In the Congressional voting records dataset, for example, all of the features are binary. In such a situation, scaling will have minimal impact.\n",
        "\n",
        "You will now explore scaling for yourself on a new dataset - White Wine Quality! Hugo used the Red Wine Quality dataset in the video. We have used the 'quality' feature of the wine to create a binary target variable: If 'quality' is less than 5, the target variable is 1, and otherwise, it is 0.\n",
        "\n",
        "The DataFrame has been pre-loaded as df, along with the feature and target variable arrays X and y. Explore it in the IPython Shell. Notice how some features seem to have different units of measurement. 'density', for instance, takes values between 0.98 and 1.04, while 'total sulfur dioxide' ranges from 9 to 440. As a result, it may be worth scaling the features here. Your job in this exercise is to scale the features and compute the mean and standard deviation of the unscaled features compared to the scaled features."
      ],
      "metadata": {
        "id": "vdY9AJogZUbp"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Instruction**"
      ],
      "metadata": {
        "id": "5dSjat8UZYGO"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- Import scale from sklearn.preprocessing.\n",
        "- Scale the features X using scale().\n",
        "- Print the mean and standard deviation of the unscaled features X, and then the scaled features X_scaled. Use the numpy functions np.mean() and np.std() to compute the mean and standard deviations"
      ],
      "metadata": {
        "id": "VbcoaUaNZd09"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Hints**"
      ],
      "metadata": {
        "id": "sYE7FqBpZ40_"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- Use the command from y import x to import x from y.\n",
        "- Scale the features by passing X to scale().\n",
        "- Use np.mean() and np.std() on X and X_scaled to compute the mean and standard deviation of the scaled and unscaled features. Place these inside the provided print() functions."
      ],
      "metadata": {
        "id": "fKW7uGfoZ--N"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "![](\n",
        "https://drive.google.com/uc?export=view&id=18XbWFpNnL-8cAKqr3mfrDdgr9FHq4doA)"
      ],
      "metadata": {
        "id": "Pzn6pnrSabM0"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **4.4.2.Centering and scaling in a pipeline**\n"
      ],
      "metadata": {
        "id": "rzDNMKshamK_"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "With regard to whether or not scaling is effective, the proof is in the pudding! See for yourself whether or not scaling the features of the White Wine Quality dataset has any impact on its performance. You will use a k-NN classifier as part of a pipeline that includes scaling, and for the purposes of comparison, a k-NN classifier trained on the unscaled data has been provided.\n",
        "\n",
        "The feature array and target variable array have been pre-loaded as X and y. Additionally, KNeighborsClassifier and train_test_split have been imported from sklearn.neighbors and sklearn.model_selection, respectively."
      ],
      "metadata": {
        "id": "HhvPnaoRa8xl"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Instruction** "
      ],
      "metadata": {
        "id": "0R44PAOubAvj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- Import the following modules:\n",
        "- StandardScaler from sklearn.preprocessing.\n",
        "- Pipeline from sklearn.pipeline.\n",
        "- Complete the steps of the pipeline with StandardScaler() - for 'scaler' and KNeighborsClassifier() for 'knn'.\n",
        "- Create the pipeline using Pipeline() and steps.\n",
        "- Create training and test sets, with 30% used for testing. Use a random state of 42.\n",
        "- Fit the pipeline to the training set.\n",
        "- Compute the accuracy scores of the scaled and unscaled models by using the .score() method inside the provided print() functions."
      ],
      "metadata": {
        "id": "FPG6oCrdbHs-"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**hints**"
      ],
      "metadata": {
        "id": "FbyQYuHSbSE0"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- Use the command from y import x to import x from y.\n",
        "- The two tuples needed in the pipeline steps are ('scaler', - StandardScaler()), and ('knn', KNeighborsClassifier()).\n",
        "- To create the pipeline, pass in steps to Pipeline().\n",
        "- Use the train_test_split() function to create the training and test sets. Pass in X and y, and specify the keyword arguments test_size=0.3 and random_state=42.\n",
        "- Use the .fit() method on pipeline with X_train, y_train as arguments.\n",
        "- Use the .score() method on knn_scaled and knn_unscaled with X_test and y_test as arguments to compute the accuracy scores."
      ],
      "metadata": {
        "id": "Kf7eNfBcbX9v"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1ugVwclwcpFPx87yzNyD6hVXz0C4YKUr2)"
      ],
      "metadata": {
        "id": "Hy6zP1sNbwGU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **4.4.3-Bringing it all together I: Pipeline for classification**\n"
      ],
      "metadata": {
        "id": "dsD7gBwmdMLj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "It is time now to piece together everything you have learned so far into a pipeline for classification! Your job in this exercise is to build a pipeline that includes scaling and hyperparameter tuning to classify wine quality.\n",
        "\n",
        "You'll return to using the SVM classifier you were briefly introduced to earlier in this chapter. The hyperparameters you will tune are  and .  controls the regularization strength. It is analogous to the  you tuned for logistic regression in Chapter 3, while  controls the kernel coefficient: Do not worry about this now as it is beyond the scope of this course.\n",
        "\n",
        "The following modules and functions have been pre-loaded: Pipeline, SVC, train_test_split, GridSearchCV, classification_report, accuracy_score. The feature and target variable arrays X and y have also been pre-loaded."
      ],
      "metadata": {
        "id": "FvN1d-ErdXy0"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Instruction** "
      ],
      "metadata": {
        "id": "7a-WU6XGdZOJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- Setup the pipeline with the following steps:\n",
        "- Scaling, called 'scaler' with StandardScaler().\n",
        "- Classification, called 'SVM' with SVC().\n",
        "- Specify the hyperparameter space using the following - notation: 'step_name__parameter_name'. Here, the step_name is SVM, and the parameter_names are C and gamma.\n",
        "- Create training and test sets, with 20% of the data used for the test set. Use a random state of 21.\n",
        "- Instantiate GridSearchCV with the pipeline and hyperparameter space and fit it to the training set. Use - 3-fold cross-validation (This is the default, so you don't have to specify it).\n",
        "Predict the labels of the test set and compute the metrics. The metrics have been computed for you."
      ],
      "metadata": {
        "id": "f5-uUpR4dfMS"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**hints**"
      ],
      "metadata": {
        "id": "ufIOmGwpdu79"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- The two tuples needed in the pipeline steps are ('scaler', StandardScaler()), and ('SVM', SVC()).\n",
        "- To specify the hyperparameter space, use 'SVM__C' and 'SVM__gamma'.\n",
        "- Use the train_test_split() function to create the training and test sets. Pass in X and y, and specify the keyword arguments test_size=0.2 and random_state=21.\n",
        "Use GridSearchCV() with the arguments pipeline and param_grid=parameters to instantiate it.\n",
        "- Use the .fit() method on the GridSearchCV object cv with - X_train and y_train as arguments.\n",
        "- To predict the labels, use the .predict() method on cv with X_test as the argument."
      ],
      "metadata": {
        "id": "phRx1PLLd0zL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "![](https://drive.google.com/uc?export=view&id=16wr8MXbiH0rGo6uT9KtC9I2a3473ZUgM)"
      ],
      "metadata": {
        "id": "fBu-7zLLea3i"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **4.4.4 -Bringing it all together II: Pipeline for regression**\n"
      ],
      "metadata": {
        "id": "ra_WFfj7eqS5"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "For this final exercise, you will return to the Gapminder dataset. Guess what? Even this dataset has missing values that we dealt with for you in earlier chapters! Now, you have all the tools to take care of them yourself!\n",
        "\n",
        "Your job is to build a pipeline that imputes the missing data, scales the features, and fits an ElasticNet to the Gapminder data. You will then tune the l1_ratio of your ElasticNet using GridSearchCV.\n",
        "\n",
        "All the necessary modules have been imported, and the feature and target variable arrays have been pre-loaded as X and y."
      ],
      "metadata": {
        "id": "80yLQX-nezka"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Instruction**"
      ],
      "metadata": {
        "id": "IEOhP5PAe2Ap"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- Set up a pipeline with the following steps:\n",
        "'imputation', which uses the Imputer() transformer and the 'mean' strategy to impute missing data ('NaN') using the mean of the column.\n",
        "'scaler', which scales the features using StandardScaler().\n",
        "'elasticnet', which instantiates an ElasticNet() regressor.\n",
        "- Specify the hyperparameter space for the  ratio using the following notation: 'step_name__parameter_name'. Here, the - step_name is elasticnet, and the parameter_name is l1_ratio.\n",
        "Create training and test sets, with 40% of the data used for the test set. Use a random state of 42.\n",
        "- Instantiate GridSearchCV with the pipeline and hyperparameter space. Use 3-fold cross-validation (This is the default, so you don't have to specify it).\n",
        "- Fit the GridSearchCV object to the training set.\n",
        "Compute  and the best parameters. This has been done for you, so hit 'Submit Answer' to see the results!"
      ],
      "metadata": {
        "id": "wzfkSlvve8qp"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**hints**\n",
        "\n",
        "- The tuples needed in the pipeline steps are:\n",
        "('imputation', Imputer(missing_values='NaN', strategy='mean', axis=0)).\n",
        "('scaler', StandardScaler()).\n",
        "('elasticnet', ElasticNet()).\n",
        "- To specify the hyperparameter space, use 'elasticnet__l1_ratio'.\n",
        "- Use the train_test_split() function to create the training and test sets. Pass in X and y, and specify the keyword arguments test_size=0.4 and random_state=42.\n",
        "- Use GridSearchCV() with the arguments pipeline and param_grid=parameters to instantiate it.\n",
        "- Use the .fit() method on the GridSearchCV object gm_cv with X_train and y_train as arguments."
      ],
      "metadata": {
        "id": "6vqvhEhTfKSo"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1UU6izX-rU3xelEHqpKx0DvXB1B3pDjZd)"
      ],
      "metadata": {
        "id": "d864MInjfpIY"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sxoxWxkeQDRi"
      },
      "source": [
        "# **References**\n",
        "\n",
        "[[1]Supervised Learning with scikit-learn](https://learn.datacamp.com/courses/supervised-learning-with-scikit-learn)\n",
        "\n",
        "[Exercise](https://github.com/wblakecannon/DataCamp/blob/master/17-supervised-learning-with-scikit-learn/01-classification/01-k-nearest-neighbors-fit.py)"
      ]
    }
  ]
}