Measuring_Performance_of_Classification_Models.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pzu_MjgDyIkG"
      },
      "source": [
        "# Performance Metrics For Classification Models\n",
        "\n",
        "Once the model is trained, the next step is to evaluate its performance. There are different performance metrices that can be used to judge output of the model. In addition to model performance evaluation, these metrices are also very useful in comparing different models and choosing best performing model for given problem domain. \n",
        "\n",
        "\n",
        "In this tutorial, various performance matrices used in classification models are demonstrated. \n",
        "This includes, confusion matrix, precision, recall, ROC curve and AUC. In order to use these metrices,  scikit learn library that supports built-in functionality of these measures will be used. \n",
        "\n",
        "The data set used for demonstration is taken from medical domain. The name of the data set is 'heart disease' available at https://www.kaggle.com/ronitf/heart-disease-uci. The size of the data set is (303 x 14). It is a binary classification data set. \n",
        "\n",
        "The key steps followed in this tutorial to make you learn how to use performances metrices on given data set is detailed below:\n",
        "\n",
        "1. Load data set\n",
        "\n",
        "2. Create train and test data sets using Hold-out method\n",
        "\n",
        "3. Use train data set to learn the model\n",
        "\n",
        "4. Apply model on the test data\n",
        "\n",
        "5. Use Scikit learn performance metrices for model evaluation\n",
        "\n",
        "6. Interpret the results\n",
        "\n",
        "It is important to note that this tutorial is to help you understand how to use different  performance metrices. The choice of classification model, selecting the models learning and testing enviornment(such as Hold-out, K-cross validation, repeated Hold-out, Leave one Out cross validation) is users specific. The performance metrices discussed in this tutorial can be applied on any classificaion model and are independent of choice of learning and testing enviornments. The key to these metrices is that they are only applicable when model is ready and is to be tested for its performance strengths. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nKZiDvEHyIkJ"
      },
      "source": [
        "# 1. Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ptqAuPNqyIkJ"
      },
      "outputs": [],
      "source": [
        "# importing Pandas for data manipulation\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "\n",
        "# importing decision tree classifier\n",
        "\n",
        "from sklearn.tree import DecisionTreeClassifier\n",
        "\n",
        "# importing method for Hold-out\n",
        "\n",
        "from sklearn.model_selection import train_test_split \n",
        "\n",
        "#importing methods for model evaluation\n",
        "from sklearn import metrics\n",
        "\n",
        "# importing matplotlib for visualization\n",
        "from matplotlib.pyplot import boxplot\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8cwL6BnDyIkK"
      },
      "source": [
        "# 2. Loading data set"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wpWUyP8UyIkK",
        "outputId": "5af8b230-430c-459d-f4d3-8287abb6d5ee"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 303 entries, 0 to 302\n",
            "Data columns (total 14 columns):\n",
            "age         303 non-null int64\n",
            "sex         303 non-null int64\n",
            "cp          303 non-null int64\n",
            "trestbps    303 non-null int64\n",
            "chol        303 non-null int64\n",
            "fbs         303 non-null int64\n",
            "restecg     303 non-null int64\n",
            "thalach     303 non-null int64\n",
            "exang       303 non-null int64\n",
            "oldpeak     303 non-null float64\n",
            "slope       303 non-null int64\n",
            "ca          303 non-null int64\n",
            "thal        303 non-null int64\n",
            "target      303 non-null int64\n",
            "dtypes: float64(1), int64(13)\n",
            "memory usage: 33.2 KB\n"
          ]
        }
      ],
      "source": [
        "dataset = pd.read_csv(\"Data sets/heart.csv\")\n",
        "dataset.info()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n5bLht3CyIkM"
      },
      "source": [
        "# 3. Creating Training and Test split\n",
        "\n",
        "Training set will be used for model learning whereas, test set for performance evaluation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OY_YSq0syIkM",
        "outputId": "a3fa85ce-0a9d-4635-f1a7-f74fbe7a44c9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The sample training data without target feature\n",
            "\n",
            "(212, 13)\n",
            "\n",
            "The sample with only target feature\n",
            "\n",
            "(91,)\n"
          ]
        }
      ],
      "source": [
        "# My_data contains all data points from My_data set from from first feature to 12th feature(indicator features)\n",
        "My_data = dataset.iloc[:,0:13] \n",
        "\n",
        "# My_target contains class information which is 13th feature in the data set of \n",
        "\n",
        "My_data_target=dataset.iloc[:,13]\n",
        "\n",
        "\n",
        "X_train, X_test, Y_train, Y_test = train_test_split(My_data, My_data_target, test_size=0.3, random_state=10)\n",
        "\n",
        "print(\"The sample training data without target feature\\n\")\n",
        "print(X_train.shape)\n",
        "print(\"\\nThe sample with only target feature\\n\")\n",
        "print(Y_test.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GhU9S7TayIkM"
      },
      "source": [
        "# 4. Learning  Model Under Hold-out method"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Q824ipOByIkN",
        "outputId": "5eaa2824-0749-4aa2-e55a-c897f432a753"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
              "                       max_features=None, max_leaf_nodes=None,\n",
              "                       min_impurity_decrease=0.0, min_impurity_split=None,\n",
              "                       min_samples_leaf=1, min_samples_split=2,\n",
              "                       min_weight_fraction_leaf=0.0, presort=False,\n",
              "                       random_state=None, splitter='best')"
            ]
          },
          "execution_count": 21,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# creating instance of Decision tree classifier\n",
        "\n",
        "DT_model_Holdout = DecisionTreeClassifier()\n",
        "\n",
        "# fitting the model to training data set\n",
        "DT_model_Holdout.fit(X_train, Y_train)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UImdYg4RyIkN"
      },
      "source": [
        "# 5. Testing Model on Test Data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gZSMTh-3yIkN"
      },
      "outputs": [],
      "source": [
        "# Getting prediction on test set\n",
        "\n",
        "DT_model_Holdout_pred_test= DT_model_Holdout.predict(X_test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cL08PE8YyIkO"
      },
      "source": [
        "# 6. Measuring Models Performance"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vJ6VV5IIyIkO"
      },
      "source": [
        "## 6.1 Estimating Models Accuracy"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Z-egK9spyIkO",
        "outputId": "7de2fc64-8768-4811-b6ad-e0402cdebd58"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 73.0 %\n"
          ]
        }
      ],
      "source": [
        "# Computing Model Accuracy\n",
        "\n",
        "print(\"Accuracy:\",round(metrics.accuracy_score(Y_test, DT_model_Holdout_pred_test),2) * 100, \"%\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0TRnBnaGyIkO"
      },
      "source": [
        "Accuracy of Decision Tree model is 75% using Hold-Out method"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r2E1WboCyIkO"
      },
      "source": [
        "## 6.2 Printing Confusion Matrix"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tsls4HYoyIkP",
        "outputId": "1d7008e6-77ff-42e0-c73d-3869958bd87c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Confusion matrix\n",
            "---------------\n",
            "[[32 18]\n",
            " [ 7 34]]\n"
          ]
        }
      ],
      "source": [
        "print (\"Confusion matrix\")\n",
        "\n",
        "print (\"---------------\")\n",
        "\n",
        "Confusion_matrix=metrics.confusion_matrix(Y_test, DT_model_Holdout_pred_test)\n",
        "print(Confusion_matrix)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CLD5Xl05yIkP"
      },
      "source": [
        "The output of confusion matrix indicates that: TN= 32, TP = 36, FP = 18 and FN = 5. The total wrong predictions made by the model are: FP + FN = 23 and total correct predictions are: TP + TN = 68. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_483tz69yIkP"
      },
      "source": [
        "### 6.2.1 Printing Confusion Matrix Using Seaborn"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OvPm12EfyIkP",
        "outputId": "02153dbd-25ed-453b-e0bb-c53b1201e5cd"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWgAAAEWCAYAAABLzQ1kAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAHpFJREFUeJzt3XucVVXdx/HPdwBBBUXNC4mFaFppivdbFlqKt7yVpllqUaQ9ZqVZVualsryUmWkmagnklSdJ8/qkqYWRggiKWpiaiaKYindgLr/nj71GjuMw58xw9jn7zHzfr9d+cc7ae6/1O3D4zZq1115bEYGZmRVPU70DMDOzzjlBm5kVlBO0mVlBOUGbmRWUE7SZWUE5QZuZFZQTtC03SStK+qOklyVNXo56DpP0f9WMrR4k3SzpiHrHYY3PCboPkfQZSTMkvSZpfkokH65C1Z8C1gbWiIiDelpJRFweEbtXIZ63kTRaUkia0qF881R+Z4X1nCrpd+WOi4g9I2JCD8M1e4sTdB8h6TjgXODHZMn0PcCvgP2qUP17gbkR0VKFuvLyPLCDpDVKyo4A5larAWX8f8qqxl+mPkDSqsAPgP+JiGsj4vWIaI6IP0bECemYgZLOlfRM2s6VNDDtGy1pnqTjJS1Ive/Pp32nAScDn04987Ede5qSRqSeav/0/khJj0t6VdITkg4rKZ9act6OkqanoZPpknYs2XenpB9KujvV83+S3tXFX8MS4A/AIen8fsCngcs7/F39QtJTkl6RdJ+knVP5HsB3Sz7n7JI4Tpd0N/AGMDKVfTHtv1DS70vqP1PS7ZJU8T+g9VlO0H3DDsAgYEoXx3wP2B4YBWwObAucVLJ/HWBVYF1gLHCBpNUi4hSyXvnVETE4Ii7tKhBJKwPnAXtGxBBgR2BWJ8etDtyYjl0DOAe4sUMP+DPA54G1gBWAb3bVNjARODy9HgPMAZ7pcMx0sr+D1YErgMmSBkXELR0+5+Yl53wOGAcMAZ7sUN/xwIfSD5+dyf7ujgivsWAVcILuG9YA/ltmCOIw4AcRsSAingdOI0s87ZrT/uaIuAl4Ddi4h/G0AZtKWjEi5kfEQ50cszfwaERMioiWiLgS+AfwiZJjfhsRcyPiTeAassS6TBHxN2B1SRuTJeqJnRzzu4h4IbX5M2Ag5T/nZRHxUDqnuUN9b5D9PZ4D/A74akTMK1OfGeAE3Ve8ALyrfYhhGd7N23t/T6ayt+rokODfAAZ3N5CIeJ1saOEoYL6kGyW9v4J42mNat+T9sz2IZxJwDLALnfxGIembkh5JwyoLyX5r6GroBOCprnZGxD3A44DIfpCYVcQJum+YBiwG9u/imGfILva1ew/v/PW/Uq8DK5W8X6d0Z0TcGhG7AcPIesUXVxBPe0xP9zCmdpOArwA3pd7tW9IQxLeAg4HVImIo8DJZYgVY1rBEl8MVkv6HrCf+TKrfrCJO0H1ARLxMdiHvAkn7S1pJ0gBJe0o6Kx12JXCSpDXTxbaTyX4l74lZwEckvSddoPxO+w5Ja0vaL41FLyYbKmnrpI6bgI3S1MD+kj4NfBC4oYcxARARTwAfJRtz72gI0EI246O/pJOBVUr2PweM6M5MDUkbAT8CPks21PEtSV0OxZi1c4LuI9J46nFkF/6eJ/u1/BiymQ2QJZEZwAPAg8DMVNaTtv4EXJ3quo+3J9WmFMczwItkyfLoTup4AdiH7CLbC2Q9z30i4r89ialD3VMjorPfDm4FbiGbevcksIi3D1+034TzgqSZ5dpJQ0q/A86MiNkR8SjZTJBJ7TNkzLoiX0w2Mysm96DNzArKCdrMrKCcoM3MCsoJ2sysoLq6caGuZv9nPV+9tHc49KLj6h2CFdDDp39judc2aXt2o4pzTtM6c2uylop70GZmBVXYHrSZWS21dXq/VOdq1bN1gjYzA5qjteJja5U4naDNzOheD7pWnKDNzIDWAt5V7QRtZga0db0oYV04QZuZAa1O0GZmxeQetJlZQTV7DNrMrJg8xGFmVlCtxcvPTtBmZtD5c9fqzQnazAxopSbrH3WLE7SZGdAcTtBmZoXkHrSZWUG1uQdtZlZM7kGbmRVUawGfX+IEbWaGhzjMzAprSfSrdwjv4ARtZga0eYjDzKyYfJHQzKygWsM9aDOzQmpzD9rMrJiWRPHSYfEiMjOrA18kNDMrqFbPgzYzK6Yi3klYvIjMzOqgLZoq3roiaZCkeyXNlvSQpNNS+fqS7pH0L0lXS1qhXExO0GZmZD3oSrcyFgO7RsTmwChgD0nbA2cCP4+IDYGXgLHlKnKCNjMDmqNfxVtXIvNaejsgbQHsCvxvKp8A7F8uJo9Bm5lR3RtVJPUD7gM2BC4AHgMWRkRLOmQesG65etyDNjMju1Gl0k3SOEkzSrZxpXVFRGtEjAKGA9sC7+9JTO5Bm5nRvR50RIwHxldw3EJJdwA7AEMl9U+96OHA0+XOdw/azIzqXSSUtKakoen1isBuwCPAHcCn0mFHANeVi8k9aDMzqrpg/zBgQhqHbgKuiYgbJD0MXCXpR8D9wKXlKnKCNjMDmqu0FkdEPABs0Un542Tj0RVzgjYzw+tBm5kVVrk7BOvBCdrMDPegzcwKyz1oM7OCKncLdz04QZuZ4WcSmpkVVhXnQVeNE7SZGcVcsN8J2swM96DNzArLD401Myuo5jYnaDOzQvI8aDOzgvKdhFbWkiVwynEDaWkWra2w/c6tHHxEM+f9ZAUem9tE//6wwcZtjPv6Evr7X6/P+NGBu/HRjUfy4utvsN95kwB4/7A1OWXfjzFwQD9a2oIfXn87D857rs6RNq4iXiQsXp++jxswAE45ezFnX7SIs369iFkzmpj7cBMf3rWFc3+ziJ+OX8SSxfDnm52d+5IpMx9m3IQpbys7fszO/OqOv3Pg+Zdz/m1/4/gxO9cput6hLZoq3moll//lksaQPbG2/aGITwPXRcQtebTXm0gwaMXsdWsLtLYICbbcru2tYzZ8fxsvPF+8n/aWn/v+/TTvHrrK28qCYOWBKwAweNBAFrz6ej1C6zXa+sIQh6RzgY2AiWRProXs+VvHStozIr5W7TZ7m7ZW+PZXBvHsM2LMvi287wNLk3NLC/z1tv4c+ZUldYzQiuCMG+/i4iMP4IQ9PkJTkzjsoqvqHVJDa24r3locefTV94qIvSLiqoiYmrargL2Bvbo6sfRJuf97xWs5hNYYmvrB2Rct4tdXvslj/2ziP08s/cl+yXkr8IEPtfKBD7V1UYP1BYdsuxln3HQXHzv7Es688S5+eMDu9Q6pobWFKt5qJY8EvUjSNp2UbwMs6urEiBgfEVtHxNaf+szgHEJrLCsPhk02b2XWjOwn++RJ/XnlZTj8qOY6R2ZFsN+WH+RPD/0LgFvmzOVDw9euc0SNrQ1VvNVKHmPQRwIXShrC0iGO9YCX0z7rwisLoV//LDkvWQwPzOzHfp9u5vab+jF7Rj9OPmsxTb60a8CCV15jm/WHM/2JeWw/cj2efGFhvUNqaEWcxVH1BB0RM4HtJK1DyUXCiHi22m31Ri+9KC44ayBtbRABO3ykha22b+OQMSuy5trB944dBMB2H27hU59rqXO0VitnH7wn245cj6ErDeLP3/oi598+jVP+cBvf2Xs0/ZqaWNLSwil/uK3eYTa0PnWjSkrITsrd9N6RwVm/fudI0FW3vlmHaKwoTrjm5k7LD/rVFTWOpPdq6UsJ2syskfSJIQ4zs0ZUxASdW59e0gaSBqbXoyUdK2loXu2ZmS2PvjLNrt3vgVZJGwLjyWZyeMDMzAqpiAk6zyGOtohokXQA8MuI+KWk+3Nsz8ysx/rErd4lmiUdChwBfCKVDcixPTOzHmsp4IL9eUb0eWAH4PSIeELS+sCkHNszM+uxPjXEEREPA8cCSFoNGBIRZ+bVnpnZ8ijiLI7cErSkO4F9Uxv3AQsk3R0Rx+XVpplZT0UBE3SeQxyrRsQrwIHAxIjYDvh4ju2ZmfVYERdLyjNB95c0DDgYuCHHdszMllu1xqAlrSfpDkkPS3pI0tdS+amSnpY0K21dLr8M+c7i+AFwKzA1IqZLGgk8mmN7ZmY91lq9WRwtwPERMTOt6nmfpD+lfT+PiJ9WWlGeFwknA5NL3j8OfDKv9szMlke1xqAjYj4wP71+VdIjLF3Zs1vyvEg4CBgLbAIMai+PiC/k1aaZWU91ZxaHpHHAuJKi8RExvpPjRgBbAPcAOwHHSDocmEHWy36pq3byHIOeBKwDjAHuInsu4as5tmdm1mMR3dmWPv0pbZ0l58FkS158PU2YuBDYABhF1sP+WbmY8kzQG0bE94HXI2IC2TMJt8uxPTOzHqvmLA5JA8iS8+URcS1ARDwXEa0R0QZcDGxbrp5cb/VOfy6UtCnZ4v1r5diemVmPVesioSQBlwKPRMQ5JeXD0vg0wAHAnHJ15Zmgx6c7CL8PXA8MBk7OsT0zsx6LqFpVOwGfAx6UNCuVfRc4VNIoIIB/A18uV1GeszguSS/vAkbm1Y6ZWTVUcRbHVOh0HOSm7tZV9QQtqctbuUu7/GZmRVHEW73z6EEPyaFOM7Nc9YnFkiLitGrXaWaWtyqOQVdNns8knFD6DEJJq0n6TV7tmZktj7a2poq3WslzFsdmEbGw/U1EvCRpixzbMzPrsQJ2oMv3oCUdmBb8QNKJkq5JU0XK1p2m2bXXszr5/kAwM+uxCFW81UolffVT04IfOwJ7AZcDv67gvJ8B0yT9UNIPgb8BZ/U8VDOzHEU3thqpJEG3pj/3AS6KiOuAgeVOioiJZIv1P5e2AyPCzyQ0s0IqYg+6kiGH+ZIuAPYAtpa0AhVeXEzPJXx4OeIzM6uJtrbiTbOrJNEeTHY34N5pabx3ASfmGpWZWa2FKt9qZJk9aEmrlLy9paTsNeDunOMyM6upIs6D7mqI4yGy4fDSHxft7wN4T45xmZnVViMl6IhYr5aBmJnVUxHX4qjoYp+kQyR9N70eLmmrfMMyM6uxRpxmJ+l8YBey9U0B3qCyedBmZg0j2lTxViuVTLPbMSK2lHQ/QES8mKbamZn1IsUb4qgkQTdLaiJ17CWtAbTlGpWZWa0V8CJhJWPQF5A9/HBNSacBU4Ezc43KzKzWCjgGXbYHHRETJd0HfDwVHRQRZR92aGbWUAo4i6PS1eX6kT2lO8hxDWkzs3op4o0qlczi+B5wJfBuYDhwhaTv5B2YmVlNtanyrUYq6UEfDmwREW8ASDoduB/4SZ6BmZnVkgrYg65oNbsOx/VPZWZmvUcjJWhJPycL+UXgIUm3pve7A9NrE56ZWY002EXC9pkaDwE3lpT/Pb9wzMzqpJF60BFxaS0DMTOrqwLefld2DFrSBsDpwAeBQe3lEbFRjnGZmdVWAYc4KpnTfBnwW7Ib1fcErgGuzjEmM7OaU1S+1UolCXqliLgVICIei4iTyBK1mVnv0Yi3egOL02JJj0k6CngaGJJvWGZmVkmC/gawMnAs2Vj0qsAX8gwK4Jsjts+7CWtAc565sN4hWCF9Y7lraMgbVSLinvTyVZYu2m9m1rtU6RZuSesBE4G1yQZExkfELyStTnb9bgTwb+DgiHipq7q6ulFlCl2MtkTEgd2O3MysqKrXg24Bjo+ImZKGAPdJ+hNwJHB7RJwh6UTgRODbXVXUVQ/6/GpFa2ZWdNUa4oiI+aTlMCLiVUmPAOsC+wGj02ETgDvpaYKOiNurEKuZWWPoRoKWNA4YV1I0PiLGd3LcCGAL4B5g7ZS8AZ4lGwLpUqXrQZuZ9W7dSNApGb8jIZeSNJjsaVRfj4hXpKVj3BERUvk+uxffNzOjujeqSBpAlpwvj4hrU/Fzkoal/cOABeXqqThBSxpY6bFmZg2nSgv2K+sqXwo8EhHnlOy6HjgivT4CuK5cSJU8UWVbSQ8Cj6b3m0v6ZbnzzMwaSRV70DuRTUneVdKstO0FnAHsJulRsme8nlGuokrGoM8D9gH+ABARsyXtUsF5ZmaNo3qzOKaSrV3UmY91p65KEnRTRDxZOsANtHanETOzomvIOwmBpyRtC4SkfsBXgbn5hmVmVmMNmqCPJhvmeA/wHHBbKjMz6zXUiAv2R8QC4JAaxGJmZiUqeaLKxXTS+Y+IcZ0cbmbWmBp0iOO2kteDgAOAp/IJx8ysPhryImFEvO3xVpImAVNzi8jMrB4aMUF3Yn0qWOTDzKyhNGKClvQSS0NvAl4kW8fUzKzXaLhZHOme8s3JnkMI0BYRBfw5Y2a2fIo4Bt3lWhwpGd8UEa1pK+BHMDOrggI+1buS1exmSdoi90jMzOqpgAm6q2cS9o+IFrKnAUyX9BjwOtkiIBERW9YoRjOz3BVxiKOrMeh7gS2BfWsUi5lZ/TRYghZARDxWo1jMzOqm0WZxrCnpuGXt7PCkADOzxtZgPeh+wGCWvfC0mVmv0Whj0PMj4gc1i8TMrJ4aLEG752xmfUeDJehuPTvLzKyRNdQQR0S8WMtAzMzqqaEStJlZn+IEbWZWUE7QZmbF5CEOM7OicoI2MyumRrvV28ysz/AQh5lZUTlBm5kVlBO0mVkxeYjDzKyg1Fa8DF3JMwnNzHq/Kj6TUNJvJC2QNKek7FRJT0ualba9ytXjBG1mRjbEUelWgcuAPTop/3lEjErbTeUqcYI2M4Oq9qAj4i/Aci845wRtZkb3etCSxkmaUbKNq7CZYyQ9kIZAVit3sBO0mRl0qwcdEeMjYuuSbXwFLVwIbACMAuYDPyt3gmdxmJmR/63eEfHcW21JFwM3lDvHPWgzM6p+kfCd9UvDSt4eAMxZ1rHt3IM2MwOI6s2DlnQlMBp4l6R5wCnAaEmjyAZK/g18uVw9TtBmZlT3TsKIOLST4ku7W48TdIEN3+jdnHTVN956v87ItZhwytVM+UXZ6ZPWyyxeDJ87FpY0Q0srjPkofPULS/ef/gu49ma475b6xdjwincjoRN0kc2b+wxHbXkCAE1NTVw57yLunnJvnaOyelhhBfjtz2HllaC5BT57DOy8HYzaBOb8A15+td4RNr4irgfti4QNYouPbcr8x55lwX/+W+9QrA6kLDkDtLRkSVqC1lY4+0L45tH1ja83UFvlW63UNEFLOrmW7fUmow/ZiTuuurveYVgdtbbCAWPhw/vDjlvD5h+Ey6fALjvBWmvUO7peIKLyrUZq3YP+Ylc7S+/OmReP1yqmwus/oD87fGJr7po8rd6hWB316wdTLoU7JsODj8D02XDrnfDZA+sdWe+Q9zS7nqj6GLSkV5a1C1ixq3PT3TjjAXZrOqiAQ/b1sc2eo/jXzCdYuODleodiBbDKENh2C7j3fvjP0zDmsKz8zUUw5jNw6xX1ja9hFTDj5HGRcCGwTeldM+0kPZVDe73eLod8mDuumlrvMKyOXlwI/ftlyXnRYpg2A8Z+Bv46ZekxW+3h5Lw8+sqC/ROB9wLvSNCAvz7dNGilgWy122ace1Qlt/pbb/X8C/CdH0NrG7QF7DEadtmx3lH1LkVcsL/qCToiTupi37er3V5vt+iNxXxyzS+UP9B6tY03gGvL3ObgOdDLqXj52fOgzcyg7wxxmJk1nr4wxGFm1pCKl5/zmwctaQNJA9Pr0ZKOlTQ0r/bMzJZHEedB53mjyu+BVkkbks1tXg/P4jCzglJbVLzVSp4Jui0iWsgWpv5lRJwADCtzjplZfVTxobHVkucYdLOkQ4EjgE+ksgE5tmdm1mOq4RoblcqzB/15YAfg9Ih4QtL6wKQc2zMz67m2bmw1klsPOiIeBo4FSI8XHxIRZ+bVnpnZ8uhTPWhJd0paRdLqwEzgYknn5NWemdlyKeAYdJ5DHKtGxCvAgcDEiNgO+HiO7ZmZ9Vhfm8XRPz1m/GDghhzbMTNbfgVcsD/PWRw/AG4FpkbEdEkjgUdzbM/MrMeK+EzCPC8STgYml7x/HPhkXu2ZmS2XAl4kzC1BSxoEjAU2AQa1l0eE1840s+IpXn7OdQx6ErAOMAa4CxgO+OHwZlZIamureKuVPBP0hhHxfeD1iJgA7A1sl2N7ZmY915duVAGa058LJW0KPAuslWN7ZmY9VsQbVfJM0OPTHYTfB64HBgMn59iemVnP9aUEHRGXpJd3ASPzasfMrCr6QoKWdFxX+yPCt3ubWfH0kXnQQ3Ko08wsV7WcnVGpqifoiDit2nWameWuikMckn4D7AMsiIhNU9nqwNXACODfwMER8VJX9eS5mt2E0mcQSlotBW1mVjzVXYvjMmCPDmUnArdHxPuA29P7LuU5D3qziFjY/ib9pNgix/bMzHquivOgI+IvwIsdivcDJqTXE4D9y9WT5zS7JkmrtXfhU/c+z/bMzHqsBvOg146I+en1s8Da5U7IM2H+DJgmqX3BpIOA03Nsz8ys57qRoCWNA8aVFI2PiPGVNxUhqWyDec6DnihpBrBrKjowPQbLzKx4WiufxZGSccUJOXlO0rCImJ/Wyl9Q7oRchxxSQnZSNrPiy3+I43rgCOCM9Od15U7I8yKhmVnjqOIsDklXAtOAjSXNkzSWLDHvJulRssf/nVGuHl+0MzMDqOKzBiPi0GXs+lh36nGCNjMDiD5wJ6GZWUPqxkXCWnGCNjODvrGanZlZQ3KCNjMrKCdoM7OC6gvLjZqZNST3oM3MCsqzOMzMiik8D9rMrKCqeCdhtThBm5mBx6DNzArLszjMzArKPWgzs2KK1tZ6h/AOTtBmZuCLhGZmheVpdmZmxRTuQZuZFZR70GZmxVTEi4SKAk4tsbeTNC495t3sLf5e9H5+qndjGFfvAKyQ/L3o5ZygzcwKygnazKygnKAbg8cZrTP+XvRyvkhoZlZQ7kGbmRWUE7SZWUE5QdeJpFMlfTOnureS9KCkf0k6T5LyaMeqL+fvxemSnpL0Wh71W/U5QfdOFwJfAt6Xtj3qG44VxB+BbesdhFXOCboGJB0u6QFJsyVN6mT/lyRNT/t/L2mlVH6QpDmp/C+pbBNJ90qalep8X4e6hgGrRMTfI7sCPBHYvwYf07qplt8LgPSdmJ//J7Nq8VocOZO0CXASsGNE/FfS6p0cdm1EXJyO/xEwFvglcDIwJiKeljQ0HXsU8IuIuFzSCkC/DnWtC8wreT8vlVmB1OF7YQ3IPej87QpMjoj/AkTEi50cs6mkv0p6EDgM2CSV3w1cJulLLP0PNw34rqRvA++NiDfzDd9y4u+FleUEXQyXAcdExIeA04BBABFxFFkvaz3gPklrRMQVwL7Am8BNknbtUNfTwPCS98NTmTWey6je98IakBN0/v4MHCRpDYBl/Co7BJgvaQBZT4l07AYRcU9EnAw8D6wnaSTweEScB1wHbFZaURpjfEXS9mn2xuHpOCuWmn4vrDE5QecsIh4CTgfukjQbOKeTw74P3EP2q+s/SsrPTtPl5gB/A2YDBwNzJM0CNiW7CNjRV4BLgH8BjwE3V+njWJXU43sh6SxJ84CVJM2TdGo1P5NVn2/1NjMrKPegzcwKygnazKygnKDNzArKCdrMrKCcoM3MCsoJ2t5BUmta02GOpMnta0D0sK7Rkm5Ir/eVdGIXxw6V9JUetNHpCnCVrAwn6TJJn+pGWyPS9Daz3DlBW2fejIhREbEpsIRsnYe3KNPt705EXB8RZ3RxyFCyOdxmhhO0lfdXYMPUc/ynpInAHLK713aXNE3SzNTTHgwgaQ9J/5A0EziwvSJJR0o6P71eW9KUtCLbbEk7AmcAG6Te+9npuBPSim4PSDqtpK7vSZoraSqwcbkPsayV4ZKPS5qR6tsnHd9P0tklbX+5kzrLriBntjycoG2ZJPUH9gQeTEXvA34VEZsAr5OtB/HxiNgSmAEcJ2kQcDHwCWArYJ1lVH8ecFdEbA5sCTwEnAg8lnrvJ0jaPbW5LTAK2ErSRyRtBRySyvYCtqng41wbEduk9h4hWxmu3YjUxt7Ar9NnGAu8HBHbpPq/JGn9DnW2ryA3Ctiat68iaLbcvNyodWbFdMswZD3oS4F3A09GxN9T+fbAB4G7syU/WIFsRbX3A09ExKMAkn4HjOukjV3J1gkhIlqBlyWt1uGY3dN2f3o/mCxhDwGmRMQbqY3rK/hMm6YlO4emem4t2XdNRLQBj0p6PH2G3YHNSsanV01tzy05bxrwPUnDyX4APFpBHGYVc4K2zryZeoVvSUn49dIi4E8RcWiH49523nIS8JOIuKhDG1/vQV2XAftHxGxJRwKjS/Z1XO8gUttfjYjSRI6kEW8dFHGFpHvIet43SfpyRPy5B7GZdcpDHNZTfwd2krQhgKSVJW1EtqjPCEkbpOMOXcb5twNHp3P7SVoVeJWsd9zuVuALJWPb60paC/gLsL+kFSUNIRtOKafTleGSgyQ1pZhHAv9MbR+djkfSRpJWLj3JK8hZ3tyDth6JiOdTT/RKSQNT8UkRMVfSOOBGSW+QDZEM6aSKrwHjJY0FWoGjI2KapLvTNLab0zj0B4BpqQf/GvDZiJgp6WqyVdwWANMrCLl9Zbjn05+lMf0HuBdYBTgqIhZJuoRsbHqmssaf552PDjsY+JykZuBZ4McVxGFWMa9mZ2ZWUB7iMDMrKCdoM7OCcoI2MysoJ2gzs4JygjYzKygnaDOzgnKCNjMrqP8Hg1iK9frpSb0AAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "\n",
        "ax= plt.subplot()\n",
        "sns.heatmap(Confusion_matrix, annot=True, ax = ax, cmap=\"viridis\"); #annot=True to annotate cells\n",
        "\n",
        "# labels, title and ticks\n",
        "ax.set_xlabel('Predicted labels');ax.set_ylabel('True labels'); \n",
        "ax.set_title('Confusion Matrix'); \n",
        "ax.xaxis.set_ticklabels(['class 0', 'class 1']); ax.yaxis.set_ticklabels(['class 0', 'class 1']);\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U72drVrQyIkP"
      },
      "source": [
        "Heatmap from Seaborn can be used for better visualization of confusion matrix. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W8_6zQ08yIkQ"
      },
      "source": [
        "## 6.3 Printing Classification Report"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XotaTZf3yIkQ",
        "outputId": "dc3a3408-7dc7-4413-c785-19cdc243d87d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The detailed Classification Report\n",
            "----------------------------------\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "     class 0       0.82      0.64      0.72        50\n",
            "     class 1       0.65      0.83      0.73        41\n",
            "\n",
            "    accuracy                           0.73        91\n",
            "   macro avg       0.74      0.73      0.73        91\n",
            "weighted avg       0.75      0.73      0.72        91\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Model detailed classification report\n",
        "target_names = ['class 0', 'class 1']\n",
        "\n",
        "\n",
        "print(\"The detailed Classification Report\")\n",
        "print(\"----------------------------------\")\n",
        "print(metrics.classification_report(Y_test, DT_model_Holdout_pred_test,target_names =target_names))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TTmeVIVmyIkQ"
      },
      "source": [
        "classfication_report() function in scikit learn allows to inspect performance of model on indicidual class. We can see that on class 0, precision is 86%(32/(32+5)) whereas, it is 67%(36/(36+18)) for class 1. It also detailed recall, f1-score and weighted average to judge model performance"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qMmw_zHcyIkQ"
      },
      "source": [
        "## 6.4  Plotting ROC curve\n",
        "\n",
        "ROC(Receiver Operating Characteristic) curve the true positive rate (Sensitivity) is plotted in function of the false positive rate (100-Specificity) for different cut-off points of a parameter. Each point on the ROC curve represents a sensitivity/specificity pair corresponding to a particular decision threshold. The area under the ROC curve (AUC) is a measure of how well a parameter can distinguish between two different classes in the present in the data set. The formula for TPR and FPR are following:\n",
        "\n",
        "TPR = TP(TP+FN) and FPR = FP(FP+TN)\n",
        "\n",
        "In order to plot ROC curve, we need to use predict_proba() method with the fitted function while model learning as it returns the probability of an instance belonging to different classes. For a binary classification problem, predict_proba() returns class probabilities for each class. The first column contains the probability of the first class and the second column contains the probability of the second class. In order to plot ROC, we use the second column to the probabilities generated by predict_proba() function and pass it to  metrics.auc()."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wD-_mEUlyIkQ",
        "outputId": "038c2779-e2b2-4e25-f881-76677f995dd3"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xm4jfUWwPHvogxJ6tKtzDIkU9K5aKIulaRUSpopuU23SYNu83Abb/OkQsOtSElUSgMShbjKmLk4SIgMmQ7r/rHeY2/HGfY5zj7vHtbnefZjD+/e79qvffbav+FdP1FVnHPOubyUCjsA55xzic0ThXPOuXx5onDOOZcvTxTOOefy5YnCOedcvjxROOecy5cnChczEblQRD4PO45EIiIbROTQEPZbW0RURPYq6X3Hg4jMFJETivA8/0yWAE8USUpEfhaRTcEX1a8i8rqI7BvPfarq26p6cjz3EU1EjhGRUSKyXkT+EJGPRKRRSe0/l3jGiEjP6PtUdV9VXRin/TUQkfdEZFXw/qeJyE0iUjoe+yuqIGHV25PXUNXGqjqmgP3slhxL+jOZrjxRJLfTVXVfoDlwJHB7yPEUSW6/ikXkaOBzYBhQFagD/AiMj8cv+ET7ZS4idYGJwBKgqapWAs4FMoCKxbyv0N57oh13lwdV9UsSXoCfgfZRtx8DPom6XRb4D7AYWAH0BcpHPd4Z+AFYBywAOgT3VwL6A8uBpcCDQOngse7AuOD6S8B/csQ0DLgpuF4VGAKsBBYB10Vtdy/wPvBWsP+euby/b4AXc7n/U+DN4PoJQCbwL2BVcEwujOUYRD33NuBX4L/AAcDHQcxrguvVg+3/DWwHNgMbgOeD+xWoF1x/HXgB+ARYj33R142K52RgDvAH8CLwdW7vPdj2rej/z1werx3s+9Lg/a0C7oh6vCXwHbA2+L98HigT9bgC1wDzgEXBfc9giWkdMAU4Pmr70sFxXhC8tylADWBs8Fobg+NyXrB9J+zztRb4FmiW47N7GzAN2ALsRdTnOYh9chDHCuDJ4P7Fwb42BJejifpMBts0Br4Afg+e+6+w/1ZT4RJ6AH4p4n/crn9Y1YHpwDNRjz8FDAf+gv0C/Qh4OHisZfBldRLWqqwGNAweGwq8DFQA/gpMAv4RPLbzjxJoE3ypSHD7AGATliBKBV8kdwNlgEOBhcApwbb3AtuAM4Nty+d4b/tgX8on5vK+ewDLg+snAFnAk1hSaBt8YR0WwzHIfu6jwXPLA5WBLsH+KwLvAR9G7XsMOb7Y2T1RrA6O717A28Cg4LEqwRff2cFj1wfHIK9E8SvQI5///9rBvl8NYj8C+9I9PHj8KKB1sK/awGzghhxxfxEcm+zkeVFwDPYCegcxlAseuwX7jB0GSLC/yjmPQXD7SOA3oBWWYC7FPq9loz67P2CJpnzUfdmf5++Ai4Pr+wKtc7znvaL21Z3IZ7IilhR7A+WC263C/ltNhUvoAfiliP9x9oe1Aft1p8BXwP7BY4J9YUb/mj2ayC/Hl4GncnnNg4Ivm+iWx/nA6OB69B+lYL/w2gS3rwBGBddbAYtzvPbtwGvB9XuBsfm8t+rBe2qYy2MdgG3B9ROwL/sKUY8PBu6K4RicAGzN/iLMI47mwJqo22MoOFH0i3qsI/BTcP0S4LuoxwRLtHklim0Erbw8Hs/+0qwedd8koFse298ADM0R998L+IytAY4Irs8BOuexXc5E8RLwQI5t5gBtoz67l+Xyec5OFGOB+4AqebznvBLF+cDUeP7dpevF+weT25mq+qWItAXewX61rgUOxH4VTxGR7G0F+3UH9ktuRC6vVwvYG1ge9bxS2BfaLlRVRWQQ9sc5FrgA6y7Jfp2qIrI26imlse6kbLu9ZpQ1wA7gEOCnHI8dgnWz7NxWVTdG3f4Fa9UUdAwAVqrq5p0PiuyDtUI6YC0kgIoiUlpVt+cTb7Rfo67/if0iJohp53sOjl9mPq+zGnuvRdqfiDTAWloZ2HHYC2vlRdvl/0BEbgYuD2JVYD/sMwX2mVkQQzxg//+Xisg/o+4rE7xurvvO4XLgfuAnEVkE3KeqH8ew38LE6ArBB7NTgKp+jf2a/U9w1yqsG6ixqu4fXCqpDXyD/ZHWzeWllmAtiipRz9tPVRvnseuBwDkiUgtrRQyJep1FUa+xv6pWVNWO0WHn8342Yt0P5+bycFes9ZTtABGpEHW7JrAshmOQWwy9sa6VVqq6H9a9BpZg8o05BsuxlpK9oGWv6nlvzpdYN1hRvYQl2frBe/kXkfeRbef7EZHjgVux43uAqu6PdU9mPyevz0xulgD/zvH/v4+qDsxt3zmp6jxVPR/r+nwUeD/4Py7o+C/BujldMfNEkTqeBk4SkSNUdQfWd/2UiPwVQESqicgpwbb9gR4i0k5ESgWPNVTV5dhMoydEZL/gsbpBi2U3qjoV+0LuB4xU1ewWxCRgvYjcJiLlRaS0iDQRkb8V4v30wX6VXiciFUXkABF5EOs+ui/HtveJSJngy64T8F4MxyA3FbHkslZE/gLck+PxFRT9i+gToKmInBnM9LkGODif7e8BjhGRx0Xk4CD+eiLylojsH8P+KmJjIhtEpCFwVQzbZ2ED+XuJyN1YiyJbP+ABEakvppmIVA4ey3lcXgWuFJFWwbYVROQ0EYlptpaIXCQiBwb/h9mfqR1BbDvI+//gY+AQEblBRMoGn5tWsezT5c8TRYpQ1ZXAm9gAMtiskvnABBFZh/1CPSzYdhI2KPwU9qvxa6y7AKwvvQwwC+sCep/8u0DeAdoH/2bHsh37wm6OzXjKTiaVCvF+xgGnYIO/y7EupSOB41R1XtSmvwZxLsMGj69U1ezuqjyPQR6exgaGVwETgM9yPP4M1oJaIyLPxvpegvezCmshPYZ1KzXCZvZsyWP7BVhSrA3MFJE/sBbbZGxcqiA3Y92B67Ev7ncL2H4k9n7nYsd6M7t2Dz2Jjf98jiWg/tixAhtzekNE1opIV1WdjI1ZPY/938zHxhJi1QF7zxuwY95NVTep6p/Y7LPxwb5aRz9JVddjEzROxz4X84ATC7Ffl4fsGSvOJZ3gTN63VDW/LpyEJCKlsOm5F6rq6LDjcS4/3qJwroSIyCkisr+IlCUyZjAh5LCcK1DcEoWIDBCR30RkRh6Pi4g8KyLzg9IELeIVi3MJ4mhsVs4qrHvkTFXdFG5IzhUsbl1PItIGm+f/pqo2yeXxjsA/sbnmrbCTxXzgyTnnEkzcWhSqOhY7jT4vnbEkoqo6AdhfRGKZN+6cc64EhXnCXTV2nVWRGdy3POeGItIL6AVQoUKFoxo2bFgiATrnXLLYuhU2bdr1snkz1NBf2J+1TCNrlaoeWJTXToozs1X1FeAVgIyMDJ08eXLIETnnXDh++w2mT4cZM+wyfTrMnAkbNkS2qVVTObo1NG0mnPXrS9Qs9xuH9L33l6LuM8xEsRQ75T5b9eA+55xLe+vXWwLITgrZ/65cGdmmShVo2hR69LB/mzSBJgcspeKtV8F558GFF7LzXMu+9xY5ljATxXDg2qBeUCvgj+DMYOecSxtbtsCcObu3En6J+v1foYIlgTPOiEoITeCgg6JeSBX69YObb4Zt2+C004otxrglChEZiFXorBIUP7sHKziHqvbFitJ1xM7a/BM7U9g551LS9u2waNHuCWHuXHsMYO+9oWFDOOYY6NUrkhRq1YJS+U09WrAArrgCRo+GE0+EV1+FurGW5ipY3BJFUNQrv8ezF05xzrmUoQrLl+/aXTRjhnUjbQrOmhGBQw+1JHD22ZGEUL8+lClThJ1Onw5TpsArr0DPnraDYpQUg9nOOZeI1qzZdRwhOzmsWRPZ5uCDLRFceWUkITRqZN1Je2TGDPjf/+CSS+DMM2HhQqhcueDnFYEnCuecK8CmTTB79u4JYWnU9Jv99rNE0LWrJYOmTaFxYxtwLlZbt8JDD9nloINsh+XKxS1JgCcK55zbKSsL5s/ftdto+nQbAtixw7YpW9ZaBH//eyQhNGkC1asXe4/P7iZOhMsvt2bMRRfBU09ZkogzTxTOubSjCkuW7D6OMHu2zUICGzyuVw+aNYMLLogkhbp1Ya8wvjmXLoXjj7dWxMcfF+uspoJ4onDOpbRVq3btLsq+vm5dZJvq1S0JnHRSJCE0bAjly+f9uiVm7lxo0ACqVYN334V27ayfqwR5onDOpYQNG2DWrN0Twq9Rq4r/5S+WBC6+eNdxhP1jWTOwpK1dC7feaudGjBkDbdrAWWeFEoonCudcUtm2zU5Qy5kQFi6MbFO+vCWAU0+NnJzWtKnNQIr7OEJxGD4crrrKstwtt8DfCrOKcPHzROGcS0g7dsDPP+/ebTRnjiULgNKl4bDD7Hu0R49IQqhTp4AT1BJZz57Qv7+9kWHDICMj7Ig8UTjnwqUKK1bsnhBmzoSNGyPb1a5tieD00yMJoUEDm4WU9LLXBRKxxFCrFtx2WxHPvit+niiccyVm3brcB5ZXrYps89e/WiLo2TPSbdS4MVSsGF7ccbVkiZ2N162bDZ5ceWXYEe3GE4Vzrtht2WJTTXMmhcWLI9vsu68lgbPOiiSEJk0sUaSFHTvg5Zet5bB9e2gD1bHwROGcK7Lt2+1ktJwJYd68XQvdHX44HHfcrpVPa9VKkoHleJg3z5pMY8dC+/ZWo6lOnbCjypMnCudcgVRh2bLdS1jMmmWrqIF96deta0ng3HMjCaF+fUsWLsqsWTBtGgwYAN27J3zG9EThnNvF779HkkF0K2Ht2sg2VataErjmmkhCaNQI9tknvLgT3o8/wg8/wKWXQufONp/3gAPCjiomniicS1N//rlrobvsf5cti2xTqZJ1F3XrFuk2atw4rvXnUs+WLfDgg/DII3DIIbbyXLlySZMkwBOFcykvK8u6xHN2Gy1YEJmVWa6ctQjat991HKFatYTvFUls331nRfxmz7Zy4E8+WSJF/IqbJwrnUoSqzSrKmRB++skqU4OdhNagATRvHilj0aSJjS2ULh1u/Cln6VJo29ZOBx8xwk4TT1KeKJxLQitX7n4uwowZsH59ZJuaNS0JdOgQaSU0bJiUP2iTy+zZNs2rWjUYPNiK+CX5SSCeKJxLYOvX2wSZnK2E336LbFO5siWCSy/ddRyhUqXw4k5La9ZA797w2ms27fX4423luRTgicK5BLB1q9UwypkQfv45ss0++1gS6NRp1wVzDjrIxxFCN3QoXH21NfVuvz30In7FzROFcyVoxw5YtGj3bqM5c2zQGWxRnIYNoXVrOycrOyHUrp3Ehe5S2WWXWSuieXP45BNo0SLsiIqdJwrn4kDVKkTnTAgzZ9q01Gx16lgi6Nw5khAaNEiYWnAuL9FF/Fq3trMKb745Zc8s9ETh3B5au9YSQM6B5dWrI9scdJAlgl69It1GjRpZvSOXZH75Bf7xD1sf9ZJL7D81xXmicC5GmzdHCt1Fn6SWmRnZZr/9LBF06bLrCmoHHhhe3K6Y7NgBL70EffpYi+Lcc8OOqMR4onAuh6ysSKG76FbCvHn2XQHWNXT44XDCCbsOLNeo4QPLKWnOHBswGjcOTj7Zqr7Wrh12VCXGE4VLW6rWGsiZEGbNsqoLYF/69epZIjjvvEhSqFfPBp1dmpgzx/oXX3/dupvS7NeAf9RdWli9OvcFc/74I7JNtWqWBNq1iySEhg290F3amjrVivj16AFnnGFF/PbfP+yoQuGJwqWUjRutRZAzISxfHtlm//0tCVx44a4L5iRRjTYXT5s3w/33w2OP2a+H88+309nTNEmAJwqXpLZtg7lzd28lLFwYmblYvrzNLDrllEgyaNrUCnimWc+Bi9X48VbEb84ca0k88YTXPMEThUtwO3bYbMScCeGnnyxZgBWza9DAznO69NJIQqhTxwvduUJYuhROPNFaESNH2qC1AzxRuAQzaZJVZs5OCjNnwoYNkcdr1bJEcNppkVZCw4ZQtmx4MbskN2uWNT2rVYMhQyxZ+Akuu/BE4RLGhx9G1pevUsVaBZddFkkIjRvbeQrOFYvff4ebboI33oCvv4Y2beD008OOKiF5onAJYfVquPJKOPJI+PRTO5PZubgZMsTWcV29Gu64A1q2DDuihOaJwiWEf/7TfuB9/rknCRdn3btbK6JFC/jsMyvm5/LlicKF7oMPYOBAm5HYrFnY0biUFF3E75hj7LT63r39rMkYxbVosYh0EJE5IjJfRPrk8nhNERktIlNFZJqIdIxnPC7xrFoFV11lP+767PYJca4YLFpkM5jefNNu9+oFt93mSaIQ4pYoRKQ08AJwKtAIOF9EGuXY7E5gsKoeCXQDXoxXPC4xXXutLQz2+uspW6HZhWX7dnj2WZsJMWFCpFXhCi2eLYqWwHxVXaiqW4FBQOcc2yiQPY+lErAsjvG4BDNkCLz7Ltx9t81wcq7YzJ5tS5Fefz20bWvzrLt3DzuqpBXPtlc1YEnU7UygVY5t7gU+F5F/AhWA9rm9kIj0AnoB1KxZs9gDdSVv5UrrcjrqKOsFcK5YzZ9vZ1f/979Wq8VPxd8jYS+seD7wuqpWBzoC/xWR3WJS1VdUNUNVMw70wv4p4dprrSCfdzm5YjNlCgwYYNdPP93GJi66yJNEMYhnolgK1Ii6XT24L9rlwGAAVf0OKAdUiWNMLgG8/z4MHgz33GPdx87tkU2bbCZEq1bwwANW1A/87MxiFM9E8T1QX0TqiEgZbLB6eI5tFgPtAETkcCxRrIxjTC5kK1fC1VdDRgbcemvY0bikN3YsHHEEPPqojUFMnepF/OIgbmMUqpolItcCI4HSwABVnSki9wOTVXU40Bt4VURuxAa2u6v61IRUds01kS4nn53o9sjSpbZ4SI0a8OWXdt3FRVz/VFV1BDAix313R12fBRwbzxhc4hg8GN57Dx56yOo2OVck06fbNLlq1WDoUCviV6FC2FGltLAHs12a+O03a0387W9wyy1hR+OS0qpVcPHFdvr+2LF2X6dOniRKgDf+Xdyp2rjEunXe5eSKQNWaotlnZ95zjw1cuxLjf7Iu7gYPtpPrHn7Yyv47VyiXXmrnQ2RkwFdf+dmZIfBE4eJqxQrrcmrZEm6+OexoXNKILuLXtq11N91wgzdHQ+JjFC5usrucNmzwLidXCAsXQvv29qEBW8P65pv9AxQiTxQubt5910qI33+/VXV2Ll/bt8PTT1vX0vffQyn/ekoUnqJdXKxYYWOPrVpZ2X/n8jVrlq17O3GiLYjety9Urx52VC7gicIVO1Ur+Jfd5VS6dNgRuYS3aBEsWADvvAPdunl9pgTjicIVu4ED7Tyoxx6Dhg3DjsYlrO+/hx9+gCuusFbEwoVQsWLYUblceCegK1a//mrrX7duDTfdFHY0LiH9+acNTrdubXOms4v4eZJIWJ4oXLFRhSuvhI0b4bXXvMvJ5WLMGJvq+sQT1pLwIn5JwbueXLF55x0YNgz+8x/vcnK5yMyEk06CWrVg1Cir0eSSgrcoXLFYvty6nI45xs6Lcm6nH3+0f6tXt18S06Z5kkgynijcHsvuctq0ybucXJSVK+GCC6B5c/j6a7uvY0fYZ59w43KF5l1Pbo+9/TYMH27dzg0ahB2NC50qDBoE111ni4/cdx8cfXTYUbk9EFOiCFaoq6mq8+Mcj0syy5bZ98Gxx8L114cdjUsIF19svx5atYL+/X3xkRRQYNeTiJwGTAe+CG43F5Gh8Q7MJT5V+Mc/rMtpwADvckprO3ZECvmdeCI8+SSMH+9JIkXEMkZxP9AKWAugqj8A9eIZlEsO//0vfPyxrVjnXU5pbP58W4b0tdfs9uWXw403+i+HFBJLotimqmtz3OfrWqe5Zcusq+nYY63ryaWhrCybC920qZ0PUaZM2BG5OIlljGK2iHQFSolIHeA6YEJ8w3KJTBV69YItW3yWU9qaMQN69IDJk6FzZ3jxRahaNeyoXJzE0qK4FjgK2AF8AGwBfNgyjb35JnzyiVVfqF8/7GhcKBYvhl9+sdlNQ4d6kkhxopp/L5KInK2qHxR0X0nJyMjQyZMnh7FrByxdauOTzZpZNQZfMiCNTJxoJ8/16mW3N2yAffcNNyYXMxGZoqoZRXluLH/md+Zy3x1F2ZlLbtldTlu32iwnTxJpYuNGq/B49NFWEnjLFrvfk0TayHOMQkROAToA1UTkyaiH9sO6oVyaeeMNGDECnnkG6vm8t/QwapQV71u40BYZeeQRKFs27KhcCctvMPs3YAawGZgZdf96oE88g3KJJzPTZjm1aWMr17k0kJkJp5wCdepYCY42bcKOyIUkz0ShqlOBqSLytqpuLsGYXIJRtR+VWVne5ZQWpk6FI4+0In4ffQRt20L58mFH5UIUy598NREZJCLTRGRu9iXukbmE8dpr8Nln1utQt27Y0bi4WbECzjsPWrSIFPHr0MGThIspUbwOvAYIcCowGHg3jjG5BLJkiZ1k27YtXHNN2NG4uFCFt96CRo3gww/hwQetXrxzgVgSxT6qOhJAVReo6p1YwnApzruc0sQFF1ghv8MOszWs77gD9t477KhcAonlzOwtIlIKWCAiVwJLAV/cNg0MGAAjR8Lzz8Ohh4YdjStWO3aAiF1OPtmmvl5zjZ9m73IVywl3rYBZwAHAv4FKwKOqOj7+4e3OT7grGUuWQJMm1l391Vfemkgpc+daU/GSS6yAn0sLe3LCXYEtClWdGFxdD1wc7LBaUXbmkkN2l9P27d7llFKysqz89z33QLlyPkjtYpbvV4CI/E1EzhSRKsHtxiLyJjAxv+e55Na/v3U5PfaYTaF3KWDaNGjdGm67DU49FWbNsrEJ52KQZ6IQkYeBt4ELgc9E5F5gNPAj4KsPpKjFi61aw4kn2jrYLkVkZlp/4nvvwZAhcMghYUfkkkh+XU+dgSNUdZOI/AVYAjRV1YWxvriIdACeAUoD/VT1kVy26Qrci61x8aOq+s+ckKhCz542ztm/v3c5Jb1vv7WWxJVXQseOVoajQoWwo3JJKL+vgs2quglAVX8H5hYySZQGXsCm0jYCzheRRjm2qQ/cDhyrqo2BGwoZvytGr74KX3wBjz/uXU5JbcMGq7dy3HHwxBORIn6eJFwR5deiOFREskuJC1An6jaqenYBr90SmJ+dXERkENZKmRW1zRXAC6q6JnjN3woZvysmv/wCvXvD3/9u62C7JPX551bid/Fim+760ENexM/tsfwSRZcct58v5GtXw7qrsmVia29HawAgIuOx7ql7VfWznC8kIr2AXgA1a9YsZBiuINldTuBdTkltyRI47TSrszJ2rLUonCsG+RUF/KqE9l8fOAGoDowVkaY51+hW1VeAV8DOoyiBuNLKK6/Al19C375Qu3bY0bhCmzIFjjoKatSwOvDHH2/TX50rJvH87bgUqBF1u3pwX7RMYLiqblPVRcBcLHG4EvLzz3DzzdC+fWThMpckfv0Vzj0XMjIiRfxOOsmThCt28UwU3wP1RaSOiJQBugHDc2zzIdaaIDhXowEQ84C52zM7dtiJuSLQr5/965KAqq0i1aiRlQF/6CEv4ufiKpZaTwCISFlV3RLr9qqaJSLXAiOx8YcBqjpTRO4HJqvq8OCxk0VkFrAduEVVVxfuLbiievllW8Ds5ZehVq2wo3Ex69YNBg+GY4+1DN+wYdgRuRQXS62nlkB/oJKq1hSRI4CeqvrPkggwJ6/1VDwWLYKmTe2H6MiR3ppIeNFF/N54A9avh6uv9pkHLmZ7Uusplk/Zs0AnYDWAqv4InFiUnbnEkN3lVKqUdzklhZ9+smVI+/e325deauvRepJwJSSWT1opVf0lx33b4xGMKxl9+8Lo0XYuls82TmDbttn4wxFHWG2mffcNOyKXpmIZo1gSdD9pcLb1P7HZSS4JLVoEt95qSxBknzvhEtAPP0CPHvbvOefAc8/BwQeHHZVLU7Ekiquw7qeawArgy+A+l2R27IDLLrO1abzLKcH9+qtdhgyBswsqguBcfMWSKLJUtVvcI3Fx99JLMGaMJYkaNQrc3JW0ceOsiN/VV0OHDrBgAeyzT9hRORfTGMX3IjJCRC4VEV8CNUktXGhdTh06WKvCJZD1621w+vjj4emnI0X8PEm4BFFgolDVusCDwFHAdBH5UES8hZFEsruc9trLKsR6l1MCGTnS1px98UWr+Pq//3kRP5dwYppfp6rfqup1QAtgHbagkUsSL7xgFR6eegqqVw87GrfTkiXQqZO1HMaNs9aEz2xyCajARCEi+4rIhSLyETAJWAl4vYAksWAB9Oljq1/26BF2NA5VmDTJrteoAZ9+ClOnegkOl9BiaVHMAFoDj6lqPVXtraq+ZnYSyO5y2ntvqxDrXU4hW74cunSBVq0iRfzat/cifi7hxTLr6VBV3RH3SFyxe/55W5ZgwADvcgqVKrz+ui1GvnkzPPqo1WlyLknkmShE5AlV7Q0MEZHdCkLFsMKdC9H8+dbl1LEjdO8edjRprmtXeP99m9XUrx80aBB2RM4VSn4tineDfwu7sp0LWXaXU5ky3uUUmu3b7cCXKgWnnx5ZY9brM7kklN8Kd8GIG4er6i7JIigfXhIr4LkieO45+OYb6+2oVi3saNLQ7NlWdbFHD7jiCrjkkrAjcm6PxPLzJrfTsy4v7kBc8Zg3D26/3ZZO9u+nErZtGzz4IDRvDnPmQKVKYUfkXLHIb4ziPGxVujoi8kHUQxWBtbk/y4Vp+3b7EVu2rHc5lbipU20waNo0OO88ePZZ+Otfw47KuWKR3xjFJGwNiurAC1H3rwemxjMoVzTPPgvjx9u6NlWrhh1NmlmxAlatgg8/hM6dw47GuWJV4Ap3icZXuMvd3Lm2bEH79jB8uLcmSsTYsTB9Olxzjd3etAnKlw83JufyEJcV7kTk6+DfNSLye9RljYj8XtRgXfHL7nIqV87Wv/YkEWfr1lmF17ZtrRmXXcTPk4RLUfl1PWUvd1qlJAJxRffMM/Dtt/Dmm97lFHcjRtg012XL7AS6++/3In4u5eXZoog6G7sGUFpVtwNHA/8AKpRAbC4Gc+bAHXfAGWfARReFHU2KW7LExh8qVbLM/MQTUMH/FFzqi2V67IfYMqh1gdeA+sA7cY1feH7HAAAdkklEQVTKxSS7y6l8eVsH27uc4kAVJkyw6zVqwOefWynwVq3Cjcu5EhRLotihqtuAs4HnVPVGwE/jSgBPPw3ffWcn2B1ySNjRpKBly+DMM+HooyNF/E480U55dy6NxJIoskTkXOBi4OPgvr3jF5KLxU8/wZ13Wk/IBReEHU2KUbWaTI0aWQviP//xIn4urcVSPfYy4GqszPhCEakDDIxvWC4/2V1O++zjXU5xcc458MEHNqupXz+oVy/siJwLVYGJQlVniMh1QD0RaQjMV9V/xz80l5cnn7Ru87ffhoMPDjuaFBFdxO/MM+Hkk61Okxfxcy6mFe6OB+YD/YEBwFwR8XZ4SGbPhrvusu+y888PO5oUMWOGdS3172+3L77YK706FyWWv4SngI6qeqyqHgOcBjwT37BcbrK7nCpUgJde8i6nPbZ1K9x3H7RoYWvGHnBA2BE5l5BiGaMoo6qzsm+o6mwR8WkfIXjiCZg4Ed55x7uc9tiUKVbEb8YMmw3w9NNw4IFhR+VcQoolUfxPRPoCbwW3L8SLApa42bPh7rvh7LOhW7ewo0kBq1fD2rXw0UfQqVPY0TiX0AosCigi5YDrgOOCu77BzqfYHOfYcpWORQGzsqwLfcECmDkTDjoo7IiS1OjRVsTvuuvs9ubNViDLuTSwJ0UB821RiEhToC4wVFUfK8oO3J574gmYNAkGDfIkUSR//AG33mqLdDRsaAPVZct6knAuRvlVj/0XVr7jQuALEcltpTsXZzNnWpdTly7QtWvY0SShjz6yE+f69YObb7axCS/i51yh5NeiuBBopqobReRAYAQ2PdaVkKwsG2/dbz948UWf5VRoS5ZYhm3Y0BYU+tvfwo7IuaSUX6LYoqobAVR1pYj4pPIS9vjjMHkyvPuur6oZM1UrgHXMMZEifscc4/WZnNsD+X35HyoiHwSXoUDdqNsf5PO8nUSkg4jMEZH5ItInn+26iIiKSJEGWlLRjBlw771WTcK7nGKUmWn11o89NlLE74QTPEk4t4fya1F0yXH7+cK8sIiUxtbaPgnIBL4XkeHR52QE21UErgcmFub1U1lWlp1Yt99+8MILBW+f9nbsgFdfhVtusYP35JNw3HEFP885F5M8E4WqfrWHr90Sqwu1EEBEBgGdgVk5tnsAeBS4ZQ/3lzIee8y6nAYP9i6nmHTpYmMQf/+7JYxDDw07IudSSjzHHaoBS6JuZ5JjHQsRaQHUUNVP8nshEeklIpNFZPLKlSuLP9IEkt3l1LUrnHtu2NEksKwsa0mAJYpXX4Uvv/Qk4VwchDZAHQyOPwn0LmhbVX1FVTNUNePAFC6zsG2bzXLaf394vlAdfWlm2jRbTOjVV+32RRdBz54+Lcy5OIk5UYhIYSefL8XW285WPbgvW0WgCTBGRH4GWgPD03lA+7HHbJr/Sy952aFcbdkC99wDRx0Fv/ziB8m5EhJLmfGWIjIdmBfcPkJEnovhtb8H6otInaCIYDdgePaDqvqHqlZR1dqqWhuYAJyhqulVnyMwbZoVMj3vPOtJcTl8/71Veb3/fquvPnu2Fb5yzsVdLC2KZ4FOwGoAVf0ROLGgJ6lqFnAtMBKYDQxW1Zkicr+InFH0kFNPdpfTAQd4l1Oe1qyBDRtgxAh4802oXDnsiJxLG7FUjy2lqr/Irv2/22N5cVUdgZ3RHX3f3Xlse0Isr5mKHnkEpk6FIUOgSpWwo0kgo0ZZEb/rr7cV5+bO9fIbzoUglhbFEhFpCaiIlBaRG4C5cY4rbUybBg88YKXDvSclsHatLUParh28/LKNTYAnCedCEkuiuAq4CagJrMAGna+KZ1DpIrrL6blYRn3SwbBhVsRvwACr+OpF/JwLXYFdT6r6GzYQ7YrZww9bl9MHH3iXEwCLF9vJI4cfDsOHQ0baToBzLqEUmChE5FVgt9WNVLVXXCJKEz/+aF1OF1wAZ50VdjQhUoVx4+D446FmTTtprnVrr8/kXAKJpevpS+Cr4DIe+CuwJZ5BpbqtW63LqXJlePbZsKMJ0eLFcNpp0KZNpIhfmzaeJJxLMLF0Pb0bfVtE/guMi1tEaeChh+CHH6w8UVrO8tyxA/r2hdtusxbFs896ET/nElgs02NzqgP4gpxF9MMP8O9/w4UXQufOYUcTkrPPtkHrk06y5Ulr1w47IudcPmIZo1hDZIyiFPA7kOfaEi5vW7fCpZfawHXadTllZUGpUnY57zzLkt27e30m55JAvolC7Cy7I4jUaNqhqrsNbLvY/Pvfdt7EsGHwl7+EHU0J+vFHuOwyOzfiyiutBIdzLmnkO5gdJIURqro9uHiSKKKpU21s4qKLbBG2tLB5M9x5p01zzcyEgw8OOyLnXBHEMuvpBxE5Mu6RpLDsWU5VqsAzz4QdTQmZNAmOPDIyIDN7Npx5ZthROeeKIM+uJxHZKyjsdyS2jOkCYCMgWGOjRQnFmPQefNC6nIYPT6Mup3XrYNMm+OwzOOWUsKNxzu2B/MYoJgEtgHTpKImL//3PupwuuQROPz3saOLs889h5ky48UZo3x7mzPHyG86lgPwShQCo6oISiiXlbNliXU4HHQRPPx12NHG0Zg3cdBO8/jo0bgxXX20JwpOEcykhv0RxoIjclNeDqvpkHOJJKQ88YFWyP/7YCv+lpA8+gGuugZUr4fbb4e67PUE4l2LySxSlgX0JWhaucCZPtnUmLr3UqlSkpMWLrT56kya2oNCRPufBuVSUX6JYrqr3l1gkKSSlu5xUYexYaNvWiviNGgWtWsHee4cdmXMuTvKbHustiSK6/34b0331Vdh//7CjKUa//AKnngonnBAp4nfccZ4knEtx+SWKdiUWRQqZPBkefdRaFB07hh1NMdmxwxbzbtzYSoI/95yVBXfOpYU8u55U9feSDCQVZHc5HXwwPPVU2NEUozPPhI8+svMhXn4ZatUKOyLnXAkqSvVYl4f77rMupxEjUqDLads2KF3aividfz6ccw5cfLEX8XMuDcVSwsPF4PvvrcvpssusGz+p/e9/0LKlrRkBliguucSThHNpyhNFMdi82abBVq0KTybz2SWbNtm5EC1bwq+/Qo0aYUfknEsA3vVUDO6912reffYZVKoUdjRFNGGCZbu5c61Z9J//pPBZgs65wvBEsYcmToTHH4fLL0/y2ncbN9q4xBdfWJ0m55wLSLItMZGRkaGTJ08OOwzAupyOPBI2bIAZM5KwNfHZZzb63ru33d66FcqUCTcm51xciMgUVc0oynN9jGIP3HMP/PQT9OuXZEli9WrrZjr1VHjjDUsQ4EnCOZcrTxRFNGGCdeP37JlEXU6q8P770KgRvPOOrT73/feeIJxz+fIxiiLYvBl69IBq1eCJJ8KOphAWL4YLLoBmzWztiCOOCDsi51wS8ERRBHffbV1On38O++0XdjQFUIXRo+Hvf7czqseMsemve/l/vXMuNt71VEjffWetiF694KSTwo6mAIsWwcknQ7t2kSJ+xxzjScI5VyieKAph0yar5VS9uk2JTVjbt8Mzz9g6ERMnwksveRE/51yR+U/LQrjrLjsf7YsvErzLqXNn+OQTK1/bt6+fYe2c2yOeKGL07bdWnuMf/0jQ89Gii/hdfLHVZ7rgAq/P5JzbY3HtehKRDiIyR0Tmi0ifXB6/SURmicg0EflKRBKyfvWmTTbLqWbNBO1ymjwZMjKsiwngvPPgwgs9STjnikXcEoWIlAZeAE4FGgHni0ijHJtNBTJUtRnwPvBYvOLZE3feaV1O/ftDxYphRxNl0ya47TZbinTlSl8nwjkXF/FsUbQE5qvqQlXdCgwCOkdvoKqjVfXP4OYEoHoc4ymS8eNtEaIrr7TJQwnju+/sPIjHHrMifrNmQadOYUflnEtB8RyjqAYsibqdCbTKZ/vLgU9ze0BEegG9AGrWrFlc8RXozz+ty6lWLfs+TiibNtkSpV9+mWAZzDmXahJiMFtELgIygLa5Pa6qrwCvgBUFLKm47rwT5s2DUaMSpMtpxAgr4nfLLXYC3ezZsPfeYUflnEtx8ex6WgpEz8usHty3CxFpD9wBnKGqW+IYT6F88w08/TRcfTWceGLIwaxaBRddBKedBm+/HSni50nCOVcC4pkovgfqi0gdESkDdAOGR28gIkcCL2NJ4rc4xlIof/5p3f61a9vypqFRhUGD4PDDYfBgK1c7aZIX8XPOlai4dT2papaIXAuMBEoDA1R1pojcD0xW1eHA48C+wHtiUzkXq+oZ8YopVv/6F8yfbyWS9t03xEAWL7Zy4EccYVOumjYNMRjnXLryhYty+OYbaNvWupyefz5uu8mbKnz1VeSsvgkT4G9/s5PpnHOuiHzhomKycaPNcqpdGx55JIQAFiywGUwnnRQp4te6tScJ51yoEmLWU6L417/su7rEu5yyi/jdeacNUL/8shfxc84lDE8UgbFj4dln4dpr4YQTSnjnp58On35qJ8y99JKVp3XOuQThYxRYl1OzZnZ92jSoUKFYXz53W7fauhClStmMpu3boVs3r8/knIsLH6PYQ7ffDgsXwmuvlVCSmDQJjjoKXnzRbnftatVePUk45xJQ2ieKMWPguefguuugTZs47+zPP6F3bzj6aFizBurWjfMOnXNuz6X1GMWGDXZiXd268NBDcd7ZuHF2TsTChbaoxaOPQqVKcd6pc87tubROFH36wM8/20zUuHc5ZS8sNHp0CKPlzjlXdGmbKEaPhhdegOuvj+NM1I8+ssJ9t95qBaNmzbIBbOecSyJpOUaR3eVUr16cupxWrrRlSM84AwYOjBTx8yThnEtCaZkobrsNfvnFZjnts08xvrAqvPOOFfF7/324/36YONGL+Dnnklra/cQdNcpmpd54Ixx3XDG/+OLFVgPkyCOtiF/jxsW8A+ecK3lp1aJYvx4uvxzq14cHHyymF92xA0aOtOu1allVwfHjPUk451JGWiWKW28t5i6nefNspbkOHawGCEDLll7EzzmXUtImUXz1FfTta11Oxx67hy+WlQWPP251P374wbqZvIifcy5FpcUYRXaXU4MGxdTl1KmTdTd17mwDHlWrFsOLOpd6tm3bRmZmJps3bw47lLRRrlw5qlevzt7FuFRyWiSKW26xceZx46B8+SK+yJYtVgK8VCno2dPm1557rtdnci4fmZmZVKxYkdq1ayP+txJ3qsrq1avJzMykTp06xfa6Kd/19OWXtrzDTTfBMccU8UUmTIAWLewMPYBzzrFCfv7Bdy5fmzdvpnLlyp4kSoiIULly5WJvwaV0oli3zrqcDjsMHnigCC+wcaMNahxzjPVf1a9f7DE6l+o8SZSseBzvlO56uuUWyMwsYpfTN99YEb9Fi2wB7Ycfhv32i0uczjmXyFK2RfHFF/DKK5Gq3oWWlWVjEl9/bV1OniScS1offvghIsJPP/20874xY8bQqVOnXbbr3r0777//PmAD8X369KF+/fq0aNGCo48+mk8//XSPY3n44YepV68ehx12GCOzz8HK4fjjj6d58+Y0b96cqlWrcuaZZwIwbNgwmjVrRvPmzcnIyGDcuHF7HE8sUrJFkd3l1LChVdGI2YcfWhG/22+3In4zZ3p9JudSwMCBAznuuOMYOHAg9913X0zPueuuu1i+fDkzZsygbNmyrFixgq+//nqP4pg1axaDBg1i5syZLFu2jPbt2zN37lxK5zj36ptvvtl5vUuXLnTu3BmAdu3accYZZyAiTJs2ja5du+6S/OIlJb8Fe/eGpUvh22+hXLkYnrBiBfzzn/DeezZo3bu31WfyJOFcsbnhBjvtqDg1bw5PP53/Nhs2bGDcuHGMHj2a008/PaZE8eeff/Lqq6+yaNEiypYtC8BBBx1E165d9yjeYcOG0a1bN8qWLUudOnWoV68ekyZN4ug8uj3WrVvHqFGjeO211wDYd999dz62cePGEhv/Sbmup5EjoV8/uPlmaNWqgI1V4b//hUaNYNgw+Pe/bYaTF/FzLmUMGzaMDh060KBBAypXrsyUKVMKfM78+fOpWbMm+8XQ5XzjjTfu7CaKvjzyyCO7bbt06VJq1Kix83b16tVZunRpnq/94Ycf0q5du13iGDp0KA0bNuS0005jwIABBcZXHFLqJ/Mff9gpDocfDjG1LhcvtidkZNjZ1Q0bxj1G59JVQb/842XgwIFcf/31AHTr1o2BAwdy1FFH5flrvLC/0p966qk9jjEvAwcOpGfPnrvcd9ZZZ3HWWWcxduxY7rrrLr788su47T9bSiWK3r1h2TL47rt8upyyi/ideqoV8Rs/3qq9en0m51LO77//zqhRo5g+fToiwvbt2xERHn/8cSpXrsyaNWt2275KlSrUq1ePxYsXs27dugJbFTfeeCOjR4/e7f5u3brRp0+fXe6rVq0aS5Ys2Xk7MzOTatWq5fq6q1atYtKkSQwdOjTXx9u0acPChQtZtWoVVapUyTfGPaaqSXU56qijNDeffqoKqrfdluvDZs4c1eOPtw3HjMlnQ+dccZg1a1ao+3/55Ze1V69eu9zXpk0b/frrr3Xz5s1au3btnTH+/PPPWrNmTV27dq2qqt5yyy3avXt33bJli6qq/vbbbzp48OA9imfGjBnarFkz3bx5sy5cuFDr1KmjWVlZuW770ksv6SWXXLLLffPmzdMdO3aoquqUKVO0atWqO29Hy+24A5O1iN+7KTFG8ccfcMUVNtRw7725bJCVBY8+akX8pk+38rFt2pR0mM65EjZw4EDOOuusXe7r0qULAwcOpGzZsrz11lv06NGD5s2bc84559CvXz8qVaoEwIMPPsiBBx5Io0aNaNKkCZ06dYppzCI/jRs3pmvXrjRq1IgOHTrwwgsv7Jzx1LFjR5YtW7Zz20GDBnH++efv8vwhQ4bQpEkTmjdvzjXXXMO7775bIgPaYokmeWRkZOjkyZN3ue/yy+H1163LqWXLXJ50yinw+edw9tl2TsTBB5dIrM6lu9mzZ3P44YeHHUbaye24i8gUVc0oyuslfYvi009hwABb3nSXJLF5M2zfbtd79bKlSYcM8SThnHOFlNSJYu1am7TUuDHcc0/UA+PH2wTr7CJ+XbrYxTnnXKEldaK48UY7V+7116FsWWDDBrjuOltEaPNmmyfrnAtVsnVvJ7t4HO+kTRSffGIJ4rbb7DQIvv4amjSB55+Ha6+FGTPgpJPCDtO5tFauXDlWr17tyaKEaLAeRbmYSlLELinPo1izxoYdmjSBu++OemCffazq6x6vdeqcKw7Vq1cnMzOTlStXhh1K2she4a44JeWspyZNJvPWWzDn4Q+ou+0n+Ne/7MHt2/3EOeecy0XCznoSkQ4iMkdE5otIn1weLysi7waPTxSR2gW95h9/wGdv/MoP9c+h7q1dYOhQ2LrVHvQk4ZxzxS5uiUJESgMvAKcCjYDzRaRRjs0uB9aoaj3gKeDRgl533aLVzC11OI0XfWyLCX37rRfxc865OIpni6IlMF9VF6rqVmAQ0DnHNp2BN4Lr7wPtpIDTDKtv/wVp1gT58Ufo08cWF3LOORc38RzMrgYsibqdCeQs/L1zG1XNEpE/gMrAquiNRKQX0Cu4uWW/H8bN8EqvAFQhx7FKY34sIvxYRPixiDisqE9MillPqvoK8AqAiEwu6oBMqvFjEeHHIsKPRYQfiwgRmVzwVrmLZ9fTUqBG1O3qwX25biMiewGVgNVxjMk551whxTNRfA/UF5E6IlIG6AYMz7HNcODS4Po5wChNtvm6zjmX4uLW9RSMOVwLjARKAwNUdaaI3I/VRR8O9Af+KyLzgd+xZFKQV+IVcxLyYxHhxyLCj0WEH4uIIh+LpDvhzjnnXMlK2lpPzjnnSoYnCuecc/lK2EQRj/IfySqGY3GTiMwSkWki8pWI1AojzpJQ0LGI2q6LiKiIpOzUyFiOhYh0DT4bM0XknZKOsaTE8DdSU0RGi8jU4O+kYxhxxpuIDBCR30RkRh6Pi4g8GxynaSLSIqYXLupi2/G8YIPfC4BDgTLAj0CjHNtcDfQNrncD3g077hCPxYnAPsH1q9L5WATbVQTGAhOAjLDjDvFzUR+YChwQ3P5r2HGHeCxeAa4KrjcCfg477jgdizZAC2BGHo93BD4FBGgNTIzldRO1RRGX8h9JqsBjoaqjVfXP4OYE7JyVVBTL5wLgAaxu2OaSDK6ExXIsrgBeUNU1AKr6WwnHWFJiORYK7BdcrwQsK8H4SoyqjsVmkOalM/CmmgnA/iJySEGvm6iJIrfyH9Xy2kZVs4Ds8h+pJpZjEe1y7BdDKirwWARN6Rqq+klJBhaCWD4XDYAGIjJeRCaISIcSi65kxXIs7gUuEpFMYATwz5IJLeEU9vsESJISHi42InIRkAG0DTuWMIhIKeBJoHvIoSSKvbDupxOwVuZYEWmqqmtDjSoc5wOvq+oTInI0dv5WE1XdEXZgySBRWxRe/iMilmOBiLQH7gDOUNUtJRRbSSvoWFQEmgBjRORnrA92eIoOaMfyucgEhqvqNlVdBMzFEkeqieVYXA4MBlDV74ByWMHAdBPT90lOiZoovPxHRIHHQkSOBF7GkkSq9kNDAcdCVf9Q1SqqWltVa2PjNWeoapGLoSWwWP5GPsRaE4hIFawramFJBllCYjkWi4F2ACJyOJYo0nF91uHAJcHsp9bAH6q6vKAnJWTXk8av/EfSifFYPA7sC7wXjOcvVtUzQgs6TmI8FmkhxmMxEjhZRGYB24FbVDXlWt0xHovewKsiciM2sN09FX9YishA7MdBlWA85h5gbwBV7YuNz3QE5gN/Aj1iet0UPFbOOeeKUaJ2PTnnnEsQniicc87lyxOFc865fHmicM45ly9PFM455/LlicIlHBHZLiI/RF1q57Nt7bwqZRZyn2OC6qM/BiUvDivCa1wpIpcE17uLSNWox/qJSKNijvN7EWkew3NuEJF99nTfLn15onCJaJOqNo+6/FxC+71QVY/Aik0+Xtgnq2pfVX0zuNkdqBr1WE9VnVUsUUbifJHY4rwB8EThiswThUsKQcvhGxH5X3A5JpdtGovIpKAVMk1E6gf3XxR1/8siUrqA3Y0F6gXPbResYTA9qPVfNrj/EYmsAfKf4L57ReRmETkHq7n1drDP8kFLICNodez8cg9aHs8XMc7viCroJiIvichksbUn7gvuuw5LWKNFZHRw38ki8l1wHN8TkX0L2I9Lc54oXCIqH9XtNDS47zfgJFVtAZwHPJvL864EnlHV5tgXdWZQruE84Njg/u3AhQXs/3RguoiUA14HzlPVplglg6tEpDJwFtBYVZsBD0Y/WVXfByZjv/ybq+qmqIeHBM/Ndh4wqIhxdsDKdGS7Q1UzgGZAWxFppqrPYiW1T1TVE4NSHncC7YNjORm4qYD9uDSXkCU8XNrbFHxZRtsbeD7ok9+O1S3K6TvgDhGpDnygqvNEpB1wFPB9UN6kPJZ0cvO2iGwCfsbKUB8GLFLVucHjbwDXAM9ja130F5GPgY9jfWOqulJEFgZ1duYBDYHxwesWJs4yWNmW6OPUVUR6YX/Xh2AL9EzL8dzWwf3jg/2UwY6bc3nyROGSxY3ACuAIrCW826JEqvqOiEwETgNGiMg/sJW83lDV22PYx4XRBQRF5C+5bRTUFmqJFZk7B7gW+Hsh3ssgoCvwEzBUVVXsWzvmOIEp2PjEc8DZIlIHuBn4m6quEZHXscJ3OQnwhaqeX4h4XZrzrieXLCoBy4P1Ay7Gir/tQkQOBRYG3S3DsC6Yr4BzROSvwTZ/kdjXFJ8D1BaResHti4Gvgz79Sqo6AktgR+Ty3PVY2fPcDMVWGjsfSxoUNs6goN1dQGsRaYit3rYR+ENEDgJOzSOWCcCx2e9JRCqISG6tM+d28kThksWLwKUi8iPWXbMxl226AjNE5AdsXYo3g5lGdwKfi8g04AusW6ZAqroZq675nohMB3YAfbEv3Y+D1xtH7n38rwN9swezc7zuGmA2UEtVJwX3FTrOYOzjCawq7I/Y+tg/Ae9g3VnZXgE+E5HRqroSm5E1MNjPd9jxdC5PXj3WOedcvrxF4ZxzLl+eKJxzzuXLE4Vzzrl8eaJwzjmXL08Uzjnn8uWJwjnnXL48UTjnnMvX/wFntzpz0E/VgwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Using predict_proba() function on test data \n",
        "DT_model_pred_prob= DT_model_Holdout.predict_proba(X_test)\n",
        "preds = DT_model_pred_prob[:,1]\n",
        "\n",
        "\n",
        "# getting fpr, tpr values to plot ROC curve at different thresholds\n",
        "fpr, tpr, threshold = metrics.roc_curve(Y_test, preds)\n",
        "roc_auc = metrics.auc(fpr, tpr)\n",
        "\n",
        "# plotting ROC curve with AUC value\n",
        "plt.title('Receiver Operating Characteristic')\n",
        "plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)\n",
        "plt.legend(loc = 'lower right')\n",
        "plt.plot([0, 1], [0, 1],'r--')\n",
        "plt.xlim([0, 1])\n",
        "plt.ylim([0, 1])\n",
        "plt.ylabel('True Positive Rate')\n",
        "plt.xlabel('False Positive Rate')\n",
        "plt.show()\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hiBJvVFfyIkR"
      },
      "source": [
        "A good model has a closer  curve that follows the left-hand border and then the top border of the ROC space. Since the ROC curve achieved by decision tree models moves slightly away from left hand border shows average performance by the model. Also, AUC which stands for area under curve provides quantitative measure of the performance of the model. The value close to 100% indicates very good performance by the model. In the output we received by the model, AUC=76% does not indicate a very good performance by the model but, can be accpeted as avegrage performing model(subjective to data set domain and user's choice)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LYTgpfhUyIkR"
      },
      "source": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.5"
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}