[PMPG_CH03].ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "[PMPG_CH03].ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xeWpEfsG7eKS"
      },
      "source": [
        "***CH03. 평가***"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0am4ZeKJ9c45"
      },
      "source": [
        "- 머신러닝은 데이터 가공/변환, 모델 학습/예측, 그리고 평가(Evaluation)의 프로세스로 구성됩니다.\r\n",
        "- 머신러닝 모델은 여러 가지 방법으로 예측 성능을 평가할 수 있습니다.\r\n",
        "- 성능 평가 지표(Evaluation Metric)는 일반적으로 모델이 분류냐 회귀냐에 따라 여러 종류로 나뉩니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GKXEwu_1-ciz"
      },
      "source": [
        "분류의 성능 평가 지표\r\n",
        "- 정확도(Accuracy)\r\n",
        "- 오차행렬(Confusion Matrix)\r\n",
        "- 정밀도(Precision)\r\n",
        "- 재현율(Recall)\r\n",
        "- F1스코어\r\n",
        "- ROC AUC"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4MPubIr3-mwb"
      },
      "source": [
        "분류는 결정 클래스 값 종류의 유형에 따라 긍정/부정과 같은 2개의 결과값만을 가지는 **이진분류**와 여러 개의 결정 클래스 값을 가지는 **멀티 분류**로 나뉠 수 있습니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5ZB3G1Hm_lcT"
      },
      "source": [
        "#**01.정확도(Accuracy)**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "agVUnkel_v2x"
      },
      "source": [
        "**정확도(Accuracy) = 예측 결과가 동일한 데이터 건수/전체 예측 데이터 건수**\r\n",
        "\r\n",
        "- 정확도는 직관적으로 모델 예측 성능을 나타내는 평가 지표입니다. 하지만 이진 분류의 경우 데이터의 구성에 따라 ML 모델의 성능을 왜곡할 수 있기 때문에 정확도 수치 하나만 가지고 성능을 평가하지 않습니다.\r\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jhBrBeay7Pzy"
      },
      "source": [
        "# 왜곡하는 예시\r\n",
        "# 단순히 성별(Sex) 피처가 1이면 0, 그렇지 않고 0이면 1로 예측하는 매우 단순한 Classifier\r\n",
        "# 사이킷런의 BaseEstimater를 상속받으면 Customized 형태의 Estimator를 개발자가 생성할 수 있습니다.\r\n",
        "import numpy as np\r\n",
        "from sklearn.base import BaseEstimator\r\n",
        "\r\n",
        "class MyDummyClassifier(BaseEstimator):\r\n",
        "  # fit() 메서드는 아무것도 학습하지않음\r\n",
        "  def fit(self,X,y=None):\r\n",
        "    pass\r\n",
        "  # predict() 메서드는 단순히 sex 피처가 1이면 0, 그렇지 않으면 1로 예측함\r\n",
        "  def predict(self,X):\r\n",
        "    pred = np.zeros( (X.shape[0],1) )\r\n",
        "    for i in range(X.shape[0]):\r\n",
        "      if X['Sex'].iloc[i] == 1:\r\n",
        "        pred[i] = 0\r\n",
        "      else:\r\n",
        "        pred[i] = 1\r\n",
        "    return pred"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_btP5dPJCiH7",
        "outputId": "9d158149-2dd4-459f-8136-d4ef941f4271"
      },
      "source": [
        "import pandas as pd\r\n",
        "from sklearn.model_selection import train_test_split\r\n",
        "from sklearn.metrics import accuracy_score\r\n",
        "\r\n",
        "# 원본 데이터를 재로딩, 데이터 가공, 학습/테스트 데이터 분할\r\n",
        "titanic_df = pd.read_csv('train.csv')\r\n",
        "# 전처리 부분은 생략함\r\n",
        "\r\n",
        "X_titanic_df = titanic_df.drop('Survived',axis = 1)\r\n",
        "y_titanic_df = titanic_df['Survived']\r\n",
        "X_train, X_test, y_train, y_test = train_test_split(X_titanic_df, y_titanic_df, test_size = 0.2, random_state = 0)\r\n",
        "\r\n",
        "# 위에서 생성한 Dummy Classifier를 이용해 학습/예측/평가 수행\r\n",
        "myclf = MyDummyClassifier()\r\n",
        "myclf.fit(X_train,y_train)\r\n",
        "mypredictions = myclf.predict(X_test)\r\n",
        "print('Dummy Classifier의 정확도는 :{0:4f}'.format(accuracy_score(y_test,mypredictions)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Dummy Classifier의 정확도는 :0.385475\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_r1tzc2bEm26"
      },
      "source": [
        "- 정확도를 평가 지표로 사용할 때는 매우 신중해야함.\r\n",
        "- 특히 정확도는 불균형한(imbalanced) 레이블 값 분포에서 ML 모델의 성능을 판단할 경우, 적합한 평가 지표가 아닙니다.\r\n",
        "- 예를 들어 100개의 데이터중에서 90개의 데이터 레이블이 0, 90개, 단 10개의 데이터 레이블이 1이라고 한다면 무조건 0으로 예측 결과를 반환하는 ML 모델의 경우라도 정확도가 90%가 됩니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G7tsOA5eFYjp"
      },
      "source": [
        "사이킷런은 load_digits() API를 통해 MNIST 데이터 세트를 제공\r\n",
        "- MNIST 데이터 세트는 레이블 값이 0~9 까지 있는 멀티 레이블 분류를 위한 것\r\n",
        "- 이것을 데이터 레이블이 7인 것만 True, 나머지값은 False인 이진분류로 바꿔보겠습니다. 즉 전체 데이터의 10%만 True, 나머지는 90% 인 불균형한 데이터 세트로 변형\r\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OiKsP1E6D7bp",
        "outputId": "da872a35-d8b3-4245-df3c-a9ec84537406"
      },
      "source": [
        "# MNIST 예시\r\n",
        "from sklearn.datasets import load_digits\r\n",
        "from sklearn.metrics import accuracy_score\r\n",
        "from sklearn.model_selection import train_test_split\r\n",
        "from sklearn.base import BaseEstimator\r\n",
        "import numpy as np\r\n",
        "import pandas as pd\r\n",
        "\r\n",
        "class MyFakeClassifier(BaseEstimator):\r\n",
        "  def fit(self,X,y):\r\n",
        "    pass\r\n",
        "  # 입력값으로 들어오는 X 데이터 세트의 크기만큼 모두 0 값으로 만들어서 반환\r\n",
        "  def predict(self,X):\r\n",
        "    return np.zeros( (len(X),1), dtype=bool )\r\n",
        "\r\n",
        "# 사이킷런의 내장 데이터 세트인 load_digits()를 이용해 MNIST 데이터 로딩\r\n",
        "digits = load_digits()\r\n",
        "\r\n",
        "# digits 번호가 7번이면 True, 이를 astype(int)로 1로 변환, 7번이 아니면 False이고 0으로 변환\r\n",
        "y = (digits.target == 7).astype(int)\r\n",
        "X_train,X_test,y_train,y_test = train_test_split(digits.data, y, random_state = 11)\r\n",
        "\r\n",
        "# 불균형한 레이블 분포도 확인\r\n",
        "print('레이블 테스트 세트 크기 :',y_test.shape)\r\n",
        "print('테스트 세트 레이블 0과 1의 분포도')\r\n",
        "print(pd.Series(y_test).value_counts())\r\n",
        "\r\n",
        "# Dummy Classifier로 학습/예측/정확도 평가\r\n",
        "fakeclf = MyFakeClassifier()\r\n",
        "fakeclf.fit(X_train,y_train)\r\n",
        "fakepred = fakeclf.predict(X_test)\r\n",
        "print('모든 예측을 0으로 하여도 정확도는 :{0:4f}'.format(accuracy_score(y_test,fakepred)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "레이블 테스트 세트 크기 : (450,)\n",
            "테스트 세트 레이블 0과 1의 분포도\n",
            "0    405\n",
            "1     45\n",
            "dtype: int64\n",
            "모든 예측을 0으로 하여도 정확도는 :0.900000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mStJgZNQIRtC"
      },
      "source": [
        "- 단순히 predict()의 결과를 np.zeros()로 모두 0값으로 반환함에도 불구하고 450개의 테스트 데이터 세트에 수행한 예측 정확도는 90%입니다. 단지 모든 것을 0으로 예측해도 정확도가 90%로 유수의 ML 알고리즘과 어깨를 겨룰 수 있다는 것은 말도 안되는 일이다.\r\n",
        "\r\n",
        "- 이처럼 정확도 평가 지표는 불균형한 레이블 데이터 세트에서 성능 수치로 사용되서는 안된다.\r\n",
        "\r\n",
        "- 정확도가 가지는 분류 평가 지표로서 이러한 한계점을 극복하기 위해 여러 가지 분류 지표와 함께 적용해햐 함\r\n",
        "- 먼저 True/false, Positive/Negative의 4분면으로 구성되는 오차 행렬(Confusion Matrix)에 대해 설명"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lXkrf70SJQjZ"
      },
      "source": [
        "#**02. 오차 행렬(Confusion Matrix)**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tk7xEpBJL-g8"
      },
      "source": [
        "- 이진 분류에서 성능 지표로 잘 활용되는 오차행렬(Confusion Matrix, 혼동행렬)은 학습된 분류 모델이 예측을 수행하면서 얼마나 햇갈리고(confused) 있는지도 함께 보여주는 지표입니다. 즉 이진 분류의 예측 오류가 얼마인지와 더불어 어떠한 유형의 예측 오류가 발생하고 있는지를 함께 나타내는 지표입니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r_iakQhuMj4h"
      },
      "source": [
        "- 오차 행렬은 4분면 행렬에서 실제 레이블 클래스 값과 예측 레이블 클래스 값이 어떠한 유형을 가지고 매핑되는지를 나타낸다. 4분면의 왼쪽, 오른쪽을 예측된 클래스 값 기준으로 Negative, Positive로 분류하고, 4분면의 위 아래를 실제 클래스 값 기준으로 Negative positive로 분류 하면 예측 클래스와 실제 클래스의 값 유형에 따라 결정되는 TN, FP FN TP 형태로 오차 행렬의 4분면을 채울 수 있다.\r\n",
        "\r\n",
        "- TN FP FN TP 값을 다양하게 결합해 분류 모델 예측 성능의 오류가 어떠한 모습으로 발생하는지 알 수 있습니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AnQJvfOvN0as"
      },
      "source": [
        "TN FP FN TP 기호가 의미하는 것은 앞 문자 True/False는 예측값과 실제값이 같은가/틀린가를 의미하고, 뒤 문자 Negative/Positive는 예측 결과값이 부정(0)/긍정(1)을 의미함.\r\n",
        "\r\n",
        ">> TN는 예측값을 Negative 값 0으로 예측했고 실제값 역시 Negative 값 0\r\n",
        "\r\n",
        ">> FP는 예측값을 Positive 값 1로 예측했고 실제값은 Negative 값 0\r\n",
        "\r\n",
        ">> FN는 예측값을 Negative 값 0으로 예측했고 실제값은 Positive 값 1\r\n",
        "\r\n",
        ">> TP는 예측값을 Positive 값 1로 예측했고 예측값은 Negative 값 1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3sOFH6TDRwKT"
      },
      "source": [
        "- 사이킷런은 오차 행렬을 구하기 위해 confusion matrix() API를 제공합니다.\r\n",
        "- 앞에서 다룬 MyFakeClassifier의 예측 성능 지표를 오차 행렬로 표현"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MEb8MesCG5dj",
        "outputId": "a06ee114-b50b-492b-8b03-aa66b57ba289"
      },
      "source": [
        "from sklearn.metrics import confusion_matrix\r\n",
        "\r\n",
        "confusion_matrix(y_test,fakepred)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[405,   0],\n",
              "       [ 45,   0]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p3LAiZfYUd_D"
      },
      "source": [
        "- TP,FN,FP,TN 값은 Classifier 성능의 여러 면모를 판단할 수 있는 기반 정보를 제공합니다. 이 값을 조합해 Classifier의 성능을 측정할 수 있는 주요 지표인 정확도(Accuracy), 정밀도(precision), 재현율(Recall)값을 알 수 있습니다.\r\n",
        "\r\n",
        "- 정확도는 오차행렬에서 True에 해당하는 값인 TN과 TP에 좌우됩니다.\r\n",
        "- **정확도 = 예측결과와 실제 값이 동일한 건수/ 전체 데이터 수 = (TN+TP)/(TN+FP+FN+TP)**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t6i4HOzbWp6p"
      },
      "source": [
        "- 정확도 보다 더 선호되는 평가 지표인 정밀도(precision)와 재현율(Recall)에 대해 알아보자"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yJACR3T-Wy9p"
      },
      "source": [
        "#**03. 정밀도와 재현율**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pDiH-dJDZrLJ"
      },
      "source": [
        "- 정밀도와 재현율은 Positive 데이터 세트의 예측 성능에 좀 더 초점을 맞춘 평가 지표이다.\r\n",
        "- 정밀도 = TP/(FP+TP)\r\n",
        "- 재현율 = TP/(FN+TP)\r\n",
        "\r\n",
        ">>- 정밀도는 예측을 Positive로 한 대상 중에 예측과 실제 값이 Positive로 일치한 데이터의 비율을 뜻한다.\r\n",
        "- Positive 예측 성능을 더욱 정밀하게 측정하기 위한 평가 지표로 양성 예측도라고도 불린다.\r\n",
        "\r\n",
        ">>- 재현율은 실제 값이 Positive인 대상 중에 예측과 실제 값이 Positive로 일치한 데이터의 비율을 뜻함.\r\n",
        "- 민감도, TPR(True Positive Rate)이라고도 불린다"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "35QE74ITeKM5"
      },
      "source": [
        "- **재현율**이 상대적으로 더 중요한 지표인 경우는 실제 Positive 양성인 데이터 예측을 Negative로 잘못 판단하게 되면 업무상 큰 영향이 발생하는 경우\r\n",
        "\r\n",
        "- **정밀도**가 상대적으로 더 중요한 지표인 경우는 실제 Negative 음성인 데이터 예측을 Positive 양성으로 잘못 판단하게 되면 업무상 큰 영향이 발생하는 경우\r\n",
        "\r\n",
        "- ** 정밀도도 중요 평가 지표지만, 업무적인 특성을 고려하면 재현율이 상대적으로 더 중요한 지표이다.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B5aCtgLKfbTX"
      },
      "source": [
        "- 사이킷런은 정밀도 계산을 위해 precision_score()를, 재현율 계산을 위해 recall_score()를 API로 제공."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lLuqpWUAgFNo"
      },
      "source": [
        "- confusion matrix,accuracy,precision,recall등의 평가를 한꺼번에 호출하는 get_clf_eval() 함수를 만들도록 하겠습니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JCEKI6LrSO_H"
      },
      "source": [
        "from sklearn.metrics import confusion_matrix, accuracy_score, precision_score, recall_score\r\n",
        "\r\n",
        "def get_clf_eval(y_test,pred):\r\n",
        "  confusion = confusion_matrix(y_test,pred)\r\n",
        "  accuracy = accuracy_score(y_test,pred)\r\n",
        "  precision = precision_score(y_test,pred)\r\n",
        "  recall = recall_score(y_test,pred)\r\n",
        "  print('오차행렬')\r\n",
        "  print(confusion)\r\n",
        "  print('정확도:{0:4f}, 정밀도:{1:4f}, 재현율:{2:4f}'.format(accuracy,precision,recall))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qJKn_QHVj_E9"
      },
      "source": [
        "def fillna(df):\r\n",
        "  df['Age'].fillna(df['Age'].mean(),inplace=True)\r\n",
        "  df['Cabin'].fillna('N',inplace=True)\r\n",
        "  df['Embarked'].fillna('N',inplace=True)\r\n",
        "  df['Fare'].fillna(0,inplace=True)\r\n",
        "  return df\r\n",
        "\r\n",
        "def drop_features(df):\r\n",
        "  df.drop(['PassengerId','Name','Ticket'],axis = 1, inplace = True)\r\n",
        "  return df\r\n",
        "\r\n",
        "def format_features(df):\r\n",
        "  df['Cabin'] = df['Cabin'].str[:1]\r\n",
        "  features = ['Cabin','Sex','Embarked']\r\n",
        "  for feature in features:\r\n",
        "    le = LabelEncoder()\r\n",
        "    le = le.fit(df[feature])\r\n",
        "    df[feature] = le.transform(df[feature])\r\n",
        "  return df\r\n",
        "\r\n",
        "def transform_features(df):\r\n",
        "  df = fillna(df)\r\n",
        "  df = drop_features(df)\r\n",
        "  df = format_features(df)\r\n",
        "  return df\r\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cPterSJ0hGOA",
        "outputId": "d368f8c1-9229-457e-d0a2-fc1062ac56f2"
      },
      "source": [
        "import pandas as pd\r\n",
        "from sklearn.model_selection import train_test_split\r\n",
        "from sklearn.linear_model import LogisticRegression\r\n",
        "from sklearn.preprocessing import LabelEncoder\r\n",
        "\r\n",
        "titanic_df = pd.read_csv('train.csv')\r\n",
        "X_titanic_df = titanic_df.drop('Survived',axis=1)\r\n",
        "X_titanic_df = transform_features(X_titanic_df)\r\n",
        "y_titanic_df = titanic_df['Survived']\r\n",
        "\r\n",
        "X_train,X_test,y_train,y_test = train_test_split(X_titanic_df,y_titanic_df,test_size = 0.20, random_state = 11)\r\n",
        "\r\n",
        "lr_clf = LogisticRegression()\r\n",
        "lr_clf.fit(X_train,y_train)\r\n",
        "pred = lr_clf.predict(X_test)\r\n",
        "get_clf_eval(y_test,pred)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "오차행렬\n",
            "[[104  14]\n",
            " [ 13  48]]\n",
            "정확도:0.849162, 정밀도:0.774194, 재현율:0.786885\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
            "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
            "\n",
            "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
            "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
            "Please also refer to the documentation for alternative solver options:\n",
            "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
            "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FyUdDVg6mXky"
      },
      "source": [
        "- 정확도에 비해 재현율, 정밀도는 낮게 나왔다. 재현율 정밀도를 좀 더 강화할 방법은 무엇일까?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QQnOIB5gmfrD"
      },
      "source": [
        "**정밀도/재현율 트레이드 오프**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7OSov9K5gdz5"
      },
      "source": [
        "- 분류하려는 업무의 특성상 정밀도 또는 재현율이 특별히 강조돼야 할 경우 분류의 결정 임곗값(Threshold)을 조정해 정밀도 또는 재현율의 수치를 높일 수 있습니다. 하지만 정밀도와 재현율은 **상호 보완적인 평가 지표**이기 때문에 어느 한쪽을 강제로 높이면 다른 하나의 수치는 떨어지기 쉽다.\r\n",
        "이를 **정밀도/재현율 트레이드 오프(Trade-off)**라고 부른다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vShfMviIhPCv"
      },
      "source": [
        "- 일반적으로 이진분류에서는 임곗값을 0.5, 즉 50%로 정하고 이 기준 값보다 확률이 크면 Positive, 작으면 Negative로 결정"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hef9Aad6juP4"
      },
      "source": [
        "- 사이킷런은 개별 데이터별로 예측 확률을 반환하는 메서드인 predict_proba()를 제공.\r\n",
        "- predict_proba() 메서드는 학습이 완료된 사이킷런 Classifier 객체에서 호출이 가능하며 테스트 피처 데이터 세트를 파라미터로 입력해주면 테스트 피처 레코드의 개별 클래스 예측 확률을 반환\r\n",
        "- predict() 메서드와 유사하지만 단지 반환 결과가 예측 결과 클래스 값이 아닌 예측 확률 결과라는 점"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w49k9Zidw60X"
      },
      "source": [
        "각 열은 개별 클래스의 예측 확률이고, 이진 분류에서 첫 번째 컬럼은 0 Negative의 확률, 두번째는 1 Positive의 확률"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mLCC72H0h149",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cd433ffe-ac09-461c-a471-1b56390de313"
      },
      "source": [
        "pred_proba = lr_clf.predict_proba(X_test)\r\n",
        "pred = lr_clf.predict(X_test)\r\n",
        "\r\n",
        "print('pred_proba()의 결과 shape : {0}', format(pred_proba.shape))\r\n",
        "print('pred_proba array에서 앞 3개만 추출 :\\n',pred_proba[:3])\r\n",
        "\r\n",
        "# 예측 확률 array와 예측 결과값 array를 병합(concatenate)하여 예측 확률과 결과값을 확인\r\n",
        "pred_proba_result = np.concatenate([pred_proba,pred.reshape(-1,1)],axis = 1)\r\n",
        "print('두 개의 class 중에서 더 큰 확률을 클래스 값으로 예측\\n',pred_proba_result[:3])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "pred_proba()의 결과 shape : {0} (179, 2)\n",
            "pred_proba array에서 앞 3개만 추출 :\n",
            " [[0.46191519 0.53808481]\n",
            " [0.878675   0.121325  ]\n",
            " [0.87716185 0.12283815]]\n",
            "두 개의 class 중에서 더 큰 확률을 클래스 값으로 예측\n",
            " [[0.46191519 0.53808481 1.        ]\n",
            " [0.878675   0.121325   0.        ]\n",
            " [0.87716185 0.12283815 0.        ]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1eGWY3rjyKmI",
        "outputId": "52c1d02d-847f-45a7-f7e4-800dfdcba849"
      },
      "source": [
        "from sklearn.preprocessing import Binarizer\r\n",
        "\r\n",
        "x = [[1,-1,2],[2,0,0],[0,1.1,1.2]]\r\n",
        "\r\n",
        "# x의 개별 원소들이 thresehold 값 보다 같거나 작으면 0, 크면 1로 반환\r\n",
        "binarizer = Binarizer(threshold = 1.1)\r\n",
        "print(binarizer.fit_transform(x))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[0. 0. 1.]\n",
            " [1. 0. 0.]\n",
            " [0. 0. 1.]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AwuzloKP0Y3X"
      },
      "source": [
        " - 입력된 X 데이터 세트에서 Binarizer의 threshold값이 1.1보다 같거나 작으면 0, 크면 1로 변환됨을 알 수 있다.\r\n",
        " - 앞 예제의 predict_proba() 메서드는 분류 결정 임곗값이 0.5로 지정한 Binarizer 클래스를 적용해 최종 예측값을 구하는 방식"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8VzTjCg40EVw",
        "outputId": "afb35de1-4206-43b5-d9ae-a6b9fcdb009b"
      },
      "source": [
        "from sklearn.preprocessing import Binarizer\r\n",
        "\r\n",
        "custom_threshold = 0.5\r\n",
        "\r\n",
        "# predict_proba() 반환값이 두 번째 컬럼, 즉 Positive 클래스 컬럼 하나만 추출해 Binarizer를 적용\r\n",
        "pred_proba_1 = pred_proba[:,1].reshape(-1,1)\r\n",
        "\r\n",
        "binarizer = Binarizer(threshold=custom_threshold).fit(pred_proba_1)\r\n",
        "custom_predict = binarizer.transform(pred_proba_1)\r\n",
        "\r\n",
        "get_clf_eval(y_test, custom_predict)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "오차행렬\n",
            "[[104  14]\n",
            " [ 13  48]]\n",
            "정확도:0.849162, 정밀도:0.774194, 재현율:0.786885\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-UyZyyWi61UU"
      },
      "source": [
        "- threshold 결정값을 0.5 -> 0.4 낮춤"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jFKk-lFD5TcJ",
        "outputId": "5269439b-c32b-4e4d-d105-041fcd71e414"
      },
      "source": [
        "from sklearn.preprocessing import Binarizer\r\n",
        "\r\n",
        "custom_threshold = 0.4\r\n",
        "\r\n",
        "# predict_proba() 반환값이 두 번째 컬럼, 즉 Positive 클래스 컬럼 하나만 추출해 Binarizer를 적용\r\n",
        "pred_proba_1 = pred_proba[:,1].reshape(-1,1)\r\n",
        "\r\n",
        "binarizer = Binarizer(threshold=custom_threshold).fit(pred_proba_1)\r\n",
        "custom_predict = binarizer.transform(pred_proba_1)\r\n",
        "\r\n",
        "get_clf_eval(y_test, custom_predict)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "오차행렬\n",
            "[[98 20]\n",
            " [10 51]]\n",
            "정확도:0.832402, 정밀도:0.718310, 재현율:0.836066\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xZ9gAHTH7Eez"
      },
      "source": [
        "- 임곗값을 낮추니 재현율 값이 올라가고 정밀도가 떨어졌습니다. why? : 분류 결정 임곗값은 Positive 예측값을 결정하는 확률의 기준, 확률이 0.5가 아닌 0.4부터 Positive로 예측을 더 너그럽게 하기에 임곗값을 낮출수록 True값이 많아지게 됩니다.\r\n",
        "- Positive 예측 값이 많아지면 상대적으로 재현율이 높아집니다. 양성 예측을 많이 하다 보니 실제 양성을 음성으로 예측하는 횟수가 상대적으로 줄어들기 때문"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L34xwz9K-lbX"
      },
      "source": [
        "- 이번 예제는 임곗값을 0.4에서 0.6까지 0.05씩 증가시키며 평가 지표를 조사"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "seNHr997687X",
        "outputId": "9f78b82e-5eac-4c7d-c033-abdc949ccd5a"
      },
      "source": [
        "# 테스트를 수행할 모든 임곗값을 리스트 객체로 저장\r\n",
        "thresholds = [0.4,0.45,0.50,0.55,0.60]\r\n",
        "\r\n",
        "def get_eval_by_threshold(y_test,pred_proba_c1,thresholds):\r\n",
        "  # thresholds list객체 내의 값을 차례로 interation하면서 Evaluation 수행.\r\n",
        "  for custom_threshold in thresholds:\r\n",
        "    binarizer = Binarizer(threshold = custom_threshold).fit(pred_proba_c1)\r\n",
        "    custom_predict = binarizer.transform(pred_proba_c1)\r\n",
        "    print('임곗값:',custom_predict)\r\n",
        "    get_clf_eval(y_test,custom_predict)\r\n",
        "\r\n",
        "get_eval_by_threshold(y_test,pred_proba[:,1].reshape(-1,1),thresholds)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "임곗값: [[1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]]\n",
            "오차행렬\n",
            "[[98 20]\n",
            " [10 51]]\n",
            "정확도:0.832402, 정밀도:0.718310, 재현율:0.836066\n",
            "임곗값: [[1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]]\n",
            "오차행렬\n",
            "[[103  15]\n",
            " [ 12  49]]\n",
            "정확도:0.849162, 정밀도:0.765625, 재현율:0.803279\n",
            "임곗값: [[1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]]\n",
            "오차행렬\n",
            "[[104  14]\n",
            " [ 13  48]]\n",
            "정확도:0.849162, 정밀도:0.774194, 재현율:0.786885\n",
            "임곗값: [[0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]]\n",
            "오차행렬\n",
            "[[109   9]\n",
            " [ 15  46]]\n",
            "정확도:0.865922, 정밀도:0.836364, 재현율:0.754098\n",
            "임곗값: [[0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]]\n",
            "오차행렬\n",
            "[[112   6]\n",
            " [ 16  45]]\n",
            "정확도:0.877095, 정밀도:0.882353, 재현율:0.737705\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wlD1YMWIAR4k"
      },
      "source": [
        "precision_recall_curve() 이용"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TXebyRip_xMp",
        "outputId": "4554c190-570d-4155-a66b-39fe0d73f164"
      },
      "source": [
        "from sklearn.metrics import precision_recall_curve\r\n",
        "\r\n",
        "# 레이블 값이 1일 때의 예측 확률을 추출\r\n",
        "pred_proba_class1 = lr_clf.predict_proba(X_test)[:,1]\r\n",
        "\r\n",
        "# 실제값 데이터 세트와 레이블 값이 1일 때의 에측 확률을 precision_recall_curve 인자로 입력\r\n",
        "precisions,recalls,thresholds = precision_recall_curve(y_test,pred_proba_class1)\r\n",
        "print('반환된 분류 임곗값 배열의 shape:',thresholds.shape)\r\n",
        "\r\n",
        "# 반환된 임곗값 배열 로우가 147건이므로 샘플로 10건만 추출하되, 임곗값을 15 step으로 추출\r\n",
        "thr_index = np.arange(0,thresholds.shape[0],15)\r\n",
        "print('샘플 추출을 위한 임곗값 배열의 index 10개 :', thr_index)\r\n",
        "print('샘플용 10개의 임곗값 :',np.round(thresholds[thr_index],2))\r\n",
        "\r\n",
        "# 15 step 단위로 추출된 임계값에 따른 정밀도,재현율 값\r\n",
        "print('샘플 임계값별 정밀도:',np.round(precisions[thr_index],3))\r\n",
        "print('샘플 임계값별 재현율:',np.round(recalls[thr_index],3))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "반환된 분류 임곗값 배열의 shape: (143,)\n",
            "샘플 추출을 위한 임곗값 배열의 index 10개 : [  0  15  30  45  60  75  90 105 120 135]\n",
            "샘플용 10개의 임곗값 : [0.1  0.12 0.14 0.19 0.28 0.4  0.56 0.67 0.82 0.95]\n",
            "샘플 임계값별 정밀도: [0.389 0.44  0.466 0.539 0.647 0.729 0.836 0.949 0.958 1.   ]\n",
            "샘플 임계값별 재현율: [1.    0.967 0.902 0.902 0.902 0.836 0.754 0.607 0.377 0.148]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 388
        },
        "id": "dioJADO5CCu_",
        "outputId": "f48b6ca2-4f31-4fac-82c4-9f8c7036f3c9"
      },
      "source": [
        "# 정밀도, 재현율의 임곗값에 따른 값 변화를 곡선 형태의 크래프로 시각화\r\n",
        "import matplotlib.pyplot as plt\r\n",
        "import matplotlib.ticker as ticker\r\n",
        "%matplotlib inline\r\n",
        "\r\n",
        "def precision_recall_curve_plot(y_test,pred_proba_c1):\r\n",
        "  precisions,recalls,thresholds = precision_recall_curve(y_test,pred_proba_c1)\r\n",
        "\r\n",
        "  # X축을 threshold 값으로 Y축을 정밀도, 재현율 값으로 각각 plot 수행, 정밀도는 점선으로 표시\r\n",
        "  plt.figure(figsize = (8,6))\r\n",
        "  threshold_boundary = thresholds.shape[0]\r\n",
        "  plt.plot(thresholds, precisions[0:threshold_boundary],linestyle = '--', label = 'precision')\r\n",
        "  plt.plot(thresholds, recalls[0:threshold_boundary],label = 'recall')\r\n",
        "\r\n",
        "  # thresold 값 x축의 scale을 0.1단위로 변경\r\n",
        "  start,end = plt.xlim()\r\n",
        "  plt.xticks(np.round(np.arange(start,end,0.1),2))\r\n",
        "  # x축,y축 label과 legend, grid 설정\r\n",
        "  plt.xlabel('Threshold value');plt.ylabel('Precision and Recall value')\r\n",
        "  plt.legend();plt.grid()\r\n",
        "  plt.show()\r\n",
        "\r\n",
        "precision_recall_curve_plot(y_test,lr_clf.predict_proba(X_test)[:,1])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfEAAAFzCAYAAAAuSjCuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3xV9f3H8df3Zu9AEnbYEPYGGQoBF6CAA1fFjVRbq611dmnVtlbrr3VvLVorolXroDgZoiJ7b2QjEALZO/n+/jgBg5LkEnLvyb15Px+P87j33HvuyTsR88n3nO8w1lpEREQk8HjcDiAiIiJ1oyIuIiISoFTERUREApSKuIiISIBSERcREQlQKuIiIiIBKtTtACcqMTHRdu7c2e0YdZKfn09MTIzbMepE2d2h7O4J5PzK7g5fZV+6dOlBa23K8d4LuCLevHlzlixZ4naMOpk7dy7p6elux6gTZXeHsrsnkPMruzt8ld0Ys6O693Q5XUREJECpiIuIiAQoFXEREZEApSIuIiISoFTERUREApSKuIiISIBSERcREQlQKuIiIiIBSkVcREQkQPmsiBtjXjLGHDDGrKnmfWOMecwYs8UYs8oYM8BXWURERIKRL1vi/wTG1vD+OKBL5TYNeNqHWURERIKOz+ZOt9bON8a0r+GQScAr1loLLDTGJBpjWlprv/NVph+xFrZ8Ck07QlInv31ZERFx16H8ElbvyWZUV2ddkfXf5bAvp+iYY8I8Hk7tkgzAmj3ZZOQVH/N+RKiH4Z2c91fuymJVRhl24wEGtmtCfGSYH74LME4N9dHJnSL+gbW213He+wB40Fq7oHL/M+BOa+2PVjcxxkzDaa2TkpIycObMmfUT0FrS553HtvaXsaP9pfVzzhrk5eURGxvr86/jC8ruDmV3TyDnV/baPbmiiMX7yvnnWGfVsRdWF7NgT9kxx8SEwZOnO+8/sbyIJfvLj3k/KdLwSHo0AH9bXMSaTOf9e4ZF0iEhpN6yjh49eqm1dtDx3guIVcystc8BzwGkpaXZelslxlqYBx3ataODH1bN0eo87lB2dwRydgjs/Mpeuzu/+pSerWJITz8NgA698zmUX3LMMaEeD73bJADQrlc+WQXHvh8W4qFXa+f9Nj3ymPfVNwwYMICuzeOIifBPeXWziO8BUqvst6l8zX+M8euXExER932XXcj+nGJuGPX9bdR2STG0S6p+LfAOyTFA9e93bhbL7sQQ+rdtUp9Ra+XmELP3gCsre6kPBbL9ej/8GL67pSAiIg3Lyl1ZAPRLTXQ5ycnzWUvcGPM6kA4kG2N2A/cAYQDW2meAWcB4YAtQAFzjqyy1Orwddi6ENoPBU3/3MUREpOHZsC+XsBBD95bxbkc5ab7snX5ZLe9b4Oe++vpei0iAVW8420XToed5bicSEREfuuX0LkwZ2o7IsMBvtGnGthvmw5S3nedZO9zNIiIiPmeMITk2wu0Y9UJFvEl76DQGwqIhd7/baURExIe2ZuTxi9eXs+VArttR6oWKODi91GObQ94+t5OIiIgPLd1+mPdX7sUEyeikgBgn7hexzSFzC+z4+sfvNesOUYHfi1FExFpLYWk5UWEhQVPITsSK3VnERYbSoYbhZIFERfyIJu1h1Qx4+TjTvXefCJe86vdIIiL17aO1+7jhX8vwGIiNCCUuMozYiFBuPzuNM3o058gsnsFa4FfszKJvm0Q8nuD4/lTEjxj3IPQ9ztSrXzwCB9b7P4+ISD06kFNEs/hIzu7ZgvvP68X+7CLyisvIKSolr6iM2EinHHy8bj+/f3cNrZtE0SI+khYJkaTERXD5kHYkRIexL7uIrMISEqPCSYwOC6ge3oUl5Wzcn8uNo4JnrQwV8SOimkCn0T9+/du58PWTUFGuMeQiEnDKKyz/98lGXlqwnfduGkGX5nFcMbRdtceHGMNpXVLYl1PIpv25zNuUQUFJOZMHtAHCeH3RTh79bPPR49OaxzGsUxK3n53mt6lG6+pAbhFdm8cxoF3w3B5t2D/xhqBpR6gohezd0KT6f/giIg1NZl4xt8xYwYItB7l0cCqpTaNr/cwZPZpzRo/mR/ettRSVVhAR6vSDntivFWkt4sgqKOVAbhFLth/m47X7+MO5PQCYtymD6PAQerSMb3BFvV1SDP+75TS3Y9SrhvUTboiOLFF6aKuKuEgQW7Mnm4zcYuKjQkmICiO7OLCnY16xK4uf/WspB/NLeOjCPlw8OLX2Dx2HMYao8O+vQnZKiaVTyrGrjFVUWDweQ3FZOb+euZKDecUYAx2SYujSPJZRXZvxk1PaAs49+fjIMJJiw0mKCScxOpyQILk/7QYV8dokVhburJ3u5hARnykuK2fyM19RVFpx9LXUOA+Tznaeb96fS2rT6IC6//vByr14PIa3bxx+dKUtXznSSSwiNITZvzyNFTuzWLs3h7V7s9lyIO/owiLlFZYb/7WUiip/H3kMXD+yI3eP605JWQVPriji8+w1JMVE0DQ2nOSYcHq0iq9xcRJvnfPYF5zVowW3nNHlpM/VUKiI1ya0clafirKajxORgLUvu4jYiDB+MaY9vVonkFNYyuYN6wB4Z/lufvXGSgDiIkJJiYsgOS6CP07sSfeW8Wzcl8sXmzNoGhNOk5hwEqLCiI8MJbVpNBGh/i36hSXlfJddCMCd47px05jOJEaH+zVDcmzEjy7JH2GAWbecxqG8Eg7ml3Aor5jM/BL6t3XuUecVl7Ert4JNK/aSXVh69HN3jE3jZ+mdyS4o5cHZ6zm9W3NGd2t2Qi34zLxi1u7NYVK/Vif9PTYkKuIi0ui1S4phye/OOHpZGGDu4U0ApHdtxsOT+3Agt5iM3GIy8orJyCkmLMQ5bsmOQzzw4Y9HsHx660g6N4vjla+3849PNxMf6Qznio8KJS4ijL9c0JsmMeEs2X6IVbuziY8KIy4ylMSoMFomRNGmSdQJDYPakZnPT19dSl5xGfcMNoSFePxewGvj8Ri6tah+0ZGmMeE8eFo06enplJZXcDi/hMz8EprGON/HpgO5vL/yO15ftIv+bRP548Se9G6d4NVwuFW7swHo2yZ4OrWBiriIyFHHK5pNYsK5aFD195MvG9yWc3u34lBBCYfyS8gpKiW3qIyWCVGAcw95fO8W5BSWkVtUSk5RGRm5eUe/1qfrD/DMvK0/Ou+6+84mOjyU6V9t55ttmaTERpASF0GzuEhS4iMYndbs6LGfrtvPr2auwGMM/7i0H+a7dSf7o3BdWIiHZvGRNIuPPPra4PZNWfb7M3l/5V7u/3AdE5/4kpYJkcy9PZ2I0BAKS8qJDPMct6gv35WFx0DvNr69teBvKuLeytkLpUUQFln7sSISUG6ZsZyOybF1ulfq8RgSosNIiA6jQ/KP79uO6JzMiM7J1X7+trO6csOojuQUOmO2DxeUkJFbTHS48+s5u7CUTfvzWLD5IDlFzm29xOgwVvzhLAD+9tFGnpizhV6t43n68oGkNo1mbhAU8eqEh3q4cGAbRndrxifr9rHzUMHR2xZTX1nMmj05dG0eS9fmcaS1iKN36wT6t23Cyl1ZdG0ed/TnGiyC67vxhZBwwDiTvuQdgElPuJ1IROpReYXl47X7uXSIO5eeQysve1d36fvm07tw8+nOHxdFpeVk5BaTU+TcL7bWMn9zBpcNSeWeCT0DquPdyWoaE84lg9se89qEPq1onxTDpv25vLdyL7nflDGqawrTrx3CsE5JhIUE33IhKuK1iUqE6z6BWb+G/WvdTiMi9ejWmSsAKCwtp2erhn+ZNTIs5Jix3sYY3rvpVBcTNSyXDmnLkXk3rbXszymmoMS5enFDEM3SVpWKuDdSB0OrAbDuXbeTiEg9qaiw7DpUwLq9OYR4DIPbN3E7ktQjYwwtEoL/9qeKuLeadoDCw1CYpRXNRALY5xv2M3djBneN68abNwynosJZ1auhzS4m4o3gu0HgK006OI97l7mbQ0TqrKCkjN+/u5avt2YS6nF+/Xk8RgVcApaKuLeSuzqPr54P5Zr4RaShK6+w/P7dNezMLDj62qOfbmZPViF/vqA34aH69SeBT/+KvdWsG3SpnIOxJNfdLCJSqzV7snl14Q5+9u+lAKz/LocXFmzjkkGpDG7f1OV0IvVDRfxEdJ/gPBariIs0dInRYQAcyCkG4L7315EYFcbd47u5GUukXulG0ImIqFy5pzjP3RwiUqsjY4LP798agL9e2Iddhwsa3FSkIidDRfxERMQ5jwc3QVjUCX88snAfHNpWz6H8w6fZIxMgWpc3pX4lRofxyEV9GdDOGTrWNimatkm1r6ctEkhUxE9EVGWhefOqOn18KMA39ZbGr3ya3RMKv94IMdVPTSniLWst763cy47MAs7v3/qYyVFEgo2K+Ilo1R8uea3O98TXb1hP927d6zmUf/gs+56lsPh5yD+oIi4nLbuglLvfWcWs1fsAWL7zMC9fM8TlVCK+oyJ+IoyB7ufW+eP7s+bSvV96/eXxI59lj4hzinh5cf2fWxqVNXuyuf6VJWTkFnPn2G5M6NuSotJyt2OJ+JSKuLgrNMJ5LC91N4cEvGZxEbRKjOLZKwbSJ8jWjBapjoq4uCvEGQZEzh7IaXWSJzMQ18K5YiKNwo7MfP751XZ+d04PmsVH8tYNw467lrRIsFIRF3eFV/b4n3ll/Zxv1J0w+jf1cy5xzZ6sQqa88A2hHkNSbDhje7bg6hHO1MdvLN5JYnQ4B3KLeXDWekI8hstPaUvnZnEq4NLoqIiLu1r1h8kv1c8EOl89AdsXnPx5xHWrdmWx7WA+p3ZOprisnILKe9uFJeXc+Z/VR48b0qEp/7ikH60ST3zIp0gwUBEXd3k80OvC+jnXvjWw8nWoqHDOKwFrb3YRAI9f1p8mMd9PzhIR6mHh3adzMK+YotJy+rdtQohHrW9pvFTEJXi06uf0dM/cAild3U4jJ2Fcrxa0bRp9dOrUIzweZ43oxrBOtIg3VMQleLTq7zzuWADRSc5zT4h7eaTOWiVG6RK5iBdUxCV4JKdBWDR88Ctnq9SnST/o8CC0G+ZiOPHW8p2HWbD5IKd1TaFfqoaKidRERVyCR0go/OQNOLDh+9fyM4j9+ll4eSy0Pw1G3g4dRmoYWgNTUWH5bMMBnpu/lcXbDxMfGUqX5rGgIi5SIxVxCS4dRjpbFQvtQEZGb4UvH4NXJkLqKTDyDuh8uop5A1BUWs6Exxew+UAerROj+MO5PbhkcCoxEfr1JFIb/V8iQa8iJBKG/RwGXQfLX4UF/4DXLnTuoY+8A9LGqZj7WVZBCQu2HOTcPq2IDAvhjB7NuWlMZ87p3ZLQEI0sEPGWirg0HmGRMOR6GHCVMxRtwf/BjMugeW8YeRt0Gg3UUswj4lTwT8KuQwW8uGAbbyzeRVFZOQPbNaFlQhR3ju3mdjSRgKQiLo1PaDgMvAr6XQ6r34Qv/ub98rKDroNz/8+3+YLUC198y59nrcdjDBP7tWLayI60TFAPdJGToSIujVdIKPS7DPpcDBtnweEdNR+/5CU4sN4/2YJMXnEZT8zZwui0Zjxwfi8Vb5F6oiIu4gmB7hNqP27XQji4xfd5glBsRCizbj6NyLAQmlaZgU1ETo56kIh4KzIRirLcTtGgWGtrPWZrRh7WWlolRqmAi9QztcRFvBXVBAoOQVmJc1+9kSsqLWfUw3MoLCmnRUIkr153Cs3jI1m64xAb9uWScaCMiK2ZTJ2+mGtGdOC2s9PcjiwSdFTERbwVnQTlxfBACrQaAEOmQc/znV7vjVBGbjH7c4oZ1jGJuMhQ4iOdec7/t3ofLyzY5hy0bCGRYR4uHpTqYlKR4KUiLuKtfpc7w8tK8mHtO/DuDfDxb2HAlTDoWkhs63ZCv8opKgXgquHtGdurxdHX7xrXjamndWTWnC9p1bknLRMiaZsU7VZMkaCmIi7irdgUGHGL8zz9btg2DxY9D18+6mxdx8GQqdBxdNCOJd+Rmc+KXVmM6JyMtdAxJYbk2GNvLYSGeGiREEnHxBDSqxR3Eal/KuIidWEMdEx3tqxdsPRlWPpP2PghJHWBwVOd4WuRCa7GrE97sgq5+Nmv2Z9TDMDtZ6fx+a/T3Q0l0sipd7rIyUpMhdP/AL9aB+c/6xTu2XfCI92d1dT2rwNra94auKyCEq56aREFJeU8M2Ugvz6zK6d0aOp2LJFGTy1xkfoSFgl9L3W2Pctg8Quw/DVnkpia9JgEF7/in4x1tPtwITmFpTx3xSCGdUo65h64iLhHRVzEF1oPgNZPwZn3w5r/QOGh4x93YB2s+y/sXgJtBvk3oxestRhj6NU6gfl3jCYyLMTtSCJShYq4iC/FJMEp06p/vzgPti+AuQ/ClLf8l8sLy3Ye5oEP1jE6rRk3jemsAi7SAPn0nrgxZqwxZqMxZosx5q7jvN/WGDPHGLPcGLPKGDPel3lEGpyIWBj+C9jyCexe6nYaAHYfLuDm15dzwVNfsetwIS0TozBB2tteJND5rIgbY0KAJ4FxQA/gMmNMjx8c9jtgprW2P3Ap8JSv8og0WIOvh6imMO9Bt5Mwc/EuTn9kHh+t3cfNYzoz97Z0Jg9s43YsEamGLy+nDwG2WGu/BTDGzAAmAeuqHGOB+MrnCcBeH+YRaZiOtMY/+yPclwwhYdD+VOh2LmElvh+iVl5hKSotJyYilK4t4hjXqwV3jO1Gq0StNCbS0PmyiLcGdlXZ3w2c8oNj7gU+Nsb8AogBzvBhHpGGa+iNUFEOpQVQnAObP4bNHzMcD+x5DrqdC93PrfdZ4bILS5k6fTGdUmJ58MI+9EtN5B+X9q/XryEivmO8WYWoTic2ZjIw1lo7tXL/CuAUa+1NVY65tTLDI8aYYcCLQC9rbcUPzjUNmAaQkpIycObMmT7J7Gt5eXnExsa6HaNOlN3PrCU2bxvxe+fTKmcZsfnOWue5sR05mDyUg8lDyY9pe1Izw+WVWB5eUsTu3Aqu7RXOiNZh9ZXeOX8g/tyrCOT8yu4OX2UfPXr0UmvtcYev+LKIDwPutdaeXbl/N4C19i9VjlmLU+h3Ve5/Cwy11h6o7rxpaWl248aNPsnsa3PnziU9Pd3tGHWi7O44mj1zK2z4ANZ/ALsXOW827VjZQp8ArQeBx/suLhm5xUx54Ru2Zebz7JSBjO7WzHfZA1Qg51d2d/gquzGm2iLuy8vpi4EuxpgOwB6cjms/+cExO4HTgX8aY7oDkUCGDzOJBKakTs687SNugdx9sOFDp6gvfAq+egxiW0C3c5xL7u1Pc+6rV8Nay9Tpi9l5qICXrx7MiM7JfvxGRKQ++ayIW2vLjDE3AR8BIcBL1tq1xpj7gCXW2veAXwPPG2N+hdPJ7Wrrq0sDIsEirgUMvs7ZCrOc++fr34eVr8OSF8ETBiHVr3dugLethQgIeaMOl+NbD4Ap/4HQiLp/DyJSL3w62Yu1dhYw6wev/aHK83XACF9mEAlqUYnQ52JnKy2ErXNg10Knk9xxlJZbwkIMdZ62pSTPWehl/t9gzG/rehYRqSeasU0kWIRFQbfxznYc1lomPraAAW0T+dP5vev+dUoLYcH/Qc/zoHnPup9HRE6aVjETaSS+2HyQ9d/l0Dc18eROdPZfnJXa3vtFtS1+EfEPFXGRIGetZfaa77j77dU0i4tgUr9WJ3fCmCQY9xDsWQrfPFM/IUWkTlTERYLco59t5oZ/LSMmIoSnpwwgIrQeFjLpdSF0ORs++g3M/g0UZZ/8OUXkhOmeuEgDUlpewTvL9uDxGJpEh0GFM1ijqLScUI8hNMS7v7sP5BRRXFZBatNoLhzQhqTYCC4bnOr152tlDEx6Ej75vTPMbfWbcOZ9zlrqWixFxG9UxEUakOlfbeeBD9cf3X/xrGgAHvhwHf9auJO4yFASo8NoEh1OcmwEL109GICP1u5jz+FCEqPD2HWokGfnb2V4p2ReuGoQqU2juWJou/oPG5sC5z8DQ6bBrNvh3RucnuvjH4aWfer/64nIj6iIizQQh/NLeOyzzZzWJZkHzutFdmEph7asAODMHi1Ijo0gq6CUrIISsgpLKa/4fkqF/yzdzcfr9h/dP7tnc+4e190/wVsPgOs+gZX/hk/ugedGwaBrCQ1P98/XF2nEVMRFXHTPf9ewfFcWd43txq7DBeQVl/G7c3rQLikGgLlbnONGdU1hVNeUas/z9JSB5BaVcrigFAO0T47xQ/oqPB7oP8WZNW7OX2Dx8wwJnQlND0H/K05oSlgR8Z6KuIhLSsoqeHPpbsoqLBi4ZHBbTumQVKcCHOIxJEaHkxhd/UxtfhHVBMY/BAOuoOD1aYS/fzN8ek+NM8hVKzTSWaJ10LXgqYfOeCJBSEVcxCVLdxymoKScZ68YyLCOSYALLWhfadGbFf3+THpyJmz/om7nyNgEs26DlTNgwqPQolf9ZhQJAiriIi75aO0+Qj2G4Z2SMMHYo9sY6D3Z2erCWqfX++y74dmRMPwmGHUnhAfJHzoi9UA3qkRc0jElhvS0ZsRF1u863kHDGGdO+JsWQ//L4ctH4amhsPkTt5OJNBgq4iJ+tDOzgLkbDwBwxdB2PHl5f5cTBYDopjDxcbh6lnOf/LXJ8OY1zpKsIo2cLqeL+Ml/V+zht++sISEqjDm3pRMe6qmf2dMai/Yj4IYFTot8/t9g02yIrmEt9E6jYeJj/ssn4gIVcRE/+GjtPm6ZsYKB7Zrw6KX9CA/VRbA6CY2AUXdAzwvgm6ehpOD4xx3cBMtfhdPvceZ6FwlSKuIiJ6ikrAJjIMzLKUzLyit4aPYGOjeLZca0oV5/TmqQ3BnOeaT69/cuh+fSndZ6/8v9FkvE31TERU5AXnEZY/42l4y8YpJiInjgvJ6M7dWSXYcKeHPpbprHR9A8LpJm8RE0j48kOTaC+Zsz2JqRz7NXDFQB95eW/SC+NWycpSIuQU1FXOQEzFi0kwO5xVw7ogOFpWW0SowCYGtGHo9/vhlrjz1++rVDGNOtOW//bDj9T3Ydb/GeMZA2Dpa/5lxyD492O5GIT9RaxI0x0cCvgbbW2uuNMV2ANGvtBz5PJ+KyotJyPt9wgPdX7uWq4e154YttDOuYxB8m9DjmuPS0Zmx+YBwH80rYn1PEgdxi9ucU0b1FHAAD2jZxI37jljYeFr8A2+ZD2li304j4hDct8ZeBpcCwyv09wJuAirgEpbLyCr7YfJD3Vu7l47X7yC8pJzk2gnG9W3L3+G60rmx9/1BoiIcWCZG0SIj0c2I5rpQ05zFvf83HiQQwb4p4J2vtJcaYywCstQUmKKeXksasosKyN7uQNk2iqbBwy4zlGGOY0LcVE/q2YmjHJEI8+mcfWI7897I1HiUSyLwp4iXGmCgq/08wxnQCin2aSsTPHv98CzOX7GLBnaMJD/UwY9owOjeL1VCwQHakrfHDjgoiQcSb31D3ALOBVGPMa8BnwB0+TSVSj77NLmf7wXyKSsurPWbTgVyyCko4skR3j1bxKuABTy1xCX61tsSttZ8YY5YBQ3H+r7jFWnvQ58lEToK19uiiIo8uK+a+r+cCkBwbQevESMb1bskNozoB8PmG/XybkU+v1gm6ZB5M1BKXRsCb3ukjK5/mVj72MMZgrZ3vu1gidfdtRh63v7WKRy7qS/vkGG7sG0FyuzT2ZhWyN6uQPVmFlFc2uUvKKrhu+hKshQsGtHY5udQvtcQl+HlzT/z2Ks8jgSE4vdXH+CSRyEn45ttMpr6yhFCP4WBeMe2TY+jWNIT0gW2Oe3yIx/D+TaeyN6uQfhrHHVzUEpdGwJvL6ROq7htjUoF/+CyRyEl4Zt5WosNDeOuG4aQ2rX2CjxCPoVfrBHq1TvBDOvEv3RqR4FeXGdt2A93rO4jIicouLGXV7ixW7MwiM7+Eeyf2JCOvmB4t470q4BLk1BKXRsCbe+KP8/1NJQ/QD1jmy1AiP1RSVkFYiMEYw78W7uClL7fxbUb+0fe7No+lrLyC6PBQmsSEu5hUGg7dE5fg501LfEmV52XA69baL32URwSAwpJyPl63jxW7slixK4u1e3OYe1s6rRKjCA/10DE5lgv6t6ZfahN6t0kgISoMgPG9WtAuKcbl9NIghFXOrFdazXKlIkHAm3vi0/0RRKSq5+Z/y98/3URUWAi9Wydw1bB2R6+OXjwolYsHpR73c1eP6ODHlNKghUVBSDgUZrmdRMRnqi3ixpjVHP86lAGstbaPz1JJo1dQUkZ4qIfV955FqJbvlLowBqKaQOFht5OI+ExNLfFz/ZZC5AfuHNuNX5+VpgIuJycyEYrUEpfgVW0Rt9bu8GcQkSO2HcyndeW9b5GTEtUE8jPdTiHiM7X+ljTGDDXGLDbG5BljSowx5caYHH+Ek8Zn16ECLnrma+5+e7XbUSQYtBkEO7+GzK1uJxHxCW+aOk8AlwGbgShgKvCkL0NJ47Rk+yEuefZrSsrKuTG9o9txJBiMuMXp3Dbvr24nEfEJr65XWmu3ACHW2nJr7cvAWN/GksZi16ECyissj366mYuf/ZqwUA+vTR1K52ZxbkeTYBDbDIZcD6tmQsZGt9OI1DtviniBMSYcWGGMecgY8ysvPydSoxW7sjjr7/N59LPNvLpwB+f1a82HN59G7zaaAlXq0YhfQngMzH3Q7SQi9c6byV6uwCnaNwG/AlKBC30ZSoLbvuwilu88zG/fXUNyXDhThrZlytC2NIuLdDuaBKOYJDjlp/DFI5B34PvpWMOi4KwHICXN3XwiJ8GbIj4Q+NBamwP80cd5JMhkFZSwanc2fdskkhAdxvSvtnPPe2sBSImLYPo1Q1S8xfeG/wIOrIei7O/nUt+5ED74FVz94feFXSTAeFPEJwB/N8bMB94AZltry3wbSwJVRm4x/12xh1W7s1m5O4sdmc6Ul89dMZCzerZgeKck7pnQg76pifRoGU9kWIjLiaVRiGoCl71+7GtLXnKK+IYPoPuE439OpIHzZtrVa4wxYcA4nF7qTxpjPrHWTvV5Ogk42YWlPPDhelomRNK3TSKXDE6lb5tE+lau1d2leRxdmqvTmjQA/a+Eb56Fj38PXc6GUC2cI4HHq6VIrbWlxpj/4UzDGgWchzPUTOQYHZNjWPTb03WJXKA4MEEAACAASURBVBq+kFA460/w2oWw+HkY9nO3E4mcMG8mexlnjPknzjjxC4EXgBY+ziUBJruwlMF/+pRn5m9VAZfA0eUM6HQ6zPkL/OtCWPuO24lETog3Q8WuBN4F0qy1V1trZ+meuFSVVVDClBe+IaughIFtm7gdR+TEjH8YUgc7s7q9ebVT0K3WIJfA4M098cv8EUQC0/aD+fzstWVsOZDHs1cM5JSOSW5HEjkxSZ3ginegrAQ++CXMexAyt+BpconbyURq5dU9cZHjKSwpZ+ITCyirsDx/1SBGdU1xO5JI3YWGw6QnIakzfPZH+savhlP6Q6z+XUvDpZnX5IR8l13I64t2Yq0lKjyEv1/Sj7m3p6uAS3AwBk67FS5+hdi8bfDCGGd8uUgDpZa41KisvILlu7J4a1MJf135Beu/cxaw694ynn6piZzevbnLCUV8oMckVvQ7wMBND8OLZ8E1s6BFb7dTifxItUXcGLMaZ0jZj94CrLW2j89SiauyCkooLbekxEXw5dZMrnppESEGBneI4+5x3Ti9ezMtUCJBLze+C1w7Gx7rD1s+VRGXBqmmlvi5fkshDca2g/lc8uzXXDI4lV+flcYpHZry1OUDYN8Gxp85zO14Iv4V19J5VG91aaCqLeLW2h3+DCLu25GZz2XPLaS8wjKmWzMAIsNCGN+7JXMztYyjNEKmstuQrXA3h0g1arqcnkvNl9PjfZZK/O6LzRncOnMlZeUVvD5tKN1a6D+vyNEiftxfhSLuq7Z3urU2zlobf5wtztsCbowZa4zZaIzZYoy5q5pjLjbGrDPGrDXG/Luu34jUXV5xGTe/vpzEqDBmTBumAi5yVOXqZrqcLg2U173TjTHNgKPzaVprd9ZyfAjwJHAmsBtYbIx5z1q7rsoxXYC7gRHW2sOVX0P8LDYilFeuPYUuzWO1qphIVZ4QCImA4ly3k4gclzdzp080xmwGtgHzgO3A/7w49xBgi7X2W2ttCTADmPSDY64HnrTWHgaw1h44gexyktbsyebhjzaQXVhK7zYJKuAiP2QMxDWHvP1uJxE5LmNruUxkjFkJjAE+tdb2N8aMBqZYa6+r5XOTgbFHliw1xlwBnGKtvanKMe8Cm4ARQAhwr7V29nHONQ2YBpCSkjJw5syZJ/AtNhx5eXnExsa6HeOop1YUsfpgOY+MiiY6zNR4bEPLfiKU3R2BnB2+z99/2Z1UeMJZ2e9+tyN5LZB/9sr+Y6NHj15qrR10vPe8uZxeaq3NNMZ4jDEea+0cY8w/6ilbKNAFSAfaAPONMb2ttVlVD7LWPgc8B5CWlmbT09Pr6cv719y5c2ko2XdmFrDkozlcP7Ij48/sXuvxDSn7iVJ2dwRydqiS/0AaHFgfUN9LIP/slf3EeDPtapYxJhaYD7xmjHkUyPfic3uA1Cr7bSpfq2o38J61ttRauw2nVd7Fi3PLSXphwbeEeAzXjujgdhSRhi2uJeTshfJSt5OI/Ig3RXwSUAD8CpgNbAUmePG5xUAXY0wHY0w4cCnw3g+OeRenFY4xJhnoCnzrVXKps0P5Jcxcsovz+7emebzW/hapUftToSQP3vsFVGi8uDQs3hTxZkC4tbbMWjsdeB6odc7NyjXHbwI+AtYDM621a40x9xljJlYe9hGQaYxZB8wBbrfWZtblGxHvFZSUMTqtGdNGdnQ7ikjD130CpP8GVr4On/7B7TQix/DmnvibwPAq++WVrw2u7YPW2lnArB+89ocqzy1wa+UmftKmSTRPTxnodgyRwDHqDig4CF89DtHJcOov3U4kAnjXEg+tHCIGQOXzcN9FEl8qr7Cs3ZvtdgyRwGIMjP0r9LoQPr0Hlr3qdiIRwLsinlHl8jfGmEnAQd9FEl9asOUg5zy2gM83aNyryAnxeOC8Z6DTGHj/Zlj7jtuJRLwq4jcAvzHG7DLG7ATuBH7q21hS38orLNO/2s7P/rWUlLgIhndKdjuSSOAJDYeLX4U2Q+DNq2H+3zQlq7iq1nvi1tqtwNDKYWZYa/N8nkrq1eb9udz5n1Us25nFaV2S+fP5vTU7m0hdRcTClf+F926Cz++HjI0w8XEI00gP8b9ai7gxpjnwZ6CVtXacMaYHMMxa+6LP00md5RaVkl9cTouESCos7Mgs4O+X9OW8fq0xpubZ2USkFmGRcMHzkNLNKeSHvoVL/+1M0SriR970Tv8n8DLw28r9TcAbgIp4A3I4v4TVe7JZvSebNXuymb8pgzHdm/P4Zf1JaxHH13efTnioN3dPRMQrxsDI2yAlDd6eBs+Phsteh5Z93U4mjYg3v9WTrbUzgQo4Ov673KeppEaH8kuYtymDt5ftPvra5S98w5UvLeLhjzaydm8O43u3ZOqp38/GpgIu4iPdJ8C1HwEGXhoLK17XfXLxG29a4vnGmCTAAhhjhgIao+SSeZsyuPrlRVgLcZGhnN/fuTx+x9g0wkM89GyVQEJ0mNsxRRqXln3g+s9h5pXw7g2w4jU45xGnlS7iQ94U8VtxpkvtZIz5EkgBJvs0lVRr8/5crIWXrx7MgHZNjt7fTk/TUuwiroprDtfMgmXT4dN74ekRMPwmGHk7hMe4nU6CVK3XWK21y4BROLO2/RToiRfTropvbNqfS2SYh5FdU0iIUotbpEHxhMCga+GmpdDnYljwd3jyFNjwoS6xi09UW8SNMSHGmMuMMbcBadbatUB7YB7whJ/ySaWKCucXQNfmcVx/WkdCPOphLtJgxabAeU/BNf+DiDiY8RN4/VI4vN3tZBJkarqc/iLOUqKLgMeNMXuBgcDd1tp3/RGusTuQW8THa/fz0dp9hId4ePHqwUw9TYuWiASMdsPhp/Phm2dgzl+cVvnI22D4zRAa4XY6CQI1FfFBQB9rbYUxJhLYB3TSKmP+sWZPNhc89RUl5RV0SI5hQp+WbkcSkboICYPhv4CeF8BHd8PnD8DKGTD+b9BptNvpJMDVVMRLrLVHhpUVGWO+VQH3n4/X7aesooIPbz6VHi3jNUGLSKBLaA0XvwJbPoVZt8Or5zkLqpz1J4jXH+lSNzV1bOtmjFlVua2usr/aGLPKXwEbq2ZxEZzaJYWerRJUwEWCSecz4MavnTXK138ATwyGhU9DeZnbySQA1dQS7+63FEJuUSmrdmezYlcWFwxozZSh7bj8lLZuxxIRXwiLhPQ7oc9FTqt89l2w/DU49/8gdYjb6SSAVFvErbU7/BmkMdp1qIDHPtvMil1ZbMnIOzoCpUuzWFomRKkFLhLsmnaEy9+C9e87hfzFM2HAlTDuYS2oIl7xZrIX8ZEQj+GzDQfol5rIhL6t6JeaSN82iZpxTaQxMQZ6THTWKZ/3IHz1OBQcgoumQ4h+RUvN9C/Ej0rKLYu2HWLpjsNkF5Zy59g0lv7uDLW4RcRZ4vSsByC+Dcy+E96/GSY+AR6teyDVUxH3gzcW7+Tf3+xkzZ4Cyu3XAKQ1j+OOs9PwaNIWEalq6A1QlAVz/wKRCXD2n53WushxVFvEK3ukVztPoLW2j08SBaG84nIiwkIY2z6M80/rS/+2iSTFaqIHEanGqDuhMAsWPgVRTWDUHW4nkgaqppb4uZWPP698fLXy8XLfxQkeZeUVfLDqO3q1TuC6Uztw3akdmDt3Luk9mrsdTUQaOmOcFnhRNsz5E4THwrCfuZ1KGqBqb7ZYa3dU9lA/01p7h7V2deV2F3CW/yIGnuzCUq58aRG/fGMF763c63YcEQlEHg9MfBy6nevM9Pa/uzSWXH7Emx4TxhgzosrOcC8/12g9NHsDi7Yd4uHJfbj1zK5uxxGRQBUS6vRSH/pz+OZpeG0yFB52O5U0IN4U4+uAp4wx240xO4CngGt9GytwZeYVM2PxLi4b0paLBqW6HUdEAl1IKIz9s9NTffsCeOEMOLjF7VTSQHiznvhSa21foC/Ogij9KtcYl+PIKiylvMIyqH0Tt6OISDAZcAVc9b7T4e2FMbDlM7cTSQNQ6xAzY0wEcCHOWuKhR8Y0W2vv82myANUyIZLp1w6he4s4t6OISLBpNwyu/xxev8y5tD7+bzD4OrdTiYu8uZz+X2ASUAbkV9nkOAyGUV1TaBavKRNFxAeatIPrPoYOo5ypWotz3U4kLvKmiLex1l5irX3IWvvIkc3nyQLQrNXfMeGJBRzOL3E7iogEs4hYZ+x4eQls/sTtNOIib4r4V8aY3j5PEuCstby3Yi8H84qJi9REeCLiY6mnQHQybPjA7STiIm+K+KnAUmPMRq0nfnzZhaVc/8oSZq/dx3n9WhMaohF4IuJjnhDoNh42fQxlxW6nEZd402Qc5/MUAW7Gop18uv4A90zowdXD27sdR0Qai+4TYdkr8NFvIKHN0ZdTd34LC5ZX/7nWA6HDSD8EFF+rtYgfWVfcGNMMUG+t40iMDmNI+6ZcM6KD21FEpDHpMNJZ9WzxC8e83Ang21o+O/J2SL/badFLwPJmiNlE4BGgFXAAaAesB3r6NlrguGRwWy4Z3NbtGCLS2IRGwC9XOR3cqpg/fz4jR1bT0i4vgdm/gfkPw94VcOHzziIrEpC8uXl7PzAU2GSt7QCcDiz0aaoAsnl/LlsOaIiHiLjEEwJhUcdsFSERP3rt6BaZAJOegHMegW/nwnOjYf86t78LqSNviniptTYT8BhjPNbaOcAgH+dq0MrKK3hxwTbGP/oFZ/59PpOe+JIvNme4HUtExDvGwOCpcPWHUFoAL5wOa952O5XUgTcd27KMMbHAfOA1Y8wBGvlkLyt3Z3P/B+vo0TKeP5zbg3P7tqRZnLoLiEiAaXsK/HQ+zLwS3roG9q2CM+51O5WcAG9a4pOAAuBXwGxgKzDBl6EauoN5znCOhyb34dpTO6iAi0jgimsBV30AfS6FBX+HzK1uJ5IT4E3v9COt7gpgum/jNFxr9mQTHxlGXGQoA9o24aWrB9E+OcbtWCIiJy803JmDfdUMyNwCSZ3cTiRe0tRiXpr2yhL2ZhcBsOH+sYzp1tzlRCIi9Sips/OYuQU429Uo4j1NLealO8d1O/r8mXm63CQiQSa6qTPULFNrlQcSFXEvPD13K2O6NeOJn/TnggGtuXBAm9o/JCISaJI6q4gHGG8mexkB3IszyUsoYABrre3o22juKygp49731jJzyW7+vWgHX9wxhnP7tHI7loiIbyR1ga2fuZ1CToA398RfxOmZvhQo922chuVgbgkzl+wG4Pz+an2LSJBr3hNW/hvyD0JMsttpxAveXE7Pttb+z1p7wFqbeWTzebIGoG1SNP+eegoA/dsmupxGRMTHmlfOpr1/jbs5xGvetMTnGGMeBt4Gjq53Z61d5rNUDcA332YSFupheOdktvxpnJYXFZHg16K387hvDXRMdzOJeMmbIn5K5WPVqVYtMKb+4zQcD320EY+BN28YrgIuIo1DTDLEtlBLPIB4M9nLaH8EaSgKSsqYsWgXy3ce5sZ0TXggIo1M6mBY/wGcuglSurqdRmpRaxPTGJNgjPk/Y8ySyu0RY0yCP8L528G8YkY+NIf7PljHoPZNuXq41gcXkUbm7L84S5zO+AkUZbudRmrhzXXil4Bc4OLKLQd42Zeh3HIwrxiPMTx0YR9m/nQYKXERbkcSEfGvxFS4+BU4vA3engYVFW4nkhp4c0+8k7X2wir7fzTGrPBVIDd1axHPot+e4XYMERF3tR8BYx+EWbfBvAdh9G/cTiTV8KYlXmiMOfXITuXkL4W+iyQiIq4bPBX6TYF5f4V1/3U7jVTDmyJ+I/CkMWa7MWYH8ARwgzcnN8aMNcZsNMZsMcbcVcNxFxpjrDFmUHXH+MOaPdlc8/IiNu3PdTOGiIj7jIFzHoHWg+DNa+DLx8Bat1PJD9RaxK21K6y1fYE+QG9rbX9r7craPmeMCQGeBMYBPYDLjDE9jnNcHHAL8M2Jhq9vB/OKmbMxg9yiMrejiIi4LywSrngHup8Ln/we3pgChVlup5Iqqi3ixpgplY+3GmNuBaYCU6vs12YIsMVa+621tgSYAUw6znH3A38Fik44fT0rK3f+ygzXuHAREUdkPFw03em1vmk2PJcO361yO5VUMraayyPGmJ9aa581xtxzvPettX+s8cTGTAbGWmunVu5fAZxirb2pyjEDgN9aay80xswFbrPWLjnOuaYB0wBSUlIGzpw506tv7kQt3lfGkyuKuX9EFKlx9V/I8/LyiI2Nrffz+oOyu0PZ3RPI+X2VPT57PT3XPkxYaQ6buv6UfS3PrPevoZ/7j40ePXqptfb4t5uttT7ZgMnAC1X2rwCeqLLvAeYC7Sv35wKDajtv165dra/8d8Ue2+7OD+zm/Tk+Of+cOXN8cl5/UHZ3KLt7Ajm/T7PnHrB2+kRr74m39p0brS0tqtfT6+f+Y8ASW01N9Gayl4eMMfHGmDBjzGfGmIwjl9prsQdIrbLfpvK1I+KAXsBcY8x2YCjwnlud2zLziokOC6FjcgwRoSFuRBARafhiU2DK2zDyDljxGix82u1EjZo314zPstbmAOcC24HOwO1efG4x0MUY08EYEw5cCrx35E1rbba1Ntla295a2x5YCEy0x7mc7ivWWmYs2sn4R79g4AOfkldcxue3pZPaNNpfEUREAo8nBMb8FtqdCktegopGtUp1g+JNET8yIcw5wJvWWq/m4bPWlgE3AR8B64GZ1tq1xpj7jDET65TWB+56ezXrvsshISqMnYcK3I4jIhI4Bl8HWTtgy2duJ2m0vJmx7QNjzAacCV5uNMak4GVPcmvtLGDWD177QzXHpntzzvo2pH1TrhnRnuYJkWQXlLoRQUQkMHU7F2Kbw+LnoetZbqdplLxZxewuY8xDQLa1ttwYk8/xh4oFHGMMM28Y5nYMEZHAFBoOA6+GeQ/BoW3QVItG+VtN48THVD5eAKQDkyqfjwWG+yWdiIg0bAOuAuOBpUG5LlaDV1NLfBTwOTDhOO9Z4G2fJBIRkcCR0Bq6jYdlr0L6b5xZ3sRvqi3i1tp7Kh+v8V8cEREJOIOvh/Xvw9p3oN9lbqdpVLwZJ/5nY0xilf0mxpgHfBtLREQCRoeRkNQFFr/gdpJGx5shZuOstUdnvLfWHgbG+y6SiIgEFGOcpUv3LIG9y91O06h4U8RDjDERR3aMMVFARA3Hi4hIY9PvMgiNhFVvup2kUfFmnPhrwGfGmCNdD68BpvsukoiIBJzIBGjRRy1xP/NmnPhfjTErgTMqX7rfWvuRb2OJiEjAadUPVvzbmYbVozUo/MHb9TbXA7OttbcBXxhj4nyYSUREAlHLflCSB5lb3E7SaHjTO/164C3g2cqXWgPv+jKUiIgEoFb9nMe9K9zN0Yh40xL/OTACyAGw1m4GmvkylIiIBKDkNAiPg9UzwVq30zQK3hTxYmttyZEdY0wozoxtIiIi3wsJhdN/D1s+hWXq/+wP3hTxecaY3wBRxpgzgTeB930bS0REAtLg66HDKJj9Gzj0rdtpgp43RfxOIANYDfwUZ2nR3/kylIiIBCiPB857Cjyh8M6NTk918Zkai7gxJgRYb6193lp7kbV2cuVzXU4XEZHjS2gD4x+GXQvhq8fcThPUaizi1tpyYKMxpq2f8oiISDDoczH0mASf3gtvT4PsPW4nCkrezNjWBFhrjFkE5B950Vo70WepREQksBkD5z3jLIzy1eOw7j049Zcw/GYIj3Y7XdDwpoj/3ucpREQk+IRHO73VB1wJn94Dc/8Cy16BM+6FXpOd++dyUqr9CRpjIo0xvwQuAroBX1pr5x3Z/JZQREQCW5N2cNE/4ZrZEJMCb18PL54Juxa7nSzg1fRn0HRgEE6v9HHAI35JJCIiwandMLh+Dpz3NGTvhhfPgP9MdZ5LndR0Ob2HtbY3gDHmRWCRfyKJiEjQ8nig30+g+0RY8Pfv75cPuR5O+7Xb6QJOTS3x0iNPrLVlfsgiIiKNRUSsc7/8F0ug92RY+BQ82pd222dCcZ7b6QJGTUW8rzEmp3LLBfoceW6MyfFXQBERCWKJbZ3JYW78CjqMpMP21+CxfvDNc1BWUvvnG7lqi7i1NsRaG1+5xVlrQ6s8j/dnSBERCXLNusOlr7Gs/0POQir/ux2eGAgr39CsbzVQ/34REWkwchLS4OoPYMp/IDIR3pkGz5wKG2e7Ha1BUhEXEZGGxRjofAZMmweTX4ayYnj9EqdVLsdQERcRkYbJ44FeF8DPv4HWg+Dj30JhltupGhQVcRERadhCwuCcRyD/IMz5s9tpGhQVcRERafha9YPB18Hi5+G7VW6naTBUxEVEJDCM+R1ENYVZt0FFhdtpGgQVcRERCQxRTeDMP8Kub2DzR26naRBUxEVEJHD0uQQiEmD9B24naRBUxEVEJHCEhEGXM2HTbE0Cg4q4iIgEmrRxUHAQdi9xO4nrVMRFRCSwdD4DPKGwcZbbSVynIi4iIoElKhHanworZ0DeAbfTuEpFXEREAs8Zf4SibJhxOZQWuZ3GNSriIiISeFr1gwuehd2L4L1fgLVuJ3KFiriIiASmHpOcCWBWz4QvHnE7jStC3Q4gIiJSZ6fdBhkb4fP7Ibkr9JjodiK/UktcREQClzEw8QlnlbP/TIWvHm9U48dVxEVEJLCFRcJPZkLn0+Hj38HL4yFzq9up/EJFXEREAl9MElz6bzj/WchYD0+PgIXPBP1CKSriIiISHIyBvpfCzxZCh9Ng9p0wfQIc2uZ2Mp9RERcRkeAS38q5vD7pSdi3ymmVL34B8g8eu5WXup30pKl3uoiIBB9joP8U6DDKGUf+4a+drar2p8HVgb0amoq4iIgEr8RUuOId2PAh5H73/eur34KDm93LVU9UxEVEJLgZA93PPfa1rB3w3Qp38tQj3RMXEZHGJzIRyooCft51FXEREWl8ohKdx6Jsd3OcJBVxERFpfKKaOo8HN7qb4yT5tIgbY8YaYzYaY7YYY+46zvu3GmPWGWNWGWM+M8a082UeERERwJndLb41zLodyordTlNnPivixpgQ4ElgHNADuMwY0+MHhy0HBllr+wBvAQ/5Ko+IiMhRkQkw4VHI2ADz/up2mjrzZUt8CLDFWvuttbYEmAFMqnqAtXaOtbagcnch0MaHeURERL7X5UzoNwUW/AP2LHM7TZ34soi3BnZV2d9d+Vp1rgP+58M8IiIixzr7TxDbDN79WUBeVjfWWt+c2JjJwFhr7dTK/SuAU6y1Nx3n2CnATcAoa+2PforGmGnANICUlJSBM2fO9ElmX8vLyyM2NtbtGHWi7O5QdvcEcn5lPzFNM5fQZ/X9bOl0LbtTJ9X+gWr4Kvvo0aOXWmsHHfdNa61PNmAY8FGV/buBu49z3BnAeqCZN+ft2rWrDVRz5sxxO0KdKbs7lN09gZxf2evg6VOtfWn8SZ3CV9mBJbaamujLy+mLgS7GmA7GmHDgUuC9qgcYY/oDzwITrbUHfJhFRESkep3GwK5voDjX7SQnxGdF3FpbhnOJ/COclvZMa+1aY8x9xpiJlYc9DMQCbxpjVhhj3qvmdCIiIr7TaQxUlML2BW4nOSE+nTvdWjsLmPWD1/5Q5fkZvvz6IiIiXmk7FMKiYevnkDbO7TRe04xtIiIioRHQ/lTYOsftJCdERVxERASc1njmZig45HYSr6mIi4iIALQe6DzuXe5ujhOgIi4iIgLQsp/zuDdwZm9TERcREQFnedKkLgE1BauKuIiIyBGtBzrDzLZ85nYSr6iIi4iIHHHarRDbHP51Afz3JijKdjtRjVTERUREjkhJgxu+gBG3wIrX4MmhsOljt1NVS0VcRESkqrAoOPM+mPqps+74vy+Cd25okEPPVMRFRESOp/VA+Ok8GHk7rJoJTw2FDR+6neoYKuIiIiLVCY2AMb+DaXMgphnM+Akse9XtVEepiIuIiNSmZV+4/nOIToLdi91Oc5SKuIiIiDdCw50i3oB6rKuIi4iIeCsyQUVcREQkIEUmQlGW2ymOUhEXERHxllriIiIiASokHMpL3U5xlIq4iIiIt4wHrHU7xVEq4iIiIt4yBmyF2ymOUhEXERHxloq4iIhIgDIeFXEREZGAFBEHxTkN5r64iriIiIi3opOhrAhK8t1OAkCo2wHqQ2lpKbt376aoqMjtKDVKSEhg/fr1bsf4kcjISNq0aUNYWJjbUUREGraYFOcxPwMiYt3NQpAU8d27dxMXF0f79u0xxrgdp1q5ubnExcW5HeMY1loyMzPZvXs3HTp0cDuOiEjDdrSIH4Sm7v/ODIrL6UVFRSQlJTXoAt5QGWNISkpq8FcxREQahJgk5zE/w90clYKiiAMq4CdBPzsRES+FRTuPZQ2j4RM0RTwYLVmyhJtvvrna9/fu3cvkyZP9mEhEpLE70uhpGL3Tg+KeeKAoLy8/oeMHDRrEoEGDqn2/VatWvPXWWycbS0REvHXkyqWGmAWX7du3061bNy6//HK6d+/O5P9v7+6jq6rOPI5/HxMgGIRGQzuMIKEzGMAkJEDUGCwBBPFloCI0MkWIWrU4pswwLMFhZswUZ00VVAaNq/UFg8WXCEu70NqCTomIgqASASEJIAFBlBpAEpyREp754+wLN5cgCeTmnhOfz1pZnNeb3z1u82Tve3L2uHF8/fXXpKSkMGPGDAYMGMArr7zC8uXLycnJYcCAAYwfP566ujoA1q1bxxVXXEH//v259NJLqa2tpaysjOuvvx6At956i8zMTDIzM8nKyqK2tpbq6mrS0tIA776AW265hfT0dLKyslixYgUAJSUljB07llGjRtG7d2/uueee2FwgY4xpE/z18WOb7Inn/2b1Sduuz+jGzTkp/O+RegqeWXvS/nEDuzN+UA/2Hz7ClEUfNNhXemdOk75vZWUlTz/9NLm5udx66608/vjjAFxwwQV8+OGHVFdXM2nSJN58800SExN54IEHePjhh5k5cyb5+fmUlpaSnZ3NoUOH6NixY4PXnjt3LsXFxeTmPHYY9wAAD1FJREFU5lJXV0dCQkKD/cXFxYgIGzdupKKigpEjR1JVVQVAeXk569evp0OHDqSmplJYWEiPHj2a9J6MMcaEsZ5429WjRw9yc3MBmDhxIqtWrQIgPz8fgLVr17J582Zyc3PJzMxk4cKF7Ny5k8rKSrp160Z2djYAnTt3Jj6+4e9Xubm5TJs2jfnz53Pw4MGT9q9atYqJEycC0KdPH3r27Hm8iA8fPpwuXbqQkJBAv3792LlzZ/QugjHGtGXn+uvu9DbZE/+2nnPH9nHfuv/8xPZN7nlHirzLO7SemJh4fNuIESN44YUXGhy3cePG0772zJkzue6663j99dfJzc1l2bJlJ/XGT6VDhw7Hl+Pi4jh69GiTzjPGGBOhYxJ06AwHdsQ6CWA98Ra1a9cuVq/2hvKff/55Bg8e3GB/dnY277zzDtu2bQPg8OHDVFVVkZqayt69e1m3bh3gPRQmstBu376d9PR0ZsyYQXZ2NhUVFQ32X3nllTz33HMAVFVVsWvXLlJTU6PyPo0x5jtLBJJ6woHqWCcBrIi3qNTUVIqLi+nbty8HDhxgypQpDfYnJydTUlLChAkTyMjIICcnh4qKCtq3b09paSmFhYX079+fESNGnPTwlXnz5pGWlkZGRgbt2rXjmmuuabD/rrvu4tixY6Snp5Ofn09JSUmDHrgxxpgWktTLN0W8TQ6nx0p8fDyLFi1qsK26urrB+rBhw473uMNlZ2ezZs2aBtvy8vLIy8sD4NFHHz3pnJSUFDZt2gR4zz9/5plnTjqmoKCAgoKC4+uvvfZaU96KMcaYU0lKgaplcOwYnBPbvrD1xI0xxpjmOL8X1H8DtXtjncSKeEsJ7xUbY4xpw5JSvH99cHObFXFjjDGmOZLc7GU++FzcirgxxhjTHF26g8TBfuuJG2OMMcES184r5J++B/V/iWkUK+LGGGNMcw26BarfhmfHQF3snt5mRdzHSkpKuPvuuwEoKipi7ty5MU5kjDEGgMH/BGOfhD0fwBN58Nn6mMSwIh4FqsqxY8diHcMYY0w0ZfwEbl3mLS8YxQ8+X9HqEayIt5Dq6mpSU1OZNGkSaWlpzJ49m+zsbDIyMrjvvvuOH/fss8+SkZFB//79ufnmmwF49dVXueyyy8jKyuKqq67iiy++iNXbMMYY0xx/nQl3lMGFg+hbMQ/++C9Q33rzU7S9J7b9YSZ8fvoJRZrlr9Lhml+d9rCtW7eycOFCDh06xJIlS1i7di2qyujRo1m5ciUJCQncf//9vPvuuyQnJ7N//34ABg8ezJo1axARnnrqKR588EEeeuihln0PxhhjoqNTV5j0O3YvKKD7mmL4YiPkL4KELlH/1m2viMdQz549ufzyy5k+fTrLly8nKysLgLq6OrZu3cr+/fsZP348ycnJAJx//vkA7N69m/z8fPbu3cuRI0fo1atXzN6DMcaYMxDXjm29b6f7oGthy6vQLvH057SAtlfEm9BjjpbQlKOqyr333sudd97ZYP+cOXMaPa+wsJBp06YxevRoysrKKCoqinZUY4wx0ZD1U8j8e2+2s1Zgn4lHwdVXX82CBQuoq6sDYM+ePezbt48hQ4awePFiampqAI4Pp3/11VdceOGFACxcuDA2oY0xxrSMVirg0BZ74j4wcuRItmzZQk5ODgCdOnVi0aJF9O3bl1mzZjFkyBDi4uLIysqipKSEoqIixo8fT1JSEsOGDWPHjtg/BcgYY4z/WRFvIZEToEydOpWpU6c2OKa2tpbJkyczefLkBtvHjBnDmDFjTnrN8GlEbYjdGGNMJBtON8YYYwLKirgxxhgTUFEt4iIySkQqRWSbiMxsZH8HESl1+98TkZRo5jHGGGPakqgVcRGJA4qBa4B+wAQR6Rdx2G3AAVX9W+AR4IEz/X6qeqanfufZtTPGmGCKZk/8UmCbqn6iqkeAF4HIu7fGAKG/qVoCDBdp/r35CQkJ1NTUWDE6A6pKTU0NCQkJsY5ijDGmmSRahU9ExgGjVPVnbv1m4DJVvTvsmE3umN1ufbs75suI17oDuAOga9euA1966aXI70ViYiJxcXFReS8tRVU5g99Roq6+vp7Dhw9/6y9BdXV1dOrUqRVTtRzLHhtBzg7Bzm/ZYyNa2YcOHfqBqg5qbF8g/sRMVZ8AngBITU3VvLy82AY6Q2VlZVj21mfZYyPI2SHY+S17bMQiezSH0/cAPcLWu7ttjR4jIvFAF6AmipmMMcaYNiOaRXwd0FtEeolIe+AmYGnEMUuB0JNPxgF/Uvtg2xhjjGmSqA2nq+pREbkbWAbEAQtU9WMR+SXwvqouBZ4Gfisi24D9eIXeGGOMMU0QtRvbokVEaoHKWOc4Q8nAl6c9yp8se2xY9tgJcn7LHhvRyt5TVbs2tiMQN7ZFqDzVXXp+JyLvW/bWZ9ljI8jZIdj5LXtsxCK7PXbVGGOMCSgr4sYYY0xABbGIPxHrAGfBsseGZY+NIGeHYOe37LHR6tkDd2ObMcYYYzxB7IkbY4wxBp8V8bOZulREMkRktYh8LCIbRaRVZ/RoQvYficiHInLUPVc+fN9FIrJcRLaIyObWnpK1CdmnuVwbROR/RKRngLL/3LWHchFZFT6Tnt/bTNhxN4qIisigsG2+zi4iBSLyZ3fdy0XkZ2H7fN1m3DE/cdk+FpHnw7b7OruIPBJ2zatE5GCAsl8kIitEZL37WXNt2D6/t/ee7mfjBhEpE5HuYfuie91V1RdfeA+E2Q78EGgPfAT0izjmLuDXbvkmoNQtxwMbgP5u/QIgzmfZU4AM4FlgXMS+MmCEW+4EnOuz7ENDmYApoesekOydw5ZHA38MSptxx50HrATWAIOCkh0oAB47xfl+bzO9gfVAklv/flCyRxxfiPeQrUBkx/s8eYpb7gdUu+UgtPfFwGS3PAz4bWtddz/1xM9m6tKRwAZV/QhAVWtUtb6VckMTsqtqtapuAI6Fb3c9w3hVfcMdV6eqX7dSbmha9hVhmdbgPQc/KNkPha0mAqGbQHzfZpzZwAPA/4VtC0r2kwShzQC3A8WqesBl3AeByR5uAvACBCa7Ap3dchfgM7cchPbeD/iTW14R2t8a191PRfxC4NOw9d1uW6PHqOpR4Cu838ouBlRElrkh63taIW+juZzGsp/KxcBBEXnZDSPNEZHWnFO1udlvA/7glgORXUT+Qbxpbh8EfuE2+77NiMgAoIeq/j7iXN9nd250w4tLRCQ0GVIQ2szFwMUi8o6IrBGRUWHb/Z4d8IZ3gV6cKCxByF4ETBSR3cDreCMJEIz2/hEw1i3fAJwnIqHaFNXr7qcifjbigcHAT92/N4jI8NhGarJ44EpgOpCNN2RTEMtApyIiE4FBwBy3KRDZVbVYVf8GmAH8q9vs6zYjIucADwP/3MhuX2d3XgVSVDUDeIMTI2hBaDPxeEPqeXi92SdF5HsEI3vITcCSsB5rELJPAEpUtTtwLd68GucQjPY+HRgiIuuBIXgzdNbTCtfdT0X8bKYu3Q2sVNUv3VDF68CAqCduJJfTWPZT2Q2Uu6Gao8Dv8GF2EbkKmAWMVtVv3OZAZA/zIvBjt+z3NnMekAaUiUg1cDmwVLyb2/yePTTkGWonTwED3XIQ2sxuYKmq/kVVdwBVeEU9CNlDbsINpTtByH4b8BKAqq4GEvCeRR6E9v6Zqo5V1Sy8n5Oo6kFa4br7qYifzdSly4B0ETnXFfchwOZWyg1Ny/5t535PREIPtx+Gz7KLSBbwG7wCvi/iXL9n7x22eh2w1S37us2o6leqmqyqKaqagncvwmhVfd/v2QFEpFvY6mhgS9i5vm4zeD9o8wBEJBlvSPQTgpEdEekDJAGrI871e/ZdwHAAEemLV8T/TDDae7IbNQC4F1gQdm50r3tL3iV3tl94QyhVeHcCznLbfon3wwu8/6iLgW3AWuCHYedOBD4GNgEP+jB7Nt5vZYfxRg8+Djt3BN7dlxuBEqC9z7K/CXwBlLuvpQHK/t+uXZTj3XBySVDaTMSxZbi704OQHfgvl+8jd937BKjNCN5HGZtdxpuCkt2tFwG/auRcX2fHuznsHddmyoGRAWrv4/A6CFV4I08dWuu62xPbjDHGmIDy03C6McYYY5rBirgxxhgTUFbEjTHGmICyIm6MMcYElBVxY4wxJqCsiBvjEyJygZyYgepzEdnjlg+KSIv/XayIFInI9GaeU3eK7SUSMTvfGWZqkdcx5rvCirgxPqHeU84yVTUT+DXwiFvOJGLinMa4B2EYY75DrIgbEwxxIvKkePMpLxeRjgDizV08T0TeB6aKyEAReUtEPnATRnRzx/1CTswJ/2LY6/Zzr/GJiIQmhwnNIb/Jff1jZBjxPCbeHMtvAt9v5Jg+IrI2bD1FRDa65X8XkXXu9Z8QEWnk/Gr3xDREZJCIlLnlRBFZICJrxZtUokkzqBnTFlkRNyYYeuNNj3kJcBC4MWxfe1UdBMwHHsWbr34g3qMf/9MdMxPIUm9Ckp+HndsHuBpvusX7RKSdiAwEbgEuw3tm++3u0bvhbgBS8Z6yNQm4IjKwqlYA7UWkl9uUD5S65cdUNVtV04COwPXNuBaz8B65fCneXPdzRCSxGecb02ZYETcmGHaoarlb/gBICdsXKoypeJOmvCEi5XgztnV3+zYAz4k3E93RsHN/r6rfqOqXwD7gB3gzRb2iqodVtQ54GW8mpnA/Al5Q1XpV/YwTU15GegmveEPDIj5URN5zPfNhwCWnvQInjARmuvdYhvc45ouacb4xbYZ9hmZMMHwTtlyP13sNOez+Fbxn8uc0cv51eIX374BZIpJ+itdt6Z8JpcBiEXkZUFXdKiIJwON4z4L/VESK8ApxpKOc6GiE7xfgRlWtbOGsxgSO9cSNaTsqga4ikgPghsYvcbMr9VDVFXhzqncBOn3L67wN/NjNGpWIN3T+dsQxK4F8EYlzn7sPbeyFVHU73i8H/8aJXnioIH8pIp3wJo9oTDUnpjAN//hgGVAY+hy9kaF+Y74zrCduTBuhqkfcn2fNF5EueP9/z8ObWWmR2ybAfFU92Mi9ZKHX+VBESvBmCgR4SlXXRxz2CiemVdxFw2kvI5UCc4Be7vUPisiTeDNSfY43XWNj/gN4WkRm4w2bh8x272uD+wVlB837TN2YNsNmMTPGGGMCyobTjTHGmICyIm6MMcYElBVxY4wxJqCsiBtjjDEBZUXcGGOMCSgr4sYYY0xAWRE3xhhjAsqKuDHGGBNQ/w8h4SLfZqcTNQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 576x432 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yjHnWtJepKAg"
      },
      "source": [
        "#**04. F1 스코어**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kYTNG5g5pP01"
      },
      "source": [
        "- F1 스코어는 정밀도와 재현율을 결합한 지표.\r\n",
        "- 공식은 생략"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3V8LwccfEMqR",
        "outputId": "0e8cea10-0d69-456b-c85f-e113a35f06d6"
      },
      "source": [
        "from sklearn.metrics import f1_score\r\n",
        "f1 = f1_score(y_test,pred)\r\n",
        "print('f1 스코어 :{0:4f}'.format(f1))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "f1 스코어 :0.780488\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Fz3-KRkepw1T",
        "outputId": "547007d0-37c9-464a-99b4-d4ffd80975f6"
      },
      "source": [
        "def get_clf_eval(y_test,pred):\r\n",
        "  confusion = confusion_matrix(y_test,pred)\r\n",
        "  accuracy = accuracy_score(y_test,pred)\r\n",
        "  precision = precision_score(y_test,pred)\r\n",
        "  recall = recall_score(y_test,pred)\r\n",
        "  f1 = f1_score(y_test,pred)\r\n",
        "  print('오차행렬\\n')\r\n",
        "  print(confusion)\r\n",
        "  print('정확도:{0:4f}, 정밀도:{1:4f}, 재현율:{2:4f}, f1 스코어:{3:4f}'.format(accuracy,precision,recall,f1))\r\n",
        "\r\n",
        "thresholds = [0.4,0.45,0.5,0.55,0.6]\r\n",
        "pred_proba = lr_clf.predict_proba(X_test)\r\n",
        "get_eval_by_threshold(y_test,pred_proba[:,1].reshape(-1,1),thresholds)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "임곗값: [[1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]]\n",
            "오차행렬\n",
            "\n",
            "[[98 20]\n",
            " [10 51]]\n",
            "정확도:0.832402, 정밀도:0.718310, 재현율:0.836066, f1 스코어:0.772727\n",
            "임곗값: [[1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]]\n",
            "오차행렬\n",
            "\n",
            "[[103  15]\n",
            " [ 12  49]]\n",
            "정확도:0.849162, 정밀도:0.765625, 재현율:0.803279, f1 스코어:0.784000\n",
            "임곗값: [[1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]]\n",
            "오차행렬\n",
            "\n",
            "[[104  14]\n",
            " [ 13  48]]\n",
            "정확도:0.849162, 정밀도:0.774194, 재현율:0.786885, f1 스코어:0.780488\n",
            "임곗값: [[0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]]\n",
            "오차행렬\n",
            "\n",
            "[[109   9]\n",
            " [ 15  46]]\n",
            "정확도:0.865922, 정밀도:0.836364, 재현율:0.754098, f1 스코어:0.793103\n",
            "임곗값: [[0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]]\n",
            "오차행렬\n",
            "\n",
            "[[112   6]\n",
            " [ 16  45]]\n",
            "정확도:0.877095, 정밀도:0.882353, 재현율:0.737705, f1 스코어:0.803571\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kGpkQutAr6bn"
      },
      "source": [
        "#**05. ROC 곡선과 AUC**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ISwJv_NHsPax"
      },
      "source": [
        "- ROC 곡선과 이에 기반한 AUC 스코어는 이진 분류의 예측 성능 측정에서 중요하게 사용되는 지표입니다.\r\n",
        "- 일반적으로 의학 분야에서 많이 사용됨\r\n",
        "- ROC 곡선은 X축인 FPR이 변할때 y축인 TPR이 어떻게 변하는지를 나타내는 곡선\r\n",
        "- TPR은 True Positive Rate의 약자로, 재현율(민감도)을 나타냄\r\n",
        "- 이에 대응하는 TNR은 특이성이라고 부른다.\r\n",
        "- **FPR = FP/(FP+TN) = 1-TNR = 1-특이성**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JTKvfFe8wLuL"
      },
      "source": [
        "- 사이킷런은 ROC곡선을 구하기위해 roc_curve() API를 제공.\r\n",
        "- 입력 파라미터 : y_true, y_score\r\n",
        "- 반환 값 : fpr, tpr, thresholds"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NL8nHw1qrBAS",
        "outputId": "3c7dc5ad-a316-4fce-86e6-c239c6ab4403"
      },
      "source": [
        "from sklearn.metrics import roc_curve\r\n",
        "\r\n",
        "pred_proba_class1 = lr_clf.predict_proba(X_test)[:,1]\r\n",
        "\r\n",
        "fpr, tpr, thresholds = roc_curve(y_test,pred_proba_class1)\r\n",
        "\r\n",
        "thr_index = np.arange(1,thresholds.shape[0],5)\r\n",
        "print('샘플 추출을 위한 임곗값 배열의 index:',thr_index)\r\n",
        "print('샘플 index로 추출한 임곗값:',np.round(thresholds[thr_index],3))\r\n",
        "print('샘플 index로 추출한 FPR:',np.round(fpr[thr_index],3))\r\n",
        "print('샘플 index로 추출한 TPR:',np.round(tpr[thr_index],3))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "샘플 추출을 위한 임곗값 배열의 index: [ 1  6 11 16 21 26 31 36 41 46 51]\n",
            "샘플 index로 추출한 임곗값: [0.965 0.654 0.63  0.563 0.454 0.4   0.348 0.148 0.125 0.112 0.108]\n",
            "샘플 index로 추출한 FPR: [0.    0.017 0.034 0.076 0.127 0.169 0.203 0.466 0.585 0.686 0.797]\n",
            "샘플 index로 추출한 TPR: [0.033 0.639 0.721 0.754 0.803 0.836 0.885 0.902 0.934 0.967 0.984]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        },
        "id": "j6niqmt0xwit",
        "outputId": "b8f07e06-68f3-41e9-82dd-6b37dc276483"
      },
      "source": [
        "def roc_curve_plot(y_test,pred_proba_c1):\r\n",
        "  fpr, tpr, thresholds = roc_curve(y_test,pred_proba_c1)\r\n",
        "  # ROC 곡선을 그래프 곡선으로 그림\r\n",
        "  plt.plot(fpr,tpr,label='ROC')\r\n",
        "  # 가운데 대각선 직선을 그림\r\n",
        "  plt.plot([0,1],[0,1],'k--',label='Random')\r\n",
        "\r\n",
        "  start, end = plt.xlim()\r\n",
        "  plt.xticks(np.round(np.arange(start,end,0.1),2))\r\n",
        "  plt.xlim(0,1);plt.ylim(0,1)\r\n",
        "  plt.xlabel('FPR(1-Sensitivity)');plt.ylabel('TPR(Recall)')\r\n",
        "  plt.legend()\r\n",
        "\r\n",
        "roc_curve_plot(y_test,pred_proba[:,1])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3hUBdbH8e9JKKHasCBdRCEQamgiTRTpoCBN6U0FcSm2F99XZV0Li7qiVGkuFhQswIpio0uXloQeWlAUkCKEAEnO+8cM2SEmJIGZuTOZ83mePMzcMveXmUvO3HauqCrGGGNCV5jTAYwxxjjLCoExxoQ4KwTGGBPirBAYY0yIs0JgjDEhzgqBMcaEOJ8VAhGZLiK/i0hMJuNFRMaJyG4R2SIiNX2VxRhjTOZ8uUUwE2hxmfEtgQrun4HARB9mMcYYkwmfFQJVXQb8cZlJ2gP/VpfVwLUiUtxXeYwxxmQsj4PLLgEc9Hie4B72a/oJRWQgrq0GChUqVKtixYp+CWiMyT1On0tm79EzTsfwu+STv5N67gykphxV1RszmsbJQpBtqjoFmAIQHR2t69evdziRMSbY/LDtN/q9v57pvaOpVLyo03F86mLrIBHh/WlTOHrkCG+89o/9mU3vZCE4BJTyeF7SPcwYY3ymWOH8FL+mgNMxfObQoUM89thjdOnShYcffphnhz8JwBuv/SPTeZwsBPOBISIyG6gLnFTVv+wWMsaYnBr2ySbifjl1ybDT55IdSuMfqsrUqVMZOXIkFy5coHXr1tme12eFQEQ+BpoAxUQkAXgByAugqpOAhUArYDeQCPTxVRZjTGj5auuvlLi2AHfeXOSS4Q1uv4EKNxXJZK7gtWfPHgYMGMDixYtp2rQp7733HuXLl8/2/D4rBKraLYvxCgz21fKNMaHt/sq38GzL0DixZOvWrWzYsIEpU6bQv39/RCRH8wfFwWJjjG8cOnGWfbnwTJrU1Nx/n5WYmBh+/vlnevbsSYcOHYiPj+eGG264oteyQmBMCOs9fS27fj/tdAyfKBKRO/+8nT9/nldeeYVXXnmFm2++mc6dOxMREXHFRQCsEBgT0hLPp9D4jhsZ3PR2p6N4VZhAVMlrnI7hdWvWrKFfv37ExsbyyCOP8NZbbxEREXHVr2uFwJgQd2OR/NQpd73TMUwWDh06RMOGDbn55pv5z3/+k6OzgrJihcCYELB81xH+vWo/6e9RfvT0OYcSmezauXMnd9xxByVKlOCTTz6hWbNmFC3q3QvirA21MbncjsN/MmjWBjYdPMGvJ5Mu+bn9psI0viPDrgPGYSdOnGDgwIFUrFiRZcuWAfDAAw94vQiAbREYk6udPHuBQbPWUyh/Hv7zxN3cXPTq9ycb35s/fz6PPfYYhw8f5qmnnqJ27do+XZ4VAmOCVOL5ZI7+ef6y07y4IJaE42eZPbCeFYEg0b9/f6ZNm0ZUVBTz5s0jOjra58u0QmBMkHpwwk9sP/xnltP9vUMVosvaweBA5tkkLjo6mjJlyvDMM8+QL18+vyzfCoExQerYmfPUu+16HqpVKtNpbiqan7tvL+bHVCanDh48yKOPPkrXrl3p0aMHjz76qN8zWCEwJoiVK1aYjrVKOh3DXIHU1FQmT57MM888Q0pKCg888IBjWawQGBNEVu4+yndxvwHwZ9IFh9OYK7Vr1y769+/PsmXLuPfee5kyZQrlypVzLI8VAmOCyIQlu1kd/weF8oWTP084USVy39WzoSAuLo4tW7Ywffp0evfuneMmcd5mhcCYIKIKNUtfy5xH73I6ismhzZs3s2nTJnr16kX79u2Jj4/nuuuuczoWYIUgpKS/qtQEH/sIg8+5c+d4+eWXee211yhevDhdunQhIiIiYIoAWCEIGQf/SKTFv5Zx5nyK01HMVaprfYGCxqpVq+jXrx/btm2jZ8+evPnmm15pEudtVghCxG+nkjhzPoUHa5Sg9A0FnY5jrkLDCnY6aDA4dOgQjRs35pZbbmHhwoW0bNnS6UiZskIQYh6oWYKGFay3jDG+sm3bNipVqkSJEiX49NNPadasGUWKBPbtMa0QBLFDJ86yKOYw2dltfPCPRJ/nMSaUHT9+nBEjRjBjxgyWLVtGw4YN6dChg9OxssUKQRCbujyeGSv3ZXv6PGHCTUUCb/+kMcHuiy++4PHHH+fIkSM899xzPm8S521WCIJYcopybcG8LH2qabamzxceRoF84T5OZUxo6du3LzNmzKB69ep89dVX1KxZ0+lIOWaFIMiFiXBNgbxOxzAmpHg2iatXrx4VKlRg5MiR5M0bnP8XrRAEgV9PnqX39HWcPpd8yfATiefJn9e+4RvjT/v372fQoEF0796dnj17MnDgQKcjXTUrBEFg79Ez7PjtTxrdcSM3Fs5/ybjqpazFgDH+kJqaysSJE3n22WdRVR566CGnI3mNFYIg8niT8tS77QanYxgTcnbs2EH//v1ZsWIFzZs3Z/LkyZQtW9bpWF5jhSDA7PrtT/YePXPJsB3ZuPmIMcZ3duzYQWxsLDNnzqRnz56ON4nzNisEAabHtLUcPpWU4biiEcF5IMqYYLRx40Y2bdpEnz59aNeuHfHx8Vx77bVOx/IJKwQB5uyFFNpULc6jjctfMrxw/jyULVbIoVTGhI6kpCRGjx7NmDFjKFGiBN26dSMiIiLXFgGwQhCQihXOTxXrM2+M361cuZJ+/fqxY8cO+vTpwxtvvBGQTeK8zQpBAPhg9X6+2vIrwF9OETXG+MehQ4do2rQpJUqUYNGiRTRv3tzpSH4T5nQAA19uPETMLydJSVVqlbmOeyre5HQkY0JGXFwcACVKlOCzzz5j69atIVUEwLYIAkbVktfwYf96TscwJmT88ccfDB8+nPfff5+lS5fSqFEj2rZt63QsR1gh8IP9x86QeJkbwiSeTyF/Xts4M8ZfPvvsMwYPHsyxY8cYNWoUderUcTqSo6wQ+NimgyfoMH5lltM1s91BxvhF7969ef/996lZsybffPMN1atXdzqS46wQ+NjJsxcAeOr+Oyl/Y+anf1YrlXtPTTPGaZ5N4u666y4qVarEiBEjyJPH/gSCjwuBiLQA3gbCgamq+lq68aWB94Fr3dM8q6oLfZnJKfVuu4FaZQLnZtXGhIq9e/cycOBAHnnkEXr16pUrmsR5m892TItIODAeaAlEAt1EJDLdZM8Dn6pqDaArMMFXefwpNVUZ8812np67manL452OY0xISklJYdy4cVSpUoXVq1enbRWYv/LlFkEdYLeqxgOIyGygPRDnMY0CRd2PrwF+8WEev/n1VBITluyhaEQeCuXPwx03F6b09XbDeGP8Zdu2bfTr149Vq1bRsmVLJk2aROnSpZ2OFbB8WQhKAAc9nicAddNN8yLwrYg8ARQC7s3ohURkIDAQCKoP8/nWkXSuXcrpGMaEnN27d7Njxw5mzZrFww8/nOuaxHmb00dKugEzVfUNEakPzBKRKqqa6jmRqk4BpgBER0f7dfsu6UIKyak5W2SiXR1sjN9t2LCBzZs307dvX9q2bcvevXspWrRo1jManxaCQ4Dn1+GS7mGe+gEtAFR1lYhEAMWA332YK9tiDp2kw/iVOS4EF4WH2bcQY3zt7NmzvPTSS4wdO5ZSpUrRvXt3IiIirAjkgC8LwTqggoiUw1UAugLd001zAGgGzBSRSkAEcMSHmXLk8MkkklOVPg3Kcus1BXI0b748YTSvfLOPkhljAJYtW0b//v3ZtWsX/fr1Y+zYsSHRJM7bfFYIVDVZRIYAi3CdGjpdVWNFZDSwXlXnAyOA90RkGK4Dx701AA/tP1ijJFElrRuoMYHk0KFDNGvWjFKlSvH999/TrFkzpyMFLZ8eI3BfE7Aw3bD/83gcBzTwZYacUlU+WnuAP06fZ8+R007HMcaks3XrVqKioihRogRffPEFTZs2pVAhu1fH1XD6YHHASTh+llFfxKQ9L5w/DzcVzX+ZOYwx/nD06FGGDRvGBx98kNYkrk2bNk7HyhWsEKST4j4wPPahanSofithIoTZQV9jHKOqzJkzhyFDhnD8+HFeeOEF6tZNfya6uRpWCDIRHgZ5wq0jqDFO69WrF7NmzSI6OpoffviBqKgopyPlOiFXCKYuj2f84t2Zjr+4RSDYVoAxTvFsEte4cWOqVq3K3/72N2sS5yMh965uTjhJcqryQI0SmU6TP08YDSsU82MqY8xF8fHxDBgwgEceeYQ+ffrQr18/pyPleiFXCABuLJyf0e2rOB3DGOMhJSWFd955h1GjRhEeHk7Pnj2djhQyQqIQnEq6wJIdR0hNVRKOJzodxxiTTlxcHH379mXNmjW0bt2aSZMmUbJkSadjhYyQKAQfrznAq19vT3sebfcFMCag7N27lz179vDRRx/RtWtXaxLnZyFRCM4lu3rYfT+8MeFhwi1F7RJ0Y5y2bt06Nm3axIABA2jdujXx8fEUKVLE6VghKaTOjyxXrBDlihWiQL5wp6MYE7ISExMZOXIk9erV49VXXyUpKQnAioCDcu0WwYFjiYycs5mk5BR+O5XkdBxjDLBkyRL69+/Pnj17GDRoEK+//ro1iQsAubYQxP5ykrX7/qB22euILF6UNlULW1toYxyUkJDAfffdR5kyZfjxxx9p2rSp05GMW64tBBf9vUMVKt5ifcmNccrmzZupVq0aJUuWZN68eTRp0oSCBe3WrYEkpI4RGGP858iRI3Tv3p3q1auzdOlSAFq1amVFIADl+i0CY4x/qSqzZ89m6NChnDx5kpdeeon69es7HctchhUCY4xX9ejRgw8//JC6desybdo0Kleu7HQkkwUrBMaYq5aamoqIICI0bdqUWrVqMXToUMLD7VTtYGDHCIwxV2X37t00a9aMGTNmANCvXz+GDRtmRSCIWCEwxlyR5ORkxo4dS1RUFBs3biRfvnxORzJXyHYNGWNyLCYmhj59+rB+/Xrat2/PhAkTuPXWW52OZa6QFQJjTI4dOHCA/fv3M3v2bDp37mxN4oKcFQJjTLasWbOGzZs3M3DgQFq1akV8fDyFCxd2OpbxAjtGYIy5rDNnzjB8+HDq16/PmDFjOHfuHIAVgVzECoExJlM//vgjVatW5a233uLRRx/l559/Jn/+/E7HMl6Wq3YNpaYqU5bHczzxPPFHzjgdx5iglpCQwP3330+5cuVYunQpjRo1cjqS8ZFcVQjij57hta+3kydMCA8Tbi6an5uKWItbY3Ji48aN1KhRg5IlS7JgwQIaN25MgQIFnI5lfChXFQJVBeBfXavTpqqdymZMTvz2228MHTqUTz/9lCVLltC4cWNatGjhdCzjB3aMwJgQp6p88MEHREZG8uWXX/Lyyy9z1113OR3L+FGu2CJoPW45u34/nbZFEGbnNBuTbd27d2f27NnUr1+fadOmUalSJacjGT/LFYUg9pdTRJe5juiy1xORN4wGtxdzOpIxAc2zSVzz5s2pX78+gwcPtv5AISpXFAKABrcXY9h9dzgdw5iAt3PnTgYMGEDPnj3p168fffr0cTqScZgdIzAmRCQnJzNmzBiqVavGli1b7EwgkybXbBEYYzK3ZcsW+vbty4YNG3jggQcYP348xYsXdzqWCRBZFgIRiQDaAA2BW4GzQAzwlarG+jaeMcYbEhISOHjwIHPmzKFjx47WJM5c4rK7hkTkJWAlUB9YA0wGPgWSgddE5DsRqXqZ+VuIyA4R2S0iz2YyTWcRiRORWBH56Ip/E2PMJX766ScmTZoEkNYkrlOnTlYEzF9ktUWwVlVfyGTcmyJyE1A6o5EiEg6MB+4DEoB1IjJfVeM8pqkAPAc0UNXj7tczxlyF06dPM2rUKN555x3Kly9Pnz59yJ8/P4UKFXI6mglQl90iUNWvshj/u6quz2R0HWC3qsar6nlgNtA+3TQDgPGqevzi62UvtjEmI99++y1VqlThnXfeYfDgwdYkzmTLZbcIRGQBoJmNV9V2l5m9BHDQ43kCUDfdNHe4l7MSCAdeVNVvMsgxEBgIULp0hhsgxoS8gwcP0rp1a8qXL8+yZcu4++67nY5kgkRWu4bG+mH5FYAmQElgmYhEqeoJz4lUdQowBSA6OjrTwmRMKNqwYQO1atWiVKlSLFy4kIYNGxIRYc0WTfZdthCo6tKreO1DQCmP5yXdwzwlAGtU9QKwV0R24ioM665iucaEhMOHD/PEE08wd+7ctCZx9913n9OxTBDKatfQVi6/ayjTM4Zw/TGvICLlcBWArkD3dNN8CXQDZohIMVy7iuKzkduYkKWq/Pvf/2bYsGEkJibyyiuvWJM4c1Wy2jXU5kpfWFWTRWQIsAjX/v/pqhorIqOB9ao63z2uuYjEASnAU6p67EqXaUwo6Nq1K59++ikNGjRg6tSpVKxY0elIJshltWto/9W8uKouBBamG/Z/Ho8VGO7+McZkwrNJXKtWrWjYsCGPP/44YWHWJcZcvWytRSJST0TWichpETkvIikicsrX4YwxsH37dho1asS0adMA6NWrF0OGDLEiYLwmu2vSu7j25e8CCgD9cV0sZozxkQsXLvDKK69QrVo14uLiKFy4sNORTC6V7a8UqrobCFfVFFWdAdg97IzxkU2bNlGnTh1GjRpFu3btiIuLo2vXrk7HMrlUdruPJopIPmCTiIwBfsVaWBvjM4cPH+bw4cN89tlnPPjgg07HMblcdv+Y93BPOwQ4g+v6gI6+CmVMKFqxYgUTJkwAoEWLFuzZs8eKgPGL7BaCo8B5VT2lqi8BTwG/+C6WMaHjzz//ZMiQITRs2JB//etfnDt3DoCCBQs6nMyEiuwWgh8Az7WyAPC99+MYE1oWLVpElSpVmDBhAk8++aQ1iTOOyO4xgghVPX3xiaqeFhH7umLMVTh48CBt2rTh9ttvZ8WKFXZ1sHFMdrcIzohIzYtPRKQWrjuVGWNyQFVZu3YtAKVKleLrr79m48aNVgSMo7JbCP4GzBGR5SKyAvgE14FjY0w2/frrr3Ts2JG6deuydKmrn+O9995rnUKN47K1a0hV14lIReBO96Ad7o6hxpgsqCozZ85k+PDhJCUl8frrr9OgQQOnYxmTJluFwH08YDhQRlUHiEgFEblTVf/j23jGBL/OnTszd+5cGjZsyNSpU7njjjucjmTMJbJ7sHgGsAHXTezB1VZ6DmCFwJgMpKSkICKEhYXRtm1b7rnnHgYNGmT9gUxAyu5aWV5VxwAXAFQ1ERCfpTImiG3bto2GDRumNYnr2bMnjz32mBUBE7Cyu2aeF5ECuG9SIyLlgXM+S2VMELpw4QIvv/wy1atXZ8eOHVxzzTVORzImW7K7a+gF4BuglIh8CDQAevsqlDHBZuPGjfTu3ZstW7bQpUsXxo0bx0033eR0LGOyJbtnDX0nIj8D9XDtEnqSS680Niak/fbbbxw9epQvv/yS9u3bOx3HmBzJshCISH2gBLBMVb8SkarAOKAhl96c3piQsmzZMrZu3crgwYNp0aIFu3fvpkCBAk7HMibHLnuMQET+CUzH1Wn0KxF5GfgWWANU8H08YwLPqVOnePzxx2ncuDHjxo1LaxJnRcAEq6y2CFoDNVQ1SUSuAw4CVVR1n8+TGROAFi5cyKBBg/jll18YPnw4o0ePtiZxJuhlVQiSVDUJQFWPi8iuQCkCe4+eYfuvdttk4z8HDx6kffv23HnnncydO5e6des6HckYr8iqENwmIvM9npfzfK6q7XwTK2tPfPwzMYf+WwiuKZDXqSgmF1NV1qxZQ7169ShVqhTffvstDRo0IF++fE5HM8ZrsioE6U9/eMNXQXIq6UIqDSsUY1TrSoSLUP5Gu7G38a5ffvmFxx57jPnz57NkyRIaN25M06ZNnY5ljNddthCo6lJ/BbkSRSPyUvGWok7HMLmMqjJt2jRGjhzJuXPnGDt2rDWJM7naZQuBiCwApgDfpO82KiK34bqobJ+qTvdZQmP8rFOnTnz++ec0btyYqVOncvvttzsdyRifymrX0ABcXUf/JSJ/AEeACKAssAd4V1Xn+TShMX7g2SSuQ4cONG/enAEDBlh/IBMSsto1dBh4GnhaRMoCxXHdmWynu/GcMUEvJiaG/v37069fPwYMGECPHj2cjmSMX2X7646q7lPVVaq6CUgSkYd9mMsYnzt//jwvvfQSNWvWZM+ePVx33XVORzLGEVldWVxURJ4TkXdFpLm4PAHEA539E9EY79uwYQO1atXixRdf5KGHHiIuLo5OnTo5HcsYR2R1jGAWcBxYBfQH/gdX07kO7i0DY4LSsWPHOHHiBAsWLKBNmzZOxzHGUVleUKaqUQAiMhX4FSh98WpjY4LJ4sWL2bp1K0OHDqV58+bs2rXLbhxvDFkfI0g7ZVRVU4AEKwIm2Jw8eZJBgwZxzz33MHHixLQmcVYEjHHJqhBUE5FTIvKniPwJVPV4bo1+TMBbsGABkZGRTJ06lZEjR7JhwwZrEmdMOlmdPhruryDGeNvBgwfp2LEjFStW5Msvv6R27dpORzImIGV11lCEiPzNfdbQQBHJ7q0tjXGEqvLTTz8BpDWJW79+vRUBYy4jq11D7wPRwFagFTlsOiciLURkh4jsFpFnLzNdRxFREYnOyesb4ykhIYF27drRoEEDli51tclq0qSJdQo1JgtZfcOP9DhraBqwNrsvLCLhwHjgPiABWCci81U1Lt10RXDdA3lNToIbc1FqairvvfceTz31FMnJybz55pvcfffdTscyJmjk5Kyh5By+dh1gt6rGq+p5YDZ/bWsN8HfgdcDORjJXpGPHjjz66KPUrl2bmJgYhg0bRni4Hd4yJruyKgTV3WcJnbqCs4ZK4Lq15UUJ7mFpRKQmUEpVv7rcC7mPT6wXkfVHjhzJYrEmFCQnJ5Oamgq4CsF7773H999/z2233eZwMmOCT1aFYLOqFnX/FFHVPB6Pr+pGACISBrwJjMhqWlWdoqrRqhp94403Xs1iTS6wZcsW6tevz3vvvQfAI488Qv/+/RERh5MZE5yyKgR6Fa99CCjl8byke9hFRYAqwBIR2QfUA+bbAWOTmXPnzvHCCy9Qq1Yt9u/fj30pMMY7sjpYfJOIDM9spKq+eZl51wEVRKQcrgLQFejuMe9JoNjF5yKyBBipquuzkduEmHXr1tG7d2/i4uLo0aMHb731FjfccIPTsYzJFbIqBOFAYVyN5nJEVZNFZAiwyP0601U1VkRGA+tVdX6O05qQdfz4cU6fPs3ChQtp2bKl03GMyVWyKgS/quroK31xVV0ILEw37P8ymbbJlS7H5E4//vgjW7du5cknn6R58+bs3LnT2kMY4wNZHSOwo2/G706cOMGAAQNo1qwZkydPTmsSZ0XAGN/IqhA080sKY9zmzZtHZGQk06dP5+mnn7Ymccb4QVZN5/7wVxBjDhw4wEMPPUSlSpWYP38+0dF2Apkx/pDtexYb4wuqyvLlywEoXbo033//PevWrbMiYIwfWSEwjjlw4ACtW7emUaNGaU3iGjVqZE3ijPEzKwTG71JTU5kwYQKVK1dm2bJljBs3zprEGeMgu7+A8bsHH3yQefPmcd999zFlyhTKli3rdCRjQpoVAuMXycnJhIWFERYWRpcuXWjfvj29e/e2/kDGBADbNWR8bvPmzdStW5cpU6YA0K1bN/r06WNFwJgAYYXA+ExSUhLPP/880dHRJCQkcMsttzgdyRiTAds1ZHxi7dq19OrVi+3bt9OrVy/efPNNrr/+eqdjGWMyYIXA+MSpU6c4e/Ys33zzDffff7/TcYwxl2GFwHjNt99+S2xsLMOGDePee+9lx44d1h7CmCBgxwjMVTt+/Dh9+vTh/vvvZ9q0adYkzpggY4XAXJXPP/+cyMhIZs2axXPPPcf69eutABgTZGzXkLliBw4coGvXrlSpUoWFCxdSo0YNpyMZY66AbRGYHFHVtL5ApUuX5scff2TNmjVWBIwJYlYITLbt37+fli1b0qRJk7RicPfdd5M3b16HkxljroYVApOl1NRU3n33XSpXrsyKFSt45513aNiwodOxjDFeYscITJY6dOjAggULuP/++5k8eTJlypRxOpIxxousEJgMXbhwgfDwcMLCwujWrRudOnWiR48e1h/ImFzIdg2Zv/j555+pU6cOkyZNAlxN4nr27GlFwJhcygqBSXP27Fmee+456tSpw+HDhylVqpTTkYwxfmC7hgwAq1evplevXuzcuZO+ffsyduxYrrvuOqdjGWP8wAqBAeDMmTNcuHCB7777jnvvvdfpOMYYP7JCEMK++eYbYmNjGTFiBM2aNWP79u1243hjQpAdIwhBx44do1evXrRs2ZL333+f8+fPA1gRMCZEWSEIIarK3LlziYyM5KOPPuL5559n3bp1VgCMCXG2ayiEHDhwgO7du1O1alW+/fZbqlWr5nQkY0wAsC2CXE5V+fHHHwEoU6YMS5YsYfXq1VYEjDFpgq4Q7Dj8J03+uZj9x844HSXg7d27l+bNm9OsWbO0JnF33XUXefLYhqAx5r+CrhAkpyrVSl1Lq6jidKtT2uk4ASklJYW3336bKlWqsGbNGiZOnGhN4owxmQq6r4b5wsN4u6v1vr+c9u3b89VXX9GqVSsmTZpkVwgbYy4r6AqByZhnk7gePXrQrVs3unfvbv2BjDFZ8umuIRFpISI7RGS3iDybwfjhIhInIltE5AcRsf7GV2D9+vVER0czceJEALp06cLDDz9sRcAYky0+KwQiEg6MB1oCkUA3EYlMN9lGIFpVqwJzgTG+ypMbnT17lmeeeYa6dety5MgRu0+AMeaK+HKLoA6wW1XjVfU8MBto7zmBqi5W1UT309VASR/myVVWrVpFtWrVGDNmDH379iUuLo42bdo4HcsYE4R8eYygBHDQ43kCUPcy0/cDvs5ohIgMBAYCFCxe3lv5gtrZs2dJTU3l+++/p1mzZk7HMcYEsYA4WCwijwDRQOOMxqvqFGAKwDWlKqofowWUhQsXEhsby1NPPcU999zDtm3b7Mbxxpir5stdQ4cAz/MWS7qHXUJE7gVGAe1U9ZwP8wSto0eP8sgjj9C6dWs+/PDDtCZxVgSMMd7gy0KwDqggIuVEJB/QFZjvOYGI1AAm4yoCv/Z+7IgAABOoSURBVPswS1BSVWbPnk2lSpX49NNPeeGFF1i7dq01iTPGeJXPdg2parKIDAEWAeHAdFWNFZHRwHpVnQ/8EygMzHGf6nhAVdv5KlOwOXDgAL169aJatWpMmzaNqKgopyMZY3IhUQ2uXe7XlKqoJw9udzqGz6gqP/zwQ9pdwlavXk3t2rUJDw93OJkxJpiJyAZVjc5oXEAcLDYue/bsYcCAASxevJglS5bQuHFj6tWr53QsYxx14cIFEhISSEpKcjpKUIiIiKBkyZI5OoZohSAAXGwS9/zzz5M3b14mT55sTeKMcUtISKBIkSKULVvWrpbPgqpy7NgxEhISKFeuXLbns0IQANq2bcvXX39NmzZtmDhxIiVL2nV1xlyUlJRkRSCbRIQbbriBI0eO5Gg+KwQOOX/+PHny5CEsLIzevXvTo0cPunbtaiu7MRmw/xfZdyXvVdDdjyA3WLt2LbVq1WLChAkAdO7cmW7dutnKboxxhBUCP0pMTGTEiBHUr1+f48ePU768tcswJhiEh4dTvXp1qlSpQtu2bTlx4kTauNjYWO655x7uvPNOKlSowN///nc8z8b8+uuviY6OJjIykho1ajBixAgnfoXLskLgJytWrCAqKoo333yTAQMGEBsbS8uWLZ2OZYzJhgIFCrBp0yZiYmK4/vrrGT9+PODq+dWuXTueffZZduzYwebNm/npp5/StvZjYmIYMmQIH3zwAXFxcaxfv57bb7/dyV8lQ3aMwE8u3jhm8eLFNGnSxOk4xgSllxbEEvfLKa++ZuStRXmhbeVsT1+/fn22bNkCwEcffUSDBg1o3rw5AAULFuTdd9+lSZMmDB48mDFjxjBq1CgqVqwIuLYsHnvsMa/m9wbbIvChBQsWMGaM6xYLTZs2JS4uzoqAMUEsJSWFH374gXbtXA0QYmNjqVWr1iXTlC9fntOnT3Pq1CliYmL+Mj4Q2RaBDxw5coQnn3ySjz/+mOrVq/O3v/2NfPnykSePvd3GXI2cfHP3prNnz1K9enUOHTpEpUqVuO+++xzJ4Su2ReBFqspHH31EpUqVmDt3LqNHj2bNmjXWJM6YIHfxGMH+/ftR1bRjBJGRkWzYsOGSaePj4ylcuDBFixalcuXKfxkfkFQ1qH6KlrxTA9W+ffs0X758WrduXY2JiXE6jjG5QlxcnNMRtFChQmmPf/75Zy1durReuHBBExMTtVy5cvrdd9+pqmpiYqK2bt1ax40bp6qqmzdv1vLly+uOHTtUVTUlJUUnTpzo87wZvWe4mn1m+HfVtgiuUmpqKosWLQKgTJkyLF++nJUrV1K5sjObsMYY36pRowZVq1bl448/pkCBAsybN4+XX36ZO++8k6ioKGrXrs2QIUMAqFq1Kv/617/o1q0blSpVokqVKsTHxzv8G/yVdR+9Crt27WLAgAEsXbqUpUuX0qhRI6cjGZPrbNu2jUqVKjkdI6hk9J5drvuobRFcgeTkZP75z39StWpVNm3axLRp06xJnDEmaNlpLFegTZs2LFq0iPbt2zNhwgRuvfVWpyMZY8wVs0KQTefOnSNv3ryEhYXRv39/+vbty0MPPWT9gYwxQc92DWXD6tWrqVmzZtopY506daJz585WBIwxuYIVgss4c+YMw4YN46677uLPP/+kQoUKTkcyxhivs11DmVi+fDm9evVi7969PP7447z66qsULVrU6VjGGON1VggykZycTN68ee20UGMM4eHhREVFkZycTLly5Zg1axbXXnvtVb/uzJkzWb9+Pe+++64XUl452zXk4csvv+TVV18FXE3iYmNjrQgYYzJtQ51b2BYB8Ntvv/HEE08wZ84catasyYgRI6xJnDEBKqMOvp07d+bxxx8nMTGRVq1a/WV879696d27N0ePHqVTp06XjFuyZEmOlu/Zhnrt2rU8+eSTJCUlUaBAAWbMmMGdd97JzJkzmT9/PomJiezZs4cHHnggrRPxjBkzePXVV7n22mupVq0a+fPnB2Dfvn307duXo0ePcuONNzJjxgxKly5N7969KVCgABs3buT3339n+vTp/Pvf/2bVqlXUrVuXmTNn5ih/RkJ6i0BVmTVrFpGRkcybN49//OMfrF692prEGWMylL4NdcWKFVm+fDkbN25k9OjR/M///E/atJs2beKTTz5h69atfPLJJxw8eJBff/2VF154gZUrV7JixQri4uLSpn/iiSfo1asXW7Zs4eGHH2bo0KFp444fP86qVat46623aNeuHcOGDSM2NpatW7eyadOmq/69Qvor74EDB+jfvz/R0dFMmzYt7eYRxpjAdblv8AULFrzs+GLFiuV4CwAyb0N98uRJevXqxa5duxARLly4kDZPs2bNuOaaawBXl9L9+/dz9OhRmjRpwo033ghAly5d2LlzJwCrVq3i888/B6BHjx48/fTTaa/Vtm1bRISoqChuvvlmoqKiAKhcuTL79u2jevXqOf6dPIXcFkFqaipff/014GoSt3LlSpYtW2ZFwBiTqczaUP/v//4vTZs2JSYmhgULFpCUlJQ2z8VdPuA62JycnHzFy7/4WmFhYZe8blhY2FW9btrrXPUrBJGdO3fSpEkTWrVqxdKlSwGIjo4mPDzc4WTGmGBQsGBBxo0bxxtvvEFycjInT56kRIkSANnaV1+3bl2WLl3KsWPHuHDhAnPmzEkbd9dddzF79mwAPvzwQ7/2LwuJQpCcnMzrr79O1apV2bp1KzNmzLCzgYwxV8SzDfXTTz/Nc889R40aNbL1zbx48eK8+OKL1K9fnwYNGlzSIfSdd95hxowZVK1alVmzZvH222/78te4REi0ob7//vv59ttvefDBBxk/fjy33HKLj9IZY7zN2lDnXE7bUOfag8VJSUnkzZuX8PBwBg4cyMCBA+nYsaPTsYwxJuDkyl1DK1eupHr16mkHdDp27GhFwBhjMpGrCsHp06cZOnQoDRs2JCkpyTYnjcklgm0XtpOu5L3KNYVg6dKlVKlShXfffZchQ4YQExOTdq6vMSZ4RUREcOzYMSsG2aCqHDt2jIiIiBzNl6uOERQsWJDly5fToEEDp6MYY7ykZMmSJCQkcOTIEaejBIWIiAhKliyZo3mC+qyhzz//nO3bt6dd1p2SkmLXBBhjTAYcu3m9iLQQkR0isltEns1gfH4R+cQ9fo2IlM3O6x4+fJhOnTrRsWNHvvjiC86fPw9gRcAYY66AzwqBiIQD44GWQCTQTUQi003WDziuqrcDbwGvZ/W658+cpFKlSvznP//h1Vdf5aeffrImccYYcxV8uUVQB9itqvGqeh6YDbRPN0174H3347lAM8niRsBJx3+jSpUqbN68mWeffZa8efN6PbgxxoQSXx4sLgEc9HieANTNbBpVTRaRk8ANwFHPiURkIDDQ/fT0ihUrdlxlk7hi6ZfhgEDIAIGRIxAyQGDkCIQMEBg5AiEDBEYOb2Qok9mIoDhrSFWnAFO89Xoisj6zgyb+EggZAiVHIGQIlByBkCFQcgRChkDJ4esMvtw1dAgo5fG8pHtYhtOISB7gGuCYDzMZY4xJx5eFYB1QQUTKiUg+oCswP90084Fe7sedgB812M5nNcaYIOezXUPuff5DgEVAODBdVWNFZDSwXlXnA9OAWSKyG/gDV7HwB6/tZroKgZABAiNHIGSAwMgRCBkgMHIEQgYIjBw+zRB0F5QZY4zxrlzTa8gYY8yVsUJgjDEhLlcVgittaSEiZUXkrIhscv9M8nGORiLys4gki0indONSPHKkP7juzQzDRSRORLaIyA8iUsZjnFcyZDPHoyKy1b2sFRevPvfmZ5JVBo/pOoqIiki0tzNkJ4eI9BaRIx7L6+8xzi/rhXuazu51I1ZEPvJ2huzkEJG3PJa1U0ROeDtHNjKUFpHFIrLR/f+klXu4v9eLMu7/o1tEZImIlPQY553PRFVzxQ+uA9J7gNuAfMBmIDLdNI8Dk9yPuwKfuB+XBWL8mKMsUBX4N9Ap3bjTfsrQFCjofvzYxffCWxlykKOox+N2wDfe/Eyyk8E9XRFgGbAaiHZovegNvJvJ/P5aLyoAG4Hr3M9vcmK9SDf9E7hONvH3ezEFeMz9OBLY59B6MQfo5X58DzDL259Jbtoi8ElLC1/kUNV9qroFSPXysnOSYbGqJrqfrsZ1nYcTOU55PC0EePvsheysFwB/x9XrKsnLy89pDl/KToYBwHhVPQ6gqr87lMNTN+BjBzIoUNT9+BrgFy9nyG6OSOBH9+PFGYy/armpEGTU0qJEZtOoajJwsaUFQDn3JuBSEWno4xyXEyEi60VktYh08FOGfsDXXs6Q7RwiMlhE9gBjgKEeo7zxmWSZQURqAqVU9asM5vf3etHRvQtgroh4XpDpr/XiDuAOEVnpXlYLL2fIbg7AtVsEKMd//xB6K0d2MrwIPCIiCcBCXFsmF/lzvdgMPOh+/ABQREQu/t3yymcSFC0m/OBXoLSqHhORWsCXIlI53bdVfymjqodE5DbgRxHZqqp7fLUwEXkEiAYaO5VBVccD40WkO/A8rosM/fKZiEgY8Cau3TLp+Xu9WAB8rKrnRGQQrq3Xe9zj/PWZ5MG1e6gJrq3EZSISpaon/JjBU1dgrqqmeAzzV45uwExVfUNE6uO65qkK/l8vRgLvikhvXLsvDwEX3w+vvBe5aYvgiltaqOo5VT0GoKobcO2zu8OHOTKlqofc/8YDS4AavsogIvcCo4B2qnrOyxmyncPDbKCDe9ne+kyyylAEqAIsEZF9QD1gvohE+3u9UNVjHp/DVKCWxzh/rRcJwHxVvaCqe4GduAqDU+tFV9LtFvLje9EP+NS9rFVABFDMgfXiF1V9UFVr4Pr/irswe+8z8caBhkD4wfVNJh7XZuTFgy6V000zmEsPFn/qfnwjEO5+fJv7g7jeVzk8pp2Jx8Fi4Dogv/txMWAXlzmIdpXvRQ1cK3CFdMO9kiEHOSp4PG6L66pzr30mOfk83NMv4b8Hi/26XgDFPR4/AKx2YL1oAbzvsayDuHaf+nW9cE9XEdiH+8JXB96Lr4He7seVcB0jEAfWi2JAmPvxP4DR3nwvVDX3FAL3m9EK1zeYPcAo97DRuL7xgquizwF2A2uB29zDOwKxwCbgZ6Ctj3PUxvXN6wyuJnux7uF3AVvdK8NWoJ8PM3wP/Ob+nTfh+hbo1QzZzPG2x3u/+OJ/Am9+JlllSDftEv5bCPy9XrzqXt5m93tR0YH1QnDtKotzL6urE+uF+/mLwGvp5vPnexEJrHQvaxPQ3KH1ohOuP/I7cW0pXvzj77X3wlpMGGNMiMtNxwiMMcZcASsExhgT4qwQGGNMiLNCYIwxIc4KgTHGhDgrBCagpeuuuMnd+bGJiJx0P98mIi+4p/Ucvl1ExqZ7rQ4i8n/ux5l2gE03z53ujo8Xl+XVO0WJq/tqT/fj3iJyq8e4qeLuxnqZ+X9y/1vWfWV2VstrI667BBqTxk4fNQFNRE6rauF0w5oAI1W1jYgUwnU+dxdcDcIuDi+Aq4tmP1Vd6Z7vJ1znZh8VVwvyorgu35+vqnMzWf4iYIKqznM/j1LVrT74VRGRJe78669g3ibuedtkMZ3gOve9gf636aAJcbZFYIKaqp4BNgC3pxt+FleBKAEgIncA51T1qHv8Ps1eB9jiuC7+u/i6W92vFy4i/xSRde4mcYPcw5u4tyDmurdKPnT/8UVEXpP/3gNirHvYiyIy0r1VEg186N76KOB+nWj3VsM/L2Zwbzm863582j34NaChe95hIrJMRKp7zLNCRKqp65vfEuCyBcOEFisEJtAV8Ngt9EX6ke4ujPVwXenpOfw6XD1ylrkHNcD1TTin3sLVzOtr9x/Ya93D+wEnVbU2rivFB4hIOfe4GsDfcF2ZehvQwJ3zAVxXTlcFXvZciHuLZD3wsKpWdxeyiz5zz3tRF1x9mTw9Cyx3z/sWMA13Iz13EYxQ1c3uadcDV9Mx0+QyVghMoDvr/uNWXVU9/xg2FJGNwLe42hDEegzfjKv/yyJVPeweXhw4ktOFq+oMXH1m5uDqyLlaRPIDzYGeIrIJWIOrH08F92xrVTVBVVNxbZWUxdXyPAmYJiIPAtneLaOqR4B4EannLigVcbU+uJw5QBsRyQv0xdXX6qLfgVszmsmEJmtDbYLV8kz2hy93HyMoh+uP9qequgk4i6vb7GWJyD+A1gCqWt397y/AdGC6iMTg6lYqwBOquijd/E2Acx6DUoA8qposInWAZrh6xwzhvy2ms2M20BnYDnyhWRzcU9VEEfkO101MOuPRyRRXz62zGc5oQpJtEZhcSV0tlF8DnnEP2ka64wiZzDfq4hYIpN1PNq/78S24vvkfAhYBj3mMu8N94DpDIlIYuEZVFwLDgGoZTPYnrrbYGfkC1x/1bvx1t1Bm804FxgHr1H3HMbc7gJjMsprQY4XA5GaTgEbuM4SWATU8DtzWFtedpx4CJotIbCav0RyIce9uWgQ85d7dNBVXh86f3VsJk7n8FnYR4D8isgVYAQzPYJqZwKSLB4s9R7j/kG/DdSOStRnMuwVIEZHNIjLMPc8G4BQwI920TYGM7sZmQpSdPmpChoi8DSxQ1e+dzuIP7msSluBqZ53qHnYz8JGqNnMymwkstkVgQskrQEGnQ/iD+yK1Nbj623ueIlsaGOFMKhOobIvAGGNCnG0RGGNMiLNCYIwxIc4KgTHGhDgrBMYYE+KsEBhjTIj7f6Jn9cNK0GquAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YFBrKQgf1IfL"
      },
      "source": [
        "- AUC값은 ROC곡선 밑의 면적을 구한 것으로서 일반적으로 1에 가까울수록 좋은 수치\r\n",
        "- AUC 수치가 커지려면 FPR이 작은 상태에서 얼마나 큰 TPR을 얻을 수 있느냐가 큰 관건이다.\r\n",
        "- 가운데 직선에서 멀어지고 왼쪽 상단 모서리 쪽으로 가파르게 곡선이 이동할수록 직사각형에 가까운 곡선이 되어 면적이 1에 가까워지는 좋은 ROC AUC 성능 수치를 얻게 된다.\r\n",
        "- 가운데 대각선 직선은 랜덤 수준의(동전 던지기 수준)이 진 분류 AUC값으로 0.5 입니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HBWV-1HZ3S8l"
      },
      "source": [
        "#**06. 피마 인디언 당뇨병 예측**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 197
        },
        "id": "lm_Pm7Tw0Vrh",
        "outputId": "4eeb6f6f-3dc3-4b46-8d04-b1fe84170698"
      },
      "source": [
        "import numpy as np\r\n",
        "import pandas as pd\r\n",
        "import matplotlib.pyplot as plt\r\n",
        "%matplotlib inline\r\n",
        "\r\n",
        "from sklearn.metrics import accuracy_score, precision_score, recall_score, confusion_matrix\r\n",
        "from sklearn.metrics import roc_curve, precision_recall_curve, f1_score\r\n",
        "from sklearn.model_selection import train_test_split\r\n",
        "from sklearn.preprocessing import StandardScaler\r\n",
        "from sklearn.linear_model import LogisticRegression\r\n",
        "\r\n",
        "diabetes = pd.read_csv('diabetes.csv')\r\n",
        "print(diabetes['Outcome'].value_counts())\r\n",
        "diabetes.head(3)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0    500\n",
            "1    268\n",
            "Name: Outcome, dtype: int64\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Pregnancies</th>\n",
              "      <th>Glucose</th>\n",
              "      <th>BloodPressure</th>\n",
              "      <th>SkinThickness</th>\n",
              "      <th>Insulin</th>\n",
              "      <th>BMI</th>\n",
              "      <th>DiabetesPedigreeFunction</th>\n",
              "      <th>Age</th>\n",
              "      <th>Outcome</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>6</td>\n",
              "      <td>148</td>\n",
              "      <td>72</td>\n",
              "      <td>35</td>\n",
              "      <td>0</td>\n",
              "      <td>33.6</td>\n",
              "      <td>0.627</td>\n",
              "      <td>50</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>85</td>\n",
              "      <td>66</td>\n",
              "      <td>29</td>\n",
              "      <td>0</td>\n",
              "      <td>26.6</td>\n",
              "      <td>0.351</td>\n",
              "      <td>31</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>8</td>\n",
              "      <td>183</td>\n",
              "      <td>64</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>23.3</td>\n",
              "      <td>0.672</td>\n",
              "      <td>32</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Pregnancies  Glucose  BloodPressure  ...  DiabetesPedigreeFunction  Age  Outcome\n",
              "0            6      148             72  ...                     0.627   50        1\n",
              "1            1       85             66  ...                     0.351   31        0\n",
              "2            8      183             64  ...                     0.672   32        1\n",
              "\n",
              "[3 rows x 9 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XsI6P5QHR0IT",
        "outputId": "c83cf02e-0c04-4313-98c3-bef62b3efab9"
      },
      "source": [
        "diabetes.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 768 entries, 0 to 767\n",
            "Data columns (total 9 columns):\n",
            " #   Column                    Non-Null Count  Dtype  \n",
            "---  ------                    --------------  -----  \n",
            " 0   Pregnancies               768 non-null    int64  \n",
            " 1   Glucose                   768 non-null    int64  \n",
            " 2   BloodPressure             768 non-null    int64  \n",
            " 3   SkinThickness             768 non-null    int64  \n",
            " 4   Insulin                   768 non-null    int64  \n",
            " 5   BMI                       768 non-null    float64\n",
            " 6   DiabetesPedigreeFunction  768 non-null    float64\n",
            " 7   Age                       768 non-null    int64  \n",
            " 8   Outcome                   768 non-null    int64  \n",
            "dtypes: float64(2), int64(7)\n",
            "memory usage: 54.1 KB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OfYJn5MhRzLN",
        "outputId": "8f97c57d-4c5b-442f-ca7e-eb66f66f6295"
      },
      "source": [
        "diabetes.isnull().sum()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Pregnancies                 0\n",
              "Glucose                     0\n",
              "BloodPressure               0\n",
              "SkinThickness               0\n",
              "Insulin                     0\n",
              "BMI                         0\n",
              "DiabetesPedigreeFunction    0\n",
              "Age                         0\n",
              "Outcome                     0\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a3NtgIngS2Ex",
        "outputId": "ec9c552e-b376-4ced-e40c-c53e5d2162bd"
      },
      "source": [
        "X = diabetes.iloc[:,:-1]\r\n",
        "y = diabetes.iloc[:,-1]\r\n",
        "\r\n",
        "X_train,X_test,y_train,y_test = train_test_split(X,y,test_size = 0.2, random_state = 156, stratify = y)\r\n",
        "\r\n",
        "# 로지스틱 회귀 학습, 예측 및 평가 수행\r\n",
        "lr_clf = LogisticRegression()\r\n",
        "lr_clf.fit(X_train,y_train)\r\n",
        "pred = lr_clf.predict(X_test)\r\n",
        "pred_proba = lr_clf.predict_proba(X_test)[:,1]\r\n",
        "\r\n",
        "get_clf_eval(y_test,pred)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "오차행렬\n",
            "\n",
            "[[88 12]\n",
            " [23 31]]\n",
            "정확도:0.772727, 정밀도:0.720930, 재현율:0.574074, f1 스코어:0.639175\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
            "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
            "\n",
            "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
            "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
            "Please also refer to the documentation for alternative solver options:\n",
            "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
            "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 388
        },
        "id": "RDGXIu7cUDM8",
        "outputId": "3f301e45-1e4f-487e-fe4e-b1b8e7295154"
      },
      "source": [
        "pred_proba_c1 = lr_clf.predict_proba(X_test)[:,1]\r\n",
        "precision_recall_curve_plot(y_test,pred_proba_c1)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfEAAAFzCAYAAAAuSjCuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3xUVf7/8ddJMumFdAIJhEDoCS10hFAFRFAEERF7+aqoK/a2ou66a/25Ku6qoGBZCy4qIAqKBKR3CL1DEjqB9Do5vz8mREpCJmEmdybzeT4e85jM3Dt33hPKZ865556jtNYIIYQQwvm4GR1ACCGEELUjRVwIIYRwUlLEhRBCCCclRVwIIYRwUlLEhRBCCCclRVwIIYRwUh5GB6ipsLAwHR4ejp+fn9FRqpWXl+fwOZ0hI0hOW3KGjCA5bc0ZcjpDRqj7nOvXrz+ltQ6vdKPW2qluXbp00YsXL9bOwBlyOkNGrSWnLTlDRq0lp605Q05nyKh13ecE1ukqaqJ0pwshhBBOSoq4EEII4aSkiAshhBBOyukGtgkhhHBMJSUlpKenU1hYWKvXBwUFsWPHDhunsj175fT29iY6OhqTyWT1a6SICyGEsIn09HQCAgKIjY1FKVXj1+fk5BAQEGCHZLZlj5xaa06fPk16ejrNmjWz+nXSnS6EEMImCgsLCQ0NrVUBd3VKKUJDQ2vciyFFXAghhM1IAa+92vzupIgLIYQQl7Fu3ToefvjhKrcfOXKEMWPG1GGiP8k5cSGEEC7FbDbj7u5u9f5JSUkkJSVVub1Ro0Z89913tohWY3ZriSulPlFKnVBKba1iu1JKvauU2quU2qKU6myvLEIIIVzDwYMHad26NRMmTKBNmzaMGTOG/Px8YmNjeeqpp+jcuTOzZs1i4cKF9OzZk86dOzN27Fhyc3MBWLt2Lb169aJDhw5069aNnJwcUlJSGDFiBABLliyhd+/edOzYkU6dOpGTk8PBgwdp3749YBkXcMcdd5CQkECnTp1YvHgxADNmzGD06NEMHTqU+Ph4nnzySZt8Xnu2xGcA7wOfVbF9GBBffusO/Lv8XgghRD0w7sOVlzw3IjGKiT1jKSg2c/unay7YZjabGdetKWOTYsjMK+b+L9ZfsP2b+3pa9b67du1i+vTp9O7dmzvvvJMPPvgAgNDQUDZs2MCpU6cYPXo0v/32G35+frz22mu8/fbbPP3004wbN45vvvmGrl27kp2djY+PzwXHfvPNN3nrrbcYPHgwubm5eHt7X7B96tSpKKVITU1l586dDBkyhN27dwOwadMmNm7ciJeXF61ateKhhx4iJibGqs9UFbu1xLXWS4HMy+wyCvisfGrYVUADpVSUvfJUqjAbtn1fp28phBDCvmJiYujduzcAt9xyC8uWLQNg3LhxAKxatYrt27dXtKhnzpzJoUOH2LVrF1FRUXTt2hWAwMBAPDwubOv27t2bZ555hnfffZezZ89esn3ZsmXccsstALRu3ZqmTZtWFPGBAwcSFBSEt7c3bdu25dChQ1f8WY08J94YSDvvcXr5c0cv3lEpdS9wL0BkZCS5ubmkpKRceYD0ecTv/Zi1B86S52/9dXnWslVOe3KGjCA5bckZMoLktLW6yBkUFEROTk7F42k3t690v3P7XLz93LnqnJwcTJVsP//YVTnXLX5u3/z8fMxmc8WCITk5OeTn55OcnMynn356wWu3bduG2Wy+5H3y8/MpLS0lJyeHBx98kKuuuopFixbRq1cvvv/+e7y8vCgrKyMnJ4fS0lLy8/MrjmE2m8nLy6OwsBClVMXzWmuys7Mvea/CwsIa/Tk5xcA2rfVHwEcASUlJ2t/fn+Tk5Cs/cH4ivDWTru67IPmOKz/eRVJSUmyT046cISNITltyhowgOW2tLnLu2LHjiiZBscUkKv7+/qSlpbF161Z69uzJDz/8QHJyMqmpqfj7+xMQEED//v15/PHHOX78OC1atCAvL4+MjAw6d+7MiRMn2LlzJ127diUnJwcfHx98fX3x8PAgICCAffv2kZiYSO/evdmyZQtpaWl07NgRNze3imN///33jBgxgt27d1ccd9euXXh6elZ8Pg8PD3x9fS/5vN7e3nTq1Mnqz2vkJWYZwPknA6LLn6s7viHQ+hrY8jWUFtXpWwshhLCPVq1aMXXqVNq0acOZM2e4//77L9geHh7OjBkzGD9+PImJifTs2ZOdO3fi6enJN998w0MPPUSHDh0YPHjwJZOvvPPOO3Tv3p3ExERMJhPDhg27YPsDDzxAWVkZCQkJjBs3jhkzZuDl5WW3z2pkS3wOMEkp9TWWAW1ZWutLutLtrtNEy3nxXfOh3fV1/vZCCCFsy8PDgy+++OKC5w4ePHjB4wEDBrB27dpLXtu1a1dWrVp1wXPJyckVvRjvvffeJT0GsbGxbN1quRDL29v7km56gNtvv53bb7+94vG8efNq8pGqZLcirpT6CkgGwpRS6cCLgAlAa/0fYD4wHNgL5AO278+2RlwyBEbDxi+kiAshhHAqdiviWuvx1WzXwIP2en+rublDx5th6RvwUfKl2/0iYOwM8PSt62RCCCFq6PxWsStwioFtdtf1Lji5E0ovmnjeXAJ7FsDGz6H7fcZkE0IIIaogRRwgoCGM+7zybZ8MhRXvQdKd4G79Gq9CCCGEvckCKNXp8yhkpUGqMfPiCiGEEFWRIl6d+CEQ0Q6WvwNlZUanEUIIISpIEa+OUpbW+MmdlkJ+dLPRiYQQQtShGTNmMGnSJACmTJnCu+++a3CiP0kRt0a76yEkDha9BB8PgNP7jE4khBCiGlpryup5D6oUcWu4e8Bdv8Ht88HNBEteNzqREEKIShw8eJBWrVpx66230r59e1555RW6du1KYmIiL774YsV+n332GYmJiXTo0IGJEycCMHfuXLp3706nTp0YNGgQx48fN+pjWE1Gp1vLLxT8ekO3eyyj1fs8ChGtjU4lhBCO6een4VhqjV7iYy61NJqq0jABhv2z2uPs2bOHmTNnkp2dzXfffceaNWvQWjNy5EiWLl1KaGgof/vb31ixYgVhYWFkZloW3OzTpw+rVq1CKcW0adN4/fXXeeutt2r0GeqaFPGa6v0XWPcJpPwDbpxpdBohhBAXadq0KT169ODxxx9n4cKFFQuK5ObmsmfPHjZv3szYsWMJCwsDICQkBID09HTGjRvH0aNHKS4uplkz269uaWtSxGvKLxR63G+Z4e1YquWboRBCiAtZ0WK+WIENVjED8PPzAyznxJ955hnuu+/Cybree++9Sl/30EMPMXnyZEaOHElKSgpTpky54iz2JkW8Nno+CKs/gs9Hg39klbsl5ebCDv+aHTuuH1z99ysMKIQQ4uqrr+aFF15gwoQJ+Pv7k5GRgclkYsCAAVx//fVMnjyZ0NBQMjMzCQkJISsri8aNGwMwc6Zz9LRKEa8Nn2C49v9B6v8uu1th6Sn8G4RZf9xjqbD1f1LEhRDCBoYMGcKOHTvo2bMnYFlr/IsvvqBdu3Y899xz9OvXD3d3dzp16sSMGTOYMmUKY8eOJTg4mAEDBnDgwAGDP0H1pIjXVvsbLLfL2JqSUrF8nVXmPwmbv76yXEII4cIuXgDlkUce4ZFHHrlkv9tuu43bbrvtgudGjRrFqFGjLtn3/GVEp0yZQk5Ojm1DXwG5xMyRePpBSZ7RKYQQQjgJKeKOxNMXykqhtNjoJEIIJ6a1ptRcvyc5ERbSne5ITJYRlWybDSY7rV/u5Q9x/S3TyQoh6qVxH65ix7FsUqdcbXQUYWdSxB1JYJTl/ns7r11+168Q082+7yGEMMyag5bJSwpLzHib3Ov0vbXWKGkk1IrWusavkSLuSNpeBw+uBbOdutPzT8Fnoyyj4KWIC1FvfXpHV+74dC2PzdrMv8Z1xMO9bs6cent7c/r0aUJDQ6WQ15DWmtOnT+Pt7V2j10kRdyRKQXhL+x1fa/D0t6zIJoSot/q3iuC54W34+/wdJDUN5o7edTPzWHR0NOnp6Zw8ebJWry8sLKxxETOCvXJ6e3sTHR1do9dIEXclSkF4Kzixw+gkQri0kzlFLNh2jGHtGxLq72Xz409dvJeIAC8+nNiFAa0jgNp11daUyWS6oqlKU1JSKqZIdWSOlFNGp7ua8DZwcpfRKYRwaR+k7OX5H7bS65+/8+ES2y9t/N36dJbuOcXV7RpicncjM6+Yf6wpZOPhMzZ/L2EsKeKuJqI15J2AT4fLpWxCGEBrza/bj5PUNJjRnaNpEmK5EuVMXjEpu07YpMVcXFqG53nnwU/nFnG6QDP63yt4fNZmjmUVXvF7CMcg3emupu0oOLgcdv8Mu+YDDYxOJIRL2Xksh/QzBUzq34KbujWpeP679en8ff4ORiRG8d74Tlc0MKyo1IyX6c8iHh8ZwMu9fdhc0pAZyw/y05aj3NM3jr8MjMfNTQagOTNpibuaBk3gpi8hsDFs+MzoNEK4nKahvvx7QmcGt71w8aTbesXyyMB45m05yuOztrB87ykKS8y1eo+ii1riAH4mxbPD27DosX4MaBPB9iPZFQX8ZE5R7T6MMJy0xF2Rmzt0ugWWvI532Dij0wjhUnw9PRiWEHXJ854ebvxlUDxZBSV8seoQ/9uQzksj23Fbr1iyCko4llVIy0h/61romgta4ueLCfFl6s2dKS61zOh2+HQ+/d5cTFLTYEZ2aMSwhCjC7DDYTtiHtMRdVadbAGh4bJHBQYRwHceyCpm6eC8ncio/J62UYsrIdmx6cQjTb0vi6nYNAVi04zhXv7OUSV9ttKp1nvrS1Tw9tPVl9/H0sPz37+vlzmODW5JVUMILP26j+6uLmDh9NftP5tbw0wkjSEvcVTVoAs0HEJX2m2X505oIbw2R7eyTS4h6bOH2Y7yxYBdD2zckIqDq/fy9PBjY5s/u9j7xYTwyMJ5/LdrD0bMFTLutKyF+npd9L2vPqYf5ezFpQDyTBsSz81g2czcf4eetxwj2tRx/ye6TlJVp+pdfqiYcixRxV9b1brz2jYfv7qzZ6/zC4fE9Mv+6EDX06/bjxIX70Tzcv0aviwjw5tHBLWnVMIBHv9nEuA9X8uXd3YkIvHTCkcISM89+n8p1HRvTt2V4jd6ndcNAWjcM5Imr/2zFf7nqEAu3H2feQ31o3zioRscT9idF3JW1Hs6q7h/SI6kGkxbs/AkWvQSn9th3djkh6pmsghJW7jvNXVfVfjKU4QlRBPt68vDXG8k4W1BpEc8vNjN7QwaJjYNqXMQr88bYDix99Tf+u+Ywr16fcMXHE7YlRdzFFfo0tMziZjVlKeJpq6SIC1EDKbtOUFqmGXLRqPSa6tk8lD+e7I+3yR2tNZ8sP8iojo0qBqMVlVrOmXvZaOGTIB8TIxIb8ePGDJ4d3gZ/LykbjkQGtomaCYsHnxA4vMroJEI4lYyzBUQFedMxJviKj3VuZbI9J3J5df4O+r6+mLd/3U1eUWnFqPOLLzG7EuO7NSGv2MycTUdsdkxhG1LERc0oBU16ShEXooYeSG7B0if7427DyVVaRgaw8NG+JLcK591Fexjzn5VknCkA/hx9bgudmzSgS9Ng2jYKtNkxhW1IERc116Q7ZO6DD3rC0c1GpxHC4WXmWaY4NtlhSdDm4f58MKELM+7oyqHTedw5cy1BPiZ8PW23jrhSiq/u6UHHGMsMj3L5meOQkxui5hLGwvHtsOVr2L8EojoYnUgIh5VfXMqgt5cwsUdTHh1sv3Ekya0i+PLu7gB0anLlXfYXO9eyn7P5CI9+s4m/X9f+gmljhTGkJS5qLrARjP4QvAIhK83oNEI4tP+uPkxmXrFNRopXp1OTYLsU8PMNaB1BnxZhPD07lbcX7qqTJU5F1aSIi9pr0ATOShEXoipFpWY+/mM/PeJC6NLUvsW1rvh7eTDttiTGJcXw7u97eXzWFkrMZUbHcllSxEXtBcXA2cNGpxDCYf1vfQbHs4t4sH8Lo6PYlMndjX/ekMCjg1ry46YM/rc+3ehILkvOiYvaa9AEDv4BW74F5QbNB4BviNGphHAYX64+RIfoIPq0CDM6is0ppXhkUDy394olyNdkdByXJUVc1F5kOyjOhdn3WB6HxsPt8yCgobG5hHAQ/72nBydziq5obXBHd66An8krJria+dyF7Ul3uqi9zrfCI5vhoQ1w87eQfQRmXgs5x41OJkSdm/bHfsb8ewU7j2WjtUZrTZCPiRYRNZsn3Rm9t2gPyW+mkF9canQUlyNFXNSeUhAcC6HNoeXVcMt3kJUBM0dIIRcu5x8/72TdoTPc9NEq/v7TDoa/u4yMswVGx6oTPZqHklVQwg8bZUa3uiZFXNhO014wYZYUcuFyikvLUMCw9g1pFOTDtGUHyCsqJTLAy+hodSKpaTDtGgUyY8UBueSsjkkRF7YV27u8kKdbutZzTxidSAi7O3Aqj9IyzdD2DZn9QC8eG9yS125IxMMOM7Q5IqUUt/WKZffxXFbuP210HJciA9uE7Z0r5F+OhfeTwLuBda/rOQm632vfbELYwc5j2QC0ahiAt8mdhwbGG5yo7o3s0Ih//ryTb9em0at5/RuN76ikiAv7iO0Dt/4I62dAmbn6/Q8sgW2zpYgLp7T7eA4eboq4sPo/iK0q3iZ33hvfiYToIKOjuBQp4sJ+YrpZbtaY8zDsnGffPELYyZ29m5HcKsKmK4c5o97l18OXmMvQ2rYrqYnKyW9YOIawlpB/GvLkfJpwPqH+XnSNlYmOAHIKS7jm3T/4aOk+o6O4BCniwjGEla/udGq3sTmEqKEtJ0uZMmcbxaUyfzhAgLfl2vj3F+8lLTPf6Dj1nhRx4RjCpYgL55NxtoCPthSxav9pzGVyadU5z1/TFoXilXnbjY5S70kRF44hKAa8g2DXfKOTCGGV4tIyHvxyA6Vl8O9buuDj6W50JIfRqIEPDw+MZ+H24yzeKZeZ2pNdi7hSaqhSapdSaq9S6ulKtjdRSi1WSm1USm1RSg23Zx7hwNzcofcjsPsXOLjc6DRCVOsfP+9gU9pZ7krwolmYn9FxHM5dfZoRF+7HV2tkpUN7slsRV0q5A1OBYUBbYLxSqu1Fuz0PfKu17gTcBHxgrzzCCXS/HwIawa9/BZn1STiwo1kFfL0mjTt6x9K1oVzkUxlPDzdm3tGNDyZ0NjpKvWbPlng3YK/Wer/Wuhj4Ghh10T4aCCz/OQiQiXddmacvDHgOMtbB9h+NTiNElaKCfJj3cB+eGdbG6CgOLSbEFw93N7IKSjiRXWh0nHrJnkW8MZB23uP08ufONwW4RSmVDswHHrJjHuEMOoyHiLaw6CUoLTY6jRAXKCg289OWowA0D/eX66CtUGouY+T7y3h1/g6jo9RLyl6T1SulxgBDtdZ3lz+eCHTXWk86b5/J5RneUkr1BKYD7bXWZRcd617gXoDIyMgu06ZNw9/f8WdGys3Ndficjpgx5PQ6ElNfYXf8/3Gk8TDAMXNWxhlyOkNGcMyc01OLWJZRysu9fYgJsBRwR8xZGSNzfri5kB2ZZbzT3/ey+8nvsnL9+/dfr7VOqnTjuXVvbX0DegILznv8DPDMRftsA2LOe7wfiLjccbt06aIXL16snYEz5HTIjGVlWv+nr9Yf9K54yiFzVsIZcjpDRq0dL+eiHcd006fm6bcW7LzgeUfLWRUjc07/Y79u+tQ8ffRswWX3k99l5YB1uoqaaM++oLVAvFKqmVLKE8vAtTkX7XMYGAiglGoDeAMn7ZhJOAOloNMtcDwVjm42Oo0QAGzNsCxy8uCAFgYncT4dYizzqW9JP2twkvrHbkVca10KTAIWADuwjELfppR6WSk1sny3x4B7lFKbga+A28u/dQhX1/4GcPeCjV8anUQIAE7lFhHkY8LLQ64Hr6m2UUG4uym2pGcZHaXeseu1EVrr+VgGrJ3/3F/P+3k70NueGYST8g2B1tdA6rcw5BWj0wiBm1LEyvXgteLj6c4bYxJp31hWOLM1ucBROK5OEyzLky56majTGtYfvHC7yRfaXAsmH0Piifpt74kcZq1P564+zYgI8GbKyHZGR3JqoztHGx2hXpIiLhxXXH8IiYOV79MKoLJp1VsNh3FfWGZ8E8KG5mw6wsdL93N3nzh+3JRBl6bBRAdffnS1qFp2YQnL9pyiW7MQwvy9jI5Tb0gRF47LzR3uXwEFZ1ixciW9eva8cPu272HBs/DL0zDsdcuAOCFsQGvNT6lH6d4sFE8PN174YSvZhaW8fkMiN3aNMTqeUzp8Op8HvtzAu+M7MbJDI6Pj1BsyU4FwbCYfCGxEsVcoBDa68NbzQeg5CdZ8BCunGp1U1CO7j+ey72QewxOjCPIxMbJjo/LncwxO5rxaNQzAy8ONLWkyQt2WpIgL5zb4FWg7ChY+B9t+MDqNqCd+2nIENwVD2zUEYHj7KAD8vKTzsrZM7m60bRQoI9RtTP5GCufm5gbXfwg5x2D2vRAQBU26G51KOLmSMs2A1pGEB1jO3fZqEcaXd3enc5Ngg5M5tw7RDfh2XRrmMo27m5z+sgUp4sL5mXzgpq9g+mCYeS34NLj8/m2uheFvyjl0USmtNU8Nbc3FU1b0bhFmUKL6IzE6iBkrDrL3RC6tGgYYHadekCIu6ge/UJj4Paz6AEovs1pS9lFYOw1ir4J219VdPuHQtNasPpDJp8sPEB7gxd+uS0DJlzybG9gmkiVPJNMkREb524oUcVF/BDeFYa9dfh9zKUwbAPOfgLh+4CPdo66ssMTMnE1H+HTFQXYczSbY18QdvZsZHaveCvIxEeRjMjpGvSID24RrcfeAke9B/mlY+LzRaYTB3liwiyf/twWtNa/dkMDKZwby8MB4o2PVa79tP84/f955yekKUTvSEheuJ6oD9HoIlr8DCWMhLtnoRKKGzhWAmnR5a63ZcPgMnyw/yB29YkmKDeG2nrEMahNJj7gQ6T6vI0ezC/nPkn0Ma9+QDjHVjF8R1ZIiLlxT8tOwYw7MfcRS0K3RpCdEytSbRlt/6AwPf7WR23vFck/fOIpKzRQUm2ng61np/kWlZn7acpQZKw6yJT2LAG8PBrWJICk2hCahvjQJlfOzdWlUx0a8+tMOvl57WIq4DUgRF67J5GPpVv9iDPz0mHWvie4Gd/9q31xOLruwhEBv+53z/H5jOk99l0oDX1PFaPFle05x92frSGwcxFXx4VwVH0anJsF4erihtWbU+8vZeSyH5uF+vHJde0Z3aizXexso0NvENYlRzNl0hOevaSt/FldIfnvCdcX2gSf2QElB9fsueA4OLLF/JieTX1zKhkNn6RMfxp7jOYx8fznjuzXhgf7NbTo/dlmZ5s2Fu/ggZR894kL494QuBPtZWt4tIvz5y8CW/LHnJP9eso/3F+8lKsibpU/2x+Tuxn394gj186JPizDc5NpkhzC+WwzfrU9n7uYj3NStidFxnJoUceHavAIst+qEtrAsi1pSCCZv++dyAhsPn2Hyt5s5craAZU8NIMDbxLUdopix4gBfrz3MP0YnMKpjY9u8V9pZPkjZx/huMbw0sj2eHn+OyW0a6scjg+J5ZFA82YUlrNp3mmV7T5GWmU9cuD/Xd5LVsxxN5ybBDGoTgbdJFi66UlLEhbBGg/JFL7IzILS5sVkMVmIu471Fe5iaso/IAC8+vaNrxcxmr4/pwP/1a85DX23ktZ93MiKx0RXNzFVYYsbb5E6XpsHMmdSbhMZBlx2AFuhtYki7hgwpny5VOCalFNNu62p0jHpBLjETwhpB5UU8K83YHAYrMZcx5j8reff3vYzq2IhfHu1Lr+YXzmQWF+7P/cnNOZZdyI6j2bV+r71nzCS/kcLyvacASIxuICPI65nCEjOpMpf6FZGWuBDWONcSP+vaRdzk7sbV7SK5v18cQ8sXBanM4LaRrHpmIBGBtTv1sGr/af65tpBGDXyJDJS1p+urv/64lV+2HmPNc4Oka72WpCUuhDUCGgHKZVviR7MK2Hj4DAAPJLe4bAEH8PJwJyLQm70ncpmx/ADbj2RTVmbd5B4nsguZ9N+NhPkofniwNy0iZI7t+uq6To3JLizl561HjY7itKQlLoQ1PDwtK6S5aEv8vd/38t26dFY+M4DQGow6X773FFPmbgcg0NuDpNgQujULYUL3JgRUcilaqbmMSV9tJK+olOe7eRPiV/m136J+6BkXSmyoL1+tTpMBiLVUbRFXSvkCjwFNtNb3KKXigVZa63l2TyeEI2kQ45It8YyzBcxal8a4rjE1KuAAt/WKZWCbCNYezGTNgUxWH8hk2Z5T3N4rFoBv16VxIruQO/s0w9fT8t9R19hgxneLIThrr60/inAwSilu7BrD67/sYu+JXKPjOCVrWuKfAuuBnuWPM4BZgBRx4VqCoiFjvdEp6sTczUfILSrlpq4xTF1sKaYPJLeo1bGig32JDvataGllFZRUnP9ceyCTWevTOZZdyMsj2+Ph7sYTV7cGICVFirgrGNMlmrcX7mbh9mO0NTqME7KmiDfXWo9TSo0H0FrnKxkiKlxRUAxsnwOr/gPuJmh/Q/VrlzshrTX/77fdNPAx0TLSn/+uPswtPZrQqIGPTY5//ipWb4ztQIC3iU9XHGDFvtO8fWNHOspUnC4lIsCbBY/2JS7MjyVL0o2O43SsGdhWrJTyATSAUqo5UGTXVEI4okadoKwEfnkKfpoMs+8FJ1+J6YlZmxn6zlLmbTlS8dyaA5nsP5nHzd2bopRiUJsIJvW338pek4e0pGGgN6dyigiVc+AuqXm4v1w+WEvWtMRfBH4BYpRSXwK9gdvtGUoIh9TuOmieBmWlsPFz+PWvkDoLEm80OlmtzE89yqz16TRu4MPJHMv38vziUj5I2UeAtwfXJETh4+lu90k5/L08mPV/PTGXaWJCZDESV/XOb7tZllpIcrLRSZxLtUVca/2rUmoD0ANQwCNa61N2TyaEI/IOtNz3nAQ75sLPT0Fcf/APNzZXDaw8UkrQ4TP89cetJDQO4vsHelW0guZsOsKS3Se5rWdTfDzr7rrd6GAp3gLWHTeTlpkvX+ZqoNrudKVUX6AdkANkA23LnxPCdbm5w8j3oTgXfn7S6DRWm7niIB9uKeL6D1ZwNqvc+q4AACAASURBVL+E125IxMPdrWJq1KHtG/Lm2A48OrilwUmFq7kxKQYFfLPW9a4AuRLWdKc/cd7P3kA3LKPVB9glkRDOIqI19H0SFv8NEsZA62uMTnRZv20/zktzt5EY5s5/HxrEuoOZtG0UeME+DXw9GdNFrtcVda9RAx8Swt2ZtT6NvwyKx8Nd5iKzhjXd6dee/1gpFQO8Y7dEQjiTPn+B7T/At7eCyY8+paWwspp/Vm5u0PcJ6Plg3WQEUtOzeOirjbRvHMSDrUvw9/IguVVEnb2/ENboF+3BexuLWLzrJIPbRhodxynUZsa2dKCNrYMI4ZTcTTDuC1g3HcylHEtPJzq6mpbs8a2w4FnwCYGO4+sk5rRl+wnx82TabUlsX7+qTt5TiJrqEO7OvX3jiAv3MzqK07Bmxrb3KL+8DMs59I7ABnuGEsKphDSDIX8DYG9KCtHVDa8tLYIvx8KcSeAXDvGD7B7xjTEdOJFTSESAN9vt/m5C1I6Hm+LZ4dJGrAlrTjqsw3IOfD2wEnhKa32LXVMJUZ95eFla7xFtLN3wNpgFrtRcxh97TpKWmV/pdk8PNxkBLpzG+kOZ/Lr9uNExnEK1RVxrPfO825da6+V1EUyIes07ECZ8B36h8OWNcHpfrQ6jteaXrccY8s5SJk5fw9Ozt1Rs25R2lqJSM9P+2E+/NxbbKrkQdvf2r7uZMmeb1SvfubIqu9OVUqn82Y1+wSZAa60T7ZZKCFcQ0BBu+R4+GQJf3ABd74ImvSC6i1Uv35x2lpfmbmPD4bM0D/fjXzd1JC7MH4Cz+cWM/mA53iZ3gnxMHM8utOcnEcKmburahIe+2sgfe0/Rr6XzzMFghMudEx9RZymEcFVhLeDmb+GL0bDweYjqAPctvexLtNYopTiRU0TG2QL+OTqBMV2iL7gkx8fTnY9vTSJl10kW7zpBbJgMFBLOY0i7SBr4mpi9IV2KeDWqLOJa60N1GUQIlxWdBI/vhXmPwp4FVe6272Qu/0nZR+NgH/4yqCWD2kRwVXz/ihXBzufl4c7ANpEMbBOJdvL53YXr8fJw5+q2Dfkp9SiFJeZK/44LC2tmbOuhlFqrlMpVShUrpcxKqey6CCeEy/DwtKxXnncSzCUVT2utWbX/NHfPXMvAt5bw4+YjlJotRVkpZdV/bkopWVxCOJ3hiVGY3BX7T+YZHcWhWXOd+PvATVjWEE8CbgVkTkYhbGzZcQ/6AOSegKDGALy+YBf/TtlHiJ8nDw+M59aeTQnz9zI0pxB1oU+LMNY8NwiTzNx2WVZN9qK13quUctdam4FPlVIbgWfsG00I1zJ9cyF9POGHP9bTvlsgLSICuCYhisYNfLihc3SdLkgihNHc3RTuKLTWaA1ubtKbVBlrini+UsoT2KSUeh04inXXlwshaqBZsxaQAcPW3onbBndwd6M90B5gkRUHcPeEfk9Bj/tBus9FPbD/ZC63fbqGF0e0Y5BMw1opa4r4RCxFexLwKBAD3GDPUEK4oufuHMu6r7bRxKeQiIBadJkf3wYLnoET2+Gaty3n2YVwYtHBvmTllzA/9agU8SpYU8S7AD9prbOBl+ycRwiXlFtUypm8Yjrf/HLtuw3LyiDlH7D0dTi9F2783KnWORfiYp4ebgxp15AFW49RVGrGy0NOKV3Mmm7xa4HdSqnPlVIjlFK1WTRFCHEZy/ee4qrXF7M5/WztD+LmBgOegzGfwJGN8PEAOLbVdiGFMMA1CVHkFJWybM8po6M4JGumXb0DaIFldPp4YJ9Sapq9gwnhSvaeyAUgPjLgyg/W/ga442coK4HpQ2DHvCs/phAG6d0ijEBvD35KPWp0FIdk7ej0EqXUz1imYfUBrgPutmcwIVzJvhO5RAV54+9lo46uxp3hnsXw9c3wzQToek/FZWsxhw/A2Tho0MQ27yWEHXl6uPHUsNY0CvIxOopDsmYp0mHAOCAZSAGmATfaNZUQLia7sJRgXxsPRAuMgjvmw9y/wNqPK55uDrDSF4b907bvJ4SdTOje1OgIDsuar/23At8A92mti+ycRwiXZJla0g5Xbpp8YPSHcO2/OLeeUcFbHfDJl/OLwrnsO5nL/pN5DJZR6heotohrrcfXRRAhXNldVzWrmE7VLkzeFT+WmALxyc+033sJYQdTF+/lt+3HWff8YDw9ZKqSc+Q3IYQBSsxl5BWVVjzu3yqizloYJaYAKJAiLpzLNQlRZBeWsnyf9CKdT4q4EHVgS/pZHvxyA9+tTwfgt+3H6f7qIp6ZncrmtLOs3n+aEzl1s+Z3qUcASEtcOJk+8WEEeHkwf4uMUj+fXPMthJ1orVm65xQfLtnHin2nCfD2oHtcCACxYX4Mbd+Q7zem89WawwA8ObQVDyS3sHuuYs9AOHEIXg6zPNG8P4z7AjxkYRXhuLw83BnUNpKF24/zqrlMFkYpV2URV0qlcm4kzEWbAK21Tqzu4EqpocC/AHdgmtb6kuGwSqkbgSnl77VZa32zddGFcGyTv93M9xszaBjozXPD23BTtxgCvE0AtIkK5M2xHfjrtW2Zs+kIv+04TnLLiDrJdaTRNcQ0awW6DAqzYN10mPsIXPdvmXNdOLThCVH8tOUou47l0L5xkNFxHMLlWuIjruTASil3YCowGEgH1iql5mitt5+3TzyW1dB6a63PKKXq5n8xIerAyI6N6Nk8lOs6Nq5yIE6gt4lbejTllh51dwlNgW8UJJ83XtU/ElJehdAW0PfxOsshRE31axnOuhcGEVj+ZVhcpohrrQ9d4bG7AXu11vsBlFJfA6OA7eftcw8wVWt9pvw9T1zhewrhMPq3cpLvpP2ehNN74PdXILQ5tLve6ERCVMrTw01Gpl9EaV35ZS1KqRwu350eeNkDKzUGGKq1vrv88USgu9Z60nn7/ADsBnpj6XKforX+pZJj3QvcCxAZGdll2rRp+Pv7W/HxjJWbm+vwOZ0hIzhXTuXlxwebCukW5UG/aMdrMVT2u3QzF9Nh8wv45+5nU8e/kxPY0qB0f3KmP3PJaRvWZDyRX8aHm4u4Pt6T9mHGLIhS17/L/v37r9daJ1W27XItcRtM4lwtDyAey2xw0cBSpVSC1vqCVSC01h8BHwEkJSVpf39/kpOT6yDelUlJSXH4nM6QEZwn54wfFzF9o+ZYluaxaxNJbu14E1NU+bvs3gmmDaDLrjeg2z2WiWK63nPBNeZ1yVn+zCWn7ViTsbDEzEurfiVdhTMpudqhWXbhSL9Lq/sllFIRSqkm525WvCQDy9rj50SXP3e+dGCO1rpEa30AS6s83tpMQtSVolIz+cWlVW7XWvPl6kP8bVUhZrPmm/t6MsABC/hl+YfDzbNAucHiv8PC52HL10anEuIC3iZ3BraJZMG2Y5SYy4yOY7hqi7hSaqRSag9wAFgCHAR+tuLYa4F4pVQzpZQncBMw56J9fsDSCkcpFQa0BPZbG16IunA8u5DBby/ljk/XVjyXV1TK+aeivlqTxnPfb6V1qDvzHr6Kzk2CjYh65SJaw+Qd8MIpCI2HLbOMTiTEJYYnRHEmv4RV+08bHcVw1rTEXwF6ALu11s2AgcCq6l6ktS4FJgELgB3At1rrbUqpl5VSI8t3WwCcVkptBxYDT2it5U9FOJSF249zODOf1QcyOXAqj9X7T9Plb79y3QcrWLnP8tf15u5NWPRYPyZ38SLEz8YLmdQ1NzdwN0HijXBoGWSlG51IiAsktwrHz9Od+bI8qVVFvKS8sLoppdy01ouBSk+wX0xrPV9r3VJr3Vxr/ffy5/6qtZ5T/rPWWk/WWrfVWidoraXvTjiciT2a8v0DvVAKftyUgQZaRPhzIruQ8R+v4vZP17D3RC7Nw/1xq0/XWSeMsdynfmdsDiEu4m1y54H+LejWLMToKIazpoifVUr5A0uBL5VS/wLy7BtLCONlF5awbM8psvJL6NQkmB7NQpm7+Qg94kKZO6kPix9P5tnhrVmx7zTv/b6HsjI7LmBihJA4iO4KqdKlLhzPg/1bcH2naKNjGM6aaVdHAQXAo8AEIAh42Z6hhHAE2zKyuWX6ar68uzu9W4Tx4si2NPCxdJUrpfA2uXNv3+aMS2rCydxC3NzqUSv8nIQb4ecnYEqD6vcNi4c7fgG/UPvnEgI4nVvEvpN5Lt0it6aIRwBHtdaFwEyllA8QCci5a1GvbUm3XOnYuIEPAK0bVj41QpCviSBfx7se3CY6TYDiHCipZnGWshJY8T78/CSMmV432YTLmzJ3Oyv2nmL1swPxcNG51K0p4rOAXuc9Npc/19UuiYRwANmFJfxnyT76tAgjNszP6DjG8fSDqx6zbl8PH8v0re2ugzbX2jeXEMDw9g2Zu/kIaw5k0qtFmNFxDGHNVxcPrXXxuQflPzv58FshLu/TZQc5k1/CU0NbGx3FeVw1GRomwLzJstSpqBPJrSLwMbnzkwuPUremiJ8875IwlFKjAFmVXdRr+07mkhgdREK0rJRkNXeTZSW0gkxLt7oQdubj6c6ANhEs2HYMc30bWGola4r4/wHPKqXSlFKHgaeA++wbSwhjvTu+E98/0NvoGM6nYQL0fdIyon3nT0anES5gePsoTuUWk5qRZXQUQ1RbxLXW+7TWPYA2QFutdS+t9V77RxOi7m08fIZX5+9Aa417fRxtXheumgwNmsCGz4xOIlzAwDYR/PFkfzrGWHEFRT1kzbSrkUqp6cAsrXWuUqqtUuquOsgmRJ3Jyi/h2e9TGf3vFfy4KYOTuUVGR3Je7iaI6gCZMoOysD9vkzsxIb5GxzCMNd3pM7BMj9qo/PFu4C/2CiSEPX24ZB/3frau4vIxgL0nchk5dRnfrE3jzt7NWPRYMhEBxqzcVW+ExMGZg1BmNjqJcAEZZwu497N1rDngegMqrSniYVrrb4EyqJgTXf5lCqc0b8tRFm4/zsj3l/Phkn2Umsu4e+Za8opK+fa+nrwwoi3+XtZceSkuKyQOzMWQffHChULYXgMfE0v3nGTu5iNGR6lz1vxvlaeUCgU0gFKqB+CaIwiEU9Naczy7kOEJDWndMJD+rSPwcHfj7XEdiQjwIjrYdbvkbC4kznKfud9yflwIO/Lz8iC5ZQS/bDvGlJHtXGo8izUt8clYlhBtrpRaDnwGPGTXVELYQV6xmTZRgfRpEc7DA+NpGRkAQOcmwVLAbe38Ii5EHRieGMXJnCLWHXStLvVqW+Ja6w1KqX5AK0ABu4Bu9g4mhC2dzCkiPMCLmXd2u2AdcGEnAY3A3UuKuKgzA1tH4OXhxvzUo3SPc535+6tsiSul3JVS45VSjwOttNbbgFhgCfB+HeUTotbKyjS/7zzOxOmr6fPa75wuH3Gu6tNyoY7KzQ1CmkHmAaOTCBfh5+XBxB5NaRrqWtMkX64lPh2IAdYA7ymljgBdgGe01j/URTghqlJWppm75QjtGwfRPNz/ku1zNx/h7V93c+BUHpGBXjw8MB6Th2sukGCYkDg4vc/oFMKFPD+irdER6tzlingSkKi1LlNKeQPHgOZaa1m9TBhuY9pZHvl6EwAdYxpwQ5doRiY2IsjXxP6TuTzy9UbaRAXy7vhODGvfEJOLrnBkqJA42Pc7lJVZWuZC1IHCEjOHM/MrxrzUd5f7l1WstT53WVkhsF8KuHAUXZoGM/uBXjwzrDUFxWZe+GErXf/+G3uO5xAX7s+Xd/fgxwd7M7JDIyngRglpBqWFkCmtcVF3nvhuCxOnr6bMReZSv1xLvLVSakv5zwrL6PQt5T9rrXWi3dMJl3BuoJm156oLis34eLrTuUkwnZsEc2/fOLYdyWbBtmMVXes9m7vOwBaH1bQ3uJlg2kAY8AIk3Qlu7kanEvXcoDYRzN18hA2Hz5AUG2J0HLu7XBFvU2cphMsqK9PM3pjBZysPMvXmzgCXnUIxt6iUmz5aSYfoBvz9+gTAUvzbNw6ifWNZccyhRLSB+5fD/Mcttw2fwTVvQ0xXo5OJemxgm0g8Pdz4KfWoaxdxrfWhugwiXIfWmk1pZ/lpy1Hmpx7lSFYhsaG+vDR3O4t2HufGLjE8dnXLS6Y+3ZqRxaT/buBwZj7/16+5QelFjYS3glvnwLbvYcGzMH0QdJoIg6aAX5jR6UQ95O/lQb+W4fyceowXrmmLWz2f+EXmlxR17mx+CWP+sxI3BX3jw3n86lYMbhtJWRm89/seZq48yLwtR/jrtW0Z17UJWmt+PVjCrF9XEOrvydf39qRbs/r/DbveUAraj4b4wbDkNVj1b9gxFwb+FbrcLl3swuauSYji1+3H2ZR+ls5Ngo2OY1dSxIVNaK2rPKddWGLms5UH2ZqRzbvjOxHs58knt3elY0wDgnxMF+z7/Ii2TOjRlBd+2MpT/0uloNjM8IQoZu8tpm/LCN4Y04FgP886+ETC5rwCYMjfoOMEmP8E/DQZNn4O17wFjbsYnU7UI4PaRvL9A71cYnlSKeLiih06nce4D1fh4+nOwNYRTB7SEl9Py1+to1kF3Pzxag6cyuOq+LCKQWn9WoZXebxmYX7MuKMr7/6+l2EJUUQEevNiTx/GDU+SiVrqg4g2cNtc2Po/WPAcfDwQ7vwFmvQwOpmoJ/y9POhUz1vg51xuxrZUpdSWqm51GVI4Hq01P27K4MCpPCZOX0NRqZnoYB9+3noMbw9L9+i3a9MY9+EqTuUU8fld3fj8ru74eFrXderh7sbkwS2JDLScF2/o5yYFvD5RChLGwIOrwd3T0r0uhA3tOZ7Diz9u5WROkdFR7OpyLfER5fcPlt9/Xn4/wX5xhLNYtT+Tt3/djcndjRM5hXx9b086xjSg1FxWMZDki9WHyCks4bO7urnMt2JRQz4NLC3w/SlGJxH1zNmCEmauPETfluEMbBNpdBy7qXZ0ulJqsNa603mbnlZKbQCetnc44Xh+Tj3K9GUHaNTAh+PZhcy6rxdpZ/Irzj15nDexyg8P9EaDSy0LKGohLhkWvQS5J8A/wug0op5oGxWIm4It6Vn1uohbM5WVUkr1Pu9BLytfJ+qhNlGBbEnPYs7mI4zs0IiE6CCGJ0RVuq+bm5ICLqoX189yf2CpsTlEveLn5UGLCH9SM7KMjmJX1gxsuwv4RCkVhGW2tjPAnXZNJRzSkbMFxIb58fKodry5cLdcqy1sI6ojeAfByqlwdHPF03FpaVC8qPLXKAWtR0CMrIosqpbQuAFL95y87NUzzs6a9cTXAx3Kizha6/r9tUZUau+JXIb/6w+mjGzHzd2bcGNSTL2fREHUETd36DAe1s+EEzsqnm5cZoajVQyELCuB5e9CjwdgwPPgWfUsf8J1JUYHsXzvKbILSgnyNVX/AidUbRFXSnkBN2BZS9zj3LcZrfXLdk0mDFdqLmPnsRzWHzrDV2sO4+PpzuC2lnNLUsCFTQ17zXI7zx8pKSQnJ1e+f1Eu/PYirJoKu3+B6z6QS9TEJSb2aMptvWKNjmFX1pzb/hEYBZQCeefdhJM5lVvE//t1N9mFJZVuLyo1s2T3STLzigH475rDjHhvGS/O2cbZ/BJeuyGR8ACvuowsROW8/C2TxNw219Iq/2Qo/PIsFOcbnUw4EFdobFhzTjxaaz3U7kmEXWmteXZ2Kgu3H2fWujReua49A9tEUlCqmZ96lF+2HuP3nSfILSrl3fGdGNmhEQNaRxDkYyIpNoTGDXyM/ghCXKpZX7h/5YWt8lFToWlPo5MJB/GPn3dwNq+E18bUz4U3rSniK5RSCVrrVLunEXbzx55TLNx+nGZhfhw4lUdWQQk5hSU8sSSf3JINhPh5MiIxiqvbNaR7nGVe8uhgX6KD5VyjcHDnWuVtR8GPk+DTYdDjfuj7hGUimSuh3OR8u5PLzC1m0c4T/LOeDm6zpoj3AW5XSh0AipD1xJ3SVfFhfHxrEv1ahrP9aDYdooM4mVvE8GYmRid3Ialp8AXXeAvhdJr1hftXWK45X/WB5WYL138IHW6yzbFEnUuMDmLW+nQyzhbUy0aJNUV8mN1TCLsxl2lWHzhNr+ZhFYPSzk3MEhHgzfA4T3rEhRoZUQjb8fKH4W9Awlg4vOrKj7d+huXLgBRxp5UQbfn/bmtGlmsW8fNmbosAvKvZXTgQrTWPfrOJOZuPMO+hPrRvHGR0JCHqRkw321xDbvKB+Y/DkY3QqFP1+wuH07phAB5uii3pWQxtX/nEVM6s2v5TpdRIpdQe4ACwBDgI/GznXMIGth3JZs7mIzyQ3FwKuBC1kTAWPHws17ALp+RtcmdEYlS9vbLGmu70V4AewG9a605Kqf7ALfaNJa7EiexC/vHzTn5KPYqnhxsTejQ1OpIQzsmnAbS7HlJnWdZC9/I3OpGohXduqr+9KNaMZCrRWp8G3JRSblrrxUCSnXOJK+Dv7cGaA5nc1DWGeQ/1kcvDhLgSXW6H4lzL+ufCaZWVaUrMZUbHsDlrivhZpZQ/sBT4Uin1L2SyF4d0PLuQolIzvp4eLH2yPy+Pak/LyACjYwnh3GK6QVAT2L/Y6CSilvadzCVhygIWbDtmdBSbs6aIjwLygUeBX4B9wLX2DCVq58EvN3DXjHVorWX1MCFsRSkIbwmZ+41OImopOtiHErOulyuaWTM6/VyruwyQ0R0OqqDYzMa0s9zbN65eTmgghKFC4iBtDWhtKerCqXh5uNM6KoDU9PpXxGV2j3qgsMTMnTPWorWmT4swo+MIUf+ExEFRNuSdMjqJqKWExkGkZmRRVqaNjmJTUsTrgWdnp7Jy/2neurEDvaWIC2F7Ic0t98c2Q8GZC24eJbl/Pi4tMjanqFJC4yByCks5lFm/Fsmx5hIz4eDc3RR/GRTP9Z2ijY4iRP0UWl7Ev7jhkk19AJaXP/ANg8k7wOMK52wXNtcjLpSHB8bjY6pijXonZc164r2BKUDT8v3PzZ0eZ99ooipZ+SX885edTOjehPaNg3h1dAImmfdcCPsJbQ43TIe8k5ds2rN3L/EtWsDBZbBzHpTkSxF3QLFhfkwe3NLoGDZnTUt8OpaR6esBs33jiKqUlWm2Hcnm67WHmb0hg4ISM/ER/rRvHCQFXIi6kDCm0qczClOI75EMbh6WIl5WWre5hNWyC0s4fDq/Xs1gaU0Rz9JayzSrBhv/8SpWH8jE08ON6zo24o7ezWgTFWh0LCHEOW7l3bRSxB3Wmwt2MXtDBlteHIJbPbkM15oivlgp9QYwG8tSpABorTfYLZUAYHPaWWLD/AjyMXFz9yaM6xpDcqsIQvykq04Ih+NmstybS4zNIaqU0DiIz1Ye4sDpPJqH148pdK0p4t3L78+falUDA6p7oVJqKPAvwB2YprX+ZxX73QB8B3TVWq+zIlO9Vlhi5l+L9vDR0v30ah7K53d1Z1THxkbHEkJcjlv5f6fSEndYieXLkqamZ7lOEdda96/NgZVS7sBUYDCQDqxVSs3RWm+/aL8A4BFgdW3ep75Jy8znvs/Xs/1oNmO6RPPMsNZGRxJCWCOo/Iv24VUQ0szYLKJSzcP98Da5sSU9i+s61Y+GkTVLkQYppd5WSq0rv72llLJmVEA3YK/Wer/Wuhj4GssUrhd7BXgNKKxR8noop7CEWz9ZQ9qZfD65PYk3x3Yg1L9+Lp8nRL0TexWEt4ZVUy0zuwmH4+HuRrtGQaRmnDU6is0oXc1fNqXU/4Ct/Dnl6kSgg9Z6dDWvGwMM1VrfXf54ItBdaz3pvH06A89prW9QSqUAj1fWna6Uuhe4FyAyMrLLtGnT8Pd3/K6Q3Nxcq3Pml2geWGSZhOCprt60Ca2baxlrktFIktN2nCEjOGfOqCMLabV7Kps6vMLZ4ESDk13IGX6fdZFx9xkznm4QG1T7/2Pr+nfZv3//9VrrylcP1Vpf9gZssua5SvYZg+U8+LnHE4H3z3vsBqQAseWPU4Ck6o7bpUsXvXjxYu0MrM2Zlpmnr5+6TDd/5ie9bM9J+4a6SH37XRrNGXI6Q0atnTRncYHWr8Vp/cVYw/JUxRl+n86QUeu6zwms01XURGsGthUopfporZdBxeQvBVa8LgOIOe9xdPlz5wQA7YGU8gU7GgJzlFIjtYsMbss4W8D7v+9h9oYMTO5u/OumTjJtqhDOzOQN3e6BlH9A+jpo0LTqfT39wNO37rIJAErMZfy89Rixob4VA92cmTVF/H5gZvl5cAVkArdb8bq1QLxSqhmW4n0TcPO5jVrrLKCiYl2uO72+KjWX8cPGI4zu3JgHklsQEyL/oIVwekl3wR9vw7SBl9/PKxAmrYOAyLrJJQBwU4pn/reFsUkxrlHEtdabgA5KqcDyx9nWHFhrXaqUmgQswHKJ2Sda621KqZexdA3MuYLcTq2wxEx+sZmmoX6se34Qfl4yhb0Q9YZ/OEycDSd2VL1PaRH8+gKsfA+G/K3usgnc3RTtGgWxJb1+DG6rsnoopW7RWn+hlJp80fMAaK3fru7gWuv5wPyLnvtrFfsmW5HX6S3acZynZ6eS3DKcN8Z2kAIuRH0U28dyu5yjm2HtJ9D7UfALrZtcAoCE6CC+WHWIUnMZHk4+bfXl0vuV3wdUcRM1lFdUyl9/3EagtwdjusiKY0K4tKsesyyWsmqq0UlcTmJ0EEWlZew5kWt0lCtWZTNQa/1h+f1LdRen/tpxNJunZ6dyNKuAmXd2o3ucfPMWwqVFtIa2I2H1R9DrIfAJNjqRy0goXwBl25Fsp1+DwprJXl5XSgUqpUxKqUVKqZNKqVvqIpyzKyz98xr8txbuJi0zn/dv7sxV8eEGphJCOIy+T0BxDqz+0OgkLiU21I9lT/Xnhs7OP2ubNScDhpQPZhsBHARaAE/YM5SzO5pVwINfbuCh3/NJy7RM4PL0sFb8/lg/hidEGZxOCOEwGiZAq2tg5QeQn2l0Gpfh5qaIDvatGOPlzKwp4ue63K8BZpVfGiYu4y9fb2LxrhMkx3jgXr7cXYuIABr4yupjQoiLDHgOirJh+TtGJ3Epaw9mMvnbQ94idQAAHpRJREFUTRSXlhkd5YpYU8TnKaV2Al2ARUqpcGSe8yrlFpWy+kAm9/aNY0IbLxo18DE6khDCkUW2g4Sxli717KNGp3EZx7IKmb0hg93Hc4yOckWqLeJa66eBXlimRC0B8qh8IRMBzNt8BIAO9WASASFEHen/jGUJ06WvG53EZSRGWwa3pWY4d+dylUVcKTWg/H40kAyMKv95KJaiLioR5u/Fo4Na0q+lDF4TQlgpJA463wYbPoPM/UancQlNQnwJ9PZgS7pzF/HLzTTSD/gduLaSbRqYbZdEdWzZnv/f3p2HV1He/R9/f8/JBmTBgARMABFRCARREIvWAq5YN6hUoaW1VuujVrvpU61brbZPXX5PrbZW26KP1qUuWCtalCJCtVYQRVnCLqIGBRd2MJDl/v0xEznEJBzgnDMzyed1XefKnJnJOZ/MnOSbmbnnvj+hsF1Wk93v1dTVs37rDroU5rX4Gi8sWsvLyz+mX7dCxg3twQnl6kZRRPbQ8J/CW4/Aq3fBqf8bdJpWz8wYWNYx8sOStnSf+M/9r+dlLk5mOee4deoSFqzeyBmHHUBuVowPNlRzcv8SvjXsQDZ+VsPQ/5nOyf1L+MHxffh0yw6WrNnE+KE9KMjLBrx/Ai74i9fd+0XDewf544hIlBV0hZ7D4L1ZQSdpMwZ178jMZR9RX++IxaLZUn23fX6a2f8AtzrnNvjP9wMud85dm+5w6WZmPHTBUdzxwnIenv0uhXnZXkM0/7aDTh1yuHhEbx6a9S5TK9cCkJMV49vDDmRzdQ2/+sdiHn/9fdrnxJl86TEc3EUd2YnIPigd7A2esmOrN8qZpNXlJx3CFScfGnSMfZJMx92nOOeubnjinFtvZl8FIl/EAQrzsrnutHKuPbXfF+4ZNDOuHNWXi4b35rE571GQl80pA7qSlx0nK2bMq9rIecf04tKRB7NfB90+JiL7qHQwuDr4cL53VC5p1RruE0+miMfNLNc5tx3AzNoBuemNlXkt7cyidtlc+JVdT5VnxWM8c+kxke88X0RC5IAjvK8fzFURz5DL/vomXQtzuebU8qCj7JVkKtDDePeHn29m5wPTgAfSGysaVMBFJKUKSqCwDN79T9BJ2owN23bw7xWfBh1jryVzn/gtwC+Bfv7jJuecbmYUEUmH8jNgybPw0m1BJ2kTBpYVsWztZqpr6oKOsleSHcx6MVDrnHvBzNqbWYFzLtrd3IiIhNGJN3n9qL/4S9ixDY6//vPGtpJ6FaUdqat3LP5wE4f3iN5IcsmMYvY9YBLQMMxOKfD3dIYSEWmz4lkw+m4YfB78+zfw/FVQH+3+vcMs6j23JXMk/n1gKDAbwDm33My6pDWViEhbFovBabdDdnuYdRfUbIPTfguxeNDJWp1uRXmM6t+VTh2i2V47mSK+3Tm3o6H1tpll4fXYJiIi6WIGJ/8Kctp718drPoPR93hH6pIyZsY93xocdIy9lsyn4V9mdjXQzsxOBC4BnklvLBERwQyOuxay28H0G2H/Q+Er/x10qlbpsx11ZMctcncdJZP2SuBjYAHwX8AUWklHLyIikXDs5dBrOLz5EDidCE21/6z4hAE3TOWt96PXj3qLRdzM4sBi59yfnXNfd86N9af1KRIRyaTDxsP6VfD+7KCTtDq9u+RTV+8iOaJZi0XcOVcHLDWzHhnKIyIiTel3utfQbd6jQSdpdUoK8+hSkMvCCLZQT+Z0+n5ApZlNN7PJDY90BxMRkQS5+dD3NKj8G9RUB52m1RlYVsT8CBbxZBq2XZf2FCIisnuHjYMFj8PyqVB+ZtBpWpWK0o5MX/IRW7bXkp8bnTsAmk1qZnnARcDBeI3a7nXO1WYqmIiINHLQCMjv6p1SVxFPqeP7daFDbpz6iDX5aunfjQeAGuBl4BSgHPhhJkKJiEgTYnGoGAuz74Gtn0KHTkEnajUGlBYxoLQo6Bh7rKVr4uXOuQnOuT8CY4FjM5RJRESac9h4qK+FhU8GnaTVWbOxmvlV0brNrKUiXtMwodPoIiIh0XUAlAyA+WqlnmrXP72QHz36VtAx9khLRfwwM9vkPzYDAxumzWxTpgKKiEgjh42D1W/AJ8uDTtKqDCwrYuUnW9lcXbP7lUOi2SLunIs75wr9R4FzLithujCTIUVEJEHF18Fi8PJvgk7SqlSUdQRg4eroHKdGq5NYERGBgq7w5R/DvEdg/uNBp2k1KkobhiWNznVxFXERkSgacTV0/xI8+2P49O2g07QKxR1yKO3YLlLdr6qIi4hEUTwLxt4L8Wx44lz14pYit58ziJ+e3DfoGElTERcRiaqiMhh9N6xZANPUuWYqDO1VTI9O7YOOkTQVcRGRKDv0FBh2Kbz2J1jyj6DTRN7m6hoemvUuy9ZuDjpKUlTERUSi7vifQ5f+MO16qK8LOk2k1dfDtX9fyPTFHwUdJSkq4iIiUZeVAyOuhE9XQOVTQaeJtKL22fTs1D4yLdRVxEVEWoO+p8P+/eCl27zDSdlrA0qLItNCXUVcRKQ1iMXgK1fAx0tg8eSg00TawNIiqtZ/xrqtO4KOslsq4iIirUX/MdCpj9dSfcfWoNNEVkWZ1+nL0jXhb9ymIi4i0lrE4t7R+Ib34JFzgk4TWUN6FjPv5ycxrHf4h3pVERcRaU0qzoYOXWDTB0EniaycrBhF7bKDjpEUFXERkdYkFoN+p0F1NFpXh9XzCz/kJ4+Hf1hSFXERkdYmryNUbwTngk4SWVXrP+Nvc1fz8ebtQUdpkYq4iEhrk1cE9bVq3LYPGkY0W7g63LeaqYiLiLQ27bxxsXVKfe/1Ly3CjNDfL64iLiLS2hT39r5WzQk2R4Tl52bRe//80PfcpiIuItLa9Dzaa6G+8Mmgk0TaUb2KyY6Hu0xmBR1ARERSLBb3On6Z+wBUb4K8wqATRdKvxlQEHWG30vovhpmNMrOlZrbCzK5qYvlPzGyRmc03s+lm1jOdeURE2owBX4Paalj6XNBJJI3SVsTNLA7cBZwClAPjzay80WpvAkOccwOBScCt6cojItKmlA2FwjKo/FvQSSKrrt4x+q5X+P2Ly4OO0qx0HokPBVY451Y653YAjwJnJq7gnJvhnNvmP50FlKUxj4hI2xGLwYAxsGI6bFwddJpIiseMbTtqeePd9UFHaVY6i3gp8H7C8yp/XnPOB3TeR0QkVY78HuDg37cHnSSyKko7smD1RlxIO86xdAUzs7HAKOfcBf7zbwFHOecubWLdCcClwHDn3Be6xzGzC4ELAUpKSgZPnDiR/Pz8tOROpS1btoQ+ZxQygnKmUhQygnKmyiFL76Lrmhd5seI3ZBWHu9lRGLflC+/W8NDiHfzv8HZ0aucd92Y658iRI99wzg1pcqFzLi0PYBgwNeH5z4CfNbHeCcBioEsyrzt48GA3Y8YMFwVRyBmFjM4pZypFIaNzypky69917hfFruqP44JOslth3JZvvLvO9bzyWff8wg8/n5fpnMDrrpmamM7T6XOAPmbWy8xygHHALiPVm9nhwB+BM5xzH6Uxi4hI29SxBxw+gW4f/hM2VgWdJnLKuxVyyoCuoR3VLG1F3DlXi3eKfCrekfbjzrlKM7vRzM7wV7sNyAeeMLO3zGxyMy8nIiJ769jLva8zbw42RwTlZce5e8JgvnRQOMcWT2tnL865KcCURvOuT5g+IZ3vLyIiQMcerC49je5vPggHDIIjLwg6UeR8smU7nTrkYGZBR9lFuPuTExGRlFh50LfhkFEw5b9hyZTdf4N87onX32fIL1+gav1nQUf5AhVxEZE2wMXiMPY+6HYYTPouVL0RdKTIOKSkAIAFIRyWVEVcRKStyOkA33gc8rvAI2fDupVBJ4qEvt0KyI6biriIiAQsvwtMeBJcHTw0FrZ+GnSi0MvNinNo1wIWhHBscRVxEZG2pnMfGP+od8vZPcfAvMegvj7oVKFWUdqR+VUbQtdzm4q4iEhb1ONLcN5zUNANnroQ7jtJ18lb8LUjSrn2tHJq61XERUQkDMoGwwXTYfTdsOE9mHgcPHUxbF4TdLLQOfLAYs4e0p3seLjKZrjSiIhIZsViMOgbcNkbcMyPYOEk+N1gePk3UFMddLpQWb52M/OrNgQdYxcq4iIiArkFcOIv4JJZ0Gs4TP8F/OEoWPwshOw6cFCueGIev56yJOgYu1ARFxGRnTr1hvGPwLeegqw8eOyb3u1odTVBJwtcRVkRCz/YSH2I/qlRERcRkS/qfRxc9AqM+Bks/yesejnoRIEbWNqRzdW1fLRNRVxERMIungVH/8A7Il/6fNBpAldRVgTAqo3huR1PRVxERJqX0x4OGgHLnmvz18b7dMknNyvGO5vqgo7yORVxERFp2SGjvFvQPlocdJJAZcVjPHzBUZx2UE7QUT6nIi4iIi07ZJT3ddlzweYIgSEHFlOQE57hSFXERUSkZYXd4IAjYNY9sLYy6DSBWrupmmdX7mD1hnAMS6oiLiIiuzf6DxCLw/+dAu++GnSawGz8rIZJy2p47Z1wDByjIi4iIrvXpR+c/0/osD88OBqWts1T6733zycnDvNDMqKZiriIiCSnYw/47lSvoD/6TXjz4aATZVw8ZvQsiIVmWFIVcRERSV6HznDuM9DrWHj6EnjljqATZdyBRTEqP9hEbV3w94uriIuIyJ7JLYBvPA79x8C062HmLUEnyqheRXHq6h1V64Nv3JYVdAAREYmgrFw4616wOLx0Kww4CzofHHSqjBhSEueKs48jJyv44+DgE4iISDTF4jDq1163rNNvCDpNxuTELRQFHFTERURkX+R3gWN+CIufgfdmBZ0mYx58dRVXP7Ug6Bgq4iIiso+GfR/yu8I/r2sz/au/t24bT75RRU3AjdtUxEVEZN/kdICRV0PVa7Do6aDTZMSA0iK219azfO2WQHOoiIuIyL47fAJ0KYdnfwyr5wadJu0GlnUEYMHqDYHmUBEXEZF9F4vDOQ9Bbj48cDqs/FfQidKqZ3F7CvKyAu+5TUVcRERSo1Nvr0e3ou7w8FivsVsrFYsZIw7tQrvseLA5An13ERFpXQoPgPOmQLfD4PFvw9wHg06UNr8bfzjXnlYeaAYVcRERSa32xfDtp+GgETD5UnjlzqATpZULsEW+iriIiKReTgcY/5jfNet18NL/CzpRyq3buoMRt83g0TnvB5ZBRVxERNIjK8frmrX/GJh5M6xbGXSilNqvfTbrt9Uwvyq4Fuoq4iIikj6xOIy6GeLZ8MINQadJKTOjorQo0BbqKuIiIpJeBV29rlkXPd3qumatKCti6ZrNVNfUBfL+KuIiIpJ+R18GBd1g6jWtqmvWgaVF1NY7lqzZHMj7q4iLiEj65XSA466F1a/DwieDTpMyg3p05Jwh3WmfE8z94iriIiKSGYeNh64V8PeLvcFSqoPt7SwVuhW145axAzmkpCCQ91cRFxGRzIjF4ZtPQsXX4T+/gzsPhzkToa426GT7pL7eUbV+WyDvrSIuIiKZU1ACo/8AF86E/fvBPy6Hu4+G5dOCTrbX7pi+nOG3zQykcZuKuIiIZN4Bg+A7z8K4R6C+Bh4ey8B5N8DaRUEn22PlBxRSV+9Y9OGmjL+3iriIiATDDPqeCpfMhpN/TcHmZXDPMfDMj2DLR0GnS9rAsiIAFgRwv3hWxt9RREQkUVYODLuE2VvK+HLtf2DOn2HBJG+M8v5joOxIiIX3mLNrYR6d83MD6fRFRVxEREKhNrsQTrwZjrwAZvwSXr8XZt8NhaVQPhr6j4bSIaEr6GbGwLIiFqzOfPerKuIiIhIunQ+Gr98P1Ztg2fNQ+ZR3dD7rLigs84p5/zFQOtg7JR8CF3y5F1t3ZL5hm4q4iIiEU14hDDzbe1RvhKXPeQV99h/h1d9DUXcoPxP6fw1Kjwi0oB99cOdA3ldFXEREwi+vCA4b5z0+29BEQe/hH6GPhgMyX9Cdc8xZtZ72OXEGlBZl7H1VxEVEJFradYRB473HZ+t3FvRZf4D/3Akde3in2w/9KpQMgNz8tEcyMy7761yO7t2Z288ZlPb3a6AiLiIi0dVuPxj0De+xbR0sneIV9Ffvglfu8NbZrxd0HQAlFf7XAV6hT/HRekVpx4yPLa4iLiIirUP7Yu+2tMMneAX9vVdhbSWsWQBrF8LiZ3aum1sIJf29gt5Q4Lv0g5z2e/32A8uKmL5kLVu215Kfm5nyqiIuIiKtT/tiryOZvqfunLd9C3y0yC/qlV5hn/dXmLPFW24xKO7tF/X+O4/cC0uTOmqvKCvCOahcvZGjDuqUph9sV2kt4mY2CrgDiAMTnXM3N1qeC/wFGAx8CpzjnFuVzkwiItJG5eZD96Heo0F9PWxYBWsWekV9bSWsnuudkm+Q19Ebfa3EK+75m7dDzZcgO2+Xl6/wG7QtaA1F3MziwF3AiUAVMMfMJjvnEjvGPR9Y75w72MzGAbcA56Qrk4iIyC5iMSg+yHuUn7FzfvVGrx/3tQt3HrnPfQBqtjEEYO5PoXOfXU7Hdy7pz98vOZpDuxZmLH46j8SHAiuccysBzOxR4EwgsYifCdzgT08Cfm9m5pxzacwlIiLSsrwi6DnMezSor4N171D54mP071TvFfj3ZsHCSZ+vMqh9J+g6ECY86Q29mmaWrnppZmOBUc65C/zn3wKOcs5dmrDOQn+dKv/52/46nzR6rQuBCwFKSkoGT5w4kfz89N8ysK+2bNkS+pxRyAjKmUpRyAjKmWpRyBmFjPDFnFk1W+iwdRX5W94hf8sqsmq3UjngqpS938iRI99wzg1palkkGrY55/4E/AlgyJAhLj8/nxEjRgQbKgkzZ84Mfc4oZATlTKUoZATlTLUo5IxCRkguZ8tLUyedvcivBronPC/z5zW5jpllAUV4DdxERERkN9JZxOcAfcysl5nlAOOAyY3WmQyc60+PBV7U9XAREZHkpO10unOu1swuBabi3WJ2n3Ou0sxuBF53zk0G7gUeNLMVwDq8Qi8iIiJJSOs1cefcFGBKo3nXJ0xXA19PZwYREZHWKlwjq4uIiEjSVMRFREQiSkVcREQkolTERUREIkpFXEREJKJUxEVERCJKRVxERCSiVMRFREQiSkVcREQkotI2FGm6mNnHwFbgk92tGwKdCX/OKGQE5UylKGQE5Uy1KOSMQkbIfM6ezrn9m1oQuSIOYGavNze2aphEIWcUMoJyplIUMoJyploUckYhI4Qrp06ni4iIRJSKuIiISERFtYj/KegASYpCzihkBOVMpShkBOVMtSjkjEJGCFHOSF4TFxERkegeiYuIiLR5oS3iZlZsZtPMbLn/db9m1jvXX2e5mZ2bMP95M5tnZpVmdo+ZxVOYbZSZLTWzFWZ2VRPLc83sMX/5bDM70J//TTN7K+FRb2aDUpVrL3J+xczmmlmtmY1NmD+yUc5qMxsdYM6fmNkiM5tvZtPNrGemcyaR8SIzW+Dn+LeZlfvzQ7XPE9Y7y8ycmQ0JY04z+46ZfZyQ5wJ/fmj2ub/O2f5ns9LMHsl0xmRymtntCVmWmdmGkObsYWYzzOxN/3f9q/78jH02k8jY0/8bNN/MZppZmT8/o9tyF865UD6AW4Gr/OmrgFuaWKcYWOl/3c+f3s9fVuh/NeBJYFyKcsWBt4GDgBxgHlDeaJ1LgHv86XHAY028TgXwdhq3XzI5DwQGAn8BxjbzOsXAOqB9gDlHNrw/cHEz2zNtOZPMWJgwfQbwfBj3ub9eAfASMAsYEsacwHeA3+/mdYLe532ANxP+5nTJZMY92ecJ618G3BfGnHjXmS/2p8uBVZn8bCaZ8QngXH/6OODBTG/Lxo/QHokDZwIP+NMPAE39V3MyMM05t845tx6YBowCcM5t8tfJwtshqbr4PxRY4Zxb6ZzbATzqZ20u+yTgeDOzRuuM9783XXab0zm3yjk3H6hv4XXGAs8557YFmHNGwvvPAsoynDOZjJsSnnag6c9b4PvcdxNwC1DdzOuEJefuBLrPge8Bd/l/e3DOfZThjMnmTDQe+GsT88OQ0wGF/nQR8EETr5POz2YyGcuBF/3pGU0sh/Rvy12EuYiXOOc+9KfXACVNrFMKvJ/wvMqfB4CZTQU+AjbjFdNUaPE9G6/jnKsFNgKdGq1zDk3/MqVKMjmTMY5w5TwfeK6J+enMmVRGM/u+mb2NdxbpB028TuD73MyOALo75/7RwusEntN3ln/acpKZdW9iedD7/BDgEDN7xcxmmdmoDGeEPfj98S9D9WJnEUoUhpw3ABPMrAqYgnfWoLF0fjaTyTgP+Jo/PQYoMLPGf9vTvS13EWgRN7MXzGxhE4/GRzmOvTiSds6dDHQDcvFOfYSCmR0FbHPOLQw6S0vMrBve6aupQWcBMLMJwBDgtkbzQ5HTOXeXc643cCVwbeKyMOxzM4sBvwEub2GdwHP6ngEOdM4NxDvD9kDiwpDs8yy8U+oj8I4Q/2xmHRsWhiRjonHAJOdcXeLMEOUcD9zvnCsDvgo86H9mgdB8Nq8AhpvZm8BwYDXw+fYMYltmZeqNmuKcO6G5ZWa21sy6Oec+9DdMU6eqVuP9AjUoA2Y2eo9qM3sa77THtH0O7b1n4lFBmT+vqXWqzCwL79TQpwnLM/GfWjI5d+ds4CnnXE3KUn1RUjnN7ATgGmC4c257o8Xpzrmn2/JR4O5G88KwzwuAAcBM/+pOV2CymZ3hnHs9RDlxziX+vkzEO7uRKAz7vAqY7Wd4x8yW4RX1ORnKmGzOBuOA7zcxPyw5z2fn5dBXzSwPr4/yhr/96f5sJvO5/AD/SNzM8oGznHMbElbJxLbcVSYuvO/NA+9oK7Fh261NrFMMvIPXqG0/f7oYyAe6+etkAY8Bl6YoVxZeA7pe7Gz80L/ROt9n14Ztjycsi+F9MA5K8/bbbc6Ede+niYZteNefRwadEzgcr8FJn2ZeI605k8zYJ2H6dOD1MO9zf/2ZJDRsC1POht9ff3oMMCuE+3wU8IA/3RnvVGynTGXck30O9AVW4fcNksltuQfb8zngO/50P7xr4g19maT9s5lkxs5AzJ/+FXBjprflF3Jn8s32cIN2AqYDy4EXgGJ//hBgYsJ63wVW+I/z/HkleP8NzwcWAr8DslKY7avAMrzCco0/70bgDH86D68V4wrgtcQPHt6Zg1mpyrKPOY/EO5rYinemoDLhew/0f2liIcj5ArAWeMt/TM50ziQy3gFU+vlmJP7yh2mfN1p3JrsW8dDkBH7tb895/vbsG8J9bniXJxYBC0i4AyZMvz/+8xuAm5v43tDkxGs09oq/z98CTsr0ZzOJjGPxatIyvDNEuUFsy8SHemwTERGJqDC3ThcREZEWqIiLiIhElIq4iIhIRKmIi4iIRJSKuIiISESpiIuEhJl1ShgFaY2ZrfanN5jZojS83w1mdsUefs+WZubfbwkj4e1DppS8jkhboSIuEhLOuU+dc4Occ4OAe4Db/elBtDxIDQB+74Ai0oaoiItEQ9zM/uyPW/1PM2sH4I9p/Fszex34oZkNNrN/mdkbZjbV77IYM/uB7RyTPXEUqHL/NVaa2eeDtpg3hnvDWAY/ahzGPL/3x15+AejSxDp9zey1hOcHmtkCf/p6M5vjv/6fmhjlDzNbZWad/ekhZjbTn+5gZveZ2WvmjT29NyOgibQKKuIi0dAHb9jL/sAG4KyEZTnOuSHAnXi9E451zg0G7sPrGhK8rosPd96AIhclfG9fvCF9hwI/N7NsMxsMnAccBXwJ+J6ZHd4ozxjgULxetr4NHN04sHNuCZBjZr38WefgdYEM3ljhRzrnBgDtgNP2YFtcA7zonBuKN9b8bWbWYQ++X6TVUBEXiYZ3nHNv+dNv4HXx2KChMB6KN8DJNDN7C28ktYax1+cDD/sjwdUmfO8/nHPbnXOf4A00UQJ8GW8Qh63OuS3A34BjG+X5CvBX51yd8waFaGp4S4DH8Yo37FrER5rZbP/I/Dig/263wE4nAVf5P+NMvG6Oe+zB94u0GrqGJhINiSO31eEdvTbY6n81vP7vhzXx/afiFd7TgWvMrKKZ103134THgCfM7G94owov90en+gNev+3vm9kNeIW4sVp2HmgkLje80aOWpjirSOToSFyk9VgK7G9mwwD8U+P9/TGZuzvnZuCNdV6EN9Jfc14GRptZe/809Rh/XqKXgHPMLO5fdx/Z1As5597G++fgOnYehTcU5E/84Ryba42+ChjsTydePpgKXNZwHb2JU/0ibYaOxEVaCefcDv/2rDvNrAjv9/u3eCMuPeTPM+BO59yGJtqSNbzOXDO7H28EPvBGDXyz0WpP4Z0GXwS8B7zaQrTH8IYW7uW//gYz+zPeCINr2Dn+dmO/AO41s5vwTps3uMn/ueb7/6C8w55dUxdpNTSKmYiISETpdLqIiEhEqYiLiIhElIq4iIhIRKmIi4iIRJSKuIiISESpiIuIiESUiriIiEhEqYiLiIhE1P8HWeZqHJZ1hvIAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 576x432 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 294
        },
        "id": "n53aZsg8V7Qe",
        "outputId": "f6304ee2-b26a-4685-dba2-f9b105b867d0"
      },
      "source": [
        "diabetes.describe()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Pregnancies</th>\n",
              "      <th>Glucose</th>\n",
              "      <th>BloodPressure</th>\n",
              "      <th>SkinThickness</th>\n",
              "      <th>Insulin</th>\n",
              "      <th>BMI</th>\n",
              "      <th>DiabetesPedigreeFunction</th>\n",
              "      <th>Age</th>\n",
              "      <th>Outcome</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>768.000000</td>\n",
              "      <td>768.000000</td>\n",
              "      <td>768.000000</td>\n",
              "      <td>768.000000</td>\n",
              "      <td>768.000000</td>\n",
              "      <td>768.000000</td>\n",
              "      <td>768.000000</td>\n",
              "      <td>768.000000</td>\n",
              "      <td>768.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>3.845052</td>\n",
              "      <td>120.894531</td>\n",
              "      <td>69.105469</td>\n",
              "      <td>20.536458</td>\n",
              "      <td>79.799479</td>\n",
              "      <td>31.992578</td>\n",
              "      <td>0.471876</td>\n",
              "      <td>33.240885</td>\n",
              "      <td>0.348958</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>3.369578</td>\n",
              "      <td>31.972618</td>\n",
              "      <td>19.355807</td>\n",
              "      <td>15.952218</td>\n",
              "      <td>115.244002</td>\n",
              "      <td>7.884160</td>\n",
              "      <td>0.331329</td>\n",
              "      <td>11.760232</td>\n",
              "      <td>0.476951</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.078000</td>\n",
              "      <td>21.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>1.000000</td>\n",
              "      <td>99.000000</td>\n",
              "      <td>62.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>27.300000</td>\n",
              "      <td>0.243750</td>\n",
              "      <td>24.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>3.000000</td>\n",
              "      <td>117.000000</td>\n",
              "      <td>72.000000</td>\n",
              "      <td>23.000000</td>\n",
              "      <td>30.500000</td>\n",
              "      <td>32.000000</td>\n",
              "      <td>0.372500</td>\n",
              "      <td>29.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>6.000000</td>\n",
              "      <td>140.250000</td>\n",
              "      <td>80.000000</td>\n",
              "      <td>32.000000</td>\n",
              "      <td>127.250000</td>\n",
              "      <td>36.600000</td>\n",
              "      <td>0.626250</td>\n",
              "      <td>41.000000</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>17.000000</td>\n",
              "      <td>199.000000</td>\n",
              "      <td>122.000000</td>\n",
              "      <td>99.000000</td>\n",
              "      <td>846.000000</td>\n",
              "      <td>67.100000</td>\n",
              "      <td>2.420000</td>\n",
              "      <td>81.000000</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       Pregnancies     Glucose  ...         Age     Outcome\n",
              "count   768.000000  768.000000  ...  768.000000  768.000000\n",
              "mean      3.845052  120.894531  ...   33.240885    0.348958\n",
              "std       3.369578   31.972618  ...   11.760232    0.476951\n",
              "min       0.000000    0.000000  ...   21.000000    0.000000\n",
              "25%       1.000000   99.000000  ...   24.000000    0.000000\n",
              "50%       3.000000  117.000000  ...   29.000000    0.000000\n",
              "75%       6.000000  140.250000  ...   41.000000    1.000000\n",
              "max      17.000000  199.000000  ...   81.000000    1.000000\n",
              "\n",
              "[8 rows x 9 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 58
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 340
        },
        "id": "FPHwgRfWWO6N",
        "outputId": "9654d019-a32b-481d-d5b7-014038582708"
      },
      "source": [
        "plt.hist(diabetes['Glucose'],bins=10)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([  5.,   0.,   4.,  32., 156., 211., 163.,  95.,  56.,  46.]),\n",
              " array([  0. ,  19.9,  39.8,  59.7,  79.6,  99.5, 119.4, 139.3, 159.2,\n",
              "        179.1, 199. ]),\n",
              " <a list of 10 Patch objects>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 59
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAR1UlEQVR4nO3dfYxldX3H8fenoP7hQwGZbjaAHTCrCW3ahUwoiWJsscqDddUmFGIUlXQ1gUSjjVk1UdPEBB/QxNRi1rABG0SwSCQBWykxEpOizuK6LAKy4BJ3s+yO2Kitxrrw7R/3jL2MMzsP92n25/uV3Nxzf+ece777u3c+e+Y35yFVhSSpLX8w6QIkScNnuEtSgwx3SWqQ4S5JDTLcJalBx0+6AICTTz65pqenJ12GJB1Tdu7c+ZOqmlps3roI9+npaWZnZyddhiQdU5I8vtQ8h2UkqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalB6+IMVWk9m952x0S2u+/qiyeyXbXBPXdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUoGXDPclpSb6R5AdJHkjyrq79pCR3JXmkez6xa0+SzyTZm2R3krNH/Y+QJD3TSvbcjwDvraozgXOBK5OcCWwD7q6qTcDd3WuAC4FN3WMrcO3Qq5YkHdWy4V5VB6vqvm76F8CDwCnAFuCGbrEbgNd301uAL1TPvcAJSTYOvXJJ0pJWNeaeZBo4C/g2sKGqDnazngA2dNOnAD/uW21/1yZJGpMVh3uS5wG3Au+uqp/3z6uqAmo1G06yNclsktm5ubnVrCpJWsaKwj3Js+gF+41V9ZWu+dD8cEv3fLhrPwCc1rf6qV3bM1TV9qqaqaqZqamptdYvSVrESo6WCXAd8GBVfapv1u3A5d305cBX+9rf0h01cy7ws77hG0nSGKzkqpAvA94M3J9kV9f2AeBq4JYkVwCPA5d08+4ELgL2Ar8E3jbUiiVJy1o23KvqW0CWmH3+IssXcOWAdUmSBuAZqpLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBq3kNns7khxOsqev7eYku7rHvvk7NCWZTvKrvnmfG2XxkqTFreQ2e9cD/wR8Yb6hqv5ufjrJNcDP+pZ/tKo2D6tASdLqreQ2e/ckmV5sXnfz7EuAvxpuWZKkQQw65n4ecKiqHulrOz3J95J8M8l5S62YZGuS2SSzc3NzA5YhSeo3aLhfBtzU9/og8KKqOgt4D/DFJC9YbMWq2l5VM1U1MzU1NWAZkqR+aw73JMcDbwRunm+rql9X1ZPd9E7gUeAlgxYpSVqdQfbcXwU8VFX75xuSTCU5rps+A9gEPDZYiZKk1VrJoZA3Af8JvDTJ/iRXdLMu5ZlDMgCvAHZ3h0b+K/DOqvrpMAuWJC1vJUfLXLZE+1sXabsVuHXwsiRJg/AMVUlqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktSglVwVUtIETG+7Y2Lb3nf1xRPbtobDPXdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIA+F1DFhkocFSsci99wlqUEruVnHjiSHk+zpa/tIkgNJdnWPi/rmvT/J3iQPJ3nNqAqXJC1tJXvu1wMXLNL+6ara3D3uBEhyJr07NP1Jt84/z992T5I0PsuGe1XdA6z0VnlbgC91N8r+EbAXOGeA+iRJazDImPtVSXZ3wzYndm2nAD/uW2Z/1yZJGqO1hvu1wIuBzcBB4JrVvkGSrUlmk8zOzc2tsQxJ0mLWFO5Vdaiqnqqqp4HP8/9DLweA0/oWPbVrW+w9tlfVTFXNTE1NraUMSdIS1hTuSTb2vXwDMH8kze3ApUmek+R0YBPwncFKlCSt1rInMSW5CXglcHKS/cCHgVcm2QwUsA94B0BVPZDkFuAHwBHgyqp6ajSlS5KWsmy4V9VlizRfd5TlPwp8dJCiJEmD8QxVSWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDlg33JDuSHE6yp6/tE0keSrI7yW1JTujap5P8Ksmu7vG5URYvSVrcSvbcrwcuWNB2F/CnVfVnwA+B9/fNe7SqNnePdw6nTEnSaiwb7lV1D/DTBW1fr6oj3ct7gVNHUJskaY2GMeb+duBrfa9PT/K9JN9Mct5SKyXZmmQ2yezc3NwQypAkzRso3JN8EDgC3Ng1HQReVFVnAe8BvpjkBYutW1Xbq2qmqmampqYGKUOStMCawz3JW4HXAm+qqgKoql9X1ZPd9E7gUeAlQ6hTkrQKawr3JBcA7wNeV1W/7GufSnJcN30GsAl4bBiFSpJW7vjlFkhyE/BK4OQk+4EP0zs65jnAXUkA7u2OjHkF8I9JfgM8Dbyzqn666BtLkkZm2XCvqssWab5uiWVvBW4dtChJ0mA8Q1WSGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGrSjck+xIcjjJnr62k5LcleSR7vnErj1JPpNkb5LdSc4eVfGSpMWtdM/9euCCBW3bgLurahNwd/ca4EJ6t9fbBGwFrh28TEnSaqwo3KvqHmDh7fK2ADd00zcAr+9r/0L13AuckGTjMIqVJK3MIGPuG6rqYDf9BLChmz4F+HHfcvu7NknSmAzlD6pVVUCtZp0kW5PMJpmdm5sbRhmSpM4g4X5ofrilez7ctR8ATutb7tSu7RmqantVzVTVzNTU1ABlSJIWGiTcbwcu76YvB77a1/6W7qiZc4Gf9Q3fSJLG4PiVLJTkJuCVwMlJ9gMfBq4GbklyBfA4cEm3+J3ARcBe4JfA24ZcsyRpGSsK96q6bIlZ5y+ybAFXDlKUJGkwKwp3Sb9fprfdMZHt7rv64olst0VefkCSGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGrfl67kleCtzc13QG8CHgBODvgfm7Xn+gqu5cc4WSpFVbc7hX1cPAZoAkx9G7CfZt9G6r9+mq+uRQKpQkrdqwhmXOBx6tqseH9H6SpAEMK9wvBW7qe31Vkt1JdiQ5cbEVkmxNMptkdm5ubrFFJElrNHC4J3k28Drgy13TtcCL6Q3ZHASuWWy9qtpeVTNVNTM1NTVoGZKkPsPYc78QuK+qDgFU1aGqeqqqngY+D5wzhG1IklZhGOF+GX1DMkk29s17A7BnCNuQJK3Cmo+WAUjyXOCvgXf0NX88yWaggH0L5kmSxmCgcK+q/wFeuKDtzQNVJEkamGeoSlKDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNWigq0JK0jBNb7tjItvdd/XFE9nuKLnnLkkNMtwlqUEDD8sk2Qf8AngKOFJVM0lOAm4GpundjemSqvqvQbclSVqZYe25/2VVba6qme71NuDuqtoE3N29liSNyaiGZbYAN3TTNwCvH9F2JEmLGEa4F/D1JDuTbO3aNlTVwW76CWDDwpWSbE0ym2R2bm5uCGVIkuYN41DIl1fVgSR/BNyV5KH+mVVVSWrhSlW1HdgOMDMz8zvzJUlrN/Cee1Ud6J4PA7cB5wCHkmwE6J4PD7odSdLKDRTuSZ6b5Pnz08CrgT3A7cDl3WKXA18dZDuSpNUZdFhmA3Bbkvn3+mJV/VuS7wK3JLkCeBy4ZMDtSNLITOrMWBjd2bEDhXtVPQb8+SLtTwLnD/LekqS18wxVSWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQcO4nrt+j0zyAkuSVs49d0lqkOEuSQ0y3CWpQWsO9ySnJflGkh8keSDJu7r2jyQ5kGRX97hoeOVKklZikD+oHgHeW1X3dbfa25nkrm7ep6vqk4OXJ0laizWHe1UdBA52079I8iBwyrAKkySt3VDG3JNMA2cB3+6arkqyO8mOJCcusc7WJLNJZufm5oZRhiSpM3C4J3kecCvw7qr6OXAt8GJgM709+2sWW6+qtlfVTFXNTE1NDVqGJKnPQOGe5Fn0gv3GqvoKQFUdqqqnqupp4PPAOYOXKUlajUGOlglwHfBgVX2qr31j32JvAPasvTxJ0loMcrTMy4A3A/cn2dW1fQC4LMlmoIB9wDsGqlCStGqDHC3zLSCLzLpz7eVIkobBM1QlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMGuZ77ujG97Y6JbHff1RdPZLuStJwmwv33zaT+M5N07BjZsEySC5I8nGRvkm2j2o4k6XeNJNyTHAd8FrgQOJPerffOHMW2JEm/a1R77ucAe6vqsar6X+BLwJYRbUuStMCoxtxPAX7c93o/8Bf9CyTZCmztXv53kocH2N7JwE8GWH9N8rFlF5lIXStgXauzXuuC9Vubda1QlyNrreuPl5oxsT+oVtV2YPsw3ivJbFXNDOO9hsm6Vse6Vm+91mZdqzOKukY1LHMAOK3v9aldmyRpDEYV7t8FNiU5PcmzgUuB20e0LUnSAiMZlqmqI0muAv4dOA7YUVUPjGJbnaEM74yAda2Oda3eeq3NulZn6HWlqob9npKkCfPaMpLUIMNdkhp0TIf7ernEQZLTknwjyQ+SPJDkXV37R5IcSLKre1w0ofr2Jbm/q2G2azspyV1JHumeTxxzTS/t65ddSX6e5N2T6LMkO5IcTrKnr23R/knPZ7rv3O4kZ4+5rk8keajb9m1JTujap5P8qq/fPjequo5S25KfXZL3d332cJLXjLmum/tq2pdkV9c+tj47SkaM7ntWVcfkg94fah8FzgCeDXwfOHNCtWwEzu6mnw/8kN5lFz4C/MM66Kt9wMkL2j4ObOumtwEfm/Bn+QS9EzLG3mfAK4CzgT3L9Q9wEfA1IMC5wLfHXNergeO76Y/11TXdv9yE+mzRz677Wfg+8Bzg9O7n9rhx1bVg/jXAh8bdZ0fJiJF9z47lPfd1c4mDqjpYVfd1078AHqR3lu56tgW4oZu+AXj9BGs5H3i0qh6fxMar6h7gpwual+qfLcAXqude4IQkG8dVV1V9vaqOdC/vpXcOydgt0WdL2QJ8qap+XVU/AvbS+/kda11JAlwC3DSKbR/NUTJiZN+zYzncF7vEwcQDNck0cBbw7a7pqu7Xqh3jHvroU8DXk+xM77IPABuq6mA3/QSwYTKlAb3zIPp/4NZDny3VP+vpe/d2ent3805P8r0k30xy3oRqWuyzWy99dh5wqKoe6Wsbe58tyIiRfc+O5XBfd5I8D7gVeHdV/Ry4FngxsBk4SO9Xwkl4eVWdTe8qnVcmeUX/zOr9HjiRY2LTO8ntdcCXu6b10me/Ncn+WUqSDwJHgBu7poPAi6rqLOA9wBeTvGDMZa27z26By3jmTsTY+2yRjPitYX/PjuVwX1eXOEjyLHof2o1V9RWAqjpUVU9V1dPA5xnRr6LLqaoD3fNh4LaujkPzv+Z1z4cnURu9/3Duq6pDXY3ros9Yun8m/r1L8lbgtcCbukCgG/J4spveSW9c+yXjrOson9166LPjgTcCN8+3jbvPFssIRvg9O5bDfd1c4qAby7sOeLCqPtXX3j9G9gZgz8J1x1Dbc5M8f36a3h/k9tDrq8u7xS4Hvjru2jrP2JtaD33WWap/bgfe0h3NcC7ws75fq0cuyQXA+4DXVdUv+9qn0ruPAknOADYBj42rrm67S312twOXJnlOktO72r4zztqAVwEPVdX++YZx9tlSGcEov2fj+EvxqB70/qL8Q3r/435wgnW8nN6vU7uBXd3jIuBfgPu79tuBjROo7Qx6Ryp8H3hgvp+AFwJ3A48A/wGcNIHangs8CfxhX9vY+4zefy4Hgd/QG9u8Yqn+oXf0wme779z9wMyY69pLbyx2/nv2uW7Zv+0+313AfcDfTKDPlvzsgA92ffYwcOE46+rarwfeuWDZsfXZUTJiZN8zLz8gSQ06lodlJElLMNwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSg/4PVggg6TurtBIAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V-r6y0-YW4xB",
        "outputId": "54b0496d-9e57-4402-f1b3-1adacf239206"
      },
      "source": [
        "# 0값을 검사할 피처 명 리스트\r\n",
        "zero_features = ['Glucose','BloodPressure','SkinThickness','Insulin','BMI']\r\n",
        "\r\n",
        "# 전체 데이터 건수\r\n",
        "total_count = diabetes['Glucose'].count()\r\n",
        "\r\n",
        "# 피처별로 반복하면서 데이터 값이 0인 데이터 건수를 추출하고 퍼센트 계산\r\n",
        "for feature in zero_features:\r\n",
        "  zero_count = diabetes[diabetes[feature]==0][feature].count()\r\n",
        "  print('{0} 0 건수는 {1}, 퍼센트는 {2:.2f}%'.format(feature, zero_count, 100*zero_count/total_count))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Glucose 0 건수는 5, 퍼센트는 0.65%\n",
            "BloodPressure 0 건수는 35, 퍼센트는 4.56%\n",
            "SkinThickness 0 건수는 227, 퍼센트는 29.56%\n",
            "Insulin 0 건수는 374, 퍼센트는 48.70%\n",
            "BMI 0 건수는 11, 퍼센트는 1.43%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8-bqI_ZqYM5w"
      },
      "source": [
        "# zero_feature 리스트 내부에 저장된 개별 피처들에 대해서 0 값을 평균값으로 대체\r\n",
        "mean_zero_features = diabetes[zero_features].mean()\r\n",
        "diabetes[zero_features] = diabetes[zero_features].replace(0, mean_zero_features)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vS2ybWseaVpp"
      },
      "source": [
        "- 로지스틱 회귀의 경우 일반적으로 숫자 데이터에 스케일링을 적용하는 것이 좋음"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1RaOemKQY2WS",
        "outputId": "eaee36b7-87dd-4580-a10a-181e364af0f1"
      },
      "source": [
        "X = diabetes.iloc[:,:-1]\r\n",
        "y = diabetes.iloc[:,-1]\r\n",
        "\r\n",
        "# standardScaler 클래스를 이용해 피처 데이터 세트에 일괄적으로 스케일링 적용\r\n",
        "scaler = StandardScaler()\r\n",
        "X_scaled = scaler.fit_transform(X)\r\n",
        "\r\n",
        "X_train,X_test,y_train,y_test = train_test_split(X_scaled,y,test_size = 0.2, random_state = 156, stratify = y)\r\n",
        "\r\n",
        "lr_clf = LogisticRegression()\r\n",
        "lr_clf.fit(X_train,y_train)\r\n",
        "pred = lr_clf.predict(X_test)\r\n",
        "pred_proba  = lr_clf.predict_proba(X_test)\r\n",
        "\r\n",
        "get_clf_eval(y_test,pred)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "오차행렬\n",
            "\n",
            "[[90 10]\n",
            " [21 33]]\n",
            "정확도:0.798701, 정밀도:0.767442, 재현율:0.611111, f1 스코어:0.680412\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hzVNzHyAbEFr",
        "outputId": "d2bde67f-3608-40ee-81fc-d53995c23073"
      },
      "source": [
        "# 0.3 부터 0.5 까지 0.03 씩 변화\r\n",
        "thresholds = [0.3, 0.33, 0.36, 0.39, 0.42, 0.45, 0.48, 0.50]\r\n",
        "pred_proba = lr_clf.predict_proba(X_test)\r\n",
        "get_eval_by_threshold(y_test,pred_proba[:,1].reshape(-1,1),thresholds)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "임곗값: [[0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]]\n",
            "오차행렬\n",
            "\n",
            "[[67 33]\n",
            " [11 43]]\n",
            "정확도:0.714286, 정밀도:0.565789, 재현율:0.796296, f1 스코어:0.661538\n",
            "임곗값: [[0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]]\n",
            "오차행렬\n",
            "\n",
            "[[72 28]\n",
            " [12 42]]\n",
            "정확도:0.740260, 정밀도:0.600000, 재현율:0.777778, f1 스코어:0.677419\n",
            "임곗값: [[0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]]\n",
            "오차행렬\n",
            "\n",
            "[[76 24]\n",
            " [15 39]]\n",
            "정확도:0.746753, 정밀도:0.619048, 재현율:0.722222, f1 스코어:0.666667\n",
            "임곗값: [[0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]]\n",
            "오차행렬\n",
            "\n",
            "[[78 22]\n",
            " [16 38]]\n",
            "정확도:0.753247, 정밀도:0.633333, 재현율:0.703704, f1 스코어:0.666667\n",
            "임곗값: [[0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]]\n",
            "오차행렬\n",
            "\n",
            "[[84 16]\n",
            " [18 36]]\n",
            "정확도:0.779221, 정밀도:0.692308, 재현율:0.666667, f1 스코어:0.679245\n",
            "임곗값: [[0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]]\n",
            "오차행렬\n",
            "\n",
            "[[85 15]\n",
            " [18 36]]\n",
            "정확도:0.785714, 정밀도:0.705882, 재현율:0.666667, f1 스코어:0.685714\n",
            "임곗값: [[0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]]\n",
            "오차행렬\n",
            "\n",
            "[[88 12]\n",
            " [19 35]]\n",
            "정확도:0.798701, 정밀도:0.744681, 재현율:0.648148, f1 스코어:0.693069\n",
            "임곗값: [[0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [1.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]\n",
            " [0.]\n",
            " [0.]\n",
            " [1.]\n",
            " [0.]]\n",
            "오차행렬\n",
            "\n",
            "[[90 10]\n",
            " [21 33]]\n",
            "정확도:0.798701, 정밀도:0.767442, 재현율:0.611111, f1 스코어:0.680412\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2gr-Sy-bc249"
      },
      "source": [
        "- 임곗값이 0.48일때가 성능 평가 지표를 유지하면서 재현율을 약간 향상시키는 좋은 임겟값으로 보임\r\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D0AjEYfncOvE",
        "outputId": "5d147003-3102-4c05-efb1-4defe2ba8b96"
      },
      "source": [
        "# 임곗값을 0.48로 설정한 Binarizer 생성\r\n",
        "binarizer = Binarizer(threshold=0.48)\r\n",
        "\r\n",
        "# 위에서 구한 lr_clf의 predict_proba() 예측 확률 array에서 1에 해당하는 컬럼값을 Binarizer 변환\r\n",
        "pred_th_048 = binarizer.fit_transform(pred_proba[:,1].reshape(-1,1))\r\n",
        "get_clf_eval(y_test,pred_th_048)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "오차행렬\n",
            "\n",
            "[[88 12]\n",
            " [19 35]]\n",
            "정확도:0.798701, 정밀도:0.744681, 재현율:0.648148, f1 스코어:0.693069\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y1gaXAO6dt4J"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}