2487_2223_Assignment1.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2487-2223 Machine Learning Assignment 1\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 1 (25 points) - Zestimate this House\n",
    "\n",
    "Purchasing a house is a very big decision for most of us. Companies such as Zillows collected tons of data regarding the listing and sold price of American houses and build the predictive model, named *Zestimate*. You are expected to build a model similar as Zestimate to predict house price in Boston. \n",
    "\n",
    "![zestimate](https://i0.wp.com/www.housesoldeasy.com/wp-content/uploads/Screen-Shot-2016-08-15-at-7.22.09-PM.png?resize=300%2C258&ssl=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.model_selection import cross_validate\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.linear_model import Ridge\n",
    "from sklearn.linear_model import Lasso\n",
    "from sklearn.linear_model import LogisticRegressionCV\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.base import BaseEstimator\n",
    "from sklearn.base import TransformerMixin\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "from sklearn.preprocessing import KBinsDiscretizer\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.metrics import make_scorer\n",
    "from sklearn.metrics import precision_score\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "import warnings \n",
    "warnings.filterwarnings('ignore')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data loaded correctly.\n",
      "Features: X. Target variable (price): y.\n",
      "X shape:  (506, 13) y shape:  (506,)\n"
     ]
    }
   ],
   "source": [
    "### DON'T MODIFY - LOAD DATA ### \n",
    "\n",
    "data_url = \"http://lib.stat.cmu.edu/datasets/boston\" \n",
    "raw_df = pd.read_csv(data_url, sep=\"\\s+\", skiprows=22, header=None)\n",
    "X = np.hstack([raw_df.values[::2, :], raw_df.values[1::2, :2]]) # FEATURES \n",
    "y = raw_df.values[1::2, 2] # TARGET VARIABLE\n",
    "assert X.shape[0] == y.shape[0], 'Mismatch in number of examples.'\n",
    "print('Data loaded correctly.')\n",
    "print('Features: X. Target variable (price): y.')\n",
    "print('X shape: ',X.shape, 'y shape: ', y.shape)\n",
    "### END ###"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The columns:\n",
    "| Index | Variable | Description                                                 |\n",
    "|-------|----------|-------------------------------------------------------------|\n",
    "| 0     | CRIM     | per capita crime rate by town                              |\n",
    "| 1     | ZN       | proportion of residential land zoned for lots over 25,000 sq.ft. |\n",
    "| 2     | INDUS    | proportion of non-retail business acres per town             |\n",
    "| 3     | CHAS     | Charles River dummy variable (= 1 if tract bounds river; 0 otherwise) |\n",
    "| 4     | NOX      | nitric oxides concentration (parts per 10 million)           |\n",
    "| 5     | RM       | average number of rooms per dwelling                         |\n",
    "| 6     | AGE      | proportion of owner-occupied units built prior to 1940       |\n",
    "| 7     | DIS      | weighted distances to five Boston employment centres        |\n",
    "| 8     | RAD      | index of accessibility to radial highways                    |\n",
    "| 9     | TAX      | full-value property-tax rate per $10,000                     |\n",
    "| 10    | PTRATIO  | pupil-teacher ratio by town                                  |\n",
    "| 11    | B        | 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town |\n",
    "| 12    | LSTAT    | % lower status of the population                             |\n",
    "| 13    | MEDV     | Median value of owner-occupied homes in $1000's              |\n",
    "\n",
    "Label = y = MEDV (price)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 1.1 (5 points) \n",
    "Create train and test set, each contains 80% and 20% of the dataset, respectively, using *train_test_split* function in scikit-learn. Train a linear model on the train set and evaluate on the test set, report the training error and test error, respectively (as mean squared error)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training error (MSE): 21.555648194527308\n",
      "Test error (MSE): 24.318238309170436\n"
     ]
    }
   ],
   "source": [
    "seed = 13\n",
    "\n",
    "# create linear regression object\n",
    "reg = LinearRegression()\n",
    "\n",
    "# split data into training and testing data sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = seed)\n",
    "\n",
    "# train the model with training set to find the best parameters for the best-fitting line\n",
    "reg.fit(X_train, y_train)\n",
    "\n",
    "# make predictions with train set and compute MSE on train data\n",
    "y_pred_test = reg.predict(X_train)\n",
    "error_train = mean_squared_error(y_train, y_pred_test)\n",
    "    \n",
    "# make predictions with test data and compute MSE on test data (unseen)\n",
    "y_pred = reg.predict(X_test)\n",
    "error_test = mean_squared_error(y_test, y_pred)\n",
    "\n",
    "print('Training error (MSE):', error_train) # how well the model fits the training data\n",
    "print('Test error (MSE):', error_test) # how well the model can predict new, unseen data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 1.2 (5 points)\n",
    "\n",
    "Perform a 10-fold cross-validation on the whole data set. Show the averaged mean sqaured error on both train and test set at each fold. Explain your findings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>train_MSE</th>\n",
       "      <th>test_MSE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>23.363228</td>\n",
       "      <td>9.286947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>22.882034</td>\n",
       "      <td>14.151283</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>23.216075</td>\n",
       "      <td>14.073606</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>20.771703</td>\n",
       "      <td>35.206924</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>21.335426</td>\n",
       "      <td>31.885117</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>22.363700</td>\n",
       "      <td>19.835878</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>23.327221</td>\n",
       "      <td>9.947269</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>11.959197</td>\n",
       "      <td>168.375380</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>21.586295</td>\n",
       "      <td>33.329745</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>23.189043</td>\n",
       "      <td>10.960411</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   train_MSE    test_MSE\n",
       "0  23.363228    9.286947\n",
       "1  22.882034   14.151283\n",
       "2  23.216075   14.073606\n",
       "3  20.771703   35.206924\n",
       "4  21.335426   31.885117\n",
       "5  22.363700   19.835878\n",
       "6  23.327221    9.947269\n",
       "7  11.959197  168.375380\n",
       "8  21.586295   33.329745\n",
       "9  23.189043   10.960411"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# perform 10-fold cross-validation on dataset with MSE\n",
    "scores = cross_validate(reg, X, y, scoring='neg_mean_squared_error', cv=10, return_train_score = True)\n",
    "\n",
    "# get train and test MSE from scores dict, multiply by -1 to get positive MSE\n",
    "train_MSE = scores[\"train_score\"] * -1  # The score array for train scores on each cv split\n",
    "test_MSE = scores[\"test_score\"] * -1    # The score array for test scores on each cv split\n",
    "\n",
    "# create dataframe containing the average MSE for train and test set for each fold\n",
    "mse_df = pd.DataFrame({'train_MSE': train_MSE, 'test_MSE': test_MSE})\n",
    "mse_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training error (MSE) over 10-fold cross-validation: 21.39939241710581\n",
      "Average testing error (MSE) over 10-fold cross-validation: 34.70525594452485\n"
     ]
    }
   ],
   "source": [
    "print(\"Average training error (MSE) over 10-fold cross-validation:\", train_MSE.mean())\n",
    "print(\"Average testing error (MSE) over 10-fold cross-validation:\", test_MSE.mean())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:orange\">**Findings:**</span>\n",
    "- The average MSE is calculated over 10 folds. The smaller the MSE, the better.\n",
    "- The model's performance is not consistent across folds. In 9 out of 10 folds get a training MSE around 21. In one fold the training MSE is significantly smaller at 11.95. \n",
    "- We can also see that the MSE on the training set varies from 9.28 to 168.37\n",
    "- Especially in cases where the test MSE is greater than the train MSE, the we can assume that the model is overfitting to the training data in some cases and then performing badly on the previously unseen test set.\n",
    "- In the 8th fold, we achieve the lowest training MSE of 11.95, but the by far largest test MSE of 168.37. This could also be explained by the model being overfit to the data it has been trained on. This fold has high variance.\n",
    "\n",
    "Overall, while the model does appear to be overfitting to the training data in many of the iterations, and is not generalizing well to new data, there are some cases where the training and test MSE values are more similar, but still relatively high. The high training error suggests, that a straight line doesn't fit well to the data (underfit)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 1.3 (5 points) \n",
    " \n",
    "Add 2-degree squared polynomial features (with no interactions) and perform 10-fold cross-validation on the whole data set. Show the mean sqaured error on both train and test set at each fold. Explain your findings.\n",
    "\n",
    "Hint: you may use sklearn.preprocessing.PolynomialFeatures and check how it produces the polynomial features with/without interaction terms."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#####  Trying out behavior of interaction terms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(506, 13)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(506, 92)\n",
      "Only interaction features (the product of different input features) and the terms with power 1 are produced\n"
     ]
    }
   ],
   "source": [
    "poly_interaction = PolynomialFeatures(degree = 2, interaction_only = True)\n",
    "X_interaction = poly_interaction.fit_transform(X)\n",
    "print(X_interaction.shape)\n",
    "print(\"Only interaction features (the product of different input features) and the terms with power 1 are produced\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(506, 105)\n",
      "Includes 78 interaction features, 13 features with power of 1, 13 features with power of 2, and intercept term\n"
     ]
    }
   ],
   "source": [
    "poly_no_interaction = PolynomialFeatures(degree = 2, interaction_only = False)\n",
    "X_no_interaction = poly_no_interaction.fit_transform(X)\n",
    "print(X_no_interaction.shape)\n",
    "print(\"Includes 78 interaction features, 13 features with power of 1, 13 features with power of 2, and intercept term\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: to achieve no interaction term at all, one must transform X by using np.hstack."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(506, 26)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_X = np.hstack((X, X**2))\n",
    "new_X.shape"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>train_MSE</th>\n",
       "      <th>test_MSE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>14.980271</td>\n",
       "      <td>10.091861</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>14.969661</td>\n",
       "      <td>8.791095</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>15.204169</td>\n",
       "      <td>11.458221</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>13.665059</td>\n",
       "      <td>22.518524</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>14.438738</td>\n",
       "      <td>14.223065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>14.940824</td>\n",
       "      <td>8.780146</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>14.859624</td>\n",
       "      <td>12.953433</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>7.419826</td>\n",
       "      <td>104.037379</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>14.605032</td>\n",
       "      <td>13.610872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>14.050919</td>\n",
       "      <td>50.771795</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   train_MSE    test_MSE\n",
       "0  14.980271   10.091861\n",
       "1  14.969661    8.791095\n",
       "2  15.204169   11.458221\n",
       "3  13.665059   22.518524\n",
       "4  14.438738   14.223065\n",
       "5  14.940824    8.780146\n",
       "6  14.859624   12.953433\n",
       "7   7.419826  104.037379\n",
       "8  14.605032   13.610872\n",
       "9  14.050919   50.771795"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# save an instance of PolynomialFeatures with the given degree\n",
    "polynomial_features = PolynomialFeatures(degree = 2, interaction_only = False)\n",
    "\n",
    "# create polynomial regression model\n",
    "model = LinearRegression()\n",
    "\n",
    "# fit model to data, using the X with no interaction terms new_X\n",
    "model.fit(new_X, y)\n",
    "\n",
    "# perform 10-fold cross-validation on dataset with MSE\n",
    "scores_poly = cross_validate(model, new_X, y, scoring='neg_mean_squared_error', cv=10, return_train_score = True)\n",
    "\n",
    "# get train and test MSE from scores dict, multiply by -1 to get positive MSE\n",
    "train_MSE_poly = scores_poly[\"train_score\"] * -1 \n",
    "test_MSE_poly = scores_poly[\"test_score\"] * -1 \n",
    "\n",
    "# create dataframe containing the average MSE for train and test set for each fold\n",
    "mse_df_poly = pd.DataFrame({'train_MSE': train_MSE_poly, 'test_MSE': test_MSE_poly})\n",
    "mse_df_poly\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average training error (MSE) over 10-fold cross-validation: 13.913412317032257\n",
      "Average testing error (MSE) over 10-fold cross-validation: 25.72363917701756\n"
     ]
    }
   ],
   "source": [
    "print(\"Average training error (MSE) over 10-fold cross-validation:\", train_MSE_poly.mean())\n",
    "print(\"Average testing error (MSE) over 10-fold cross-validation:\", test_MSE_poly.mean())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:orange\">**Findings:**</span>\n",
    "- The training error (MSE) of the 2-degree polynomial model (no interaction) with 13.92 has improved compared to the MSE of the linear model (21.4) \n",
    "- Also the testing error (MSE) of the 2-degree polynomial model (no interaction) with 25.72 has improved compared to the MSE of the linear model (34.7) \n",
    "\n",
    "Both errors are smaller than in the linear model. This means that the polynomial model fits the data better than a straight line.\n",
    "\n",
    "- Again, we have folds with high variance that are significantly worse on the unseen data than other folds, such as fold 8 and 10"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 1.4 (10 points)\n",
    "\n",
    "Perform cross-validation using ridge regression and lasso regression on different feature combinations (linear features vs. polynomial features obtained earlier respectively. Explain which method works better in this case. Check the coefficients and explain the differences between ridge regression and lasso regression."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Linear features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ridge Regression - Linear Features:\n",
      "Average training error over 10-k cv with ridge regression (linear): 21.58114506497026\n",
      "Average testing error over 10-k cv with ridge regression (linear): 34.07824620925931\n",
      "Coefficients\n",
      "    feature  coefficient\n",
      "0      CRIM    -0.150575\n",
      "1        ZN     0.058798\n",
      "2     INDUS    -0.035243\n",
      "3      CHAS     2.424039\n",
      "4       NOX    -8.099813\n",
      "5        RM     3.543438\n",
      "6       AGE    -0.002831\n",
      "7       DIS    -1.472918\n",
      "8       RAD     0.351650\n",
      "9       TAX    -0.015446\n",
      "10  PTRATIO    -0.821765\n",
      "11        B     0.007758\n",
      "12    LSTAT    -0.570019\n"
     ]
    }
   ],
   "source": [
    "# Ridge Regression\n",
    "X_names = ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT']\n",
    "\n",
    "ridge_reg = Ridge()\n",
    "cv_scores_rl = cross_validate(ridge_reg, X, y, cv=10, scoring='neg_mean_squared_error', return_train_score=True)\n",
    "\n",
    "# get train and test MSE from scores dict, multiply by -1 to get positive MSE\n",
    "train_MSE_rl = cv_scores_rl[\"train_score\"] * -1 \n",
    "test_MSE_rl = cv_scores_rl[\"test_score\"] * -1 \n",
    "\n",
    "print(\"Ridge Regression - Linear Features:\")\n",
    "print(\"Average training error over 10-k cv with ridge regression (linear):\", train_MSE_rl.mean())\n",
    "print(\"Average testing error over 10-k cv with ridge regression (linear):\", test_MSE_rl.mean())\n",
    "\n",
    "# Create dataframe with feature names and coefficients\n",
    "ridge_reg.fit(X_train, y_train)\n",
    "ridge_coefs = pd.DataFrame({'feature': X_names, 'coefficient': ridge_reg.coef_})\n",
    "print(\"Coefficients\")\n",
    "print(ridge_coefs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Lasso Regression - Linear Features:\n",
      "Average training error over 10-k cv with lasso regression (linear): 26.446016110077032\n",
      "Average testing error over 10-k cv with lasso regression (linear): 34.46408458830231\n",
      "Coefficients\n",
      "    feature  coefficient\n",
      "0      CRIM    -0.100439\n",
      "1        ZN     0.061775\n",
      "2     INDUS    -0.000000\n",
      "3      CHAS     0.000000\n",
      "4       NOX    -0.000000\n",
      "5        RM     0.737827\n",
      "6       AGE     0.025951\n",
      "7       DIS    -0.752248\n",
      "8       RAD     0.334002\n",
      "9       TAX    -0.018118\n",
      "10  PTRATIO    -0.694069\n",
      "11        B     0.006781\n",
      "12    LSTAT    -0.793181\n"
     ]
    }
   ],
   "source": [
    "# Lasso regression\n",
    "lasso_reg = Lasso(max_iter=5000)\n",
    "\n",
    "cv_scores_ll = cross_validate(lasso_reg, X, y, cv=10, scoring='neg_mean_squared_error', return_train_score=True)\n",
    "\n",
    "# get train and test MSE from scores dict, multiply by -1 to get positive MSE\n",
    "train_MSE_ll = cv_scores_ll[\"train_score\"] * -1 \n",
    "test_MSE_ll = cv_scores_ll[\"test_score\"] * -1 \n",
    "\n",
    "print(\"\\nLasso Regression - Linear Features:\")\n",
    "print(\"Average training error over 10-k cv with lasso regression (linear):\", train_MSE_ll.mean())\n",
    "print(\"Average testing error over 10-k cv with lasso regression (linear):\", test_MSE_ll.mean())\n",
    "\n",
    "lasso_reg.fit(X_train, y_train)\n",
    "lasso_coefs = pd.DataFrame({'feature': X_names, 'coefficient': lasso_reg.coef_})\n",
    "print(\"Coefficients\")\n",
    "print(lasso_coefs)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Polynomial features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ridge Regression - Polynomial Features:\n",
      "Average training error over 10-k cv with ridge regression (poly): 14.201739336116793\n",
      "Average testing error over 10-k cv with ridge regression (poly): 26.942921093602394\n"
     ]
    }
   ],
   "source": [
    "# Ridge Regression\n",
    "cv_scores_rp = cross_validate(ridge_reg, new_X, y, cv=10, scoring='neg_mean_squared_error', return_train_score=True)\n",
    "\n",
    "# get train and test MSE from scores dict, multiply by -1 to get positive MSE\n",
    "train_MSE_rp = cv_scores_rp[\"train_score\"] * -1 \n",
    "test_MSE_rp = cv_scores_rp[\"test_score\"] * -1 \n",
    "\n",
    "print(\"Ridge Regression - Polynomial Features:\")\n",
    "print(\"Average training error over 10-k cv with ridge regression (poly):\", train_MSE_rp.mean())\n",
    "print(\"Average testing error over 10-k cv with ridge regression (poly):\", test_MSE_rp.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Lasso Regression - Polynomial Features:\n",
      "Average training error over 10-k cv with lasso regression (poly): 18.548501008872663\n",
      "Average testing error over 10-k cv with lasso regression (poly): 30.776932337350466\n"
     ]
    }
   ],
   "source": [
    "# Lasso regression\n",
    "cv_scores_lp = cross_validate(lasso_reg, new_X, y, cv=10, scoring='neg_mean_squared_error', return_train_score=True)\n",
    "\n",
    "# get train and test MSE from scores dict, multiply by -1 to get positive MSE\n",
    "train_MSE_lp = cv_scores_lp[\"train_score\"] * -1 \n",
    "test_MSE_lp = cv_scores_lp[\"test_score\"] * -1 \n",
    "\n",
    "print(\"\\nLasso Regression - Polynomial Features:\")\n",
    "print(\"Average training error over 10-k cv with lasso regression (poly):\", train_MSE_lp.mean())\n",
    "print(\"Average testing error over 10-k cv with lasso regression (poly):\", test_MSE_lp.mean())\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:orange\">**Findings:**</span>\n",
    "Which method works better?\n",
    "- For the linear model: The testing error is almost the same for both ridge regression (34.08) and lasso regression (34.46). The training error with ridge regression is performing slightly better (21.58) than the training error with lasso regression (26.4)\n",
    "- For the 2-degree polynomial model: The ridge regression performed slightly better with a testing error of 26.9 and a training error of 14.2 than the lasso regression with a testing error of 30.8 and a training error of 18.5.\n",
    "\n",
    "<span style=\"color:orange\">**Difference between ridge and lasso regression:**</span>\n",
    "Both ridge and lasso regression are regularization techniques that can be used to prevent overfitting and improve model generalization. They work by penalizing high values of parameters θ to reduce the effect of some features on the model’s predicted value. More precisely, a penalty term is added to the loss function. The MSE is then calculated with this penalty term and then the best line is chosen for the regularized cost function.\n",
    "\n",
    "- Ridge regression is a L2 regularized version of Linear regression. The weights are penalized in proportion to the sum of the squares of the weights. If parameter lambda is set very high, the parameters (/ model weights) will shrink close to zero, but they will never be 0.\n",
    "\n",
    "- Lasso regression is a L1 regularized version of Linear Regression. The weights are penalized in proportion to sthe um of the absolute values of the weights. L1 can (unlike L2) push certain weights to be 0. L1 therefore also performs feature selection. This can be seen by the coefficient of the features CHAS, INDUS, and NOX being set to 0.000000, so these features were removed."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 2 (25 points) - Cancer Detection\n",
    "\n",
    "Given a dataset with features that are computed from a digitized image of a fine needle aspirate (FNA) of a breast mass, which describes characteristics of the cell nuclei present in the image, let's try to predict whether the patients are diagnosed as Malignant (M) or Benign (B)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# LOAD DATA \n",
    "\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "\"\"\"\n",
    "DOCS:\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html#sklearn.datasets.load_breast_cancer\n",
    "\"\"\"\n",
    "X, y = load_breast_cancer(return_X_y=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:orange\">**Performance Metric**</span>\n",
    "\n",
    "see class distribution as stated here: https://scikit-learn.org/stable/datasets/toy_dataset.html#breast-cancer-dataset\n",
    "\n",
    "0 = negative class = Malignant (=Cancer) = 212\n",
    "\n",
    "1 = positive class = Benign (=No cancer)= 357"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normally, one would choose Recall as performance metric for sick patient detection, when cost of **False Negative** is high, meaning we predict an actual positive (cancer) as negative. However in this dataset, an actual positive is represented by benign tumor (no cancer). \n",
    "\n",
    "So for our dataset, the cost of **False Positive** (actually negative (0 = cancer) but we predicted positive (1 = no cancer) is high. \n",
    "\n",
    "Therefore I'm using precision as a score, measuring how accurate is our model out of those being predicted as positive.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 2.1 (5 points) \n",
    "Use logistic regression to train the dataset through cross-validation, report the score on train and test set, respectively. Explain your findings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean Precision on Training set: 0.9613558441408451\n",
      "Mean Precision on Testing set: 0.9574495360021675\n"
     ]
    }
   ],
   "source": [
    "# instantiate the model\n",
    "logreg = LogisticRegression(max_iter=10000)\n",
    "\n",
    "# perform 10-fold cross-validation on dataset with recall\n",
    "scores_log = cross_validate(logreg, X, y, scoring='precision', cv=10, return_train_score = True)\n",
    "\n",
    "# get train and test recall from scores dict\n",
    "train_MSE_log = scores_log[\"train_score\"]\n",
    "test_MSE_log = scores_log[\"test_score\"]\n",
    "\n",
    "print(\"Mean Precision on Training set:\", train_MSE_log.mean())\n",
    "print(\"Mean Precision on Testing set:\", test_MSE_log.mean())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:orange\">**Findings:**</span>\n",
    "After training the dataset through 10-fold cross-validation using logistic regression, we get an average precision of 0.96 on the training set, and a slightly worse recall of 0.95 on the testing set.\n",
    "\n",
    "The results show that the logistic regression model achieved high precision scores on both the training set (0.961) and testing set (0.957). This suggests that the model is performing well and is able to generalize to new, unseen data. But with the extremely high cost of misdiagnosing cancer as no cancer, we want to get the precision as high as possible."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 2.2 (5 points) \n",
    "By default, sklearn's logistic regression uses the L2 regularization. Now use the logistic regression without any regularzation to perform cross validation, report what do you find on train and test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean Precision on Training set: 0.91765770934469\n",
      "Mean Precision on Testing set: 0.9180184203017021\n"
     ]
    }
   ],
   "source": [
    "# instantiate the model without any regularization\n",
    "logreg2 = LogisticRegression(max_iter=10000, penalty='none', solver=\"saga\")\n",
    "\n",
    "# perform 10-fold cross-validation on dataset with recall\n",
    "scores_log2 = cross_validate(logreg2, X, y, cv=10, scoring='precision', return_train_score=True)\n",
    "\n",
    "# print train and test recall from scores dict\n",
    "print(\"Mean Precision on Training set:\", scores_log2[\"train_score\"].mean())\n",
    "print(\"Mean Precision on Testing set:\", scores_log2[\"test_score\"].mean())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:orange\">**Findings:**</span>\n",
    "The model that is not using any regularization is performing worse than the one using l2 regularization (ridge regression). The new model only achieves an average precision of 0.92 on the training set (vs. 0.96 with l2) and 0.92 (vs. 0.96 with l2) on the testing set.\n",
    "\n",
    "Reducing the complexity of the model produces a better precision."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 2.3 (15 points) \n",
    "Check how many Benign and Malignant cases in the dataset. What might be the problem if we use the default score of the logistic regression cross-validation? Now adjust the class weight of M and L and retrain the model again to bias toward Malignant, using the relative weight of M and L as 2:1. What about the relaive weight to be 5:1, or 10:1? Explain what you find.\n",
    "\n",
    "Hint: you can use LogisticRegressionCV to combine LogisticRegression and cross-validation. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:orange\">**Finding out how many benign and malignant cases are in the dataset**</span>\n",
    "\n",
    "see class distribution as stated here: https://scikit-learn.org/stable/datasets/toy_dataset.html#breast-cancer-dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 212, 1: 357}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unique, counts = np.unique(y, return_counts=True)\n",
    "dict(zip(unique, counts))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "0 = negative class = Malignant (=Cancer) = 212\n",
    "\n",
    "1 = positive class = Benign = 357\n",
    "\n",
    "We have class imbalance. Model might struggle to learn the patterns of the minority class (Malignant), may become biased towards the majority class (Benign). \n",
    "\n",
    "<span style=\"color:orange\">**The problem with the default score**</span>\n",
    "The default score of the logistic regression cross-validation is mean accuracy. Accuracy is not a good metric for imbalanced dataset. A model might have good accuracy by achieving the high accuracy with always predicts the majority class - a model like this is essentially useless.\n",
    "\n",
    "Also for the case of sick patient detection, we are more interested in the model actually predicting cancer if the patient has cancer (normally recall, in this case as discussed earlier precision).\n",
    "\n",
    "<span style=\"color:orange\">**Adjusting the class weight**</span>\n",
    "One solution is to modify the class weights to bias the model towards the Malignant class. We may do this by giving the Malignant class a heavier weight than the Benign class during training. The class weight parameter in the LogisticRegression model can be used to change the relative weight of the classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of malignant cases: 212 (37.26%)\n",
      "Number of benign cases: 357 (62.74%)\n",
      "Score with 2:1 class weight ratio: 0.956140350877193\n",
      "Score with 5:1 class weight ratio: 0.9473684210526315\n",
      "Score with 10:1 class weight ratio: 0.9473684210526315\n"
     ]
    }
   ],
   "source": [
    "# Split data into training and testing sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=seed)\n",
    "\n",
    "# Count number of samples in each class\n",
    "n_samples = y.shape[0]\n",
    "n_malignant = sum(y == 0) # 0 = negative class = Malignant (=Cancer)\n",
    "n_benign = sum(y == 1) # 1 = positive class = Benign\n",
    "\n",
    "print(f\"Number of malignant cases: {n_malignant} ({n_malignant / n_samples:.2%})\")\n",
    "print(f\"Number of benign cases: {n_benign} ({n_benign / n_samples:.2%})\")\n",
    "\n",
    "# Define logistic regression model with class weights\n",
    "clf_2to1 = LogisticRegressionCV(class_weight={0: 2, 1: 1}, cv=10, max_iter=10000)\n",
    "clf_5to1 = LogisticRegressionCV(class_weight={0: 5, 1: 1}, cv=10, max_iter=10000)\n",
    "clf_10to1 = LogisticRegressionCV(class_weight={0: 10, 1: 1}, cv=10, max_iter=10000)\n",
    "\n",
    "# train model with 2:1 class weight ratio and evaluate model on test set\n",
    "clf_2to1.fit(X_train, y_train)\n",
    "score_2to1 = clf_2to1.score(X_test, y_test)\n",
    "print(f\"Score with 2:1 class weight ratio: {score_2to1}\")\n",
    "\n",
    "# Train model with 5:1 class weight ratio and evaluate model on test set\n",
    "clf_5to1.fit(X_train, y_train)\n",
    "score_5to1 = clf_5to1.score(X_test, y_test)\n",
    "print(f\"Score with 5:1 class weight ratio: {score_5to1}\")\n",
    "\n",
    "# Train model with 10:1 class weight ratio and evaluate model on test set\n",
    "clf_10to1.fit(X_train, y_train)\n",
    "score_10to1 = clf_10to1.score(X_test, y_test)\n",
    "print(f\"Score with 10:1 class weight ratio: {score_10to1}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      0.93      0.94       212\n",
      "           1       0.96      0.97      0.97       357\n",
      "\n",
      "    accuracy                           0.96       569\n",
      "   macro avg       0.96      0.95      0.95       569\n",
      "weighted avg       0.96      0.96      0.96       569\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# normal class weight ratio (with l2 regularization)\n",
    "logreg.fit(X, y)\n",
    "y_pred_normal = logreg.predict(X)\n",
    "print(classification_report(y, y_pred_normal))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      0.98      0.96       212\n",
      "           1       0.99      0.96      0.97       357\n",
      "\n",
      "    accuracy                           0.97       569\n",
      "   macro avg       0.96      0.97      0.97       569\n",
      "weighted avg       0.97      0.97      0.97       569\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 2:1 class weight ratio\n",
    "y_pred_2to1 = clf_2to1.predict(X)\n",
    "print(classification_report(y, y_pred_2to1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.92      0.99      0.95       212\n",
      "           1       0.99      0.95      0.97       357\n",
      "\n",
      "    accuracy                           0.96       569\n",
      "   macro avg       0.96      0.97      0.96       569\n",
      "weighted avg       0.97      0.96      0.96       569\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 5:1 class weight ratio\n",
    "y_pred_5to1 = clf_5to1.predict(X)\n",
    "print(classification_report(y, y_pred_5to1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.91      0.99      0.95       212\n",
      "           1       0.99      0.94      0.97       357\n",
      "\n",
      "    accuracy                           0.96       569\n",
      "   macro avg       0.95      0.96      0.96       569\n",
      "weighted avg       0.96      0.96      0.96       569\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 10:1 class weight ratio\n",
    "y_pred_10to1 = clf_10to1.predict(X)\n",
    "print(classification_report(y, y_pred_10to1))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:orange\">**Findings**</span>\n",
    "Since the dataset is balanced towards benign cases, we tried changing the weight. We retrained the model again to bias toward Malignant by setting the class_weight={0: 2, 1: 1}. This means that the model will weigh the malignant class samples twice as much as the benign class samples during training.\n",
    "\n",
    "Since the model is performing worse the more we changed the weight towards Malignant, it seems like we overcompensated for the class imbalance and end up biasing the model too much towards the malignant class."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 3 (50 points) - Call Me Maybe? \n",
    "\n",
    "\n",
    "\n",
    "![telemarketing](https://neilpatel.com/wp-content/uploads/2019/08/profissional-de-telemarketing-sorridente.jpeg)\n",
    "\n",
    "Telemarketing is a method of direct marketing in which a salesperson solicits prospective customers to buy products or services over the phone. It has become one of the most widely used marketing campaign methods to engage with customers with product and service opportunity. We have collected real data from a Portuguese retail bank, from May 2008 to June 2013 with thousands of phone contacts. \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "The current practice of many data teams is to build such propensity models and predict customer's probability to adopt the product and target them from the highest probability to the lowest probability. Note that telemarketing may incur some costs for contacting the customer, thus the success (i.e., the generated profit) of using machine learning model requries further inspection.  As the data scientist, you are asked to build a propensity model to evaluate the effectiveness of their telemarketing campaigns, i.e. whether the customer subscribed to the term deposit.  \n",
    "\n",
    "**Telemarketing Dataset (bank.csv)**\n",
    "All customers are contained in the file bank.csv. Each line of this file after the header row represents one customer of the Portuguese bank, and has the following format:\n",
    "\n",
    "### bank client data:\n",
    "- age (numeric)\n",
    "- job : type of job (categorical: 'admin.','blue-collar','entrepreneur','housemaid','management','retired','self-employed','services','student','technician','unemployed','unknown')\n",
    "- marital : marital status (categorical: 'divorced','married','single','unknown'; note: 'divorced' means divorced or widowed)\n",
    "- education (categorical: 'primary', 'secondary', 'tertiary')\n",
    "- balance: amcount of bank account balance\n",
    "- default: has credit in default? (categorical: 'no','yes','unknown')\n",
    "- housing: has housing loan? (categorical: 'no','yes','unknown')\n",
    "- loan: has personal loan? (categorical: 'no','yes','unknown')\n",
    "\n",
    "### related with the last contact of the current campaign:\n",
    "- contact: contact communication type (categorical: 'cellular','telephone', 'unknown')\n",
    "- day: last contact day of month\n",
    "- month: last contact month of year (categorical: 'jan', 'feb', 'mar', ..., 'nov', 'dec')\n",
    "- duration: last contact duration, in seconds (numeric). Important note: this attribute highly affects the output target (e.g., if duration=0 then y='no'). Yet, the duration is not known before a call is performed. \n",
    "\n",
    "### other attributes:\n",
    "- campaign: number of contacts performed during this campaign and for this client (numeric, includes last contact)\n",
    "- pdays: number of days that passed by after the client was last contacted from a previous campaign (numeric; -1 means client was not previously contacted)\n",
    "- previous: number of contacts performed before this campaign and for this client (numeric)\n",
    "- poutcome: outcome of the previous marketing campaign (categorical: 'failure','nonexistent','success')\n",
    "- y - has the client subscribed a term deposit? (binary: 'yes','no')\n",
    "\n",
    "\n",
    "Answer the following questions using the provided dataset. You can write down intermediate results towards the final answers. If any model invovles random_state, set it to be 42."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>job</th>\n",
       "      <th>marital</th>\n",
       "      <th>education</th>\n",
       "      <th>default</th>\n",
       "      <th>balance</th>\n",
       "      <th>housing</th>\n",
       "      <th>loan</th>\n",
       "      <th>contact</th>\n",
       "      <th>day</th>\n",
       "      <th>month</th>\n",
       "      <th>duration</th>\n",
       "      <th>campaign</th>\n",
       "      <th>pdays</th>\n",
       "      <th>previous</th>\n",
       "      <th>poutcome</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>30</td>\n",
       "      <td>unemployed</td>\n",
       "      <td>married</td>\n",
       "      <td>primary</td>\n",
       "      <td>no</td>\n",
       "      <td>1787</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>cellular</td>\n",
       "      <td>19</td>\n",
       "      <td>oct</td>\n",
       "      <td>79</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>unknown</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>33</td>\n",
       "      <td>services</td>\n",
       "      <td>married</td>\n",
       "      <td>secondary</td>\n",
       "      <td>no</td>\n",
       "      <td>4789</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>cellular</td>\n",
       "      <td>11</td>\n",
       "      <td>may</td>\n",
       "      <td>220</td>\n",
       "      <td>1</td>\n",
       "      <td>339</td>\n",
       "      <td>4</td>\n",
       "      <td>failure</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>35</td>\n",
       "      <td>management</td>\n",
       "      <td>single</td>\n",
       "      <td>tertiary</td>\n",
       "      <td>no</td>\n",
       "      <td>1350</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>cellular</td>\n",
       "      <td>16</td>\n",
       "      <td>apr</td>\n",
       "      <td>185</td>\n",
       "      <td>1</td>\n",
       "      <td>330</td>\n",
       "      <td>1</td>\n",
       "      <td>failure</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>30</td>\n",
       "      <td>management</td>\n",
       "      <td>married</td>\n",
       "      <td>tertiary</td>\n",
       "      <td>no</td>\n",
       "      <td>1476</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>unknown</td>\n",
       "      <td>3</td>\n",
       "      <td>jun</td>\n",
       "      <td>199</td>\n",
       "      <td>4</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>unknown</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>59</td>\n",
       "      <td>blue-collar</td>\n",
       "      <td>married</td>\n",
       "      <td>secondary</td>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>unknown</td>\n",
       "      <td>5</td>\n",
       "      <td>may</td>\n",
       "      <td>226</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>unknown</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age          job  marital  education default  balance housing loan  \\\n",
       "0   30   unemployed  married    primary      no     1787      no   no   \n",
       "1   33     services  married  secondary      no     4789     yes  yes   \n",
       "2   35   management   single   tertiary      no     1350     yes   no   \n",
       "3   30   management  married   tertiary      no     1476     yes  yes   \n",
       "4   59  blue-collar  married  secondary      no        0     yes   no   \n",
       "\n",
       "    contact  day month  duration  campaign  pdays  previous poutcome   y  \n",
       "0  cellular   19   oct        79         1     -1         0  unknown  no  \n",
       "1  cellular   11   may       220         1    339         4  failure  no  \n",
       "2  cellular   16   apr       185         1    330         1  failure  no  \n",
       "3   unknown    3   jun       199         4     -1         0  unknown  no  \n",
       "4   unknown    5   may       226         1     -1         0  unknown  no  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bank = pd.read_csv('bank.csv', sep=';')\n",
    "bank.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "age           int64\n",
       "job          object\n",
       "marital      object\n",
       "education    object\n",
       "default      object\n",
       "balance       int64\n",
       "housing      object\n",
       "loan         object\n",
       "contact      object\n",
       "day           int64\n",
       "month        object\n",
       "duration      int64\n",
       "campaign      int64\n",
       "pdays         int64\n",
       "previous      int64\n",
       "poutcome     object\n",
       "y            object\n",
       "dtype: object"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bank.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>balance</th>\n",
       "      <th>day</th>\n",
       "      <th>duration</th>\n",
       "      <th>campaign</th>\n",
       "      <th>pdays</th>\n",
       "      <th>previous</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>4521.000000</td>\n",
       "      <td>4521.000000</td>\n",
       "      <td>4521.000000</td>\n",
       "      <td>4521.000000</td>\n",
       "      <td>4521.000000</td>\n",
       "      <td>4521.000000</td>\n",
       "      <td>4521.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>41.170095</td>\n",
       "      <td>1422.657819</td>\n",
       "      <td>15.915284</td>\n",
       "      <td>263.961292</td>\n",
       "      <td>2.793630</td>\n",
       "      <td>39.766645</td>\n",
       "      <td>0.542579</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>10.576211</td>\n",
       "      <td>3009.638142</td>\n",
       "      <td>8.247667</td>\n",
       "      <td>259.856633</td>\n",
       "      <td>3.109807</td>\n",
       "      <td>100.121124</td>\n",
       "      <td>1.693562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>19.000000</td>\n",
       "      <td>-3313.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>33.000000</td>\n",
       "      <td>69.000000</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>104.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>39.000000</td>\n",
       "      <td>444.000000</td>\n",
       "      <td>16.000000</td>\n",
       "      <td>185.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>49.000000</td>\n",
       "      <td>1480.000000</td>\n",
       "      <td>21.000000</td>\n",
       "      <td>329.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>87.000000</td>\n",
       "      <td>71188.000000</td>\n",
       "      <td>31.000000</td>\n",
       "      <td>3025.000000</td>\n",
       "      <td>50.000000</td>\n",
       "      <td>871.000000</td>\n",
       "      <td>25.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               age       balance          day     duration     campaign  \\\n",
       "count  4521.000000   4521.000000  4521.000000  4521.000000  4521.000000   \n",
       "mean     41.170095   1422.657819    15.915284   263.961292     2.793630   \n",
       "std      10.576211   3009.638142     8.247667   259.856633     3.109807   \n",
       "min      19.000000  -3313.000000     1.000000     4.000000     1.000000   \n",
       "25%      33.000000     69.000000     9.000000   104.000000     1.000000   \n",
       "50%      39.000000    444.000000    16.000000   185.000000     2.000000   \n",
       "75%      49.000000   1480.000000    21.000000   329.000000     3.000000   \n",
       "max      87.000000  71188.000000    31.000000  3025.000000    50.000000   \n",
       "\n",
       "             pdays     previous  \n",
       "count  4521.000000  4521.000000  \n",
       "mean     39.766645     0.542579  \n",
       "std     100.121124     1.693562  \n",
       "min      -1.000000     0.000000  \n",
       "25%      -1.000000     0.000000  \n",
       "50%      -1.000000     0.000000  \n",
       "75%      -1.000000     0.000000  \n",
       "max     871.000000    25.000000  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bank.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "no     4000\n",
       "yes     521\n",
       "Name: y, dtype: int64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bank.y.value_counts()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:orange\">**y:**</span> Only 11.5% (521/4521) of customers subscribed to a term deposit -> unbalanced data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3.1 (15 points)\n",
    "\n",
    "Split the data into 80% training set and 20% test set. **Build a pipeline to preprocess the indicated numerical features and categorical features separately**. For numerical features 'balance', 'campaign', standardize these features. For categorical features 'job', 'marital', 'education', 'default', transform them through one-hot encoding. For the numeric feature 'age', convert it into the quartile categorical variable and transform it through one-hot encoding. \n",
    "\n",
    "Train a Logistic regression model with L2 regularization using 5-fold cross validation (default hyperparameter) on the train set and show the accuracy, precision, recall on the train set. Explain whether the model is useful for the bank to identify the customer propensity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# replaces the values 'yes' and 'no' in the y variable with 1 and 0 respectively\n",
    "X = bank.drop('y', axis = 1)\n",
    "y = bank[\"y\"] .replace({\"yes\": 1, \"no\": 0})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# visualization of pipeline\n",
    "from sklearn import set_config\n",
    "set_config(display=\"diagram\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each subpipeline starts with a selector transformer: it simply transforms the data by selecting the desired attributes (numerical or categorical), dropping the rest, and converting the resulting DataFrame to a NumPy array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a class to select numerical or categorical columns \n",
    "class DataFrameSelector(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self, attribute_names):\n",
    "        self.attribute_names = attribute_names\n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    def transform(self, X):\n",
    "        return X[self.attribute_names].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.12107186, -0.57682947,  0.        , ...,  0.        ,\n",
       "         0.        ,  0.        ],\n",
       "       [ 1.1186443 , -0.57682947,  0.        , ...,  1.        ,\n",
       "         0.        ,  0.        ],\n",
       "       [-0.02414438, -0.57682947,  0.        , ...,  1.        ,\n",
       "         0.        ,  0.        ],\n",
       "       ...,\n",
       "       [-0.37472364,  2.63916021,  0.        , ...,  0.        ,\n",
       "         0.        ,  1.        ],\n",
       "       [-0.09492484,  0.38796743,  0.        , ...,  0.        ,\n",
       "         0.        ,  0.        ],\n",
       "       [-0.09525714, -0.2552305 ,  0.        , ...,  0.        ,\n",
       "         1.        ,  0.        ]])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "seed = 42\n",
    "\n",
    "# define attributes\n",
    "num_attribs = [\"balance\", \"campaign\"]\n",
    "cat_attribs = [\"job\", \"marital\", \"education\", \"default\"]\n",
    "age_attribs = [\"age\"]\n",
    "\n",
    "# define pipeline to apply feature scaling to numerical features\n",
    "num_pipeline = Pipeline([\n",
    "        ('selector', DataFrameSelector(num_attribs)),\n",
    "        ('std_scaler', StandardScaler())\n",
    "])\n",
    "\n",
    "# define pipeline to apply one-hot encoding to categorical values\n",
    "cat_pipeline = Pipeline([\n",
    "        ('selector', DataFrameSelector(cat_attribs)),\n",
    "        ('cat_encoder', OneHotEncoder(sparse=False, handle_unknown='ignore')),\n",
    "    ])\n",
    "\n",
    "# define pipeline to convert age into the quartile categorical variable and transform it through one-hot encoding. \n",
    "age_pipeline = Pipeline([\n",
    "        ('selector', DataFrameSelector(age_attribs)),\n",
    "        ('age_quantile_transform', KBinsDiscretizer(n_bins=4, encode='ordinal', strategy='quantile')),\n",
    "        ('cat_encoder', OneHotEncoder(sparse=False, handle_unknown='ignore')),\n",
    "    ])\n",
    "\n",
    "# apply different operations on num and cat by combining pipelines with ColumnTransformer\n",
    "preprocessing = ColumnTransformer([\n",
    "        (\"num\", num_pipeline, num_attribs),\n",
    "        (\"cat\", cat_pipeline, cat_attribs),\n",
    "        (\"age\", age_pipeline, age_attribs)\n",
    "    ])\n",
    "\n",
    "X_processed = preprocessing.fit_transform(bank)\n",
    "X_processed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 {color: black;background-color: white;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 pre{padding: 0;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-toggleable {background-color: white;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-item {z-index: 1;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-parallel::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-parallel-item:only-child::after {width: 0;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;position: relative;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-e66cf075-a750-43aa-891b-afcbc9000eb1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-e66cf075-a750-43aa-891b-afcbc9000eb1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;num&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;selector&#x27;,\n",
       "                                                  DataFrameSelector(attribute_names=[&#x27;balance&#x27;,\n",
       "                                                                                     &#x27;campaign&#x27;])),\n",
       "                                                 (&#x27;std_scaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;balance&#x27;, &#x27;campaign&#x27;]),\n",
       "                                (&#x27;cat&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;selector&#x27;,\n",
       "                                                  DataFrameSelector(attribute_names=[&#x27;job&#x27;,\n",
       "                                                                                     &#x27;marital&#x27;,\n",
       "                                                                                     &#x27;education&#x27;,\n",
       "                                                                                     &#x27;default&#x27;])),\n",
       "                                                 (&#x27;cat_encoder&#x27;,\n",
       "                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;,\n",
       "                                                                sparse=False))]),\n",
       "                                 [&#x27;job&#x27;, &#x27;marital&#x27;, &#x27;education&#x27;, &#x27;default&#x27;]),\n",
       "                                (&#x27;age&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;selector&#x27;,\n",
       "                                                  DataFrameSelector(attribute_names=[&#x27;age&#x27;])),\n",
       "                                                 (&#x27;age_quantile_transform&#x27;,\n",
       "                                                  KBinsDiscretizer(encode=&#x27;ordinal&#x27;,\n",
       "                                                                   n_bins=4)),\n",
       "                                                 (&#x27;cat_encoder&#x27;,\n",
       "                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;,\n",
       "                                                                sparse=False))]),\n",
       "                                 [&#x27;age&#x27;])])</pre><b>Please rerun this cell to show the HTML repr or trust the notebook.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"0d4e0d2c-3df2-4ac9-8b95-77babfce840d\" type=\"checkbox\" ><label for=\"0d4e0d2c-3df2-4ac9-8b95-77babfce840d\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;num&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;selector&#x27;,\n",
       "                                                  DataFrameSelector(attribute_names=[&#x27;balance&#x27;,\n",
       "                                                                                     &#x27;campaign&#x27;])),\n",
       "                                                 (&#x27;std_scaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;balance&#x27;, &#x27;campaign&#x27;]),\n",
       "                                (&#x27;cat&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;selector&#x27;,\n",
       "                                                  DataFrameSelector(attribute_names=[&#x27;job&#x27;,\n",
       "                                                                                     &#x27;marital&#x27;,\n",
       "                                                                                     &#x27;education&#x27;,\n",
       "                                                                                     &#x27;default&#x27;])),\n",
       "                                                 (&#x27;cat_encoder&#x27;,\n",
       "                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;,\n",
       "                                                                sparse=False))]),\n",
       "                                 [&#x27;job&#x27;, &#x27;marital&#x27;, &#x27;education&#x27;, &#x27;default&#x27;]),\n",
       "                                (&#x27;age&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;selector&#x27;,\n",
       "                                                  DataFrameSelector(attribute_names=[&#x27;age&#x27;])),\n",
       "                                                 (&#x27;age_quantile_transform&#x27;,\n",
       "                                                  KBinsDiscretizer(encode=&#x27;ordinal&#x27;,\n",
       "                                                                   n_bins=4)),\n",
       "                                                 (&#x27;cat_encoder&#x27;,\n",
       "                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;,\n",
       "                                                                sparse=False))]),\n",
       "                                 [&#x27;age&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"42ff9620-44d6-43dd-bd70-4d5f4a2cc394\" type=\"checkbox\" ><label for=\"42ff9620-44d6-43dd-bd70-4d5f4a2cc394\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">num</label><div class=\"sk-toggleable__content\"><pre>[&#x27;balance&#x27;, &#x27;campaign&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"81bd42bc-25b1-4384-88a7-ac8e61b816d3\" type=\"checkbox\" ><label for=\"81bd42bc-25b1-4384-88a7-ac8e61b816d3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">DataFrameSelector</label><div class=\"sk-toggleable__content\"><pre>DataFrameSelector(attribute_names=[&#x27;balance&#x27;, &#x27;campaign&#x27;])</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"91b309a2-8428-4ca2-9f7c-7e4d2e6a2f4c\" type=\"checkbox\" ><label for=\"91b309a2-8428-4ca2-9f7c-7e4d2e6a2f4c\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"87e72361-15b2-4bd7-9ff1-3207510472a5\" type=\"checkbox\" ><label for=\"87e72361-15b2-4bd7-9ff1-3207510472a5\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">cat</label><div class=\"sk-toggleable__content\"><pre>[&#x27;job&#x27;, &#x27;marital&#x27;, &#x27;education&#x27;, &#x27;default&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"be37eb76-49c1-40ff-8c17-5dd6a81d1c4f\" type=\"checkbox\" ><label for=\"be37eb76-49c1-40ff-8c17-5dd6a81d1c4f\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">DataFrameSelector</label><div class=\"sk-toggleable__content\"><pre>DataFrameSelector(attribute_names=[&#x27;job&#x27;, &#x27;marital&#x27;, &#x27;education&#x27;, &#x27;default&#x27;])</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"0174d6d4-6706-46eb-981b-4a167df4b911\" type=\"checkbox\" ><label for=\"0174d6d4-6706-46eb-981b-4a167df4b911\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;, sparse=False)</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"b4fc60d5-1c24-4891-a8dd-b4ff66675385\" type=\"checkbox\" ><label for=\"b4fc60d5-1c24-4891-a8dd-b4ff66675385\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">age</label><div class=\"sk-toggleable__content\"><pre>[&#x27;age&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"d45fe0e1-cfd9-4312-9c12-03d092214c33\" type=\"checkbox\" ><label for=\"d45fe0e1-cfd9-4312-9c12-03d092214c33\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">DataFrameSelector</label><div class=\"sk-toggleable__content\"><pre>DataFrameSelector(attribute_names=[&#x27;age&#x27;])</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"949c2d7b-9ff4-416b-8078-e9d8d75eb7e3\" type=\"checkbox\" ><label for=\"949c2d7b-9ff4-416b-8078-e9d8d75eb7e3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">KBinsDiscretizer</label><div class=\"sk-toggleable__content\"><pre>KBinsDiscretizer(encode=&#x27;ordinal&#x27;, n_bins=4)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"63f091b0-b911-4279-8d9a-52f54df1b120\" type=\"checkbox\" ><label for=\"63f091b0-b911-4279-8d9a-52f54df1b120\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;, sparse=False)</pre></div></div></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('num',\n",
       "                                 Pipeline(steps=[('selector',\n",
       "                                                  DataFrameSelector(attribute_names=['balance',\n",
       "                                                                                     'campaign'])),\n",
       "                                                 ('std_scaler',\n",
       "                                                  StandardScaler())]),\n",
       "                                 ['balance', 'campaign']),\n",
       "                                ('cat',\n",
       "                                 Pipeline(steps=[('selector',\n",
       "                                                  DataFrameSelector(attribute_names=['job',\n",
       "                                                                                     'marital',\n",
       "                                                                                     'education',\n",
       "                                                                                     'default'])),\n",
       "                                                 ('cat_encoder',\n",
       "                                                  OneHotEncoder(handle_unknown='ignore',\n",
       "                                                                sparse=False))]),\n",
       "                                 ['job', 'marital', 'education', 'default']),\n",
       "                                ('age',\n",
       "                                 Pipeline(steps=[('selector',\n",
       "                                                  DataFrameSelector(attribute_names=['age'])),\n",
       "                                                 ('age_quantile_transform',\n",
       "                                                  KBinsDiscretizer(encode='ordinal',\n",
       "                                                                   n_bins=4)),\n",
       "                                                 ('cat_encoder',\n",
       "                                                  OneHotEncoder(handle_unknown='ignore',\n",
       "                                                                sparse=False))]),\n",
       "                                 ['age'])])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'fit_time': array([0.21530294, 0.07925606, 0.03249907, 0.03486705, 0.03722978]),\n",
       " 'score_time': array([0.00728607, 0.00346804, 0.00579715, 0.00444388, 0.00468612]),\n",
       " 'test_accuracy': array([0.88259669, 0.88381743, 0.88381743, 0.8824343 , 0.8824343 ]),\n",
       " 'test_precision': array([0., 0., 0., 0., 0.]),\n",
       " 'test_recall': array([0., 0., 0., 0., 0.])}"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# split data into training and testing data sets\n",
    "# use processed X\n",
    "X_train, X_test, y_train, y_test = train_test_split(X_processed, y, test_size = 0.2, random_state=seed)\n",
    "\n",
    "# instantiate the model\n",
    "l2_reg = LogisticRegression(penalty='l2', random_state=seed)\n",
    "\n",
    "# perform cross-validation on the training data\n",
    "scoring = {\n",
    "    'accuracy': 'accuracy',\n",
    "    'precision': make_scorer(precision_score, zero_division=0),\n",
    "    'recall': 'recall'\n",
    "}\n",
    "cv_scores = cross_validate(l2_reg, X_train, y_train, cv=5, scoring=scoring)\n",
    "cv_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.8830200285795069\n",
      "Precision: 0.0\n",
      "Recall: 0.0\n"
     ]
    }
   ],
   "source": [
    "# display mean accuracy, precision, and recall over the 5 folds\n",
    "print(\"Accuracy:\", cv_scores['test_accuracy'].mean())\n",
    "print(\"Precision:\", cv_scores['test_precision'].mean())\n",
    "print(\"Recall:\", cv_scores['test_recall'].mean())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:orange\">**Explain whether the model is useful for the bank to identify the customer propensity.**</span>\n",
    "\n",
    "This model achieves an accuracy of 0.88. However, Precision and Recall is 0, which gives us a hint that the model always outputs \"not adopt\" (0) and is therefore completely useless.\n",
    "\n",
    "This is a common problem amongst imbalanced datasets. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3.2 (10 points)\n",
    "\n",
    "Now add more features to the model to see if we can improve the performance (categorical features: 'housing', 'loan' and numerical features: 'day', 'duration'). Use the preprocess pipeline built previously to transform the data. \n",
    "\n",
    "Train a Logistic regression model with L1 regularization using 5-fold cross validation on the train set, by fine-tuning the hyperparameter alpha, i.e. the regularization strength from [0.001, 0.01, 0.1, 1]. Choose the correct score function that reflect the current data team's practice. Report the average score with the best hyperparameter. Does model performance improve, and if so, how?\n",
    "\n",
    "Expalin whether all features are useful for making prediction and why. List top 5 features that contribute to the prediction the most. If not all features are useful, list those unuseful features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4521, 33)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add more attributes\n",
    "num_attribs2 = [\"balance\", \"campaign\", \"day\", \"duration\"]\n",
    "cat_attribs2 = [\"job\", \"marital\", \"education\", \"default\", \"housing\", \"loan\"]\n",
    "age_attribs2 = ['age']\n",
    "\n",
    "# define pipeline to apply feature scaling to numerical features\n",
    "num_pipeline2 = Pipeline([\n",
    "        ('selector', DataFrameSelector(num_attribs2)),\n",
    "        ('std_scaler', StandardScaler())\n",
    "])\n",
    "\n",
    "# define pipeline to apply one-hot encoding to categorical values\n",
    "cat_pipeline2 = Pipeline([\n",
    "        ('selector', DataFrameSelector(cat_attribs2)),\n",
    "        ('cat_encoder', OneHotEncoder(sparse=False, handle_unknown='ignore')),\n",
    "    ])\n",
    "\n",
    "# define pipeline to convert age into the quartile categorical variable and transform it through one-hot encoding. \n",
    "age_pipeline2 = Pipeline([\n",
    "        ('selector', DataFrameSelector(age_attribs2)),\n",
    "        ('age_quantile_transform', KBinsDiscretizer(n_bins=4, encode='ordinal', strategy='quantile')),\n",
    "        ('cat_encoder', OneHotEncoder(sparse=False, handle_unknown='ignore')),\n",
    "    ])\n",
    "\n",
    "# apply different operations on num, cat, and age by combining pipelines with ColumnTransformer\n",
    "preprocessing2 = ColumnTransformer([\n",
    "        (\"num\", num_pipeline2, num_attribs2),\n",
    "        (\"cat\", cat_pipeline2, cat_attribs2),\n",
    "        (\"age\", age_pipeline2, age_attribs2)\n",
    "    ])\n",
    "\n",
    "X_processed2 = preprocessing2.fit_transform(bank)\n",
    "X_processed2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 {color: black;background-color: white;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 pre{padding: 0;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-toggleable {background-color: white;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-estimator:hover {background-color: #d4ebff;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-item {z-index: 1;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-parallel::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-parallel-item:only-child::after {width: 0;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;position: relative;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-9db0d5ff-16be-4fe1-8f81-c8ef47de0027\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;num&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;selector&#x27;,\n",
       "                                                  DataFrameSelector(attribute_names=[&#x27;balance&#x27;,\n",
       "                                                                                     &#x27;campaign&#x27;,\n",
       "                                                                                     &#x27;day&#x27;,\n",
       "                                                                                     &#x27;duration&#x27;])),\n",
       "                                                 (&#x27;std_scaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;balance&#x27;, &#x27;campaign&#x27;, &#x27;day&#x27;, &#x27;duration&#x27;]),\n",
       "                                (&#x27;cat&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;selector&#x27;,\n",
       "                                                  DataFrameSelector(attribute_names=[&#x27;job&#x27;,\n",
       "                                                                                     &#x27;marital&#x27;,\n",
       "                                                                                     &#x27;education&#x27;,\n",
       "                                                                                     &#x27;default&#x27;,\n",
       "                                                                                     &#x27;housing&#x27;,\n",
       "                                                                                     &#x27;loan&#x27;])),\n",
       "                                                 (&#x27;ca...coder&#x27;,\n",
       "                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;,\n",
       "                                                                sparse=False))]),\n",
       "                                 [&#x27;job&#x27;, &#x27;marital&#x27;, &#x27;education&#x27;, &#x27;default&#x27;,\n",
       "                                  &#x27;housing&#x27;, &#x27;loan&#x27;]),\n",
       "                                (&#x27;age&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;selector&#x27;,\n",
       "                                                  DataFrameSelector(attribute_names=[&#x27;age&#x27;])),\n",
       "                                                 (&#x27;age_quantile_transform&#x27;,\n",
       "                                                  KBinsDiscretizer(encode=&#x27;ordinal&#x27;,\n",
       "                                                                   n_bins=4)),\n",
       "                                                 (&#x27;cat_encoder&#x27;,\n",
       "                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;,\n",
       "                                                                sparse=False))]),\n",
       "                                 [&#x27;age&#x27;])])</pre><b>Please rerun this cell to show the HTML repr or trust the notebook.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"3fd209b9-20c4-4c4a-8aa1-a36330ac728c\" type=\"checkbox\" ><label for=\"3fd209b9-20c4-4c4a-8aa1-a36330ac728c\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;num&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;selector&#x27;,\n",
       "                                                  DataFrameSelector(attribute_names=[&#x27;balance&#x27;,\n",
       "                                                                                     &#x27;campaign&#x27;,\n",
       "                                                                                     &#x27;day&#x27;,\n",
       "                                                                                     &#x27;duration&#x27;])),\n",
       "                                                 (&#x27;std_scaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;balance&#x27;, &#x27;campaign&#x27;, &#x27;day&#x27;, &#x27;duration&#x27;]),\n",
       "                                (&#x27;cat&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;selector&#x27;,\n",
       "                                                  DataFrameSelector(attribute_names=[&#x27;job&#x27;,\n",
       "                                                                                     &#x27;marital&#x27;,\n",
       "                                                                                     &#x27;education&#x27;,\n",
       "                                                                                     &#x27;default&#x27;,\n",
       "                                                                                     &#x27;housing&#x27;,\n",
       "                                                                                     &#x27;loan&#x27;])),\n",
       "                                                 (&#x27;ca...coder&#x27;,\n",
       "                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;,\n",
       "                                                                sparse=False))]),\n",
       "                                 [&#x27;job&#x27;, &#x27;marital&#x27;, &#x27;education&#x27;, &#x27;default&#x27;,\n",
       "                                  &#x27;housing&#x27;, &#x27;loan&#x27;]),\n",
       "                                (&#x27;age&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;selector&#x27;,\n",
       "                                                  DataFrameSelector(attribute_names=[&#x27;age&#x27;])),\n",
       "                                                 (&#x27;age_quantile_transform&#x27;,\n",
       "                                                  KBinsDiscretizer(encode=&#x27;ordinal&#x27;,\n",
       "                                                                   n_bins=4)),\n",
       "                                                 (&#x27;cat_encoder&#x27;,\n",
       "                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;,\n",
       "                                                                sparse=False))]),\n",
       "                                 [&#x27;age&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"960ba777-71df-4d7f-9bd1-43dd1f187a69\" type=\"checkbox\" ><label for=\"960ba777-71df-4d7f-9bd1-43dd1f187a69\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">num</label><div class=\"sk-toggleable__content\"><pre>[&#x27;balance&#x27;, &#x27;campaign&#x27;, &#x27;day&#x27;, &#x27;duration&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"62e5b623-ea87-4c20-b8d0-7054204aff29\" type=\"checkbox\" ><label for=\"62e5b623-ea87-4c20-b8d0-7054204aff29\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">DataFrameSelector</label><div class=\"sk-toggleable__content\"><pre>DataFrameSelector(attribute_names=[&#x27;balance&#x27;, &#x27;campaign&#x27;, &#x27;day&#x27;, &#x27;duration&#x27;])</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"dfa8e162-a29a-482e-a6e4-731f744da243\" type=\"checkbox\" ><label for=\"dfa8e162-a29a-482e-a6e4-731f744da243\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"866d8a15-de2b-4a97-a147-2576dedb628d\" type=\"checkbox\" ><label for=\"866d8a15-de2b-4a97-a147-2576dedb628d\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">cat</label><div class=\"sk-toggleable__content\"><pre>[&#x27;job&#x27;, &#x27;marital&#x27;, &#x27;education&#x27;, &#x27;default&#x27;, &#x27;housing&#x27;, &#x27;loan&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"f945cba5-08dd-4ea0-a71d-5c7304d05705\" type=\"checkbox\" ><label for=\"f945cba5-08dd-4ea0-a71d-5c7304d05705\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">DataFrameSelector</label><div class=\"sk-toggleable__content\"><pre>DataFrameSelector(attribute_names=[&#x27;job&#x27;, &#x27;marital&#x27;, &#x27;education&#x27;, &#x27;default&#x27;,\n",
       "                                   &#x27;housing&#x27;, &#x27;loan&#x27;])</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"d321ea09-487c-42f5-a2db-2917e77199eb\" type=\"checkbox\" ><label for=\"d321ea09-487c-42f5-a2db-2917e77199eb\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;, sparse=False)</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"8a8e4c1d-fc25-4e37-92d6-c96839d46372\" type=\"checkbox\" ><label for=\"8a8e4c1d-fc25-4e37-92d6-c96839d46372\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">age</label><div class=\"sk-toggleable__content\"><pre>[&#x27;age&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"6960b8cd-6413-417e-9457-3d8bef47fa5b\" type=\"checkbox\" ><label for=\"6960b8cd-6413-417e-9457-3d8bef47fa5b\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">DataFrameSelector</label><div class=\"sk-toggleable__content\"><pre>DataFrameSelector(attribute_names=[&#x27;age&#x27;])</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"f383b9f3-0a91-4783-b7b4-8806f304e2af\" type=\"checkbox\" ><label for=\"f383b9f3-0a91-4783-b7b4-8806f304e2af\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">KBinsDiscretizer</label><div class=\"sk-toggleable__content\"><pre>KBinsDiscretizer(encode=&#x27;ordinal&#x27;, n_bins=4)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"89c81e8e-6034-456b-9c28-1710e1a670f6\" type=\"checkbox\" ><label for=\"89c81e8e-6034-456b-9c28-1710e1a670f6\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;, sparse=False)</pre></div></div></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('num',\n",
       "                                 Pipeline(steps=[('selector',\n",
       "                                                  DataFrameSelector(attribute_names=['balance',\n",
       "                                                                                     'campaign',\n",
       "                                                                                     'day',\n",
       "                                                                                     'duration'])),\n",
       "                                                 ('std_scaler',\n",
       "                                                  StandardScaler())]),\n",
       "                                 ['balance', 'campaign', 'day', 'duration']),\n",
       "                                ('cat',\n",
       "                                 Pipeline(steps=[('selector',\n",
       "                                                  DataFrameSelector(attribute_names=['job',\n",
       "                                                                                     'marital',\n",
       "                                                                                     'education',\n",
       "                                                                                     'default',\n",
       "                                                                                     'housing',\n",
       "                                                                                     'loan'])),\n",
       "                                                 ('ca...coder',\n",
       "                                                  OneHotEncoder(handle_unknown='ignore',\n",
       "                                                                sparse=False))]),\n",
       "                                 ['job', 'marital', 'education', 'default',\n",
       "                                  'housing', 'loan']),\n",
       "                                ('age',\n",
       "                                 Pipeline(steps=[('selector',\n",
       "                                                  DataFrameSelector(attribute_names=['age'])),\n",
       "                                                 ('age_quantile_transform',\n",
       "                                                  KBinsDiscretizer(encode='ordinal',\n",
       "                                                                   n_bins=4)),\n",
       "                                                 ('cat_encoder',\n",
       "                                                  OneHotEncoder(handle_unknown='ignore',\n",
       "                                                                sparse=False))]),\n",
       "                                 ['age'])])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessing2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best hyperparameter alpha: {'C': 0.1}\n",
      "Mean cross-validated score of the best estimator: 0.5955192189547527\n",
      "\n",
      "Coefficients\n",
      "[[ 0.02060661 -0.23432841  0.          0.965484    0.         -0.340325\n",
      "   0.          0.          0.          0.20871636  0.          0.\n",
      "   0.          0.          0.          0.          0.         -0.20962352\n",
      "   0.          0.         -0.0162182   0.17158427  0.         -0.34468764\n",
      "   0.          0.         -0.74461899  0.         -0.51525403  0.\n",
      "   0.         -0.10028963  0.        ]]\n"
     ]
    }
   ],
   "source": [
    "# split data into training and testing data sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X_processed2, y, test_size = 0.2, random_state=seed)\n",
    "\n",
    "# instantiate the model\n",
    "l1_reg = LogisticRegression(penalty='l1', random_state=seed, solver='liblinear')\n",
    "\n",
    "# Use the GridSearchCV object with cross_val_score to perform cross-validation and hyperparameter tuning\n",
    "params = {'C': [0.001, 0.01, 0.1, 1]}\n",
    "grid_search = GridSearchCV(l1_reg, params, cv=5, scoring=\"precision\", refit=True).fit(X_train, y_train)\n",
    "\n",
    "# display best hyperparameters and average score on the test set\n",
    "print(\"Best hyperparameter alpha:\", grid_search.best_params_)\n",
    "print(\"Mean cross-validated score of the best estimator:\", grid_search.best_score_)\n",
    "\n",
    "print(\"\\n\"+\"Coefficients\")\n",
    "print(grid_search.best_estimator_.coef_)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:orange\">**Scoring the model**</span>\n",
    "The current practice of many data teams is to build such propensity models and predict customer's probability to adopt the product and target them from the highest probability to the lowest probability.\n",
    "\n",
    "It is likely that the data team is interested in maximizing **precision** (out of the customers predicted to adopt, how many actually adopt) as targeting customers with a high probability of adopting the product is to be costly, and they want to minimize the number of customers who are targeted but do not end up adopting the product.\n",
    "\n",
    "<span style=\"color:orange\">**Report the average score with the best hyperparameter.**</span>\n",
    "\n",
    "The best hyperparameter was 0.1\n",
    "The best score (precision) was 0.6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['balance', 'campaign', 'day', 'duration', 'job_admin.',\n",
       "       'job_blue-collar', 'job_entrepreneur', 'job_housemaid',\n",
       "       'job_management', 'job_retired', 'job_self-employed',\n",
       "       'job_services', 'job_student', 'job_technician', 'job_unemployed',\n",
       "       'job_unknown', 'marital_divorced', 'marital_married',\n",
       "       'marital_single', 'education_primary', 'education_secondary',\n",
       "       'education_tertiary', 'education_unknown', 'default_no',\n",
       "       'default_yes', 'housing_no', 'housing_yes', 'loan_no', 'loan_yes',\n",
       "       'age_0.0', 'age_1.0', 'age_2.0', 'age_3.0'], dtype=object)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get feature names for numerical attributes\n",
    "num_features = np.array(num_attribs2)\n",
    "\n",
    "# get feature names for categorical attributes\n",
    "cat_encoder = preprocessing2.named_transformers_['cat'].named_steps['cat_encoder']\n",
    "cat_features = cat_encoder.get_feature_names(cat_attribs2)\n",
    "\n",
    "# get feature names for age attribute\n",
    "age_encoder = preprocessing2.named_transformers_['age'].named_steps['cat_encoder']\n",
    "age_features = age_encoder.get_feature_names(age_attribs2)\n",
    "\n",
    "# concatenate all feature names\n",
    "feature_names = np.concatenate((num_features, cat_features, age_features), axis=0)\n",
    "feature_names"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:orange\">**Finding the top 5 features**</span>\n",
    "\n",
    "The larger the magnitude of the coefficient (whether positive or negative), the more important the corresponding feature is for predicting the target variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>feature</th>\n",
       "      <th>coef</th>\n",
       "      <th>coef_abs</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>duration</td>\n",
       "      <td>0.965484</td>\n",
       "      <td>0.965484</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>housing_yes</td>\n",
       "      <td>-0.744619</td>\n",
       "      <td>0.744619</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>loan_yes</td>\n",
       "      <td>-0.515254</td>\n",
       "      <td>0.515254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>default_no</td>\n",
       "      <td>-0.344688</td>\n",
       "      <td>0.344688</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>job_blue-collar</td>\n",
       "      <td>-0.340325</td>\n",
       "      <td>0.340325</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>campaign</td>\n",
       "      <td>-0.234328</td>\n",
       "      <td>0.234328</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>marital_married</td>\n",
       "      <td>-0.209624</td>\n",
       "      <td>0.209624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>job_retired</td>\n",
       "      <td>0.208716</td>\n",
       "      <td>0.208716</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>education_tertiary</td>\n",
       "      <td>0.171584</td>\n",
       "      <td>0.171584</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>age_2.0</td>\n",
       "      <td>-0.100290</td>\n",
       "      <td>0.100290</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>balance</td>\n",
       "      <td>0.020607</td>\n",
       "      <td>0.020607</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>education_secondary</td>\n",
       "      <td>-0.016218</td>\n",
       "      <td>0.016218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>housing_no</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>default_yes</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>education_primary</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>education_unknown</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>loan_no</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>age_0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>age_1.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>marital_divorced</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>marital_single</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>job_unknown</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>job_unemployed</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>job_technician</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>job_student</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>job_services</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>job_self-employed</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>job_management</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>job_housemaid</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>job_entrepreneur</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>job_admin.</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>day</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>age_3.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                feature      coef  coef_abs\n",
       "3              duration  0.965484  0.965484\n",
       "26          housing_yes -0.744619  0.744619\n",
       "28             loan_yes -0.515254  0.515254\n",
       "23           default_no -0.344688  0.344688\n",
       "5       job_blue-collar -0.340325  0.340325\n",
       "1              campaign -0.234328  0.234328\n",
       "17      marital_married -0.209624  0.209624\n",
       "9           job_retired  0.208716  0.208716\n",
       "21   education_tertiary  0.171584  0.171584\n",
       "31              age_2.0 -0.100290  0.100290\n",
       "0               balance  0.020607  0.020607\n",
       "20  education_secondary -0.016218  0.016218\n",
       "25           housing_no  0.000000  0.000000\n",
       "24          default_yes  0.000000  0.000000\n",
       "19    education_primary  0.000000  0.000000\n",
       "22    education_unknown  0.000000  0.000000\n",
       "27              loan_no  0.000000  0.000000\n",
       "29              age_0.0  0.000000  0.000000\n",
       "30              age_1.0  0.000000  0.000000\n",
       "16     marital_divorced  0.000000  0.000000\n",
       "18       marital_single  0.000000  0.000000\n",
       "15          job_unknown  0.000000  0.000000\n",
       "14       job_unemployed  0.000000  0.000000\n",
       "13       job_technician  0.000000  0.000000\n",
       "12          job_student  0.000000  0.000000\n",
       "11         job_services  0.000000  0.000000\n",
       "10    job_self-employed  0.000000  0.000000\n",
       "8        job_management  0.000000  0.000000\n",
       "7         job_housemaid  0.000000  0.000000\n",
       "6      job_entrepreneur  0.000000  0.000000\n",
       "4            job_admin.  0.000000  0.000000\n",
       "2                   day  0.000000  0.000000\n",
       "32              age_3.0  0.000000  0.000000"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "coef = grid_search.best_estimator_.coef_[0]\n",
    "coef_df = pd.DataFrame({'feature': feature_names, 'coef': coef, 'coef_abs': np.abs(coef) })\n",
    "coef_df_sorted = coef_df.sort_values('coef_abs', ascending=False)\n",
    "\n",
    "coef_df_sorted"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The top 5 features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>feature</th>\n",
       "      <th>coef</th>\n",
       "      <th>coef_abs</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>duration</td>\n",
       "      <td>0.965484</td>\n",
       "      <td>0.965484</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>housing_yes</td>\n",
       "      <td>-0.744619</td>\n",
       "      <td>0.744619</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>loan_yes</td>\n",
       "      <td>-0.515254</td>\n",
       "      <td>0.515254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>default_no</td>\n",
       "      <td>-0.344688</td>\n",
       "      <td>0.344688</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>job_blue-collar</td>\n",
       "      <td>-0.340325</td>\n",
       "      <td>0.340325</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            feature      coef  coef_abs\n",
       "3          duration  0.965484  0.965484\n",
       "26      housing_yes -0.744619  0.744619\n",
       "28         loan_yes -0.515254  0.515254\n",
       "23       default_no -0.344688  0.344688\n",
       "5   job_blue-collar -0.340325  0.340325"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "coef_df_sorted.head(5)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:orange\">**Not all features are useful**</span>\n",
    "Some features were removed by l1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>feature</th>\n",
       "      <th>coef</th>\n",
       "      <th>coef_abs</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>housing_no</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>default_yes</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>education_primary</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>education_unknown</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>loan_no</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>age_0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>age_1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>marital_divorced</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>marital_single</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>job_unknown</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>job_unemployed</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>job_technician</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>job_student</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>job_services</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>job_self-employed</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>job_management</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>job_housemaid</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>job_entrepreneur</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>job_admin.</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>day</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>age_3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              feature  coef  coef_abs\n",
       "25         housing_no   0.0       0.0\n",
       "24        default_yes   0.0       0.0\n",
       "19  education_primary   0.0       0.0\n",
       "22  education_unknown   0.0       0.0\n",
       "27            loan_no   0.0       0.0\n",
       "29            age_0.0   0.0       0.0\n",
       "30            age_1.0   0.0       0.0\n",
       "16   marital_divorced   0.0       0.0\n",
       "18     marital_single   0.0       0.0\n",
       "15        job_unknown   0.0       0.0\n",
       "14     job_unemployed   0.0       0.0\n",
       "13     job_technician   0.0       0.0\n",
       "12        job_student   0.0       0.0\n",
       "11       job_services   0.0       0.0\n",
       "10  job_self-employed   0.0       0.0\n",
       "8      job_management   0.0       0.0\n",
       "7       job_housemaid   0.0       0.0\n",
       "6    job_entrepreneur   0.0       0.0\n",
       "4          job_admin.   0.0       0.0\n",
       "2                 day   0.0       0.0\n",
       "32            age_3.0   0.0       0.0"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "coef_df_sorted[coef_df_sorted[\"coef\"] == 0]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:orange\">**The model performance did improve**</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision of previous model: 0.0\n"
     ]
    }
   ],
   "source": [
    "print(\"Precision of previous model:\", cv_scores['test_precision'].mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision of current model: 0.5955192189547527\n"
     ]
    }
   ],
   "source": [
    "print(\"Precision of current model:\", grid_search.best_score_)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3.3 (10 points)\n",
    "\n",
    "Now use the best model found in the cross-validation to predict the test set, show the obtained confusion matrix. Assume that targeting each customer would cost 10 euros and if the customer subscribe, the company would earn 50 euros. If we perform targeted telemarketing to all customers that are predicted to subscribe in the test set, what's the resulting profit?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 {color: black;background-color: white;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 pre{padding: 0;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-toggleable {background-color: white;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-estimator:hover {background-color: #d4ebff;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-item {z-index: 1;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-parallel::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-parallel-item:only-child::after {width: 0;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;position: relative;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-63dad6b6-3761-4219-81e5-2de14bfcbc25 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-63dad6b6-3761-4219-81e5-2de14bfcbc25\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LogisticRegression(C=0.1, penalty=&#x27;l1&#x27;, random_state=42, solver=&#x27;liblinear&#x27;)</pre><b>Please rerun this cell to show the HTML repr or trust the notebook.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"f135d02a-e010-4b0e-aac3-2cd79301cc47\" type=\"checkbox\" checked><label for=\"f135d02a-e010-4b0e-aac3-2cd79301cc47\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression(C=0.1, penalty=&#x27;l1&#x27;, random_state=42, solver=&#x27;liblinear&#x27;)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "LogisticRegression(C=0.1, penalty='l1', random_state=42, solver='liblinear')"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_model = grid_search.best_estimator_\n",
    "y_pred = best_model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAEGCAYAAABFBX+4AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAzRklEQVR4nO3dd3gUVdvA4d+zSSihBwEhIEVCR5BmQZEOokhvItIUFAG7BFCwIdg/9cWCAiIIGKUXaZEqIESKUgXpvUMghWxyvj92iAGSZQPZbCZ57veaa2fPzJw5kzc+OZw5RYwxKKWUsg+HrwuglFIqdTRwK6WUzWjgVkopm9HArZRSNqOBWymlbMbf1wVISc67+2t3F3Wds+v/5+siqAwohz9yq3mkJuZEb/zfLd/vVmTYwK2UUulK7NMAoYFbKaUAxKeV6FTRwK2UUqA1bqWUsh2tcSullM04/HxdAo9p4FZKKdCmEqWUsh1tKlFKKZvRGrdSStmM1riVUspmtMatlFI2o71KlFLKZrTGrZRSNuPQNm6llLIXrXErpZTNaK8SpZSyGX05qZRSNqNNJUopZTPaVKKUUjajNW6llLIZrXErpZTNaI1bKaVsxka9SuzzJ0YppbxJHJ5v7rIRKS8im5JsF0TkBREJEpHFIrLL+iyQ5JrBIrJbRHaKSLMbFVUDt1JKgauN29PNDWPMTmNMdWNMdaAmEAXMAEKBcGNMCBBufUdEKgGdgcpAc+BLEXFb/dfArZRSkGY17ms0Av41xuwHWgETrPQJQGtrvxUw1RgTa4zZC+wG6rjLVAO3UkpBqmrcItJHRCKSbH1SyLUzMMXaL2KMOQpgfRa20oOBg0muOWSlpUhfTiqlFKSqJm2MGQOMcZudSDbgMWDwje6c3C3cXaCBWymlAHGkeQPEw8AGY8xx6/txESlqjDkqIkWBE1b6IaBEkuuKA0fcZaxNJUopBYirCcSjzUNd+K+ZBGA20N3a7w7MSpLeWUSyi0hpIARY5y5jrXErpRQk32Bxs1mJBAJNgL5JkkcBYSLSGzgAdAAwxmwVkTBgG+AEnjPGxLvLXwO3UkpBamrSN2SMiQIKXpN2Glcvk+TOHwGM8DR/DdxKKUXaBm5v08CtlFKAI+1fTnqNBm6llII0beP2Ng3cSimFNpUopZTtaOBWSimb0cCtlFI2o4E7iwrKl4v53wwAoEjBvCQkJHDy7EUAHnziQ+KcbvvUe2Tht8+TKzA7D3T9AIAale5g5IttaPb0Z7ect/KOu6tWJCSkXOL3T78YTXBw8WTPvbfW3ayN2HhL93tjSCgREevIkzsP4nAw5PVhVKt+9y3lmRWIQwN3lnTm/CXu7TwKgKF9W3ApKpb/mxieeNzPz0F8fMIt36dwgdw0rVuJRb9vu+W8lPdlz56DsOmzbnxiGnrp5ddo0qw5q39fxTtvDeOXGXPS9f52pDVulWjMW09w9kIU1coXZ9OOg0Reir0qoEf8PIS2A7/mwNEzdG5Rm+e6PERAgD/r/97H8yN/IiHh+knCPv0hnNCnml0XuB0O4d2BrahXK4RsAf58E7aCsdN+R0T4NLQDD9YMYd/h0zgcwg+z1jBjyab0+BGoa0RdusTzA/px4cIFnE4n/Qc+T4OGja865+TJE7z28otcungRZ3w8rw97kxo1a7H691V8NfoLLl++TIkSJXj73ZEE5sqV4r1q1qrNwQMHAPjh+/HMnDENgLbt2vPEkz2IioritZdf4PixY8QnJNDnmX40f7iF9x4+A9PAra5S9o7CtHjmCxISDEP7Jv8fRfnSRWjftAYNen6C05nA/w3uSOcWtZk89/q5Zv74ay+PNbiLerVCuBgVm5jeo/X9nL8YzQNPfEi2AH9++/4llqzZQY1KJShZrCC1OrxH4aDcbJz+Bj/MWuO151VXi42NoWPbVgAUK16cjz75jE8/H03u3Lk5e/YM3bp0on6DRlcFjvnz5nJ/3Qd4uu+zxMfHExMTzdmzZ/j2m6/45rvxBAYGMu67MfwwYTzP9Ouf4r2XL/uNsiHl2LZ1C7NmTmfSlDAwhq5dOlKzdh0OHzxIoUKF+d9XrhlKIyMjvfvDyMjsE7c1cKeH6Us2JltzTqpBnfLUqHQHqya9BkDO7AGcPHMxxfNHfbeQ0Kea8/rn//0TvPF9FagSEkybxq72zHy5c1D2jkLcX/1Opi/eiDGG46cjWbH+nzR4KuWpa5tK4uLi+Pz/PmHDn+txiIMTJ45z+tQpbitUKPGcKlWqMvz1ITidTho0bEyFihWJWL+UPf/upscTXRLzuat69WTv+cnHH/DtN19RICiIN98Zwbq1a2jYqDGBgYEANGrchA1/RlD3gQf5+KP3+fTjD3mofgNq1KzlvR9EBqc1bnWVqOj/asXO+HgcSV6C5MgWALh+aSbN+YNhX8z2KM/l6/9heL9HqFO1VGKaiPDS+z+zZM32q859+MHKt1B6ldbmz53D2bNnmBI2nYCAAB5u0pDYy7FXnVOzVm3G/TCJlcuXM3Twa/To2Zs8efNy7311ef+jT254jytt3Ff8sWZ1sueVKlWaqWHTWblyOZ99+jH33V/XbQ0+M7NT4LbP4PxMYv+RM1Sv6JozvXqF4pQKdk0gtnTdTto0rk6hArkBKJA3kDuKFkgxH4D3xy7kpe7/tY0uXr2dPh0ewN/f9X9r2TsKE5gjG6s37qF1o+qICIWD8vBgrRBvPJry0MWLkQQFFSQgIIB1f6zlyJHD151z5MhhgoIK0q5DR9q0bcf2bVu5q1p1Nm3cwIH9+wGIjo5m3769Ht2zZq3aLP1tCdHR0URFRfFb+BJq1KzFiRPHyZEzJ4+2bEX3nr3ZsT3rvvB2OBweb76mNe50NjN8E10frcPaqaH8uXU/u/a7FsHYsecYb42ey5yv+uMQIc4Zz4ujwjhw9GyKeS1ctS2xuyHA+BmrKVksiDWTQxGBU2cv0vGlMcwI30T9e8rz5y9D2L3/BOu37ON8ZIzXn1Ulr8WjLRn43LN06diW8hUqUrpMmevOiVi3ju/Hj8Xf35/AwEDeHfk+QUFBvD1iJKGvvsTluMsA9B/wAqVKlb7hPStWqsxjrdrStXMHwPVysmLFSvy+aiWffvwBDnHg7+/P0GFvpumz2op9KtyIMe7bXn0l5939M2bBbCpXzmxcir5MUL5crJz4Cg17fsLx0/Z7EXV2/f98XQSVAeXwv/WwG/zsDI9jzuGv2vg0zGuNO4uY/vmz5MuTk2wBfoz8doEtg7ZS3mSnNm4N3FmEjqxUyj0N3EopZTM65F15LKRkYSa+3yvxe+nggrzz1TyWR+zii6GdyZUzO/uPnKbn0AlEXnK9UHylV1N6tLqP+IQEXv7gl+u6/6nMZ9jrg1mxfBlBQQWZPmsuAK++/AL797p6lURGRpInT550H1qfmaRljVtE8gPfAVUAA/QCdgI/AaWAfUBHY8xZ6/zBQG8gHhhojFnoLn8N3D62a/+JxPlNHA7h34UjmL10M5M/fIrQT2ew6s/dPNnqXl7s3oi3v5xHhTK306FZDWq0H0HRQvmY/3V/qrZ++4YDfJS9tWrdli6PP8HQwYMS0z78+P8S9z/6YBS5c+f2QckyjzRuKvkMWGCMaS8i2YBAYAgQbowZJSKhQCgwSEQqAZ2BykAxYImIlHO30rvvOySqRA3qlGfvoZMcOHqWkJKFWfXnbgB+W7uD1o2qA/Bo/bv4eeEGLsc52X/kNP8ePEXtKqV8V2iVLmrWqk3efPmSPWaMYdHCX3n4kUfTuVSZi4h4vN0gn7xAPWAsgDHmsjHmHNAKmGCdNgFobe23AqYaY2KNMXuB3UAdd/fwWuAWkQoiMkhEPheRz6z9it66X2bQoVlNwhb8CcC2f4/yaP2qALRtUoPiRVyDcYIL5ePQsf/6dh8+cZZihZP/D1plDRv+jKBgwYKULFnK10WxN/F8E5E+IhKRZOuTJKcywElgvIhsFJHvRCQXUMQYcxTA+ixsnR8MHExy/SErLUVeCdwiMgiYiusx1wHrrf0p1j8RUrou8YfhPLXVG0XLsAL8/XjkoapMX+yai7nvmz/St2M9fv/xNXIHZudynPWvpmT+2mfQrvgqnfw6fy7NW2ht+1alpsZtjBljjKmVZBuTJCt/oAbwlTHmbuASrmaRFG+dTJrb/6q91cbdG6hsjIlLmiginwBbgVHJXWQ9/BjIegNwmj1QiU07DnLijKt/9T/7jtOy32jANXT9ynwjh0+co/jt/w2FDy5cgKMnz6d/gVWG4HQ6CV+ymKlh031dFNtzpF2vkkPAIWPMH9b3X3AF7uMiUtQYc1REigInkpxfIsn1xYEjbsuaViW9RgKuRvZrFbWOqWt0bF4rsZkESJyzREQIfboZ3/6yCoB5y/6iQ7MaZAvwp2SxgpS9oxDrt+zzRZFVBvDHmtWULl2GIrff7uui2F5atXEbY44BB0WkvJXUCNgGzAa6W2ndgStdgGYDnUUku4iUBkJwtVSkyFs17heAcBHZxX9tN3cAZYGsOfWYGzlzBNDwngr0f3dKYlrH5rXo26keALN+28QPs9YCsH3PMaYt2sjGaUNxxifwwqgw7VGSBQx65SUi1q/j3LmzNGlYj2efG0Dbdh1Y8Ot8mrd4xNfFyxTSePzNAOBHq0fJHqAnropymIj0Bg4AHQCMMVtFJAxXcHcCz7nrUQJenKtERBy43owG42rDOQSsv1GBrshqTSXKMzpXiUpOWsxVUn7QQo9jzs73m2XOuUqMMQnAWm/lr5RSaclGI951AI5SSkGavpz0Og3cXjagawN6tLkfYwxbdx+hz/BJfPd2N0JKFQEgf56cnIuMThw9eUXxIvn57p0nKVIwLwnGMG7a74yesgyAquWCkx0Of1+1Mnw2pBOX45w8OXg8ew6eIl/unEx8vxePPTc6vR9deSC5oexJRUZGMmTQqxw7egRnfDzde/aidZt2AEyc8D3Tp/2MiBASUo63R4wke/bsfPrxh/y+agXlK1RkxMgPAJgzeyYXzp+na7fu191DudgpcOvISS8qVigf/bo8RN2uH1Crw3v4ORx0aFaTbqHjubfzKO7tPIqZ4ZuY9dum6651xicQ+sl07m73Lg89+RF9O9WjQhlXz4Gvhj3O65/PonbH95i9dDMvdm8EwPPdGtLl1e8Y9sUc+nR4EIDBfZrzwTi30x4oH2rVui1fffNdisd/mvIjZe68k59nzGbs9xP5+IP3ibt8mePHjzP5xx+YEjaN6bPmkpAQz4L584iMjGTzpo38MmMOCfHx7PpnJzExMcyeOYOOnR9PxyezHxHPN1/TwO1l/n5+5MwegJ+fg5w5sl3X57pdkxpXdQO84tipC2zacQiAi1Gx7Nh7jGKF8gOkOBw+zhlPzuwBBOYMIM4ZT+nit1GscP7Ec1XG424oO7i6qEVduoQxhqioS+TLlw8/f9c/lOPj44mNicHpdBIdE0OhwoVxOIS4uDiMMcTExuLv78/3477j8Se6ERAQkF6PZUtp1R0wPWjg9qIjJ8/zfz+E88+v77B38QguXIwmfO2OxON1a9zJ8TOR/HvgpNt87igaRPXyxRP7a6c0HP7DcYsY/XoX+j/egK+nruCt/i1568vr//mt7KPz413Zs+dfGtd/kPatH+O1wUNxOBwUKVKE7j160axxAxrXf4A8uXNzf90HyJUrN42bNKVTu9YEBxcnd548bN2yhQYNG9/4ZlmcBm4FuNqvH61flYqPDqdM06HkypmNzi1qJx7v2LwWPy+IcJtHrpzZmPLRU7z60bTEaV1TGg7/1z+Heaj7xzTv8zmlihfk6MnzCMLEUT0Z9+6TFA7K472HVV6xetUqKlSoyJJlKwmbNpORI97m4sWLXDh/nqW/hTN/UTiLl64kOjqauXNc4zl69n6asOmzeOW1UEZ/8Rn9Bgxk+i8/8+pLzzPm6y99/EQZlzaVKAAa3lOBfUdOc+rsRZzOBGb+tpl7q7kWdvXzc9CqYTV+Wbghxev9/R1M+ehpfvo1glm/bU5MvzIcvm7XDwhb8Cd7D11fYw99qjkjx/zK0L4P887X85kyfz39utRP82dU3jVr5nQaNWmKiHBHyZIEBxdn7549rF27muDixQkKCiIgIIBGjZuyeePGq67dbq3YXrJkKebMnsmHn3zG7t272L9/nw+eJONzOMTjzdc0cHvRwWNnqFO1NDlzuNoWG9Qpz869xwFoeE95/tl3nMMnzqV4/dfDu7Jz7zE+n/TbVekpDYe/4omW97Bg5VbORUYTmCMbCQmGhARDYA5t47Sb24sW5Y+1awA4feoU+/btpXiJ4txetBh/bd5MdHQ0xhj+WLuG0nfeedW1o7/4jH79B+J0OkmId/2rzCEOYqJj0v057MBOTSXaHdCL1m/Zz4wlG1kzeRDO+AQ27zjE2Gm/A1dP4XpF0UL5+HLY47QZ8BX3Vy9D10fv4e9/DrN2qmtiseH/m83CVdtSHA4PruHzT7S8h0f7uUYYfj7pN6Z89BSX45x0H/x9Ojy1So3khrI7nU4AOnbqQp9n+vHG0MG0a90SYwwvvPQKBQoEUaBAEE2aNqNzhzb4+flToWJF2nfolJjvb+FLqFKlKoULu7qd3lX9btq1bkm5cuUoX6GCT541o8sA8dhjXhvyfqt0yLtKjg55V8lJiyHvtUcs8zjmrB9aP3MOeVdKKTuxU41bA7dSSpHma056lQZupZTCXkPeNXArpRTaVKKUUrajTSVKKWUzNorbqQvcIlIAKGGM+ctL5VFKKZ+wU437hiMnRWSZiOQVkSBgMzDeWq1dKaUyDTuNnPRkyHs+Y8wFoC0w3hhTE9CpxpRSmUpazlUiIvtE5G8R2SQiEVZakIgsFpFd1meBJOcPFpHdIrJTRJrdsKwePI+/iBQFOgI6R6hSKlPywuyADYwx1Y0xtazvoUC4MSYECLe+IyKVgM5AZaA58KWI+LnL2JPA/TawENhtjFkvImWAXR4XXSmlbCAdmkpaAROs/QlA6yTpU40xscaYvcBuoI67jG4YuI0xPxtj7jLG9LO+7zHGtLvZkiulVEaUmhq3iPQRkYgkW59rsjPAIhH5M8mxIsaYowDWZ2ErPRg4mOTaQ1ZailLsVSIiX1g3T5YxZqC7jJVSyk4cqahJG2PGAGPcnFLXGHNERAoDi0Vkh5tzk7ux2wmv3HUHdL80i1JKZSJpOeTdGHPE+jwhIjNwNX0cF5Gixpij1nvDE9bph4ASSS4vDhxxl3+KgdsYMyHpdxHJZYy5dBPPoJRSGV5axW0RyQU4jDGR1n5TXO8KZwPdgVHW5yzrktnAZKubdTEgBFjn7h43HIAjIvcBY4HcwB0iUg3oe6XNWymlMoM07J9dBJhh5ecPTDbGLBCR9UCYiPQGDgAdAIwxW0UkDNgGOIHnjDHx7m7gycjJ/wOa4fqrgDFms4jUu7nnUUqpjCmt4rYxZg9QLZn000CjFK4ZAYzw9B4eDXk3xhy85q+R278GSillN3Lri+ikG08C90ERuR8wIpINGAhs926xlFIqfdloOm6PAvczwGe4+hUexjUY5zlvFkoppdJbplpIwRhzCuiaDmVRSimfSU0/bl/zZHbAMiIyR0ROisgJEZllDXtXSqlMwwtzlXiNJ3OVTAbCgKK4+hj+DEzxZqGUUiq9ZbZpXcUYM9EY47S2SdxgOKZSStmNnWrc7uYqCbJ2l4pIKDAVV8DuBMxLh7IppVS68csIEdlD7l5O/okrUF95mr5JjhngHW8VSiml0ltGaALxlLu5SkqnZ0GUUsqXbNQb0LORkyJSBagE5LiSZoz5wVuFUkqp9JYpatxXiMhwoD6uwD0feBhYBWjgVkplGjaK2x71KmmPa2KUY8aYnrgmT8nu1VIppVQ6s1N3QE+aSqKNMQki4hSRvLgm/9YBOEqpTMXPRo3cngTuCBHJD3yLq6fJRW4wybdSStmNfcK2Z3OVXFkw4WsRWQDkBU55tVRKKZXO7DRXiUe9Sq4wxuwDEJEDwB3eKJBSSvmCjeJ26gJ3EjZ6RKWUurGM8NLRUzcbuHWuEqVUpmKjuO12rpIvSD5AC5DfWwVSSilfSOteJSLiB0QAh40xj1rzP/0ElAL2AR2NMWetcwcDvXEtCznQGLPQXd7uatwRN3lMKaVsxwtNJc/jWuYxr/U9FAg3xoyyJu4LBQaJSCWgM1AZ19TZS0SknLuV3t3NVTIhrUp/M/Yu/9SXt1dKZTGejEb0lIgUBx7BtXL7S1ZyK1yj0AEmAMuAQVb6VGNMLLBXRHYDdYA16VFWpZSyrdSMnBSRPiISkWTrc012/we8BiQkSStijDkKYH0WttKDgYNJzjtkpaXoZl9OKqVUppKaJm5jzBhgTHLHRORR4IQx5k8Rqe9Bdsnd2W0HEA3cSilFmr6crAs8JiItcM2omldEJgHHRaSoMeaoiBTFNX0IuGrYJZJcXxw44u4GN9OrBABjzEDPnkEppTK+tIrbxpjBwGAAq8b9ijHmCRH5EOgOjLI+Z1mXzAYmi8gnuF5OhnCDaUVutleJUkplKunQj3sUECYivYEDQAcAY8xWEQkDtgFO4Dl3PUrAtRCwtwt7U45diMuYBVM+lT8wwNdFUBlQDv9bH80dOv8fj2POqBblfDpcx5OFFArh6rJy7Qo4Db1YLqWUSld26mLnSVl/xNWJvDTwFq4RP+u9WCallEp3Ip5vvuZJ4C5ojBkLxBljlhtjegH3erlcSimVrvwc4vHma550B4yzPo+KyCO4uqkU916RlFIq/WWAeOwxTwL3uyKSD3gZ+ALXuPsXvVoqpZRKZ5lqIQVjzFxr9zzQwLvFUUop37BR3PaoV8l4khmIY7V1K6VUppDZmkrmJtnPAbThBsMxlVLKbsRGC3t50lQyLel3EZkCLPFaiZRSygf8bdSR+2YmmQpBFwpWSmUymWrNSRGJ5Oo27mO4RlIqpVSmkanauI0xedKjIEop5Us2qnDfeOSkiIR7kqaUUnbmEPF48zV383HnAAKB20SkAP+t0pAX15yxSimVafhlkpeTfYEXcAXpP/kvcF8ARnu3WEoplb4cmaE7oDHmM+AzERlgjPkiHcuklFLpLgO0gHjMk38cJIhI/itfRKSAiPTzXpGUUir9OcTzzdc8CdxPG2POXflijDkLPO21EimllA9kipeTSThERIy1xpmI+AHZvFsspZRKXxkgHnvMkxr3QlwLXDYSkYbAFGCBd4ullFLpK60WUhCRHCKyTkQ2i8hWEXnLSg8SkcUissv6LJDkmsEisltEdopIsxuV1ZPAPQgIB54FnrP2X/XgOqWUsg1HKrYbiAUaGmOqAdWB5iJyLxAKhBtjQnDF0VAAEakEdAYqA82BL62WDbdldcsYk2CM+doY094Y0w7YimtBBaWUyjRExOPNHeNy0foaYG0GaAVMsNInAK2t/VbAVGNMrDFmL7AbqOPuHh51OReR6iLyvojsA94BdnhynVJK2YWkZhPpIyIRSbY+V+Ul4icim4ATwGJjzB9AEWPMUQDrs7B1ejBwMMnlh6y0FLkbOVkOV/W9C3Aa+AkQY4yugqOUynRS01vEGDMGGOPmeDxQ3epKPUNEqrjJLrkbX7d4TVLuepXsAFYCLY0xuwFERNeaVEplSt7oVGKMOSciy3C1XR8XkaLGmKMiUhRXbRxcNewSSS4rzg0Wq3HXVNIO1xSuS0XkWxFphHeeTSmlfM7hEI83d0Sk0JVBiyKSE2iMqyI8G+hundYdmGXtzwY6i0h2ESmNa82Dde7u4W7I+wxcVfxcuBrRXwSKiMhXwAxjzCK3pVdKKRtJwzmmigITrJ4hDiDMGDNXRNbg6lrdGzgAdAAwxmwVkTBgG+AEnrOaWlIk1rgaj4hIkHWzTsaYhjfzRJ46diHO84KpLCN/YICvi6AyoBz+t94aELbpiMcxp2P1Yj5tfUjVHxljzBljzDfeDtpKKZXeUtOrxNduZs1JpZTKdDLVmpNKKZUV+GngVkope7FP2NbAnWYa3HMXZe4MSfz+7kefU7RY8oOfmterzYIV62/pfiPfHErEujVMmbmAbNmyce7cWfo+2YmfZmtnn4zo3Lmz9OnVA4BTp07h8HMQVCAIgB+n/kxAtlufcLN3j26cPHmC7NmyExgYyFvvvkep0mVuOd+swkYVbg3caSV79uyMnTwtXe/pcDiYP3s6rdt3Ttf7qtTLn78AYdNd3Xa/Gv0FgYGBdO/ZO/G40+nE3//W/3Mc+f5HVK5SlV/CfuKTjz7g89Ff33KeWUWmWLpM3ZqoqCiGvjyAyMgLOJ1Onnp2AA88dHVnnNOnTvLmkFeIuniR+Ph4Xgx9g2p312T92t8ZN+ZL4i5fpljxEoQOe5fAwMDr7tG+Szd+njKRR1u3v+7YlInjWLp4IXFxl3mwfiN69e0PwITvvmbJgnkUKlKEfPkLUL5CJTp36+mdH4Jy640hoeTNl48d27dRsVJlcuXKdVVAb9vqUb748muCg4szd84sJk+aiDMujip3VWPoG8Px80t5ArmatWrx48QJGGP49OMPWLVyJSLC032fpfnDLTh58gSvvfwily5exBkfz+vD3qRGzVrp9egZkta4s6DY2Fh6P94OgNuDg3lr5Ce8++Fn5Mqdm3PnztKv5+PUrdfgqjfXSxbMo86999OtV1/i4+OJjYnh3Lmz/DBuDJ+M/pacOQOZPGEsYT9OoMfTz153zyK3F6VqtbtZNH8O99ern5i+fu3vHDpwgG8mTMUYw5CX+7N5QwTZc+RgxW+L+W7Sz8THx/NUtw6Ur1DJ6z8blbL9+/cxZuz3+Pn58dXo5Cfd3PPvvyz89VcmTJpCQEAAI95+k/lz59CyVesU812+bClly5UjfPEidu7Ywc/TZ3Hu7Fke79SemrVqMX/eXO6v+wBP932W+Ph4YmKivfOANiJa4856rm0qcTrj+PbLz9i8MQKHODh58gRnTp+m4G23JZ5ToVIV3n/nDZxOJw881IiQ8hXYtHI9+/f8S//e3QCIc8ZRuWq1FO/7RM8+DHm5P/c9UC8xbf3a1UT8sZqnurpq4tHRURw6uJ+oS1HUfagB2XPkAOD+B+un5Y9A3YSmTZu7rTkD/LF2Ddu3baFrJ9f/nzGxMQQVLJjsuYMHvUKO7DkoFhxM6JA3mDhhPM1bPIKfnx8Fb7uNmrVrs/Xvv6lSpSrDXx+C0+mkQcPGVKhYMc2fzW60V4li8a/zOHf2DN9ODMPfP4BOjzXl8uXYq86pVqMWn4+ZwJpVKxgxfDCdu/UgT5581LznPoaP+NCj+xQvcQdly1Vg6ZKFiWnGQNceT/FY245XnRs2+YdbfzCVpnLmzJm47+fnR0JCQuL3y7Gu3xeDoWWrNjz/4ss3zO9KG/cVKY2MrlmrNuN+mMTK5csZOvg1evTs7bYGnxXYKG6n5fB8ldSli5HkDyqIv38AGyLWcezo9ZN9HTt6hPwFgmjZpj2PPNaWXTu2U7nqXWzZvJFDBw8AEBMTzcH9+9zeq1vPPvw06fvE73Xuu5/5s2cQFRUFwMkTxzl75jRVq93N6pXLiY2NJSoqirWrVqTZ86pbVyw4mO3btwGwfdtWDh8+BMA999zHkkULOX36NADnz53jyJHDHuVZo1ZtFv76K/Hx8Zw5c4YNERFUqXoXR44cJiioIO06dKRN23Zs37bVOw9lIyKeb76mNW4vafzwowx+qT99nuxI2XIVuKNU6evO2fTneqZOHI+fvz85AwMZ+uZ75C8QxODhI3h76KvExV0G4KlnBlKiZKkU71X6zrKElK/Irp3bAah9b132791Dv15dAcgZGMjrb4+kYuWq1K1Xn96Pt6NI0aKUr1iZXLnzpP3Dq5vSuEkz5syeRce2rahcpSolS5UC4M6yZXlu4As8+3QvEkwC/v4BDHl9GMVS6G6aVKPGTfhr80Y6tG2FiPDCy69yW6FCzJ45g+/Hj8Xf35/AwEDeHfm+l58u47NTG3eqJplKTzrJlHdERUURGBhITEw0A/t055Uhb1LORi8odZIplZy0mGQqfMcpj2NOowq3+TTKa407i/novTfZv+dfLl++TLNHHrNV0FbKm1KzAo6vaY1b2YrWuFVy0qLGvWznGY9jTv3yQVrjVv85sG8vbw15JfH7kSOH6NWnP3fXqs3Ho94hOiqK24sW44133idX7tw+LKlKT8NeH8yK5csICirI9FlzAdi5Ywfvvj2cqKgoihULZuQHH5Fbfydu2g0WtslQtMadgcXHx9O+RUO++n4Kwwa9SL/nX6F6zdrMmz2dY4cP0/vZAb4uYrrLqjXuPyPWExgYyNDBgxID9+Md2/HSq4OoVbsOM6b/wuFDh+g/8AXfFtRH0qLGvfKfsx7HnAfLFbDPQgoqfW1Yv5ZixUtwe9FiHDywj2o1XEOSa9e5j+VLF/u4dCo91axVm7z58l2Vtm/fXmrWqg3AfffVJXyxTjB2K+zUHVADdwYWvuhXGjVrAUDpMmX5fcVSAJaGL+LE8WO+LJrKAMqGlGPZ0nAAFi1cwLFjR31cInuz0wo46R64RSTFGY1EpI+IRIhIxMTx36VnsTKcuLg4Vq9YRv1GTQEYNOwdZvw8hae7dSQ66hIBAVmzyUD95613RjB1ymQ6d2hLVNQlAgJufWrYrMxPxOPNHREpISJLRWS7iGwVkeet9CARWSwiu6zPAkmuGSwiu0Vkp4g0u1FZffFy8i1gfHIHjDFjgDGgbdx/rF5JSIWKBBV0zW1SslQZPv7ftwAc3L+PNTrqMcsrXeZOvvl2HOBqNlmxfJlvC2R3aVeVdgIvG2M2iEge4E8RWQz0AMKNMaNEJBQIBQaJSCWgM1AZKAYsEZFy7lZ690rgFpG/UjoEFPHGPTOb8IXzadS0ReL3s2dOUyCoIAkJCfww7hsea9fRzdUqKzh9+jQFC7p+J7795is6dNJ52W9FWo2cNMYcBY5a+5Eish0IBloB9a3TJgDLgEFW+lRjTCywV0R2A3WANSndw1s17iJAM+DsNekCrPbSPTONmJhoItat4eUhwxPTwhfOZ8YvUwGoV78xLVq28VXxlA8MeuUlItav49y5szRpWI9nnxtAdFQUU6dMBlxD21u3aefjUtpbal46ikgfoE+SpDFWi8G155UC7gb+AIpYQR1jzFERKWydFgysTXLZISst5ft7ozugiIwFxhtjViVzbLIx5vEb5ZHVm0pU8rJqd0DlXlp0B1y/57zHMad2mXw3vJ+I5AaWAyOMMdNF5JwxJn+S42eNMQVEZDSwxhgzyUofC8w3xqS4pJZXatzGmN5ujt0waCulVLpLw+4iIhIATAN+NMZMt5KPi0hRq7ZdFDhhpR8CSiS5vDhw/XSiSWh3QKWUwjVXiaebO+Ja5mossN0Y80mSQ7OB7tZ+d2BWkvTOIpJdREoDIcA6d/fQIe9KKUWaVrjrAt2Av0Vkk5U2BBgFhIlIb+AA0AHAGLNVRMKAbbh6pDznrkcJ6JD3dBU2+QfmzZyGiFC6bAihw94le/bsicenTBzHkl/nAa7h7vv37WHWopVkz5GDgX26Exd3mXhnPA81apK4+O/XX3zCH6tXUrZcBYa+NRKAhfNnE3n+PO27dEv/h/SyzNbGfezoUYYOfo3Tp08h4qB9h4507db9qnP27vmXYa8PYfu2rQx4/sWrVof/feUK3h81goT4BNq060Dvp13vyz79+EN+X7WC8hUqMmLkBwDMmT2TC+fPX5d/ZpAWbdwb9l/wOObUKJlXh7xnBSdPHGfaTz8y5oef+P6nmSQkJPDbol+vOqdLt16MnTyNsZOn8fRzL1CtRi3y5stHtmzZ+PSrcYybPJ2xk39h3Zrf2fr3Zi5ejGTLX5sYP2UGCfEJ/Lv7H2JjYlgwZxatO2jXMDvw8/fjlddCmTnnVyZN+YmpUybz7+7dV52TN19+Bg0eelXABtcf9/dGvM2XX3/HjNnzWDB/Lv/u3k1kZCSbN23klxlzSIiPZ9c/O4mJiWH2zBl07KyvmFIiqfifr2ngTkfxTiexsbE4nU5iY6K5rVChFM8NX/RfP24RITAwEACn04nT6UREcIgDZ1wcxhhiY2Pw9/dnysTxtOvcFX//zFUzzawKFSpMxUqVAciVKzdlypThxInjV51TsGBBqlS9C3//q1s2t/z9FyVKlKR4iRIEZMtG8xaPsGxpOA6HEGf9XsTExuLv78/3477j8Se66YhbN3SuEnWdQoWL0PmJHnRs2Zi2DzcgV6481L63brLnxsREs27NKh5q2CQxLT4+nt6Pt6N103rUuuc+KlW5i8BcuajXsAlPdW1P0WLFyZ07Dzu2beGBhxqm12OpNHT48CF2bN9O1buqeXT+iePHub3o7YnfCxcpwvHjx8mVKzeNmzSlU7vWBAcXJ3eePGzdsoUGDRt7q+iZgp0Ct76cTCeRF86zasVSps5aSO48eRge+jKL5s+haYuW1527esUyqtx191Wzwfn5+TF28jQiIy/w+qvPs2f3LsqUDeHxJ3vx+JO9APjg3WH0eqY/c2f+wvo/1nBn2XI82btvuj2junlRly7x8gsDeTV0iMdzahuub5IVK6r07P00PXs/DcCbw4bSb8BApv/yM2tWryKkXHn6PNMv7QqfSWSEJhBPaY07nUSsW0vRYsHkLxCEv38ADzZoxJa/NiV7bvji/2YFvFaePHm5u2Zt1q25emzTP9ZCwSXuKMnC+XN4a+TH7P13F4cO7E/T51BpLy4ujpdeGEiLR1rSuElTj68rUuR2jh39b5bIE8ePU7hw4avOubJqfMmSpZgzeyYffvIZu3fvYv/+fWlS9szETjVuDdzppMjtRdn291/ExERjjGHD+j8oWbrMdeddvBjJ5g0RPPBQg8S0c2fPEBl5AYDYmBgi1q29btX4cV9/Qa++/XE6nSTEu3oSicNBTEy0F59K3SpjDG8OG0qZMmV4skeKE2cmq3KVqhw4sI9Dhw4Sd/kyC+bP46EGVzeTjf7iM/r1H3jV74VDHMREx6TZM2QWdprWVZtK0kmlKnfxUKMmPP1ER/z8/ChbvgIt23Rg1rSfAGjVrhMAK5eGU/ue+8mZMzDx2tOnTvLem0NJSIjHJBjqN27G/Q/WTzy+clk4FSpV4bZCrtpW5arV6NG5DXeWLUfZchXS7yFVqm3c8CdzZ88ipFw5OrZtBcCAF17i6FHXwLmOnbpw6uRJunRqx6WLF3E4HEyaOIEZs+eTO3duBg8dxrN9niIhIZ7WbdpRtmxIYt6/hS+hSpWqFC7smtftrup30651S8qVK0f5Cvp7cZ2MEJE9pP24la1ktn7cKm2kRT/urYcveRxzKgfn0sWClVLK1+y0WLAGbqWUAls1lWjgVkop7NUdUAO3UkqRMbr5eUoDt1JKYauWEg3cSikF2Cpya+BWSim44QIJGYkGbqWUwlYVbg3cSikF2Cpya+BWSim0O6BSStmOjZq4dXZApZSCtJ3WVUTGicgJEdmSJC1IRBaLyC7rs0CSY4NFZLeI7BSRZjfKXwO3UkqR5mtOfg80vyYtFAg3xoQA4dZ3RKQS0BmobF3zpYj4uctcA7dSSpG2NW5jzArgzDXJrYAJ1v4EoHWS9KnGmFhjzF5gN1DHXf4auJVSitQtpCAifUQkIsnWx4NbFDHGHAWwPq8sVxQMHExy3iErLUX6clIppUjdy0ljzBhgTFrdOrlbuLtAa9xKKQWkw+Jlx0WkKID1ecJKPwSUSHJeceCIu4w0cCulFK6FFDzdbtJsoLu13x2YlSS9s4hkF5HSQAiwzl1G2lSilFKkbT9uEZkC1AduE5FDwHBgFBAmIr2BA0AHAGPMVhEJA7YBTuA5Y0y82/x1zUllJ7rmpEpOWqw5eey85zHn9nwBuuakUkr5nI1GTmrgVkopbBW3NXArpRTYa64SDdxKKQWIjSK3Bm6llEKbSpRSynZsVOHWwK2UUqALKSillO1ojVsppWxGA7dSStmMNpUopZTNaI1bKaVsxkZxWwO3UkoBtorcGriVUgpt41ZKKdu5hQUS0p0GbqWUAm0qUUopu9GmEqWUshk7dQfMsEuXqf+ISB9jzBhfl0NlLPp7kXXpKu/20MfXBVAZkv5eZFEauJVSymY0cCullM1o4LYHbcdUydHfiyxKX04qpZTNaI1bKaVsRgO3UkrZjAbuDE5EmovIThHZLSKhvi6P8j0RGSciJ0Rki6/LonxDA3cGJiJ+wGjgYaAS0EVEKvm2VCoD+B5o7utCKN/RwJ2x1QF2G2P2GGMuA1OBVj4uk/IxY8wK4Iyvy6F8RwN3xhYMHEzy/ZCVppTKwjRwZ2zJTXuj/TeVyuI0cGdsh4ASSb4XB474qCxKqQxCA3fGth4IEZHSIpIN6AzM9nGZlFI+poE7AzPGOIH+wEJgOxBmjNnq21IpXxORKcAaoLyIHBKR3r4uk0pfOuRdKaVsRmvcSillMxq4lVLKZjRwK6WUzWjgVkopm9HArZRSNqOBW11FROJFZJOIbBGRn0Uk8Bby+l5E2lv737mbIEtE6ovI/Tdxj30icpun6Snk0UNE/pcW91UqPWjgVteKNsZUN8ZUAS4DzyQ9aM1YmGrGmKeMMdvcnFIfSHXgVior0sCt3FkJlLVqw0tFZDLwt4j4iciHIrJeRP4Skb4A4vI/EdkmIvOAwlcyEpFlIlLL2m8uIhtEZLOIhItIKVx/IF60avsPikghEZlm3WO9iNS1ri0oIotEZKOIfEPy87kkS0TqiMhq69rVIlI+yeESIrLAmvt8eJJrnhCRdVa5vrn2D5eI5BKRedazbBGRTqn9ISuVWv6+LoDKmETEH9c84AuspDpAFWPMXhHpA5w3xtQWkezA7yKyCLgbKA9UBYoA24Bx1+RbCPgWqGflFWSMOSMiXwMXjTEfWedNBj41xqwSkTtwjR6tCAwHVhlj3haRR4A+qXisHdZ9nSLSGHgPaJf0+YAoYL31h+cS0Amoa4yJE5Evga7AD0nybA4cMcY8YpU7XyrKo9RN0cCtrpVTRDZZ+yuBsbiaMNYZY/Za6U2Bu660XwP5gBCgHjDFGBMPHBGR35LJ/15gxZW8jDEpzSvdGKgkklihzisieax7tLWunSciZ1PxbPmACSISgmuWxYAkxxYbY04DiMh04AHACdTEFcgBcgInrsnzb+AjEXkfmGuMWZmK8ih1UzRwq2tFG2OqJ02wgtalpEnAAGPMwmvOa8GNp50VD84BVzPefcaY6GTKcrPzNLwDLDXGtLGaZ5YlOXZtnsYq6wRjzOCUMjTG/CMiNYEWwEgRWWSMefsmy6eUR7SNW92MhcCzIhIAICLlRCQXsALobLWBFwUaJHPtGuAhESltXRtkpUcCeZKctwjXBFtY51W3dlfgaq5ARB4GCqSi3PmAw9Z+j2uONRGRIBHJCbQGfgfCgfYiUvhKWUWkZNKLRKQYEGWMmQR8BNRIRXmUuila41Y34zugFLBBXFXgk7iC3QygIa7mg3+A5ddeaIw5abWRTxcRB66mhybAHOAXEWkFDAAGAqNF5C9cv6crcL3AfAuYIiIbrPwPuCnnXyKSYO2HAR/gaip5Cbi2GWcVMBEoC0w2xkQAiMjrwCKrrHHAc8D+JNdVBT607hMHPOumPEqlCZ0dUCmlbEabSpRSymY0cCullM1o4FZKKZvRwK2UUjajgVsppWxGA7dSStmMBm6llLKZ/wc4kDMVSjwltAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "cf_matrix = confusion_matrix(y_test, y_pred)\n",
    "\n",
    "group_names = [\"True Neg\",\"False Pos\",\"False Neg\",\"True Pos\"]\n",
    "group_counts = [\"{0:0.0f}\".format(value) for value in\n",
    "                cf_matrix.flatten()]\n",
    "group_percentages = [\"{0:.2%}\".format(value) for value in\n",
    "                     cf_matrix.flatten()/np.sum(cf_matrix)]\n",
    "labels = [f\"{v1}\\n{v2}\\n{v3}\" for v1, v2, v3 in\n",
    "          zip(group_names,group_counts,group_percentages)]\n",
    "labels = np.asarray(labels).reshape(2,2)\n",
    "sns.heatmap(cf_matrix, annot=labels, fmt=\"\", cmap='Blues')\n",
    "plt.xlabel('Predicted Labels')\n",
    "plt.ylabel('Actual Labels')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "FN = cf_matrix[0][0]\n",
    "FP = cf_matrix[0][1]\n",
    "FN = cf_matrix[1][0]\n",
    "TP = cf_matrix[1][1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The profit will be 590 €\n"
     ]
    }
   ],
   "source": [
    "n_targeted = TP + FP # target all customers that were predicted to subscribe\n",
    "n_subscribers = TP # the FN will not be targeted\n",
    "targeting_cost = 10\n",
    "subscription_earn = 50\n",
    "\n",
    "revenue = n_subscribers * subscription_earn\n",
    "cost = n_targeted * targeting_cost\n",
    "\n",
    "profit = revenue - cost\n",
    "print(f\"The profit will be {profit} €\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3.4 (10 points)\n",
    "\n",
    "Now adjust the decision threshold in order to optimize the obtained profit. What would be the resulting threshold and profit? Is the propensity model built based on the targeting predicted probability useful in terms of profit maximizing? Explain."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Default decision threshold = 0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_recall_curve\n",
    "\n",
    "# Create an array of possible thresholds\n",
    "y_pred_positive = best_model.predict_proba(X_test)[:, 1]\n",
    "\n",
    "# takes the true output values and the probabilities for the positive class as input and returns the precision, recall and threshold values.\n",
    "precisions, recalls, thresholds = precision_recall_curve(y_test, y_pred_positive)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:orange\">**Precision-Recall Curve**</span>\n",
    "showing the trade-off between precision and recall for different classification thresholds. The best model would be one with high precision and high recall, but there is usually a trade-off between the two."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.0, 1.0, 0.0, 1.0)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAEdCAYAAAD0NOuvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAv4ElEQVR4nO3dd5hU5dnH8e9NU0CsxAYiqKAgtgCKYMEWhSRWEruxBRVLYmJiLzEaTYwa39iCxp5oohhLRCyxoEEiqFhQVBRElEhRURFYF+73j/tMZnZ2dnfOltnd4fe5rrnmtDnnmcNy7nm6uTsiIiJptGnuBIiISOuj4CEiIqkpeIiISGoKHiIikpqCh4iIpKbgISIiqSl4CGZ2tJl5zutLM3vVzE4xs3YlSkPP5NpHp/hMJt09my5lNV47936tMLMFZvagmW1Z6rTkpWtYkqZhOdueMbNnUpxjRzP7u5l9bGYVZrbQzJ4wsx+ZWdsmSLa0QiV5MEir8QNgDrB6svxHYF3gghJcey6wI/Beis88knxmbpOkqG63AX8i/h9tDfwKGG9mW7n7582UpgYxs58CVwFPAWcCHwBrAd8BbgA+Bx5spuRJC6LgIbmmuvuMZPlxM9sM+CkFgoeZtQcqvZF6mbr7MmBSys/MB+Y3xvXr6SN3z6T5eTNbBNwF7APc03zJqh8z24UIHNe6+2l5ux80s6uAzo1wnUb925HmoWIrqc1koIuZbZ8UhYw2s9+Z2cfAMmBNADM70MwmmdnXZva5md1rZj3yT2ZmPzazl81siZl9ZmbPmtmQZF+1YiszG5QUlyxMzv2+mV2fs79asZWZtTezS8xsVlLkMitZb59zTOZaJ5jZxWY2N0n3w2bWvQH36+Xkvcp3b4z7k+z/VbJ/UVJM9pSZDW5AevOdBXwK/LLQTnd/z91fS9JykZlVe/ib2W1mNitnPXOv8/92Mn9T3y9wjhvMbH7ev9mPk6LUpcl3/7OZrd3gbyz1puAhtekFLAe+StbPBfoAo4ADgKVmdiIwFngTGAmcAPQHnjWzLpkTmdnvgTHEA/aHwBHABPIetDnHrwY8llz/aGAEcDF155ZvJx6CdwDfA24lil9uL3Ds2cBmwLHAT4gisL/Ucf7a9Eze/1f01sj3pxtwNbA/cU/mARPMbOsGpDlz/bbAMOBxd1/a0PMVkP+38xrwNnBkXjo6EN//Hnf/Jtl2OXA98CSwL/ALInf3qOpgmpG767WSv4gHkQObEw/ntYiH3HLgAeKh6MSDzXI+txqwCLgl73w9gQrgp8n6Zsm5rqolDZlrHJ2sD0zWty4i3T2T9f7J+kV5x52Xe66caz2bd9wZyfYNi7hnDlya3K9VgEHAG8ALQPvGvj8Frt82ufbbwDU524claRuWs+0Z4Jk6zrde8rnLirz+RfH4qLb9NmBWgX/XKn87yb5zgSXAGjnb9k+O3z7n88uBC/I+OzQ5bv/m/v+zsr6U85Bc04FviKKL64lf4cfm7H/Ak/+5iR2JyvW/mFm7zIuodJ8O7JIctyeRyx2TIi3vEpWzfzKzI8xsoyI+k7neXXnbM+u75m1/JG/99eS9B4CZtcn9XmaW///lHOJ+LQVeJOoD9vXkFzONfH/MbE8ze9rMFgKVybX7EEG/aHnfqZ2ZWZrP11P+3w7Ev8sqROOMjCOBt939xWR9L+Le5N/D/wBfkL2HUmIKHpLrAOIX9BZAZ3c/yt0/zdmf36pp3eT9SeJBlvvaClgn2Z95n1NsQtx9EbAb8DERyGab2RtmdlAtH8uUgeen8795+zM+zVtflryvmrzfQtXvdEve8bcQ92tn4pd4D+CenIdxo90fM/s2MI4oQjwOGJxc+9Wc9BYrPy27AguJXMDGKc9VrGot4tz9A6Jo7kgAM1sT+C5wZ85hmXs4o0C6Vyd776TE1NpKcr3h2dZWheT/clyYvB8NTCtw/JfJ+4LkvRtRzFIUd58KHJT80hxI1FH83cy2cfc3CnwkEwzWp2qT3/Xz0lusi4Brc9YX5O2f6+5TkuXnk6BxIVG3cS+Ne38OInIbB+bkbDCztYgcWhqD8tbfdvdKi74ge5nZKh6t32qzNLl+B3evyNle08O8ppZVdwI3mdnGwN5AB6rWO2Xu4XeAzwp8Pu2/qTQSBQ9piInEA3Azdy9UIZ3xJLCCqCz9edqLuHslMMnMzicqTPsS9Qv5nk3eDyHqIzIOT94npLzuLGBWio/8FvgxcKGZ3Ufj3p9ORNn//x7CZrY7kduZmSKN5AS8fJcT9SNXAPlNdTGzXkAXjxZXHySb+5O0MktyDkPIBsVi3Ev0JzocGA5MSO57xhPEvenh7k+kOK80MQUPqTd3/8LMfgFcZ2bfAh4lKoi7EUUhz7j7X939PTO7GvhZ0sLoIeJBuD0w3d3/ln9uM/se8TB9gHg4diYeaF8SldKF0jPNzO4GLkpyKxOJeofzgbuTh16TcfclZvYbIrdyoLuPbcT7M57oc3Obmd1K1HWcD3zUiOmfYGY/A64ys75E5fdsogHFHsDxwGFES6nMd7nJzC4k6i5+SbZlXrHX/MLMHgJOBjYggm/u/vfM7LfAtWa2OfEDYSmwEVEfcrO7P12/bywN0tw19no1/4tsq6XNatjfM9l/fA37RwBPExWYS4jy6VuAfnnHnUg8eJYRRUzPADvmXePoZH1z4G9E4FhKdAYcB+xQIN09c7a1By4hfhl/k7xfQtICqrbvQ4GWSrXcMwcuKbC9A5FbeYWkdVFj3J9k/6nJ/VhC9MHZk7yWVIW+Q/4xRXy3IUSOYC7ZBhSPE82H2+Qct1OSjq+Bd5L9t1G4tVXBv53kmO8mx1RpeZV3zJFEJ9LFRIB6iwjS3Zv7/8/K+sr8cYuIiBRNra1ERCS1kgYPM7vFzOaZWaHKTiz8n5nNMLPXkuaJIiLSwpQ653EbMaxATYYDvZPXKGIUTxERaWFKGjzcfQLVO2bl2g+4w8MkYE0z26A0qRMRkWK1tKa63YAPc9bnJNuq9U41s1FE7gRYZ0Dfvj3p1KkEKRQRKRMvvfTSAnf/Vn0+29KCR6Exdgo2B3P3MSRjAZkN9DvvnMKAAU2ZNBGR8mJmH9R9VGEtrbXVHKLzT0Z3YmwjERFpQVpa8HgIOCppdTUYWOTuzTXFqIiI1KCkxVbJ0BHDgK5mNocYRK49gLvfSPQgHkH0wP0aOKaU6RMRkeKUNHi4+6F17HdijBsREWnBWlqxlYiItAIKHiIikpqCh4iIpKbgISIiqSl4iIhIagoeIiKSmoKHiIikpuAhIiKpKXiIiEhqCh4iIpKagoeIiKSm4CEiIqkpeIiISGoKHiIikpqCh4iIpKbgISIiqSl4iIhIagoeIiKSmoKHiIikpuAhIiKpKXiIiEhqCh4iIpKagoeIiKSm4CEiIqkpeIiISGoKHiIikpqCh4iIpKbgISIiqSl4iIhIagoeIiKSmoKHiIikpuAhIiKpKXiIiEhqCh4iIpKagoeIiKRW8uBhZvuY2dtmNsPMziqwfw0ze9jMXjWzaWZ2TKnTKCIitStp8DCztsB1wHCgH3ComfXLO+xk4E133wYYBlxpZh1KmU4REaldqXMe2wMz3P19d68A7gH2yzvGgS5mZsBqwKdAZWmTKSIitSl18OgGfJizPifZlutaoC/wMfA68BN3X5F/IjMbZWZTzGxKUyVWREQKK3XwsALbPG99b2AqsCGwLXCtma1e7UPuY9x9oLsPbOxEiohI7UodPOYAG+WsdydyGLmOAe73MAOYCWxRovSJiEgRSh08JgO9zaxXUgl+CPBQ3jGzgT0AzGw9YHPg/ZKmUkREatWulBdz90ozOwV4DGgL3OLu08zsxGT/jcCvgdvM7HWimOtMd19QynSKiEjtSho8ANx9HDAub9uNOcsfA98pdbpERKR46mEuIiKpKXiIiEhqCh4iIpKagoeIiKSm4CEiIqkpeIiISGoKHiIikpqCh4iIpKbgISIiqSl4iIhIagoeIiKSmoKHiIikpuAhIiKpKXiIiEhqCh4iIpKagoeIiKSm4CEiIqkpeIiISGoKHiIikpqCh4iIpKbgISIiqSl4iIhIagoeIiKSmoKHiIikpuAhIiKpKXiIiEhqCh4iIpKagoeIiKTWrj4fMrP1gR7Aqvn73H1CQxMlIiItW6rgYWbdgLuAXQrtBhxo2wjpEhGRFixtzuMGoD/wS+B1YFmjp0hERFq8tMFjZ+A0d7+zKRIjIiKtQ9oK8yXAvKZIiIiItB5pg8dNwJFNkRAREWk90hZbfQQcaWZPAeOAT/MPcPdbajuBme0DXENUrN/s7pcXOGYY8AegPbDA3XdNmU4REWlCaYPHjcl7T2BYgf0O1Bg8zKwtcB2wFzAHmGxmD7n7mznHrAlcD+zj7rPNbN2UaRQRkSaWNnj0auD1tgdmuPv7AGZ2D7Af8GbOMYcB97v7bAB3Vx2LiEgLkyp4uPsHDbxeN+DDnPU5wA55x/QB2pvZM0AX4Bp3vyP/RGY2ChgVawMamCwREUmjvj3M+wO7AmsDC4EJ7v5GMR8tsM0LpGkAsAfQEXjBzCa5+ztVPuQ+BhgT6RmYfw4REWlCaXuYtwNuAw6laiBwM/srcLS7L6/lFHOAjXLWuwMfFzhmgbsvBhab2QRgG+AdRESkRUjbVPdC4IfABUT9R8fk/QLg4OS9NpOB3mbWy8w6AIcAD+Ud8yCws5m1M7NORLHWWynTKSIiTShtsdURwK/d/dKcbR8AlyYtqY4hAkxB7l5pZqcAjxFNdW9x92lmdmKy/0Z3f8vMxgOvASuI5rzFFImJiEiJmHvx1QVmtgz4rrs/WWDfnsAj7r5KI6avyHQN9ClTpjBA9eYiIkUzs5fcfWB9Ppu22OpjYGgN+4ZQvf5CRETKUNpiq78A55rZimR5LrA+UXdxLvDbxk2eiIi0RGmDx0XAJsCvkuUMA+5Otksr9Pjj8O67MHo0WKEG1SIiOVIVW7l7pbsfBmwFnEK0rjoF6O/uh9fRTFdaoEWL4JhjYO+94ZRT4I034NJLYb314C9/ae7UiUhLVa9Ogu4+DZjWyGmREnv6aTj6aJg9O7tt771h7tzs/sMPb5akiUgLV2fwMLMewFx3/yZZrlVmTCopnS+/hK++gg02KO74JUvg7LPhmmtifeBA+O9/Yc6cCBzt2kFlZdOlV0Rav2KKrWYC2yXLs5L12l6SwiefwMcNaKM2aRJsthn06QOLF8f6EUfAWzV0q5w8Gb797QgcbdvCRRfBxImwww7Qpg2ccQZceWX90yMiK4diiq2OBd7LWdY4Uo3kzTdhyBBYZZUIIK+/DjfdBOedV1wu4t574aijYOnSWL/yyqivqKiAnj3hkkuyx65YAZddBhdeCMuXQ9++cMcdkesA+Otf4bPPoq7j5psb/auKSJmpM3i4++05y7c1aWpWIvPmwXe/GxXWEIFjjz3g009hiy3g1FNr/qw7XH45nHNO1e0X5vTtX57TdGH+/MiNPP54rJ9+egSZjh2zx3ToEIFDRKQYaTsJVmNm/czsIDPbsDEStDJYuhT23x9mzcpuGzEiAgfAN9/U/Nnly+HkkyNwmMEVV8CGyZ1v3z6Kn3I9/zxst10EjnXWgUcfhauuqho4RETSShU8zOxaM7sxZ/1A4FXgXuBNMxvUyOkrO+7RNPaFF6BHj6hngGwLp9osXQoHHww33BBFXffdF3UUw4bBxhvDM8/AfvvFsStWRGAZNgw++iiKx155BfbZJ32a586Nc4iIZKTNeQwHJuas/wr4JzFk+ovUMiiihN/8Bu65B7p0gX/+M1o2AfTvH81ma7JoEQwfDmPHwhprRE7iwANj3113wcyZESAyrr8efvnLyKn84hcRWDbaqOCpa7R0aVSo9+wJ224bAUlEBNL381ifaHGFmXUHtgSOc/fXzez/gD83bvLKy2OPwfnnR3HT3XfDVlvBrrtGZfljj8Hvf1/4cwsXRv+Ll16C9dePY7feOru/UI/wr76CtdaC22+H73+/funN7SS4YEE03+3QoX7nEpHykjbnsQRYLVneFfgCmJKsf0VMGysFzJoFhx0WxVYXXhiV5RCB4LXXsvUW+T75BHbbLQLHpptGs9rcwJEv00pr++3h5ZfrHzgy+vatebiSf/0LdtopcidNzT3qbx58sOmvJSJ1Sxs8XgZOTqahPRl4wt0zhRm9iIESJc+SJVHE9OmnETTOPz+7zyxb75Hvo48iZ/L669ECa8IE6NWr9msddVQEmuefj+Km+th9dxg6NJr6Tp2aLVrL+OADGDkS9twT/v3vph3GpLIymiTvsAPsvHM0NHjvvTo/JiJNLG3wOBcYTFSSbw78Omff/kS9R9maNCk62P3739X33XVXtGqaPTuKoXbaCf74x9h3yilRWb3JJnDnnTUHi1xz5kTgePvtyGk8+2zNuZNcbdpEGtu3T/fdcm2ySQSfc8+tWky1ZAlcfHHkRsaOjU6GGR9+GHU2AwbE97/uOthyS7jtNvjPfyJo7rlnNBs+++wIhs8+W3MaFi+Ga6+Nzo8//GF0bsz48sv6fzcRaSTunuoFdAYGAKvnbf8u0Cft+RrjBQN8yhRvUitWuA8e7A7uZ55Zdd+cOe6dO8e+O+5w/973YnmHHdzvvjuWV13V/dVXa7/G6afHsWec4d67dywPGOC+cGHTfa9itG8faenRI97B/ZBD3J96KvvdVl01u6/Y19lnV7/WJ5+4n3ee+9prZ4/bdFP3665z32KLWH/llZLfApGyBEzxej53Uw+M6O6LgZcKbH+kIUGspXv22ch5FHLGGfFLGaII57HHYnnuXDjhhFi++ura6ypyXXllPDa33RaeeCIqvluC2bOjkv+Pf4xc0bvvxvZMD/eadOoEX3+dXe/aNSrgc82ZEw0GxoyJHA7A4MHRUmy//SKXM2ZM1c+sWFFcLk5EGl+d//XM7CgzWydnudZX0ye5eVx+eeHtTz8dTW8zMoED4mH7xRdRTp8JIsVwj4d0Swkc/fvDmmtG0Hj55QgcEP1UMi3GXnghmgdvuSX86U/ZYrfTToP334/7d9BBUeF/+unZc7/3HowaFUVl11wTgeN734PnnotjDzywavEYxPaRI6Ovy003lew2iEiuurImwApg+5zl2l7L65sFasirqYutXnqpanFLptiqosK9X7/YlinaAffu3bPL3bq5L1hQ3HXOOSc+s8UWUXzTUixd6r5sWeOd79JLs/epTZtYNnM/+ODai/a22aZ60dfxx1c95vXX3U88MY594AH3s85y32gj95NOarz0i5QLmrjYKrcVVR1tfcrTb5PJdTt3zhZPQVTovvlmNKHdbrvo8d2lSxRR/eAH0ZLqrrtiWJBinHRSFPEcdxysu27jf4/6WmWVpjnvnDnRkuuoo+Css2DzzWs/fvXV471r16i0f+65WP/mG/jHP6KSfsKE7PH7759dfvLJRk26yEqvmIERPyi0vLJ4990ICu3bR2ui666L7fPnZwcivOaaKLaBGDdq332jhdBee8XwIMXq3j1aOJW7YcOiyfE++0Qv+GKbFP/5z9FsecSIaLX23HNRF7XxxtnhXVZbLTpIZgwfHuN5Qfyb3XlnBK38gSFFJJ1UFeZmNhjo4e5/L7DvB8Bsd/9PYyWuJbjiiqiYPeaYeLhnXHJJNBkdPjyaoX7nO3HMppvG/r/9rXnS2xoMGRL1IGn17h2vXJlK+379YsDII46IOpInnoBddomK+UcfjU6a3bplB50cMQKWLYOHHoIf/ajq0C4iUre0ra0uAybUsK8vcBKwe4NS1IIsXBhzXphFq59//CO2z5wZy2bZivT27bOBQ5re4MGRYxk4MILGrrtW7QmfKbL6/PN4/+abaJnVsWNUyu+7b7ZV18KFCh4iaaUNHtsAv6th34vAaQ1LTvNxrz4Mx623xq/T4cOrlsffe28cf9RRxTe/lca11VYRxOvSq1f06F9llchhjBoVOZElS2KAyUWLohf7u+9G/VSHDitH0aFIQ6UNHqtSc/PetkQHwlbnoouimemUKdH8FKKo6sZk8PmTTqp6vHs8ZC6+uKTJlHowq/rvdO65UcR1wAExIdeBB8L48VXHzBo6NLY9/3w0fhiUTDRQ6AeGyMoqbfB4C9gXKNQhcF/g7QanqMRWrIjAMX9+tJzKBI8nnog+CD16RPl4vpNPjopaaV2GDo0XwLhx8b5sWbSkW7YsciG77ZY9/rLLorXcfffF+Frjx5c+zSItUdrgcSPwJzP7ArgJmAN0A0YBxwGjGzd5TeObb+Dww+MX5W67ReDId/318X7CCdU7qa2+evUpYKX12W23aDG3ySbRgXHgQJg+PTpErr9+LGfquSDG6BKRkCp4uPtNZrY5cDrws9xdwNXuPqbwJ1uWSZOi3uKll7KVprlmz46Jmtq3jz4XGVtuGe+//nX0NZDWrWPHqsPJjx0b9Sh77BGtsA4+OCrl99svmmND5FQnTYrmviNHangUWXnVZ2yrM8zsBmAvYG1gAfCku9ej8WXzmJjMheie7QOQa8yYeEgcfDCst152+/e/HxWsmc5qUl769YsXRD+dXXeNzpqffx7B46uvogJ+9uw4Zvz4mKRLZGWUOngAuPt7QKudVSETPD77LNr/56qogJtvjuX8inJQ4FiZZH44ZIotKyuzgQOiiW9me/6cJyLlLnWm28w6m9lpZnafmT1lZr2T7YeY2RaNn8TG5Z4NHp9/Huu5Hn44Zu/r3z/m5BBZffUY6fgnP4kWWAcfHNsffjjqTVZdNUZWnjcvBmq8/fbmTa9IKaTtYb4R8AzQHZgO9Cc79exuwJ7A8Y2YvkY3Y0b14cBz3XVXvB97rJplStbPcmr4MkPU5I6mfOWV0ax3RTKv5k47Ve00+t578MgjsNlmhVvvibQ2aTPbVwLLgN7Ax0BFzr5ngYsaJ1lNJ5PryNWzZxRfffppNN9s0wYOOaTUKZPWYo89oq5syJCY6fAPf4jtbdtG8VVFRQxd88or8MAD0WLr9dfjmI4dY19+Cz6R1iZt8NgLGOXus80s/8//I6LZbouWHzw22SR+Dc6aFW35KypiQMMNNmiW5EkrcNxx2VZ47lHE2aFDNKjYdVd47bX4G8rN4a6+esztsmRJ9aJSkdYobZ1HB6CmGaTXAL6p6wRmto+ZvW1mM8zsrFqOG2Rmy81sZMo01io/eIwYkS2eeiTp+njEEY15RSlnZhFIjjwy+od06hTbFyyIviInnBCtsubPz+Y2br89erZvt13MUS/SGqXNebwGHAQU6mc7nALT0+ZKcivXETmYOcBkM3vI3d8scNxvgceqn6V4S5ZES5guSa3MokUwbVpeoodnR2atqIhihQMOaMhVZWV29dUxOsEee8TgjYX6gRyfUys4fnzd85iItERpcx5XAMeZ2U3ALsm2fmb2K6KH+RV1fH57YIa7v+/uFcA9wH4FjjsVGAvMS5m+KgYNigH0MpWY//lPFBlkmmCuumr1+Tb22y8bbETSGjw4BmIcMqR64Bg0KIq39tkHtt++edIn0ljS9jC/38xGA5cDxyab7yCKsk5x97pG/ukGfJizPgfYIfcAM+sGHEAM7T6ophOZ2ShiWBRgQLX9X32VzWVUVESgyBRZHXoo/Pe/MGBAtpgh4/DD6/gGIvU0cWL8La6ySjT7ffHFGALlwgvhmWeix/qppzZ3KkWKk7ap7hrArcCdwI7AusBCYKK711QXUuUUBbblVx/+ATjT3ZdbLW1lk6FQxkS6Blargvzoo+qfyQSPnXaKsYzyrbOOegxL0zGrPqVvZuRmiP5Fp54axahLlmi4f2nZig4eZtaOCBQHuPvDQH1mhZ4DbJSz3p1o8ptrIHBPEji6AiPMrNLdH0hzoY/zzrp8eYxJBLDjjoU/c/DBMZ6VSFPbZpt433jjKMK6994YL2uzzaJPCMBbb8EWLb7braysiq7zcPdK4BNgeQOuNxnobWa9zKwDcAjwUN51erl7T3fvCdwHjE4bOKB6zmPatGhf37MnbLhh1X1bbRUV5aNGpb2KSP0ce2w03Z05E36XTK+2eHE2cEAEE5GWKm2F+V00oAd5EoBOIVpRvQX83d2nmdmJZnZifc9bSH7wyBRZFZpu9He/iyKDzK9BkVLo0iWKsnr2hCuugPPOg3//O/qKiLR0aZvqzgION7PJwIPAXPLqLNz9ltpO4O7jgHF5226s4dijU6bvf/KDx+TJ8b7DDtWPNVMLK2leZ5yRXdYgi9IapP0zTUb1YUMKNXGKQFJr8CiV/OCRGR5CuQsRkYZLGzx2IJrlFphCqWXJDR4rVmSb7W61VfOkR6ShKiqir9Laa2cnJhNpLnUGj6S39/nAT4kRdJcDDwPHufvnTZm4hshtbfX++/D111FRvvbazZcmkTTGjoUbboD774/1zp2jUr1z5+intNpqzZs+WbkVk/M4EbiAGIp9MrAJ0YnvC+CYJktZA6xYAXPnZtczRVbKdUhrcmNeTeDixdn3L79MFzyWLo2OsiKNpZjWVj8GbnL33d39THf/AXAycETS3LbFmTcvxrTKUPCQ1mT//aFHj+jIev31MZfIHXdEUez668cxY8fCj38cY2hl/r5zffIJ/P3vcPLJUcTVsSP8/Oexb9GiGIFBpCGKyXlsApyRt+1vwA3AxsC7jZ2ohqqpslzBQ1qDU06JV21yhzG56KLoXDh+fAwHv8UWMexJvjvugKefhqlTo+hr9mxYa63GTLmsTIoJHqsRRVS5MkORtMgGrgoeUq722CMmmBo6NIYwee65bJ1IxvTpMWbb0KHRZ2TttWH06BgmPjPHyFdfwYcfKnhI/RXb2qqbmW2Ss942Z/vnuQe6+/uNkbCGyB+a5IMPYi6Fvn2bJz0ijeWuu2JkaLOYT33kyBjiZLfdYs6QXr0iwAwYECP4QtQBzp8fyzvvDCedpHlEpOGKDR731bD9gQLbmn2CzUKDIvbpowpDKQ+Z8UJ32ilaXdWlTRu44ILseiao3Hpr5D6efTYmRevYMZoCDxoEY8Zkj1+wIMaFmzoV9twzhp2fNy+CUqYORlY+xQSPFtmiqjaFgoeKrESqysy9DlEfkjF1anSmffFFeOGF7GRpEHOV9OoVY3K1aRP7NsktkyhSRUUUJy9ZEsVrZhGMCk2eJS1TncHD3W8vRUIak4KHSM2OOy5yHd/+Nmy0UeQydtghHuJnnhnFYrkV9h07wgYbRH8piMAB8bCfOTNahn30UZzLLAZ3XGMN6No1tn/wQeR2pk6FKVPgpZcicFRUxHl69IgK/HffhV/8Ag47DF55Jepjvve9kt4aScHcq02F0eqYDfQpU6YwIBkwpX//6tPNPvBAzBIoIjU75pioS9l++5i6YMcdY16R9u2jefCnn0agOe20KO7q0yeCxfJkrO0114TPP4/l1VePkYMLMYsgVZcPPojgIk3DzF5y94H1+WxZDsGWX2EOynmIFOPWW2velzuB2pprxvs771Q9JhM4oGrg6NMnKvEHDoz37baLXMktt8Cmm8bU0AceGMf26hX9VL7+OnIj06fH8PRDh0YdzWuvxf/nvfZqyDeVhiq7nMeSJdFMMVMpWFERWeIvvlB5qkhjeecd+Oc/oV+/CAb/+lc0SBk0KP4PvvoqbLtttj6klklB/+ezz+J9rbUiOLzxRs3HZv5Pf/ppdp6eYq4hVSnnkSNT37HhhtkhSvr3V+AQaUx9+kTP94xDDqm6f7PN0p8zt89J374RPLp0iWBUWRk/CrfdNubmWbwYunfP/h8fOTLW334bDj0Uvv/9+Hz79oWnYZCGK9vg0a1b9g9LRVYircvdd8NVV8WPwDZtYmyuDh1ieb31oqlw7vh19+V0Jnj00arn2m23qNyfPRsuuSQq5KXhyu73eG7wyFDwEGld2raNnESmxGDVVbPLzzwD//gHzJiRnXnxkEOqTqiV26fr6aejcn/mzOq98aX+yi7nkaksz52nXMFDpHz07ZsdLWLTTSOYZPzsZ1EHsumm0U/lzjtjed687FzxK1bArFkxbEum4l/SK7vgkZvzaN8eli1T8BBZWWywQbwg29QYssVa48fHUPZLkunsjj8++q989ln0dxlYr6rjlVNZB4+rroqy0q5dmzdNItK8MgElMydKxs03Z5cfekjBI42yDh677NK8aRGRlmHIEJgwIepS+vWDJ5+Exx+PVmOTJ8fcJ5JOWQcPERGIPiA775xdHzkyXgAXXxzvc+fGSBTvvhsdEocMKXkyW5WyCh4rVhSuMBcRqcvNN2eLsXr2zI7hJYWVVfBYuBC++SY6G3Xs2NypEZHWYJddouXVGmtE0Hj66fgRuvfe0ZN+1qwYKHLZsugrctJJMYT9rFkxfhdEbmXBghiyPs3c8q1ZWQ1P0rZtjJnTv3/heZ1FRGrz5ZcRSCoraz+uXbvCx1x4YUwL3FpoeJKE6jtEpCG6dIlmve+8A5tvHmNnPfBAzNbYti1cfXUclxlFGCL3sdpqkVuZN69Zkt0syip4ZGZVyzTLExFJK3/qhqOPzi6PHh3N/zfdNHqxf/FFBI8bboCTT475SkaPjmHqd94ZzjuvpEkvqbIKHplROddeu3nTISLlKX/AxzXWiPf27eN98uR4ATzxBJx7bvmO9ltWwSMzl4CGHBCRUjrggMh1dOgQuZLTTy9usqvWTMFDRKSBunaFP/0pu3766fH+y19Gk9/nnoPvfCfmF5ozJ4rCfvzjZklqo1HwEBFpZB06RKD4/e+z2+66K7v8+ecKHi1KJnjkTiojIlJqN94Yk1b16hXrU6fGAK3t2sE550SRVkVFTKvbsWPr7NRcVsEjU2GunIeINKdjjolXvjffjOAxfXq01nKPeUpmzMgGmtairCaDUrGViLRkG2wQ86+7Ryustm1jWKXBg2P4+D594N57mzuVxSl58DCzfczsbTObYWZnFdh/uJm9lrwmmtk2xZ5bwUNEWrK11oohTmbOjP4i3/1ubJ83DyZNimFOfvjDGBqlb1/4wQ9abqutkhZbmVlb4DpgL2AOMNnMHnL3N3MOmwns6u6fmdlwYAxQ1BT2Ch4i0tKtvXa2L9rvfx+dCddfP4qyLr00tj/+eLxPnw7z58O66zZPWmtT6jqP7YEZ7v4+gJndA+wH/C94uPvEnOMnAd2LOXFFBXz9dWQDO3duxBSLiDSR3r2zc69XVsYwKJWV0KMHHH44LFoUg73Onh05ld69W06nw1IHj27Ahznrc6g9V3Ec8GihHWY2ChgVawNYtCiW1lyz5dxcEZFitWtXtfluptd695yfzyNGwO67xzhae+0F++xT2jTmKnXwKPRYL1iiZ2a7EcFjp0L73X0MUaSF2UBXkZWIlJOttorh4XONGxcviIr12bNLn66MUgePOcBGOevdgY/zDzKzrYGbgeHuvrCYEyt4iEg5eeKJmN1wvfXg/ffh5z+P5r1rrRWTVi1Z0rzpK3XwmAz0NrNewEfAIcBhuQeYWQ/gfuBId3+n2BMreIhIOWnbNltktfnm8M9/xvL8+dkZD5tTSYOHu1ea2SnAY0Bb4BZ3n2ZmJyb7bwQuANYBrreovKgsZrISdRAUESmdkvcwd/dxwLi8bTfmLB8PHJ/2vMp5iMjKbvHimMCqe/embzhUNj3MNa6ViKxMFiyAQYOygcIsZjTs0SOGPBk5EnbaKbYPHRq917t3h0ceaZzrl83YVsp5iMjKoHNn6NQp+rVNmVLzcWPHZpcn5vSeGzcu27O9IRQ8RERakU6d4IUXYqrbDTaI3umVlTGnyBprwJ13xvNw/fVjrvXKSthoo+i1ftlljZcOBQ8RkVZm663jVchRRxXePm1a46ZBwUNEZCVy/fXR3Hf+/IadR8FDRGQlsPrq2eXGGPa9bIJHRUW8K3iIiFR30EHw5ZdRD/Ktb8Vrzz3rf76yCR4ZCh4iItV16gSjRzfe+cqmn0eGgoeISNMrq+DRrl1EVxERaVplFTw0l4eISGmUXfAQEZGmV1bBQ+NaiYiURlkFD+U8RERKQ8FDRERSU/AQEZHUFDxERCQ1BQ8REUlNwUNERFJT8BARkdQUPEREJDUFDxERSU3BQ0REUlPwEBGR1MoqeGhsKxGR0iib4NGhA6y6anOnQkRk5VA2wUNzeYiIlE5ZBQ8RESkNBQ8REUlNwUNERFJT8BARkdQUPEREJDUFDxERSU3BQ0REUlPwEBGR1EoePMxsHzN728xmmNlZBfabmf1fsv81M/t2MedV8BARKZ2SBg8zawtcBwwH+gGHmlm/vMOGA72T1yjghmLOrXGtRERKp9Q5j+2BGe7+vrtXAPcA++Udsx9wh4dJwJpmtkFdJ1bOQ0SkdNqV+HrdgA9z1ucAOxRxTDdgbu5BZjaKyJkALNtxR3ujcZPaanUFFjR3IloI3Yss3Yss3Yuszev7wVIHj0JDF3o9jsHdxwBjAMxsirsPbHjyWj/diyzdiyzdiyzdiywzm1Lfz5a62GoOsFHOenfg43ocIyIizajUwWMy0NvMeplZB+AQ4KG8Yx4CjkpaXQ0GFrn73PwTiYhI8ylpsZW7V5rZKcBjQFvgFnefZmYnJvtvBMYBI4AZwNfAMUWcekwTJbk10r3I0r3I0r3I0r3Iqve9MPdq1QkiIiK1Kpse5iIiUjoKHiIiklqrCh5NNbRJa1TEvTg8uQevmdlEM9umOdJZCnXdi5zjBpnZcjMbWcr0lVIx98LMhpnZVDObZmbPljqNpVLE/5E1zOxhM3s1uRfF1K+2OmZ2i5nNMyvcF67ez013bxUvooL9PWAToAPwKtAv75gRwKNEX5HBwH+aO93NeC+GAGsly8NX5nuRc9xTRIOMkc2d7mb8u1gTeBPokayv29zpbsZ7cQ7w22T5W8CnQIfmTnsT3ItdgG8Db9Swv17PzdaU82iyoU1aoTrvhbtPdPfPktVJRH+ZclTM3wXAqcBYYF4pE1dixdyLw4D73X02gLuX6/0o5l440MXMDFiNCB6VpU1m03P3CcR3q0m9nputKXjUNGxJ2mPKQdrveRzxy6Ic1XkvzKwbcABwYwnT1RyK+bvoA6xlZs+Y2UtmdlTJUldaxdyLa4G+RCfk14GfuPuK0iSvRanXc7PUw5M0RKMNbVIGiv6eZrYbETx2atIUNZ9i7sUfgDPdfXn8yCxbxdyLdsAAYA+gI/CCmU1y93eaOnElVsy92BuYCuwObAo8YWbPufsXTZy2lqZez83WFDw0tElWUd/TzLYGbgaGu/vCEqWt1Iq5FwOBe5LA0RUYYWaV7v5ASVJYOsX+H1ng7ouBxWY2AdgGKLfgUcy9OAa43KPgf4aZzQS2AF4sTRJbjHo9N1tTsZWGNsmq816YWQ/gfuDIMvxVmavOe+Huvdy9p7v3BO4DRpdh4IDi/o88COxsZu3MrBMxqvVbJU5nKRRzL2YTOTDMbD1ihNn3S5rKlqFez81Wk/PwphvapNUp8l5cAKwDXJ/84q70MhxJtMh7sVIo5l64+1tmNh54DVgB3OzuZTedQZF/F78GbjOz14mimzPdveyGajezu4FhQFczmwNcCLSHhj03NTyJiIik1pqKrUREpIVQ8BARkdQUPEREJDUFDxERSU3BQ0REUlPwEAHM7Ggz85xXhZm9Z2a/MbNVmzFdt5nZrJz1nkn6jm6uNIlAK+rnIVIiPyB63HYhxsM6O1k+tTkTJdLSKHiIVDXV3Wcky0+YWW/gODNbWQfNEylIxVYitXuZGECwK4CZdTKz35rZzKRoa6aZnWtmVf4vmdm3zOx6M/vQzJYl73ea2SrJ/s2S9ZlmtsTM3jezG8xsrdJ/RZH0lPMQqV1PYBGw0MzaEcNd9COGtnidmDznfGBt4OcASQCYmGy7hBgKZF1i3oQOwDJgQ6J47KfAZ8SkRecQQ0XsWIovJtIQCh4iVbVNgkSmzuMg4KfJcO5HEkPb75pMsAPwr2TssAvN7LfJ5EqnE8FgoLu/knPuuzMLyecz58DMJhJjCz1nZtvlfU6kxVGxlUhV04FviJnX/gz8yd2vTfbtA3wATExGpW2XBJrHiYHmBifHfQeYXFsAMLMOZnaOmU03syXJNZ9Ldm/e6N9KpJEp5yFS1QFEcdK3gJ8Bo83sP+5+B1H0tDHxoC9knZz3V+u4zmVEC66LiSKuL4l5FO4Hmq1psEixFDxEqnoj09rKzJ4i6iuuMLOxwEJgJvDDGj47K3lfQN3TeB5CzBt9SWaDma3WgHSLlJSCh0gN3H2Zmf2CmEBpNDCeqAP5yt2n1/LRx4HzzGwbd68pB9KJ6jmYspx/RsqTgodILdz9ITObDJwBbEY84P9lZlcSRVMdiPmv9wX2d/evgauBw4AnzewSolVWV6K11Ynu/iURiH6UTEQ0AzgQGFLSLyfSAAoeInU7j2iiezywN3AWMAroBSwG3gMeASoA3P1zMxtKNNM9i6gD+QR4KnMMUd9hwKXJ+jjgUFa++bOlldJMgiIikpqa6oqISGoKHiIikpqCh4iIpKbgISIiqSl4iIhIagoeIiKSmoKHiIikpuAhIiKp/T/ujOBZxKAjQAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot precision-recall-curve\n",
    "plt.plot(recalls, precisions, \"b-\", linewidth=2)\n",
    "plt.xlabel(\"Recall\", fontsize=16)\n",
    "plt.ylabel(\"Precision\", fontsize=16)\n",
    "plt.title(\"Precision-Recall-Curve\", fontsize=16)\n",
    "plt.axis([0, 1, 0, 1])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:orange\">**Precision and Recall as function of threshold**</span> We can observe how the binary classification model's performance varies when we modify the threshold. The precision and recall numbers denote, respectively, the proportion of correctly classified positive cases and the fraction of correctly categorized positive instances. The best model is one with high recall and high precision."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeMAAAEdCAYAAAA/7w9RAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABV6ElEQVR4nO3dd3gUVffA8e9JpSWU0Am9CgjSpKiACkhR8BULdvSnoGDDV6yo+GLBjg0FFUFREKWq2FCKSJeqFAk99E4gIfX+/ri72ZRNskk2uynn8zz7ZGfmzszZ2c2enTt37hVjDEoppZTynwB/B6CUUkqVdJqMlVJKKT/TZKyUUkr5mSZjpZRSys80GSullFJ+pslYKaWU8jNNxnkgIoNFxKR5xIjIBhF5QESCfBhHPcf+B+diHWfs9QouMv8Rkd0iMjmHMvUyvH/JInJIRL4Ukdo+CjWr2DK9P568poImItVFZJ6InHDE94if4qgnIqNFpIGbZX4/TtkRkbtFZLuIJIjIqWzKPSIi17mZP9px7H32HZMVEVkkIku9vL1FHpQbLSLF8n5cv7+pRdwNQDQQ7nj+HlAVeM5H+z8IdAZ25GKdHxzrHCyQiIqWV4B5QAjQCXgeuEBEOhpjEv0aWeHzHNANGIz97Oz2Uxz1sO/TUmBnhmX/Ac74OiBPiEhNYCLwJXAXcD6b4o9gX9+sgo9MFRaajPNnvTEmyvH8FxFphP1HcpuMRSQYSDJe6mnFGBMPrMjlOkeBo97YfzGw0xjjPH5LHO/Pi0A7cnlcS4ALgA3GmNn+DiQrxph1/o4hG42BQGCKMcZrZ5SeEpFQx/eFKqS0mtq7VgNhIlI1TVXoMBF5TUQOAPFABQARuU5EVohIrIicEpFvRKROxg2KyL0islZE4kTkpIgsFpEujmWZqqlFpIOI/Coixx3b3iki49Msd1cNGiwiLzqq+RIcf190JCcy7GuoiPxPRA464v5ORCJzOjAi0ktE5jvWixWRv0XkvyISmKHcbhGZKiKDRGSLiJwTkTUicqmbbT7sKH/eUeaynOLIwVrH33Tvg+M92ODYzzER+VREKmUoEyQiT4jIZke5oyLyk4g0cywvJSJvO173WUe1+HfO5fnl6fYd1c1TROSAiMQ73o/vRaRqFtut56gW7A5cJq6qfWd1caYfliIyWUR2Z9yGp5+drD7zItIdWOgo9muaWLo71stUTS0iF4vIAscxOSciv4nIxW7ijRaRNiLyh+PzuV1E7svhsDvXbyoisx2vKU7s/3XvtNsHFjkmf3PEPNnNpnAct7rArWleX8ay9UXkB8dr2iMiz4lIQJptdHesd52IfCwiR4HDaZZ78nl+2PH/53wP1ojIf9zE28PxXjn/p691U6a3iCx3bOu0iMwRkabZHVPHes7347yI7BeRZwHJab2iSpOxd9UHkoGzaeY9AzQBhmCr0c47/slnApuB64GhQEtgsYiEOVcUkTewVVtrgRuB24AlZEgWacqXA352xDAY6Av8j5xrQKYATwKfA1cDnwFPOOZn9BTQCLgbeBhb5f1lDtsHaAD85livn2Pbo4GX3JS9DPgv8CxwE/aM4nsRqeAsICL/B4zDfjlfC0wGpgEVPYglK/Ucf1Or/UVkLDAeWAD0B0YCvYEfJf0PiemO1zLfEc+92Pe3hmN5KBCGPfPuB9wPlAJWiEj1fMTs5On2v8C+ZyOBnsBD2EstZbLYrvNSyEZgneN5Xi9z5PjZyeEzvxYY7ij6UJpY1uKGiLQCFmM/E4OBO7CXlBaLSOsMxcOBr4CpwADsD+sPReTy7F6Q2OrnpUBr4AFHzKeAH0Skj6PYGEe8OOLv7Jjnzn+AQ9j/485ZlJ0N/I79nM0BXgDudLOt97DJ63bs6/fo8ywitwJvYv+f+gK3At8CldJvnobAO8BbwHXYz8S3YmsIncenN/bS2Fns//L92O+6pSJSK4tjgIhUdrzGyo7XNtwR591ZrVPkGWP0kcsH9oNtgKbYRFcRm1CTgTmOMvUcZdYCkmbdcsBpYFKGbdYDEoBHHNONHNt7K5s4nPsY7Jhu75hu5UHs9RzTLR3TozOUG5V2W2n2tThDuccc82vm4viJ47g9A5wEAtIs2+2YVzHNPOfrusUxHQDsA37KsN2bHOUm57B/52sZ4oijDHAFNil9m6FcMvBchvUvcax/rWP6Csf0Q7k4BoGO/cYAI7J6f9Ick2xfUy62fzY3caZZbymwKMO80YBxU3YysNvN8c72s+PhZ767Y50ebpalO07YBHIKqJBmXjhwApiVIV4DXJ5mXihwDJiYw3F5A0gCGmU49tuAtWnm9XDso7sHx3o3MNXN/NGObdyVYf4m4Bc3x2i2m8+9J5/n99PGnkWMi4BEoHGaeVUd2386zbw1wHYgKM28+o5138qwvUVppl/Cfh/WSTOvrOM9yfSZKw4PPTPOn63YD9UJ7K/NL8n8y22OcXySHDpjvxC+FFu1GSS2dWS0Y3tdHeV6YJPOxFzEsx375TNBRG4Tz1oGO/c3NcN853S3DPN/yDC9yfHX7dm6k4jUEJEJIrIH+0+WiD2Lq4D9J05ruTHmZDb7iHQ8ZmRYbyb2i9FTExxxnMOetR/Gnok59cS+Bxnfq5XYhkLOY9cL+2X2cXY7E5EbRWSl2Ja0SY79lsP+qMs3D7e/GhjpqIa8UER8We2X02cnL5/57HQFvjfGnHLOMMacwTbay/i5jjXGLExTLh77/5Tt59qxjxXG1XYEY0wy9qzyIhEJz9crcC/jcfwb93FmvL7v6ed5NTb29xzV0FnVmmw3xmx3ThhjjgBHnLGISFmgLfC1MSYpTbldwJ9kfg/S6ow9rnvTrHcO+C6bdYo0Tcb58x+gA9AMKGuMucMYcyJDmYzVec7EswCbCNI+LgQiHMudf6M9DcYYcxq4HDiA/XGw13EdZ2A2qzmrnjLGeSjDcqeMr8/ZKKRUVjtwXM+ah60CfxF7JtkBVxV1xnXT7cO4Gp44yzmrfg9nKJcEHM8qDjdedMTRDXs20BZ73Jyc71UUmd+rcNK/VyeMMXFZ7UhErgG+BrYAtwAdHfs+SjbHzlO52P5N2PficWzV8/6M1xwLUE6fnVx/5nNQCffV6YfIfDnjpJty8eT83mS3D3GzH29wdxzdxZnVd09On+fPsdXJHbHV5SdEZJZkvh0yYxwZY6mIPQZZHZ+M3y1p1SDD/7eDu3nFgramzp+/0/4izkLGBi7OZDEY+MdN+RjH32OOv7WwVV4eMcasBwY6fvG2x16nmyEirY0xf7tZxfkPVZ30t0g5rzPmJrllpaEjltuNMaln4I4EkhfOf+5qaWc6XnNE5uJZ2mOMWeN4vsRxvf4uEfnIGLMK12vvhfsva+fyY0AlESmdTUIeBEQZYwaniTeY7L+QcsOj7TvOXoYDwx2NaO7EXnM8CnyYy32ed+wnxBiTkGZ+bt6DtPL0mc/GCVyf47Sq4z6ReHsfxov7yYusvnuy/Tw7avImYGvYKjrKv4n9sdcxF/s/6Yghq+OT3XfLQTL8fzu4m1cs6Jmx7y3DJtxGxpg1bh7OL6EFQAr2umauGWOSjL1t51ns+3xBFkUXO/4OyjD/VsffJXnZfwbOaq7Ue3cdieJW98VzFI29ZnxjhvkDyd8PzCeBOOx9rAC/Yt+DOlm8V7sc5X7BngHck822y5C5Cv127PVFb8j19o0x24wxT2O/NFvmYZ97HH9T13U0suuSh22BZ59559l0aQ+2txjol6FRZBhwDa7PfX4tBjpJ+rsTArE1EOuMMTFZrZiNeDx7fbnl6ec5lTHmpDHma+wloVx9RhzVyn8BN6Rt7CgidbGfkezeg+XY45p6qc1R7Z3XH/CFnp4Z+5gx5oyIjAQ+EJEqwI/YBl21sNWli4wxXxljdojI28Cjji+QedjGERcDWx3/IOmIyNXYL7I5wC5sg4eHsMl/eRbx/CMi04DRjjPLZdjrNc8C04wxG73wsrdgv7hfEpFkbFIekdeNGWNSROQF4BMR+QzbkrkRthYgz50+GGMOicgHwGMi0s4Y85eIvAq87ziLXIw9G6yNvf72iTFmoTFmoYjMBN5yfHn8DgRjr8H9YIxZBPwEXOt4T7/H3sv8EPYavzfkuH0RKY9NeF/iau8wAFud+Ese9un87H4sIs9jGz09Tvq7CTzm4Wf+X+yPjrtF5AQ2cW3LIumNwV4a+c3xPhrsXQJlsHcZeMPb2FquXx3H4AwwDHsHRb88bnMz9jayq7HVuceMMbvzG6jj+Ob4eRaRibi+M45gX8vt5O0z8iz2Gvf3Ym+xLIetiTmNPdvOytvY4/iLiIzGvs8jsT+Wiyd/tyArig9cLV4bZVOmnqPMPVks74u9LecM9gMWBUwCmmcodx/22l48tsprEdA5wz4GO6abYquSdmH/yY5ib7Xp6Cb2emnmOTu72IP9gt7jmA7O6fXgarnZPYdjdhG2RW4s9sz2f9gzSXcth921JHXX4vthR6znsa02L8WDlsfZvTfYKtYzwNw0827HdgJyDptotmCvMUemKeNsHf4vtoGa89g3dSwPcBzTA45jsBhokzHeLN4fT15TjtvHJssJ2MsjZx2vczWOVuo5bD9Ta2rH/Esd24h1vPbbyLo1tUefHbL5zDuWD8X2vpWUdn13xwlbrbrA8XqdDfUuzlBmMhDt5rUtcvea3ZRriv0BfNrxWVwB9M5QJjetqZsBfziOqUnz/o12TAe5iT/t8XYe10wtzj35PGMvXSzCJuJ47PfJ20B4hmOz1M223b0HvbGJPc5xjObi+L/I7lhj23D84Tim+7GJ/QWKaWtqcbxopZRSSvmJXjNWSiml/CzHZCwik0TkiIi4a4mLWO+KSJSIbBSRtt4PUymllCq+PDkznoyt889KH2wn6I2xjYdye3uEUkopVaLlmIyNMUvI/l65AcDnxloBVBCRGtmUV0oppVQa3ri1qRb2nk+naMe8TL2uiMgQHPcQli1btl2zZl4ZsIYDMQc4GHOQGmE1qBlW0yvbVEoppbztr7/+OmaMqZJxvjeSsbu+bd020TbGTMTR72z79u3NmjVr3BXLtVeXvsqTvz1JiwYtGNB0AADlQspxQ/MbKBtS1iv7UEoppfLL0T9/Jt5IxtHYm8adIrH3OvpMRBnb+96CnQtYsHNB6vwTcSd4tPOjvgxFKaWUyjVvJON5wAMiMh17g/1pY0xexjrNsxtb3MieU3s4ed52t7ru0DqW7VvGnlNuf4AopZRShUqOnX44ukrsjh3k+TC2395gAGPMR44h2N7HtriOxY61mWP9szerqTN6b+V7PPTTQ5QNLsvZp/PUM59SSinldSLylzGmfcb5OZ4ZG2NuzmG5wY4CU2g0iWgCQJ3yOQ1FqpRSSvlfseyBq37F+gBsObaFldEr/RyNUkoplb1imYzrlq+b+nzR7kX+C0QppZTyQKEeQvH06dMcO3aMhISEnAtnsKz/Mk6fP035UuXZsmVLAUSnCkpISAiVK1emfPny/g5FKaV8otAm4/Pnz3P48GEiIyMpXbo0tp2Y5w6fPcy+M/uoVrYatcvXznkFVSgYY4iLiyM6OprQ0FBKlSrl75CUUqrAFdpkfPToUapUqUKZMmXytH6A2Br4M/Fn2Hfa1UFYaFAoVcpUyXVyV74hIpQpU4bKlStz9OhRatfWH1JKqeKv0Cbj8+fPU7169TyvHxwYDEBcUhxxSXHplpULKUeZ4LwleeUbYWFhHD9+3N9hKKWUTxTaZJyUlERQUN7DKx9annoV6pGUkpQ671jsMc4nnedcwjlNxoVcUFAQSUlJORdUSqlioNAmYyBfVckiQuUyldPNi02M5XzSefad2UeVspn66VaFiF5GUEqVJMXy1qasOJNzgASQkJxATr2PKaWUUr5QopJx2WA7glNSShIbD29ky7EtmpCVUkr5XYlKxoEBgVQqXYngANu4KzYxlhST4rP9T548GRFJfYSFhdG6dWvef/99n10f3b17NyLC5MmTPV7HGffu3bsLLC6llCrJCvU144LQoGIDADYd3kR8cjy7T+2mYaWGPo3hm2++ITIykjNnzvDNN9/w4IMPcuTIEf73v/8V+L5r1KjB8uXLadjQ89fcr18/li9fTo0aNQowMqWUKrlKXDJ2Kh1cmvjkeGISYny+74suuohGjRoB0KtXL6Kiohg3bpzbZJyYmEhQUJDXGjSFhobSqVOnXK1TpUoVqlTRBm9KKVVQSlQ1dVrO/quTUpKIPhPt12vHHTp0ICYmhlWrViEijB8/nscff5yaNWsSGhrKqVOnAJg1axadOnWiTJkyVKhQgRtuuIG9e/dm2t7HH39M27ZtKV26NBUrVqRbt24sW7YMcF9NvXr1anr27ElERARlypShQYMGDBs2LHW5u2rqxMRERo0aRb169QgJCaFevXqMGjWKxMTE1DLOfU2YMIHnnnuOGjVqUKFCBa655hqio6O9exCVUqoIK5LJWCTrx8SJrnITJ2ZdLiQoOLWXrkNnD9G2ncmy7JAhrm3+9Zf3X8+uXbsIDAykXLlyALz00kv8+++/TJw4kdmzZ1OqVCk++ugjBg4cSPPmzfn222+ZMGECf//9N926dSMmxnV2/9hjjzFkyBDatm3LjBkzmDp1Kl27dnWbtAHOnj3LVVddRWBgIJMnT2b+/Pk899xzOV7DvvPOOxk7dix33HEH33//PXfddRevvvoqd955Z6ayr7zyClFRUUyaNIl33nmH5cuXc+utt+bjiCmlVDFjjPHLo127diY7mzdvznIZZP2YMMFVbsKE7MvGJcSZ1ftXm9X7V5uWreOzLHfvva5trlmTbdjZ+uyzzwxgtm7dahITE82JEyfMRx99ZAICAsyAAQPMrl27DGDatGljUlJSUteLiYkx4eHh5q677kq3vV27dpng4GDz9ttvG2OM2b59uwkICDAjRozIMgbnPj777DNjjDGrV682gNmwYUOOce/atcsYY8ymTZsMYJ5//vl05caMGZNuW859de3aNV25119/3QBm//792R2ubD8DSilVFAFrjJucWCTPjLNLsWnPYocMyb5sqeBSlA+1IwN9Nn8jq/evISb+bKZyac+227XLf/zNmjUjODiYSpUqMWzYMG699VYmTZqUuvzaa69Nd414+fLlnDlzhltvvZWkpKTUR2RkJM2aNWPJkiUALFiwgJSUFIakPQg5aNy4MRUqVGDo0KFMnTqVffv25biOc3+33XZbuvnO6cWLF6eb369fv3TTF154IUCWZ+tKKVXSFMlk7E2R4ZFUL1edUkF2dKD4pPgC3+fs2bNZvXo1W7du5dy5c3z++edUqlQpdXnGVstHjhwBoEePHgQHB6d7bNq0KbUPZ+ffyMhIj2MpX748CxcupGbNmgwbNow6derQsmVLZs6cmeU6J06ccBunsy9x53KntK8NbCMysP2PK6WUKsGtqZ1KB5cmMjiS5JRkzied53jccSLKRBToPlu2bJnamtqdjC2nIyJsPJMnT6ZFixaZyoeFhQFQubLtYWz//v00bdrU43guuugiZs6cSVJSEmvWrOGVV17hxhtvZMOGDbRs2TJTeWdyPXToULpbpA4dOpQuXqWUUp4p8WfGTs5RngQpdL1ydenShbCwMKKiomjfvn2mhzPx9ujRg4CAACamrVfPhaCgIDp16sSYMWNISUlhy5Ytbst169YNgOnTp6eb/+WXXwLQtWvXPO1fKaVKqhJ/Zuzk7CrzdPxp1h5cS/2K9alUulIOa/lGeHg4r7/+OsOHD+fo0aP06dOH8uXLs3//fhYvXkz37t255ZZbaNiwISNGjOCtt94iJiaG/v37ExgYyKpVq2jWrBk33XRTpm1///33TJw4kWuvvZb69etz7tw53n33XcLCwujcubPbeFq0aMHNN9/M6NGjSUpKokuXLixfvpwxY8Zw880306pVq4I+JEopVaxoMnYoG1yWkMAQO4AEhpj4mEKTjAGGDh1K7dq1ef311/nqq69ITEykVq1adO3alYsuuii13BtvvEGjRo0YP348U6ZMoWzZsrRq1YpevXq53W7jxo0pXbo0Y8aM4eDBg4SFhdGhQwd+/fXXbK89T5kyhQYNGjBp0iRefPFFatasyRNPPMHzzz/v7ZeulFLFnvirSrZ9+/ZmzZo1WS7fsmULF1xwgQ8jso6eO8qe03sICgiiVFApaoXVIiw0zOdxKP99BpRSqqCIyF/GmPYZ5+s14wxCg2xL36SUJM4mnOVY3DE/R6SUUqq402ScQXhoOBdWvZBaYbUAOBV3yr8BKaWUKvY0GbsRGhRKuRDbNWWySS50rauVUkoVL5qMs+BMxmATslJKKVVQNBlnQUQQbOcbB2MO+jkapZRSxZkm42yUL2X7rT6fpN02KqWUKjiajLNRuYztXvJ0/GmSUrIfUlAppZTKK03G2Uh73TghOcGPkSillCrONBlnIyggiDLBZQDYfny7tqpWSilVIDQZ58B5dpyYkkhiSqKfo1FKKVUcaTLOQZ3ydQgQe5h2ndzFibgTOayRtcmTJ9tW2o5HSEgIDRs25Omnn/br2L6DBw+mXr16qdO7d+9GRJg8ebLfYlJKqZJEB4rwQOmg0pxLPEdMQgwJyQn5HkDim2++ITIykpiYGGbPns0rr7xCTEwM7733npciVkopVZR4dGYsIr1FZJuIRInIk26WlxeR70Rkg4j8IyJ3eT9U/2lcqTH1KtQDID45Pt8tqy+66CI6depEz549GT9+PD169ODTTz8lJSXFC9EqpZQqanJMxiISCHwA9AGaAzeLSPMMxYYDm40xrYHuwJsiEuLlWP0mKDCICqUqpE4fOXfEq9tv27YtcXFxHDtmB6WIjY3liSeeoH79+oSEhFC/fn1eeumlTMn66NGjDBs2jNq1axMaGkrt2rW5/fbbiY+PByAqKorbb7+d+vXrU7p0aRo0aMD999/PyZMnvRq/Ukqp/PGkmvpiIMoYsxNARKYDA4DNacoYIExEBCgHnAAK5MZceUEKYrO5dvaps5QJLoN9yfmze/duypcvT0REBElJSVx11VVs3ryZZ599lgsvvJAVK1YwZswYTpw4wZtvvgnAyZMn6dKlCydOnGDUqFG0atWKI0eOMHfuXBISEggNDeXAgQNERkYybtw4KlasyM6dO3n55Zfp27cvy5cvz3fcSimlvMOTZFwL2JdmOhromKHM+8A84AAQBtxkjMlU5yoiQ4AhAHXq1MlLvIXGlmNbaFypcWovXbmRnJxMUlJS6jXjmTNnMm7cOAIDA/niiy9YunQpixcvpmvXrgBceeWVALzwwgs88cQTVK1albfffpudO3eyZs0a2rRpk7rtm2++OfV5165dU7cB0KVLFxo1asRll13GunXr0q2nlFLKfzxJxu5O/TLecHsVsB64AmgI/CoifxhjzqRbyZiJwESA9u3b5+mmXfO8/+71TU5JZs/pPZxNOEtCcgLnk85Tntwn42bNmqWbHjZsGA888AAAP/30E3Xr1qVLly4kJbkqF3r16sWoUaNYsWIF/fv355dffqFDhw7ZJtSEhATeeOMNPv/8c/bs2ZOuxfa2bds0GSulVCHhSQOuaKB2mulI7BlwWncBs4wVBewCmlHMBAYE0qBig9TW1PvO7MthDfdmz57N6tWrmT9/Pj169GD8+PF8/vnnABw5coQ9e/YQHByc7nHxxRcDcPz48dS/kZGR2e7nqaeeYvTo0dx222388MMPrFq1ilmzZgH49VYqpZRS6XlyZrwaaCwi9YH9wCDglgxl9gJXAn+ISDWgKbDTm4EWJuGh4Rw6ewgAY0yurxu3bNmSRo0aAXDFFVfQqlUrRo4cycCBA4mIiKB+/frMmDHD7brO+4ErV67M/v37s93P9OnTueOOOxg1alTqvLNnz+YqVqWUUgUvx2RsjEkSkQeAn4FAYJIx5h8Ruc+x/CNgDDBZRDZhq7WfMMYcK8C4/So8NJwACSDFpHA87njqgBJ5ERoayuuvv86AAQMYP348vXv3ZubMmZQrVy5TdXZavXr14sUXX2TDhg20bt3abZnY2FiCg4PTzfvss8/yHKtSSqmC4VGnH8aY+cD8DPM+SvP8ANDLu6EVbs5kfCLuRL6SMUD//v3p0KEDb7zxBlFRUXz22WdceeWV/Pe//6V169YkJCSwY8cO5s2bx5w5cyhTpgwjRozgq6++okePHowaNYoLL7yQY8eOMXfuXD766CPCwsLo3bs3U6ZM4cILL6RRo0bMmjWLZcuWeekIKKWU8hbtgSuPGlZsyLbj2zgTf4aTcSepWLpivrb34osvctVVV/HJJ5/w888/M3bsWCZOnMiuXbsoW7YsDRs2pF+/foSE2Nu3K1SowJ9//smoUaMYO3Ysx48fp1q1alxxxRWpZd577z2MMTzzzDMA9O3bl2nTpqVef1ZKKVU4iL9GImrfvr1Zs2ZNlsu3bNnCBRdc4MOIcichOYGNhzcC9iy5TfU2XrnnWLkU9s+AUkrlloj8ZYxpn3G+DhSRRyGBITSNaApAiknhWOwxklOS/RyVUkqpokiTcT6EhYZRKqgUAHtO7+Hg2YN+jkgppVRRpMk4n+qUr0P5UNvxx7mEc36ORimlVFGkDbjyKTw0nOCAYE4fPU1MQoy/w1FKKVUEFeozY381LsstZ1U12OvHKv+KynuvlFLeUGiTcXBwMHFxcf4OwyMiQnCA7Vxj4+GN+R7vWEFcXFymDkuUUqq4KrTJuGrVquzfv5/Y2NgicZbkHL0pKSWJ2MRYP0dTdBljiI2NZf/+/VStWtXf4SillE8U2mvG4eHhABw4cIDExEQ/R+OZ2HOxxCbGcjr6NDXDavo7nCIrODiYatWqpX4GlFKquCu0yRhsQi5KX8jzl83nsV8fA+D0k6cJDy06sSullPKfQltNXRQ90umR1OcLdy30XyBKKaWKFE3GXhQYEMgFlW33jQ/++KCfo1FKKVVUaDL2ss8G2CEKD549qK2qlVJKeUSTsZd1jOxIrbBaJKUkse/0Pn+Ho5RSqgjQZFwAwkLDADifdN7PkSillCoKNBkXgACxh/Wp357ycyRKKaWKAk3GBaBRpUYAHDl3xM+RKKWUKgo0GReAZy57BoDl0cuLRO9hSiml/EuTcQFoULFB6nMd41gppVRONBkXgMplKtO8SnMA7v3uXj07VkoplS1NxgXkktqXADB/+3yiTkT5ORqllFKFmSbjAvJ6z9dTn8/4Z4YfI1FKKVXYaTIuIOVLladng54AjFo4ij2n9vg5IqWUUoWVJuMCNLbH2NTnGw9v9GMkSimlCjNNxgWobY22dKndBYD+0/vr2bFSSim3NBkXsIcufij1+fLo5X6MRCmlVGGlybiA3dTyJp6+9GlAq6qVUkq5p8nYBy6rexkAUzdOJTE50c/RKKWUKmw0GftAr4a9aFa5GfvO7KPTp5345p9v/B2SUkqpQkSTsQ8ESEBqVfXag2t5c/mbfo5IKaVUYaLJ2Edua3Ubn/b/FICV+1ey7/Q+P0eklFKqsNBk7CMiQvd63VOn9exYKaWUkyZjH2pQsQHXXXAdAOsOrePbzd/y7eZvOXT2kJ8jU0op5U8eJWMR6S0i20QkSkSezKJMdxFZLyL/iMhi74ZZfNzZ+k4AluxZwg3f3MAN39zANdOu8XNUSiml/CkopwIiEgh8APQEooHVIjLPGLM5TZkKwHigtzFmr4hULaB4i7weDXowvMNwDp09RFxSHPO3z2fNgTWkmBQCRCsqlFKqJMoxGQMXA1HGmJ0AIjIdGABsTlPmFmCWMWYvgDHmiLcDLS7KBJfh/b7vA3Ag5gC13qoFQL+v+vHjrT/6MzSllFJ+4smpWC0gbdPfaMe8tJoAFUVkkYj8JSJ3uNuQiAwRkTUisubo0aN5i7gYqV6uOv0a9wPg56ifOX3+tJ8jUkop5Q+eJGNxM89kmA4C2gH9gKuAZ0WkSaaVjJlojGlvjGlfpUqVXAdb3ARIAN/f8j0da3XEYFgRvcLfISmllPIDT5JxNFA7zXQkcMBNmZ+MMeeMMceAJUBr74RY/LWv2R6AzUc351BSKaVUceRJMl4NNBaR+iISAgwC5mUoMxe4TESCRKQM0BHY4t1Qi696FeoB8OgvjxJ1Isq/wSillPK5HJOxMSYJeAD4GZtgZxhj/hGR+0TkPkeZLcBPwEZgFfCJMebvggu7eGlVrVXq80snXcrfR/TQKaVUSSLGZLz86xvt27c3a9as8cu+CxtjDL/t+o3nFj7H8ujlRJSO4LEujxEeGs4tF95ChVIV/B2iUkopLxCRv4wx7TPO9+TWJlXARIQeDXpwaZ1LuX7G9fyw/Qee+u0pAI6cO8Lo7qP9G6BSSqkCpb1MFCKlgkox66ZZvNP7Ha6sfyUAGw9vZGX0Sh0HWSmlijFNxoVMSGAID3V8iFsvvBWA2Vtn0+nTTjzz+zN+jkzlxZo18MILsHy5vyNRShVmmowLqb6N+zKg6QA61uoIwOT1k0lOSfZzVMoTxsCAAdC+PfTpA6NHw6xZruWHD8Mff0BCgt9CVEoVMpqMC6lq5aoxZ9Aclv/fchpWbMjR2KNc8fkV/g5LeSApCebNg7/+gmPHoHlz+M9/XMtnzYKuXaFiRejdG157DbZv91+8Sin/02RcyIkIIzqNAODPvX+SkKynU4VdcDA0buyanjcPunRxTZcuDS1aQGws/PwzPPEEXHghrCiADtiMsYk+UZscKFWoaTIuAoZfPJyGFRuSbJJpP7F9ukfHTzry9d9f+ztElcGwYfbv+PHQsGH6ZYMHw99/w6FDMG0a9OsH8fFw3XVw8KBn29+6FZ59Fm6/HRYudF8mMRHuuw+aNIGnn87zS8mTEydg5Ur7Y0AplTNNxkVEt7rdANh0ZBN/Hfwr9bFq/ypGLx7t3+BUqu+/h8cft9XSBw/C/fdnXbZaNRg0CGbPttXWMTE2yWZl/354801o2xYuuABefBGmToUrroB7701f9uRJe7164kQ7PX9+7l/LwYPw0EOw2YNeWo1Jfw38gw+gUyd71q+UypneZ1xETLhmAg9c/ADJxtWIyxjD5VMuZ+uxrdz//f10q9eNQS0H+TFKNXYs/Pkn1K0Lw4d7tk5wMHzzjev6clZeftmeaQOEh8P110PNmvD++zYhO23dahuQ/fuv3Xbz5vDAA7l7HcZAZCSkpNi4vvoqc5nYWHtW/sMP9jF0qOsMvGtX+/f1121Svu663O1fqRLHGOOXR7t27YzKv35f9jOMxjAaE/BCgDkRe8LfIZVYa9YYA8aUL29MTEz+thUdbcz11xszdapr3vLlxlx3nTHffmtMXJxr/smTxiQluaZ79bJxtG5tzJ49edv/hx/abYAx/fu75u/ebcwHHxjTt68xpUq5yoAx/fql38Zbb9n5YWHGbN2atziUKm6ANcZNTtRq6iJuwtUTGN93PE0jmpJiUhi3YhyHzh7yd1gl0jvv2L/33APlyuV+/eRkOHIENmyAXr3g229h8mTX8k6dYOZMGDgQSpVyza9QAQID7fNz52DtWltm6VKoUydvryUszP3zJ5+0Z/zz58P58/b2reefh1WrbEO1tB55BG680Va/DxwIx4/nLRalSgLtm7qYGPHTCMatHAfY68uLBi/yazzFXXQ0vPsuvPKKTYSHDtnEl5wMO3ZAvXq53+bSpXDZZa7p5s3hp5+gdu2s13EnJQUC0vzMPnfObuf4cRgyxPPtjBsHI0ZA69awfr2dN3Omq9FZnz5QvXr22zh7Fi6+GLZsscdp27bMDdqUKkm0b+pibkTnEcQmxjJp/ST+2PsHGw5toHq56lQrV83foRU7e/fC5ZfDzp1Qtqw9M/zwQ9t6+T//yVsiBqha1fU8MjJviRjSJ2KwnYxcfz2UL28bjIWHZ71u2kTer5/tPaxfP9fygQPtw1Plytkz5vvusz9SGjRwLbv2WnuvdZ8+0LOnfa5USaVnxsVMt8ndWLJnSer0T7f+xFWNrvJjRMWDMfZWnS++sGeGJ09Cu3bw6682iQwcaDvzWLQIunXL2z5iY6FGDXs2uWlT9o25cuvSS23Dsqefhpdecl8mIQE6d7av5b//hdBQ+7pFvBNDYqJtUAa2UVjVqq5bnwICbDV8nz720aZN5h8VShUHWZ0Z68e9mHms82NcUPkCaobVBGDUwlH46weXJ7Zv9++9qCkp9vafGTOyLjN/PjRtahPV+PE2EXfrBgsW2ERsDERE2GupzlbEeVGmjL1efPCgdxMxwBtv2L9vvWWr2N356CN7vfnzz12J0FuJGFyJGOzx2rABXn0Vune3+1u2zN473b69rQ53KsQfX6W8x12rLl88tDV1wTqXcM5Uea2KYTTml6hf/B2OWzt2GBMaaszAgcYkJBhz5Igxw4enLxMdbcwDDxjTsKEx//ufMWfPut/WZ58Z06aNMfv2pZ+fkmLMrl2u6eRkY7780phrrjHm9Glj5s2zLX4rVTJmw4b06zktXWrLVK9uzKOPGrNuXfrlRcUNN9jXMWhQ5vg3bbLHAIyZO9f3sZ0+bcysWcbce68x9eoZc/y4a9k99xjTs6cxs2cbk5iY/XZ+/NGYVq2MWb++QMPNsxUrjGnRwn72VclEFq2pNRkXY8/9/pxhNObB+Q/6OxS3HnzQfgJvvtkmh6uustNHjtjlKSnGNG2a/vaZ6tWNeeghY15+2bWdJUtcy1u2NObZZ13L5s83JjDQmPvus1/2bdq4yj76qN1Hnz52ulw5Y6ZPN+axx2zickpJMea339LfPlQURUUZU7q0fa0ff+yaP3OmMWXL2vm9evn/h0ba/aekGFO3rus9q1PHvvfOz0hGN95oy82enfX29+415uGHjXnqKS8GnYWEBGNWrXJNv/uuje+zzwp+36pw0mRcAi3atSj1HuQh84aY2IRYf4eUTkSE/QSuW2enu3XLnCg++cQmxmnTjOnQIX1iXr/enilVrpx+funSrjOrV14xJiAg/fJatYz59FPXWVZcnDG33JK+jIgx//zjy6PhG1Om2Nd38cX29T/7rOs133KLMefO+TvCzE6cMObNN23tiDPWkBBjbrst8/3L111nl3/9tZ1Om9iTkmwSj4qyZSIjCy7m48eNGTvW7iM01PXjwfkDdNy4gtu3Ktw0GZdAsQmxpsLYCqkJ+Y0/3/B3SKmcZ7Ohoa4vzKlTXV+27qqjU1KM+eYbY3r0cJ1ZJCfbhH3RRbYTiquvNmbhwvRfwps3244rqlWzydldwklJMWbMGJu4O3UyZvVqb7/iwmPKFHsMzp+3STkgwJg33vD/GXFOkpNtNfTVV9sfS5D5fbrySjv/l19sEr/sMvvcGGOeeMKYmjXtZYfgYFvO2z8+/vnHmKFDXTUQYGt3nHGOGmXnjR7t3f2qokOTcQm148QO8+ayN1MT8toDa326/5077RntBx+45q1b5/qi6tjRNT852XWWW6uWT8NMdfp04U9K3hQd7UpWRcnOnZnPLq+7zpgKFeznZ9Uqe2YK9kfayJH2eWCgMYsWuS5/pG0nkB/x8cb07p2+duWqq+yPh+RkV7k33rDLRozwzn5V0ZNVMtbW1MVcg4oNGNFpBI0r2TH9Rvw8wqf7f/JJWL3atjT+739tr02xsbY1Ldj7T50CAuDmm+3z/ft9Gmaq8HDvtiAu7GrVsvf4FjX168PDD7umt2+3t5adOmWnK1aEkSPtABrnz9s+ssEOtNGtmx3JyrleXsXGulp6h4TYv6VL2z66//nH3ifeu3f6W7QqVLB/nXEq5aSdfpQAIsKbvd6k//T+7Dq1iwU7FwBwYdULvdopSEwMjB4NjRrZYQIDAuztK05vvWV7clqwwN5Hu369/bJK64UXYN066N/fa2GpEqBxY3sf+IQJtiOWhg3tj6qPPrIJ85NP7HCTDz3kKg/Zj5KVlV27bCcvH39sR+m65BI7/733oFIl+8iKJmOVFU3GJUSbGm0A2Ht6Lz2/sKdCtcJqsW/EPsQLp4LG2K4TP/3UTj/5JJw5Y59Pn257fgJYvtx+SdaoYR8ZVawIf/yR73BUCXTxxfaRVkCAvY/88cftj0TnR72N/XdgxQrPtp2SYs90x4+39507z4jnz3cl40aNct5O+fL27+nTnu1XlRyajEuIWmG1GNllJGsPrgVg8Z7F7I/Zz6GzhwgPdd8/YmhQKEEB2X9Ejh6FypXtGcKnn9rqupYtbecRABdeCDfcYOfddFPuh/JTKr9EXGfCTpdeahPyRRflvP7EibZzkp077XRIiOuznDH55+TSS+HAAdcZcnS0vYxz9dXpO0VRJY8m4xJCRHit52up0/Xfqc/uU7up+VbNLNepUKoCb/V6i8EXDU49e96wwSbdMWNg40bbdWKLFrabyMGD4ZprbP/MixfbMXqHDrVnJy1awN9/F/SrVMoz9eq5fjC6k7aP7p077aNuXbj/frj7bqhSJW/7LVUqfY3QJ5/YSzNVq8Idd8D//R80a5a3bauiTfumLqGe+e0Z3ln5Dgb3778xhrikOABuaH4DE66eQMXSFSlXzo4ClFavXnYwgNDQgo5aqYITF2cvqYwfb39YDh9u5+/bZ9s39O3rGqrSWyZPto3LNm92zevSxSblG2/M21CcqnDLqm9qTcbKrYcfMUxY8TnJVz1AUsBZygdHUL1sDXZtrE3C1K8hIYyqVeGuu+C552y/ykoVJcZAVJRto7B5M0yaZPsdB3sdeOlS38WxcqWtcZo+3Q4UAnDrrTB1qm9iANi92579JybaOFassD9MlHdpMlZuxcfb1svVqtnbRcB+Kdxzj6NAxR0w8FaIXJm6Th/zPve1HU6fPnqdSxVdf/9t2zSk1aEDDBtmrwmXLu37mM6etZd3Pv3UDppxlWPAtcWL7bXlO+5IP9SmNxgD778Pjz1mW4lff70dwjMmxn43ZLyuvm0bPPggjB0Lbdt6N5aSQEdtUun88YetWm7d2o5G9MUXrmV16kBQkK1+HjW8ITXmL4P3t3D5+XEA/CgPcEXvs5qIVZHWvLl9lCplq6VXrbKPwYP9k4jBVkvfdZc9K78qzcin48bZ+6Zr1YLrroMffoCkpPzv79gxGDDA3vKVkGB/oISH2+viAO++m3mdrl3t0KGDB+d//yoNdz2B+OKhPXD5R0qK7Zc5bU9BtWvbDuydEhLswykx0ZhDh4yJS4xL7cnr8/Wf+z54pbzs/PnC2R93RvPm2W5A0/azXrOmMU8/nfcRoH7/3W4DbM9l33zjWhYVZbscDQkx5vDh9Os5B7to2TLvr6ckI4seuLQ1dQnibCH65ZeuedWq2WtDNdM0qs54xhsUZMtBKfo06sOPUT/y3b/fkZCckFqmbY22qfcyK1VUFJVGh9dcYx8HDsCUKfb6dlQUvPyyba/xzDOebysx0bbgfvllm9YvucR+J9St6yrTsKHd37x5tiOVZ591LevQwf4tVco7r01Zes24BFi92g7W/uGHcNtttgOO55+HCy6Al15y3fPoiTeXvcljvz6WaX65kHIceewIpYP9VL+nVAlijL3U9OmnNqnWquX5unFx9v7ozZth1CibaIPcnJb9/jtceSVUrw579ri6/Ny3z17KqlHD/jhQuaMNuEqotWvtL9/z513z8vOWH4s9xst/vMzp864uhL7f/j1Hzh1h+sDptK7emsaVGhMY4OV7QJRSWTIG/vzTJuinnnJfJinJlXT/+cdeL+7WLftttm5tu6794gv7Q75fP3uP9vjxtpYtPt59IldZ02RcwsyZA3v3wnff2b6gnT791NU4w1vunns3n63/LHW6bY22fHPDNzSo2MC7O1JKuXXypL3UdP687W+7adP0y0eOhEOH0jfU9MTcuXD4sE3EBw7YnszKl7cJ+Phxe5YcGem911ESZJWM9TdNMXT+vO0p6NAh17xJk+w/z223eX9/Q9sNZdORTcQmxnL47GHWHlxLu4ntmHLtFPo31REflCpoFSvagTA+/ti2gP7gA9ey1avhjTdshyXPPZe5a9DsDBjgev7dd/Zvnz729qfQULtf5R0enRmLSG/gHSAQ+MQYMzaLch2AFcBNxphvs9umnhl7X2ysvR1i5UrbSKN1a9vY4/hxuOIK38Rw6vwpBs8ZzNxtcwHbpearPV5lSLshvglAqRLqn39sH/Blytg+r52JsmdPWzs2ciS89lr228jOJZfAsmXw1VeuoU5V7uX5PmMRCQQ+APoAzYGbRaR5FuVeBX7Of7gqL1autPcnJiba6bfftgnZV4kYbPKdfdNsXuth/+tPnT/FtL+n+S4ApUqoFi1s4o2NtX1eg22EtWCBrVp+8sm8b/u++2wihszDnhYUY0rWUJOedPpxMRBljNlpjEkApgMD3JR7EJgJHPFifCoX9u1zPR8/Hi6/3D9xiAgjLxnJxvs2ArBo9yJ+3P6jf4JRqgR55BH797337I9yZ2OukSOzH2c5J85BNSIi7Bn3hg12sJjZs/MVbrZSUuyJxNy5BbePwsSTZFwLSPM1T7RjXioRqQX8B/gouw2JyBARWSMia44ePZrbWFUWoqNtP7Z//glHjsD27faasb81jmhMmWDbafWQ74ekuy9ZKeV9vXtDkyb2h/mdd9oexapVg4cfzt92p0+3l8BmzrTT69bZ68/OaW85fdoOywp22xUq2O5BSwJPkrG7keczXmgeBzxhjEnObkPGmInGmPbGmPZV8joGmUpn+XJo1cpex+nRw44t7Mkg575QKqgUW4dvpXRQaaLPRPPVpq/8HZJSxVpAADz9NDzxhO3eEuy9xPkd/alBA/jpJ9etUM77mvfvz992nYyxLb2bNrVdc4JtaLZwof2OKwk8aU0dDdROMx0JZLzVuz0w3THmbWWgr4gkGWPmeCNI5d6UKa7+YSMj7cDl4u6nkx/VLl+bCVdP4I45d/Dqn69yR+s7CBDtEl2pgnLnna7ny5ZB+0xNhfLPm8l40yY7XOUff9jpvXttxyQXXghly9qxpA8fdvYCWHx58q24GmgsIvVFJAQYBMxLW8AYU98YU88YUw/4FhimibhgnTxpf/063Xln+kHLC5NBLQdRt3xdth7bymt/vsaSPUtIMSn+DkupYq9LF1fPWd6UNhnntauKM2fg0UehTRubiKtUseM7//GHHagjKMj2FAYl4+w4x2RsjEkCHsC2kt4CzDDG/CMi94nIfQUdoHJv5kz7axFgxIise90pDIIDg3msi+1C86nfnqLb5G7cPvt2/NXhjFIqf8LD7VlrbKy9zptbZ8/a1t9vv22T+fDhdmjGO++0Ve1OnTvbvyUhGXvU6YcxZj4wP8M8t421jDGD8x+Wysk999gb72Nj0w/yUFjd0/Ye/j7yN/vO7GPx7sV8tekrWlRpwdB2QwkMCKRCqQr+DlEp5SEReyZ77pztxyA3/duDvYY9YAD89ZftoCSrcZFLUjLW7jCLmNOnISws/a/Hombu1rlc+/W16ebd1OImPv/P54QEFkCdmlLK67p2tdd3Fy6E+vWzLxsTY0eKuuIK6NvXzouLs714ZfddduyYTfrNmtmBLQpbm5i8yHOnH8q/UlLsQN7nzsHu3Xb4slGj/B1V/gxoNoD3+7xPjXI1qFS6EsEBwXz9z9dcP+N64pPi/R2eUsoDS5bY76TsErEx9raoZs3gzTftfdDJjntuSpfO+aSicmXbre+WLcUjEWdHk3EhN2UK9Oplq3Xq17f3EM+fb39VFmXDLx7Ogf8e4Pjjx1l5z0oqla7Ed/9+x11z7/J3aEopL9i82Q7BePPNdpCJjh1h2jTbR3ZuFPdW1E6ajAu5a65JP12zJixaZH9VFhdtarThtzt+I1ACmf73dGITY/0dklIqj86ds3d6tG5tq7AjIuwAFsuWQbt2ed9uUT8ByYkmYz87dSr9WMNO8+fb6yv//mtHXale3d5O8OefuW8sURRcVP0iGkc0xmDYdmybv8NRSuWga1ebaP/5J/38lBT48ktbHT10qG0lfc89eW/nsn+/7QykRYv8x1yYaTL2k+RkGD3a9vPaty9MmGDvE378cXj2WTuI98KF8P339qb9gwdtt5f16vk78oLTqlorABbsXJBDSaWUv50+DSdOQFKSHUP57Fk7PywMPvvMDlzz0Uc2YedH9er2+2/XLvu3uNJk7CeffGJbF4K9DjxsmG2o8Prr8OKLdv6oUa7nJcHtrW4HYMJfE7RTEKUKOeeZ7uOP2y55035X9expG5t6Q2Bgyej8Q5Oxn8yaZf9GRNhfj6++am+AHzrU1bH7mDFF+xam3OrTqA91y9dlx8kdLNmzxN/hKKWysX69/fvLL/bsOCYm771x5aQk3G/sUacfKu9iY22Dq9277ZigsbG2VfSxY/YX39attvl+v36udT7Kduyr4iswIJAbmt/AG8vfYOrGqYSFhKUuKxdSjiYRTZDifn+DUkXQ8uW2tXRBKQnJuASdd/mGMfYX46FDdrpzZ5tohw+3VS0rV7p6zHKOsqRc+ja2PQJ8uu5T2n/cPvXR7INmXDfjOo6e06E3lSpMHnywYBMxQKdO9u+aNa7RqIobTcZetH+/rVZu08Y2xpo3DzZudC3/6y/bKKt8eXvv3Sef+C/Wwuqyupdxe6vbaVujbbpHuZByzNk6h5YftuT7f7/3d5hKKR+qVMl2HBIf76oeL260mtqL3n8//fSAAfb2pN9/tw2zPv3UjjVckhpl5VZQQBCf/+fzTPP3nNrDnXPuZPGexVwz7RqCAoIIlEBua3Ub7/Z5lzLBZfwQrVKqbl3f7Oell2z3mRdc4Jv9+Zr2Te0F8fF2XGFj4Ouv0y9LSSn+3bj5SopJ4e3lbzN68WjOJpxNnd+6Wmtm3jiThpUa+jE6pUqWc+fsd1/ZsjZJKs9k1Te1JuN8MMYOfH399a6qk8OHbcfmixdDkyZFY0SloibFpJCcksyWY1sYOGMgUSeiCA0MpXej3nw24DMqlq7o7xCVUsotHSjCy5KTbQ80jRqlv4YRHm7PhLt310RcUAIkgODAYFpVa8Xqe1fTv2l/4pPjmbttLl//83XOG1BK5dsff9hx1Y8c8d0+P/kEBg2yfV0XN5qM82DaNNsIa+lS17z337e90ZQq5b+4SqIKpSow56Y5/F+b/wNg4l8TGfnLSEb+MpLHf32cn6N+9nOEShVPXbvaWsEJE3y3zxkz7KXAZct8t09f0QZcuWQM3HKLa/r11+1ZcPtMlQ7KV0SEBy9+kE/Xfcq6Q+tYd2hd6rLXl73OjS1u5N3e71KtXAkZ/kUpHzrqw7sNO3e2Q8ouX25/CBQnmoxz6dVXXc+PH7dN7pX/ta5uG3HtOLEjdd6JuBO8u+pdZvwzg193/Mrz3Z4nMjwy07rta7anbgUfNQlVSuVZce78Q5NxLqRtFf3cc5qIC5vrLrgu07yh7Ydy3/f38fOOn3nk50fcrle9XHW2PbCN8NDwAo5QqeKndm3f7cvZ+cdff9mW3MWpFbe2pvbQqVN2hCWnxEQI0p8yRYIxhul/T2f21tkkm+R0y9YfWs/OkzsZ2WUkr/V8zU8RKlX0/PmnbcD18su+bSvTogVs3mzPjp3JuSjRW5u84M03YcMG+OADO0yYKvrWHFhDh487EBwQzJxBc1K741RKFU733GM7UHrzTXj0UX9Hk3t6a1M+/PyzHVf4v/+Fzz/XRFyctK/ZnhGdRpCYkkj/af35fEPm3r+UUoVHjx52DPjiNra7nhnn4Px5KF3aPv/tN9u9pSpejDE88/szvLL0FQBqhdXyeHSoyPBIxvcdT5sabQoyRKVUMaHV1Hmwaxc0aOCaPnXK3l+siqd3V77LY788RmJKYq7WKx1Umk/6f8ItF96Sc2GlVImmyTiXjLEjMDmNHw/33++/eJRvnD5/mjPxZzwqm2JS+N/i/zFp/SQArmp4FWGhrmsYl9a+lIc7PVwgcSpVksXG2uEUq1QpegNHaDLOBWPsm12unJ2+8kp7o7kO+KAyMsbw4ZoPefinh0lKScq0/OjIo1Quo4NWK+VNL7wAo0fDI4/A22/7O5rcySoZ6805GezYYfub7tnTtpyOiIBatfwdlSqsRIRhHYbRo0EPNhzakDp/9OLRbD66mf8t/h8tqrTgkjqX0LxKcwJE20wqlV8dO9q/69ZlX64o0WScRkqKTcRgz4TXrIG77/ZvTKpoaBLRhCYRTVKnF+9ZzOajm3lv1Xup8yqXqUz3et3pXrc73et1p3mV5h43FFNKuThbUu/f79cwvEqTcRppu1ibOROuy9yhk1IeefqypykfWp7zSec5dO4Qi3Yv4kDMAb7d/C3fbv4WgCplqtjkXK87Vze5mjrl6/g5aqWKBueIeAcO2MuKxeE3rV4zdti7F+o6uie+806YPNmv4ahixhhD1IkoFu1exKI9i1i4ayEHzx5MXV46qDS/3/k7nSKLYJdCSvmYMba/h3Pn4ORJqFDB3xF5Tq8Z5+DDD13P77nHf3Go4klEaBzRmMYRjbm33b0YY9h+YjuLdi/im83fsGDnAq6Zdg0r/m8FDSs19He4ShVqIvbsePt2e3ZclJJxVkp8a5LYWDv60r332ukff4RLL/VvTKr4ExGaRDRhSLshzL9lPlc1vIpjscfoP70/KSbF3+EpVeg5G9YeOODfOLylRCdjY6BsWahcGaKj7XTv3v6OSpU0wYHBzLhhBrXCarH56GZW7V/l75CUKvQmTbINuC6/3N+ReEeJTcaHD6fv1CMhwX+xKBUeGp46BOTcrXP9HI1ShV/9+raqOjDQ35F4h0fJWER6i8g2EYkSkSfdLL9VRDY6HstEpLX3Q/WeuDioXt01/fHHtvNxpfypZ4OeAKw+sNrPkSilfC3HBlwiEgh8APQEooHVIjLPGLM5TbFdQDdjzEkR6QNMBDoWRMDeMGOG6/m4cdpgSxUOdSvY5vwbDm/gwfkPerROeGg4N194My2rtizI0JQqdNauhTFjoFkzeOUVf0eTf560pr4YiDLG7AQQkenAACA1GRtjlqUpvwKI9GaQ3nb77bB7tx344WHtOlgVEnXL1yUkMIRjscd4f/X7Hq/38tKXubTOpdzX7j6ub349oUGhBRilUoVDbCzMmQOdisndgJ4k41rAvjTT0WR/1vt/wI/uFojIEGAIQJ06vu/gYMIEiIqC116D55/3+e6Vylb5UuVZcPsC1h9a7/E6W45tYerGqSzdu5Sle5fyyM+PcPdFd9OqWqvUMsGBwfRu1Jvw0PACiFop/0jb8UdxkGOnHyJyA3CVMeYex/TtwMXGmEz1aCJyOTAeuNQYczy77fq604+//oL2jtus58yBAQN8tmulClRMfAxfbfqKD9d8yIbDG9yWubftvUy8ZqKPI1Oq4DjHmg8Ots8Dikhz5Px0+hEN1E4zHQlk+i0iIq2AT4A+OSViX4uLg4EDXdPaWEsVJ2GhYQxtP5Qh7YawInoFUzdO5VT8KQDOJZxj7ra5/L7rd/8GqZSXlSoFlSrBiRO2r4gqVfwdUf54koxXA41FpD6wHxgEpBtFXUTqALOA240x/3o9yny6807Ys8f20rJ7t723WKniRkToXLsznWt3Tp2XlJJEhbEV2HFyB/vP7KdWuA5BpoqPmjVtMt6/v+gn4xxP7I0xScADwM/AFmCGMeYfEblPRO5zFHsOiADGi8h6ESk0nU5/+y18840dm3jxYihf3t8RKeU7QQFB9Gxob5ma9vc0P0ejlHcVp+vGHtWyG2PmG2OaGGMaGmNecsz7yBjzkeP5PcaYisaYixyPTPXh/pCUZAefBhg7Flq1yra4UsXS4NaDAfh47cfEJ8X7NxilvOiKK2DQIFtdXdQVkUveufPBB7Yj8bvvhjffhP/8B+6/399RKeUffRv3pV6Fevx7/F/u++E+/DVSm1Le9sQTMG1a8bi9qdgl46lT4YEH7POaNeGmm2xVdVFpaaeUtwUHBjPrxlmUCS7D5PWTeWXpK5qQlSpkilWK2roV/u//7PNGjeDFF+1zTcSqpGtTow1Trp0CwDO/P8PQ74eSkKwdsquiLTERdu6EzZtzLlvYFZvxjLdvt/cOJyTYHramTLFV1Uop6/rm1zNt4DTumnsXH6/9mM1HN9OnUZ9s1ykVVIrbW99O1bJVfRSlUp5btw46doS2bW1fEkVZsUjGKSm2avrff20/pe+/r4lYKXcGtRxEo0qNuHb6tfy570/+3PdnjuscPneY13q+5oPolMqd4tSaulgk46+/hkOHoE4dWL4cwrXXP6Wy1L5me9YMWcOkdZM4l3Auy3IbDm/gh+0/sOnIJn7b+Vuu91MupBwdanUgQPQ6kSoY1arZE6/Dh+3dM0FFOKPl2B1mQfFmd5gnT8JXX8H119s3RymVf19t+opbZ92ar21cWf9Kplw7RTsbUQWmenWbjKOjoVYR+JjlpzvMQq9iRRg+3N9RKFW89G7Um0EtB3Hk3JE8rb/h0AZ+2/UbrT5qxcSrJzKw+cCcV1Iql2rVssn4wIGikYyzUiySsVLK+yqVrsS0gXnvtevQ2UPcNfcufor6ieu/uZ6I0hE0jmhM40qOR0RjmkQ0oXGlxoSFhnkxclWS1Kxpxzbevx86dPB3NHmnyVgpVSCql6vO/Fvm8/6q93lu0XMcjzvO8ejjrIhekalstbLVbHKu1MSVsCMa06hSI8oEl/FD9KqoKC6NuIrFNWOlVOFmjOHg2YNsP76d7Se28+/xf9l+Yjvbj28n6kQU8clZd9MZGR5JiyotaFO9DW1qtKFN9TY0rNRQG4YpALZtgzNnbN8SFSv6O5qcZXXNWJOxUsqvUkwK0WeibYJ2JGtnot5xcgdJKUmZ1gkLCaN19dY2QTuSdPMqzQkJDPHDK1DKc5qMlVJFTlJKErtP7Wbj4Y2sO7iOdYfs40BM5jrJ4IBgWlZtSbsa7bjzoju5pPYliHY4oAoZTcZKqWLjyLkj6ZLzuoPr2H5ie7oybWu05eGOD3NTi5sIDQr1U6SqoB05AmPGQGAgjBvn72hypslYKVWsxcTHsOHwBn7c/iMT107kWOwxAKqWrcq9be+lfoX6qWWDAoLo16QflctU9le4ykuOHLH9S1SqBMeP+zuanGkyVkqVGHGJcUz7exrvrHyHjYc3ui1zbbNrmX3TbB9HprwtJQVCQ20PXHFxUKqUvyPKniZjpVSJY4xhyZ4lzPhnBueTztt5GL7Y+AXGGKIfjaZ6uep+jlLlV506sG8f7NgBDRr4O5rsFeseuJRSyh0RoVu9bnSr1y3d/BNxJ5i7bS4frPqAMVeM8VN0yltq1rTJ+MCBwp+Ms6I36imlSpxHOz8KwNg/x7Lu4Do/R6Pyy9kNZlHu+EPPjJVSJU7Xul158OIHeW/Ve3T8pKNPW1tHlI5gwR0LaFSpkc/2WdwVh164NBkrpUqksT3GsmTPEjYc3kBiQqLP9ns24Sxzts7hsS6P+WyfxV3LlnDZZVC5CDeO1wZcSqkSK8WkZDums7dN3TiVYfOH0bhSY/o27kutsFrUDKtJrXDH37BalA0p67N4lO9pAy6llMogQAJ8OmJUz4Y9EYTtJ7bzzsp33JYJDw1Pn6TLuZK1M2FXL1ed4MBgn8WtCp4mY6WU8pFGlRrx97C/2Xh4I/vP7OdAzAH2x7j+7j+znzPxZzgTf4Ytx7ZkuR1BqFq2arqEfUGVC7i7zd2Eh4b78BUVHvHxcPQoREb6O5K80WpqpZQqJIwxnDx/0iZnN8naOf/wucOkmJRM60eUjuCZy57h/g73UyqokPd+4UWnT0OFClCuHMTE+Dua7GmnH0opVUwkpSRx+OzhdGfU0/6exp/7/gSgdnhtnrr0KeqUr5On7ZcOLs2ldS4tMqNgGWMTcWysHU4xzHdXHnJNk7FSShVjxhjmb5/PU789xaYjm/K9vRrlanB/+/sZ0m4I1cpV80KEBatxY4iKgq1boWlTf0eTNU3GSilVAqSYFL7a9BXfbv6WxJS83bK18+ROth7bCkBIYAg3tbiJ4R2Gp94bHRoUSrmQcl6L2Ru6dYMlS+C33+CKK/wdTda0NbVSSpUAARLAba1u47ZWt+V5G8YYFu5eyHur3mPu1rl8sfELvtj4Rery4IBg3rrqLR64+AFvhOwVRb3jD+0OUymlVDoiwhX1r2D2TbPZ8dAOHuv8GJHhkVQqXYlKpSuRmJLIiJ9HsGr/Kn+Hmqqod4mpyVgppVSW6lesz+u9XmffiH0cf/w4xx8/zsMdHyYpJYlbZt5CYrLvei/Ljp4ZK6WUKlFe7fEqkeGR7Di5I9v7oX3pmmtg9mwYPtzfkeSNXjNWSimVK6FBoTSr3IzoM9H895f/FtiY0CEBIXSt25V+TfpRuUz2HU83bmwfRZUmY6WUUrl2QeULWLBzAQt2LijQ/UxaP4kACeCS2pfQv2l/+jftT5OIJgW6T3/QW5uUUkrl2pn4M/y4/UcSkhMKbB/H447zY9SPLNy1MN1tWk0jmtK/aX8GtRxE2xptAUhJgddeg4MHYdw4ECmwsPIlX/cZi0hv4B0gEPjEGDM2w3JxLO8LxAKDjTFrs9umJmOllFKeOBN/hp+jfmbev/P44d8fOHn+ZOqydjXaMaTdEG5ueTO1q4Vx+jQcOwYREX4MOBt5TsYiEgj8C/QEooHVwM3GmM1pyvQFHsQm447AO8aYjtltV5OxUkqp3EpKSWLp3qXM3DyTLzd9mZqYywaXJXjrLZz64xZeGR2WOmBEjZpQobx9fuIkHD7kfrsi0KyZa3rHTqhaBS6q25AKpSp4Lf78JOPOwGhjzFWO6acAjDGvpCkzAVhkjJnmmN4GdDfGHMxqu5qMlVJK5UdcYhwzt8xk4l8T+WPvHwWyjzk3zWFAswFe215+euCqBexLMx2NPfvNqUwtIF0yFpEhwBDH5FlH0q4MHPMgDuU5Pabep8fU+/SYep8eU++qfO3oa719POu6m+lJMnZ3GTzj6bQnZTDGTAQmpltRZI27Xwkq7/SYep8eU+/TY+p9eky9y5fH05NOP6KB2mmmI4GMfZx4UkYppZRSbniSjFcDjUWkvoiEAIOAeRnKzAPuEKsTcDq768VKKaWUcsmxmtoYkyQiDwA/Y29tmmSM+UdE7nMs/wiYj21JHYW9temuXMQwMeciKpf0mHqfHlPv02PqfXpMvctnx9NvnX4opZRSytKBIpRSSik/02SslFJK+ZnPkrGI9BaRbSISJSJPulkuIvKuY/lGEWnrq9iKIg+O562O47hRRJaJSGt/xFmU5HRM05TrICLJInK9L+Mrijw5piLSXUTWi8g/IrLY1zEWNR7875cXke9EZIPjmOamDU+JJCKTROSIiPydxfKCz0/GmAJ/YBt+7QAaACHABqB5hjJ9gR+x9yx3Alb6Irai+PDweHYBKjqe99Hjmf9jmqbc79hGi9f7O+7C/PDwc1oB2AzUcUxX9Xfchfnh4TF9GnjV8bwKcAII8XfshfkBdAXaAn9nsbzA85OvzowvBqKMMTuNMQnAdCBj/2IDgM+NtQKoICI1fBRfUZPj8TTGLDPGOHtTX4G991tlzZPPKNg+2GcCR3wZXBHlyTG9BZhljNkLYIzR45o9T46pAcIcA/iUwybjJN+GWbQYY5Zgj1NWCjw/+SoZZ9VdZm7LKCu3x+r/sL/qVNZyPKYiUgv4D/CRD+Mqyjz5nDYBKorIIhH5S0Tu8Fl0RZMnx/R94AJsx0ubgIeNMSm+Ca/YKvD85El3mN7gtS41FZCLYyUil2OT8aUFGlHR58kxHQc8YYxJlsI6WGrh4skxDQLaAVcCpYHlIrLCGPNvQQdXRHlyTK8C1gNXAA2BX0XkD2PMmQKOrTgr8Pzkq2SsXWp6l0fHSkRaAZ8AfYwxx30UW1HlyTFtD0x3JOLKQF8RSTLGzPFJhEWPp//3x4wx54BzIrIEaI0dtlVl5skxvQsYa+zFzigR2QU0A1b5JsRiqcDzk6+qqbVLTe/K8XiKSB1gFnC7nmV4JMdjaoypb4ypZ4ypB3wLDNNEnC1P/u/nApeJSJCIlMGOCLfFx3EWJZ4c073YmgZEpBrQFNjp0yiLnwLPTz45MzYF36VmieLh8XwOiADGO87kkoyO5pIlD4+pygVPjqkxZouI/ARsBFKAT4wxbm8vUR5/TscAk0VkE7Z69QljjA6rmA0RmQZ0ByqLSDTwPBAMvstP2h2mUkop5WfaA5dSSinlZ5qMlVJKKT/TZKyUUkr5mSZjpZRSys80GSullFJ+pslYqQIgIsaDx25H2cmO2yn8TkR2i8hUL29vsgflJjuPh1Ilka964FKqpOmcYXo2doSd0WnmxfssGqVUoabJWKkC4BjZJZWIxGO7fVyRxSp5IiKhxhhN6koVcVpNrVQhISJtROQPEYkVke3OXpXSLB/sqN7uKiLfiMgpYKVjWZCIPCUiW0UkXkQOiMibIlIqzfpBIjJGRHaIyHkROSYiS0Uk0yAiIjJIRLaIyDkRWZNFmdscA9g7t/WFJ8PKiciVIrLWsd4OERmal+OlVHGiZ8ZKFQ7hwFfYkaH+h+1u70MR2WaMWZih7JfANOB6XP/DU4FrgFeBZdgh9MYA9YCBjjJPACOAZ7Cj+oRjB7+olGH7l2H7M34WOO/YzvciUs8YcwpARIYAE4CvgaeAmsDLQEcRaWuMOevuRYrIBdiuBddg+1UOxVbdlwOSsz9EShVfmoyVKhzCsANPLARwjF7UC7gZyJiMvzXGPO6cEJHLgJuAO40xnztmLxCRE8BUEbnIGLMeex37F2PMO2m29Z2bWMKBi4wxJx3bP4QdoKAv8JWIBGIT9CJjzKA0cWwF/gDuBt7N4nWOAmKAXo6RmhCRZcAOdJQ2VYJpNbVShUNs2jNgx3Xg7UAdN2VnZ5juDSQAMx1V0UEiEgT84lje1fF3NXbYx5dE5FLHqD/uLHcmYodNjr/OWJoCVbFn6KmMMUuBPUC3rF4k9gfBfGcidqy3D/gzm3WUKvY0GStVOJx0My8eKOVmfsah26oCIcBZIDHN44hjeYTj78vY0Wj6Y89gj4vIZyJSOcP2TqSdSNNAzBmLs1rb3RByh8hc7Z1WDeCwm/nu5ilVYmg1tVJFT8ah1o5jr+1elkX5AwDGmETsNeVXRaQ6cDXwFlAGW83tKWeyru5mWXXs9eCsHASquZnvbp5SJYaeGStV9P2EPWstb4xZ4+aR6VqsMeaQMeYTYAHQMpf724Y9kx2UdqaIdAHqAouzWXc5tqq8bJr1agOX5DIGpYoVPTNWqogzxixyDI7+rYi8BawCUrAtqftiB5f/V0TmYjseWYutFm+Dvd48IZf7SxaR54AJjt66pgK1gJew17k/y2b1F4EbgF9E5HVs9foLaDW1KuE0GStVPNwGPIhtyfwM9nrzbuBnXIluCTYRDsdWTe8FXsMm0VwxxkwUkVhgJDAXe716PvB4Vrc1OdbbIiJ9gdext0Xtx1addwa65zYOpYoLMSbj5SellFJK+ZJeM1ZKKaX8TJOxUkop5WeajJVSSik/02SslFJK+ZkmY6WUUsrPNBkrpZRSfqbJWCmllPIzTcZKKaWUn/0/xwJ/d03LssIAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot precision and recall as a function of threshold\n",
    "\n",
    "def plot_precision_recall_vs_threshold(precisions, recalls, thresholds):\n",
    "    # remove last value from precisions and recalls array to get cleaner plot\n",
    "    plt.plot(thresholds, precisions[:-1], \"b--\", label=\"Precision\", linewidth=2)\n",
    "    plt.plot(thresholds, recalls[:-1], \"g-\", label=\"Recall\", linewidth=2)\n",
    "    plt.xlabel(\"Threshold\", fontsize=16)\n",
    "    plt.title(\"Precision and Recall as function of threshold\", fontsize=16)\n",
    "    plt.legend(loc=\"upper left\", fontsize=16)\n",
    "    plt.ylim([0, 1])\n",
    "\n",
    "plt.figure(figsize=(8, 4))\n",
    "plot_precision_recall_vs_threshold(precisions, recalls, thresholds)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>threshold</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.043684</td>\n",
       "      <td>0.149847</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.043698</td>\n",
       "      <td>0.148545</td>\n",
       "      <td>0.989796</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.043914</td>\n",
       "      <td>0.148773</td>\n",
       "      <td>0.989796</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.044042</td>\n",
       "      <td>0.149002</td>\n",
       "      <td>0.989796</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.044183</td>\n",
       "      <td>0.149231</td>\n",
       "      <td>0.989796</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>649</th>\n",
       "      <td>0.924749</td>\n",
       "      <td>0.400000</td>\n",
       "      <td>0.020408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>650</th>\n",
       "      <td>0.941895</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.010204</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>651</th>\n",
       "      <td>0.955003</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>652</th>\n",
       "      <td>0.964185</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>653</th>\n",
       "      <td>0.999771</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>654 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     threshold  precision    recall\n",
       "0     0.043684   0.149847  1.000000\n",
       "1     0.043698   0.148545  0.989796\n",
       "2     0.043914   0.148773  0.989796\n",
       "3     0.044042   0.149002  0.989796\n",
       "4     0.044183   0.149231  0.989796\n",
       "..         ...        ...       ...\n",
       "649   0.924749   0.400000  0.020408\n",
       "650   0.941895   0.250000  0.010204\n",
       "651   0.955003   0.000000  0.000000\n",
       "652   0.964185   0.000000  0.000000\n",
       "653   0.999771   0.000000  0.000000\n",
       "\n",
       "[654 rows x 3 columns]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_prec_recall = pd.DataFrame({\"threshold\" : thresholds, \"precision\" : precisions[:-1], \"recall\": recalls[:-1]})\n",
    "df_prec_recall"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The threshold at which recall and precision are equal and intersect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>threshold</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>556</th>\n",
       "      <td>0.253801</td>\n",
       "      <td>0.44898</td>\n",
       "      <td>0.44898</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     threshold  precision   recall\n",
       "556   0.253801    0.44898  0.44898"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_prec_recall[(df_prec_recall[\"precision\"] == df_prec_recall[\"recall\"]) & (df_prec_recall[\"precision\"] != 0)]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:orange\">**Generate the predicted probability of adoption for each customer**</span> pos_prediction does the following: y_pred_pos returns a boolean array with True for samples whose probability of belonging to the positive class is greater than the threshold and False for those whose probability is less than or equal to the threshold.\n",
    "\n",
    "Finally, it converts the boolean array to integers by multiplying it by 1 and returns it as the predicted classes for the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pos_prediction(X, threshold):\n",
    "    y_pred_pos = best_model.predict_proba(X)[:, 1]\n",
    "    return (y_pred_pos > threshold).astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0,\n",
       "       1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0])"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pos_prediction(X_test, 0.5)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:orange\">**Profit for different thresholds**</span> calculate the expected profits based on different thresholds:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Threshold</th>\n",
       "      <th>Profit</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>-4150.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.001</td>\n",
       "      <td>-4150.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.002</td>\n",
       "      <td>-4150.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.003</td>\n",
       "      <td>-4150.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.004</td>\n",
       "      <td>-4140.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>995</th>\n",
       "      <td>0.995</td>\n",
       "      <td>-10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>996</th>\n",
       "      <td>0.996</td>\n",
       "      <td>-10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>997</th>\n",
       "      <td>0.997</td>\n",
       "      <td>-10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>998</th>\n",
       "      <td>0.998</td>\n",
       "      <td>-10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999</th>\n",
       "      <td>0.999</td>\n",
       "      <td>-10.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1000 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    Threshold  Profit\n",
       "0         0.0 -4150.0\n",
       "1       0.001 -4150.0\n",
       "2       0.002 -4150.0\n",
       "3       0.003 -4150.0\n",
       "4       0.004 -4140.0\n",
       "..        ...     ...\n",
       "995     0.995   -10.0\n",
       "996     0.996   -10.0\n",
       "997     0.997   -10.0\n",
       "998     0.998   -10.0\n",
       "999     0.999   -10.0\n",
       "\n",
       "[1000 rows x 2 columns]"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "thresholds = np.arange(0.0, 1.0, 0.001)\n",
    "\n",
    "# Create an empty dataframe to store results\n",
    "profits_df = pd.DataFrame(columns=['Threshold', 'Profit'])\n",
    "\n",
    "# Calculate profit for each threshold\n",
    "for t in thresholds:\n",
    "    y_pred_custom = pos_prediction(X_test, t)\n",
    "    tn, fp, fn, tp = confusion_matrix(y_test, y_pred_custom).ravel()\n",
    "    profit = (tp * 50) - ((tp + fp) * 10)\n",
    "    profits_df = profits_df.append({'Threshold': t, 'Profit': profit}, ignore_index=True)\n",
    "\n",
    "profits_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Profits')"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAEGCAYAAABCa2PoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAqYUlEQVR4nO3deZwU1bn/8c/DDAzrgGwKDApERFkMyoRoEkTUIEaNaDTi1biRcDUaE5PcROO9v8SXMVezuGCiXhWjUeMSjYoaoyYgaNwCiGwCoqiMAwLDMiPDNjPP749TAw3MPt1TPd3f9+vVr6o+1VX9nBZ5OEudMndHRESkudrEHYCIiGQGJRQREUkKJRQREUkKJRQREUkKJRQREUmK3LgDiEvPnj19wIABcYchItKqzJ07d72796rpWNYmlAEDBjBnzpy4wxARaVXM7KPajqnLS0REkkIJRUREkkIJRUREkiJrx1BEJLPs3LmToqIitm3bFncoGaF9+/YUFBTQtm3bBp+jhCIiGaGoqIguXbowYMAAzCzucFo1d6ekpISioiIGDhzY4PPU5SUiGWHbtm306NFDySQJzIwePXo0urWnhCIiGUPJJHma8lsqoWSKt+6GWb8BPY5ARGKihJIJdpTD334MM38JH72Wuu9RshKpVUlJCSNHjmTkyJEccMAB9OvXj5EjR9KtWzeGDh2a9O/7xS9+wW9/+9tGndO5c+cayy+88EIef/zxZsekhNJaVVbA/D/D8z+FX/XZXb702eRcf0sJFM2FZX+H7WWw8UO4thu8cUdyri+SYXr06MH8+fOZP38+l1xyCVdeeeWu923a1P9XbUVFRQtEmVpKKK3RumVwXQ946lJ4887d5fkF8MbtUPYplG+Az9btPrbpY3juR/D0ZSFB1GVHOfxmENxzHDx8NvxvAfzx5HDs71fBL7rCC9dAaTHs3Aol7+97jcqdsG558+sqkgEqKyv5zne+w7Bhwxg/fjxbt24F4Nhjj+VnP/sZY8eO5dZbb2Xu3LmMHTuWUaNGceKJJ7J69WoApk6dytChQzn88MOZNGnSrusuWbKEY489lkGDBjF16tRd5TfddBPDhw9n+PDh3HLLLfvE4+5cfvnlDB06lJNPPpm1a9cmpZ6aNtwaLfvbnu8PPQVy28OQk+CJyXDXWChbDe06w88+CZ955FxYsyDs9zwEvvz92q//xu37lrXPh5E/gdm/Du9f/314VTvkJPhk7u73W6I/oD+Kksr8h6DfKBg0tuH1FGmia59ZzJLi0qRec2jffH5+6rAmnfvee+/x8MMPc/fdd/PNb36TJ554gvPOOw+ATZs2MWvWLHbu3MnYsWN5+umn6dWrF48++ijXXHMN9957LzfccAMrV64kLy+PTZs27bru0qVLmTlzJmVlZQwZMoRLL72UBQsW8Mc//pE333wTd+eLX/wiY8eO5Ygjjth13pNPPsmyZctYuHAhn376KUOHDuXiiy9u1u8DSiit08pXoOcQuPytPcurqkJCKQv/qmHHZ/Dns2HLOlizcPfnPl0cxkPMYPtn8JcLod+RMPYq+PfdMOM66HEwDD8zbHPbwdDTwrnl62HeA1C1c8/vXv48DD4R8vuG96vegrWLYdoJoXUE0LU/XLkIqirhvZegawEcMDzpP49Iuhk4cCAjR44EYNSoUXz44Ye7jp199tkALFu2jEWLFvHVr34VCK2aPn1Cd/bhhx/Oueeey8SJE5k4ceKuc08++WTy8vLIy8ujd+/efPrpp7z66qucfvrpdOrUCYAzzjiDV155ZY+EMnv2bM455xxycnLo27cvxx13XFLqqYTSmlTuhDvHwLp34cgL9j3epg0MGAMfvgL9j4JVb8Dyv+8+/rXfwgs/gwWPQr/C8PnnfhSOrXgJZt0Y9nsPhXMehv0G7Psdp9wcXlWVsPqd0Nr5x8/h8/8BBaP2jPX5n4RkNmBM6BZb9UboLuvUK5QDjDgLeh8GY34En8wLSaZz76T8XJK9mtqSSJW8vLxd+zk5Obu6vIBdf/G7O8OGDeP111/f5/znnnuO2bNnM336dK677joWL15c43UrKirwBk6eScUUa42htCar3gzJBOCwU2v+zAXPwM83wdkPQk5e6PaCkBxGngvfuCe8f+223ckk0cEnwCWv1pxMErXJCa2avM5w8u/2TCYAOW1D4jn7QZh4e0hQnXqHV+/Ddn9u4V9gxi9DPHePg5vVYpHsNGTIENatW7croezcuZPFixdTVVXFqlWrGDduHL/+9a/ZtGkTn332Wa3XOeaYY3jqqacoLy9ny5YtPPnkk4wZM2afzzzyyCNUVlayevVqZs6cmZQ6qIXSmqxZFLbffAAGf7Xmz1T/q6NzL/jx8pBQKrZBTrvdXVfjfxm6rQ6ZAEMnQt+RsPEjePuBcO0GzEhptI7d4crFIdGYhQH9D2bBzi0hsb343+Fzldth9QLoc3jyYxBJY+3atePxxx/niiuuYPPmzVRUVPCDH/yAQw45hPPOO4/Nmzfj7lx55ZV069at1usceeSRXHjhhYwePRqAb3/723t0dwGcfvrpzJgxgxEjRnDIIYcwdmxyxjatoc2jTFNYWOit7gFbL/5PmNX132t3J47WrqoK1i2Fyh2w4X14/OLQsvrPWWE8pl0XWPIUeBWMODPuaCWNvfvuuxx22GH1f1AarKbf1MzmunthTZ+PtcvLzO41s7VmtiihrLuZvWRm70Xb/RKOXW1mK8xsmZmdmFA+yswWRsemWqauv1C2Grr0yZxkAqE1tP/Q0EoadkbocqvcDrcfBbd/Kcwqe/yiMNmgtDiM3YhIWop7DOU+YMJeZVcB/3T3wcA/o/eY2VBgEjAsOud2M8uJzrkDmAIMjl57XzMzlLxf/9hGa2YGZ9y9e9yntAhe/t/dx286DB79VjyxiUi9Yh1DcffZZjZgr+LTgGOj/fuBl4GfRuWPuPt2YKWZrQBGm9mHQL67vw5gZn8CJgLPpzj8lrVlPXy6CAqbP1c8rXXsDv+1AqxNWAmgqiKM8yx/Poz7rH4n7ghFpBbpOCi/v7uvBnD31WZWPYe0H/BGwueKorKd0f7e5fswsymElgwHHnhgksNOsfdeDOMMw78RdySp17ZD2BZetLvsyPPD8i+v/C7c6Z/XJZbQRKR26ZhQalPTwIHXUb5voftdwF0QBuWTF1oLmPkraJMLfY+MO5L4VHf3/W9BuI9mZ3mYNXba7bpBUiQNxD2GUpNPzawPQLStXmSmCOif8LkCoDgqL6ihPDNU7AjraG1eBQOPgZzW9G+AJBvxzd37ny6G7oPCCgCzfx3WFrvvFHj6cq2KLBKTdPzbaTpwAXBDtH06ofzPZnYT0Jcw+P6Wu1eaWZmZHQW8CZwP3NbyYafI+mXw0ath/5RbYg0ldm3bw+SX4O0H4eSbQnKdNh6WRH9E8rqGVQIKvhASTcEX4PNnxxuzZJWcnBxGjBhBRUUFhx12GPfffz8dO3Zs8PnnnHMOixcv5qKLLmLjxo0cc8wxnHDCCdxyyy1MmTKlUdeKQ6wJxcweJgzA9zSzIuDnhETymJlNBj4GzgJw98Vm9hiwBKgALnP36jmklxJmjHUgDMZnzoB8abQu1wXPwH4HxRtLOug/OryqnT89LOPSJiesS/aHL8AzV4Rj/74biqL1zgYes3s9MpEU6dChA/Pnzwfg3HPP5c477+SHP/zhruOVlZXk5OTUeO6aNWt47bXX+Oijj/Y5dsstt3DeeecpodTF3c+p5dDxtXz+euD6GsrnAJnZib55Vdhm8nTh5mjbHrol9IRe8TZsKw0zxN76P1jwl3Bfy7vPwP7D4aN/weaiMP7ywcuwY0tYOeDQk2OrgmSmMWPGsGDBAl5++WWuvfZa+vTpw/z585k3bx6XXnopc+bMITc3l5tuuolx48Yxfvx41q5dy8iRI7ntttuYNm0ap5xyCsXFxRQXFzNu3Dh69uyZtGVSUiEdu7wk0ZKnoNuB4VknUr/ug8K292HwhcnQa0h4PPLffgy37TWhofMB4XkuT18GB30ZOnSr/bo7t4YFL9vn71W+LSQ1SS/PX7XnCtvJcMAIOOmGBn20oqKC559/ngkTwi1xb731FosWLWLgwIH87ne/A2DhwoUsXbqU8ePHs3z5cqZPn84pp5yyq4Uzbdo0AK644gpuuukmZs6cSc+ePZNbpyRTQklnq9+BlbPhmJ+kZn2tTJabF5IJwBHnQaeeISHk5Yd10NpE3Q5PXx7WMPv3PaF8/XthRYKPXtt9V/6OLWGl5KoKOHUqjLoAVvwDXr4hPAPm3Mfh4Bob1ZJltm7dumuZ+jFjxjB58mRee+01Ro8ezcCBAwF49dVX+d73vgfAoYceykEHHcTy5cvJz8+v7bKthhJKOvvndWE74qx442jt2naAYafXfOzUqbDin+EZMDOu212el7+7tQMweHxIHv+6FV69GTau3H3swTNg0Dg4/6mUhC9N0MCWRLIljqEkql6iHmjw8vKtkRJKOlv1ZrhLvNchcUeSudq0gTP+L6x83K4TfG4ctO0E3QeGe1wSzfoNzLs/JKivXAlHfw9mXg9zpsEHM8PU5b3XWWvbEb50RVjmX4SwdPxDDz3Ecccdx/Lly/n4448ZMmTIrsf91qRLly6UlZWpy0uaaOc22F6qG/ZawsBjwqs+Y/8rvBKdchMc+S144HSYc++ex9yhYmu4q//oyzNrUU9psu9+97tccskljBgxgtzcXO677749HpRVkylTpnDSSSfRp0+ftB6U1/L16apsDfxuSLjf4guT445GmqKyAm48KDyKecKNcNQlcUeU0bR8ffK1quXrpQ5bN4Zth/3q/pykr5zcMGAP4THJlRXxxiOSYurySkeVFeF57KCE0toddDQc8a0wk2zaCbCjPMwi21tOOzj3L+GxyiKtlBJKOlr1RpguDND3iLo/K+lv3M9CQil+G7odBMMm7n7mC4SxljfvhBm/DMvF4PDx67Dp43C820HwrSd3T3WWWrk7mfp8vZbWlOEQJZR0tHpB2F74t7pvtpPWIb8vfPcN6HHwvjPHqq1bCu//M7wgzA4beEy4B2blLFj+d93NX4/27dtTUlJCjx49lFSayd0pKSmhffvG3bSrhJKO1iyEzvvDgC/HHYkkS+96Botru4dlcxHcPAzm3q+EUo+CggKKiopYt25d3KFkhPbt21NQ0LgVOpRQ0tGni2D/YXFHIemga0G4KXPxk+Eu/p6D444obbVt23bX3egSD83ySjdVVVCyAnoOiTsSSReF0bTxh87aPa6yt5L3Yc0iKPu05eIS2YtaKOmmrDishNvz4LgjkXQx4CvQ/6gwWePVW8Ld/CXvh2NF/w6D/aWfhPftusAlr0BuA/q+O++vNeIkqZRQ0s3698K2p5ZbkYgZTH4Bpp0YlnmZM233sTa50P+LMPwM6NAd/nktTB3ZsOuOuhBOvTUVEUuWUkJJN2vfDVslFNnb128Lz3Np1ymsjJyTF6YS50bLdriHRx1sL6v/Wm8/CIufgm2b4eATworMIs2khJJuPpkLXfpClwPijkTSTa9D6l4o1AxGnNmwa3XqFVZXfn8mfPivho/Z9fgcdOzesM9K1lFCSSce3dB24BfjjkQy3WGnhNece+HZK8Nd/A1x0FfgoudSG5u0Wkoo6WTjh2Fw9SDdfyItZOR54bkvDVlnbP5D8O50uLuOh4n1OBhOv3P3ysrby8Crwn5evlZcznBKKOlk3v1hO+jYWMOQLJLbruF/3rocEO7cr6ol+ZSvhwWPwJa1YW2yTatg7eLdx488P4wDScZSQkkna9+F3sN085qkpwOGw7mP1X58Swn85QIoLwnvc3LDPTQ9PgeL/grz/gTH/wI69WiRcKXlKaGkkzWLYP+hcUch0jSdesCFz9Z8rMN+8MkceGIyHDCings5fPR6WHamIcb8EL74n40KVVJDCSVdrH0XSovCTWwimebz58BL/y88KvmDmZDboe7Pt+sIA8eGp13W5f2ZsOAxJZQ0oYSSLtYvD9uR/xFvHCKpYAYXPAsb3o8SRef6z2mIF/8nLP2/Y0sYt2l8YKFrTpJCv2S62BwtnVFvd4BIK9X70PBKpoJCeG0H/KpvEy9gYVba5yftLnKHj16D7aW1n9a+Kxz0pSZ+Z+ZSQkkXpZ+E9Zf0hEaRhjtkApz4q7D+XVO8cWdYH61oTuglKJ4POz4Dr6z/3CmzoO/IkIA+XQQ7t0F+n7BCdJZSQkkXpcXhQUyapy/ScLl5cPRlTT9/RznMvQ8WPRGWsRk0FvL7hcQwYEzN/z+Wb4AHzwirP7frBFs3hCVsAPK6wk/er/1BahkuYxKKmU0AbgVygHvc/YaYQ2qc0k/CH2QRaTkn/Dy8GsMdxl4FGz7YXdbjc1BVCbN/DbcfBef+BTo2Z3q0Qfv8Zpwfj4xIKGaWA/wB+CpQBPzbzKa7+5J4I2uE0mLdIS/SGpjBuKv3Ld+6MTxO4IOZMPWI5n/PCb+Ar1zZ/Ou0oIxIKMBoYIW7fwBgZo8ApwFJTyjXPrOYJcV1DNY1gXklD24u5pn3nUf+7/WkXltEWtJPKdxvDL0r1jTrKieWP0POzNtZ9K/X9iif0/5o5rQ/ulnXBhjaN5+fn5r8p8JmSkLpB6xKeF8E7LPCoplNAaYAHHjggS0TWQN0q9pELpVsyOkZdygi0kxz2jd/9lel5fL1z/7CiO1v7yrr7KUM2bGYtTn7A9DOd3D49nl08HK6VW3ksB0LyfEGrMkGzMq7DEh+QjF3T/pFW5qZnQWc6O7fjt5/Cxjt7t+r7ZzCwkKfM2dOS4VYt9d+Dy9eA996Ej53XNzRiEg6mv3b8MiBveXkgbWB/YeFV0Mc/s0m30RtZnPdvbCmY5nSQikC+ie8LwCKY4ql8V68Jmx7a9kVEanFUZdC78PC4H+1/Q6CPp+PL6a9ZEpC+Tcw2MwGAp8Ak4DWc8t5247h/hM9VEtEatOuExx6ctxR1KlN3AEkg7tXAJcDLwDvAo+5++K6z0oTO7fBzq1haW8RkVYsU1oouPvfgL/FHUejffw64NA3CdMMRURilBEtlFatOJrF0V+P/RWR1k0JJW7rlkGXvtChW9yRiIg0ixJK3DYXQbf0uSdGRKSplFDiVlYcFqITEWnllFDi5A6lq0OXl4hIK6eEEqdtm6Biq1ooIpIRlFDiVLo6bPPVQhGR1k8JJU6l0WN/9RwUEckASihxWv9e2Hb/XLxxiIgkgRJKnEreg/bdoJOWrReR1k8JJU7r34Oeg/UceRHJCEoocdr4IXQfFHcUIiJJoYQSp/IN0KlX3FGIiCSFEkpcKrbDzi1aw0tEMoYSSly2bgrbDvvFGoaISLIoocRl64awVUIRkQyhhBKX6psatY6XiGQIJZS4bFoVtt36xxuHiEiSKKHEZfMqaJMLXbQwpIhkBiWUuGxaFRaFbJMTdyQiIkmhhBKXzaugq7q7RCRzKKHEZZMSiohkFiWUOFTuDI/+1YC8iGQQJZQ4lBaDV6mFIiIZRQklDps1ZVhEMo8SSktzh4/fCPtdD4w3FhGRJFJCaWmLn4QZ14X9/QbEGoqISDLFklDM7CwzW2xmVWZWuNexq81shZktM7MTE8pHmdnC6NhUs/BUKjPLM7NHo/I3zWxAC1encYrn7d7PyY0vDhGRJIurhbIIOAOYnVhoZkOBScAwYAJwu5lV3/l3BzAFGBy9JkTlk4GN7n4wcDNwY8qjb442bcN28kvxxiEikmSxJBR3f9fdl9Vw6DTgEXff7u4rgRXAaDPrA+S7++vu7sCfgIkJ59wf7T8OHF/deklLZashvwD6j447EhGRpEq3MZR+wKqE90VRWb9of+/yPc5x9wpgM9Cjpoub2RQzm2Nmc9atW5fk0BuotBjytX6XiGSelCUUM/uHmS2q4XVaXafVUOZ1lNd1zr6F7ne5e6G7F/bqFdOjd0uLtSCkiGSklI0Ku/sJTTitCEi8OaMAKI7KC2ooTzynyMxyga7AhiZ8d+pV7gz3oBxyYv2fFRFpZRrUQjGz75tZvgXTzGyemY1PQTzTgUnRzK2BhMH3t9x9NVBmZkdF4yPnA08nnHNBtH8mMCMaZ0k/65ZCxTboe0TckYiIJF1Du7wudvdSYDzQC7gIuKGpX2pmp5tZEXA08JyZvQDg7ouBx4AlwN+By9y9MjrtUuAewkD9+8DzUfk0oIeZrQB+CFzV1LhSrnR12Or+ExHJQA3t8qoep/ga8Ed3f6c5M6nc/UngyVqOXQ9cX0P5HGB4DeXbgLOaGkuLKl8fth27xxuHiEgKNLSFMtfMXiQklBfMrAtQlbqwMlR5Sdh2rHESmohIq9bQFspkYCTwgbuXm1kPQreXNEZ5SbixMS8/7khERJKuoS2Ul9x9nrtvAnD3EsJd6dIY5SWhdZLG912KiDRVnS0UM2sPdAR6mtl+7B5LyQf6pji2zLOlRN1dIpKx6uvy+k/gB4TkkbCqIaXAH1IUU+YqL9GAvIhkrDoTirvfCtxqZt9z99taKKbMVV4CB+wzUU1EJCPU1+V1nLvPAD4xszP2Pu7uf01ZZJmofL26vEQkY9XX5XUMMAM4tYZjDiihNFRlBWzdpIQiIhmrvoSyMdpOc/dXUx1MRtu2CXDo2DPuSEREUqK+acPV95pMTXUgGW9ztPp+597xxiEikiL1tVDeNbMPgV5mtiCh3AB398NTFlmmWbc0bHsfFm8cIiIpUt8sr3PM7ADgBeDrLRNShtoUPTdMC0OKSIaqd+kVd18DfN7M2gGHRMXL3H1nSiPLNFvWQV5XyM2LOxIRkZRo0FpeZjaW8Bz3DwndXf3N7AJ3n53C2DLLlnXQSQPyIpK5Gro45E3AeHdfBmBmhwAPA6NSFVjGKV+vhCIiGa2hi0O2rU4mAO6+HGibmpAy1Jb10Cmm59iLiLSAhrZQ5prZNOCB6P25wNzUhJShtqyH/qPjjkJEJGUamlAuAS4DriCMocwGbk9VUBmnqipadkVdXiKSuepNKGbWBpjr7sMJYynSWFvWgldB5/3jjkREJGXqHUNx9yrgHTM7sAXiyUwbVoZt90HxxiEikkIN7fLqAyw2s7eALdWF7q6bHRti08dhq5saRSSDNTShXJvSKDLd1g1h20krDYtI5mrII4AvAQ4GFhJWHa5oicAyytaNgIU75UVEMlR9Yyj3A4WEZHIS8LuUR5SJtm6EDt2gTUNv+xERaX3q6/Ia6u4jAKL7UN5KfUgZaOtG6LBf3FGIiKRUff9k3rUApLq6mqF8gxKKiGS8+loonzez0mjfgA7R++rnoeSnNLpMsXUjdOwedxQiIilVZwvF3XPcPT96dXH33IT9JicTM/uNmS01swVm9qSZdUs4drWZrTCzZWZ2YkL5KDNbGB2bamYWleeZ2aNR+ZtmNqCpcaWMurxEJAvENUr8EjA8euLjcuBqADMbCkwChgETgNvNLCc65w5gCjA4ek2IyicDG939YOBm4MaWqkSDKaGISBaIJaG4+4sJYzJvAAXR/mnAI+6+3d1XAiuA0WbWB8h399fd3QnPZpmYcM790f7jwPHVrZe0UFUJ2zYroYhIxkuHeawXA89H+/2AVQnHiqKyftH+3uV7nBMlqc1A+txBuG0z4EooIpLxGnqnfKOZ2T+AA2o4dI27Px195hqgAnio+rQaPu91lNd1Tk0xTSF0m3HggS20NNnWjWGrhCIiGS5lCcXdT6jruJldAJwCHB91Y0FoefRP+FgBUByVF9RQnnhOkZnlAl2BDbXEdBdwF0BhYWGNSSfptm4KWyUUEclwsXR5mdkE4KfA1929POHQdGBSNHNrIGHw/S13Xw2UmdlR0fjI+cDTCedcEO2fCcxISFDxUwtFRLJEyloo9fg9kAe8FI2fv+Hul7j7YjN7DFhC6Aq7zN0ro3MuBe4DOhDGXKrHXaYBD5jZCkLLZFKL1aIhlFBEJEvEklCiKb61HbseuL6G8jnA8BrKtwFnJTXAZFJCEZEskQ6zvDJbdUJp3y3WMEREUk0JJdW2boC8fMiJq3dRRKRlKKGkWtkaPUteRLKCEkqqlRZDfp+4oxARSTkllFT7bA10UUIRkcynhJJq20qhvR79KyKZTwklldxhx2fQrnPckYiIpJwSSipVbIeqCshTQhGRzKeEkko7toStWigikgWUUFJpR1nYKqGISBZQQkml7VFCUZeXiGQBJZRU+mxt2HbqHW8cIiItQAkllcrWhG2Xmp4zJiKSWZRQUukzJRQRyR5KKKlUtiasMty2Q9yRiIiknBJKKpWtVutERLKGEkoqla1RQhGRrKGEkkplWhhSRLKHEkqqVFWphSIiWUUJJVW2boCqndBZCUVEsoMSSqpsWRe2nXvFG4eISAtRQkmV8pKw7dgz3jhERFqIEkqqbFkfth17xBuHiEgLUUJJleoWSie1UEQkOyihpMquLi+1UEQkOyihpMqW9ZDXFXLaxh2JiEiLUEJJlfIS6KTWiYhkj1gSipldZ2YLzGy+mb1oZn0Tjl1tZivMbJmZnZhQPsrMFkbHppqZReV5ZvZoVP6mmQ2IoUr7Kl+v7i4RySpxtVB+4+6Hu/tI4Fng/wGY2VBgEjAMmADcbmY50Tl3AFOAwdFrQlQ+Gdjo7gcDNwM3tlQl6rSlRFOGRSSrxJJQ3L004W0nwKP904BH3H27u68EVgCjzawPkO/ur7u7A38CJiacc3+0/zhwfHXrJVbq8hKRLJMb1xeb2fXA+cBmYFxU3A94I+FjRVHZzmh/7/Lqc1YBuHuFmW0GegDrUxZ8fdzV5SUiWSdlLRQz+4eZLarhdRqAu1/j7v2Bh4DLq0+r4VJeR3ld59QU0xQzm2Nmc9atW9e4CjXG9jKo3KEuLxHJKilrobj7CQ386J+B54CfE1oe/ROOFQDFUXlBDeUknFNkZrlAV2BDLTHdBdwFUFhYWGPSSQrd1CgiWSiuWV6DE95+HVga7U8HJkUztwYSBt/fcvfVQJmZHRWNj5wPPJ1wzgXR/pnAjGicJT66qVFEslBcYyg3mNkQoAr4CLgEwN0Xm9ljwBKgArjM3Sujcy4F7gM6AM9HL4BpwANmtoLQMpnUUpWolRaGFJEsFEtCcfdv1HHseuD6GsrnAMNrKN8GnJXUAJuremFIzfISkSyiO+VToVwrDYtI9lFCSYXyEsjJg3ad445ERKTFKKGkwpaS0DpJg/srRURaihJKKpSv1/iJiGQdJZRUKNc6XiKSfZRQUmGLll0RkeyjhJIK5SW6S15Eso4SSrJV7IDtperyEpGso4SSbLvuku8ebxwiIi1MCSXZtDCkiGQpJZRk013yIpKllFCSrXodL42hiEiWUUJJtvLoUSzq8hKRLKOEkmzl6wGDDvvFHYmISItSQkm20mLo1Ava5MQdiYhIi1JCSbbNq6Bb//o/JyKSYZRQkskd1r8H+w2IOxIRkRanhJJMG1dC6Sdw0JfjjkREpMUpoSRTyfthu/8+TyoWEcl4SijJVJ1Qug+KNw4RkRgooSTThg+gXRfdgyIiWUkJJZk2fADdB+rRvyKSlZRQkmnDB+ruEpGspYSSLJUVsOkjJRQRyVpKKMmyeRVUVUCPz8UdiYhILJRQkmXDB2GrFoqIZCkllGRRQhGRLBdrQjGzH5uZm1nPhLKrzWyFmS0zsxMTykeZ2cLo2FSzMJXKzPLM7NGo/E0zGxBDVWDDSmjbETrvH8vXi4jELbaEYmb9ga8CHyeUDQUmAcOACcDtZla9bO8dwBRgcPSaEJVPBja6+8HAzcCNLVKBva1ZAD0O1pRhEclacbZQbgZ+AnhC2WnAI+6+3d1XAiuA0WbWB8h399fd3YE/ARMTzrk/2n8cOL669dJitpfBx2/AoGNb9GtFRNJJLAnFzL4OfOLu7+x1qB+wKuF9UVTWL9rfu3yPc9y9AtgMtOwD3Ve+AlU7YfBXW/RrRUTSSW6qLmxm/wAOqOHQNcDPgPE1nVZDmddRXtc5NcU0hdBtxoEHHljTR5pm6bPQrjP0Pyp51xQRaWVSllDc/YSays1sBDAQeCfqmSoA5pnZaELLI/HpVAVAcVReUEM5CecUmVku0BXYUEtMdwF3ARQWFtaYdBptzSJ452EovBhy2yXlkiIirVGLd3m5+0J37+3uA9x9ACEhHOnua4DpwKRo5tZAwuD7W+6+Gigzs6Oi8ZHzgaejS04HLoj2zwRmROMsqVdZATN+CTl5MO6aFvlKEZF0lbIWSlO4+2IzewxYAlQAl7l7ZXT4UuA+oAPwfPQCmAY8YGYrCC2TSS0SbPHb8PefwcevwdGXQ8fuLfK1IiLpylrqH/PpprCw0OfMmdP4E+c9AC/fAKXRHIFx18Ax/6XpwiKSFcxsrrsX1nQsrVoorULH7lAwCrqcCqO/o7W7REQiSiiNdejJ4SUiInvQWl4iIpIUSigiIpIUSigiIpIUSigiIpIUSigiIpIUSigiIpIUSigiIpIUSigiIpIUWbv0ipmtAz5q4uk9gfVJDKc1UJ2zg+qcHZpT54PcvVdNB7I2oTSHmc2pbS2bTKU6ZwfVOTukqs7q8hIRkaRQQhERkaRQQmmau+IOIAaqc3ZQnbNDSuqsMRQREUkKtVBERCQplFBERCQplFAaycwmmNkyM1thZlfFHU8ymFl/M5tpZu+a2WIz+35U3t3MXjKz96LtfgnnXB39BsvM7MT4om8eM8sxs7fN7NnofUbX2cy6mdnjZrY0+u99dBbU+croz/UiM3vYzNpnWp3N7F4zW2tmixLKGl1HMxtlZgujY1PNGvlsc3fXq4EvIAd4HxgEtAPeAYbGHVcS6tUHODLa7wIsB4YCvwauisqvAm6M9odGdc8DBka/SU7c9Whi3X8I/Bl4Nnqf0XUG7ge+He23A7plcp2BfsBKoEP0/jHgwkyrM3AMcCSwKKGs0XUE3gKOBgx4HjipMXGohdI4o4EV7v6Bu+8AHgFOizmmZnP31e4+L9ovA94l/I94GuEvIKLtxGj/NOARd9/u7iuBFYTfplUxswLgZOCehOKMrbOZ5RP+4pkG4O473H0TGVznSC7QwcxygY5AMRlWZ3efDWzYq7hRdTSzPkC+u7/uIbv8KeGcBlFCaZx+wKqE90VRWcYwswHAEcCbwP7uvhpC0gF6Rx/LlN/hFuAnQFVCWSbXeRCwDvhj1M13j5l1IoPr7O6fAL8FPgZWA5vd/UUyuM4JGlvHftH+3uUNpoTSODX1J2bMvGsz6ww8AfzA3Uvr+mgNZa3qdzCzU4C17j63oafUUNaq6kz4l/qRwB3ufgSwhdAVUptWX+do3OA0QtdOX6CTmZ1X1yk1lLWqOjdAbXVsdt2VUBqnCOif8L6A0Hxu9cysLSGZPOTuf42KP42awUTbtVF5JvwOXwa+bmYfEroujzOzB8nsOhcBRe7+ZvT+cUKCyeQ6nwCsdPd17r4T+CvwJTK7ztUaW8eiaH/v8gZTQmmcfwODzWygmbUDJgHTY46p2aKZHNOAd939poRD04ELov0LgKcTyieZWZ6ZDQQGEwbzWg13v9rdC9x9AOG/4wx3P4/MrvMaYJWZDYmKjgeWkMF1JnR1HWVmHaM/58cTxggzuc7VGlXHqFuszMyOin6r8xPOaZi4Zye0thfwNcIsqPeBa+KOJ0l1+gqhabsAmB+9vgb0AP4JvBdtuyecc030GyyjkTNB0u0FHMvuWV4ZXWdgJDAn+m/9FLBfFtT5WmApsAh4gDC7KaPqDDxMGCPaSWhpTG5KHYHC6Hd6H/g90WoqDX1p6RUREUkKdXmJiEhSKKGIiEhSKKGIiEhSKKGIiEhSKKGIiEhSKKGINICZ9TCz+dFrjZl9Eu1vMrMlKfi+X5jZjxt5zme1lN9nZmcmJzKR2imhiDSAu5e4+0h3HwncCdwc7Y9kz7XAahQtTCiS0ZRQRJovx8zujp658aKZdQAws5fN7FdmNgv4fvSsiVlmNtfMXkhYFuMKM1tiZgvM7JGE6w6NrvGBmV1RXWhmP4ye7bHIzH6wdzAW/D665nPsXhQQM7sh4bt+m6ofRLKT/tUk0nyDgXPc/Ttm9hjwDeDB6Fg3dx8brZU2CzjN3deZ2dnA9cDFhAUaB7r7djPrlnDdQ4FxhGfULDOzO4DDgYuALxIW83vTzGa5+9sJ550ODAFGAPsTlle518y6R8cOdXff67tEmk0JRaT5Vrr7/Gh/LjAg4dij0XYIMBx4KXoIXg5hqQwIy6A8ZGZPEZZDqfacu28HtpvZWkJy+ArwpLtvATCzvwJjgMSEcgzwsLtXAsVmNiMqLwW2AfdELZdnm15lkX2py0uk+bYn7Fey5z/UtkRbAxZXj8O4+wh3Hx8dOxn4AzAKmJsw3lLTdRv6SNZ91lRy9wrCw6KeIDw46e8NvJZIgyihiLSMZUAvMzsawuMCzGyYmbUB+rv7TMLDvroBneu4zmxgYrR6bidCF9YrNXxmkpnlROM046Lv7Ax0dfe/AT8gTCgQSRp1eYm0AHffEU3dnWpmXQn/791CWLn6wajMCLPHNkXdYjVdZ56Z3cfuJdXv2Wv8BOBJ4DhgYXT9WVF5F+BpM2sffdeVSaqeCIBWGxYRkeRQl5eIiCSFEoqIiCSFEoqIiCSFEoqIiCSFEoqIiCSFEoqIiCSFEoqIiCTF/wd086JnzQO9DgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "profits_df.plot()\n",
    "plt.xlabel(\"Thresholds\")\n",
    "plt.ylabel(\"Profits\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a low threshold we achieve negative profit (wrong predictions). When the threshold increases, we the profit also increases, but only up to a certain point."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:orange\">**Maximum profit**</span> gives us optimal threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1340.0"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_profit = profits_df[\"Profit\"].max()\n",
    "max_profit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Threshold</th>\n",
       "      <th>Profit</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>123</th>\n",
       "      <td>0.123</td>\n",
       "      <td>1340.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>125</th>\n",
       "      <td>0.125</td>\n",
       "      <td>1340.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Threshold  Profit\n",
       "123     0.123  1340.0\n",
       "125     0.125  1340.0"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_profit_row = profits_df.loc[profits_df['Profit'] == max_profit]\n",
    "max_profit_row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimal_threshold = 0.123"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:orange\">**Propensity model**</span> Is the propensity model built based on the targeting predicted probability useful in terms of profit maximizing? Explain your findings.\n",
    "\n",
    "\n",
    "A propensity model is a predictive model that estimates the probability of an event occurring, such as the probability of a customer purchasing a product or responding to a marketing campaign. As we could see by the plotted curves, in terms of profit maximizing changing the threshold makes sense. It can help the bank to make decisions on how many customers to target."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3.5 (5 points)\n",
    "\n",
    "Now train a random forest model, with 10 decision trees and max_depth=2, what is the profit that can be achieved given the threshold that you identified earlier? Do you need to increase or decrese the threshold to maximize the profit using random forest model? Explain your result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "With the random forest model can achieve a profit of 670 €\n"
     ]
    }
   ],
   "source": [
    "random_forest = RandomForestClassifier(n_estimators=10, max_depth=2, random_state = seed)\n",
    "\n",
    "random_forest.fit(X_train, y_train)\n",
    "\n",
    "# Calculate predicted probabilities for positive class\n",
    "y_pred_pos_rf = random_forest.predict_proba(X_test)[:, 1]\n",
    "\n",
    "# Apply custom threshold\n",
    "y_pred_rf = (y_pred_pos_rf > optimal_threshold).astype(int)\n",
    "\n",
    "# Calculate profit\n",
    "tn, fp, fn, tp = confusion_matrix(y_test, y_pred_rf).ravel()\n",
    "profit = (tp * 50) - ((tp + fp) * 10)\n",
    "print(f\"With the random forest model can achieve a profit of {profit} €\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>threshold</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.087840</td>\n",
       "      <td>0.130319</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.088176</td>\n",
       "      <td>0.129252</td>\n",
       "      <td>0.969388</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.088199</td>\n",
       "      <td>0.129428</td>\n",
       "      <td>0.969388</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.088412</td>\n",
       "      <td>0.129604</td>\n",
       "      <td>0.969388</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.088416</td>\n",
       "      <td>0.129959</td>\n",
       "      <td>0.969388</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>288</th>\n",
       "      <td>0.215460</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>0.010204</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>289</th>\n",
       "      <td>0.220764</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.010204</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290</th>\n",
       "      <td>0.224557</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.010204</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>291</th>\n",
       "      <td>0.232286</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.010204</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>292</th>\n",
       "      <td>0.249292</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>293 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     threshold  precision    recall\n",
       "0     0.087840   0.130319  1.000000\n",
       "1     0.088176   0.129252  0.969388\n",
       "2     0.088199   0.129428  0.969388\n",
       "3     0.088412   0.129604  0.969388\n",
       "4     0.088416   0.129959  0.969388\n",
       "..         ...        ...       ...\n",
       "288   0.215460   0.200000  0.010204\n",
       "289   0.220764   0.250000  0.010204\n",
       "290   0.224557   0.333333  0.010204\n",
       "291   0.232286   0.500000  0.010204\n",
       "292   0.249292   0.000000  0.000000\n",
       "\n",
       "[293 rows x 3 columns]"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision_rf, recall_rf, threshold_rf = precision_recall_curve(y_test, y_pred_pos_rf)\n",
    "\n",
    "# Populate dataframe with precision and threshold\n",
    "df_precision_rf = pd.DataFrame({\"threshold\" : threshold_rf, \"precision\" : precision_rf[:-1], \"recall\": recall_rf[:-1]})\n",
    "df_precision_rf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeMAAAEdCAYAAAA/7w9RAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABXgUlEQVR4nO3dd3hU1dbA4d9KhUBI6CC9iaL0IIoKqEiz4LUhKqBeRcV+bd9VLIBeC/ZOEVCwgyKioiiCIiBFQEFEeu81JCGk7O+PPUMmySSZJJM5M5P1Pk+ezJw5ZZ1pa/Y+u4gxBqWUUko5J8LpAJRSSqnyTpOxUkop5TBNxkoppZTDNBkrpZRSDtNkrJRSSjlMk7FSSinlME3GJSAiN4iI8fhLFpEVInKniEQFMI7GruPfUIxt3LE3LrvInCMim0RkYhHrNM7z+mWJyC4R+UBEGgQo1IJiy/f6+HJOZU1E6ojIdBE54IrvXofiaCwiT4pIUy+POf48FUZEbhKRtSJyXEQOFbLevSJyuZflT7qe+4B9xxREROaIyDw/72+OD+s9KSJh2R/X8Rc1xF0FbAOquG6/DtQCHg/Q8XcCZwHri7HN165tdpZJRKHlGWA6EAOcCTwBnCoinY0xGY5GFnweB7oBN2DfO5sciqMx9nWaB2zI89i/gCOBDsgXInISMAb4ALgROFbI6vdiz+/zso9MBQtNxqWz3BizznX7exFpjv0geU3GIhINZBo/jbRijEkHFhZzm73AXn8cPwxsMMa4n7+fXa/PU0BHivm8lgOnAiuMMV84HUhBjDHLnI6hEC2ASOA9Y4zfSpS+EpFY1/eFClJaTe1fi4F4EanlURU6VESeF5EdQDqQCCAil4vIQhFJFZFDIvKZiDTMu0MRuUVEfheRNBE5KCJzRaSL67F81dQi0klEZonIfte+N4jIWx6Pe6sGjRaRp1zVfMdd/59yJSfyHOtWERkhIjtdcX8lIvWLemJEpKeIfOPaLlVEVorI/SISmWe9TSIyWUSuEZHVIpIiIktE5Bwv+7zHtf4x1zrnFhVHEX53/c/1OrhegxWu4+wTkXdFpFqedaJE5GER+cu13l4RmSkip7geryAiL7vO+6irWvwr9+Ol5ev+XdXN74nIDhFJd70eM0SkVgH7beyqFuwOnCs5Vfvu6uJ8PyxFZKKIbMq7D1/fOwW950WkO/CTa7VZHrF0d22Xr5paRM4QkR9cz0mKiPwoImd4iXebiLQXkV9c78+1InJbEU+7e/uWIvKF65zSxH6ue3vuH5jjuvujK+aJXnaF63lrBFzncX55120iIl+7zmmziDwuIhEe++ju2u5yERkrInuB3R6P+/J+vsf1+XO/BktE5F9e4u3heq3cn+nLvKzTW0QWuPZ1WESmiUjLwp5T13bu1+OYiGwXkccAKWq7UKXJ2L+aAFnAUY9ljwInA0Ow1WjHXB/yqcBfwJXArcDpwFwRiXdvKCIvYKu2fgeuBq4HfiZPsvBYvzLwnSuGG4C+wAiKrgF5D/g/4H3gYmAC8LBreV7/BZoDNwH3YKu8Pyhi/wBNgR9d213k2veTwNNe1j0XuB94DOiPLVHMEJFE9woi8m/gFeyX82XAROAjoKoPsRSksev/iWp/EXkWeAv4AbgUeBDoDXwruX9IfOw6l29c8dyCfX3ruh6PBeKxJe+LgNuBCsBCEalTipjdfN3/JOxr9iBwIXA39lJLXAH7dV8K+QNY5rpd0sscRb53injP/w7c4Vr1bo9YfscLEWkDzMW+J24ABmEvKc0VkbZ5Vq8CfAhMBvphf1i/LSLnFXZCYquf5wFtgTtdMR8CvhaRPq7VRrrixRX/Wa5l3vwL2IX9HJ9VwLpfALOx77NpwHBgsJd9vY5NXgOx5+/T+1lErgNexH6e+gLXAVOAarl3TzPgVeAl4HLse2KK2BpC9/PTG3tp7Cj2s3w79rtunojUK+A5QERquM6xhuvc7nDFeVNB24Q8Y4z+FfMP+8Y2QEtsoquKTahZwDTXOo1d6/wOiMe2lYHDwPg8+2wMHAfudd1v7trfS4XE4T7GDa77Sa77bXyIvbHr/umu+0/mWW+Y5748jjU3z3oPuJafVIznT1zP26PAQSDC47FNrmVVPZa5z+ta1/0IYCswM89++7vWm1jE8d3nMsQVRxxwPjYpTcmzXhbweJ7tz3Ztf5nr/vmu+3cX4zmIdB03GbivoNfH4zkp9JyKsf+jxYnTY7t5wJw8y54EjJd1JwKbvDzfhb53fHzPd3dt08PLY7meJ2wCOQQkeiyrAhwAPs8TrwHO81gWC+wDxhTxvLwAZALN8zz3a4DfPZb1cB2juw/P9SZgspflT7r2cWOe5X8C33t5jr7w8r735f38hmfsBcQ4B8gAWngsq+Xa/yMey5YAa4Eoj2VNXNu+lGd/czzuP439PmzosayS6zXJ954Lhz8tGZfO39g31QHsr80PyP/LbZpxvZNczsJ+IXwgtmozSmzryG2u/XV1rdcDm3TGFCOetdgvn9Eicr341jLYfbzJeZa773fLs/zrPPf/dP33Wlp3E5G6IjJaRDZjP2QZ2FJcIvZD7GmBMeZgIceo7/r7NM92U7FfjL4a7YojBVtq340tibldiH0N8r5Wv2EbCrmfu57YL7OxhR1MRK4Wkd/EtqTNdB23MvZHXan5uP/FwIOuasjWIhLIar+i3jslec8XpiswwxhzyL3AGHME22gv7/s61Rjzk8d66djPU6Hva9cxFpqctiMYY7Kwpcp2IlKlVGfgXd7ncSXe48x7fd/X9/NibOyvu6qhC6o1WWuMWeu+Y4zZA+xxxyIilYAOwCfGmEyP9TYCv5L/NfB0FvZ53eKxXQrwVSHbhDRNxqXzL6ATcApQyRgzyBhzIM86eavz3InnB2wi8PxrDVR3Pe7+v83XYIwxh4HzgB3YHwdbXNdxrihkM3fVU944d+V53C3v+bkbhVQo6ACu61nTsVXgT2FLkp3IqaLOu22uY5ichifu9dxVv7vzrJcJ7C8oDi+ecsXRDVsa6IB93tzcr9U68r9WVcj9Wh0wxqQVdCARuQT4BFgNXAt0dh17L4U8d74qxv77Y1+Lh7BVz9vzXnMsQ0W9d4r9ni9CNbxXp+8i/+WMg17WS6fo16awY4iX4/iDt+fRW5wFffcU9X5+H1ud3BlbXX5ARD6X/N0h88aRN5aq2OegoOcn73eLp7rk+Xy7eFsWFrQ1dems9PxFXIC8DVzcyeIGYJWX9ZNd//e5/tfDVnn5xBizHLjC9Ys3CXud7lMRaWuMWellE/cHqg65u0i5rzMWJ7kVpJkrloHGmBMlcFcCKQn3h7u250LXOVfPv3qBNhtjlrhu/+y6Xn+jiLxjjFlEzrn3xPuXtfvxfUA1EalYSEK+BlhnjLnBI95oCv9CKg6f9u8qvdwB3OFqRDMYe81xL/B2MY95zHWcGGPMcY/lxXkNPJXoPV+IA+S8jz3VwXsi8fcxjB+PUxIFffcU+n521eSNxtawVXWt/yL2x17nYhz/oCuGgp6fwr5bdpLn8+3ibVlY0JJx4M3HJtzmxpglXv7cX0I/ANnY65rFZozJNLbbzmPY1/nUAlad6/p/TZ7l17n+/1yS4+fhruY60XfXlSiu8756kbZhrxlfnWf5FZTuB+b/AWnYfqwAs7CvQcMCXquNrvW+x5YAbi5k33Hkr0IfiL2+6A/F3r8xZo0x5hHsl+bpJTjmZtf/E9u6Gtl1KcG+wLf3vLs0XdGH/c0FLsrTKDIeuISc931pzQXOlNy9EyKxNRDLjDHJBW1YiHR8O7/i8vX9fIIx5qAx5hPsJaFivUdc1cpLgas8GzuKSCPse6Sw12AB9nk9canNVe1d0h/wQU9LxgFmjDkiIg8Cb4pITeBbbIOuetjq0jnGmA+NMetF5GXgP64vkOnYxhFnAH+7PiC5iMjF2C+yacBGbIOHu7HJf0EB8awSkY+AJ10ly/nY6zWPAR8ZY/7ww2mvxn5xPy0iWdikfF9Jd2aMyRaR4cA4EZmAbcncHFsLUOJBH4wxu0TkTeABEelojFkqIs8Bb7hKkXOxpcEG2Otv44wxPxljfhKRqcBLri+P2UA09hrc18aYOcBM4DLXazoD25f5buw1fn8ocv8ikoBNeB+Q096hH7Y68fsSHNP93h0rIk9gGz09RO7eBD7z8T3/D/ZHx00icgCbuNYUkPRGYi+N/Oh6HQ22l0ActpeBP7yMreWa5XoOjgBDsT0oLirhPv/CdiO7GFudu88Ys6m0gbqe3yLfzyIyhpzvjD3YcxlIyd4jj2Gvcc8Q28WyMrYm5jC2tF2Ql7HP4/ci8iT2dX4Q+2M5PDndgiwU/8hp8dq8kHUau9a5uYDH+2K75RzBvsHWAeOBVnnWuw17bS8dW+U1BzgrzzFucN1via1K2oj9kO3FdrXp7CX2xh7L3INdbMZ+QW923Y8u6nzIabnZvYjnrB22RW4qtmQ7AluS9NZy2FtLUm8tvu9xxXoM22rzHHxoeVzYa4OtYj0CfOmxbCB2EJAUbKJZjb3GXN9jHXfr8H+wDdTcz31L1+MRrud0h+s5mAu0zxtvAa+PL+dU5P6xyXI09vLIUdd5LsbVSr2I/edrTe1afo5rH6muc7+egltT+/TeoZD3vOvxW7Gjb2V6bu/tecJWq/7gOl93Q70z8qwzEdjm5dzmeDtnL+u1xP4APux6Ly4EeudZpzitqU8BfnE9p8bj9XvSdT/KS/yez7f7ec3X4tyX9zP20sUcbCJOx36fvAxUyfPczPOyb2+vQW9sYk9zPUdf4vpcFPZcY9tw/OJ6TrdjE/twwrQ1tbhOWimllFIO0WvGSimllMOKTMYiMl5E9oiIt5a4iPWaiKwTkT9EpIP/w1RKKaXCly8l44nYOv+C9MEOgt4C23iouN0jlFJKqXKtyGRsjPmZwvvK9QPeN9ZCIFFE6hayvlJKKaU8+KNrUz1sn0+3ba5l+UZdEZEhuPoQVqpUqeMpp/hlwhp2JO9gZ/JO6sTXoV58gWOPK6WUUo5aunTpPmNMzbzL/ZGMvY1t67WJtjFmDK5xZ5OSksySJUu8rVZs7694n8HTBtPttG58fOXHftmnUkop5W+u8fnz8Udr6m3YTuNu9bF9HQOmWdVmAGw4uCGQh1VKKaX8wh/JeDowyNWq+kzgsDGmJHOdllizajYZrz+4vog1lVJKqeBTZDW1a6jE7kANEdmGHbc3GsAY8w52lKG+2BGkUoEbyyrYgtSuVJu46DgOpB3g0LFDJFZIDHQISimlVIkVmYyNMQOKeNxgZ4FxjIjQtGpTVu5ZyYaDG+hQV7s6K6WUCh1hM1FEs6rNWLlnJUljkogog6lZB7UdxPh+4/2+X6WUUipshsO87JTLiIqIwmDIMll+//tyzZdOn6JSSqkwFdQl48OHD7Nv3z6OHz9e5LqdYzuz4ooVfo/BYNh6eCsiwurVq/2+f5VfTEwMNWrUICEhwelQlFIqIII2GR87dozdu3dTv359KlasiIi37sxlzxhD+q50sk02jWs2pmJ0Wcz5rdyMMaSlpbFt2zZiY2OpUKGC0yEppVSZC9pq6r1791KzZk3i4uIcS8RgG4dVr1gdgM2HN7PtyDZSM1IdiyfciQhxcXHUqFGDvXv3Oh2OUkoFRNAm42PHjlG5cmWnwwCgZpwduezo8aPsOrqLbUe2ORxR+IuPj+fYsWNOh6GUUgERtNXUmZmZREUFR3hxMXE0r9ac5PRkdqfsJj0znQNphc2dkV+FqArERceVUYThJyoqiszMTKfDUEqpgAiObFcAJ6un80qskEiFqAo2GWell2jozba12xIdGV0G0YWfYHrtlVKqrAV1Mg42sZGx1K1cl2OZxas+PZx+mGyTzfGs45qMlVJK5aPJuBhEhHpVij9F46o9q0jLTGP1vtW0r9OeyIjIMohOKaVUqAraBlzhpHqcbY391SdfERUZhYggIsTHx9O2bVveeOONgF0f3bRpEyLCxIkTfd5m4sSJiAibNm0qs7iUUqo805JxANSpXIf9qftP3P/ss8+oX78+R44c4bPPPuOuu+5iz549jBgxosxjqVu3LgsWLKBZs2Y+b3PRRRexYMEC6tatW4aRKaVU+aXJOEA8GyS1a9eO5s2bA9CzZ0/WrVvHK6+84jUZZ2RkEBUV5bcGTbGxsZx55pnF2qZmzZrUrFnTL8dXSimVn1ZTB4FOnTqRnJzMokWLEBHeeustHnroIU466SRiY2M5dOgQAJ9//jlnnnkmcXFxJCYmctVVV7Fly5Z8+xs7diwdOnSgYsWKVK1alW7dujF//nzAezX14sWLufDCC6levTpxcXE0bdqUoUOHnnjcWzV1RkYGw4YNo3HjxsTExNC4cWOGDRtGRkbGiXXcxxo9ejSPP/44devWJTExkUsuuYRt27SvtlJKuYVkMhYp+G/MmJz1xowpfF1PHTsWvN6QITnrLV1aspijIwpuRb1x40YiIyNPDHLy9NNP888//zBmzBi++OILKlSowDvvvMMVV1xBq1atmDJlCqNHj2blypV069aN5OTkE/t64IEHGDJkCB06dODTTz9l8uTJdO3a1WvSBjh69Ci9evUiMjKSiRMn8s033/D4448XeQ178ODBPPvsswwaNIgZM2Zw44038txzzzF48OB86z7zzDOsW7eO8ePH8+qrr7JgwQKuu+46X542pZQqH4wxjvx17NjRFOavv/4q8DEo+G/06Jz1Ro8ufF1PHToUvN4tt+Sst2RJoWEX6Gj6UfPES08YwMxZMsdsPbjV7Nyz07zzzjsmIiLC9OvXz2zcuNEApn379iY7O/vEtsnJyaZKlSrmxhtvzLXPjRs3mujoaPPyyy8bY4xZu3atiYiIMPfdd1+BcbiPMWHCBGOMMYsXLzaAWbFiRYHbTJgwwQBm48aNxhhj/vzzTwOYJ554Itd6I0eOzLUv97G6du2aa71Ro0YZwGzfvr2wp6zQ94BSSoUiYInxkhNDsmRcWIr1LMUOGVL4up6WLi14Pc/SdseOJYu5UkwlqlasCkD3pO40qNqAurXqMnToUK677jrGj8+ZK/myyy7LdY14wYIFHDlyhOuuu47MzMwTf/Xr1+eUU07h559/BuCHH34gOzubIZ5PQhFatGhBYmIit956K5MnT2br1q1FbuM+3vXXX59rufv+3Llzcy2/6KKLct1v3bo1QIGldaWUKm+0AVcAVYmtAsDbk94mrnocVROqcuZpZ56Ymejo3qMA+Vot79mzB4AePXp43W/VqjbJ799vW2zXr1/f55gSEhL46aefGDlyJEOHDiU5OZnTTjuN4cOHc8UVV3jd5sCBA17jrFOnTq7H3apVq5brfmxsLICOPa2UUi6ajAPIXdo9p+M5HEuwiWhzymZIsY/vOLQDgGyTnWu76tVtP+WJEydy2mmn5dtvfHw8ADVq1ABg+/bttGzZ0ue42rVrx9SpU8nMzGTJkiU888wzXH311axYsYLTTz893/ru5Lpr165cXaR27dqVK16llFK+Cclq6lAXGxVLjbgaVI6pnOsvQuzLkZmdu/FUly5diI+PZ926dSQlJeX7cyfeHj16EBERwRjPevViiIqK4swzz2TkyJFkZ2ezevVqr+t169YNgI8//jjX8g8++ACArl27luj4SilVXmnJ2AEiQuPExvmWb9tsu/uYPBe0q1SpwqhRo7jjjjvYu3cvffr0ISEhge3btzN37ly6d+/OtddeS7Nmzbjvvvt46aWXSE5O5tJLLyUyMpJFixZxyimn0L9//3zHnDFjBmPGjOGyyy6jSZMmpKSk8NprrxEfH89ZZ53lNf7TTjuNAQMG8OSTT5KZmUmXLl1YsGABI0eOZMCAAbRp06b0T5JSSpUjmoyDSGy0vZaalpmW77Fbb72VBg0aMGrUKD788EMyMjKoV68eXbt2pV27difWe+GFF2jevDlvvfUW7733HpUqVaJNmzb07NnT6zFbtGhBxYoVGTlyJDt37iQ+Pp5OnToxa9asQq89v/feezRt2pTx48fz1FNPcdJJJ/Hwww/zxBNPlO5JUEqpckjylsICJSkpySxZsqTAx1evXs2pp54awIicl5yezJr9a4iQCCpGVaRO5TonWmCXR+XxPaCUCm8istQYk5R3uV4zDiKVYyoTFx1HtskmJSOF3Sm7nQ5JKaVUAGgyDiIiQsvqLWlW1bZQTstIY8vhLWw/sp2s7CyHo1NKKVVW9JpxkImMiKRKbBUEIctksSfF9jF2t8BWSikVfjQZB6HIiEhOrn4yaZlpHEw7SPLxZJLTk4mOiLaJ2k8zOCmllAoOWk0dpOJj46lVqRaVYioBsD9tP2sPrOXQsUPOBqaUUsrvtGQc5GrG1SQrO4ujx4+SlpnGsUwdQlIppcKNloyDXGxULI0SG1E9zg4xmXd0LqWUUqFPk3GIiIqwlRgZ2RkOR6KUUsrfNBmHiOiIaAAOpB1gw8ENDkejlFLKnzQZh4i46DgiJRKwCTkjS0vISikVLjQZB9DEiRMRkRN/MTExNGvWjEceeaTIuX2jI6NpW6ctcdFxAKzcs5LU46l+ieuGG26gcePGJ+5v2rQJEWHixIl+2b9SSqnCaWtqB3z22WfUr1+f5ORkvvjiC5555hmSk5N5/fXXC90uQiKoVrEaqRmpZJksdqfspklMkwBFrZRSqqz4VDIWkd4iskZE1onI/3l5PEFEvhKRFSKySkRu9H+o4aNdu3aceeaZXHjhhbz11lv06NGDd999l+zs7CK3rVWpFnUq1wHg6PGj7Dq6S1tYK6VUiCsyGYtIJPAm0AdoBQwQkVZ5VrsD+MsY0xboDrwoIjF+jjVsdejQgbS0NPbt2wdAamoqDz/8ME2aNCEmJoYmTZrw9NNPk52dTYREnBgWc9eeXdx5x500bNiQ2NhYGjRowMCBA0lPTwdg3bp1DBw4kCZNmlCxYkWaNm3K7bffzsGDBx07V6WUUvn5Uk19BrDOGLMBQEQ+BvoBf3msY4B4seM0VgYOAGVSXJPhwTEUpHnCf1NPbtq0iYSEBKpXr05mZia9evXir7/+4rHHHqN169YsXLiQkSNHcuDAAV588UUqRFUg0SRyZb8rOXTwEEPvH8qZHc9k3959fDPjG9LT04mNjWXHjh3Ur1+fV155hapVq7Jhwwb+97//0bdvXxYsWOC3+JVSSpWOL8m4HrDV4/42oHOedd4ApgM7gHigvzEmX52riAwBhgA0bNiwJPGGhaysLDIzM09cM546dSqvvPIKkZGRTJo0iXnz5jF37ly6du0KwAUXXADA8OHDefjhh6lVqxbvj36frZu38v6379Py9JYAnMRJtOnRhuwY+9R37dr1xD4AunTpQvPmzTn33HNZtmwZ7du3D/CZK6WU8saXZOytKJq3WNgLWA6cDzQDZonIL8aYI7k2MmYMMAYgKSmpREVLf5ZInXLKKafkuj906FDuvPNOAGbOnEmjRo3o0qULmZk5lQs9e/Zk2LBhLFy4kEsvvZTvv/+epE5JnNXprBPdnI5nHSc9K53k9GQSKyRy/PhxXnjhBd5//302b96cq8X2mjVrNBkrpVSQ8CUZbwMaeNyvjy0Be7oReNYYY4B1IrIROAVY5Jcow8wXX3xB/fr12bt3Ly+99BJvvfUWnTt3ZtCgQezZs4fNmzcTHR3tddv9+/ef+N+2bVuaVm164jH3gCC7U3ZTJbYKIx4dweuvv87jjz9Oly5diI+PZ9u2bVx++eVFdqVSSikVOL4k48VACxFpAmwHrgGuzbPOFuAC4BcRqQ20BHSYqAKcfvrpNG/eHIDzzz+fNm3a8OCDD3LFFVdQvXp1mjRpwqeffup1W3d/4Bo1arB9+/Zcj1WJqXLi9tHjR/n4448ZNGgQw4YNy1l+9Kifz0YppVRpFZmMjTGZInIn8B0QCYw3xqwSkdtcj78DjAQmisif2Grth40x+8ow7rARGxvLqFGj6NevH2+99Ra9e/dm6tSpVK5cOV91tqeePXvy1FNPsWLFCtq2bQtAVGQU9avUZ9uRbew8upPU1NR8JewJEyaU6fkopZQqPp8G/TDGfAN8k2fZOx63dwA9/Rta+XHppZfSqVMnXnjhBdatW8eECRO44IILuP/++2nbti3Hjx9n/fr1TJ8+nWnTphEXF8d9993Hhx9+SI8ePRg2bBitW7dm3759TPl8CneMuINKlStxwYUX8N5779G6dWuaN2/O559/zvz5850+XaWUUnnoCFxB4qmnnqJXr16MGzeO7777jmeffZYxY8awceNGKlWqRLNmzbjooouIibHdtxMTE/n1118ZNmwYzz77LPv376d27dqcf/75xFeMJ5tsRr00iqiIKB599FEA+vbty0cffcQZZ5zh5KkqpZTKQ2ybq8BLSkoyS5YsKfDx1atXc+qppwYwovDxz/5/OJJ+hBbVWpBQIcHpcEpM3wNKqXAjIkuNMUl5l+tEEWFIXL3RDqcfZn/qfrKysxyOSCmlVGG0mjoMRYj9jbUnZQ8AlaIrcXL1k4mMiHQyLKWUUgXQknEYqhtfl5pxNakRV4OYyBhSMlLYcHAD2fkHRVNKKRUEtGQchuKi42iU2AiAtIw01uxfw+H0w+xJ2XNixiellFLBI6hLxk41LgsnFaMrUrdyXQDSM9MdjsZ3+torpcqToC0ZR0dHk5aWRlxcnNOhhDz3teK9qXvZn7b/xPKKURVpWaPliWvMwSQtLa3AIUGVUircBN+3sEutWrXYvn07qampWkoqpbjouBMJN9tkn/hLyUjheNZxh6PLzRhDamoq27dvp1atWk6Ho5RSARG0JeMqVew4yzt27CAjI8PhaEJfBVMB4zHZ1q6ju8jIymDNgTXERMY4GFl+0dHR1K5d+8R7QCmlwl3QJmOwCVm/kMvG/ZPv57v133Fbx9t4++K3nQ5HKaXKtaCtplZl67kezxETGcM7S99hzNIxToejlFLlmibjcqptnbYM7z4cgCfnPOlsMEopVc5pMi7H7u58NwAH0g44HIlSSpVvmozLsQpRFQBIz0onNSPV4WiUUqr80mRcjkVIBGfUs9Mpvrf8Pe1CppRSDtFkXM4NOH0AAEO/GcrFH13scDRKKVU+aTIu5wa2GUjb2m0B+GbtN6zcs9LhiJRSqvwJ6n7GquxVj6vO8tuWc/uM23ln6Tuc9955JMQmFLj+LR1u4eFzHg5ghEopFf7EqeuESUlJZsmSJY4cW+X35+4/6TimIxnZhY921jChIZvv3RygqJRSKryIyFJjTFLe5VoyVgC0rt2aXQ/sKrCb096UvXQZ34U9KXu44+s7eOjsh05M06iUUqp0NBmrE6pVrEa1itW8PlYvvh5x0XGkZqTy1pK3+GXLLyy6ZdGJ7lFKKaVKTpOx8knF6Ios+PcC/tj9B8PnDufPPX8yeNpgzm14LgBdG3WlTe02DkeplFKhSa8Zq2JbvH0xXcZ3ITM788SyqhWqsvGejSRUKLjxl1JKlXcFXTPWrk2q2DrV68T313/PHZ3u4I5Od3BqjVM5eOwgry963enQlFIqJGnJWJXaTxt/4vz3z6dqhapsuncTVWJ12kullPJGS8aqzHRv3J1zG55rS8e/aelYKaWKS5OxKjUR4cnuTwIw7KdhRI2IImpEFDWer8HvO393NjillAoBmoyVX5zX+DwuP/VyALJMFlkmi/1p+/lhww8OR6aUUsFPrxkrv3K3sH5x/ov834//R734ejRObHzi8ejIaB7v+jjnNTnPoQiVUso5OgKXCoioCPuWal+3PQDbk7ezPXl7rnVqV6qtyVgppTxoMlZlomeznqwauirX8JpzN81l2E/DOJ513MHIlFIq+GgyVmWmVc1Wue7vT90PkGuwEKVKYuRIWL4chg+H0093OhqlSk8bcKmAiYyIBGDBtgUOR6JC2eefw+OP2/8VdGh0FSZ8SsYi0ltE1ojIOhH5vwLW6S4iy0VklYjM9W+YKhxUjKoIwL7UfSzevtjhaFQo2rULbr3V3u7eHZo3t7eNgUsugQcegDlzIKPwmUCVCjpFJmMRiQTeBPoArYABItIqzzqJwFvApcaY04Cr/B+qCnXnNDyHk+JPAuC5X59zOBoVStasgXHj4PLLYd8+6NEDfvwx5/Gff4YZM+DFF+G886BmTejfHyZNsusDHD0Kl11mq7aPa7MFFWSK7NokImcBTxpjernu/xfAGPOMxzpDgZOMMcN8PbB2bSqf1h1YR4vXWwAw6V+TuL7N9Q5HpELBv/8N48fb24mJ8Mcf0KBBzuPZ2TBvnk3IX38Nf/2V85gILF0Ka9faBA3QoYNN1K1yN2tQqsyVZjjMesBWj/vbXMs8nQxUFZE5IrJURAYVEMQQEVkiIkv27t3ra+wqjDSv1pwrW10JwH9//C/HMo85HJEKBb16wTXXwEsv2YZbnokYICICunaF55+HVatg/Xp47TXo2RNq14bWreGXX+y6kZHw++82If/0U8BPRSmvfCkZXwX0Msbc7Lo/EDjDGHOXxzpvAEnABUBFYAFwkTHmn4L2qyXj8ivbZNN+dHv+2P0HNeNqEhMZA0DVilX5ov8XNK/W3OEIVTA47TSIj7cJs2LFku8nIwOio6F9e5vIp0+HadNsSfr336FSJX9FrFTRSlMy3gZ4/g6tD+zwss5MY0yKMWYf8DPQtqTBqvAWIRGMunAUERLB3tS9JwYGWblnJc//+rzT4akgsHatrWpevrz0Laajo20DryuugCZN4IIL4N13YdEiTcQqePjSz3gx0EJEmgDbgWuAa/Os8yXwhohEATFAZ+BlfwaqwkvPZj3Z/9B+jh4/CsCWw1s4e/zZTP5jcq6BQjx1qNuBh85+6MQoXyr8GANjx8J//mPvd+lir/mWlgiccgq8+SbExdllCQn2f1YW/P23LYkr5ZQiv9WMMZkicifwHRAJjDfGrBKR21yPv2OMWS0iM4E/gGxgnDFmZVkGrkJfYoVEEiskAlC/Sn16N+/NzHUzmbp6qtf1p66eyqLti/joio+oGF2KeksVlLZvh1tugW+/tff797fJ01+uvDL/soMHbUl540ZbGq9Rw3/HU6o4dKIIFTQOph1kzqY5ZJmsfI8dST/CA98/wMFjBzm34bl8NeArEiokOBCl8pfUVFsFHeG6WHb22TB/PlStCm+9ZRtslTVjbOOwWbNsl6hvvtGBRFTZKuiasSZjFTJW7VlFr8m92J68nYFtBvL+v953OqRyKTUVdu+GPXts4mrTxreq5OPH4bffYPZs+7dwISxZYls6AzzxhO2y9OabcNJJZXsOnjZutNXhu3bBxRfbkb2io/PHPnMmdOuWU72tVEloMlZh4Z/9/9Dm7TakZ6Uze9Bsnf0pQP76y45wtXs3pKTkfmz+fDjrLO/bJSfD22/DDz/Y1stpaTmPicB778HAgfa+Mf65PlwSK1faRHvggK0e/+AD2wUKbB/mM86wfZUfeQSeftqZGFV4KE1raqWCxsnVT+bRcx8F4Pavbyc9M93hiMLHjh1w4YWwbFn+x+LiYMMGm4hjYmw/X3cJccOG/Ptxi4mxyWvWLJuITz8d7roLvvjCjozlTsTgXCIGG9d339muVJ98AkOG2CQMNgEvXWpvew4mopQ/abNUFXIeOvshJv85mTX71/D8r8/z33P/e+IxQU5MSKGKlp4OsbH2dmKiTbYXX2y7/dSrl1NarV/fDklZq5ZNwiJw332QmWnHiP7jD5gyxf5t3gx799oEHhtrB+JITLTXZGvVcvBki5CUZEfv6tXLJmJjYPRoeM5j5NbatZ2LT4U3raZWIemnjT9x/vvn51seFRHF2EvGckO7GwIfVIj59Vf417/g+++hXTt7XfScc2DxYujY0SaiIUPgo4/g5JPzb+9O1HfdBW+8kbO8alVbEu7YMWCn4lerVsGpp9ofKqedZq8pP/igfQ5OOcU+R0qVlFZTq7ByXpPzuPuMu4mKiCJSIomUSAQhMzuTHzf+WPQOFCNH2hLsW2/Z+zExtjVx06a2WjYpyY5QNWKE9+1FbFXz22/b66u33GKrenfvDt1EDDYBR0TYUb/mzbPn9/zzcPPNmohV2dGSsQobn676lP5T+pNYIZFmVZsBtqT8eLfH6duir8PRBZdjx6B6ddsyeteu3NWvf/1lG2QdOQJ9+tiqZ/dAGXkNHgzvv29Htsp77ThcrVplq9tr1nQ6EhWKtGSswt7J1W1d6qFjh1i6cylLdy7lt+2/8fQv2vw1rzFjbCLu2DH/ddBWreyUhG++acdwLigRQ8715ltuKbNQg8rnn9vn7LbbnI5EhRstGauwsuXwFvak7AEgPTOdbhO7ISIceOgA8bHxDkcXHJKToVkzW0X95Zdw6aUl39fhw7Yqu3t3Z1tDB8rmzbYaOyXFDqHZsqXTEalQoyVjVS40TGhI0klJJJ2UxNkNzybppCQyszOZt2We06E5zhj79/LLNhGfdZbtO1waCQm2lXR5SMQAjRrZCScAPvvM2VhUeNFkrMLa+U1si+uxv49l8h+T+ejPj9ifut/hqJzx8su2OnnhQnv/2WfLTxL1p/797f9PP7UtrtescTYeFR40Gauw5k7GX/z9BQO/GMi1n1/LbV+Xzwt+s2fbqQO//db2Ee7a1emIQlOPHrbf9J9/wrXX2u5OH37odFQq1GkyVmHt/CbnM+zcYVzX+jouP/VyABZuW8j6A+vJNtkORxc42dmwZYu9/cMP8NJLzsYTymJibP9ssA26AK67zrl4VHjQBlyq3MjMzqTy/yqTnmWH0Ly+zfVM+tckh6MqW1lZMGmSLRVPmmS75KxdC1WqOB1ZaJs5045UluUxwdiaNd4HR1HKkzbgUuVeVEQUD539EI0SGgG2hBzOjh+3Jbgbb7SJGOwAH5qIS++CC3LGq3Z7911nYlHhQUvGqtw5mHaQas9XA6B6xer5Hm+Q0ICfBv9EYoXEAEfmX8bAHXfAxx/DPffYyRDcLYFV6bmr/r/+2s72dOONdgzv4vr2W/i//4M77ywf/bWzs+30m3XqOB2JM3QKRaVcjDEkjU3i952/F7jO99d/z4XNLgxgVP6zY0fOfMAZGbB9OzRu7GhI5c7ff9sRuqrn/62Xy//+B4/aScg4/3z4McxHcl292nanS0jIX7NQXhSUjHXWJlXuiAiLbl7EwWMH8z02YOoAftjwA4/OfpTftv/GsK7DHIiw+DIzYepUeOUVmwi2boXKlSE6WhNxIB08aFtZP/KInYjjxx9h/XpbI1GtWv71zz7bjoOdnQ2HDgU83IBr0sQ+R+vX28FiOnRwOqLgodeMVbkUGRFJjbga+f5aVGsBwOIdi3nsp8dYtH2Rw5EWbc8e273mmmtsH2IRmxBUYO3aZWe/uvhicFf6TZtmZ74aNChnWsbFi3O26dYN5syxt8M5GWdkwNix9r3pnsN63DhnYwo2moyV8vBizxeZed1MBrax3xj3f38/T/38FEt2BMcllSNH7CxKH31kv9jBljDWr7fVom+9ZUvFZ53lbJzlUe3a0KmTHW40Pd1Ow3j//XZKya+/hscesz+YzjjDTlvpduqp9n84J+Nx4+yPkn797OxXAB98YIcVVZYmY6U8VIyuSK/mvRhx3ggiJIJ5W+bx2E+PcdVnVzkdGsbAv/8NTzwBL76YM3rW8eP2f+fOcPvtUKmSczGWZyK29Neggb1/9tl2+MzJk+39//3PjtpVubId09stIcH+P3Qo5wdWOElJyZmG85ZbbEPCM8+0Pyx1SNEcmoyV8qJxYmOm9Z/Go+c+SmxkLJsObWL30d2OxjR+vJ3OEHKGZIScwTyiowMfk8qtalU7EEifPrZ1NEDfvvD44/Z2UhIsWwZXefy2i462je6OHg18vIHwyiu2Cv+MM+ByO+7OiVbjWlWdQ1tTK1WE9qPbs3zXcqpXrM7swbNpU7tNQI+/fTtMnGhLVqmpts/w9dfnPD59OvznPzB8uI4EFcz+/tvOllWefjTt2wdNm9qq+9mz7aQiYH941K1r+7yvXGl/xJQX2rVJqRKavmY6Q78eyvbk7cRGxvJSr5cY1HYQgvdZFqIiooiNii3x8dLT7ZSE551nGwOde65t/ANwww0wYUKJd61UQP3nP3aCkl697Khlnv74w86dHVXO+vRoMlaqFFIzUrl35r2M/X1sketGRUQx+V+T6X96/yLX9WbGDNsXc/hw+O9/bWmqc2fb8OXCC21XGBVerr7aliI/+6zovsmhYutWaN7ctmlYtsy2NFfaz1ipUomLjmPMJWPo0bQHD3z/APvTvE/DmJGVQUZ2Br9s+aXEyXjatJzb0dF2LOnYkhe0VQiYNw927rS1IuGiXj07ROiffxaeiPftg/37oWXLgIUWlLRkrJQfjVk6hltn3ApA2qNpVIiqUKzts7LstbS9e201XuvWZRGlCjb169u2AVu3lmxIzVD13Xdw6aV2Os9Zs5yOJjB0ogilAqBVzVYnbr+68NVib//aazYRN21qu4Co8iEy0v73nAUqlO3d69t6nTvbc//hB9iwoWxjCnaajJXyo3MansN7l70HwNO/PM2elD0+b/vdd/DAA/b2c8/l9CNW4c/dDiA7DKbYnjfP9rV+4omi101MzOnmVd5nvdJkrJSfDWo7iL4t+pJ8PJknfvLhGwk7XOD119sv48cegyuvLOMgVVAJl2RsDDz8sL327euPSfeIXBMm2DHWyytNxkqVgRcufIFIiWTM72N47bfX+G7dd17XMybni2vGDNt468knAxqqCgLuaupQT8ZffQXz59uhWe+/37dtzjnHNt7auRO++aZs4wtmmoyVKgOn1jyV25JuI9tkc8/Me+j9QW++X/99vvWmTbOTPMyYYa+f9eunXZfKoz59bPemUB7KNCvLdsUDGDYM4uN9204kp3Q8tuieg2FLuzYp5We7dtnGKI+f/TQxkTEs27GCOVtm89+ZI1gYWY8NG+zj69fbYRAxsHZrPSDR4ciVU14tflu/oPP++/DXX3aaxFtvLd62gwfbBJ6Zaf/K20AgoF2blPIrY6BtW9u3slIlOPlkqFrnCEu7NuRw+uECt0uITWDzvZtJqJAQwGiV8o9jx+x7fetWOzFGSYZl3bcPatTwf2zBRgf9UCoA/v7bJuKICDtbzbJlULlyFZ78z6u8vvRFDh3OIjYWYmLtQB4xMbD58CYOpx9m5Z6VnN3wbKdPQTlg2zab0Bo0CM0BXg4dgjZtoFo1GDCgZPsoD4m4MD4lYxHpDbwKRALjjDHPFrBeJ2Ah0N8YM8VvUSoVIoYPt/+zs20pYdkyO1tN7dqDub/HYK/bXDPlGj5Z9QkbDm7QZFxOXXIJLF9u56Zu397paIqvTh3b7uHIkdK1eTDGNgCLj7fJvTwp8mkTkUjgTaAP0AoYICKtCljvOcB7s1GlQsjRo/aLpbj69rX/r7nGjqR0ySV20vnCNKvaDID1B9cX/4AqLIRL16YqVUq3/Ztv2tbVTz3ln3hCiS+/Yc4A1hljNhhjjgMfA/28rHcXMBXwfZQDpYLQsWN2LN1mzeDtt21pBWDzZnt771477dvnn8Mzz8CNN8LUqXadQYPs/MIffuj78ZpVs8l4+NzhTFimUzKVR6GajHfsgIsugsWL/bO/yy6zz8W0ab6P4hUufEnG9YCtHve3uZadICL1gH8B7xS2IxEZIiJLRGTJ3vL2TKuQ8e67tqXzvn0wdKidHB3soAQdO0KtWnbM6CuugEcesXMNDxniahmNve5XnNGzkk7KacvxxBzfBglR4SVUk/GIEbZv8LNeL1wWX/36tnYpI8O2zi5PfEnG3r5W8jbBfgV42BhT6MiqxpgxxpgkY0xSzZo1fQxRqcBJT8/9xXLmmXZuYbBD951+OiQk2KnhLroI7rvPlp6nTLGNV0qiTe02zB40G4D9aft5bPZjPDb7MZbs0N4G5UUoJuM1a2DcOBv700/7b7+33GL/jx1rryGXF7404NoGNPC4Xx/YkWedJOBjscWBGkBfEck0xkzzR5BKBcrEibZl6+mnw4oVuRuj3Huv/SsLnet3JiYyhtSMVJ76xV4wGzV/FItuWUSb2uWsJUs5FIrJeNgwO9DHzTfbgWv8pW9fO3PZmjV2nOtzz/XfvoOZLyXjxUALEWkiIjHANcB0zxWMMU2MMY2NMY2BKcBQTcQqFKWk2FGwnn46sCNhxUXH8eU1XzKi+whGdB9B3xZ9Sc9Kp88Hfdh8aHPgAlGOcHfrWbfO2Th8tXixrQ2qUMH/w7dGRdl2GGBL3uWFT4N+iEhfbFV0JDDeGPO0iNwGYIx5J8+6E4EZRXVt0kE/lFMyM2HJEttAq0YNOHgQNm60pZJOnZyOzkrNSCVpTBKr962mSmwVPr7iY/q06ON0WKqMLFxoS5mdOwf/6FPGQI8eMHu2nRTCX9eLPa1fD1262JG8Rozw//6dVNCgHzoClwpbO3fC7bfba1AXXZSzvE8fmDnT3k5IgMOugbHOPRd+/jnwcRZk3YF13DDtBn7d+iuCMLz7cG7vdDtREVEkVkh0OjxVTm3YYBswxsTY21Wrls1xwnVYzIKSsQ5Jr8LWAw/Al1/CxRfbX/G33mrvd+uWs87hw1C5sv1yadnSuVi9aV6tOT/f+DMjzxsJwONzHqfmqJpUfa4qncd15sM/P+R41nGHo1TlTdOmtjr900/LLhFDeCbiwmjJWIWt2rVhT55e7z16wLff2u5Lmzfba1PNmxevK5ITvl37Lfd+dy/7UveRmpHKscxjANStXJc7Ot3BkI5DqFlJeyiEsp9/toNddOmi02i6HT5s++yfc479wRwOtJpalTvz58Nvv9lp3SpVsjPD3HwztMo3flxoSc1IZfIfk3n1t1f5a+9fAMRGxtIgoYHX9QVhcNvBPNr10UCGqYrphx/gwgvtMJArVjgdTX7Hj8Mnn9ixpwNVan3gAXjxRduPf/TowByzrGkyVuXWkSO21WdMjNOR+Jcxhh83/sirv73K1/98jcnX/T+HIPx28290qhckLdRUPunpUL26bdG/dasdACOYvPkm3Hmn7W0wbVpgjrl6tf3xXLmybQNSuXJgjluWNBmrsJeenjMcZbt2djCO8mJPyh4OH/M+ReMbi97gtUWv0axqM85qcBaCMKjtIHo07RHgKFVR+vWD6dPtgBc33+x0NDmOHrW9D/bsscPA/utfgTv2OefAr7/aS0s33RS445YVTcYq7D30EIwaZW/XqgW7dgX/teBAOHr8KK3ebMXWIzmj2nao24GlQ5Y6GJXyZvRouO02m+w+/9zpaHKMGAFPPGFHpJs/P7Cfq4kTbduOs86yxw51moxV2Ni9O/dMSBdfbKef2749Z9nGjdC4caAjC16bDm3il82/sDtlNw/OepCacTV5olv+cbArxVRiwOkDiI0KwUl1w8CWLdCoka2O3b8/OC6t7N1rW1AfPQpz50LXroE9fkoKnHSSvdy0ciWcdlpgj+9vBSXjctZ4XIWyzEy4+26YNMm2snSPkLVtm03EERG2D/GYMfbDq3I0TmxM48TG7E3Zy4OzHmRv6l7u/PZOr+uu2LWCl3u/HOAIFUDDhnYo1pUr7VCQrVrZUundd/t3yMnieOopm4j79g18Igbb+PK66+xlp7FjcyZuCTeajFVIOHIErr4avvsOIiPtr3V36XjSJFuSqF8foqOdjTPY1axUk7GXjGXZzmX5HsvMzmTcsnG8tug1rm19rTb2csjTT9vGSo0a2UFrpk2DGTNsqTnQNm+Gd96x1dLPPBP447vdfDOsWhXe41RrNbUKCT17wqxZdvjK6dPt9SPlfw98/wAvLniRtrXbsviWxURH6q8bJzVoYGt+wJkZjDIy7NSh//wDL7wQ+OOHIx2BS4W0uXPt//nzNRGXpeHdh9M4sTErdq/g5YVaVe009+AfTl07jo62fXw1EZc9TcYq6N10kx1wAGz3ClV2KsVU4p2L7NwvD//wMNEjo4keGU3CswkM+mIQszfOJtuE0Dx/ISgtDd57D15+OWdM9SpVAh/HkSOBP2ZRfv/dDmsbKrNbFYcmYxX03NVzl18e2GkNy6tezXsxNGkoYK8jZ2ZnciT9CJP+mMQF719Ak1eb8Njsx1i7f63DkYavG26wXfWqV7ejXe3bZ5N0oCxdCvXq2evXweT1120DzXffdToS/9NrxioopaXZhloxMbYRSYUKubszqbKXmZ2J+/th8+HNTFoxifdWvMfmwznzK3dr1I0pV0+hRlwNp8IMS3Xr2n7yAwfaBopDhsBzz0FiYmCO36sXfP89PPggPP98YI7pi/nz4eyz7XfB1q2h2WBT+xmroLZ8uR0Q/rff7Fym27dDUpL9Ajr/fKejU27ZJpufN//Meyve49NVn5KakcqnV37KVadd5XRoYaVLF1iwwN4+6aTcfejL2pw5cN55tmp8wwZbOg8Wxth+xqtXB34kMH/RBlwqaH3wgU28o0bZmWvcfYYPH7bzDavgESERdG/cnQn9JnD5qZcDkJYZwPrTcqJJk5zbgRyj2hg7sQrYSRqCKRGD7WJ1yy329rhxzsbib5qMleO++w6ysqBtW1sl9vffcOyY7U7RsaPT0amCxMfEA7D+wHqHIwk/nqPHJSTYmqNANFr66itYuBBq1oR77y3745XEwIH28tXMmbaqOlxoMlYBl/fKyD332H7EP/1kr1G1bBma14LKG3fJePzy8WRmZzocTXjxLBnPmgXt29vW1WUpKwsedc2y+cgjEB9ftscrqRo1bPV0drbtAx0uNBmrgLrzTlsFnZBgu23MmmVLv999B1WrOh2dKo7zm5xP82rN2XZkG/O2zHM6nLDiWTI+80z73z34h1u2n3uYHThgE12DBnayimA2dCjcdRdccYXTkfiPDoepAiI93Q5g8Oab9v6RI/DNN7YEcOGFjoamSihCImhTuw3rDqzjn/3/0KJaC78fI6FCApVjwmAS22I6/3y48kqYMsUm44ULcyfjv/+Gyy6D116ztUr+ULMmzJ5tJ2KpUME/+ywrXbs6M052WdJkrMrExo22tePVV9tf2lFR8OyzOY+fcYYdyCMUW0OqHBWjKgJw64xby2T/sZGx3NDuBh7o8gDNqzUvk2MEo4gImxCrVoXOne0ydzLOzrZjNa9ZA+PH2wZerVr557giUKeOf/alike7Nim/SkmB99+31Uhgu0ccds15//zz9jrUZZfZfpQq9E1fM517Z95Lela63/dtjGHn0Z0ACMIVra7g4bMfJumkfL1CwlpWFsTG2v/HjtmZi+66y5Zk9+6166SmQsWKJdt/aqr9vN5/P7Ru7b+4A+G99+zzMXVq6IxDoP2MVZk77zzbNzLd43v5kkvsxA5KlcTqvat5Yf4LTPpjEhnZGQCcFH8SERJBw4SGPNTlIS5teSkSyNnuHdCwoW05PHu2/UylpMAXX9gGj+vWlW6e3+efh4cfhk6dbD//UHoq+/Wz3y/PPWdHLAsF2s9Y+cWWLTB4sG3g0alTTtXZ3Ll2sID0dGjXDj7+GHbsgC+/dDJaFepOrXkq7/Z7l433bOTBLg8SHxPPjuQdbDuyjflb53PZJ5fRYUwHvvz7S5wqWASCu69x3742EV91la1hauG6TF/Sbk+HD+dcPho5MrQSMdjqerB9jkP95ddrxspnH38MAwbkXvbmm3ae09RU2/+vSRMYPtyZ+FT4qlelHs9f+DzDuw9nX+o+sk0209dM55l5z7B813Iu++Qy2tVpx03tbqLfKf1omNDQ6ZD9aswYO0LdM89AtWp2jGaA5q7L6GtLOEz4Cy/AwYPQrZv/GoIFUp8+doSytWvtgEHdujkdUclpyVj55Isvcifia66x14bdU7z16WPvayJWZalidEUaJDSgUWIj7up8Fxvu2cBrvV+jbuW6LN+1nLtn3k2jVxrRYXQHhs8ZzvJdy8OixHz66XDOObatxSuv5FwfLU3JeMeOnL7L//tf6JWKwTYMvekme3vsWGdjKS29Zqx88sIL9vpU5cq2P6IOyqGCybHMY0z5awrT/p7GzHUzSclIOfFYo4RG9GvZj4fOfoh6Veo5GGXpJSfbz6A7cX77ra26Pv98+PFH3/fz55/22vPmzXDxxXbkrVC1aRM0bWpH5dqxw9YcBDNtwKV8sm+fvS68ZYsdjL1jR1t9tXq1bdwxcKCdTUmpYHUs8xg/bviRL9d8yfQ109mdshuA1rVas+zWZURGhM8beO1aOPlk28Br8+ai13f75x/bvbBlS9uuI9S7M/XsaQcQeu0129I8mGkyVkV64gkYMSL3sgED7LUqpUJRtsnmt22/MWDqADYf3sztSbcz6sJRVIqp5HRofpGRYcdoPvlkm1gLk51tS9TuUvWyZXDqqcE/wIcvZs+2M0xdc42tOQhmmoxVoXbutFU9x47ZN3O3bnZIvj597LCVSoWyT1d9Sv8p/QGoF1+PsZeMpU+LPg5HFTjJybZWq0cPOyStck5ByVhbUysAXnzRJuKOHWHx4tBszKFUQa5qdRULOi/g878/Z8vhLfT9sC83truRl3q9RGKFRKfDK1Pr19v+uKtWwfz5tmtisE4C4Q/Z2XYEs1ATgiGrsnDTTbalZij2NVSqKCLCy71fZsPdG3i+x/PERsYyYfkEksYkkXI8pegdBLFZs+Df/4ZPP83/2A8/2PEAVq2yVdK//hq+iTgry84A16SJrQkINZqMy6n0dDtgx9Gj9n6rVrYxSJ/yU3OnyqHIiEgePPtBlt+2nBbVWrD+4Hq+XBPaI9OsXm3HqJ49O2eZMfDqq9C7t+1HfMkldrKJFv6fyyNoREbC77/bxqeffOJ0NMWnyTjEGWOT6JQp8NJLtsFVnTpw7bW5p1jbscOu85//wFln2TGjGzSAZs3s7DAAlcKjTYtSRTqlxince+a9AEz+Y7KzwZSSt77GTz0F995rS4uPPALTptnPfLi75Rb7PxT7HPvUgEtEegOvApHAOGPMs3kevw542HX3KHC7MWZFYfvUBlwlZ4y9DtSkiZ35KCHBtqrMa8IEuOEGe/uKK+wsSp5q17bXiW+91Y7tqlR5si91H3VfrIsxhu3/2U7tyiEy00Ae3ro3bdgA3bvDqFHQv7+j4QVUaqodkevwYVixAtq0cTqi/Eo8NrWIRAJvAn2AVsAAEck7YddGoJsxpg0wEhhT+pCVp+xs27Dq3/+2SbhvX5uAK1a0rZ0vucT2r7vxxpxt3OPZgk26vXrZEbNmzrRVV7t2waFDmohV+VQjrgZ9mvchy2TxyaoQrNd0adzYVtFu2WI/52B7RqxdW74SMUBcHFx/vb09bpyzsRRXkSVjETkLeNIY08t1/78AxphnCli/KrDSGFPoUDdaMi6e4cNzhp50u+UWO2atNykp9o2pjbGUKtgnKz/hmqnXcEa9M/jt5t+cDqfEEhLgyBH7Yz3UkpC/LV8O7dtDYqK9PFfSqSXLSmlmbaoHbPW4v821rCD/Br4tIIghIrJERJbsdU/EqYr0wQe5E3FcnL0eVNg40JUqaSJWqiiXtLyE+Jh4Fm1fxD/7/3E6nBLr3dv+Nyb0Zy8qrXbtICnJ1vp9953T0fjOl2Ts7Svd68stIudhk/HD3h43xowxxiQZY5Jq1qzpe5TlzNGj8McfOfcvvtj+r13bXhNKSbEDvNet60x8SoWLuOg4rmh1BQD9p/Rn48GNDkdUMmPG2MtY776rP8LBjpuwaJHtXx0qfEnG24AGHvfrAzvyriQibYBxQD9jzH7/hFe+rF5tS7z16kHbtjB0qF2ekGBHyNq1yzbSUEr5zxPdnqBZ1WYs37WcpLFJfLcuhIpTLgkJtjSorK5dbf/qUPph4ksyXgy0EJEmIhIDXANM91xBRBoCnwMDjTGhW9fjkB074IILbF/fV1+1134gd+f8UB/IXalg1TixMUuGLOGiFhdxIO0AfT7ow9M/P022yS56YxX0Dh1yOgLfFJmMjTGZwJ3Ad8Bq4FNjzCoRuU1EbnOt9jhQHXhLRJaLiLbMKkByMvz0E7zxRk6/wGefzd1hf9QoWLLETiSulCp7iRUSmT5gOsO724YYw34axuWfXM7hY4cdjkyV1MGDtntXy5a2C2iw04kiAuDQIXj0UViwwLb0M8aWej//3A7cPnOmHbDj3HNt96TERIcDVqoc+2btN1z3+XUcOnaIuOg44qLjAKgQVYHRF4+mb4u+DkeofGEMtG5thwKdMsWOtRAMStOaWvkoPd2Wdv/xqKifNQuqVoW33rJTlhlj+wUmJ0Pnznad3r3h++/hscc0ESvltL4t+rLkliV0qNuB1IxU9qXuY1/qPrYd2cb0NdOL3oEKCiKhNSKXJuMSWrMGnn7attZr3962dK5QwQ5Nd//9OevFxOTcfuAB2xJ648ac0rFSKvg0q9aMJbcsYc8De9jzwB6e7/E8AJnZmQ5Hporj+uvtd/D33+eMThasNBn7aOtWO/0YwP798MorMGwYTJ9uq5737LGj4DRqBJ69ts46y167yMqy14Lj4pyIXilVXCJCzUo1qVmpJqfWPBWASX9M4rNVnzkcmfJV9eq2etoYO5lGMNNk7IOZM+2ECu4qj+rVbTeCCy6Ajz+2/fu2b7fV1Js25X7RY2Js1XMozq+plLL6tujL3WfczfGs4/Sf0p/RS0Y7HZLykft7e/x4WygKVtqAC/urKTvblmzBDrI+fz4cOABpafB//2eXd+sGc+bkbBNKfdiUUqVjjOGZec/w6OxHEYSlQ5bSvm57p8NSRTDGTqSxbZudRrJtW2fj0QZcHg4etN2JevSw13qjonJf4J81CwYOtBNVuxNxtWq2S5KbJmKlyhcR4ZFzH+HezvdiMNz17V04VZhRvhOx8xvv3Ol8Ii5MlNMBOGHkSDucpKeUlJzbrVvb2U5q1IDYWIiOtrMkaQJWSj3Z/Uk+XPkhv279lQ///JDr2lzndEiqCB06OB1B0cpdNfVrr9kSL9ipCN94Ay680CZcpZTyxYRlE7hp+k3UqlSL1XesplrFak6HpHxw/Lht13Pyyc7FUFA1ddiUjP/8014POHwYdu+2f3v22L969eCrr+x6zZrZ/82bw7x5tppaKaWKY3C7wbyz9B0WbV9E9eer8+1139K7eW+nw1KFWLPGDqxUpYodCyLYGtWGTTL++GP43/+8P7bfY9qKjh3tMJP33mv7BSulVHFFSATvXPQOZ48/m7TMNK6dei1LhyylSdUmToemCtC8ub3suH69bYh7/vlOR5Rb2FRTf/ih7YJUvbot7br/ataEWrVs/1+llPKnbJPNZR9fxlf/fEWb2m2YevVUmldr7nRYqgBPPAEjRsCAATZnOKGgauqwScZKKeWEg2kH6TS2E+sPrqdCVAVGnjeSe8+8l6iIsKl4DBubN9u2QtHRdra86tUDH4N2bVJKqTJQtWJVFt68kIFtBnIs8xgPznqQM8aewdCvhzJ301ynw1MeGjWCnj1tQ67Jk52OJjdNxkopVUo14mrw/r/e55trv6FBlQYs27WMt5e8Tff3unPZx5fxz36d5j1YeE4eEUzdxLWaWiml/Cg5PZkv13zJ6r2refW3V0nJSCEqIopbO95K+zreR+yKjIikb4u+1KpUK8DRlj/Hj0ODBnYgp7lzbZuiQNJrxkopFWA7k3fy+E+PM375eLJNdqHrXtnqSj67SiehCIQtW2xCdmIgJ03GSinlkD93/8nY38eScjwl32MpGSl8suoTmiQ2YcM9GxyITgVS2A/6oZRSwap17da81uc1r49lZGUwdfVUNh3aRFpGGhWjKwY4uvJr0ybYuxc6dXI6Em3ApZRSjoqOjCaxQiIGQ/LxZKfDKTdmzYKmTeH2252OxNJkrJRSDouOsIPjZ2RlOBxJ+XHOOZCQAEuXwrJlTkejyVgppRznHiAkI1uTcaBUrGinygUYN87ZWECTsVJKOS46UkvGTrj5Zvv/gw8gNdXZWDQZK6WUw9zV1JnZmQ5HUr60aQNnnGFn+5syxdlYNBkrpZTDTpSMtZo64DxH5HKSJmOllHLYiWvGWk0dcP372wkj6tWzo3M5RfsZK6WUw060ptaSccDFx8O2bc7Pb68lY6WUclhcdBwALy14iYNpBx2OpvxxOhGDJmOllHLcQ2c/RFx0HJ/99Rmt327NDxt+cDqkciczE2bMgHnznDm+JmOllHJY3xZ9WX7rcjrX68z25O1cOOlCJi6f6HRY5cq778Ill8CIEc4cX5OxUkoFgRbVWzDvpnk83vVxAG7/+nb+2P2Hw1GVH1dfDbGxdpjMjRsDf3xNxkopFSSiIqIYft5wbmp3E8cyj3HuhHNp+05bvvz7S6dDC3tVq8KVV9rb48cH/viajJVSKsi83vd1OtfrzJH0I/yx+w+unnI1szfOdjqssOfuczx+vL2GHEg6n7FSSgWhbJPNhoMbeO2313h90evEx8RzactLEZET68RExDCk4xA61+/sYKThwxho2RLWroXp0+01ZH8raD5jTcZKKRXEsk02A6YO4NNVn3p9PCYyhrGXjGVQ20EBjiw8Pf88PPwwXHopfFkGVwcKSsY66IdSSgWxCIngg8s/YGCbgfn6IM/bMo8xv49h8LTBLNy2kNNqnuZQlL4TEeKi46gSW4X4mHjiY+OJj4m39123IyMiHYtv8GBbTX322YE9rk8lYxHpDbwKRALjjDHP5nlcXI/3BVKBG4wxvxe2Ty0ZK6VU6b2z5B3u/OZOskyW06H4TcWoirmSs+ftfPdj43Ml9rxJPjYqttjHNwY8rgb4VYlLxiISCbwJXAhsAxaLyHRjzF8eq/UBWrj+OgNvu/4rpZQqQ7cl3Uab2m34eOXHZGUHf0LONtmkZKSQfDyZ5PRkjqQfOXHb/T8tM420zDR2p+wu9fFiImMKLIEXVjp3J/ZmVZuRUCHBD2deOF+qqc8A1hljNgCIyMdAP8AzGfcD3je2mL1QRBJFpK4xZqffI1ZKKZVLlwZd6NKgi9Nh+EW2ySY1I9Vroj6SfiRX0j7xeCGPHc86zv60/exP21+ieCZd8hnXd7jSz2eZny/JuB6w1eP+NvKXer2tUw/IlYxFZAgwxHX3qIjsB/YVJ+AgVQM9j2ATLuei5xFcwuU8IHzOpUzPY+CTVzHQv7ts5G2hL8nYW8153gvNvqyDMWYMMObERiJLvNWdhxo9j+ATLuei5xFcwuU8IHzOJVzOw5dBP7YBDTzu1wd2lGAdpZRSSnnhSzJeDLQQkSYiEgNcA0zPs850YJBYZwKH9XqxUkop5Zsiq6mNMZkicifwHbZr03hjzCoRuc31+DvAN9huTeuwXZtu9PH4Y4peJSToeQSfcDkXPY/gEi7nAeFzLmFxHo6NwKWUUkopSyeKUEoppRymyVgppZRyWJkkYxHpLSJrRGSdiPyfl8dFRF5zPf6HiHTweOw+EVklIitF5CMRqVAWMfrCh/M4RUQWiEi6iDxQnG0DraTnIiINROQnEVntel3uCWzk+eIs8WviejxSRJaJyIzAROxdKd9biSIyRUT+dr0uZwUu8vxKeS6h9Hm/zvV99YeIzBeRtr5uG0glPY8Q/KwX+Hq4Hg+Kz7rPjDF+/cM28loPNAVigBVAqzzr9AW+xfZPPhP4zbW8HrARqOi6/yl2nGu/x+mn86gFdAKeBh4ozrYhdC51gQ6u2/HAP06dS2nOw+Px/wAfAjNC8fVwPfYecLPrdgyQGIrnEoKf9y5AVdftPh7fW0HzeS/leYTaZ93reXg87vhnvTh/ZVEyPjF8pjHmOOAePtPTieEzjTELgUQRqet6LAqoKCJRQBzO9Vcu8jyMMXuMMYuBjOJuG2AlPhdjzE7jmvTDGJMMrMZ+iTqhNK8JIlIfuAgYF4hgC1Hi8xCRKkBX4F3XeseNMYcCErV3pXpNCK3P+3xjjHvapIXY8RR82jaASnweIfhZL+j1CKbPus/KIhkXNDRmkesYY7YDLwBbsENpHjbGfF8GMfrCl/Moi23Lgl/iEZHGQHvgN/+EVWylPY9XgIeAbD/GVBKlOY+mwF5ggqsKbpyIVPJ3gMVQ4nMJ8c/7v7G1eyXZtiyV5jxOCMHPet7zeIXg+Kz7rCyScYmHzxSRqthfP02Ak4BKInK9n+PzlU9DfJbBtmWh1PGISGVgKnCvMeaIX6IqvhKfh4hcDOwxxiz1b0glUprXIwroALxtjGkPpABOXqMszWsSkp93ETkP++X/cHG3DYDSnId7eUh91vOeR5B91n1WFsm4NMNn9gA2GmP2GmMygM+x1wWcUJohPoNteNBSxSMi0dgP5wfGmM/9HFtxlOY8zgYuFZFN2Cqv80Vksn/D81lp31vbjDHuEssUbHJ2SmnOJeQ+7yLSBlv12c8Ys7842wZIac4j5D7rBZxHMH3WfVYWybg0w2duAc4UkTgREeAC7HULJ/hyHmWxbVkocTyu1+FdYLUx5qUyjNEXJT4PY8x/jTH1jTGNXdvNNsY4VQorzXnsAraKSEvXogvIPZ1poJXmvR5Sn3cRaYj9wTDQGPNPcbYNoBKfR6h91gs6jyD7rPuuLFqFYVtL/4NtDfeoa9ltwG2u2wK86Xr8TyDJY9vhwN/ASmASEFsWMfrpPOpgf8EdAQ65blcpaFsn/0p6LsA52OqhP4Dlrr++oXYeefbRHYdbWJbyvdUOWOJ6TabhalEaoucSSp/3ccBBj8/BksK2DbXzCMHPeoGvh8c+HP+s+/qnw2EqpZRSDtMRuJRSSimHaTJWSimlHKbJWCmllHKYJmOllFLKYZqMlVJKKYdpMlaqDIiI8eFvk2vdiSKyzeGQARCRTf4cIMG1v4k+rDfR/XwoVR5FOR2AUmEq77SGX2BnnnnSY1l6wKJRSgU1TcZKlQFjZyM7QUTSgX15l5eWiMQaYzSpKxXitJpaqSAhIu1F5BcRSRWRtSJyW57Hb3BVb3cVkc9E5BCuWXVEJEpE/isif4tIuojsEJEXRaSCx/ZRIjJSRNaLyDER2Sci80TkHC+xXCN2kvkUEVlSwDrXi8gKj31NkpypUAs7zwtE5HfXdutF5NaSPF9KhRMtGSsVHKpgJ0J/BRgB3Ai8LSJrjDE/5Vn3A+Aj4EpyPsOTgUuA54D5wKnASKAxcIVrnYeB+4BHscMHVgGSgGp59n8u0BJ4DDjm2s8MEWlsXHMni8gQYDTwCfBf7KxL/wM6i0gHY8xRbycpIqcC32CH87wGiMVW3VcGsgp/ipQKX5qMlQoO8cBQd+IVkZ+BnsAAIG8ynmKMech9R0TOBfoDg40x77sW/yAiB4DJItLOGLMcex37e2PMqx77+spLLFWAdsY1cbuI7MIO3N8X+FBEIrEJeo4x5hqPOP4GfgFuAl4r4DyHAclAT2NMimu7+djxh52c2UwpR2k1tVLBIdWzBOy6DrwWaOhl3S/y3O8NHAemuqqio0QkCvje9XhX1//FQF8ReVpEznHNhuPNAncidvnT9d8dS0ugFraEfoIxZh6wGehW0ElifxB8407Eru22Ar8Wso1SYU+TsVLB4aCXZelABS/Ld+a5XwuIAY4CGR5/e1yPV3f9/x/wBHAptgS7X0QmiEiNPPs74HnHo4GYOxZ3tXbeOAB2kb/a21NdYLeX5d6WKVVuaDW1UqEn71Rr+7HXds8tYP0dAMaYDOw15edEpA5wMfASEIet5vaVO1nX8fJYHez14ILsBGp7We5tmVLlhpaMlQp9M7Gl1gRjzBIvf/muxRpjdhljxgE/AKcX83hrsCXZazwXikgXoBEwt5BtF2Cryit5bNcAOLuYMSgVVrRkrFSIM8bMEZGPgCki8hKwCMjGtqTuCzxsjPlHRL7EDjzyO7ZavD32evPoYh4vS0QeB0a7RuuaDNQDnsZe555QyOZPAVcB34vIKGz1+nC0mlqVc5qMlQoP1wN3YVsyP4q93rwJ+I6cRPczNhHega2a3gI8j02ixWKMGSMiqcCDwJfY69XfAA8V1K3Jtd1qEekLjMJ2i9qOrTo/C+he3DiUChdiTN7LT0oppZQKJL1mrJRSSjlMk7FSSinlME3GSimllMM0GSullFIO02SslFJKOUyTsVJKKeUwTcZKKaWUwzQZK6WUUg77f0d8hLvkPOjfAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot precision and recall as function of threshold\n",
    "plt.figure(figsize=(8, 4))\n",
    "plot_precision_recall_vs_threshold(precision_rf, recall_rf, threshold_rf)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Threshold</th>\n",
       "      <th>Profit</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>-4150.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.01</td>\n",
       "      <td>-4150.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.02</td>\n",
       "      <td>-4150.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.03</td>\n",
       "      <td>-4150.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.04</td>\n",
       "      <td>-4150.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>0.95</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>0.96</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>0.97</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>0.98</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>0.99</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Threshold  Profit\n",
       "0        0.0 -4150.0\n",
       "1       0.01 -4150.0\n",
       "2       0.02 -4150.0\n",
       "3       0.03 -4150.0\n",
       "4       0.04 -4150.0\n",
       "..       ...     ...\n",
       "95      0.95     0.0\n",
       "96      0.96     0.0\n",
       "97      0.97     0.0\n",
       "98      0.98     0.0\n",
       "99      0.99     0.0\n",
       "\n",
       "[100 rows x 2 columns]"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "thresholds = np.arange(0.0, 1.0, 0.01)\n",
    "\n",
    "# Create an empty dataframe to store results\n",
    "profits_df_rf = pd.DataFrame(columns=['Threshold', 'Profit'])\n",
    "\n",
    "# Calculate profit for each threshold\n",
    "for t in thresholds:\n",
    "    y_pred_pos_rf = random_forest.predict_proba(X_test)[:, 1]\n",
    "    y_pred_rf = (y_pred_pos_rf > t).astype(int)\n",
    "    tn, fp, fn, tp = confusion_matrix(y_test, y_pred_rf).ravel()\n",
    "    profit = (tp * 50) - ((tp + fp) * 10)\n",
    "    profits_df_rf = profits_df_rf.append({'Threshold': t, 'Profit': profit}, ignore_index=True)\n",
    "\n",
    "profits_df_rf"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How the profits evolve with different decision threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAfO0lEQVR4nO3de3xV1Z338c/PAAk3QSAIJCCxIsqtXFJ0Oo8iooDVUWy9YKVeW6ZW62inT6u1T6edvpzH1lYtbbWP9W6t6NCx8GjttCqt2qKISrnJJa0okQABDYRLIAm/+WPvpCdwIEDOOXvvk+/79Tqvc87aZ++9VoB8WWvtvY65OyIiIk2OiroCIiISLwoGERFpQcEgIiItKBhERKQFBYOIiLTQIeoKtFWfPn188ODBUVdDRCRR3nzzzc3uXpxuW+KDYfDgwSxatCjqaoiIJIqZvXegbRpKEhGRFhQMIiLSgoJBRERaSPwcg4jkl/r6eiorK6mrq4u6KnmhqKiI0tJSOnbseMj7KBhEJFYqKyvp3r07gwcPxsyirk6iuTtbtmyhsrKSsrKyQ95PQ0kiEit1dXX07t1boZABZkbv3r0Pu/elYBCR2FEoZM6R/Cw1lJQE26th8S+g27HQYyD0GQLd+0VdKxHJUwqGJFjwY/jTj/7+3o6C838CYy6Prk4ieWrLli1MmjQJgA0bNlBQUEBxcTFr165lwIABrFixIqPn+/a3v023bt346le/esj7dOvWje3bt+9XftVVV3Heeedx0UUXtalOGkqKO3d451komwBffguumAtlp8PcL8Gih6OunUje6d27N4sXL2bx4sV88Ytf5Oabb25+f9RRrf/KbGhoyEEtsysjwWBmD5nZJjNbllLWy8x+b2ZrwudjUrbdamYVZrbKzKaklI8zs6XhtlmmgUaoXgkf/hWGnQ+9PwbHnwGXPQVDJsOzN8ErP4Q3HoA518C9n4QNS6OusUjeamxs5Atf+ALDhw9n8uTJ7Nq1C4AzzjiDb3zjG0yYMIEf/ehHvPnmm0yYMIFx48YxZcoUqqqqAJg1axbDhg1j1KhRTJ8+vfm4K1as4IwzzuD4449n1qxZzeV33XUXI0aMYMSIEdxzzz371cfdueGGGxg2bBjnnnsumzZtykg7MzWU9AjwE+CxlLJbgBfd/Q4zuyV8/3UzGwZMB4YDA4AXzOxEd28E7gNmAq8BvwGmAs9nqI7JtPLZ4HnouX8v61gEl/4C/vNqePHfg7Lu/aFuG7x8J1zy2P7HEUmg7/z/5axYvy2jxxw24Gj+7Z+GH9G+a9as4cknn+TnP/85l1xyCb/61a+YMWMGADU1Nfzxj3+kvr6eCRMmMHfuXIqLi3nqqae47bbbeOihh7jjjjt49913KSwspKampvm4K1euZP78+dTW1jJ06FCuu+46lixZwsMPP8zrr7+Ou3PKKacwYcIExowZ07zfM888w6pVq1i6dCkbN25k2LBhXHPNNW36+UCGgsHdXzazwfsUXwCcEb5+FPgD8PWwfLa77wbeNbMKYLyZrQWOdvcFAGb2GDCN9h4M7zwLpZ+Ao/u3LO9QCJc8CmtfhWMGB48X/x3+dA98+DfodXwElRXJb2VlZYwePRqAcePGsXbt2uZtl156KQCrVq1i2bJlnH322UDQy+jfP/j3O2rUKC6//HKmTZvGtGnTmvc999xzKSwspLCwkL59+7Jx40ZeffVVLrzwQrp27QrApz/9aV555ZUWwfDyyy9z2WWXUVBQwIABAzjzzDMz0s5sTj4f6+5VAO5eZWZ9w/ISgh5Bk8qwrD58vW/5fsxsJkHPgkGDBmW42jFSsw6qFsNZ30m/vaAjfGzi39+f8s+w4Cew4Kdw7g9zUkWRbDrS/9lnS2FhYfPrgoKC5qEkoPkXuLszfPhwFixYsN/+zz33HC+//DLz5s3ju9/9LsuXL0973IaGBtz9kOqUjRH3KCaf07XCD1K+f6H7/e5e7u7lxcVplxPPDyufC55P/qdD+3z3fjDqUnj7CdixJXv1EpEDGjp0KNXV1c3BUF9fz/Lly9m7dy/r1q1j4sSJfP/736empibtlUVNTj/9dH7961+zc+dOduzYwTPPPMNpp52232dmz55NY2MjVVVVzJ8/PyNtyGaPYaOZ9Q97C/2BplmRSmBgyudKgfVheWma8vZr5bNQfHIw6XyoPvllePtxeOPncMYt2aubiKTVqVMn5syZw4033sjWrVtpaGjgpptu4sQTT2TGjBls3boVd+fmm2+mZ8+eBzzO2LFjueqqqxg/fjwAn//851sMIwFceOGFvPTSS4wcOZITTzyRCRMmZKQNdqjdlVYPFMwxPOvuI8L3dwJbUiafe7n718xsOPBLYDzB5POLwBB3bzSzN4AvA68TTD7/2N1/c7DzlpeXe15+Uc+OLfCDE+C0f4Uzv3l4+/7yUqh8A25aBp26ZKd+IlnyzjvvcPLJJ0ddjbyS7mdqZm+6e3m6z2fqctUngQXAUDOrNLNrgTuAs81sDXB2+B53Xw48DawAfgtcH16RBHAd8ABQAfyV9jzxvPq34HvhpPMOf99P3gg7t8BDk2HFXNi7N/P1E5G8lamrki47wKZJB/j87cDtacoXASMyUafEW/cadO4F/T9++PsO/kf4zIMw/z/g6Sug+KTg8tY+QzJfTxHJO7rzOa6qlkD/UXCkVxyMvAhueCMIiI/WwhsPZrR6IpK/FAxx1FgPm1ZAv5FtO85RBUFADBgLlQszUzcRyXsKhjiqXgWNe6DfEQwjpTPwE0EPpF7fiCUirVMwxNGGJcFz/1GZOV7peNhbH9wsJyLSCgVDHFUtgY5doPcJmTnewOA6aCrfyMzxRPJcQUEBo0ePZsSIEVx88cXs3LnzsPa/7LLLGDVqFHfffTff+ta3eOGFFwC45557DvtYUdD3McTRhiVw7PBgjiATuvWFnsfBOs0ziByKzp07s3jxYgAuv/xyfvazn/GVr3yleXtjYyMFBen/fW7YsIE///nPvPfee/ttu+eee5gxYwZdusT7/iL1GOJm795g6ex+GRpGajJwfNBjyNANjSLtxWmnnUZFRQV/+MMfmDhxIp/97GcZOXIkdXV1XH311YwcOZIxY8Y0L0cxefJkNm3axOjRo3nllVe46qqrmDNnDrNmzWL9+vVMnDiRiRMntnLWaKnHEDc1a2H3tszNLzQp/QQs/U/YWgk9B7b+eZE4eP6WzH/HSL+RcM4dh/TRhoYGnn/+eaZOnQrAwoULWbZsGWVlZfzwh8FClUuXLmXlypVMnjyZ1atXM2/ePM4777zmHseDDwaXit94443cddddzJ8/nz59+mS2TRmmHkPcVIUTz229VHVfpZ8InnXZqkirdu3axejRoykvL2fQoEFce+21AIwfP56ysjIAXn31VT73uc8BcNJJJ3HcccexevXqyOqcSeoxxM2GpWAF0DfDyw33GwkdOsO6N2DEZzJ7bJFsOcT/2Wda6hxDqqaltYFDXhY7idRjiJsNS6B4aPAtbZlU0BEGjFGPQSRDTj/9dJ544gkAVq9ezfvvv8/QoUMPuk/37t2pra3NRfXaRMEQN1VLMj/x3EQ3uolkzJe+9CUaGxsZOXIkl156KY888kiLL9xJZ+bMmZxzzjmxn3zO2LLbUcmrZbe3b4IfDIEp/wH/cH3mj//Os/DU5XDN72DQKZk/vkgGaNntzItk2W3JkOaJ5yz1GDQBLSKHQMEQJxv+Ejxn+oqkJt2PhR6DoDJPelgikhUKhjjZ9A4cXQqde2bvHCVjYf1b2Tu+SAYkfYg7To7kZ6lgiJPNa6D4xOyeo2Qc1LwP26uzex6RI1RUVMSWLVsUDhng7mzZsoWiosO7ylH3McSFO2ypgEEzsnueknHB8/q34MQp2T2XyBEoLS2lsrKS6mr95yUTioqKKC0tPax9FAxxUVsFe7ZnbkXVA+n/cbCj4IM3FQwSSx07dmy+u1iioaGkuNgc3krfJ8tDSYXdoPjkIBhERNJQMMTF5jXBc7aDAYIJ6A/e1EqrIpKWgiEuNq+BTt2ge7/sn6tkHOz6CD56N/vnEpHEUTDExebV0GcImGX/XE0T0B/oslUR2Z+CIS42r8nNMBJA35ODlVYVDCKShoIhDvbsgG2V0HtIbs5X0DG4OkkT0CKShoIhDrZUBM99chQMEAwnVf0FGutzd04RSQQFQxzk8oqkJiVjoWFXsAyHiEgKBUMcbF4DGPQ6PnfnLBkbPGs4SUT2EbtgMLOpZrbKzCrM7Jao65MTm1fDMcdl/lvbDuaYMujaF167Fz56L3fnFZHYi1UwmFkB8FPgHGAYcJmZDYu2VjmwJYdXJDUxg888ALUb4YFJf1+Ku2EPrFsI1fnxpeYicvjitlbSeKDC3f8GYGazgQuAFZk+0abaOjZszd1XXB7wJmPfy8jqNWzpPZ7162qyX4/UNx0/TuF5v+L4311Nx4c/xY6+Y+i6aTFHNQY/l5rjprBx9JfZ1Wck7K2n044qCuqOvI6O7rSO7mbzePzso7zZPh4/ATJakb4DP0Zxv4GZO2AobsFQAqxLeV8JZOU7KJ956wP+7/Mrs3How1JCNX8qquOuxfDkm3+KpA69uI07O/4/jv1gAwv3nsHCvSdx8lHvc/Xa3zL0vf9mo/ekD1spsNj80xIR4PVh36T4kv+d8ePGLRjS3fa7328jM5sJzAQYNGjQEZ1o6oh+nNC32xHte6TS3dTcq+pV+CNcPOVMzu6b9utXM1+PtD/mSVQDZeEDYPGeb1Ly11/SZetfeb9rCXVdB1Bf2Iv0f0yHfPJYiLQaubi7Pd1pIznr/qKtRzx+Cpn6K1D2sdGZOdA+4hYMlUBqv6gUWL/vh9z9fuB+gPLy8iP6b+xxvbtyXO+uR7JrZtUEa86PHXsKdCuOuDL76guj/k/UlRCRHIvV5DPwBjDEzMrMrBMwHZgXcZ2ya/NqKOoBXftEXRMRESBmPQZ3bzCzG4D/BgqAh9x9ecTVyq5t66HHoMiGF0RE9hWrYABw998Av4m6HjmzY1MMh5BEpD2L21BS+7O9OrjRTEQkJhQMUXJXj0FEYkfBEKXdtdBQpx6DiMSKgiFKO4JLVemmYBCR+FAwRKkpGHSpqojEiIIhSts3Bc8aShKRGFEwRGlHGAwaShKRGFEwRGl7NWDQRUNJIhIfCoYo7dgEXXpBQezuMxSRdkzBEKXtmzS/ICKxo2CI0o5qXZEkIrGjYIjSjmpNPItI7CgYoqR1kkQkhhQMUanfBXtqtU6SiMSOgiEqurlNRGJKwRAVrZMkIjGlYIhKc49BQ0kiEi8KhqjsUDCISDwpGKLSvLKqgkFE4kXBEJXt1VDYAzoWRV0TEZEWFAxR0Vd6ikhMKRiiopvbRCSmFAxRUY9BRGJKwRCV7Zs08SwisaRgiELDHqir0VCSiMSSgiEKOzcHzxpKEpEYUjBEQeskiUiMKRiioHWSRCTGFAxR0DpJIhJjbQoGM7vYzJab2V4zK99n261mVmFmq8xsSkr5ODNbGm6bZWYWlhea2VNh+etmNrgtdYu1pnWS1GMQkRhqa49hGfBp4OXUQjMbBkwHhgNTgXvNrCDcfB8wExgSPqaG5dcCH7n7CcDdwPfaWLf42l4NHbtAp65R10REZD9tCgZ3f8fdV6XZdAEw2913u/u7QAUw3sz6A0e7+wJ3d+AxYFrKPo+Gr+cAk5p6E3lnh+5hEJH4ytYcQwmwLuV9ZVhWEr7et7zFPu7eAGwFeqc7uJnNNLNFZraouro6w1XPgZ1boGufqGshIpJWh9Y+YGYvAP3SbLrN3eceaLc0ZX6Q8oPts3+h+/3A/QDl5eVpPxNrddugqEfUtRARSavVYHD3s47guJXAwJT3pcD6sLw0TXnqPpVm1gHoAXx4BOeOv9210KOk9c+JiEQgW0NJ84Dp4ZVGZQSTzAvdvQqoNbNTw/mDK4C5KftcGb6+CHgpnIfIP7u3QeHRUddCRCStVnsMB2NmFwI/BoqB58xssbtPcfflZvY0sAJoAK5398Zwt+uAR4DOwPPhA+BB4HEzqyDoKUxvS91ibXetgkFEYqtNweDuzwDPHGDb7cDtacoXASPSlNcBF7elPomwtxH2bIciBYOIxJPufM613bXBc2H3aOshInIACoZcUzCISMwpGHJt97bgWXMMIhJTCoZcU49BRGJOwZBrTcGgG9xEJKYUDLlWtzV4Vo9BRGJKwZBrGkoSkZhTMOSaJp9FJOYUDLm2uxbsKH0Xg4jEloIh13bXBsNIefpVEyKSfAqGXKvTAnoiEm8KhlzbvU0TzyISawqGXNOS2yIScwqGXGuaYxARiSkFQ67trtWS2yISawqGXKvTHIOIxJuCIdc0lCQiMadgyKXGemjYBYVaQE9E4kvBkEtaJ0lEEkDBkEtN6yRp8llEYkzBkEt1TQvoqccgIvGlYMglDSWJSAIoGHJJS26LSAIoGHKpucegYBCR+FIw5JImn0UkARQMuaTJZxFJAAVDLu2uhaM6QIeiqGsiInJACoZcalpyW9/eJiIx1qZgMLM7zWylmS0xs2fMrGfKtlvNrMLMVpnZlJTycWa2NNw2yyz4LWlmhWb2VFj+upkNbkvdYknrJIlIArS1x/B7YIS7jwJWA7cCmNkwYDowHJgK3GtmBeE+9wEzgSHhY2pYfi3wkbufANwNfK+NdYsfLbktIgnQpmBw99+5e0P49jWgNHx9ATDb3Xe7+7tABTDezPoDR7v7And34DFgWso+j4av5wCTmnoTeUPf9ywiCZDJOYZrgOfD1yXAupRtlWFZSfh63/IW+4RhsxXone5EZjbTzBaZ2aLq6uqMNSDr9H3PIpIArQaDmb1gZsvSPC5I+cxtQAPwRFNRmkP5QcoPts/+he73u3u5u5cXFxe31oT40Pc9i0gCdGjtA+5+1sG2m9mVwHnApHB4CIKewMCUj5UC68Py0jTlqftUmlkHoAfw4SG0ITk0+SwiCdDWq5KmAl8Hznf3nSmb5gHTwyuNyggmmRe6exVQa2anhvMHVwBzU/a5Mnx9EfBSStDkB00+i0gCtNpjaMVPgELg9+E88Wvu/kV3X25mTwMrCIaYrnf3xnCf64BHgM4EcxJN8xIPAo+bWQVBT2F6G+sWL/V10LhHPQYRib02BUN4aemBtt0O3J6mfBEwIk15HXBxW+oTa1pAT0QSQnc+54qW3BaRhFAw5MpuLaAnIsmgYMiVpqEkTT6LSMwpGHJFS26LSEIoGHJF3/csIgmhYMiV5mDoEW09RERaoWDIld1bg2f1GEQk5hQMubK7Nvjmtg6doq6JiMhBKRhypU4rq4pIMigYckUL6IlIQigYcmV3re56FpFEUDDkir6kR0QSQsGQK3XboEiXqopI/CkYcqWuBop6Rl0LEZFWKRhyZVcNdO4ZdS1ERFqlYMiFxnqo36GhJBFJBAVDLtSFdz1rKElEEkDBkAvNwaAeg4jEn4IhF3bVBM+aYxCRBFAw5EJdTfCsHoOIJICCIReag6FnlLUQETkkCoZc0ByDiCSIgiEXNMcgIgmiYMiFuhoo6BR8H4OISMwpGHKhbmswv2AWdU1ERFqlYMgFLYchIgmiYMiFuq2aeBaRxFAw5IJWVhWRBFEw5IJ6DCKSIG0KBjP7rpktMbPFZvY7MxuQsu1WM6sws1VmNiWlfJyZLQ23zTILZmTNrNDMngrLXzezwW2pW6xojkFEEqStPYY73X2Uu48GngW+BWBmw4DpwHBgKnCvmRWE+9wHzASGhI+pYfm1wEfufgJwN/C9NtYtHtzVYxCRRGlTMLj7tpS3XQEPX18AzHb33e7+LlABjDez/sDR7r7A3R14DJiWss+j4es5wKSm3kSi7dkO3qg5BhFJjA5tPYCZ3Q5cAWwFJobFJcBrKR+rDMvqw9f7ljftsw7A3RvMbCvQG9ic5pwzCXodDBo0qK1NyC4thyEiCdNqj8HMXjCzZWkeFwC4+23uPhB4Arihabc0h/KDlB9sn/0L3e9393J3Ly8uLm6tCdHSchgikjCt9hjc/axDPNYvgeeAfyPoCQxM2VYKrA/LS9OUk7JPpZl1AHoAHx7iueNLK6uKSMK09aqkISlvzwdWhq/nAdPDK43KCCaZF7p7FVBrZqeG8wdXAHNT9rkyfH0R8FI4D5FsGkoSkYRp6xzDHWY2FNgLvAd8EcDdl5vZ08AKoAG43t0bw32uAx4BOgPPhw+AB4HHzayCoKcwvY11iwcNJYlIwrQpGNz9MwfZdjtwe5ryRcCINOV1wMVtqU8sqccgIgmjO5+zra4GMChUMIhIMigYsq1uKxQeDUfpRy0iyaDfVtm2qwY6q7cgIsmhYMg2LYchIgmjYMg2LbktIgmjYMg2rawqIgmjYMg2DSWJSMIoGLJNQ0kikjAKhmxq2AP1OxUMIpIoCoZsarrrWXMMIpIgCoZs0nIYIpJACoZs0pLbIpJACoZsag4G9RhEJDkUDNmkJbdFJIEUDNmkOQYRSSAFQzZpjkFEEkjBkE27aqBDEXQsiromIiKHTMGQTVoOQ0QSSMGQTVoOQ0QSSMGQTeoxiEgCKRiySUtui0gCKRiyacdm6HxM1LUQETksCoZs2fkhbKuEvsOiromIyGFRMGTL+reC55Kx0dZDROQwKRiyZf3bwXP/j0dbDxGRw6RgyJYP3obeQ3RVkogkjoIhW9a/pWEkEUkkBUM2bKuC2ioYMCbqmoiIHLaMBIOZfdXM3Mz6pJTdamYVZrbKzKaklI8zs6XhtllmZmF5oZk9FZa/bmaDM1G3SDTNLwxQj0FEkqfNwWBmA4GzgfdTyoYB04HhwFTgXjMrCDffB8wEhoSPqWH5tcBH7n4CcDfwvbbWLTLr3wYrgH4jo66JiMhhy0SP4W7ga4CnlF0AzHb33e7+LlABjDez/sDR7r7A3R14DJiWss+j4es5wKSm3kTirH8L+p4MnbpEXRMRkcPWpmAws/OBD9z9L/tsKgHWpbyvDMtKwtf7lrfYx90bgK1A7wOcd6aZLTKzRdXV1W1pQua5Bz2GAaOjromIyBHp0NoHzOwFoF+aTbcB3wAmp9stTZkfpPxg++xf6H4/cD9AeXl52s9EpuZ92LlF8wsiklitBoO7n5Wu3MxGAmXAX8IRn1LgLTMbT9ATGJjy8VJgfVhemqaclH0qzawD0AP48HAaEwtNdzzriiQRSagjHkpy96Xu3tfdB7v7YIJf7GPdfQMwD5geXmlURjDJvNDdq4BaMzs1nD+4ApgbHnIecGX4+iLgpXAeIlnWvw0FneDYEVHXRETkiLTaYzgS7r7czJ4GVgANwPXu3hhuvg54BOgMPB8+AB4EHjezCoKewvRs1C2r6rbCuoVBKHToFHVtRESOSMaCIew1pL6/Hbg9zecWAfv9d9rd64CLM1WfVr31OCz4SWaO5Xth+8YgGADG/3NmjisiEoGs9BgSoUsvKB6aueOVTYCeA6HHQDhhUuaOKyKSY+03GE46N3iIiEgLWitJRERaUDCIiEgLCgYREWlBwSAiIi0oGEREpAUFg4iItKBgEBGRFhQMIiLSgiVxnbpUZlYNvHeEu/cBNmewOknRHtvdHtsM7bPd7bHNcPjtPs7di9NtSHwwtIWZLXL38qjrkWvtsd3tsc3QPtvdHtsMmW23hpJERKQFBYOIiLTQ3oPh/qgrEJH22O722GZon+1uj22GDLa7Xc8xiIjI/tp7j0FERPahYBARkRbabTCY2VQzW2VmFWZ2S9T1yQYzG2hm883sHTNbbmb/Epb3MrPfm9ma8PmYqOuaaWZWYGZvm9mz4fv20OaeZjbHzFaGf+b/kO/tNrObw7/by8zsSTMrysc2m9lDZrbJzJallB2wnWZ2a/i7bZWZTTnc87XLYDCzAuCnwDnAMOAyMxsWba2yogH4V3c/GTgVuD5s5y3Ai+4+BHgxfJ9v/gV4J+V9e2jzj4DfuvtJwMcJ2p+37TazEuBGoNzdRwAFwHTys82PAFP3KUvbzvDf+HRgeLjPveHvvEPWLoMBGA9UuPvf3H0PMBu4IOI6ZZy7V7n7W+HrWoJfFCUEbX00/NijwLRIKpglZlYKnAs8kFKc720+GjgdeBDA3fe4ew153m6CryfubGYdgC7AevKwze7+MvDhPsUHaucFwGx33+3u7wIVBL/zDll7DYYSYF3K+8qwLG+Z2WBgDPA6cKy7V0EQHkDfCKuWDfcAXwP2ppTle5uPB6qBh8MhtAfMrCt53G53/wD4AfA+UAVsdfffkcdt3seB2tnm32/tNRgsTVneXrdrZt2AXwE3ufu2qOuTTWZ2HrDJ3d+Mui451gEYC9zn7mOAHeTHEMoBhWPqFwBlwACgq5nNiLZWsdDm32/tNRgqgYEp70sJuqB5x8w6EoTCE+7+X2HxRjPrH27vD2yKqn5Z8I/A+Wa2lmCI8Ewz+wX53WYI/k5Xuvvr4fs5BEGRz+0+C3jX3avdvR74L+CT5HebUx2onW3+/dZeg+ENYIiZlZlZJ4KJmnkR1ynjzMwIxpzfcfe7UjbNA64MX18JzM113bLF3W9191J3H0zw5/qSu88gj9sM4O4bgHVmNjQsmgSsIL/b/T5wqpl1Cf+uTyKYR8vnNqc6UDvnAdPNrNDMyoAhwMLDOrK7t8sH8ClgNfBX4Lao65OlNv4vgi7kEmBx+PgU0JvgKoY14XOvqOuapfafATwbvs77NgOjgUXhn/evgWPyvd3Ad4CVwDLgcaAwH9sMPEkwj1JP0CO49mDtBG4Lf7etAs453PNpSQwREWmhvQ4liYjIASgYRESkBQWDiIi0oGAQEZEWFAwiItKCgkFERFpQMIiISAv/A5uWVAD9HEZ6AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "profits_df_rf.plot()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The company achieves a lower maximum profit of 1130 compared to 1340 €"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1130.0"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_profit = profits_df_rf[\"Profit\"].max()\n",
    "max_profit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Threshold</th>\n",
       "      <th>Profit</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0.14</td>\n",
       "      <td>1130.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Threshold  Profit\n",
       "14      0.14  1130.0"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_profit_row_rf = profits_df_rf.loc[profits_df_rf['Profit'] == max_profit]\n",
    "max_profit_row_rf"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To maximize the profit using random forest model, we would need to increase the threshold to 0.14 (before optimal_threshold was 0.123)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Random forest builds multiple decision trees and merges them together to get a more accurate and stable prediction. It can be used for both classification and regression problems. Generally, random forest models perform better than a single model (wisdom of the crowd). \n",
    "\n",
    "However, the random forest model in this case performs worse (lower achievable profit). This could be due to the the max_depth=2 parameter used in the random forest model which may limit its ability to capture complex relationships, potentially leading to reduced performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "a81a1412cade015a653d21208291244715c6a021abd2ef05b33165e8b2673496"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}