03_Dropping_Constant_Features.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <b><center>Feature Selection</center><b>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Dropping constant features**\n",
    "\n",
    "Dropping constant features, also known as `constant variance` or `zero variance` features, is a common step in feature selection during data preprocessing. It involves identifying and removing features that have the `same` value for all samples in the dataset. These constant features typically provide `no useful information` and *can potentially hinder the performance of machine learning models*.\n",
    "\n",
    "Here are a few reasons why dropping constant features is beneficial:\n",
    "\n",
    "- **No Discriminatory Power**: Features with constant values do not vary across different samples or instances in the dataset. Since these features do not change, they do not contribute any discriminatory power to distinguish between different classes or patterns in the data. `Keeping such features may introduce noise or bias into the model`.\n",
    "\n",
    "- **Redundant Information**: Constant features do not provide any additional information beyond the single value they possess. Including such features in the analysis does not add any value to the model's understanding of the data or its ability to make accurate predictions. Removing them simplifies the dataset and reduces redundancy.\n",
    "\n",
    "- **Computational Efficiency**: Dropping constant features can lead to computational efficiency gains, especially when dealing with large datasets. These features do not contribute to the learning process, yet they require computational resources for processing, memory storage, and model training. Removing them reduces the overall computational burden.\n",
    "\n",
    "- **Model Stability**: Including constant features in a machine learning model may cause instability or overfitting. Overfitting occurs when a model learns noise or irrelevant patterns in the training data, resulting in poor generalization performance on unseen data. Removing constant features reduces the risk of overfitting and promotes better model stability.\n",
    "\n",
    "While dropping constant features is generally considered a good practice, it's important to note that there may be exceptions. For example, in certain specific cases or domains, constant features might carry some unique significance or have a specific purpose. Therefore, it's always essential to carefully analyze and understand the data before making decisions about feature selection and removal."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Importance of Feature Selection in High-Dimensional Data Analysis:**\n",
    "\n",
    "High-dimensional data poses problems such as increased computational complexity, overfitting, and sparsity. As the number of features increases, the data becomes increasingly sparse, making it difficult to find meaningful patterns or relationships. This sparsity can lead to poor generalization and decreased model performance.\n",
    "\n",
    "Feature selection aims to address the curse of dimensionality by reducing the number of irrelevant or redundant features, thereby improving model efficiency and reducing overfitting. By selecting the most informative features, we can mitigate the challenges associated with high-dimensional data and focus on the most relevant information for solving the problem at hand."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Without Pre-Processing & Feature Selection**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1000, 100) (1000,)\n",
      "Accuracy without feature selection: 83.0\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import make_classification\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "# Generate a random dataset with 1000 samples and 100 features\n",
    "X, y = make_classification(n_samples=1000, n_features=100, random_state=42)\n",
    "\n",
    "# Shape of the dataset\n",
    "print(X.shape, y.shape)\n",
    "\n",
    "# Split the dataset into train and test sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)\n",
    "\n",
    "# Train a logistic regression classifier without feature selection\n",
    "lr = LogisticRegression(random_state=42)\n",
    "lr.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions on the test set\n",
    "y_pred = lr.predict(X_test)\n",
    "\n",
    "# Calculate the accuracy of the classifier\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "print(\"Accuracy without feature selection:\", accuracy*100)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**With Pre-Processing & Feature Selection**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy with feature selection: 88.0\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import make_classification\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.feature_selection import SelectKBest\n",
    "from sklearn.feature_selection import chi2\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "# Generate a random dataset with 1000 samples and 100 features\n",
    "X, y = make_classification(n_samples=1000, n_features=100, random_state=42)\n",
    "\n",
    "# Split the dataset into train and test sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)\n",
    "\n",
    "# Preprocess the data by scaling it between 0 and 1\n",
    "scaler = MinMaxScaler()\n",
    "X_train_scaled = scaler.fit_transform(X_train)\n",
    "X_test_scaled = scaler.transform(X_test)\n",
    "\n",
    "# Perform feature selection using the chi-squared test\n",
    "k = 10  # Select the top 10 features\n",
    "selector = SelectKBest(score_func=chi2, k=k)\n",
    "X_train_selected = selector.fit_transform(X_train_scaled, y_train)\n",
    "X_test_selected = selector.transform(X_test_scaled)\n",
    "\n",
    "# Train a logistic regression classifier with feature selection\n",
    "lr = LogisticRegression(random_state=42)\n",
    "lr.fit(X_train_selected, y_train)\n",
    "\n",
    "# Make predictions on the test set\n",
    "y_pred = lr.predict(X_test_selected)\n",
    "\n",
    "# Calculate the accuracy of the classifier with feature selection\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "print(\"Accuracy with feature selection:\", accuracy*100)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Variance Threshold**\n",
    "\n",
    "\n",
    "The VarianceThreshold class from the `sklearn.feature_selection` module in scikit-learn is used for feature selection based on `variance`. It's primarily used to remove features (`columns`) from a dataset that have `low variance`, assuming that such features contain less useful or redundant information.\n",
    "\n",
    "The idea behind this approach is that if a feature has very little variance, it means that it does not vary much across the samples in the dataset. In other words, the feature has almost the same value for all or most of the samples. Such features often provide little discriminatory power and may not contain much useful information for building predictive models.\n",
    "\n",
    "Here's a simple example to illustrate the usage of VarianceThreshold:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of constant columns: 2\n",
      "C\n",
      "D\n"
     ]
    }
   ],
   "source": [
    "# Import pandas to create DataFrame\n",
    "from sklearn.feature_selection import VarianceThreshold\n",
    "import pandas as pd\n",
    "\n",
    "# Make DataFrame of the given data\n",
    "data = pd.DataFrame({\"A\": [1, 2, 4, 1, 2, 4],\n",
    "                     \"B\": [4, 5, 6, 7, 8, 9],\n",
    "                     \"C\": [0, 0, 0, 0, 0, 0],\n",
    "                     \"D\": [1, 1, 1, 1, 1, 1]})\n",
    "\n",
    "\n",
    "# Create VarianceThreshold object with threshold=0\n",
    "var_thres = VarianceThreshold(threshold=0)\n",
    "\n",
    "# Fit the object to the data\n",
    "var_thres.fit(data)\n",
    "\n",
    "# Get the support (boolean mask) of the features\n",
    "feature_mask = var_thres.get_support()\n",
    "\n",
    "# Get the column names of the constant columns\n",
    "constant_columns = [\n",
    "    column for column in data.columns if column not in data.columns[feature_mask]]\n",
    "\n",
    "print(\"Number of constant columns:\", len(constant_columns))\n",
    "\n",
    "for feature in constant_columns:\n",
    "    print(feature)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial Dataset:\n",
      "   A  B  C  D\n",
      "0  1  4  0  1\n",
      "1  2  5  0  1\n",
      "2  4  6  0  1\n",
      "3  1  7  0  1\n",
      "4  2  8  0  1\n",
      "5  4  9  0  1\n",
      "\n",
      "Feature Mask:\n",
      "If the particular feature is selected (True), if not selected (False) \n",
      "[ True  True False False]\n",
      "\n",
      "Constant Columns:\n",
      "Index(['C', 'D'], dtype='object')\n",
      "\n",
      "Updated Dataset:\n",
      "   A  B\n",
      "0  1  4\n",
      "1  2  5\n",
      "2  4  6\n",
      "3  1  7\n",
      "4  2  8\n",
      "5  4  9\n"
     ]
    }
   ],
   "source": [
    "# Import pandas to create DataFrame\n",
    "from sklearn.feature_selection import VarianceThreshold\n",
    "import pandas as pd\n",
    "\n",
    "# Make DataFrame of the given data\n",
    "data = pd.DataFrame({\"A\": [1, 2, 4, 1, 2, 4],\n",
    "                     \"B\": [4, 5, 6, 7, 8, 9],\n",
    "                     \"C\": [0, 0, 0, 0, 0, 0],\n",
    "                     \"D\": [1, 1, 1, 1, 1, 1]})\n",
    "\n",
    "# Display the initial dataset\n",
    "print(\"Initial Dataset:\")\n",
    "print(data)\n",
    "\n",
    "# Create VarianceThreshold object with threshold=0\n",
    "var_thres = VarianceThreshold(threshold=0)\n",
    "\n",
    "# Fit the object to the data\n",
    "var_thres.fit(data)\n",
    "\n",
    "# Get the support (boolean mask) of the features\n",
    "feature_mask = var_thres.get_support()\n",
    "print(\"\\nFeature Mask:\")\n",
    "print(\"If the particular feature is selected (True), if not selected (False) \")\n",
    "print(feature_mask)\n",
    "\n",
    "# Get the column names of the constant columns\n",
    "constant_columns = data.columns[~feature_mask]\n",
    "\n",
    "print(\"\\nConstant Columns:\")\n",
    "print(constant_columns)\n",
    "\n",
    "# Drop the constant columns from the dataset\n",
    "data = data.drop(constant_columns, axis=1)\n",
    "\n",
    "# Display the updated dataset\n",
    "print(\"\\nUpdated Dataset:\")\n",
    "print(data)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Applying the Above technique on Santander Customer Dataset**\n",
    "\n",
    "[Downlode Dataset](https://www.kaggle.com/c/santander-customer-satisfaction/data?select=train.csv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 371)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>var3</th>\n",
       "      <th>var15</th>\n",
       "      <th>imp_ent_var16_ult1</th>\n",
       "      <th>imp_op_var39_comer_ult1</th>\n",
       "      <th>imp_op_var39_comer_ult3</th>\n",
       "      <th>imp_op_var40_comer_ult1</th>\n",
       "      <th>imp_op_var40_comer_ult3</th>\n",
       "      <th>imp_op_var40_efect_ult1</th>\n",
       "      <th>imp_op_var40_efect_ult3</th>\n",
       "      <th>...</th>\n",
       "      <th>saldo_medio_var33_hace2</th>\n",
       "      <th>saldo_medio_var33_hace3</th>\n",
       "      <th>saldo_medio_var33_ult1</th>\n",
       "      <th>saldo_medio_var33_ult3</th>\n",
       "      <th>saldo_medio_var44_hace2</th>\n",
       "      <th>saldo_medio_var44_hace3</th>\n",
       "      <th>saldo_medio_var44_ult1</th>\n",
       "      <th>saldo_medio_var44_ult3</th>\n",
       "      <th>var38</th>\n",
       "      <th>TARGET</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>23</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>39205.170000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>34</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>49278.030000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>23</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>67333.770000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>37</td>\n",
       "      <td>0.0</td>\n",
       "      <td>195.0</td>\n",
       "      <td>195.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>64007.970000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "      <td>39</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>117310.979016</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 371 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   ID  var3  var15  imp_ent_var16_ult1  imp_op_var39_comer_ult1  \\\n",
       "0   1     2     23                 0.0                      0.0   \n",
       "1   3     2     34                 0.0                      0.0   \n",
       "2   4     2     23                 0.0                      0.0   \n",
       "3   8     2     37                 0.0                    195.0   \n",
       "4  10     2     39                 0.0                      0.0   \n",
       "\n",
       "   imp_op_var39_comer_ult3  imp_op_var40_comer_ult1  imp_op_var40_comer_ult3  \\\n",
       "0                      0.0                      0.0                      0.0   \n",
       "1                      0.0                      0.0                      0.0   \n",
       "2                      0.0                      0.0                      0.0   \n",
       "3                    195.0                      0.0                      0.0   \n",
       "4                      0.0                      0.0                      0.0   \n",
       "\n",
       "   imp_op_var40_efect_ult1  imp_op_var40_efect_ult3  ...  \\\n",
       "0                        0                        0  ...   \n",
       "1                        0                        0  ...   \n",
       "2                        0                        0  ...   \n",
       "3                        0                        0  ...   \n",
       "4                        0                        0  ...   \n",
       "\n",
       "   saldo_medio_var33_hace2  saldo_medio_var33_hace3  saldo_medio_var33_ult1  \\\n",
       "0                      0.0                      0.0                     0.0   \n",
       "1                      0.0                      0.0                     0.0   \n",
       "2                      0.0                      0.0                     0.0   \n",
       "3                      0.0                      0.0                     0.0   \n",
       "4                      0.0                      0.0                     0.0   \n",
       "\n",
       "   saldo_medio_var33_ult3  saldo_medio_var44_hace2  saldo_medio_var44_hace3  \\\n",
       "0                     0.0                      0.0                      0.0   \n",
       "1                     0.0                      0.0                      0.0   \n",
       "2                     0.0                      0.0                      0.0   \n",
       "3                     0.0                      0.0                      0.0   \n",
       "4                     0.0                      0.0                      0.0   \n",
       "\n",
       "   saldo_medio_var44_ult1  saldo_medio_var44_ult3          var38  TARGET  \n",
       "0                     0.0                     0.0   39205.170000       0  \n",
       "1                     0.0                     0.0   49278.030000       0  \n",
       "2                     0.0                     0.0   67333.770000       0  \n",
       "3                     0.0                     0.0   64007.970000       0  \n",
       "4                     0.0                     0.0  117310.979016       0  \n",
       "\n",
       "[5 rows x 371 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from sklearn.feature_selection import VarianceThreshold\n",
    "\n",
    "df=pd.read_csv('_dataset\\santander.csv',nrows=10000)\n",
    "print(df.shape)\n",
    "display(df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.drop(labels=['TARGET'], axis=1)\n",
    "y = df['TARGET']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(7000, 370) (3000, 370)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# separate dataset into train and test\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    df.drop(labels=['TARGET'], axis=1),\n",
    "    df['TARGET'],\n",
    "    test_size=0.3,\n",
    "    random_state=42)\n",
    "\n",
    "print(X_train.shape, X_test.shape)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Lets apply the variance threshold**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "VarianceThreshold(threshold=0)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "var_thres = VarianceThreshold(threshold=0)\n",
    "var_thres.fit(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True, False, False,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True, False, False,  True,  True,  True,  True,  True, False,\n",
       "       False,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True, False, False, False, False,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "       False, False,  True,  True,  True,  True,  True,  True,  True,\n",
       "       False,  True,  True,  True, False, False,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True, False, False,  True,  True,  True,\n",
       "        True,  True, False, False,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True, False, False, False,\n",
       "       False,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True, False, False,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True, False,  True,  True,  True,  True,  True,\n",
       "       False, False,  True,  True,  True,  True,  True,  True,  True,\n",
       "       False,  True,  True,  True, False,  True,  True,  True,  True,\n",
       "       False, False,  True,  True,  True,  True,  True, False,  True,\n",
       "        True, False,  True,  True, False,  True, False, False,  True,\n",
       "        True,  True,  True,  True, False, False, False, False,  True,\n",
       "       False,  True,  True,  True,  True,  True,  True, False, False,\n",
       "       False, False,  True, False,  True, False, False, False, False,\n",
       "        True,  True,  True,  True,  True,  True, False,  True,  True,\n",
       "        True, False,  True, False, False, False, False,  True,  True,\n",
       "       False, False, False, False, False,  True, False, False, False,\n",
       "        True, False,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True, False, False,  True,  True,  True,  True,  True,  True,\n",
       "       False,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True, False,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True, False,  True,\n",
       "       False, False, False, False,  True,  True,  True,  True, False,\n",
       "       False, False, False, False,  True, False, False, False,  True,\n",
       "        True,  True,  True,  True, False,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True, False, False,\n",
       "       False, False,  True,  True,  True,  True,  True, False,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "var_thres.get_support()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "281"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Finding non constant features\n",
    "sum(var_thres.get_support())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "281"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Lets Find non-constant features \n",
    "len(X_train.columns[var_thres.get_support()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "89\n"
     ]
    }
   ],
   "source": [
    "constant_columns = [column for column in X_train.columns\n",
    "                    if column not in X_train.columns[var_thres.get_support()]]\n",
    "\n",
    "print(len(constant_columns))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ind_var2_0\n",
      "ind_var2\n",
      "ind_var13_medio_0\n",
      "ind_var13_medio\n",
      "ind_var18_0\n",
      "ind_var18\n",
      "ind_var27_0\n",
      "ind_var28_0\n",
      "ind_var28\n",
      "ind_var27\n",
      "ind_var34_0\n",
      "ind_var34\n",
      "ind_var41\n",
      "ind_var46_0\n",
      "ind_var46\n",
      "num_var13_medio_0\n",
      "num_var13_medio\n",
      "num_var18_0\n",
      "num_var18\n",
      "num_var27_0\n",
      "num_var28_0\n",
      "num_var28\n",
      "num_var27\n",
      "num_var34_0\n",
      "num_var34\n",
      "num_var41\n",
      "num_var46_0\n",
      "num_var46\n",
      "saldo_var13_medio\n",
      "saldo_var18\n",
      "saldo_var28\n",
      "saldo_var27\n",
      "saldo_var34\n",
      "saldo_var41\n",
      "saldo_var46\n",
      "delta_imp_amort_var18_1y3\n",
      "delta_imp_amort_var34_1y3\n",
      "delta_imp_reemb_var17_1y3\n",
      "delta_imp_reemb_var33_1y3\n",
      "delta_imp_trasp_var17_in_1y3\n",
      "delta_imp_trasp_var17_out_1y3\n",
      "delta_imp_trasp_var33_out_1y3\n",
      "delta_num_reemb_var17_1y3\n",
      "delta_num_reemb_var33_1y3\n",
      "delta_num_trasp_var17_in_1y3\n",
      "delta_num_trasp_var17_out_1y3\n",
      "delta_num_trasp_var33_out_1y3\n",
      "imp_amort_var18_hace3\n",
      "imp_amort_var18_ult1\n",
      "imp_amort_var34_hace3\n",
      "imp_amort_var34_ult1\n",
      "imp_var7_emit_ult1\n",
      "imp_reemb_var13_hace3\n",
      "imp_reemb_var17_hace3\n",
      "imp_reemb_var17_ult1\n",
      "imp_reemb_var33_hace3\n",
      "imp_reemb_var33_ult1\n",
      "imp_trasp_var17_in_hace3\n",
      "imp_trasp_var17_in_ult1\n",
      "imp_trasp_var17_out_hace3\n",
      "imp_trasp_var17_out_ult1\n",
      "imp_trasp_var33_in_hace3\n",
      "imp_trasp_var33_out_hace3\n",
      "imp_trasp_var33_out_ult1\n",
      "imp_venta_var44_hace3\n",
      "ind_var7_emit_ult1\n",
      "num_var2_0_ult1\n",
      "num_var2_ult1\n",
      "num_var7_emit_ult1\n",
      "num_meses_var13_medio_ult3\n",
      "num_reemb_var13_hace3\n",
      "num_reemb_var17_hace3\n",
      "num_reemb_var17_ult1\n",
      "num_reemb_var33_hace3\n",
      "num_reemb_var33_ult1\n",
      "num_trasp_var17_in_hace3\n",
      "num_trasp_var17_in_ult1\n",
      "num_trasp_var17_out_hace3\n",
      "num_trasp_var17_out_ult1\n",
      "num_trasp_var33_in_hace3\n",
      "num_trasp_var33_out_hace3\n",
      "num_trasp_var33_out_ult1\n",
      "num_venta_var44_hace3\n",
      "saldo_var2_ult1\n",
      "saldo_medio_var13_medio_hace2\n",
      "saldo_medio_var13_medio_hace3\n",
      "saldo_medio_var13_medio_ult1\n",
      "saldo_medio_var13_medio_ult3\n",
      "saldo_medio_var29_hace3\n"
     ]
    }
   ],
   "source": [
    "for column in constant_columns:\n",
    "    print(column)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>var3</th>\n",
       "      <th>var15</th>\n",
       "      <th>imp_ent_var16_ult1</th>\n",
       "      <th>imp_op_var39_comer_ult1</th>\n",
       "      <th>imp_op_var39_comer_ult3</th>\n",
       "      <th>imp_op_var40_comer_ult1</th>\n",
       "      <th>imp_op_var40_comer_ult3</th>\n",
       "      <th>imp_op_var40_efect_ult1</th>\n",
       "      <th>imp_op_var40_efect_ult3</th>\n",
       "      <th>...</th>\n",
       "      <th>saldo_medio_var29_ult3</th>\n",
       "      <th>saldo_medio_var33_hace2</th>\n",
       "      <th>saldo_medio_var33_hace3</th>\n",
       "      <th>saldo_medio_var33_ult1</th>\n",
       "      <th>saldo_medio_var33_ult3</th>\n",
       "      <th>saldo_medio_var44_hace2</th>\n",
       "      <th>saldo_medio_var44_hace3</th>\n",
       "      <th>saldo_medio_var44_ult1</th>\n",
       "      <th>saldo_medio_var44_ult3</th>\n",
       "      <th>var38</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9069</th>\n",
       "      <td>18258</td>\n",
       "      <td>2</td>\n",
       "      <td>23</td>\n",
       "      <td>0.0</td>\n",
       "      <td>300.9</td>\n",
       "      <td>813.75</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>117310.979016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2603</th>\n",
       "      <td>5211</td>\n",
       "      <td>2</td>\n",
       "      <td>29</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>69825.090000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7738</th>\n",
       "      <td>15572</td>\n",
       "      <td>2</td>\n",
       "      <td>25</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>88115.130000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1579</th>\n",
       "      <td>3125</td>\n",
       "      <td>2</td>\n",
       "      <td>24</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>150490.350000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5058</th>\n",
       "      <td>10101</td>\n",
       "      <td>2</td>\n",
       "      <td>33</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>117310.979016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5734</th>\n",
       "      <td>11502</td>\n",
       "      <td>2</td>\n",
       "      <td>23</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>78929.940000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5191</th>\n",
       "      <td>10394</td>\n",
       "      <td>2</td>\n",
       "      <td>49</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>127000.770000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5390</th>\n",
       "      <td>10815</td>\n",
       "      <td>2</td>\n",
       "      <td>24</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>95723.880000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>860</th>\n",
       "      <td>1703</td>\n",
       "      <td>2</td>\n",
       "      <td>17</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>101492.340000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7270</th>\n",
       "      <td>14585</td>\n",
       "      <td>2</td>\n",
       "      <td>23</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>195326.610000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>7000 rows × 281 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         ID  var3  var15  imp_ent_var16_ult1  imp_op_var39_comer_ult1  \\\n",
       "9069  18258     2     23                 0.0                    300.9   \n",
       "2603   5211     2     29                 0.0                      0.0   \n",
       "7738  15572     2     25                 0.0                      0.0   \n",
       "1579   3125     2     24                 0.0                      0.0   \n",
       "5058  10101     2     33                 0.0                      0.0   \n",
       "...     ...   ...    ...                 ...                      ...   \n",
       "5734  11502     2     23                 0.0                      0.0   \n",
       "5191  10394     2     49                 0.0                      0.0   \n",
       "5390  10815     2     24                 0.0                      0.0   \n",
       "860    1703     2     17                 0.0                      0.0   \n",
       "7270  14585     2     23                 0.0                      0.0   \n",
       "\n",
       "      imp_op_var39_comer_ult3  imp_op_var40_comer_ult1  \\\n",
       "9069                   813.75                      0.0   \n",
       "2603                     0.00                      0.0   \n",
       "7738                     0.00                      0.0   \n",
       "1579                     0.00                      0.0   \n",
       "5058                     0.00                      0.0   \n",
       "...                       ...                      ...   \n",
       "5734                     0.00                      0.0   \n",
       "5191                     0.00                      0.0   \n",
       "5390                     0.00                      0.0   \n",
       "860                      0.00                      0.0   \n",
       "7270                     0.00                      0.0   \n",
       "\n",
       "      imp_op_var40_comer_ult3  imp_op_var40_efect_ult1  \\\n",
       "9069                      0.0                        0   \n",
       "2603                      0.0                        0   \n",
       "7738                      0.0                        0   \n",
       "1579                      0.0                        0   \n",
       "5058                      0.0                        0   \n",
       "...                       ...                      ...   \n",
       "5734                      0.0                        0   \n",
       "5191                      0.0                        0   \n",
       "5390                      0.0                        0   \n",
       "860                       0.0                        0   \n",
       "7270                      0.0                        0   \n",
       "\n",
       "      imp_op_var40_efect_ult3  ...  saldo_medio_var29_ult3  \\\n",
       "9069                        0  ...                     0.0   \n",
       "2603                        0  ...                     0.0   \n",
       "7738                        0  ...                     0.0   \n",
       "1579                        0  ...                     0.0   \n",
       "5058                        0  ...                     0.0   \n",
       "...                       ...  ...                     ...   \n",
       "5734                        0  ...                     0.0   \n",
       "5191                        0  ...                     0.0   \n",
       "5390                        0  ...                     0.0   \n",
       "860                         0  ...                     0.0   \n",
       "7270                        0  ...                     0.0   \n",
       "\n",
       "      saldo_medio_var33_hace2  saldo_medio_var33_hace3  \\\n",
       "9069                      0.0                      0.0   \n",
       "2603                      0.0                      0.0   \n",
       "7738                      0.0                      0.0   \n",
       "1579                      0.0                      0.0   \n",
       "5058                      0.0                      0.0   \n",
       "...                       ...                      ...   \n",
       "5734                      0.0                      0.0   \n",
       "5191                      0.0                      0.0   \n",
       "5390                      0.0                      0.0   \n",
       "860                       0.0                      0.0   \n",
       "7270                      0.0                      0.0   \n",
       "\n",
       "      saldo_medio_var33_ult1  saldo_medio_var33_ult3  saldo_medio_var44_hace2  \\\n",
       "9069                     0.0                     0.0                      0.0   \n",
       "2603                     0.0                     0.0                      0.0   \n",
       "7738                     0.0                     0.0                      0.0   \n",
       "1579                     0.0                     0.0                      0.0   \n",
       "5058                     0.0                     0.0                      0.0   \n",
       "...                      ...                     ...                      ...   \n",
       "5734                     0.0                     0.0                      0.0   \n",
       "5191                     0.0                     0.0                      0.0   \n",
       "5390                     0.0                     0.0                      0.0   \n",
       "860                      0.0                     0.0                      0.0   \n",
       "7270                     0.0                     0.0                      0.0   \n",
       "\n",
       "      saldo_medio_var44_hace3  saldo_medio_var44_ult1  saldo_medio_var44_ult3  \\\n",
       "9069                      0.0                     0.0                     0.0   \n",
       "2603                      0.0                     0.0                     0.0   \n",
       "7738                      0.0                     0.0                     0.0   \n",
       "1579                      0.0                     0.0                     0.0   \n",
       "5058                      0.0                     0.0                     0.0   \n",
       "...                       ...                     ...                     ...   \n",
       "5734                      0.0                     0.0                     0.0   \n",
       "5191                      0.0                     0.0                     0.0   \n",
       "5390                      0.0                     0.0                     0.0   \n",
       "860                       0.0                     0.0                     0.0   \n",
       "7270                      0.0                     0.0                     0.0   \n",
       "\n",
       "              var38  \n",
       "9069  117310.979016  \n",
       "2603   69825.090000  \n",
       "7738   88115.130000  \n",
       "1579  150490.350000  \n",
       "5058  117310.979016  \n",
       "...             ...  \n",
       "5734   78929.940000  \n",
       "5191  127000.770000  \n",
       "5390   95723.880000  \n",
       "860   101492.340000  \n",
       "7270  195326.610000  \n",
       "\n",
       "[7000 rows x 281 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.drop(constant_columns,axis=1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **On Iris Dataset**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.simplefilter(action='ignore', category=FutureWarning)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before preprocessing:\n",
      "Number of samples: 891\n",
      "Number of features: 10\n",
      "Accuracy before preprocessing: 0.7374301675977654\n",
      "After feature selection:\n",
      "Number of samples: 891\n",
      "Number of features: 9\n",
      "Accuracy after feature selection: 0.7430167597765364\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from sklearn.feature_selection import VarianceThreshold\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.metrics import accuracy_score\n",
    "import seaborn as sns\n",
    "\n",
    "# Load the Titanic dataset from Seaborn\n",
    "titanic_data = sns.load_dataset('titanic')\n",
    "\n",
    "# Preprocessing\n",
    "titanic_data = titanic_data.drop(['class', 'deck', 'embark_town', 'alive', 'alone'], axis=1)\n",
    "titanic_data['sex'] = titanic_data['sex'].map({'female': 0, 'male': 1})\n",
    "titanic_data['age'] = titanic_data['age'].fillna(titanic_data['age'].mean())\n",
    "titanic_data['fare'] = titanic_data['fare'].fillna(titanic_data['fare'].mean())\n",
    "titanic_data['family_size'] = titanic_data['sibsp'] + titanic_data['parch'] + 1\n",
    "titanic_data = titanic_data.drop(['sibsp', 'parch'], axis=1)\n",
    "\n",
    "# Perform one-hot encoding on categorical variables\n",
    "titanic_data = pd.get_dummies(titanic_data, drop_first=True)\n",
    "\n",
    "X = titanic_data.drop('survived', axis=1)\n",
    "y = titanic_data['survived']\n",
    "\n",
    "# Split the data into training and testing sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.2, random_state=42)\n",
    "\n",
    "# Print the shape of the dataset before preprocessing\n",
    "print(\"Before preprocessing:\")\n",
    "print(\"Number of samples:\", X.shape[0])\n",
    "print(\"Number of features:\", X.shape[1])\n",
    "\n",
    "# Train a KNN classifier on the original dataset\n",
    "clf_original = KNeighborsClassifier()\n",
    "clf_original.fit(X_train, y_train)\n",
    "y_pred_original = clf_original.predict(X_test)\n",
    "accuracy_original = accuracy_score(y_test, y_pred_original)\n",
    "print(\"Accuracy before preprocessing:\", accuracy_original)\n",
    "\n",
    "# Apply VarianceThreshold with threshold 0.2\n",
    "selector = VarianceThreshold(threshold=0.2)\n",
    "X_selected = selector.fit_transform(X)\n",
    "\n",
    "# Split the selected features into training and testing sets\n",
    "X_train_selected, X_test_selected, y_train, y_test = train_test_split(\n",
    "    X_selected, y, test_size=0.2, random_state=42)\n",
    "\n",
    "# Print the shape of the dataset after feature selection\n",
    "print(\"After feature selection:\")\n",
    "print(\"Number of samples:\", X_selected.shape[0])\n",
    "print(\"Number of features:\", X_selected.shape[1])\n",
    "\n",
    "# Train a KNN classifier on the selected features\n",
    "clf_selected = KNeighborsClassifier()\n",
    "clf_selected.fit(X_train_selected, y_train)\n",
    "y_pred_selected = clf_selected.predict(X_test_selected)\n",
    "accuracy_selected = accuracy_score(y_test, y_pred_selected)\n",
    "print(\"Accuracy after feature selection:\", accuracy_selected)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before preprocessing:\n",
      "Number of samples: 891\n",
      "Number of features: 10\n",
      "After feature selection:\n",
      "Number of samples: 891\n",
      "Number of features: 9\n",
      "Accuracy after feature selection and hyperparameter tuning: 0.7206703910614525\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from sklearn.feature_selection import VarianceThreshold\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.metrics import accuracy_score\n",
    "import seaborn as sns\n",
    "\n",
    "# Load the Titanic dataset from Seaborn\n",
    "titanic_data = sns.load_dataset('titanic')\n",
    "\n",
    "# Preprocessing\n",
    "titanic_data = titanic_data.drop(\n",
    "    ['class', 'deck', 'embark_town', 'alive', 'alone'], axis=1)\n",
    "titanic_data['sex'] = titanic_data['sex'].map({'female': 0, 'male': 1})\n",
    "titanic_data['age'] = titanic_data['age'].fillna(titanic_data['age'].mean())\n",
    "titanic_data['fare'] = titanic_data['fare'].fillna(titanic_data['fare'].mean())\n",
    "titanic_data['family_size'] = titanic_data['sibsp'] + titanic_data['parch'] + 1\n",
    "titanic_data = titanic_data.drop(['sibsp', 'parch'], axis=1)\n",
    "\n",
    "# Perform one-hot encoding on categorical variables\n",
    "titanic_data = pd.get_dummies(titanic_data, drop_first=True)\n",
    "\n",
    "X = titanic_data.drop('survived', axis=1)\n",
    "y = titanic_data['survived']\n",
    "\n",
    "# Split the data into training and testing sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.2, random_state=42)\n",
    "\n",
    "# Print the shape of the dataset before feature selection\n",
    "print(\"Before preprocessing:\")\n",
    "print(\"Number of samples:\", X.shape[0])\n",
    "print(\"Number of features:\", X.shape[1])\n",
    "\n",
    "# Apply VarianceThreshold with threshold 0.2\n",
    "selector = VarianceThreshold(threshold=0.2)\n",
    "X_selected = selector.fit_transform(X)\n",
    "\n",
    "# Print the shape of the dataset after feature selection\n",
    "print(\"After feature selection:\")\n",
    "print(\"Number of samples:\", X_selected.shape[0])\n",
    "print(\"Number of features:\", X_selected.shape[1])\n",
    "\n",
    "# Split the selected features into training and testing sets\n",
    "X_train_selected, X_test_selected, y_train, y_test = train_test_split(\n",
    "    X_selected, y, test_size=0.2, random_state=42)\n",
    "\n",
    "# Define the parameter grid for grid search\n",
    "param_grid = {\n",
    "    'n_neighbors': [3, 5, 7, 9, 11],  # Vary the number of neighbors\n",
    "    'weights': ['uniform', 'distance'],  # Explore different weight options\n",
    "}\n",
    "\n",
    "# Perform grid search to find the best hyperparameters\n",
    "clf_selected = KNeighborsClassifier()\n",
    "grid_search = GridSearchCV(clf_selected, param_grid, cv=5)\n",
    "grid_search.fit(X_train_selected, y_train)\n",
    "\n",
    "# Get the best model from grid search\n",
    "best_model = grid_search.best_estimator_\n",
    "\n",
    "# Make predictions on the test set using the best model\n",
    "y_pred_selected = best_model.predict(X_test_selected)\n",
    "accuracy_selected = accuracy_score(y_test, y_pred_selected)\n",
    "print(\"Accuracy after feature selection and hyperparameter tuning:\", accuracy_selected)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}