Arvato Project Workbook.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3.8.10 64-bit ('dsenv': conda)"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.10"
    },
    "interpreter": {
      "hash": "a7e01aac2c9bcb8fe1e7851228a633d1435a51d01cb4372225780e1a41da0658"
    },
    "colab": {
      "name": "Arvato Project Workbook.ipynb",
      "provenance": [],
      "collapsed_sections": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-RNcCEhtbUGr"
      },
      "source": [
        "# Capstone Project: Create a Customer Segmentation Report for Arvato Financial Services\n",
        "\n",
        "In this project, you will analyze demographics data for customers of a mail-order sales company in Germany, comparing it against demographics information for the general population. You'll use unsupervised learning techniques to perform customer segmentation, identifying the parts of the population that best describe the core customer base of the company. Then, you'll apply what you've learned on a third dataset with demographics information for targets of a marketing campaign for the company, and use a model to predict which individuals are most likely to convert into becoming customers for the company. The data that you will use has been provided by our partners at Bertelsmann Arvato Analytics, and represents a real-life data science task.\n",
        "\n",
        "If you completed the first term of this program, you will be familiar with the first part of this project, from the unsupervised learning project. The versions of those two datasets used in this project will include many more features and has not been pre-cleaned. You are also free to choose whatever approach you'd like to analyzing the data rather than follow pre-determined steps. In your work on this project, make sure that you carefully document your steps and decisions, since your main deliverable for this project will be a blog post reporting your findings."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "id": "x6GKbUm7gDpR",
        "outputId": "5ddc6bd9-9ffd-4f29-b819-a02708d4e2ca"
      },
      "source": [
        "# When using Google Colab we want to mount the data drive\n",
        "try:\n",
        "    from google.colab import drive\n",
        "    drive.mount('/content/drive')\n",
        "    IN_COLAB = True\n",
        "except:\n",
        "    IN_COLAB = False\n"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "id": "OEIRP6RoUWWL",
        "outputId": "ba7a2db7-d523-46cc-b1e9-63f41c3ced66"
      },
      "source": [
        "if IN_COLAB:\n",
        "    !pip3 install -U scikit-learn"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (1.0.1)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn) (3.0.0)\n",
            "Requirement already satisfied: numpy>=1.14.6 in /usr/local/lib/python3.7/dist-packages (from scikit-learn) (1.19.5)\n",
            "Requirement already satisfied: scipy>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn) (1.4.1)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn) (1.1.0)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3mAkhKUJZkbF",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "486f0291-d1b2-440d-f5ea-2f402f2c7994"
      },
      "source": [
        "#Install kneed package if not there\n",
        "kneed_installed = !pip list | grep kneed\n",
        "if not kneed_installed:\n",
        "    !pip install kneed"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting kneed\n",
            "  Downloading kneed-0.7.0-py2.py3-none-any.whl (9.4 kB)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from kneed) (3.2.2)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from kneed) (1.4.1)\n",
            "Requirement already satisfied: numpy>=1.14.2 in /usr/local/lib/python3.7/dist-packages (from kneed) (1.19.5)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->kneed) (1.3.2)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->kneed) (2.8.2)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->kneed) (2.4.7)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->kneed) (0.11.0)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.1->matplotlib->kneed) (1.15.0)\n",
            "Installing collected packages: kneed\n",
            "Successfully installed kneed-0.7.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NBWy9JrpbUGv",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "defd158e-c400-4a9c-fe75-100139b0ace6"
      },
      "source": [
        "# import libraries here; add more as necessary\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from scipy.stats import entropy\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "# from functools import reduce\n",
        "from timeit import default_timer as timer\n",
        "from datetime import datetime, timedelta\n",
        "import gc\n",
        "\n",
        "pd.options.mode.chained_assignment = None  # default='warn'\n",
        "\n",
        "# magic word for producing visualizations in notebook\n",
        "%matplotlib inline\n",
        "\n",
        "# Google Colab environment:\n",
        "if IN_COLAB:\n",
        "    # Get pandas version\n",
        "    print(\"Pandas version: \", pd.__version__)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Pandas version:  1.1.5\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VuCRP43NbUGx"
      },
      "source": [
        "## Part 0: Get to Know the Data\n",
        "\n",
        "There are four data files associated with this project:\n",
        "\n",
        "- `Udacity_AZDIAS_052018.csv`: Demographics data for the general population of Germany; 891 211 persons (rows) x 366 features (columns).\n",
        "- `Udacity_CUSTOMERS_052018.csv`: Demographics data for customers of a mail-order company; 191 652 persons (rows) x 369 features (columns).\n",
        "- `Udacity_MAILOUT_052018_TRAIN.csv`: Demographics data for individuals who were targets of a marketing campaign; 42 982 persons (rows) x 367 (columns).\n",
        "- `Udacity_MAILOUT_052018_TEST.csv`: Demographics data for individuals who were targets of a marketing campaign; 42 833 persons (rows) x 366 (columns).\n",
        "\n",
        "Each row of the demographics files represents a single person, but also includes information outside of individuals, including information about their household, building, and neighborhood. Use the information from the first two files to figure out how customers (\"CUSTOMERS\") are similar to or differ from the general population at large (\"AZDIAS\"), then use your analysis to make predictions on the other two files (\"MAILOUT\"), predicting which recipients are most likely to become a customer for the mail-order company.\n",
        "\n",
        "The \"CUSTOMERS\" file contains three extra columns ('CUSTOMER_GROUP', 'ONLINE_PURCHASE', and 'PRODUCT_GROUP'), which provide broad information about the customers depicted in the file. The original \"MAILOUT\" file included one additional column, \"RESPONSE\", which indicated whether or not each recipient became a customer of the company. For the \"TRAIN\" subset, this column has been retained, but in the \"TEST\" subset it has been removed; it is against that withheld column that your final predictions will be assessed in the Kaggle competition.\n",
        "\n",
        "Otherwise, all of the remaining columns are the same between the three data files. For more information about the columns depicted in the files, you can refer to two Excel spreadsheets provided in the workspace. [One of them](./DIAS Information Levels - Attributes 2017.xlsx) is a top-level list of attributes and descriptions, organized by informational category. [The other](./DIAS Attributes - Values 2017.xlsx) is a detailed mapping of data values for each feature in alphabetical order.\n",
        "\n",
        "In the below cell, we've provided some initial code to load in the first two datasets. Note for all of the `.csv` data files in this project that they're semicolon (`;`) delimited, so an additional argument in the [`read_csv()`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html) call has been included to read in the data properly. Also, considering the size of the datasets, it may take some time for them to load completely.\n",
        "\n",
        "You'll notice when the data is loaded in that a warning message will immediately pop up. Before you really start digging into the modeling and analysis, you're going to need to perform some cleaning. Take some time to browse the structure of the data and look over the informational spreadsheets to understand the data values. Make some decisions on which features to keep, which features to drop, and if any revisions need to be made on data formats. It'll be a good idea to create a function with pre-processing steps, since you'll need to clean all of the datasets before you work with them."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W3qRTNYjIR3H"
      },
      "source": [
        "## Strategy for exploring the data:\n",
        "\n",
        "* Load datasets. Initially load random samples, finally load the full datasets.\n",
        "* Load excel information files.\n",
        "* Get the attributes from excel files and compare them to the attributes appearing in the datasets.\n",
        "* Get the value pools for each known attribute in the excel files and check against the values for each attribute in the datasets.\n",
        "* Replace the values outside the pools with NaNs.\n",
        "* Replace values in the pools representing unknowns, missing, not given, etc. with NaNs.\n",
        "* Explore NaNs, analysing and understanding missing values in the two datasets.\n",
        "* Identify significance of categorical, ordinal attributes."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5MKY46QFSBv4"
      },
      "source": [
        "### Load data and excel files"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uiL6n6PU1faD"
      },
      "source": [
        "# Start timer notebook run\n",
        "start_run = timer()\n",
        "\n",
        "# Current date/time suffix for filenames\n",
        "ts = datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n",
        "\n",
        "# figure bbox caption\n",
        "cprt = '© 2021 · Chris Liatas'\n",
        "# copyright bbox properties\n",
        "bb_cprt_args = dict(boxstyle='square,pad=47.8', mutation_aspect=.015, facecolor='grey', alpha=0.3)"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3ogvB6EVbUGy"
      },
      "source": [
        "# Assign path names to variables for easier manipulation later on.\n",
        "colab_dir = \"/content/drive/MyDrive/workspace/arvato_data/\"\n",
        "remote_dir = \"../../data/Term2/capstone/arvato_data/\"\n",
        "\n",
        "azdias_file = f\"{colab_dir if IN_COLAB else remote_dir}Udacity_AZDIAS_052018.csv\"\n",
        "customers_file = f\"{colab_dir if IN_COLAB else remote_dir}Udacity_CUSTOMERS_052018.csv\"\n",
        "mailout_train_file = f\"{colab_dir if IN_COLAB else remote_dir}Udacity_MAILOUT_052018_TRAIN.csv\"\n",
        "mailout_test_file = f\"{colab_dir if IN_COLAB else remote_dir}Udacity_MAILOUT_052018_TEST.csv\"\n",
        "\n",
        "# Files with information about the columns depicted in the csv files\n",
        "dias_attrib = f\"{colab_dir if IN_COLAB else remote_dir}DIAS Attributes - Values 2017.xlsx\"\n",
        "dias_info = f\"{colab_dir if IN_COLAB else remote_dir}DIAS Information Levels - Attributes 2017.xlsx\"\n",
        "\n",
        "# Kaggle submission file\n",
        "kg_submission = colab_dir if IN_COLAB else \"\"\n",
        "# Kaggle competition\n",
        "kg_competition = 'udacity-arvato-identify-customers'"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z5hxSvIhbUG0"
      },
      "source": [
        "We will create a function to load a sample of the datasets to speed things up during the data exploration phase, particularly for the larger azdias dataset. This will reduce the memory used for holding the whole dataset initially, but also greatly improve the speed of calculations when exploring and cleaning the data. We will later use the whole dataset for best results.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VDBER8ePsw3o"
      },
      "source": [
        "def get_df_sample(n_samples, seed=None, *args, **kwargs):\n",
        "    \"\"\"Load a random sample out of an input csv file or the whole file.\n",
        "\n",
        "    Args:\n",
        "        n_samples (int): Size of sample (number of rows) to randomly load from the input file.\n",
        "        args, kwargs: Pass arguments and keyword arguments as in pandas `read_csv` method.\n",
        "        Ref: https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html\n",
        "\n",
        "    Returns:\n",
        "        The dataframe loaded from the input file of n_samples rows\n",
        "    \n",
        "    Reference:\n",
        "        Stack Overflow - `Read a small random sample from a big CSV file into a Python data frame`:\n",
        "        https://stackoverflow.com/a/61631765/10074873\n",
        "        Pandas - `Iterating through files chunk by chunk`:\n",
        "        https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#iterating-through-files-chunk-by-chunk\n",
        "        Numpy - `Random Generator`:\n",
        "        https://numpy.org/doc/stable/reference/random/generator.html#random-generator\n",
        "    \"\"\"\n",
        "    rng = np.random.default_rng(seed=seed)\n",
        "\n",
        "    samples_reader = pd.read_csv(*args, **kwargs)\n",
        "    df = samples_reader.get_chunk(n_samples)\n",
        "\n",
        "    for chunk in samples_reader:\n",
        "        chunk.index = rng.integers(n_samples, size=len(chunk))\n",
        "        df.loc[chunk.index] = chunk\n",
        "    \n",
        "    return df\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8kliWVB50HXY"
      },
      "source": [
        "* Read first the attributes as strings, to get the categories and then use that to read them directly as categorical."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PdORfEzW6w8N"
      },
      "source": [
        "# Set to `True` to go through all the cells.\n",
        "RUN_ALL = False"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r1YX1l9S6aPb"
      },
      "source": [
        "if RUN_ALL:\n",
        "    # Try loading a sample\n",
        "    azdias_try1 = get_df_sample(200000, 42, azdias_file, sep=';', chunksize=256000)\n",
        "\n",
        "    azdias_try1.head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ts54Y4ZyFKDY"
      },
      "source": [
        "if RUN_ALL:\n",
        "    # Check the types of columns\n",
        "    azdias_try1.dtypes.value_counts()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dKve66KCFS5B"
      },
      "source": [
        "if RUN_ALL:\n",
        "    # Check `object` columns\n",
        "    azdias_try1.select_dtypes(include=object).head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HxJkGQjy8dEw"
      },
      "source": [
        "####Check columns 18, 19 that produced warnings when reading csv data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0mGiZ5Gn8dE3"
      },
      "source": [
        "if RUN_ALL:\n",
        "    # Check the two columns to identify why pandas \"complained\" about mixed types\n",
        "    azdias_try1_1819 = azdias_try1.iloc[:,[18, 19]]\n",
        "\n",
        "    azdias_try1_1819.columns"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8QUsT3by9wSj"
      },
      "source": [
        "The two attributes are `'CAMEO_DEUG_2015', 'CAMEO_INTL_2015'`. We should check their values range to further explore them."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "neGj2Cm38dE3"
      },
      "source": [
        "if RUN_ALL:\n",
        "    azdias_try1_1819.CAMEO_DEUG_2015.unique(), azdias_try1_1819.CAMEO_INTL_2015.unique()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JuWaPndOEppa"
      },
      "source": [
        "if RUN_ALL:\n",
        "    # Clean up\n",
        "    del azdias_try1, azdias_try1_1819"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VLr5AMniCUyy"
      },
      "source": [
        "In these columns there are two values `X` and `XX` respectively that fall outside the range of values for this attributes as seen in `DIAS Attributes - Values 2017.xlsx`.\n",
        "\n",
        "#### Importing values as NaNs at loading with `read_csv`\n",
        "\n",
        "By observing the helper files `DIAS Attributes - Values 2017.xlsx` and `DIAS Information Levels - Attributes 2017.xlsx` we can see that zeros (`0` or `0.0`) and `-1` (`-1` or `-1.0`) values may be converted to NaNs as they offer no information even in the fields that are considered as valid input. `na_values` argument of `read_csv` method will be used for this to do this conversion during data import. This will leave the value `9` that is considered as `unknown` in some variables to be processed later.\n",
        "\n",
        "We will take advantage of the `na_values` argument in `read_csv`method to pass a tuple of values that will:\n",
        "\n",
        "* convert specific values to NaNs\n",
        "* allow auto importing of attributes that were mixed types as specific types (numerical).\n",
        "\n",
        "We will also use `parse_dates` argument to import attribute `EINGEFUEGT_AM` as datetime.\n",
        "\n",
        "There are there more categorical attributes (`'CAMEO_DEU_2015', 'D19_LETZTER_KAUF_BRANCHE', 'OST_WEST_KZ'`) that we can identify which will be converted later on.\n",
        "\n",
        "We will initially use samples of the supplied datasets to be able to explore the data and perform all necessary computations faster, preventing notebook crashes due to insufficient memory.\n",
        "\n",
        "A sample of 200,000 rows will be used for `AZDIAS` and 150,000 rows for `CUSTOMERS` datasets to keep a relative balance between the two datasets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QaCzkHiQyQlK"
      },
      "source": [
        "azd_sample_size = 300000\n",
        "cus_sample_size = 150000\n",
        "\n",
        "# Specific values to replace as np.NaN while loading the dataframe.\n",
        "na_tupl = (-1, -1.0, 0, 0.0, 'X', 'XX')\n",
        "\n",
        "# `read_csv` parameters dictionary\n",
        "read_csv_kwargs = {\n",
        "    'sep': ';',\n",
        "    'na_values': na_tupl,\n",
        "    'chunksize': 256000,\n",
        "    'parse_dates': ['EINGEFUEGT_AM'],\n",
        "}\n",
        "\n",
        "# Read samples from azdias, customers datasets of size n rows, (use parse_dates to convert `EINGEFUEGT_AM` to datetime).\n",
        "azdias = get_df_sample(azd_sample_size, None, azdias_file, **read_csv_kwargs)\n",
        "# custrs = pd.read_csv(customers_file, sep=';', na_values=na_tupl, parse_dates=['EINGEFUEGT_AM'])\n",
        "custrs = get_df_sample(cus_sample_size, None, customers_file, **read_csv_kwargs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FiZMkNa0bUG2"
      },
      "source": [
        "Let us check the dataframes now:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kKhi531fbUG3",
        "outputId": "6b5bd5d7-c836-4746-8420-7666a5f7729e"
      },
      "source": [
        "azdias.shape, custrs.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((300000, 366), (150000, 369))"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 315
        },
        "id": "eQhf-DIIapyF",
        "outputId": "c91a5cd7-d7df-4f3f-9b09-4b45934d5ac1"
      },
      "source": [
        "azdias.sample(7)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>LNR</th>\n",
              "      <th>AGER_TYP</th>\n",
              "      <th>AKT_DAT_KL</th>\n",
              "      <th>ALTER_HH</th>\n",
              "      <th>ALTER_KIND1</th>\n",
              "      <th>ALTER_KIND2</th>\n",
              "      <th>ALTER_KIND3</th>\n",
              "      <th>ALTER_KIND4</th>\n",
              "      <th>ALTERSKATEGORIE_FEIN</th>\n",
              "      <th>ANZ_HAUSHALTE_AKTIV</th>\n",
              "      <th>ANZ_HH_TITEL</th>\n",
              "      <th>ANZ_KINDER</th>\n",
              "      <th>ANZ_PERSONEN</th>\n",
              "      <th>ANZ_STATISTISCHE_HAUSHALTE</th>\n",
              "      <th>ANZ_TITEL</th>\n",
              "      <th>ARBEIT</th>\n",
              "      <th>BALLRAUM</th>\n",
              "      <th>CAMEO_DEU_2015</th>\n",
              "      <th>CAMEO_DEUG_2015</th>\n",
              "      <th>CAMEO_INTL_2015</th>\n",
              "      <th>CJT_GESAMTTYP</th>\n",
              "      <th>CJT_KATALOGNUTZER</th>\n",
              "      <th>CJT_TYP_1</th>\n",
              "      <th>CJT_TYP_2</th>\n",
              "      <th>CJT_TYP_3</th>\n",
              "      <th>CJT_TYP_4</th>\n",
              "      <th>CJT_TYP_5</th>\n",
              "      <th>CJT_TYP_6</th>\n",
              "      <th>D19_BANKEN_ANZ_12</th>\n",
              "      <th>D19_BANKEN_ANZ_24</th>\n",
              "      <th>D19_BANKEN_DATUM</th>\n",
              "      <th>D19_BANKEN_DIREKT</th>\n",
              "      <th>D19_BANKEN_GROSS</th>\n",
              "      <th>D19_BANKEN_LOKAL</th>\n",
              "      <th>D19_BANKEN_OFFLINE_DATUM</th>\n",
              "      <th>D19_BANKEN_ONLINE_DATUM</th>\n",
              "      <th>D19_BANKEN_ONLINE_QUOTE_12</th>\n",
              "      <th>D19_BANKEN_REST</th>\n",
              "      <th>D19_BEKLEIDUNG_GEH</th>\n",
              "      <th>D19_BEKLEIDUNG_REST</th>\n",
              "      <th>...</th>\n",
              "      <th>REGIOTYP</th>\n",
              "      <th>RELAT_AB</th>\n",
              "      <th>RETOURTYP_BK_S</th>\n",
              "      <th>RT_KEIN_ANREIZ</th>\n",
              "      <th>RT_SCHNAEPPCHEN</th>\n",
              "      <th>RT_UEBERGROESSE</th>\n",
              "      <th>SEMIO_DOM</th>\n",
              "      <th>SEMIO_ERL</th>\n",
              "      <th>SEMIO_FAM</th>\n",
              "      <th>SEMIO_KAEM</th>\n",
              "      <th>SEMIO_KRIT</th>\n",
              "      <th>SEMIO_KULT</th>\n",
              "      <th>SEMIO_LUST</th>\n",
              "      <th>SEMIO_MAT</th>\n",
              "      <th>SEMIO_PFLICHT</th>\n",
              "      <th>SEMIO_RAT</th>\n",
              "      <th>SEMIO_REL</th>\n",
              "      <th>SEMIO_SOZ</th>\n",
              "      <th>SEMIO_TRADV</th>\n",
              "      <th>SEMIO_VERT</th>\n",
              "      <th>SHOPPER_TYP</th>\n",
              "      <th>SOHO_KZ</th>\n",
              "      <th>STRUKTURTYP</th>\n",
              "      <th>TITEL_KZ</th>\n",
              "      <th>UMFELD_ALT</th>\n",
              "      <th>UMFELD_JUNG</th>\n",
              "      <th>UNGLEICHENN_FLAG</th>\n",
              "      <th>VERDICHTUNGSRAUM</th>\n",
              "      <th>VERS_TYP</th>\n",
              "      <th>VHA</th>\n",
              "      <th>VHN</th>\n",
              "      <th>VK_DHT4A</th>\n",
              "      <th>VK_DISTANZ</th>\n",
              "      <th>VK_ZG11</th>\n",
              "      <th>W_KEIT_KIND_HH</th>\n",
              "      <th>WOHNDAUER_2008</th>\n",
              "      <th>WOHNLAGE</th>\n",
              "      <th>ZABEOTYP</th>\n",
              "      <th>ANREDE_KZ</th>\n",
              "      <th>ALTERSKATEGORIE_GROB</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>9372</th>\n",
              "      <td>771330</td>\n",
              "      <td>NaN</td>\n",
              "      <td>7.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>20.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>6B</td>\n",
              "      <td>6.0</td>\n",
              "      <td>43.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>9</td>\n",
              "      <td>6.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>9</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>...</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>7</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>7</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>7</td>\n",
              "      <td>5</td>\n",
              "      <td>7</td>\n",
              "      <td>7</td>\n",
              "      <td>5</td>\n",
              "      <td>6</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>7.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>150835</th>\n",
              "      <td>482166</td>\n",
              "      <td>NaN</td>\n",
              "      <td>9.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>7.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3D</td>\n",
              "      <td>3.0</td>\n",
              "      <td>25.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>7</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>6</td>\n",
              "      <td>4</td>\n",
              "      <td>7</td>\n",
              "      <td>3.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>8.0</td>\n",
              "      <td>11.0</td>\n",
              "      <td>11.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>92034</th>\n",
              "      <td>364325</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>12.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>10.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>8B</td>\n",
              "      <td>8.0</td>\n",
              "      <td>41.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6</td>\n",
              "      <td>3</td>\n",
              "      <td>6</td>\n",
              "      <td>6</td>\n",
              "      <td>7</td>\n",
              "      <td>2</td>\n",
              "      <td>7</td>\n",
              "      <td>3</td>\n",
              "      <td>5</td>\n",
              "      <td>4</td>\n",
              "      <td>7</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>12.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>254110</th>\n",
              "      <td>525434</td>\n",
              "      <td>NaN</td>\n",
              "      <td>9.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>6B</td>\n",
              "      <td>6.0</td>\n",
              "      <td>43.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>...</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>7</td>\n",
              "      <td>6</td>\n",
              "      <td>3</td>\n",
              "      <td>5</td>\n",
              "      <td>6</td>\n",
              "      <td>3</td>\n",
              "      <td>6</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>2.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>11.0</td>\n",
              "      <td>10.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>179919</th>\n",
              "      <td>701438</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.0</td>\n",
              "      <td>20.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>20.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>2B</td>\n",
              "      <td>2.0</td>\n",
              "      <td>13.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5</td>\n",
              "      <td>3.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>7.0</td>\n",
              "      <td>8</td>\n",
              "      <td>5</td>\n",
              "      <td>10.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>...</td>\n",
              "      <td>4.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>7</td>\n",
              "      <td>6</td>\n",
              "      <td>1</td>\n",
              "      <td>5</td>\n",
              "      <td>7</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>2.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>228622</th>\n",
              "      <td>600002</td>\n",
              "      <td>NaN</td>\n",
              "      <td>9.0</td>\n",
              "      <td>18.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>18.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1D</td>\n",
              "      <td>1.0</td>\n",
              "      <td>15.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>7</td>\n",
              "      <td>5</td>\n",
              "      <td>2</td>\n",
              "      <td>7</td>\n",
              "      <td>7</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>5</td>\n",
              "      <td>6</td>\n",
              "      <td>4</td>\n",
              "      <td>5</td>\n",
              "      <td>7</td>\n",
              "      <td>4</td>\n",
              "      <td>3.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.0</td>\n",
              "      <td>7.0</td>\n",
              "      <td>7.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>7.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>277041</th>\n",
              "      <td>1054086</td>\n",
              "      <td>NaN</td>\n",
              "      <td>9.0</td>\n",
              "      <td>15.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>15.0</td>\n",
              "      <td>14.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>16.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5A</td>\n",
              "      <td>5.0</td>\n",
              "      <td>31.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>...</td>\n",
              "      <td>7.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>6</td>\n",
              "      <td>6</td>\n",
              "      <td>1</td>\n",
              "      <td>7</td>\n",
              "      <td>7</td>\n",
              "      <td>1</td>\n",
              "      <td>6</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>5</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>2.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>11.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>6</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>7 rows × 366 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "            LNR  AGER_TYP  ...  ANREDE_KZ  ALTERSKATEGORIE_GROB\n",
              "9372     771330       NaN  ...          1                     1\n",
              "150835   482166       NaN  ...          1                     4\n",
              "92034    364325       NaN  ...          2                     3\n",
              "254110   525434       NaN  ...          2                     3\n",
              "179919   701438       NaN  ...          2                     3\n",
              "228622   600002       NaN  ...          2                     2\n",
              "277041  1054086       NaN  ...          2                     3\n",
              "\n",
              "[7 rows x 366 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MxqF--BtKQFg",
        "outputId": "af89adf6-c01a-46b0-8713-788232ab638f"
      },
      "source": [
        "azdias.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 300000 entries, 0 to 299999\n",
            "Columns: 366 entries, LNR to ALTERSKATEGORIE_GROB\n",
            "dtypes: datetime64[ns](1), float64(320), int64(42), object(3)\n",
            "memory usage: 837.7+ MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gcoK-sQDc7fV",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 315
        },
        "outputId": "59f6874b-5b56-422b-fa69-1431622ef142"
      },
      "source": [
        "custrs.sample(7)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>LNR</th>\n",
              "      <th>AGER_TYP</th>\n",
              "      <th>AKT_DAT_KL</th>\n",
              "      <th>ALTER_HH</th>\n",
              "      <th>ALTER_KIND1</th>\n",
              "      <th>ALTER_KIND2</th>\n",
              "      <th>ALTER_KIND3</th>\n",
              "      <th>ALTER_KIND4</th>\n",
              "      <th>ALTERSKATEGORIE_FEIN</th>\n",
              "      <th>ANZ_HAUSHALTE_AKTIV</th>\n",
              "      <th>ANZ_HH_TITEL</th>\n",
              "      <th>ANZ_KINDER</th>\n",
              "      <th>ANZ_PERSONEN</th>\n",
              "      <th>ANZ_STATISTISCHE_HAUSHALTE</th>\n",
              "      <th>ANZ_TITEL</th>\n",
              "      <th>ARBEIT</th>\n",
              "      <th>BALLRAUM</th>\n",
              "      <th>CAMEO_DEU_2015</th>\n",
              "      <th>CAMEO_DEUG_2015</th>\n",
              "      <th>CAMEO_INTL_2015</th>\n",
              "      <th>CJT_GESAMTTYP</th>\n",
              "      <th>CJT_KATALOGNUTZER</th>\n",
              "      <th>CJT_TYP_1</th>\n",
              "      <th>CJT_TYP_2</th>\n",
              "      <th>CJT_TYP_3</th>\n",
              "      <th>CJT_TYP_4</th>\n",
              "      <th>CJT_TYP_5</th>\n",
              "      <th>CJT_TYP_6</th>\n",
              "      <th>D19_BANKEN_ANZ_12</th>\n",
              "      <th>D19_BANKEN_ANZ_24</th>\n",
              "      <th>D19_BANKEN_DATUM</th>\n",
              "      <th>D19_BANKEN_DIREKT</th>\n",
              "      <th>D19_BANKEN_GROSS</th>\n",
              "      <th>D19_BANKEN_LOKAL</th>\n",
              "      <th>D19_BANKEN_OFFLINE_DATUM</th>\n",
              "      <th>D19_BANKEN_ONLINE_DATUM</th>\n",
              "      <th>D19_BANKEN_ONLINE_QUOTE_12</th>\n",
              "      <th>D19_BANKEN_REST</th>\n",
              "      <th>D19_BEKLEIDUNG_GEH</th>\n",
              "      <th>D19_BEKLEIDUNG_REST</th>\n",
              "      <th>...</th>\n",
              "      <th>RT_KEIN_ANREIZ</th>\n",
              "      <th>RT_SCHNAEPPCHEN</th>\n",
              "      <th>RT_UEBERGROESSE</th>\n",
              "      <th>SEMIO_DOM</th>\n",
              "      <th>SEMIO_ERL</th>\n",
              "      <th>SEMIO_FAM</th>\n",
              "      <th>SEMIO_KAEM</th>\n",
              "      <th>SEMIO_KRIT</th>\n",
              "      <th>SEMIO_KULT</th>\n",
              "      <th>SEMIO_LUST</th>\n",
              "      <th>SEMIO_MAT</th>\n",
              "      <th>SEMIO_PFLICHT</th>\n",
              "      <th>SEMIO_RAT</th>\n",
              "      <th>SEMIO_REL</th>\n",
              "      <th>SEMIO_SOZ</th>\n",
              "      <th>SEMIO_TRADV</th>\n",
              "      <th>SEMIO_VERT</th>\n",
              "      <th>SHOPPER_TYP</th>\n",
              "      <th>SOHO_KZ</th>\n",
              "      <th>STRUKTURTYP</th>\n",
              "      <th>TITEL_KZ</th>\n",
              "      <th>UMFELD_ALT</th>\n",
              "      <th>UMFELD_JUNG</th>\n",
              "      <th>UNGLEICHENN_FLAG</th>\n",
              "      <th>VERDICHTUNGSRAUM</th>\n",
              "      <th>VERS_TYP</th>\n",
              "      <th>VHA</th>\n",
              "      <th>VHN</th>\n",
              "      <th>VK_DHT4A</th>\n",
              "      <th>VK_DISTANZ</th>\n",
              "      <th>VK_ZG11</th>\n",
              "      <th>W_KEIT_KIND_HH</th>\n",
              "      <th>WOHNDAUER_2008</th>\n",
              "      <th>WOHNLAGE</th>\n",
              "      <th>ZABEOTYP</th>\n",
              "      <th>PRODUCT_GROUP</th>\n",
              "      <th>CUSTOMER_GROUP</th>\n",
              "      <th>ONLINE_PURCHASE</th>\n",
              "      <th>ANREDE_KZ</th>\n",
              "      <th>ALTERSKATEGORIE_GROB</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>136940</th>\n",
              "      <td>728</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>...</td>\n",
              "      <td>4.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6</td>\n",
              "      <td>3</td>\n",
              "      <td>6</td>\n",
              "      <td>6</td>\n",
              "      <td>7</td>\n",
              "      <td>3</td>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>4</td>\n",
              "      <td>7</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3</td>\n",
              "      <td>COSMETIC_AND_FOOD</td>\n",
              "      <td>MULTI_BUYER</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75636</th>\n",
              "      <td>157987</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>17.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>4A</td>\n",
              "      <td>4.0</td>\n",
              "      <td>22.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5</td>\n",
              "      <td>3.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>7.0</td>\n",
              "      <td>...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>5</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>7</td>\n",
              "      <td>1</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>6</td>\n",
              "      <td>4</td>\n",
              "      <td>7</td>\n",
              "      <td>3.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>7.0</td>\n",
              "      <td>1</td>\n",
              "      <td>COSMETIC</td>\n",
              "      <td>MULTI_BUYER</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75953</th>\n",
              "      <td>3467</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>...</td>\n",
              "      <td>4.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6</td>\n",
              "      <td>3</td>\n",
              "      <td>6</td>\n",
              "      <td>6</td>\n",
              "      <td>7</td>\n",
              "      <td>3</td>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>4</td>\n",
              "      <td>7</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3</td>\n",
              "      <td>COSMETIC_AND_FOOD</td>\n",
              "      <td>MULTI_BUYER</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>54865</th>\n",
              "      <td>2272</td>\n",
              "      <td>2.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>16.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>11.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>2B</td>\n",
              "      <td>2.0</td>\n",
              "      <td>13.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>6</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>7</td>\n",
              "      <td>6</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>6</td>\n",
              "      <td>4</td>\n",
              "      <td>6</td>\n",
              "      <td>3.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>7.0</td>\n",
              "      <td>1</td>\n",
              "      <td>COSMETIC</td>\n",
              "      <td>MULTI_BUYER</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>105606</th>\n",
              "      <td>36931</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>8.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>1A</td>\n",
              "      <td>1.0</td>\n",
              "      <td>13.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>7</td>\n",
              "      <td>6</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>5</td>\n",
              "      <td>6</td>\n",
              "      <td>4</td>\n",
              "      <td>6</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>25.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1</td>\n",
              "      <td>COSMETIC</td>\n",
              "      <td>SINGLE_BUYER</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>98948</th>\n",
              "      <td>109437</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>18.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>12.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>2C</td>\n",
              "      <td>2.0</td>\n",
              "      <td>14.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>9</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>7.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2.0</td>\n",
              "      <td>...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>7</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>7</td>\n",
              "      <td>4</td>\n",
              "      <td>7</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>6</td>\n",
              "      <td>7</td>\n",
              "      <td>5</td>\n",
              "      <td>7</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>2</td>\n",
              "      <td>FOOD</td>\n",
              "      <td>MULTI_BUYER</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>142772</th>\n",
              "      <td>11872</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>12.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>4A</td>\n",
              "      <td>4.0</td>\n",
              "      <td>22.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>...</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>4</td>\n",
              "      <td>7</td>\n",
              "      <td>6</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>6</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>7.0</td>\n",
              "      <td>1</td>\n",
              "      <td>COSMETIC_AND_FOOD</td>\n",
              "      <td>MULTI_BUYER</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>7 rows × 369 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "           LNR  AGER_TYP  ...  ANREDE_KZ  ALTERSKATEGORIE_GROB\n",
              "136940     728       NaN  ...          2                     1\n",
              "75636   157987       NaN  ...          1                     4\n",
              "75953     3467       NaN  ...          1                     2\n",
              "54865     2272       2.0  ...          1                     4\n",
              "105606   36931       1.0  ...          1                     4\n",
              "98948   109437       3.0  ...          1                     2\n",
              "142772   11872       NaN  ...          1                     4\n",
              "\n",
              "[7 rows x 369 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dKI90x7crBOI",
        "outputId": "a6715920-182f-42eb-afb3-9a6dc1b81373"
      },
      "source": [
        "custrs.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 150000 entries, 0 to 149999\n",
            "Columns: 369 entries, LNR to ALTERSKATEGORIE_GROB\n",
            "dtypes: datetime64[ns](1), float64(321), int64(42), object(5)\n",
            "memory usage: 422.3+ MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bD8MlEsyPuEh"
      },
      "source": [
        "So, in `custrs` dataset the three extra columns (`'CUSTOMER_GROUP', 'ONLINE_PURCHASE', and 'PRODUCT_GROUP'`) are imported as one `float64` and two `object` dtypes, comparing to `azdias` columns."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "CwoMvCuVVIIA",
        "outputId": "c7e0befa-f08b-4df7-f63c-68ad40095dbe"
      },
      "source": [
        "azd_obj_cols = azdias.select_dtypes(include=object)\n",
        "\n",
        "azd_obj_cols.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>CAMEO_DEU_2015</th>\n",
              "      <th>D19_LETZTER_KAUF_BRANCHE</th>\n",
              "      <th>OST_WEST_KZ</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>6B</td>\n",
              "      <td>D19_VERSAND_REST</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>7B</td>\n",
              "      <td>D19_UNBEKANNT</td>\n",
              "      <td>W</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>5B</td>\n",
              "      <td>D19_UNBEKANNT</td>\n",
              "      <td>W</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3C</td>\n",
              "      <td>D19_BUCH_CD</td>\n",
              "      <td>W</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1B</td>\n",
              "      <td>D19_VERSAND_REST</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  CAMEO_DEU_2015 D19_LETZTER_KAUF_BRANCHE OST_WEST_KZ\n",
              "0             6B         D19_VERSAND_REST           O\n",
              "1             7B            D19_UNBEKANNT           W\n",
              "2             5B            D19_UNBEKANNT           W\n",
              "3             3C              D19_BUCH_CD           W\n",
              "4             1B         D19_VERSAND_REST           O"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "DwscKVZiQTDV",
        "outputId": "f755d9bd-7a85-4db1-d157-2fcdf66f7084"
      },
      "source": [
        "cus_obj_cols = custrs.select_dtypes(include=object)\n",
        "\n",
        "cus_obj_cols.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>CAMEO_DEU_2015</th>\n",
              "      <th>D19_LETZTER_KAUF_BRANCHE</th>\n",
              "      <th>OST_WEST_KZ</th>\n",
              "      <th>PRODUCT_GROUP</th>\n",
              "      <th>CUSTOMER_GROUP</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1A</td>\n",
              "      <td>D19_UNBEKANNT</td>\n",
              "      <td>W</td>\n",
              "      <td>COSMETIC_AND_FOOD</td>\n",
              "      <td>MULTI_BUYER</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>NaN</td>\n",
              "      <td>D19_BANKEN_GROSS</td>\n",
              "      <td>NaN</td>\n",
              "      <td>FOOD</td>\n",
              "      <td>SINGLE_BUYER</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>8A</td>\n",
              "      <td>D19_TELKO_REST</td>\n",
              "      <td>W</td>\n",
              "      <td>COSMETIC</td>\n",
              "      <td>MULTI_BUYER</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>COSMETIC_AND_FOOD</td>\n",
              "      <td>MULTI_BUYER</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>7B</td>\n",
              "      <td>D19_SCHUHE</td>\n",
              "      <td>W</td>\n",
              "      <td>FOOD</td>\n",
              "      <td>MULTI_BUYER</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  CAMEO_DEU_2015 D19_LETZTER_KAUF_BRANCHE  ...      PRODUCT_GROUP CUSTOMER_GROUP\n",
              "0             1A            D19_UNBEKANNT  ...  COSMETIC_AND_FOOD    MULTI_BUYER\n",
              "1            NaN         D19_BANKEN_GROSS  ...               FOOD   SINGLE_BUYER\n",
              "2             8A           D19_TELKO_REST  ...           COSMETIC    MULTI_BUYER\n",
              "3            NaN                      NaN  ...  COSMETIC_AND_FOOD    MULTI_BUYER\n",
              "4             7B               D19_SCHUHE  ...               FOOD    MULTI_BUYER\n",
              "\n",
              "[5 rows x 5 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Amx6vfU944fa"
      },
      "source": [
        "Convert the remaining **object** columns (`'CAMEO_DEU_2015', 'D19_LETZTER_KAUF_BRANCHE', 'OST_WEST_KZ', etc.`) to **categorical**:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "42h5zoCi44fa",
        "outputId": "e3b36977-6d3a-47ae-bcd3-05690e39f799"
      },
      "source": [
        "# Get the names of the columns to convert\n",
        "azd_cat_cols = azd_obj_cols.columns.tolist()\n",
        "cus_cat_cols = cus_obj_cols.columns.tolist()\n",
        "# Convert all three columns using astype method\n",
        "azdias[azd_cat_cols] = azdias[azd_cat_cols].astype('category')\n",
        "custrs[cus_cat_cols] = custrs[cus_cat_cols].astype('category')\n",
        "\n",
        "azdias[azd_cat_cols].dtypes"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "CAMEO_DEU_2015              category\n",
              "D19_LETZTER_KAUF_BRANCHE    category\n",
              "OST_WEST_KZ                 category\n",
              "dtype: object"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dAlDDT59WC37"
      },
      "source": [
        "## Explore attributes in the dataset(s) and excel files\n",
        "\n",
        "### 1. Compare attributes between the two excel files\n",
        "\n",
        "Read in the excel files with attributes related information"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 266
        },
        "id": "hzcj2Z9U4qQl",
        "outputId": "9f3694f6-b916-4d1c-d293-890d4305c403"
      },
      "source": [
        "# Read in the mapping of data values for each feature in the DIAS dataset.\n",
        "attr_cols = range(1,5)  # skip the first column as it is empty\n",
        "dfdias_attrib = pd.read_excel(dias_attrib, header=1, usecols=attr_cols)\n",
        "dfdias_attrib.head(7)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Attribute</th>\n",
              "      <th>Description</th>\n",
              "      <th>Value</th>\n",
              "      <th>Meaning</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>AGER_TYP</td>\n",
              "      <td>best-ager typology</td>\n",
              "      <td>-1</td>\n",
              "      <td>unknown</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>no classification possible</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1</td>\n",
              "      <td>passive elderly</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2</td>\n",
              "      <td>cultural elderly</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3</td>\n",
              "      <td>experience-driven elderly</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>ALTERSKATEGORIE_GROB</td>\n",
              "      <td>age classification through prename analysis</td>\n",
              "      <td>-1, 0</td>\n",
              "      <td>unknown</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1</td>\n",
              "      <td>&lt; 30 years</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "              Attribute  ...                     Meaning\n",
              "0              AGER_TYP  ...                     unknown\n",
              "1                   NaN  ...  no classification possible\n",
              "2                   NaN  ...             passive elderly\n",
              "3                   NaN  ...            cultural elderly\n",
              "4                   NaN  ...   experience-driven elderly\n",
              "5  ALTERSKATEGORIE_GROB  ...                     unknown\n",
              "6                   NaN  ...                  < 30 years\n",
              "\n",
              "[7 rows x 4 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 266
        },
        "id": "Pu5PJGsSR3Qn",
        "outputId": "46a40032-718f-42ae-c5bf-636e70f3488a"
      },
      "source": [
        "# DIAS Information Levels - Attributes 2017.xlsx\n",
        "dfdias_info = pd.read_excel(dias_info, header=1, usecols=attr_cols)\n",
        "dfdias_info.head(7)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Information level</th>\n",
              "      <th>Attribute</th>\n",
              "      <th>Description</th>\n",
              "      <th>Additional notes</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>NaN</td>\n",
              "      <td>AGER_TYP</td>\n",
              "      <td>best-ager typology</td>\n",
              "      <td>in cooperation with Kantar TNS; the informatio...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Person</td>\n",
              "      <td>ALTERSKATEGORIE_GROB</td>\n",
              "      <td>age through prename analysis</td>\n",
              "      <td>modelled on millions of first name-age-referen...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>NaN</td>\n",
              "      <td>ANREDE_KZ</td>\n",
              "      <td>gender</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>NaN</td>\n",
              "      <td>CJT_GESAMTTYP</td>\n",
              "      <td>Customer-Journey-Typology relating to the pref...</td>\n",
              "      <td>relating to the preferred information, marketi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>NaN</td>\n",
              "      <td>FINANZ_MINIMALIST</td>\n",
              "      <td>financial typology: low financial interest</td>\n",
              "      <td>Gfk-Typology based on a representative househo...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>NaN</td>\n",
              "      <td>FINANZ_SPARER</td>\n",
              "      <td>financial typology: money saver</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>NaN</td>\n",
              "      <td>FINANZ_VORSORGER</td>\n",
              "      <td>financial typology: be prepared</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  Information level  ...                                   Additional notes\n",
              "0               NaN  ...  in cooperation with Kantar TNS; the informatio...\n",
              "1            Person  ...  modelled on millions of first name-age-referen...\n",
              "2               NaN  ...                                                NaN\n",
              "3               NaN  ...  relating to the preferred information, marketi...\n",
              "4               NaN  ...  Gfk-Typology based on a representative househo...\n",
              "5               NaN  ...                                                NaN\n",
              "6               NaN  ...                                                NaN\n",
              "\n",
              "[7 rows x 4 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dl6bzueqWL3m",
        "outputId": "f4f01d7f-9c54-41bc-eadc-3321f4a0aea1"
      },
      "source": [
        "# Get attributes' names from the Attribute column in the two excel files\n",
        "\n",
        "# `DIAS Information Levels - Attributes 2017.xlsx`\n",
        "dfdias_info_attrs = dfdias_info.Attribute.value_counts().index.values\n",
        "\n",
        "# `DIAS Attributes - Values 2017.xlsx`\n",
        "attrbf_names = dfdias_attrib.Attribute.value_counts().index.values\n",
        "\n",
        "# Get the attribute names from `azdias` df columns:\n",
        "azd_attr_names = azdias.columns.values\n",
        "\n",
        "# Get the attribute names from `custrs` df columns:\n",
        "cus_attr_names = custrs.columns.values\n",
        "\n",
        "dfdias_info_attrs.size, attrbf_names.size, azd_attr_names.size, cus_attr_names.size"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(313, 314, 366, 369)"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "y4kljTWMESiB",
        "outputId": "404e02c1-3097-418b-d61b-c02304def734"
      },
      "source": [
        "# Get the set intersection of the two arrays:\n",
        "# We are comparing names as they were originally written in the files.\n",
        "xcl_attr_intersc = np.intersect1d(attrbf_names, dfdias_info_attrs)\n",
        "\n",
        "xcl_attr_intersc.size"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "300"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SIEjV7gVGhoR",
        "outputId": "928c7057-61d8-4fae-c56f-745c08e23cee"
      },
      "source": [
        "# Attributes values excel file contains some different attributes compared to information excel file:\n",
        "xcl_attr_diff_vals_info = np.setdiff1d(attrbf_names, dfdias_info_attrs)\n",
        "\n",
        "xcl_attr_diff_vals_info, xcl_attr_diff_vals_info.size"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array(['BIP_FLAG', 'D19_BANKEN_ANZ_12', 'D19_BANKEN_ANZ_24',\n",
              "        'D19_GESAMT_ANZ_12', 'D19_GESAMT_ANZ_24', 'D19_LOTTO_RZ',\n",
              "        'D19_TELKO_ANZ_12', 'D19_TELKO_ANZ_24', 'D19_VERSAND_ANZ_12',\n",
              "        'D19_VERSAND_ANZ_24', 'D19_VERSI_ANZ_12', 'D19_VERSI_ANZ_24',\n",
              "        'KBA13_CCM_3000', 'KBA13_CCM_3001'], dtype=object), 14)"
            ]
          },
          "metadata": {},
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xKErHY9pHtNh",
        "outputId": "bf61ec09-ca43-46bd-e997-7f10335f4d33"
      },
      "source": [
        "# Information excel file contains some different attributes compared to attributes excel file:\n",
        "xcl_attr_diff_info_vals = np.setdiff1d(dfdias_info_attrs, attrbf_names)\n",
        "\n",
        "xcl_attr_diff_info_vals, xcl_attr_diff_info_vals.size"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array(['ARBEIT', 'D19_BANKEN_ ANZ_12             D19_BANKEN_ ANZ_24',\n",
              "        'D19_GESAMT_ANZ_12                                    D19_GESAMT_ANZ_24',\n",
              "        'D19_TELKO_ ANZ_12                  D19_TELKO_ ANZ_24',\n",
              "        'D19_VERSAND_ ANZ_12          D19_VERSAND_ ANZ_24',\n",
              "        'D19_VERSI_ ANZ_12                                       D19_VERSI_ ANZ_24',\n",
              "        'D19_VERSI_DATUM', 'D19_VERSI_OFFLINE_DATUM',\n",
              "        'D19_VERSI_ONLINE_DATUM', 'EINWOHNER', 'GKZ', 'PLZ', 'PLZ8'],\n",
              "       dtype=object), 13)"
            ]
          },
          "metadata": {},
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fpy0m26gLLzw"
      },
      "source": [
        "Taking the symmetric difference of the attributes in excel files, we can observe that there are more common attributes, but we must perform some string operations to bring them in similar forms for comparison."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ulbAe_CfJoRs",
        "outputId": "c83482fb-7b91-44a5-bb58-bd3c1cca234b"
      },
      "source": [
        "# Symmetric difference of the two attributes' sets.\n",
        "xcl_attrs_symtrc_diff = np.union1d(xcl_attr_diff_vals_info, xcl_attr_diff_info_vals)\n",
        "\n",
        "xcl_attrs_symtrc_diff, xcl_attrs_symtrc_diff.size"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array(['ARBEIT', 'BIP_FLAG',\n",
              "        'D19_BANKEN_ ANZ_12             D19_BANKEN_ ANZ_24',\n",
              "        'D19_BANKEN_ANZ_12', 'D19_BANKEN_ANZ_24', 'D19_GESAMT_ANZ_12',\n",
              "        'D19_GESAMT_ANZ_12                                    D19_GESAMT_ANZ_24',\n",
              "        'D19_GESAMT_ANZ_24', 'D19_LOTTO_RZ',\n",
              "        'D19_TELKO_ ANZ_12                  D19_TELKO_ ANZ_24',\n",
              "        'D19_TELKO_ANZ_12', 'D19_TELKO_ANZ_24',\n",
              "        'D19_VERSAND_ ANZ_12          D19_VERSAND_ ANZ_24',\n",
              "        'D19_VERSAND_ANZ_12', 'D19_VERSAND_ANZ_24',\n",
              "        'D19_VERSI_ ANZ_12                                       D19_VERSI_ ANZ_24',\n",
              "        'D19_VERSI_ANZ_12', 'D19_VERSI_ANZ_24', 'D19_VERSI_DATUM',\n",
              "        'D19_VERSI_OFFLINE_DATUM', 'D19_VERSI_ONLINE_DATUM', 'EINWOHNER',\n",
              "        'GKZ', 'KBA13_CCM_3000', 'KBA13_CCM_3001', 'PLZ', 'PLZ8'],\n",
              "       dtype=object), 27)"
            ]
          },
          "metadata": {},
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z3_w1vf4PPlc"
      },
      "source": [
        "We will fix the names of the attributes that seem different in the `DIAS Information Levels - Attributes 2017.xlsx` (or info) file.\n",
        "\n",
        "* Parse lines containing more than one attribute"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d5uYN0XWyDF2",
        "outputId": "ebe9fa29-93cf-4b2b-a487-78e4dd0b3463"
      },
      "source": [
        "# We will create a new list splitting the lines containing more than one attribute and stripping white space:\n",
        "xcl_attr_diff_info_vals_fxd = [i.strip() for x in xcl_attr_diff_info_vals for i in x.split(\"  \", maxsplit=1)]\n",
        "\n",
        "xcl_attr_diff_info_vals_fxd, len(xcl_attr_diff_info_vals_fxd)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(['ARBEIT',\n",
              "  'D19_BANKEN_ ANZ_12',\n",
              "  'D19_BANKEN_ ANZ_24',\n",
              "  'D19_GESAMT_ANZ_12',\n",
              "  'D19_GESAMT_ANZ_24',\n",
              "  'D19_TELKO_ ANZ_12',\n",
              "  'D19_TELKO_ ANZ_24',\n",
              "  'D19_VERSAND_ ANZ_12',\n",
              "  'D19_VERSAND_ ANZ_24',\n",
              "  'D19_VERSI_ ANZ_12',\n",
              "  'D19_VERSI_ ANZ_24',\n",
              "  'D19_VERSI_DATUM',\n",
              "  'D19_VERSI_OFFLINE_DATUM',\n",
              "  'D19_VERSI_ONLINE_DATUM',\n",
              "  'EINWOHNER',\n",
              "  'GKZ',\n",
              "  'PLZ',\n",
              "  'PLZ8'],\n",
              " 18)"
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kBNy939it5JY"
      },
      "source": [
        "* Expand the initial attributes list to include the newly identified attributes from above."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aJPb5TeJz5um",
        "outputId": "a2bcf0d9-7f33-4c14-b904-4d33f01630ab"
      },
      "source": [
        "# Our initial info attributes' list needs to be expanded to compensate for elements that were packed together.\n",
        "dfdias_info_attrs_new = []\n",
        "# Create a deep copy of the clean names list so it may be reused.\n",
        "attrs_to_append = [i for i in xcl_attr_diff_info_vals_fxd]\n",
        "# For every element in the imported attributes from information file check against the \n",
        "# fixed attribute names list and create a new list including fixed names.\n",
        "for i in dfdias_info_attrs:\n",
        "    if i in xcl_attr_diff_info_vals and len(attrs_to_append) > 0:\n",
        "        for idx, x in enumerate(attrs_to_append):\n",
        "            if i.startswith(x):\n",
        "                if len(i) > len(x):\n",
        "                    dfdias_info_attrs_new.append(''.join(x.split()))\n",
        "                    dfdias_info_attrs_new.append(''.join(attrs_to_append[idx + 1].split()))\n",
        "                    del attrs_to_append[idx + 1]\n",
        "                    del attrs_to_append[idx]\n",
        "                else:\n",
        "                    dfdias_info_attrs_new.append(x)\n",
        "                    del attrs_to_append[idx]\n",
        "                break\n",
        "    else:\n",
        "        # just copy the rest of the elements.\n",
        "        dfdias_info_attrs_new.append(i)\n",
        "\n",
        "# Convert to numpy array\n",
        "dfdias_info_attrs_new = np.array(dfdias_info_attrs_new)\n",
        "\n",
        "dfdias_info_attrs_new[60:80], dfdias_info_attrs_new.size"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array(['D19_BEKLEIDUNG_GEH_RZ', 'PLZ8_ANTG3', 'KBA05_SEG3', 'KBA05_MOD1',\n",
              "        'D19_REISEN_RZ', 'KBA13_HERST_FORD_OPEL', 'KBA05_CCM3',\n",
              "        'KBA05_KRSKLEIN', 'FINANZTYP', 'CAMEO_DEU_2015',\n",
              "        'D19_VERSICHERUNGEN_RZ', 'PLZ8_ANTG1', 'RETOURTYP_BK_S',\n",
              "        'KBA13_CCM_0_1400', 'KBA05_ANTG2', 'KBA13_HALTER_66',\n",
              "        'FINANZ_ANLEGER', 'KBA13_SEG_OBERKLASSE', 'EWDICHTE',\n",
              "        'FINANZ_MINIMALIST'], dtype='<U27'), 318)"
            ]
          },
          "metadata": {},
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JBefRNxWS3tp"
      },
      "source": [
        "Now that we have fixed the names we can compare the attributes between the two attributes' information files.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jpVLmLZBah1Y",
        "outputId": "8858d52f-b1ee-43e5-96bd-4636fe4c1aaf"
      },
      "source": [
        "# Get the set intersection of the two arrays:\n",
        "xcl_attr_intersc_fxd = np.intersect1d(attrbf_names, dfdias_info_attrs_new)\n",
        "\n",
        "xcl_attr_intersc_fxd.size"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "310"
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rL9-ERNARBTF"
      },
      "source": [
        "Common attributes between the two files increased to 310. After this different attributes in each file become as follows:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "P_7jBzzleywY",
        "outputId": "1757d6b3-cf33-48f6-97e2-01f47fc53ce4"
      },
      "source": [
        "# DIAS Attributes - Values 2017.xlsx unique values:\n",
        "xcl_attr_diff_vals_info = np.setdiff1d(attrbf_names, dfdias_info_attrs_new)\n",
        "\n",
        "xcl_attr_diff_vals_info, xcl_attr_diff_vals_info.size"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array(['BIP_FLAG', 'D19_LOTTO_RZ', 'KBA13_CCM_3000', 'KBA13_CCM_3001'],\n",
              "       dtype=object), 4)"
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3SmwEWXqc8Eo",
        "outputId": "65f0d437-e4ef-400e-e22a-c2888ec527f7"
      },
      "source": [
        "# DIAS Information Levels - Attributes 2017.xlsx contains some different attributes compared to attributes excel file:\n",
        "xcl_attr_diff_info_vals = np.setdiff1d(dfdias_info_attrs_new, attrbf_names)\n",
        "\n",
        "xcl_attr_diff_info_vals, xcl_attr_diff_info_vals.size"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array(['ARBEIT', 'D19_VERSI_DATUM', 'D19_VERSI_OFFLINE_DATUM',\n",
              "        'D19_VERSI_ONLINE_DATUM', 'EINWOHNER', 'GKZ', 'PLZ', 'PLZ8'],\n",
              "       dtype='<U27'), 8)"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3iABVdP2TYxs",
        "outputId": "b7f94e2e-0fd9-4d4c-eed4-a6882c5bdef9"
      },
      "source": [
        "# Symmetric difference of the two attributes' sets.\n",
        "xcl_attrs_symtrc_diff = np.union1d(xcl_attr_diff_vals_info, xcl_attr_diff_info_vals)\n",
        "\n",
        "xcl_attrs_symtrc_diff, xcl_attrs_symtrc_diff.size"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array(['ARBEIT', 'BIP_FLAG', 'D19_LOTTO_RZ', 'D19_VERSI_DATUM',\n",
              "        'D19_VERSI_OFFLINE_DATUM', 'D19_VERSI_ONLINE_DATUM', 'EINWOHNER',\n",
              "        'GKZ', 'KBA13_CCM_3000', 'KBA13_CCM_3001', 'PLZ', 'PLZ8'],\n",
              "       dtype=object), 12)"
            ]
          },
          "metadata": {},
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KPiyGR2bWjCH",
        "outputId": "75fee581-e38c-47f3-edc4-1574fdcf636c"
      },
      "source": [
        "# Take the union for the total of unique values in both files together.\n",
        "xcl_attrs_unique = np.union1d(attrbf_names, dfdias_info_attrs_new)\n",
        "\n",
        "xcl_attrs_unique.size"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "322"
            ]
          },
          "metadata": {},
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GJBadEFNeJgw"
      },
      "source": [
        "The two Excel spreadsheets provided in the workspace (`DIAS Information Levels - Attributes 2017.xlsx` and `DIAS Attributes - Values 2017.xlsx`) contain 310 common attributes. In addition the first file contains 8 unique attributes while the second 4 unique attributes. In total there are 322 unique attributes present in both files together."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "axWSd155af2U"
      },
      "source": [
        "###Comparing `azdias`, `custrs` datasets to the unique attributes from the two excel files.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rlia9PK0Y_5y",
        "outputId": "d0475e52-0d89-46d8-9503-377e520fab25"
      },
      "source": [
        "attrs_intesc = np.intersect1d(azd_attr_names, xcl_attrs_unique)\n",
        "\n",
        "azd_attr_names.size, xcl_attrs_unique.size, attrs_intesc.size"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(366, 322, 276)"
            ]
          },
          "metadata": {},
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YVfldGk3fYMo"
      },
      "source": [
        "For the customers dataset, they only different attributes should be the 3 extra ones in the customers dataset, so the intersection should be the same as for `azdias` dataset:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OivZGO8UfHMf",
        "outputId": "e8036040-f60e-41d7-fc94-81013080f2f8"
      },
      "source": [
        "attrs_intesc = np.intersect1d(cus_attr_names, xcl_attrs_unique)\n",
        "\n",
        "cus_attr_names.size, xcl_attrs_unique.size, attrs_intesc.size"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(369, 322, 276)"
            ]
          },
          "metadata": {},
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1jKz-npDaqPj"
      },
      "source": [
        "At first look, we see that only 276 attributes are common. To explore what happens with the rest, we will look at the different attributes from the excel files."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MlqeX1HjBdTW",
        "outputId": "b9671ffe-948b-4a92-a38a-4d80ea14f4b9"
      },
      "source": [
        "# Set difference of excel unique attributes - azdias set attributes will give us the different excel attributes:\n",
        "difrnt_xcl_attrs = np.setdiff1d(xcl_attrs_unique, azd_attr_names)\n",
        "\n",
        "difrnt_xcl_attrs, difrnt_xcl_attrs.size"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array(['BIP_FLAG', 'CAMEO_DEUINTL_2015', 'D19_BANKEN_DIREKT_RZ',\n",
              "        'D19_BANKEN_GROSS_RZ', 'D19_BANKEN_LOKAL_RZ', 'D19_BANKEN_REST_RZ',\n",
              "        'D19_BEKLEIDUNG_GEH_RZ', 'D19_BEKLEIDUNG_REST_RZ',\n",
              "        'D19_BILDUNG_RZ', 'D19_BIO_OEKO_RZ', 'D19_BUCH_RZ',\n",
              "        'D19_DIGIT_SERV_RZ', 'D19_DROGERIEARTIKEL_RZ', 'D19_ENERGIE_RZ',\n",
              "        'D19_FREIZEIT_RZ', 'D19_GARTEN_RZ', 'D19_HANDWERK_RZ',\n",
              "        'D19_HAUS_DEKO_RZ', 'D19_KINDERARTIKEL_RZ', 'D19_KK_KUNDENTYP',\n",
              "        'D19_KOSMETIK_RZ', 'D19_LEBENSMITTEL_RZ', 'D19_LOTTO_RZ',\n",
              "        'D19_NAHRUNGSERGAENZUNG_RZ', 'D19_RATGEBER_RZ', 'D19_REISEN_RZ',\n",
              "        'D19_SAMMELARTIKEL_RZ', 'D19_SCHUHE_RZ', 'D19_SONSTIGE_RZ',\n",
              "        'D19_TECHNIK_RZ', 'D19_TELKO_MOBILE_RZ', 'D19_TELKO_REST_RZ',\n",
              "        'D19_TIERARTIKEL_RZ', 'D19_VERSAND_REST_RZ',\n",
              "        'D19_VERSICHERUNGEN_RZ', 'D19_VOLLSORTIMENT_RZ',\n",
              "        'D19_WEIN_FEINKOST_RZ', 'EINWOHNER', 'GEOSCORE_KLS7', 'GKZ',\n",
              "        'HAUSHALTSSTRUKTUR', 'KBA13_CCM_1400_2500', 'PLZ', 'PLZ8',\n",
              "        'SOHO_FLAG', 'WACHSTUMSGEBIET_NB'], dtype=object), 46)"
            ]
          },
          "metadata": {},
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nxld0l32klVV"
      },
      "source": [
        "Examining the `azdias` (and `customers`) columns' names we can observe that most of the above names are not actually missing but have been named slightly different. So, for example `CAMEO_INTL_2015` in azdias dataset is `CAMEO_DEUINTL_2015` in the attributes’ explanations file. This means we have to perform some operations to match the columns that appear different. We can create a dictionary to match old names with the new names, to rename the attributes' from excel files as in the azdias dataset. For most of the columns names with the prefix `D19_` if we remove the suffix `_RZ` we have the desired names."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lkCpI95gCh5a"
      },
      "source": [
        "We can also try to identify the rest of attributes, for example with `FLAG` in name and do the same for the rest, but not for `D19_` as we already know that only the suffix `_RZ` is missing from the name, hence it is seen as different string in the code comparisons."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Fd5AzGJfqQ76",
        "outputId": "621879fe-1afb-493d-a2a8-f6c4bcc81d98"
      },
      "source": [
        "attr_flag = [x for x in azd_attr_names if 'FLAG' in x]\n",
        "attr_bip = [x for x in azd_attr_names if 'BIP' in x]\n",
        "attr_soho = [x for x in azd_attr_names if 'SOHO' in x]\n",
        "attr_kba131400 = [x for x in azd_attr_names if 'KBA13_CCM_14' in x]\n",
        "attr_geosc =[ x for x in azd_attr_names if 'GEO' in x]\n",
        "attr_haus = [ x for x in azd_attr_names if 'HAUS' in x]\n",
        "attr_wach = [ x for x in azd_attr_names if 'WACH' in x]\n",
        "attr_gkz = [x for x in azd_attr_names if 'GKZ' in x]\n",
        "attr_plz = [x for x in azd_attr_names if 'PLZ' in x]\n",
        "\n",
        "attr_flag, attr_bip, attr_soho, attr_kba131400, attr_geosc, \\\n",
        "attr_haus, attr_wach, attr_gkz, attr_plz"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(['DSL_FLAG', 'HH_DELTA_FLAG', 'UNGLEICHENN_FLAG'],\n",
              " [],\n",
              " ['SOHO_KZ'],\n",
              " ['KBA13_CCM_1400', 'KBA13_CCM_1401_2500'],\n",
              " ['KBA13_PEUGEOT'],\n",
              " ['ANZ_HAUSHALTE_AKTIV',\n",
              "  'ANZ_STATISTISCHE_HAUSHALTE',\n",
              "  'D19_HAUS_DEKO',\n",
              "  'FINANZ_HAUSBAUER'],\n",
              " [],\n",
              " [],\n",
              " ['PLZ8_ANTG1',\n",
              "  'PLZ8_ANTG2',\n",
              "  'PLZ8_ANTG3',\n",
              "  'PLZ8_ANTG4',\n",
              "  'PLZ8_BAUMAX',\n",
              "  'PLZ8_GBZ',\n",
              "  'PLZ8_HHZ'])"
            ]
          },
          "metadata": {},
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "urHD3UFhC1Bb"
      },
      "source": [
        "From the above we see that we also have attributes `'SOHO_KZ', 'KBA13_CCM_1401_2500'` we can match. These matches will greatly reduce the number of different attributes in the azdias dataset when compared to the attributes' excel files. However, azdias dataset has some more attributes in addition to the ones present in the attributes files.\n",
        "\n",
        "There are no additional attributes for `GKZ` and `PLZ` or `PLZ8` in azdias dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5ddd5ZBxpt8q",
        "outputId": "81dce357-931a-4098-f512-85f60d018237"
      },
      "source": [
        "# Create a dictionary with key: value the old names replacing all `D19_*` names containing `_RZ`\n",
        "xcl_attr_names_new = {i : i[:-3] if '_RZ' in i else i for i in difrnt_xcl_attrs}\n",
        "# Replace SOHO_FLAG:\n",
        "xcl_attr_names_new[\"SOHO_FLAG\"] = 'SOHO_KZ'\n",
        "# Replace CAMEO_DEUINTL_2015:\n",
        "xcl_attr_names_new[\"CAMEO_DEUINTL_2015\"] = 'CAMEO_INTL_2015'\n",
        "# Replace KBA13_CCM_1400_2500:\n",
        "xcl_attr_names_new[\"KBA13_CCM_1400_2500\"] = 'KBA13_CCM_1401_2500'\n",
        "\n",
        "xcl_attr_names_new.items()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "dict_items([('BIP_FLAG', 'BIP_FLAG'), ('CAMEO_DEUINTL_2015', 'CAMEO_INTL_2015'), ('D19_BANKEN_DIREKT_RZ', 'D19_BANKEN_DIREKT'), ('D19_BANKEN_GROSS_RZ', 'D19_BANKEN_GROSS'), ('D19_BANKEN_LOKAL_RZ', 'D19_BANKEN_LOKAL'), ('D19_BANKEN_REST_RZ', 'D19_BANKEN_REST'), ('D19_BEKLEIDUNG_GEH_RZ', 'D19_BEKLEIDUNG_GEH'), ('D19_BEKLEIDUNG_REST_RZ', 'D19_BEKLEIDUNG_REST'), ('D19_BILDUNG_RZ', 'D19_BILDUNG'), ('D19_BIO_OEKO_RZ', 'D19_BIO_OEKO'), ('D19_BUCH_RZ', 'D19_BUCH'), ('D19_DIGIT_SERV_RZ', 'D19_DIGIT_SERV'), ('D19_DROGERIEARTIKEL_RZ', 'D19_DROGERIEARTIKEL'), ('D19_ENERGIE_RZ', 'D19_ENERGIE'), ('D19_FREIZEIT_RZ', 'D19_FREIZEIT'), ('D19_GARTEN_RZ', 'D19_GARTEN'), ('D19_HANDWERK_RZ', 'D19_HANDWERK'), ('D19_HAUS_DEKO_RZ', 'D19_HAUS_DEKO'), ('D19_KINDERARTIKEL_RZ', 'D19_KINDERARTIKEL'), ('D19_KK_KUNDENTYP', 'D19_KK_KUNDENTYP'), ('D19_KOSMETIK_RZ', 'D19_KOSMETIK'), ('D19_LEBENSMITTEL_RZ', 'D19_LEBENSMITTEL'), ('D19_LOTTO_RZ', 'D19_LOTTO'), ('D19_NAHRUNGSERGAENZUNG_RZ', 'D19_NAHRUNGSERGAENZUNG'), ('D19_RATGEBER_RZ', 'D19_RATGEBER'), ('D19_REISEN_RZ', 'D19_REISEN'), ('D19_SAMMELARTIKEL_RZ', 'D19_SAMMELARTIKEL'), ('D19_SCHUHE_RZ', 'D19_SCHUHE'), ('D19_SONSTIGE_RZ', 'D19_SONSTIGE'), ('D19_TECHNIK_RZ', 'D19_TECHNIK'), ('D19_TELKO_MOBILE_RZ', 'D19_TELKO_MOBILE'), ('D19_TELKO_REST_RZ', 'D19_TELKO_REST'), ('D19_TIERARTIKEL_RZ', 'D19_TIERARTIKEL'), ('D19_VERSAND_REST_RZ', 'D19_VERSAND_REST'), ('D19_VERSICHERUNGEN_RZ', 'D19_VERSICHERUNGEN'), ('D19_VOLLSORTIMENT_RZ', 'D19_VOLLSORTIMENT'), ('D19_WEIN_FEINKOST_RZ', 'D19_WEIN_FEINKOST'), ('EINWOHNER', 'EINWOHNER'), ('GEOSCORE_KLS7', 'GEOSCORE_KLS7'), ('GKZ', 'GKZ'), ('HAUSHALTSSTRUKTUR', 'HAUSHALTSSTRUKTUR'), ('KBA13_CCM_1400_2500', 'KBA13_CCM_1401_2500'), ('PLZ', 'PLZ'), ('PLZ8', 'PLZ8'), ('SOHO_FLAG', 'SOHO_KZ'), ('WACHSTUMSGEBIET_NB', 'WACHSTUMSGEBIET_NB')])"
            ]
          },
          "metadata": {},
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vjJoHZvj-KAS"
      },
      "source": [
        "Let us rename the corresponding attribute names rows as per above dictionary and compare to `azdias` attributes again:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZLZCfrJLkaux",
        "outputId": "cd788b78-44dd-492e-a23e-6868a38a9a36"
      },
      "source": [
        "# Create a copy of the `xcl_attrs_unique`\n",
        "xcl_attrs_unique_new = pd.Series(xcl_attrs_unique)\n",
        "\n",
        "# Rename specific attributes based on the created dictionary\n",
        "xcl_attrs_unique_new.replace(xcl_attr_names_new, inplace=True)\n",
        "\n",
        "# Compare to `azdias` attributes again taking the sets intersection\n",
        "attrs_common = np.intersect1d(azd_attr_names, xcl_attrs_unique_new)\n",
        "\n",
        "azd_attr_names.size, xcl_attrs_unique_new.size, attrs_common.size"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(366, 322, 312)"
            ]
          },
          "metadata": {},
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KSy2c0AsLIAP"
      },
      "source": [
        "There are 312 attributes in `azdias` dataset which are also present in the attributes explanatory excel files. Azdias dataset contains 54 additional attributes:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2jS4388ipBIl",
        "outputId": "c3715531-cd9b-4eec-9ec1-f37dae6f3200"
      },
      "source": [
        "# Let's see for the attributes' file what are the different attributes according to the previous intersection\n",
        "azdias_only_attrs = np.setdiff1d(azd_attr_names, xcl_attrs_unique_new)\n",
        "\n",
        "azdias_only_attrs, azdias_only_attrs.size"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array(['AKT_DAT_KL', 'ALTERSKATEGORIE_FEIN', 'ALTER_KIND1', 'ALTER_KIND2',\n",
              "        'ALTER_KIND3', 'ALTER_KIND4', 'ANZ_KINDER',\n",
              "        'ANZ_STATISTISCHE_HAUSHALTE', 'CJT_KATALOGNUTZER', 'CJT_TYP_1',\n",
              "        'CJT_TYP_2', 'CJT_TYP_3', 'CJT_TYP_4', 'CJT_TYP_5', 'CJT_TYP_6',\n",
              "        'D19_BUCH_CD', 'D19_KONSUMTYP_MAX', 'D19_LETZTER_KAUF_BRANCHE',\n",
              "        'D19_SOZIALES', 'D19_TELKO_ONLINE_QUOTE_12',\n",
              "        'D19_VERSI_ONLINE_QUOTE_12', 'DSL_FLAG', 'EINGEFUEGT_AM',\n",
              "        'EINGEZOGENAM_HH_JAHR', 'EXTSEL992', 'FIRMENDICHTE', 'GEMEINDETYP',\n",
              "        'HH_DELTA_FLAG', 'KBA13_ANTG1', 'KBA13_ANTG2', 'KBA13_ANTG3',\n",
              "        'KBA13_ANTG4', 'KBA13_BAUMAX', 'KBA13_GBZ', 'KBA13_HHZ',\n",
              "        'KBA13_KMH_210', 'KK_KUNDENTYP', 'KOMBIALTER', 'KONSUMZELLE',\n",
              "        'LNR', 'MOBI_RASTER', 'RT_KEIN_ANREIZ', 'RT_SCHNAEPPCHEN',\n",
              "        'RT_UEBERGROESSE', 'STRUKTURTYP', 'UMFELD_ALT', 'UMFELD_JUNG',\n",
              "        'UNGLEICHENN_FLAG', 'VERDICHTUNGSRAUM', 'VHA', 'VHN', 'VK_DHT4A',\n",
              "        'VK_DISTANZ', 'VK_ZG11'], dtype=object), 54)"
            ]
          },
          "metadata": {},
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I6VT0tCrt2vY"
      },
      "source": [
        "And `customers` dataframe has the above plus the three additional columns, as described in the project description."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "A7aHmmWrtsQC",
        "outputId": "a6916dc4-613f-4cc0-874a-f57b9f3b6885"
      },
      "source": [
        "custrs_only_attrs = np.setdiff1d(cus_attr_names, xcl_attrs_unique_new)\n",
        "\n",
        "custrs_only_attrs, custrs_only_attrs.size"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array(['AKT_DAT_KL', 'ALTERSKATEGORIE_FEIN', 'ALTER_KIND1', 'ALTER_KIND2',\n",
              "        'ALTER_KIND3', 'ALTER_KIND4', 'ANZ_KINDER',\n",
              "        'ANZ_STATISTISCHE_HAUSHALTE', 'CJT_KATALOGNUTZER', 'CJT_TYP_1',\n",
              "        'CJT_TYP_2', 'CJT_TYP_3', 'CJT_TYP_4', 'CJT_TYP_5', 'CJT_TYP_6',\n",
              "        'CUSTOMER_GROUP', 'D19_BUCH_CD', 'D19_KONSUMTYP_MAX',\n",
              "        'D19_LETZTER_KAUF_BRANCHE', 'D19_SOZIALES',\n",
              "        'D19_TELKO_ONLINE_QUOTE_12', 'D19_VERSI_ONLINE_QUOTE_12',\n",
              "        'DSL_FLAG', 'EINGEFUEGT_AM', 'EINGEZOGENAM_HH_JAHR', 'EXTSEL992',\n",
              "        'FIRMENDICHTE', 'GEMEINDETYP', 'HH_DELTA_FLAG', 'KBA13_ANTG1',\n",
              "        'KBA13_ANTG2', 'KBA13_ANTG3', 'KBA13_ANTG4', 'KBA13_BAUMAX',\n",
              "        'KBA13_GBZ', 'KBA13_HHZ', 'KBA13_KMH_210', 'KK_KUNDENTYP',\n",
              "        'KOMBIALTER', 'KONSUMZELLE', 'LNR', 'MOBI_RASTER',\n",
              "        'ONLINE_PURCHASE', 'PRODUCT_GROUP', 'RT_KEIN_ANREIZ',\n",
              "        'RT_SCHNAEPPCHEN', 'RT_UEBERGROESSE', 'STRUKTURTYP', 'UMFELD_ALT',\n",
              "        'UMFELD_JUNG', 'UNGLEICHENN_FLAG', 'VERDICHTUNGSRAUM', 'VHA',\n",
              "        'VHN', 'VK_DHT4A', 'VK_DISTANZ', 'VK_ZG11'], dtype=object), 57)"
            ]
          },
          "metadata": {},
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ui_PbIzHADzT",
        "outputId": "efaba461-be20-4836-e3bc-878eea137f0e"
      },
      "source": [
        "# Attributes unique to excel files.\n",
        "xcl_only_attrs = np.setdiff1d(xcl_attrs_unique_new, azd_attr_names)\n",
        "\n",
        "xcl_only_attrs, xcl_only_attrs.size"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array(['BIP_FLAG', 'D19_BUCH', 'D19_KK_KUNDENTYP', 'EINWOHNER',\n",
              "        'GEOSCORE_KLS7', 'GKZ', 'HAUSHALTSSTRUKTUR', 'PLZ', 'PLZ8',\n",
              "        'WACHSTUMSGEBIET_NB'], dtype=object), 10)"
            ]
          },
          "metadata": {},
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p6FiuM0NjOQb"
      },
      "source": [
        "* `attrs_common` - 312 common attributes between `azdias` dataset and both excel files combined.\n",
        "* `azdias_only_attrs` - 54 attributes appearing only in `azdias` dataset.\n",
        "* `xcl_attrs_unique_new` - 322 unique attributes in both excel files combined.\n",
        "* `xcl_only_attrs` - 10 attributes appearing only in excel files."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mvqmjw_nzfSy"
      },
      "source": [
        "For 322 unique attributes present in both excel files combined there are 312 in common with the `azdias` dataset, leaving out 54 attributes known only in `azdias` dataset. Most of these seem to be related to attributes found in the excel files. For example `CJT` attribute in excel files seems to be related to the series of `CJT_TYP_*` attributes in azdias dataset. These are probably derived attributes to allow more granularity of collected information in certain fields."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TPW-luJf0XPq",
        "outputId": "0673e6b0-d804-482b-b3e4-cbe8b74b840b"
      },
      "source": [
        "azdias[\"CJT_TYP_1\"].value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "5.0    89881\n",
              "2.0    66243\n",
              "3.0    57721\n",
              "4.0    55071\n",
              "1.0    29508\n",
              "Name: CJT_TYP_1, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ixfj1WlpUrqQ"
      },
      "source": [
        "###Identify and replace out of range and unknown values with NaNs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cf-2qGa8uWGz"
      },
      "source": [
        "We have already made some value replacements when loading the `.csv` files. We observe that some attributes include values outside the pool of values described in the information excel files. These values that are outside the allowed values described in the relevant excel files will be identified and replaced with NaNs. The process we will follow is:\n",
        "\n",
        "* We will get the allowed values pool for each attribute from the excel file `DIAS Attributes - Values 2017.xlsx`.\n",
        "* We will create a dictionary, mapping attribute names to value pools.\n",
        "* We will in turn search the relevant columns in `azdias` and `customers` datasets and replace all the values outside these pools with NaNs.\n",
        "\n",
        "In addition, there are some special cases where the allowed values are not specified, or the attributes do not have corresponding information in any of the excel files. These will have to be treated case by case."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 266
        },
        "id": "5yKP1PVGa54S",
        "outputId": "17ea65a8-3d86-4e66-ebf4-0b593d817b9f"
      },
      "source": [
        "# In attributes values excel replace NaNs in attribute names with forward fill method ffill\n",
        "# to make it easier to extract specific attribute names and values\n",
        "dfdias_attrib_ffil = dfdias_attrib.fillna(method='ffill')\n",
        "\n",
        "dfdias_attrib_ffil.head(7)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Attribute</th>\n",
              "      <th>Description</th>\n",
              "      <th>Value</th>\n",
              "      <th>Meaning</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>AGER_TYP</td>\n",
              "      <td>best-ager typology</td>\n",
              "      <td>-1</td>\n",
              "      <td>unknown</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>AGER_TYP</td>\n",
              "      <td>best-ager typology</td>\n",
              "      <td>0</td>\n",
              "      <td>no classification possible</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>AGER_TYP</td>\n",
              "      <td>best-ager typology</td>\n",
              "      <td>1</td>\n",
              "      <td>passive elderly</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>AGER_TYP</td>\n",
              "      <td>best-ager typology</td>\n",
              "      <td>2</td>\n",
              "      <td>cultural elderly</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>AGER_TYP</td>\n",
              "      <td>best-ager typology</td>\n",
              "      <td>3</td>\n",
              "      <td>experience-driven elderly</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>ALTERSKATEGORIE_GROB</td>\n",
              "      <td>age classification through prename analysis</td>\n",
              "      <td>-1, 0</td>\n",
              "      <td>unknown</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>ALTERSKATEGORIE_GROB</td>\n",
              "      <td>age classification through prename analysis</td>\n",
              "      <td>1</td>\n",
              "      <td>&lt; 30 years</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "              Attribute  ...                     Meaning\n",
              "0              AGER_TYP  ...                     unknown\n",
              "1              AGER_TYP  ...  no classification possible\n",
              "2              AGER_TYP  ...             passive elderly\n",
              "3              AGER_TYP  ...            cultural elderly\n",
              "4              AGER_TYP  ...   experience-driven elderly\n",
              "5  ALTERSKATEGORIE_GROB  ...                     unknown\n",
              "6  ALTERSKATEGORIE_GROB  ...                  < 30 years\n",
              "\n",
              "[7 rows x 4 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A7lzPWQQz4gz"
      },
      "source": [
        "1. Create a dictionary with attributes' names as keys and allowed input (`Value` column) as values."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vykPIslNZQ7t",
        "outputId": "8ace2c36-ffe5-4823-cc2a-b4fba8090d5a"
      },
      "source": [
        "# First get the allowed value ranges per attribute from the attributes values file (DIAS Attributes - Values 2017.xlsx)\n",
        "# Group by Attribute and get values in a multinex object, then transform to np.ndarray by chaining `values` method:\n",
        "attrs_values_arr = dfdias_attrib_ffil.groupby(\"Attribute\")[\"Value\"].value_counts().index.values\n",
        "\n",
        "# Create a helper function to parse values in the form '-1, 0'\n",
        "def get_nums(x):\n",
        "    \"\"\"Return a list of integers from a string\n",
        "    or the same input if not a string\"\"\"\n",
        "    try:\n",
        "        li = x.split(',')\n",
        "        return list(map(int, li))\n",
        "    except (AttributeError, ValueError):\n",
        "        return [x]\n",
        "\n",
        "# Create a dictionary mapping attributes as keys and allowed values in a list to key:[values]\n",
        "# we are simultaneously injecting np.NaN in values' range as this might already be present\n",
        "# among the attributes' values in `azdias`.\n",
        "attrs_val_dict = {}\n",
        "for i in attrs_values_arr:\n",
        "    attrs_val_dict.setdefault(i[0],[np.NaN]).extend(get_nums(i[1]))\n",
        "\n",
        "list(attrs_val_dict.items())[:2]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('AGER_TYP', [nan, -1, 0, 1, 2, 3]),\n",
              " ('ALTERSKATEGORIE_GROB', [nan, 1, 2, 3, 4, 9, -1, 0])]"
            ]
          },
          "metadata": {},
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BlytOkLuBHJ4"
      },
      "source": [
        "This dictionary must have its keys (attributes) names match the corresponding attributes names in `azdias` dataset for the cases where the attributes have differences in names, according to the previously created dictionary mapping `xcl_attr_names_new`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AaYrGec2Bm60"
      },
      "source": [
        "# We will use the `xcl_attr_names_new` to get the corresponding names and create a new dictionary.\n",
        "attrs_val_pools = {xcl_attr_names_new.get(k, k): v for k, v in attrs_val_dict.items()}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RBriqwyKkfb7"
      },
      "source": [
        "We will create a different dictionary with the special cases with key values equal to `'…'`. These are some special cases where the input is described as \"numeric value\" which is attribute context related. So, this can be a year, a number typically coded from 1-10, or 1-3, but with the range not exactly specified. This will allow us to process these cases separately."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uI9xYLc6nQ-j",
        "outputId": "9cd1ee12-fabf-4b7d-f355-be7157ddea3e"
      },
      "source": [
        "# Create new dict while removing the related items from the previous\n",
        "# Iterate through a list of keys \n",
        "attrs_val_dict_any = {k: attrs_val_pools.pop(k) for k in list(attrs_val_pools) if '…' in attrs_val_pools[k]}\n",
        "\n",
        "attrs_val_dict_any"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'ANZ_HAUSHALTE_AKTIV': [nan, '…'],\n",
              " 'ANZ_HH_TITEL': [nan, '…'],\n",
              " 'ANZ_PERSONEN': [nan, '…'],\n",
              " 'ANZ_TITEL': [nan, '…'],\n",
              " 'GEBURTSJAHR': [nan, '…'],\n",
              " 'KBA13_ANZAHL_PKW': [nan, '…'],\n",
              " 'MIN_GEBAEUDEJAHR': [nan, '…']}"
            ]
          },
          "metadata": {},
          "execution_count": 51
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Am6RwFr2oQX0"
      },
      "source": [
        "We now have two dictionaries mapping attributes to possible values for each attribute. There are some unique attributes (`azdias_only_attrs`) in `azdias` dataset that have not been accounted for. For these we will adopt the mapping of similar attributes based on context and add them in the first mapping dictionary. We can identify at least two value categories `CJT_*` and `KBA13_` that can be assigned to values' range list `[-1, 0, 1, 2, 3, 4, 5]` as can be observed from similar attributes in excel attributes values file. We cannot do the same for `D19_*` attributes or any of the rest attributes in the `azdias_only_attrs`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ea-KBCyJpYGH",
        "outputId": "061eaa0f-357c-4070-9896-b1153ce803da"
      },
      "source": [
        "# we are then left with an undefined values range list of attributes for unique to `azdias` as:\n",
        "azdias_only_attrs_undef = [i for i in azdias_only_attrs if not i.startswith(('CJT_', 'KBA13_'))]\n",
        "\n",
        "azdias_only_attrs_undef = np.array(azdias_only_attrs_undef)\n",
        "\n",
        "azdias_only_attrs_undef, azdias_only_attrs_undef.size"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array(['AKT_DAT_KL', 'ALTERSKATEGORIE_FEIN', 'ALTER_KIND1', 'ALTER_KIND2',\n",
              "        'ALTER_KIND3', 'ALTER_KIND4', 'ANZ_KINDER',\n",
              "        'ANZ_STATISTISCHE_HAUSHALTE', 'D19_BUCH_CD', 'D19_KONSUMTYP_MAX',\n",
              "        'D19_LETZTER_KAUF_BRANCHE', 'D19_SOZIALES',\n",
              "        'D19_TELKO_ONLINE_QUOTE_12', 'D19_VERSI_ONLINE_QUOTE_12',\n",
              "        'DSL_FLAG', 'EINGEFUEGT_AM', 'EINGEZOGENAM_HH_JAHR', 'EXTSEL992',\n",
              "        'FIRMENDICHTE', 'GEMEINDETYP', 'HH_DELTA_FLAG', 'KK_KUNDENTYP',\n",
              "        'KOMBIALTER', 'KONSUMZELLE', 'LNR', 'MOBI_RASTER',\n",
              "        'RT_KEIN_ANREIZ', 'RT_SCHNAEPPCHEN', 'RT_UEBERGROESSE',\n",
              "        'STRUKTURTYP', 'UMFELD_ALT', 'UMFELD_JUNG', 'UNGLEICHENN_FLAG',\n",
              "        'VERDICHTUNGSRAUM', 'VHA', 'VHN', 'VK_DHT4A', 'VK_DISTANZ',\n",
              "        'VK_ZG11'], dtype='<U26'), 39)"
            ]
          },
          "metadata": {},
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6O8slta7yQ79"
      },
      "source": [
        "So, for the `customers` the respective array will be:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "19My-1UPwXfE",
        "outputId": "fbb20c45-b585-40fe-f047-9b04855b6678"
      },
      "source": [
        "cus_only_attrs_undef = np.append(azdias_only_attrs_undef,\n",
        "                                 ['CUSTOMER_GROUP', 'ONLINE_PURCHASE', 'PRODUCT_GROUP'])\n",
        "\n",
        "cus_only_attrs_undef, cus_only_attrs_undef.size"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array(['AKT_DAT_KL', 'ALTERSKATEGORIE_FEIN', 'ALTER_KIND1', 'ALTER_KIND2',\n",
              "        'ALTER_KIND3', 'ALTER_KIND4', 'ANZ_KINDER',\n",
              "        'ANZ_STATISTISCHE_HAUSHALTE', 'D19_BUCH_CD', 'D19_KONSUMTYP_MAX',\n",
              "        'D19_LETZTER_KAUF_BRANCHE', 'D19_SOZIALES',\n",
              "        'D19_TELKO_ONLINE_QUOTE_12', 'D19_VERSI_ONLINE_QUOTE_12',\n",
              "        'DSL_FLAG', 'EINGEFUEGT_AM', 'EINGEZOGENAM_HH_JAHR', 'EXTSEL992',\n",
              "        'FIRMENDICHTE', 'GEMEINDETYP', 'HH_DELTA_FLAG', 'KK_KUNDENTYP',\n",
              "        'KOMBIALTER', 'KONSUMZELLE', 'LNR', 'MOBI_RASTER',\n",
              "        'RT_KEIN_ANREIZ', 'RT_SCHNAEPPCHEN', 'RT_UEBERGROESSE',\n",
              "        'STRUKTURTYP', 'UMFELD_ALT', 'UMFELD_JUNG', 'UNGLEICHENN_FLAG',\n",
              "        'VERDICHTUNGSRAUM', 'VHA', 'VHN', 'VK_DHT4A', 'VK_DISTANZ',\n",
              "        'VK_ZG11', 'CUSTOMER_GROUP', 'ONLINE_PURCHASE', 'PRODUCT_GROUP'],\n",
              "       dtype='<U26'), 42)"
            ]
          },
          "metadata": {},
          "execution_count": 53
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1l1c0LrUk_6u"
      },
      "source": [
        "* We now have `attrs_val_pools` which is a dictionary mapping of known attribute names and \n",
        "lists with accepted values per variable.\n",
        "* `attrs_val_dict_any` which is a dictionary with attributes' names that do not have specific values defined but instead accept a wider range of numerical values.\n",
        "* `azdias_only_attrs_undef`, `cus_only_attrs_undef` which are two arrays with all attributes known in `azdias` and `customers` only that do not have associated information about their accepted values.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tg002pusJW9b"
      },
      "source": [
        "####Replace unknown and specific values with NaNs\n",
        "\n",
        "Using the `attrs_val_pools` we can check against the values in respective attributes in `azdias` and `customers` datasets. If values are found outside the accepted ones, these will be replaced by NaNs. We will use the keys in this dictionary to iterate through the attributes and make changes only in the common attributes (`attrs_common`) between `azdias` dataset and excel files."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KuH1ZF7c_R0C",
        "outputId": "6e0ec108-9632-488b-f4ff-6700102dfb8c"
      },
      "source": [
        "# for every key (attribute) in the dictionary created earlier\n",
        "for k in attrs_val_pools:\n",
        "    # if the attribute is among the common ones\n",
        "    if k in attrs_common:\n",
        "        # select all those rows in the column that fall outside the attribute's\n",
        "        # specified values pool and replace them with NaN.\n",
        "        azdias.loc[~azdias[k].isin(attrs_val_pools[k]), k] = np.NaN\n",
        "        custrs.loc[~custrs[k].isin(attrs_val_pools[k]), k] = np.NaN\n",
        "\n",
        "azdias['CAMEO_DEUG_2015'].value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "8.0    45507\n",
              "9.0    36443\n",
              "6.0    35479\n",
              "4.0    35077\n",
              "3.0    29259\n",
              "2.0    28097\n",
              "7.0    25994\n",
              "5.0    18443\n",
              "1.0    11960\n",
              "Name: CAMEO_DEUG_2015, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 54
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YGdqtIeHKU2k"
      },
      "source": [
        "We want to find the attributes where 9 is used for unknown/NaN values, to replace it in `azdias`, `custrs` datasets with NaNs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vf5wKAYs-yW-"
      },
      "source": [
        "# We will use the forward filled attribute dataframe to make it easier to extract the attributes' names\n",
        "attrs_nine_name = dfdias_attrib_ffil[dfdias_attrib_ffil[\"Value\"] == '-1, 9'][\"Attribute\"].tolist()\n",
        "\n",
        "# Create a dict to use as input for the pandas replace method\n",
        "replace9_nan_dict = {k:9 for k in attrs_nine_name}\n",
        "# Add 'KOMBIALTER' attribute in the dict that is not in excel files\n",
        "replace9_nan_dict['KOMBIALTER'] = 9\n",
        "\n",
        "# We can now use this dictionary of attributes to replace value 9 (unknown) with\n",
        "# NaN for specific attributes in the azdias, custrs dataframe\n",
        "azdias.replace(replace9_nan_dict, np.NaN, inplace=True)\n",
        "custrs.replace(replace9_nan_dict, np.NaN, inplace=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PeNGbxZp1sAc"
      },
      "source": [
        "Special cases where values' pool was `'…'`:\n",
        "\n",
        "* `ANZ_HAUSHALTE_AKTIV` - number of households in the building numeric value, typically coded from 1-10 but higher values can be accepted (large building blocks)\n",
        "* `ANZ_HH_TITEL` - number of academic title holder in building, numeric value typically coded from 1-10 but higher values can be accepted (large building blocks)\n",
        "* `ANZ_PERSONEN` - number of adult persons in the household, numeric value typically coded from 1-3 but higher values can be accepted depending on the area and family size. This has some extreme values values in the range [23, 45] that may be proven outliers.\n",
        "* `ANZ_TITEL` - number of professional title holders in a household, numeric value typically coded from 1-10 but higher values can be accepted depending on the family size.\n",
        "* `GEBURTSJAHR` - year of birth, with values ranging from 1900 to 2017.\n",
        "* `KBA13_ANZAHL_PKW` - number of cars in the PLZ8, numeric value. This may also have a large range based on the area (postcode) rural, urban areas.\n",
        "* `MIN_GEBAEUDEJAHR` - year the building was first mentioned in the database, numeric values of years ranging from 1985 to 2016.\n",
        "\n",
        "By observation these attributes seem to have acceptable values related to their context.\n",
        "\n",
        "For the attributes known only to `azdias` dataset that cannot be related to existing information from the excel files there is a number of attributes that appear to be related to one another - `'ALTERSKATEGORIE_FEIN', 'ALTER_KIND1', 'ALTER_KIND2', 'ALTER_KIND3', 'ALTER_KIND4'` - and by context might provide categorization for number of children for age group bins. From `ALTERSKATEGORIE_FEIN` we can understand that there are 25 bins that correspond to an age range. `ANZ_KINDER` provides the number of children, while `''ALTER_KIND1', 'ALTER_KIND2', 'ALTER_KIND3', 'ALTER_KIND4'`, must provide the ages for the 1st, 2nd, 3rd and 4th child. `ANZ_STATISTISCHE_HAUSHALTE` seems to provide some budget categorization in a range 1 - 449 with a mean 7.6.\n",
        "\n",
        "`'D19_BUCH_CD'` is probably related to books and or CDs ownership, `'D19_KONSUMTYP_MAX'` is a categorization of consumer type,`'D19_LETZTER_KAUF_BRANCHE'` provides a categorization based on recent products purchases, `'D19_SOZIALES'` probably provides a social type related categorization, `'D19_TELKO_ONLINE_QUOTE_12'` (few records) is telecoms related probably for the last 12 months average spending, `'D19_VERSI_ONLINE_QUOTE_12'` (few records) is another 12 months spending record, `'DSL_FLAG'` is probably internet line capacity related (DSL line) indicating if person has DSL (value 1), `'EINGEFUEGT_AM'` is probably the date and time the database entry for this entry was inserted (all times are set to 00:00:00), `'EINGEZOGENAM HH JAHR'` is possibly the year someone moved to a household, `'EXTSEL992'` is some kind of age related attribute, `'FIRMENDICHTE'` is a company size categorization probably related to the company each person works for, `'GEMEINDETYP'` is a community type categorization, `'HH_DELTA_FLAG'` cannot be associated with something known, `'KK_KUNDENTYP'` is a customer type categorization, `'KOMBIALTER'` is another age related categorization, `'KONSUMZELLE'` is another flag probably indicating somekind of consumer feature, `'MOBI_RASTER'` should be an attribute related to mobiles (or mobility) range 1-6, `'RT_KEIN_ANREIZ'` refers lack of incentives on something with values range 1-5, `'RT_SCHNAEPPCHEN'` describes bargain response on something with values range 1-5, `'RT_UEBERGROESSE'` describes over-size response on something with values range 1-5, `'SOHO_KZ'` could be related to small office / home office binary attribute, `'STRUKTURTYP'` provides structure type for homes (?) with values range 1-3, `'UMFELD_ALT'` records old relatives(?) with values range 1-5, `'UMFELD_JUNG'` records young relatives with values range 1-5, `'UNGLEICHENN_FLAG'` binary attribute recording unequality, `'VERDICHTUNGSRAUM'` probably describes the density of the living area (neibourhood, block?) values 1-45, `'VHA', 'VHN', 'VK_DHT4A', 'VK_DISTANZ','VK_ZG11'` cannot be identified."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0DLFyqxkFKOo"
      },
      "source": [
        "# Based on the above these are additional categoricals:\n",
        "extra_cat_cols = np.array(['ALTERSKATEGORIE_FEIN', 'D19_BUCH_CD', 'D19_KONSUMTYP_MAX',\n",
        "                          'D19_SOZIALES', 'D19_TELKO_ONLINE_QUOTE_12', 'D19_VERSI_ONLINE_QUOTE_12',\n",
        "                          'FIRMENDICHTE', 'GEMEINDETYP', 'KK_KUNDENTYP', 'KOMBIALTER',\n",
        "                          'MOBI_RASTER', 'RT_KEIN_ANREIZ', 'RT_SCHNAEPPCHEN',\n",
        "                          'RT_UEBERGROESSE', 'STRUKTURTYP', 'UMFELD_ALT', 'UMFELD_JUNG',\n",
        "                          'KONSUMZELLE', 'SOHO_KZ', 'UNGLEICHENN_FLAG', 'VHA', 'VHN',\n",
        "                          'VK_DHT4A', 'VK_DISTANZ', 'VK_ZG11'])\n",
        "# ('KOMBIALTER' needs to remove 9 as NaN)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n-W1VaD1Q8uI"
      },
      "source": [
        "All known categorical (includes all ordinal) attributes `cat_attrs`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sOtvwq1wQfu9",
        "outputId": "85ca67f9-9351-4735-ff93-a24828cf2f36"
      },
      "source": [
        "cat_attrs = np.union1d(attrs_common, extra_cat_cols)\n",
        "\n",
        "cat_attrs.size"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "336"
            ]
          },
          "metadata": {},
          "execution_count": 57
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WHjWie9dT_OW"
      },
      "source": [
        "##Analysing and understanding missing values in `azdias` and `custrs` dfs\n",
        "\n",
        "* Total NaNs:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MDCgqY6DHKWA",
        "outputId": "1f722e96-64d0-4677-d081-f1c6cc38e343"
      },
      "source": [
        "# count NaNs in the dataframes\n",
        "azdias_nan = azdias.isna().sum().sum()\n",
        "cus_df_nan = custrs.isna().sum().sum()\n",
        "\n",
        "(azdias.size, azdias_nan, round(azdias_nan/azdias.size, 2)), \\\n",
        "(custrs.size, cus_df_nan, round(cus_df_nan/custrs.size, 2))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((109800000, 30782105, 0.28), (55350000, 19690956, 0.36))"
            ]
          },
          "metadata": {},
          "execution_count": 58
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NtnheqHMPu8f"
      },
      "source": [
        "There are about 28% NaNs in `azdias` df and about 35-36% NaNs in the `customers` \n",
        "dataset out of the total data sizes.\n",
        "\n",
        "* NaNs per column:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TSYfy-4yeJDY",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 555
        },
        "outputId": "2c08dd9b-a9e1-4a2b-a633-21a22dcae271"
      },
      "source": [
        "def cols_nans_info(df, cols=None):\n",
        "    \"\"\"Return a new dataframe with NaNs summary per column of the input df\n",
        "\n",
        "    Creates and returns a new dataframe with three columns that correspond to\n",
        "    the attributes (columns) of the initial dataframe, the NaN counts per column\n",
        "    and portion of NaNs per column.\n",
        "\n",
        "    Args:\n",
        "        df (pd.DataFrame): The input dataframe to process.\n",
        "        cols (tuple): 3 elements tuple with the names of the three columns in the\n",
        "        output dataframe. If left None the three default names ('Columns',\n",
        "        'NaNcounts', 'NaNportions') will be used.\n",
        "    \n",
        "    Returns:\n",
        "        A new dataframe with three columns providing counts and portions of NaNs\n",
        "        per column in the input df. Example:\n",
        "\n",
        "            Columns\t     NaNcounts\tNaNportions\n",
        "        0\tAGER_TYP\t   685843\t    0.769554\n",
        "        1\tAKT_DAT_KL     73499\t    0.082470\n",
        "        2\tALTER_HH\t   310267\t    0.348137\n",
        "        3\tALTER_KIND1\t   810163\t    0.909048\n",
        "        4\tALTER_KIND2\t   861722\t    0.966900\n",
        "    \"\"\"\n",
        "    cols = cols or ('Columns', 'NaNcounts', 'NaNportions')\n",
        "    new_df = df.isna().sum().rename_axis(cols[0]).reset_index(name=cols[1])\n",
        "    new_df[cols[2]] = new_df[cols[1]] / df.shape[0]\n",
        "\n",
        "    print('Average NaNs per column: ', round(new_df[cols[2]].mean(), 4))\n",
        "    print('Max NaNs per column: ', round(new_df[cols[2]].max(), 4))\n",
        "    print('Min NaNs per column: ', round(new_df[cols[2]].min(), 4))\n",
        "\n",
        "    return new_df\n",
        "\n",
        "# Get NaNs info for the `azdias` dataframe\n",
        "print(\"AZDIAS\")\n",
        "azdias_nans = cols_nans_info(azdias)\n",
        "# Get NaNs info for the `custrs` dataframe\n",
        "print(\"CUSTOMERS\")\n",
        "cus_df_nans = cols_nans_info(custrs)\n",
        "\n",
        "pd.merge(azdias_nans, cus_df_nans, on='Columns', suffixes=(\"_azd\", \"_cus\"))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "AZDIAS\n",
            "Average NaNs per column:  0.2803\n",
            "Max NaNs per column:  0.9991\n",
            "Min NaNs per column:  0.0\n",
            "CUSTOMERS\n",
            "Average NaNs per column:  0.3558\n",
            "Max NaNs per column:  0.999\n",
            "Min NaNs per column:  0.0\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Columns</th>\n",
              "      <th>NaNcounts_azd</th>\n",
              "      <th>NaNportions_azd</th>\n",
              "      <th>NaNcounts_cus</th>\n",
              "      <th>NaNportions_cus</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>LNR</td>\n",
              "      <td>0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>AGER_TYP</td>\n",
              "      <td>230705</td>\n",
              "      <td>0.769017</td>\n",
              "      <td>75850</td>\n",
              "      <td>0.505667</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>AKT_DAT_KL</td>\n",
              "      <td>24857</td>\n",
              "      <td>0.082857</td>\n",
              "      <td>36528</td>\n",
              "      <td>0.243520</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ALTER_HH</td>\n",
              "      <td>104758</td>\n",
              "      <td>0.349193</td>\n",
              "      <td>53876</td>\n",
              "      <td>0.359173</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ALTER_KIND1</td>\n",
              "      <td>272842</td>\n",
              "      <td>0.909473</td>\n",
              "      <td>140828</td>\n",
              "      <td>0.938853</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>361</th>\n",
              "      <td>WOHNDAUER_2008</td>\n",
              "      <td>24857</td>\n",
              "      <td>0.082857</td>\n",
              "      <td>36528</td>\n",
              "      <td>0.243520</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>362</th>\n",
              "      <td>WOHNLAGE</td>\n",
              "      <td>33900</td>\n",
              "      <td>0.113000</td>\n",
              "      <td>39977</td>\n",
              "      <td>0.266513</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>363</th>\n",
              "      <td>ZABEOTYP</td>\n",
              "      <td>0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>364</th>\n",
              "      <td>ANREDE_KZ</td>\n",
              "      <td>0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>365</th>\n",
              "      <td>ALTERSKATEGORIE_GROB</td>\n",
              "      <td>0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>366 rows × 5 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                  Columns  NaNcounts_azd  ...  NaNcounts_cus  NaNportions_cus\n",
              "0                     LNR              0  ...              0         0.000000\n",
              "1                AGER_TYP         230705  ...          75850         0.505667\n",
              "2              AKT_DAT_KL          24857  ...          36528         0.243520\n",
              "3                ALTER_HH         104758  ...          53876         0.359173\n",
              "4             ALTER_KIND1         272842  ...         140828         0.938853\n",
              "..                    ...            ...  ...            ...              ...\n",
              "361        WOHNDAUER_2008          24857  ...          36528         0.243520\n",
              "362              WOHNLAGE          33900  ...          39977         0.266513\n",
              "363              ZABEOTYP              0  ...              0         0.000000\n",
              "364             ANREDE_KZ              0  ...              0         0.000000\n",
              "365  ALTERSKATEGORIE_GROB              0  ...              0         0.000000\n",
              "\n",
              "[366 rows x 5 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 59
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gIE9eiJB29-U"
      },
      "source": [
        "A first look at the output indicates that there are attributes with a lot of NaNs and even some full of NaNs. Let us visualize these results to get a better picture of the data. We will create a histogram dividing the percentage of NaNs in 10 bins counting corresponding attributes in each bin."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6rbuO8UV2fVS"
      },
      "source": [
        "# Save image location\n",
        "nan_col_distrb_file = '/content/drive/MyDrive/workspace/arvato_data/nan_col_distrb_{}.png'.format(ts) \\\n",
        "if IN_COLAB else 'nan_col_distrb_{}.png'.format(ts)\n",
        "\n",
        "with sns.axes_style(\"whitegrid\"):\n",
        "    fig, ax = plt.subplots(1, 2, figsize=(10,5))\n",
        "    sns.histplot(ax=ax[0], data=azdias_nans, x=\"NaNportions\", bins=np.linspace(0, 1, 11)).set(title='[AZDIAS] Column counts per NaN portions')\n",
        "    sns.histplot(ax=ax[1], data=cus_df_nans, x=\"NaNportions\", bins=np.linspace(0, 1, 11)).set(title='[CUSTOMERS] Column counts per NaN portions')\n",
        "    # copyright text\n",
        "    fig.suptitle(cprt, y=-.03, verticalalignment='bottom', bbox=bb_cprt_args)\n",
        "    fig.tight_layout()\n",
        "    fig.savefig(nan_col_distrb_file, transparent=False, bbox_inches=\"tight\")\n",
        "    plt.close(fig)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wJQfPTzCHR0I"
      },
      "source": [
        "We can see that both datasets have similar distributions of NaNs. The first plot for `azdias` shows that there are about 50+ attributes with NaNs above 80%. Around 280 attributes have less than 30% NaNs while the rest are between 30% - 80% NaNs. The second for `customers` shows that there are around 235 attributes with less than 30% NaNs and around 50 with more than 80% NaNs.\n",
        "\n",
        "* `AZDIAS`\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 359
        },
        "id": "1Mjg0zktJvbf",
        "outputId": "1ccaec2e-0878-4106-d9a4-76f912120773"
      },
      "source": [
        "azd_col_nan_bins = azdias_nans.NaNportions.value_counts(bins=np.linspace(0, 1, 11), sort=False)\n",
        "\n",
        "azd_col_nan_bins = azd_col_nan_bins.rename_axis(\"Bins\").reset_index(name=\"Count\")\n",
        "\n",
        "azd_col_nan_bins[\"ColPortion\"] = azd_col_nan_bins.Count / azdias.shape[1]\n",
        "\n",
        "azd_col_nan_bins"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Bins</th>\n",
              "      <th>Count</th>\n",
              "      <th>ColPortion</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>(-0.001, 0.1]</td>\n",
              "      <td>67</td>\n",
              "      <td>0.183060</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>(0.1, 0.2]</td>\n",
              "      <td>169</td>\n",
              "      <td>0.461749</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>(0.2, 0.3]</td>\n",
              "      <td>40</td>\n",
              "      <td>0.109290</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>(0.3, 0.4]</td>\n",
              "      <td>4</td>\n",
              "      <td>0.010929</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>(0.4, 0.5]</td>\n",
              "      <td>10</td>\n",
              "      <td>0.027322</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>(0.5, 0.6]</td>\n",
              "      <td>7</td>\n",
              "      <td>0.019126</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>(0.6, 0.7]</td>\n",
              "      <td>6</td>\n",
              "      <td>0.016393</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>(0.7, 0.8]</td>\n",
              "      <td>10</td>\n",
              "      <td>0.027322</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>(0.8, 0.9]</td>\n",
              "      <td>21</td>\n",
              "      <td>0.057377</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>(0.9, 1.0]</td>\n",
              "      <td>32</td>\n",
              "      <td>0.087432</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            Bins  Count  ColPortion\n",
              "0  (-0.001, 0.1]     67    0.183060\n",
              "1     (0.1, 0.2]    169    0.461749\n",
              "2     (0.2, 0.3]     40    0.109290\n",
              "3     (0.3, 0.4]      4    0.010929\n",
              "4     (0.4, 0.5]     10    0.027322\n",
              "5     (0.5, 0.6]      7    0.019126\n",
              "6     (0.6, 0.7]      6    0.016393\n",
              "7     (0.7, 0.8]     10    0.027322\n",
              "8     (0.8, 0.9]     21    0.057377\n",
              "9     (0.9, 1.0]     32    0.087432"
            ]
          },
          "metadata": {},
          "execution_count": 61
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "evEpfSVv1VPW",
        "outputId": "e796e2da-1142-4433-c191-a0eb755e39de"
      },
      "source": [
        "azd_col_nan_bins.loc[:2,].sum()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Count         276.000000\n",
              "ColPortion      0.754098\n",
              "dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 62
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FWAnwTMKRbRI"
      },
      "source": [
        "* `CUSTOMERS`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 359
        },
        "id": "UzJoHJdiL6Tg",
        "outputId": "85f1ae6d-18e1-44be-e9cf-a0227a4531c5"
      },
      "source": [
        "cus_col_nan_bins = cus_df_nans.NaNportions.value_counts(bins=np.linspace(0, 1, 11), sort=False)\n",
        "\n",
        "cus_col_nan_bins = cus_col_nan_bins.rename_axis(\"Bins\").reset_index(name=\"Count\")\n",
        "\n",
        "cus_col_nan_bins[\"ColPortion\"] = cus_col_nan_bins.Count / custrs.shape[1]\n",
        "\n",
        "cus_col_nan_bins"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Bins</th>\n",
              "      <th>Count</th>\n",
              "      <th>ColPortion</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>(-0.001, 0.1]</td>\n",
              "      <td>59</td>\n",
              "      <td>0.159892</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>(0.1, 0.2]</td>\n",
              "      <td>0</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>(0.2, 0.3]</td>\n",
              "      <td>176</td>\n",
              "      <td>0.476965</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>(0.3, 0.4]</td>\n",
              "      <td>40</td>\n",
              "      <td>0.108401</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>(0.4, 0.5]</td>\n",
              "      <td>20</td>\n",
              "      <td>0.054201</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>(0.5, 0.6]</td>\n",
              "      <td>10</td>\n",
              "      <td>0.027100</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>(0.6, 0.7]</td>\n",
              "      <td>8</td>\n",
              "      <td>0.021680</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>(0.7, 0.8]</td>\n",
              "      <td>9</td>\n",
              "      <td>0.024390</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>(0.8, 0.9]</td>\n",
              "      <td>18</td>\n",
              "      <td>0.048780</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>(0.9, 1.0]</td>\n",
              "      <td>29</td>\n",
              "      <td>0.078591</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            Bins  Count  ColPortion\n",
              "0  (-0.001, 0.1]     59    0.159892\n",
              "1     (0.1, 0.2]      0    0.000000\n",
              "2     (0.2, 0.3]    176    0.476965\n",
              "3     (0.3, 0.4]     40    0.108401\n",
              "4     (0.4, 0.5]     20    0.054201\n",
              "5     (0.5, 0.6]     10    0.027100\n",
              "6     (0.6, 0.7]      8    0.021680\n",
              "7     (0.7, 0.8]      9    0.024390\n",
              "8     (0.8, 0.9]     18    0.048780\n",
              "9     (0.9, 1.0]     29    0.078591"
            ]
          },
          "metadata": {},
          "execution_count": 63
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "f-weU28oQdYi",
        "outputId": "790ce384-b760-46f3-ab1f-fb72c760f17c"
      },
      "source": [
        "cus_col_nan_bins.loc[:2,].sum()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Count         235.000000\n",
              "ColPortion      0.636856\n",
              "dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 64
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r8mPD3c3RiBE"
      },
      "source": [
        "####Format data for plotting\n",
        "\n",
        "We need to convert the dataframe to [long-form](http://seaborn.pydata.org/tutorial/data_structure.html#long-form-vs-wide-form-data) to use the `hue` argument in `sns.barplot` to plot the data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "d2NL0Nk_MZsD",
        "outputId": "8b2eba38-0187-43f0-fb25-fe2cd5bd513c"
      },
      "source": [
        "merge_col_nan_bins = pd.merge(azd_col_nan_bins, cus_col_nan_bins, on='Bins', suffixes=(\"_azd\", \"_cus\"))\n",
        "\n",
        "merge_col_nan_bins.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Bins</th>\n",
              "      <th>Count_azd</th>\n",
              "      <th>ColPortion_azd</th>\n",
              "      <th>Count_cus</th>\n",
              "      <th>ColPortion_cus</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>(-0.001, 0.1]</td>\n",
              "      <td>67</td>\n",
              "      <td>0.183060</td>\n",
              "      <td>59</td>\n",
              "      <td>0.159892</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>(0.1, 0.2]</td>\n",
              "      <td>169</td>\n",
              "      <td>0.461749</td>\n",
              "      <td>0</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>(0.2, 0.3]</td>\n",
              "      <td>40</td>\n",
              "      <td>0.109290</td>\n",
              "      <td>176</td>\n",
              "      <td>0.476965</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>(0.3, 0.4]</td>\n",
              "      <td>4</td>\n",
              "      <td>0.010929</td>\n",
              "      <td>40</td>\n",
              "      <td>0.108401</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>(0.4, 0.5]</td>\n",
              "      <td>10</td>\n",
              "      <td>0.027322</td>\n",
              "      <td>20</td>\n",
              "      <td>0.054201</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            Bins  Count_azd  ColPortion_azd  Count_cus  ColPortion_cus\n",
              "0  (-0.001, 0.1]         67        0.183060         59        0.159892\n",
              "1     (0.1, 0.2]        169        0.461749          0        0.000000\n",
              "2     (0.2, 0.3]         40        0.109290        176        0.476965\n",
              "3     (0.3, 0.4]          4        0.010929         40        0.108401\n",
              "4     (0.4, 0.5]         10        0.027322         20        0.054201"
            ]
          },
          "metadata": {},
          "execution_count": 65
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "cPFgga5kNuuK",
        "outputId": "d2b73af8-9a08-4f3d-968a-5ddda99bad3f"
      },
      "source": [
        "cols_nan_portns_melt = merge_col_nan_bins[\n",
        "                                          ['Bins', 'ColPortion_azd', 'ColPortion_cus']\n",
        "                                         ].melt(id_vars=('Bins'))\n",
        "cols_nan_portns_melt.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Bins</th>\n",
              "      <th>variable</th>\n",
              "      <th>value</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>(-0.001, 0.1]</td>\n",
              "      <td>ColPortion_azd</td>\n",
              "      <td>0.183060</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>(0.1, 0.2]</td>\n",
              "      <td>ColPortion_azd</td>\n",
              "      <td>0.461749</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>(0.2, 0.3]</td>\n",
              "      <td>ColPortion_azd</td>\n",
              "      <td>0.109290</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>(0.3, 0.4]</td>\n",
              "      <td>ColPortion_azd</td>\n",
              "      <td>0.010929</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>(0.4, 0.5]</td>\n",
              "      <td>ColPortion_azd</td>\n",
              "      <td>0.027322</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            Bins        variable     value\n",
              "0  (-0.001, 0.1]  ColPortion_azd  0.183060\n",
              "1     (0.1, 0.2]  ColPortion_azd  0.461749\n",
              "2     (0.2, 0.3]  ColPortion_azd  0.109290\n",
              "3     (0.3, 0.4]  ColPortion_azd  0.010929\n",
              "4     (0.4, 0.5]  ColPortion_azd  0.027322"
            ]
          },
          "metadata": {},
          "execution_count": 66
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Gdjv9iEg10f9"
      },
      "source": [
        "# Save image location\n",
        "nan_col_perc_file = '/content/drive/MyDrive/workspace/arvato_data/nan_col_perc_{}.png'.format(ts) \\\n",
        "if IN_COLAB else 'nan_col_perc_{}.png'.format(ts)\n",
        "\n",
        "with sns.axes_style(\"whitegrid\"):\n",
        "    plt.figure(figsize=(8,5))\n",
        "    sns.barplot(data=cols_nan_portns_melt, x=\"value\", y=\"Bins\", hue=\"variable\", palette=\"Blues_d\").set(title='[AZDIAS, CUSTOMERS] Columns portions per NaN bin')\n",
        "    # copyright text\n",
        "    plt.suptitle(cprt, y=-.03, verticalalignment='bottom', bbox=bb_cprt_args)\n",
        "    plt.tight_layout()\n",
        "    plt.savefig(nan_col_perc_file, transparent=False, bbox_inches=\"tight\")\n",
        "    plt.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4cKl8oIKIEsw"
      },
      "source": [
        "###Rows NaNs analysis\n",
        "\n",
        "* Identify missing values per row as percentage (portions).\n",
        "* Drop rows with all NaNs (zero rows will be dropped):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yU-jD-FHmHD0"
      },
      "source": [
        "def rows_nans_info(df, cols=None):\n",
        "    \"\"\"Return a new dataframe with NaNs summary per row of the input df\n",
        "\n",
        "    Creates and returns a new dataframe with three columns that correspond to\n",
        "    the rows of the input dataframe, the NaN counts per row\n",
        "    and portion of NaNs per row.\n",
        "\n",
        "    Args:\n",
        "        df (pd.DataFrame): The input dataframe to process.\n",
        "        cols (tuple): 3 elements tuple with the names of the three columns in the\n",
        "        output dataframe. If left None the three default names ('Rows',\n",
        "        'NaNcounts', 'NaNportions') will be used.\n",
        "    \n",
        "    Returns:\n",
        "        A new dataframe with three columns providing counts and portions of NaNs\n",
        "        per row in the input df. Example:\n",
        "\n",
        "            Rows    NaNcounts\tNaNportions\n",
        "        0\t0\t    685843\t    0.769554\n",
        "        1\t1       73499\t    0.082470\n",
        "        2\t2\t    310267\t    0.348137\n",
        "        3\t3\t    810163\t    0.909048\n",
        "        4\t4       861722\t    0.966900\n",
        "    \"\"\"\n",
        "    cols = cols or ('Rows', 'NaNcounts', 'NaNportions')\n",
        "    new_df = df.isna().sum(axis=1).rename_axis(cols[0]).reset_index(name=cols[1])\n",
        "    new_df[cols[2]] = new_df[cols[1]] / df.shape[1]\n",
        "\n",
        "    print(\"Average NaNs per row: \", round(new_df[cols[2]].mean(), 4))\n",
        "    print(\"Max NaNs per row: \", round(new_df[cols[2]].max(), 4))\n",
        "    print(\"Min NaNs per row: \", round(new_df[cols[2]].min(), 4))\n",
        "\n",
        "    return new_df"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rwGZI1TiuVwX"
      },
      "source": [
        "* `AZDIAS`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XQH1JJQhpAqB",
        "outputId": "2937e542-6207-4872-c640-760563eff46d"
      },
      "source": [
        "azdias_row_nans = rows_nans_info(azdias)\n",
        "\n",
        "azdias_row_nans.NaNportions.describe()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Average NaNs per row:  0.2803\n",
            "Max NaNs per row:  0.8579\n",
            "Min NaNs per row:  0.071\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "count    300000.000000\n",
              "mean          0.280347\n",
              "std           0.198368\n",
              "min           0.071038\n",
              "25%           0.183060\n",
              "50%           0.213115\n",
              "75%           0.243169\n",
              "max           0.857923\n",
              "Name: NaNportions, dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 69
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R9OlKWVFuZby"
      },
      "source": [
        "* `CUSTOMERS`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "T4Ge91SCldMK",
        "outputId": "5b0cd59f-7cff-4c39-adfc-4d16547012e2"
      },
      "source": [
        "custrs_row_nans = rows_nans_info(custrs)\n",
        "\n",
        "custrs_row_nans.NaNportions.describe()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Average NaNs per row:  0.3558\n",
            "Max NaNs per row:  0.8537\n",
            "Min NaNs per row:  0.0759\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "count    150000.000000\n",
              "mean          0.355753\n",
              "std           0.287805\n",
              "min           0.075881\n",
              "25%           0.165312\n",
              "50%           0.195122\n",
              "75%           0.747967\n",
              "max           0.853659\n",
              "Name: NaNportions, dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 70
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9hFDrCkIrquH"
      },
      "source": [
        "There is on average a 28% missing values per row in `AZDIAS` and around 35% NaNs per row in `CUSTOMERS`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hu35Ukta5tHV"
      },
      "source": [
        "# Save image location\n",
        "nan_rows_distrb_file = '/content/drive/MyDrive/workspace/arvato_data/nan_rows_distrb_{}.png'.format(ts) \\\n",
        "if IN_COLAB else 'nan_rows_distrb_{}.png'.format(ts)\n",
        "\n",
        "with sns.axes_style(\"whitegrid\"):\n",
        "    fig, ax = plt.subplots(1, 2, figsize=(10,5))\n",
        "    sns.histplot(ax=ax[0], data=azdias_row_nans, x=\"NaNportions\", bins=np.linspace(0, 1, 11)).set(title='[AZDIAS] Row counts per NaN portions')\n",
        "    sns.histplot(ax=ax[1], data=custrs_row_nans, x=\"NaNportions\", bins=np.linspace(0, 1, 11)).set(title='[CUSTOMERS] Row counts per NaN portions')\n",
        "    # copyright text\n",
        "    fig.suptitle(cprt, y=-.03, verticalalignment='bottom', bbox=bb_cprt_args)\n",
        "    fig.tight_layout()\n",
        "    fig.savefig(nan_rows_distrb_file, transparent=False, bbox_inches=\"tight\")\n",
        "    plt.close(fig)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PH4GtlQA8Uap"
      },
      "source": [
        "As seen, both datasets have most rows with missing values in the range 10% - 30% per row."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4s8zwqfUptnq"
      },
      "source": [
        "* `AZDIAS`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 359
        },
        "id": "ooDvb_K7lTBj",
        "outputId": "bab8eff3-aebd-4671-ac19-f6687b039e62"
      },
      "source": [
        "azd_row_nan_bins = azdias_row_nans.NaNportions.value_counts(bins=np.linspace(0, 1, 11), sort=False)\n",
        "\n",
        "azd_row_nan_bins = azd_row_nan_bins.rename_axis(\"Bins\").reset_index(name=\"Count\")\n",
        "\n",
        "azd_row_nan_bins[\"RowPortion\"] = azd_row_nan_bins.Count / azdias.shape[0]\n",
        "\n",
        "azd_row_nan_bins"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Bins</th>\n",
              "      <th>Count</th>\n",
              "      <th>RowPortion</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>(-0.001, 0.1]</td>\n",
              "      <td>243</td>\n",
              "      <td>0.000810</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>(0.1, 0.2]</td>\n",
              "      <td>116662</td>\n",
              "      <td>0.388873</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>(0.2, 0.3]</td>\n",
              "      <td>131302</td>\n",
              "      <td>0.437673</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>(0.3, 0.4]</td>\n",
              "      <td>15065</td>\n",
              "      <td>0.050217</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>(0.4, 0.5]</td>\n",
              "      <td>1140</td>\n",
              "      <td>0.003800</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>(0.5, 0.6]</td>\n",
              "      <td>1695</td>\n",
              "      <td>0.005650</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>(0.6, 0.7]</td>\n",
              "      <td>1949</td>\n",
              "      <td>0.006497</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>(0.7, 0.8]</td>\n",
              "      <td>6741</td>\n",
              "      <td>0.022470</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>(0.8, 0.9]</td>\n",
              "      <td>25203</td>\n",
              "      <td>0.084010</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>(0.9, 1.0]</td>\n",
              "      <td>0</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            Bins   Count  RowPortion\n",
              "0  (-0.001, 0.1]     243    0.000810\n",
              "1     (0.1, 0.2]  116662    0.388873\n",
              "2     (0.2, 0.3]  131302    0.437673\n",
              "3     (0.3, 0.4]   15065    0.050217\n",
              "4     (0.4, 0.5]    1140    0.003800\n",
              "5     (0.5, 0.6]    1695    0.005650\n",
              "6     (0.6, 0.7]    1949    0.006497\n",
              "7     (0.7, 0.8]    6741    0.022470\n",
              "8     (0.8, 0.9]   25203    0.084010\n",
              "9     (0.9, 1.0]       0    0.000000"
            ]
          },
          "metadata": {},
          "execution_count": 72
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YmmRkr6dzV3D",
        "outputId": "7cb65f6e-4d67-4cb9-f9ad-620cf437130d"
      },
      "source": [
        "# 20% NaN counts and dataset portion - AZDIAS\n",
        "azd_row_nan_bins.loc[:2,].sum()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Count         248207.000000\n",
              "RowPortion         0.827357\n",
              "dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 73
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MTaGrnrTqPMG"
      },
      "source": [
        "* `CUSTOMERS`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 359
        },
        "id": "YkUdfIzLp7wl",
        "outputId": "ee052ecd-019c-46b9-fc3f-661683c5f416"
      },
      "source": [
        "cus_row_nan_bins = custrs_row_nans.NaNportions.value_counts(bins=np.linspace(0, 1, 11), sort=False)\n",
        "\n",
        "cus_row_nan_bins = cus_row_nan_bins.rename_axis(\"Bins\").reset_index(name=\"Count\")\n",
        "\n",
        "cus_row_nan_bins[\"RowPortion\"] = cus_row_nan_bins.Count / custrs.shape[0]\n",
        "\n",
        "cus_row_nan_bins"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Bins</th>\n",
              "      <th>Count</th>\n",
              "      <th>RowPortion</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>(-0.001, 0.1]</td>\n",
              "      <td>250</td>\n",
              "      <td>0.001667</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>(0.1, 0.2]</td>\n",
              "      <td>80654</td>\n",
              "      <td>0.537693</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>(0.2, 0.3]</td>\n",
              "      <td>24487</td>\n",
              "      <td>0.163247</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>(0.3, 0.4]</td>\n",
              "      <td>4342</td>\n",
              "      <td>0.028947</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>(0.4, 0.5]</td>\n",
              "      <td>234</td>\n",
              "      <td>0.001560</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>(0.5, 0.6]</td>\n",
              "      <td>260</td>\n",
              "      <td>0.001733</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>(0.6, 0.7]</td>\n",
              "      <td>833</td>\n",
              "      <td>0.005553</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>(0.7, 0.8]</td>\n",
              "      <td>2376</td>\n",
              "      <td>0.015840</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>(0.8, 0.9]</td>\n",
              "      <td>36564</td>\n",
              "      <td>0.243760</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>(0.9, 1.0]</td>\n",
              "      <td>0</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            Bins  Count  RowPortion\n",
              "0  (-0.001, 0.1]    250    0.001667\n",
              "1     (0.1, 0.2]  80654    0.537693\n",
              "2     (0.2, 0.3]  24487    0.163247\n",
              "3     (0.3, 0.4]   4342    0.028947\n",
              "4     (0.4, 0.5]    234    0.001560\n",
              "5     (0.5, 0.6]    260    0.001733\n",
              "6     (0.6, 0.7]    833    0.005553\n",
              "7     (0.7, 0.8]   2376    0.015840\n",
              "8     (0.8, 0.9]  36564    0.243760\n",
              "9     (0.9, 1.0]      0    0.000000"
            ]
          },
          "metadata": {},
          "execution_count": 74
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6QjGUX2JgA9d",
        "outputId": "37880551-f93e-480f-b1f6-65b0566b8b26"
      },
      "source": [
        "# 20% NaN counts and dataset portion - CUSTOMERS\n",
        "cus_row_nan_bins.loc[:2,].sum()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Count         105391.000000\n",
              "RowPortion         0.702607\n",
              "dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 75
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GkEPSY3ksGEo"
      },
      "source": [
        "####Format data in long-form for plotting with `sns.barplot`. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "Pxev9AVQqbIS",
        "outputId": "6dfa9212-d7d9-42d1-d801-b2de641ab6cc"
      },
      "source": [
        "merge_row_nan_bins = pd.merge(azd_row_nan_bins, cus_row_nan_bins,\n",
        "                              on='Bins', suffixes=(\"_azd\", \"_cus\"))\n",
        "\n",
        "rows_nan_portns_melt = merge_row_nan_bins[\n",
        "                                          ['Bins', 'RowPortion_azd', 'RowPortion_cus']\n",
        "                                         ].melt(id_vars=('Bins'))\n",
        "\n",
        "rows_nan_portns_melt.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Bins</th>\n",
              "      <th>variable</th>\n",
              "      <th>value</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>(-0.001, 0.1]</td>\n",
              "      <td>RowPortion_azd</td>\n",
              "      <td>0.000810</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>(0.1, 0.2]</td>\n",
              "      <td>RowPortion_azd</td>\n",
              "      <td>0.388873</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>(0.2, 0.3]</td>\n",
              "      <td>RowPortion_azd</td>\n",
              "      <td>0.437673</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>(0.3, 0.4]</td>\n",
              "      <td>RowPortion_azd</td>\n",
              "      <td>0.050217</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>(0.4, 0.5]</td>\n",
              "      <td>RowPortion_azd</td>\n",
              "      <td>0.003800</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            Bins        variable     value\n",
              "0  (-0.001, 0.1]  RowPortion_azd  0.000810\n",
              "1     (0.1, 0.2]  RowPortion_azd  0.388873\n",
              "2     (0.2, 0.3]  RowPortion_azd  0.437673\n",
              "3     (0.3, 0.4]  RowPortion_azd  0.050217\n",
              "4     (0.4, 0.5]  RowPortion_azd  0.003800"
            ]
          },
          "metadata": {},
          "execution_count": 76
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "267iYIqashPU"
      },
      "source": [
        "# Save image location\n",
        "nan_rows_perc_file = '/content/drive/MyDrive/workspace/arvato_data/nan_rows_perc_{}.png'.format(ts) \\\n",
        "if IN_COLAB else 'nan_rows_perc_{}.png'.format(ts)\n",
        "\n",
        "with sns.axes_style(\"whitegrid\"):\n",
        "    plt.figure(figsize=(8,5))\n",
        "    sns.barplot(data=rows_nan_portns_melt, x=\"value\", y=\"Bins\", hue=\"variable\", palette=\"Blues_d\").set(title='[AZDIAS, CUSTOMERS] Rows portions per NaN bin')\n",
        "    # copyright text\n",
        "    plt.suptitle(cprt, y=-.03, verticalalignment='bottom', bbox=bb_cprt_args)\n",
        "    plt.tight_layout()\n",
        "    plt.savefig(nan_rows_perc_file, transparent=False, bbox_inches=\"tight\")\n",
        "    plt.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gxx6Eb3JzqYi"
      },
      "source": [
        "\n",
        "In both datasets most rows have NaNs in the range 10% - 30%.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QRFY80lVsTbv"
      },
      "source": [
        "####Column type and exploration for the attributes with values range `'…'`\n",
        "\n",
        "Finally we have the special cases where the values' pool was `'…'`, described with the `attrs_val_dict_any` dictionary mapping. These include a wide range of numerical values:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IvFtJAUiisMf"
      },
      "source": [
        "* `AZDIAS`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Nf8NKh-ipMPs",
        "outputId": "c07f3d36-f651-4e43-fb7a-a3c0a1641cba"
      },
      "source": [
        "# Check `attrs_val_dict_any` special cases types:\n",
        "azdias.loc[:, attrs_val_dict_any.keys()].dtypes"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "ANZ_HAUSHALTE_AKTIV    float64\n",
              "ANZ_HH_TITEL           float64\n",
              "ANZ_PERSONEN           float64\n",
              "ANZ_TITEL              float64\n",
              "GEBURTSJAHR            float64\n",
              "KBA13_ANZAHL_PKW       float64\n",
              "MIN_GEBAEUDEJAHR       float64\n",
              "dtype: object"
            ]
          },
          "metadata": {},
          "execution_count": 78
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 297
        },
        "id": "P52gpw_Jtn7Q",
        "outputId": "36ccb1ed-79ae-4c98-d9cb-400ccfe83b7d"
      },
      "source": [
        "azdias.loc[:, attrs_val_dict_any.keys()].describe()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ANZ_HAUSHALTE_AKTIV</th>\n",
              "      <th>ANZ_HH_TITEL</th>\n",
              "      <th>ANZ_PERSONEN</th>\n",
              "      <th>ANZ_TITEL</th>\n",
              "      <th>GEBURTSJAHR</th>\n",
              "      <th>KBA13_ANZAHL_PKW</th>\n",
              "      <th>MIN_GEBAEUDEJAHR</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>266258.000000</td>\n",
              "      <td>8166.000000</td>\n",
              "      <td>263595.000000</td>\n",
              "      <td>1077.000000</td>\n",
              "      <td>167731.000000</td>\n",
              "      <td>264109.000000</td>\n",
              "      <td>268489.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>8.395676</td>\n",
              "      <td>1.371296</td>\n",
              "      <td>1.800823</td>\n",
              "      <td>1.073352</td>\n",
              "      <td>1967.093161</td>\n",
              "      <td>618.351438</td>\n",
              "      <td>1993.292340</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>15.846538</td>\n",
              "      <td>1.456124</td>\n",
              "      <td>1.117183</td>\n",
              "      <td>0.274717</td>\n",
              "      <td>17.848186</td>\n",
              "      <td>339.685636</td>\n",
              "      <td>3.358749</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1900.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1985.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>2.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1955.000000</td>\n",
              "      <td>382.000000</td>\n",
              "      <td>1992.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>4.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1967.000000</td>\n",
              "      <td>548.000000</td>\n",
              "      <td>1992.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>10.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1981.000000</td>\n",
              "      <td>776.000000</td>\n",
              "      <td>1993.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>595.000000</td>\n",
              "      <td>23.000000</td>\n",
              "      <td>40.000000</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>2017.000000</td>\n",
              "      <td>2300.000000</td>\n",
              "      <td>2016.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       ANZ_HAUSHALTE_AKTIV  ANZ_HH_TITEL  ...  KBA13_ANZAHL_PKW  MIN_GEBAEUDEJAHR\n",
              "count        266258.000000   8166.000000  ...     264109.000000     268489.000000\n",
              "mean              8.395676      1.371296  ...        618.351438       1993.292340\n",
              "std              15.846538      1.456124  ...        339.685636          3.358749\n",
              "min               1.000000      1.000000  ...          1.000000       1985.000000\n",
              "25%               2.000000      1.000000  ...        382.000000       1992.000000\n",
              "50%               4.000000      1.000000  ...        548.000000       1992.000000\n",
              "75%              10.000000      1.000000  ...        776.000000       1993.000000\n",
              "max             595.000000     23.000000  ...       2300.000000       2016.000000\n",
              "\n",
              "[8 rows x 7 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 79
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xgnuwrZEwsCn"
      },
      "source": [
        "We may observe that values for the above attributes look acceptable."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RA1in0-vxcGA",
        "outputId": "beb67bbe-e566-4157-cbfa-d14832cbf7e9"
      },
      "source": [
        "# Count unique values per attribute (column)\n",
        "azdias.loc[:, attrs_val_dict_any.keys()].nunique()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "ANZ_HAUSHALTE_AKTIV     269\n",
              "ANZ_HH_TITEL             20\n",
              "ANZ_PERSONEN             19\n",
              "ANZ_TITEL                 4\n",
              "GEBURTSJAHR             115\n",
              "KBA13_ANZAHL_PKW       1260\n",
              "MIN_GEBAEUDEJAHR         32\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 80
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kvr5HOBQymPd",
        "outputId": "e599c619-bab7-4cc4-dec1-151dcbe21446"
      },
      "source": [
        "# Discrete unique values per column\n",
        "azdias.loc[:, attrs_val_dict_any.keys()].apply(lambda col: col.unique())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "ANZ_HAUSHALTE_AKTIV    [1.0, 4.0, 10.0, 8.0, 2.0, 9.0, 80.0, 11.0, na...\n",
              "ANZ_HH_TITEL           [nan, 1.0, 3.0, 2.0, 7.0, 8.0, 10.0, 4.0, 13.0...\n",
              "ANZ_PERSONEN           [3.0, 4.0, 2.0, 1.0, nan, 9.0, 5.0, 6.0, 7.0, ...\n",
              "ANZ_TITEL                                      [nan, 1.0, 2.0, 3.0, 4.0]\n",
              "GEBURTSJAHR            [nan, 1946.0, 1988.0, 2005.0, 1985.0, 1961.0, ...\n",
              "KBA13_ANZAHL_PKW       [539.0, 578.0, 1179.0, 563.0, 507.0, 330.0, 38...\n",
              "MIN_GEBAEUDEJAHR       [1994.0, 1992.0, 1993.0, 1995.0, nan, 1996.0, ...\n",
              "dtype: object"
            ]
          },
          "metadata": {},
          "execution_count": 81
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-cDc1oZjiwx0"
      },
      "source": [
        "* `CUSTOMERS`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "99JzwJ8Piy-k",
        "outputId": "c6ff05c1-11b3-401c-8bb0-b9ee7554a64f"
      },
      "source": [
        "# Check `attrs_val_dict_any` special cases types:\n",
        "custrs.loc[:, attrs_val_dict_any.keys()].dtypes"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "ANZ_HAUSHALTE_AKTIV    float64\n",
              "ANZ_HH_TITEL           float64\n",
              "ANZ_PERSONEN           float64\n",
              "ANZ_TITEL              float64\n",
              "GEBURTSJAHR            float64\n",
              "KBA13_ANZAHL_PKW       float64\n",
              "MIN_GEBAEUDEJAHR       float64\n",
              "dtype: object"
            ]
          },
          "metadata": {},
          "execution_count": 82
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 297
        },
        "id": "zRayo-nki8df",
        "outputId": "012969b7-7ccc-400b-a19d-5612d5246203"
      },
      "source": [
        "custrs.loc[:, attrs_val_dict_any.keys()].describe()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ANZ_HAUSHALTE_AKTIV</th>\n",
              "      <th>ANZ_HH_TITEL</th>\n",
              "      <th>ANZ_PERSONEN</th>\n",
              "      <th>ANZ_TITEL</th>\n",
              "      <th>GEBURTSJAHR</th>\n",
              "      <th>KBA13_ANZAHL_PKW</th>\n",
              "      <th>MIN_GEBAEUDEJAHR</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>108992.000000</td>\n",
              "      <td>4779.000000</td>\n",
              "      <td>107914.000000</td>\n",
              "      <td>2170.000000</td>\n",
              "      <td>76998.000000</td>\n",
              "      <td>109830.000000</td>\n",
              "      <td>110885.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>5.030690</td>\n",
              "      <td>1.528144</td>\n",
              "      <td>2.387429</td>\n",
              "      <td>1.076498</td>\n",
              "      <td>1949.740500</td>\n",
              "      <td>666.902841</td>\n",
              "      <td>1993.052126</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>14.246347</td>\n",
              "      <td>2.065431</td>\n",
              "      <td>1.324896</td>\n",
              "      <td>0.284291</td>\n",
              "      <td>14.909137</td>\n",
              "      <td>340.248578</td>\n",
              "      <td>3.062568</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1900.000000</td>\n",
              "      <td>5.000000</td>\n",
              "      <td>1985.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1939.000000</td>\n",
              "      <td>430.000000</td>\n",
              "      <td>1992.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1948.000000</td>\n",
              "      <td>593.000000</td>\n",
              "      <td>1992.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>4.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>3.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1960.000000</td>\n",
              "      <td>828.000000</td>\n",
              "      <td>1992.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>395.000000</td>\n",
              "      <td>23.000000</td>\n",
              "      <td>21.000000</td>\n",
              "      <td>5.000000</td>\n",
              "      <td>2017.000000</td>\n",
              "      <td>2300.000000</td>\n",
              "      <td>2016.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       ANZ_HAUSHALTE_AKTIV  ANZ_HH_TITEL  ...  KBA13_ANZAHL_PKW  MIN_GEBAEUDEJAHR\n",
              "count        108992.000000   4779.000000  ...     109830.000000     110885.000000\n",
              "mean              5.030690      1.528144  ...        666.902841       1993.052126\n",
              "std              14.246347      2.065431  ...        340.248578          3.062568\n",
              "min               1.000000      1.000000  ...          5.000000       1985.000000\n",
              "25%               1.000000      1.000000  ...        430.000000       1992.000000\n",
              "50%               1.000000      1.000000  ...        593.000000       1992.000000\n",
              "75%               4.000000      1.000000  ...        828.000000       1992.000000\n",
              "max             395.000000     23.000000  ...       2300.000000       2016.000000\n",
              "\n",
              "[8 rows x 7 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 83
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4po6xQp_jG06",
        "outputId": "5a3277a0-66a5-4cd3-aee2-73e0e4188bde"
      },
      "source": [
        "# Count unique values per attribute (column)\n",
        "custrs.loc[:, attrs_val_dict_any.keys()].nunique()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "ANZ_HAUSHALTE_AKTIV     209\n",
              "ANZ_HH_TITEL             19\n",
              "ANZ_PERSONEN             16\n",
              "ANZ_TITEL                 4\n",
              "GEBURTSJAHR             112\n",
              "KBA13_ANZAHL_PKW       1246\n",
              "MIN_GEBAEUDEJAHR         32\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 84
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vS1isDPOjLXn",
        "outputId": "d34fd382-8cfd-4c29-a354-4d65e825e0dc"
      },
      "source": [
        "# Discrete unique values per column\n",
        "custrs.loc[:, attrs_val_dict_any.keys()].apply(lambda col: col.unique())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "ANZ_HAUSHALTE_AKTIV    [1.0, nan, 27.0, 7.0, 74.0, 2.0, 9.0, 29.0, 14...\n",
              "ANZ_HH_TITEL           [nan, 2.0, 1.0, 13.0, 5.0, 4.0, 20.0, 17.0, 9....\n",
              "ANZ_PERSONEN           [2.0, 3.0, nan, 4.0, 1.0, 5.0, 7.0, 6.0, 8.0, ...\n",
              "ANZ_TITEL                                      [nan, 1.0, 2.0, 3.0, 5.0]\n",
              "GEBURTSJAHR            [nan, 1979.0, 1960.0, 1942.0, 1971.0, 1914.0, ...\n",
              "KBA13_ANZAHL_PKW       [1201.0, nan, 428.0, 513.0, 1300.0, 422.0, 110...\n",
              "MIN_GEBAEUDEJAHR       [1992.0, nan, 1994.0, 1997.0, 1995.0, 1996.0, ...\n",
              "dtype: object"
            ]
          },
          "metadata": {},
          "execution_count": 85
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JHl1KCtMkkhp"
      },
      "source": [
        "## Identifying (if any) columns to drop\n",
        "\n",
        "At this point we will proceed concatenating the two samples in a single dataframe, creating a new feature `TARGET` that will be `1` for customers and `-1` for unknown.\n",
        "\n",
        "This will combine attribute characteristics from both datasets and help us make decisions best on both samples.\n",
        "\n",
        "First let us check if there are any common user IDs in LNR columns:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "alC6YufSIpVe",
        "outputId": "bf77301a-2552-4edb-a055-d580abcdeca8"
      },
      "source": [
        "np.intersect1d(azdias.LNR.values, custrs.LNR.values)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([], dtype=int64)"
            ]
          },
          "metadata": {},
          "execution_count": 86
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4nTnKw9QKdrG"
      },
      "source": [
        "Concatenate the dataframes:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HXPVl039KibG",
        "outputId": "a7c15902-5bc4-41e0-d04e-39bc4eee2a08"
      },
      "source": [
        "# Create a new dataframe from `azdias` and `customers` dataframes creating a `target` attribute:\n",
        "df_train = pd.concat([azdias.assign(TARGET=-1),\n",
        "                      custrs.drop(columns=[\"CUSTOMER_GROUP\", \"ONLINE_PURCHASE\", \"PRODUCT_GROUP\"])\n",
        "                      .assign(TARGET=1)])\n",
        "\n",
        "# Shuffle dataframe\n",
        "df_train = df_train.sample(frac=1).reset_index(drop=True)\n",
        "\n",
        "# Keep \"LNR\" IDs\n",
        "clientIDs = df_train[\"LNR\"]\n",
        "\n",
        "# Remove \"LNR\" and \"EINGEFUEGT_AM\" columns which are client IDs and entry datetimes\n",
        "df_train.drop(columns=[\"LNR\", \"EINGEFUEGT_AM\"], inplace=True)\n",
        "\n",
        "df_train.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(450000, 365)"
            ]
          },
          "metadata": {},
          "execution_count": 87
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0NUYUSt2MuLt",
        "outputId": "078b0938-308f-4f77-dc47-6075f594f1be"
      },
      "source": [
        "df_train.dtypes.value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "float64     357\n",
              "int64         5\n",
              "category      1\n",
              "category      1\n",
              "category      1\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 88
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AQQalTZFe7Dx"
      },
      "source": [
        "# Convert float columns to int.\n",
        "df_train_int_cols = df_train.select_dtypes(include=['float64']).fillna(-1.0).astype(int)\n",
        "df_train.loc[:, df_train_int_cols.columns] = df_train_int_cols"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4eIj7QSkfg91",
        "outputId": "2b04ec5a-3efb-481f-d4ed-63ec81843db4"
      },
      "source": [
        "df_train.dtypes.value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "int64       362\n",
              "category      1\n",
              "category      1\n",
              "category      1\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 90
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zFbsHASkJbQM"
      },
      "source": [
        "### One-hot encoding\n",
        "\n",
        "One-hot encode categorical attributes `CAMEO_DEU_2015, D19_LETZTER_KAUF_BRANCHE, OST_WEST_KZ`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "KJF4WWxnN1iq",
        "outputId": "a30b532f-331f-4f47-a9b1-e7935a23c380"
      },
      "source": [
        "df_train.select_dtypes(include='category').head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>CAMEO_DEU_2015</th>\n",
              "      <th>D19_LETZTER_KAUF_BRANCHE</th>\n",
              "      <th>OST_WEST_KZ</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>6F</td>\n",
              "      <td>D19_VERSICHERUNGEN</td>\n",
              "      <td>W</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>5F</td>\n",
              "      <td>NaN</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>4B</td>\n",
              "      <td>D19_SONSTIGE</td>\n",
              "      <td>W</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1D</td>\n",
              "      <td>D19_UNBEKANNT</td>\n",
              "      <td>W</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  CAMEO_DEU_2015 D19_LETZTER_KAUF_BRANCHE OST_WEST_KZ\n",
              "0             6F       D19_VERSICHERUNGEN           W\n",
              "1             5F                      NaN           O\n",
              "2             4B             D19_SONSTIGE           W\n",
              "3             1D            D19_UNBEKANNT           W\n",
              "4            NaN                      NaN         NaN"
            ]
          },
          "metadata": {},
          "execution_count": 91
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zgX9vrcnkoW1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0febe430-8ea0-41a4-b3f0-8dccc2857e52"
      },
      "source": [
        "df_train = pd.get_dummies(df_train)\n",
        "\n",
        "df_train.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(450000, 443)"
            ]
          },
          "metadata": {},
          "execution_count": 92
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zRiU-CQLosIP"
      },
      "source": [
        "irrelevant features, highly correlated features, and missing values can prevent the model from learning and decrease generalization performance on the testing data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x6zhEN80ow9m"
      },
      "source": [
        "## Feature selection to keep only the most useful attributes\n",
        "\n",
        "* Start with **collinear** attributes\n",
        "(Ref: https://chrisalbon.com/code/machine_learning/feature_selection/drop_highly_correlated_features/)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CYq3EONpwRSG"
      },
      "source": [
        "Calculate correlation on a stratified sample of the data, since `azdias` dataframe is large enough that notebook runs out of memory while calculating the correlation matrix."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SqpHOJZx23Ef"
      },
      "source": [
        "def get_stratified_samples(df, fracs=np.array([0.2, 0.8])):\n",
        "    \"\"\"Split an input df into two stratified subsets of sizes indicated\n",
        "    by the fracs array.\n",
        "    \"\"\"\n",
        "    # shuffle input df\n",
        "    df_ = df.sample(frac=1) \n",
        "    # split into 2 parts, return df_smpl_a, df_smpl_b\n",
        "    return np.array_split(df_, (fracs[:-1].cumsum() * len(df_)).astype(int))\n",
        "\n",
        "# azd_cp_smpl_a, azd_cp_smpl_b = get_stratified_samples(azd_cp)\n",
        "\n",
        "# azd_cp_smpl_a.shape, azd_cp_smpl_b.shape"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qXQ7xITAAIEy"
      },
      "source": [
        "def drop_collinear_cols(df, threshold=0.95):\n",
        "    \"\"\"Ιdentifies the highly correlated variables based on the absolute\n",
        "    magnitude of the Pearson correlation coefficient being greater than threshold (0.9)\n",
        "\n",
        "    Args:\n",
        "        df (pd.Dataframe): Input dataframe to work on\n",
        "        threshold (float): threshold value above which the columns will be selected\n",
        "    \n",
        "    Returns:\n",
        "        np.array of selected columns based on the threshold value.\n",
        "    \"\"\"\n",
        "    # Calculate absolute value correlation matrix from the input df.\n",
        "    # corr_mat = df.corr().abs()\n",
        "    corr_mat = np.abs(np.corrcoef(df, rowvar=False))\n",
        "    # Convert to dataframe\n",
        "    corr_mat = pd.DataFrame(corr_mat, index=df.columns, columns=df.columns)\n",
        "    # Select upper triangle of correlation matrix\n",
        "    upr = corr_mat.where(np.triu(np.ones(corr_mat.shape), k=1).astype(np.bool))\n",
        "    # Find index of attribute columns with correlation greater than `threshold`\n",
        "    return np.array([c for c in upr.columns if any(upr[c] > threshold)])\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2QfKzZJGPor4",
        "outputId": "bed87452-b537-48a0-cd86-f45cb365a235"
      },
      "source": [
        "# Find attribute columns with correlation greater than 0.95\n",
        "to_drop = drop_collinear_cols(df_train)\n",
        "\n",
        "to_drop, len(to_drop)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array(['ANZ_STATISTISCHE_HAUSHALTE', 'CAMEO_INTL_2015', 'KBA05_KRSHERST2',\n",
              "        'KBA05_KRSHERST3', 'KBA05_SEG2', 'KBA13_HALTER_30',\n",
              "        'KBA13_HALTER_40', 'KBA13_HALTER_55', 'KBA13_HALTER_66',\n",
              "        'KBA13_HERST_SONST', 'KBA13_KMH_250', 'KBA13_MERCEDES',\n",
              "        'KBA13_SEG_VAN', 'KBA13_VW', 'LP_FAMILIE_GROB',\n",
              "        'LP_LEBENSPHASE_GROB', 'LP_STATUS_GROB', 'PLZ8_BAUMAX', 'PLZ8_GBZ',\n",
              "        'PLZ8_HHZ'], dtype='<U26'), 20)"
            ]
          },
          "metadata": {},
          "execution_count": 95
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kmYZXXzGmpTa",
        "outputId": "1a7a6db2-0f9c-4bab-ce17-9dc4a74e2590"
      },
      "source": [
        "df_train.drop(columns=to_drop, inplace=True)\n",
        "\n",
        "df_train.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(450000, 423)"
            ]
          },
          "metadata": {},
          "execution_count": 101
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KzBDP0AHnmA5"
      },
      "source": [
        "###Search for more columns with high NaN ratio, check their significance.\n",
        "\n",
        "References:\n",
        "* https://www.kaggle.com/questions-and-answers/131935\n",
        "* https://stats.stackexchange.com/questions/221332/variance-of-a-distribution-of-multi-level-categorical-data\n",
        "\n",
        "Check attributes with NaN ratio more than **0.8**."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 470
        },
        "id": "dm8b90HXpw8l",
        "outputId": "ca9ef3c5-c6f8-40bb-9a63-e40a7c9dc0c4"
      },
      "source": [
        "df_train_nans = cols_nans_info(df_train.replace(-1, np.NaN))\n",
        "\n",
        "df_train_nans"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Average NaNs per column:  0.256\n",
            "Max NaNs per column:  0.9991\n",
            "Min NaNs per column:  0.0\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Columns</th>\n",
              "      <th>NaNcounts</th>\n",
              "      <th>NaNportions</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>AGER_TYP</td>\n",
              "      <td>306555</td>\n",
              "      <td>0.681233</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>AKT_DAT_KL</td>\n",
              "      <td>61385</td>\n",
              "      <td>0.136411</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>ALTER_HH</td>\n",
              "      <td>158634</td>\n",
              "      <td>0.352520</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ALTER_KIND1</td>\n",
              "      <td>413670</td>\n",
              "      <td>0.919267</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ALTER_KIND2</td>\n",
              "      <td>436210</td>\n",
              "      <td>0.969356</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>418</th>\n",
              "      <td>D19_LETZTER_KAUF_BRANCHE_D19_VERSICHERUNGEN</td>\n",
              "      <td>0</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>419</th>\n",
              "      <td>D19_LETZTER_KAUF_BRANCHE_D19_VOLLSORTIMENT</td>\n",
              "      <td>0</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>420</th>\n",
              "      <td>D19_LETZTER_KAUF_BRANCHE_D19_WEIN_FEINKOST</td>\n",
              "      <td>0</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>421</th>\n",
              "      <td>OST_WEST_KZ_O</td>\n",
              "      <td>0</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>422</th>\n",
              "      <td>OST_WEST_KZ_W</td>\n",
              "      <td>0</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>423 rows × 3 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                         Columns  NaNcounts  NaNportions\n",
              "0                                       AGER_TYP     306555     0.681233\n",
              "1                                     AKT_DAT_KL      61385     0.136411\n",
              "2                                       ALTER_HH     158634     0.352520\n",
              "3                                    ALTER_KIND1     413670     0.919267\n",
              "4                                    ALTER_KIND2     436210     0.969356\n",
              "..                                           ...        ...          ...\n",
              "418  D19_LETZTER_KAUF_BRANCHE_D19_VERSICHERUNGEN          0     0.000000\n",
              "419   D19_LETZTER_KAUF_BRANCHE_D19_VOLLSORTIMENT          0     0.000000\n",
              "420   D19_LETZTER_KAUF_BRANCHE_D19_WEIN_FEINKOST          0     0.000000\n",
              "421                                OST_WEST_KZ_O          0     0.000000\n",
              "422                                OST_WEST_KZ_W          0     0.000000\n",
              "\n",
              "[423 rows x 3 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 102
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "McGEhcx9qskK",
        "outputId": "916defcf-4d98-414e-e3a0-b8a4dcaeeb20"
      },
      "source": [
        "# Select the columns with NaNs more than 80% to check for variability\n",
        "df_train_nans_gt80 = df_train_nans.loc[(df_train_nans[\"NaNportions\"] > .8), \"Columns\"].values\n",
        "\n",
        "df_train_nans_gt80, df_train_nans_gt80.size"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array(['ALTER_KIND1', 'ALTER_KIND2', 'ALTER_KIND3', 'ALTER_KIND4',\n",
              "        'ANZ_HH_TITEL', 'ANZ_KINDER', 'ANZ_TITEL', 'D19_BANKEN_ANZ_12',\n",
              "        'D19_BANKEN_ANZ_24', 'D19_BANKEN_DIREKT', 'D19_BANKEN_GROSS',\n",
              "        'D19_BANKEN_LOKAL', 'D19_BANKEN_ONLINE_QUOTE_12',\n",
              "        'D19_BANKEN_REST', 'D19_BEKLEIDUNG_GEH', 'D19_BILDUNG',\n",
              "        'D19_BIO_OEKO', 'D19_DIGIT_SERV', 'D19_DROGERIEARTIKEL',\n",
              "        'D19_ENERGIE', 'D19_FREIZEIT', 'D19_GARTEN', 'D19_HANDWERK',\n",
              "        'D19_KINDERARTIKEL', 'D19_KOSMETIK', 'D19_LEBENSMITTEL',\n",
              "        'D19_NAHRUNGSERGAENZUNG', 'D19_RATGEBER', 'D19_SAMMELARTIKEL',\n",
              "        'D19_SCHUHE', 'D19_TELKO_ANZ_12', 'D19_TELKO_ANZ_24',\n",
              "        'D19_TELKO_MOBILE', 'D19_TELKO_ONLINE_QUOTE_12', 'D19_TELKO_REST',\n",
              "        'D19_TIERARTIKEL', 'D19_VERSAND_REST', 'D19_VERSI_ANZ_12',\n",
              "        'D19_VERSI_ANZ_24', 'D19_VERSI_ONLINE_QUOTE_12',\n",
              "        'D19_WEIN_FEINKOST', 'HH_DELTA_FLAG', 'KBA05_ANTG4', 'KBA05_SEG6',\n",
              "        'KONSUMZELLE', 'SOHO_KZ', 'TITEL_KZ', 'UNGLEICHENN_FLAG'],\n",
              "       dtype=object), 48)"
            ]
          },
          "metadata": {},
          "execution_count": 103
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pZi7T3KkrFUC",
        "outputId": "da2b97e1-7e39-46bd-c55c-0f0b9c69f3b1"
      },
      "source": [
        "df_train.filter(df_train_nans_gt80).dtypes.value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "int64    48\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 104
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MicCKx4VRw4y"
      },
      "source": [
        "We will calculate the **entropy** on the above (ordinal variables) as a measure of the amount of information in a categorical variable. We will attempt to identify columns with little to no information at all.\n",
        "\n",
        "To do this we will create a function to calculate the entropy in a given column."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wzGWpU4AH3mu"
      },
      "source": [
        "def get_entropy(labels):\n",
        "    \"\"\"Calculate the entropy of a data series.\n",
        "\n",
        "    Calculate the entropy based on Shannon's entropy (log2).\n",
        "    Scipy entropy() will normalize values' frequencies if\n",
        "    they don’t sum to 1.\n",
        "\n",
        "    Args:\n",
        "        labels (pd.Series): pandas Series to calculate entropy\n",
        "    \n",
        "    Returns:\n",
        "        ent: The calculated entropy of the input labels\n",
        "    \"\"\"\n",
        "    # get values' frequencies\n",
        "    vals_freq = labels.value_counts(sort=False)\n",
        "    ent = entropy(vals_freq, base=2)\n",
        "    \n",
        "    return ent"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ELI9YWHMxsqq"
      },
      "source": [
        "A mapping of attribute names (columns) and entropy can be created. This will allow us to decide if we should drop some more columns with a high percentage of NaNs that also offer little to no information."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WLMdidX3Ty4J"
      },
      "source": [
        "def drop_nan_cols(df, df_nans_cols, nan_thres=.8, use_entropy=True, ent_thres=1.0):\n",
        "    \"\"\"Get df columns with higher NaN percent than  nan_thres (> 80%) and return\n",
        "    the ones with entropy < ent_thres. If use_entropy=False return all columns\n",
        "    with NaN portion higher than nan_thres.\n",
        "\n",
        "    Returns:\n",
        "        drop_cols (np.array): Higher percentage NaN columns list (with low entropy)\n",
        "        ent (dict): Mapping of attributes names with entropy values for selected\n",
        "        columns range (or None if entropy is not used). \n",
        "    \"\"\"\n",
        "    # Select the columns with NaNs more than nan_thres to check for variability\n",
        "    df_nans_cols_gt80 = df_nans_cols.loc[(df_nans_cols[\"NaNportions\"] > nan_thres),\n",
        "                                         \"Columns\"].values\n",
        "    # Filter the dataset to select the above columns only.\n",
        "    df_var_nan80 = df.filter(df_nans_cols_gt80)\n",
        "    drop_cols = df_var_nan80.values\n",
        "    ent = None\n",
        "    if use_entropy:\n",
        "        # Calculate entropy & create mapping\n",
        "        ent = {i: round(get_entropy(df_var_nan80[i]), 4) for i in df_var_nan80}\n",
        "        # From the dictionary we will select the columns with entropy less than ent_thres\n",
        "        df_ent_drop = [i for i in ent if ent[i] < ent_thres]\n",
        "        drop_cols = np.array(df_ent_drop)\n",
        "\n",
        "    return drop_cols, ent\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mW5cyXyssQV7",
        "outputId": "7d490720-9275-416c-aade-5a46faa2e64e"
      },
      "source": [
        "# Get df_train columns with higher NaN percent (> 80%) and return the ones with entropy < 1.0\n",
        "df_train_ent_drop, df_train_var_nan80_ent = drop_nan_cols(df_train, df_train_nans)\n",
        "\n",
        "df_train_ent_drop, len(df_train_ent_drop)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array(['ALTER_KIND1', 'ALTER_KIND2', 'ALTER_KIND3', 'ALTER_KIND4',\n",
              "        'ANZ_HH_TITEL', 'ANZ_KINDER', 'ANZ_TITEL', 'D19_BANKEN_ANZ_12',\n",
              "        'D19_BANKEN_ANZ_24', 'D19_BANKEN_GROSS', 'D19_BANKEN_LOKAL',\n",
              "        'D19_BANKEN_ONLINE_QUOTE_12', 'D19_BANKEN_REST',\n",
              "        'D19_BEKLEIDUNG_GEH', 'D19_BILDUNG', 'D19_BIO_OEKO',\n",
              "        'D19_DIGIT_SERV', 'D19_DROGERIEARTIKEL', 'D19_ENERGIE',\n",
              "        'D19_FREIZEIT', 'D19_GARTEN', 'D19_HANDWERK', 'D19_KINDERARTIKEL',\n",
              "        'D19_KOSMETIK', 'D19_LEBENSMITTEL', 'D19_NAHRUNGSERGAENZUNG',\n",
              "        'D19_RATGEBER', 'D19_SAMMELARTIKEL', 'D19_SCHUHE',\n",
              "        'D19_TELKO_ANZ_12', 'D19_TELKO_ANZ_24', 'D19_TELKO_MOBILE',\n",
              "        'D19_TELKO_ONLINE_QUOTE_12', 'D19_TELKO_REST', 'D19_TIERARTIKEL',\n",
              "        'D19_VERSI_ANZ_12', 'D19_VERSI_ANZ_24',\n",
              "        'D19_VERSI_ONLINE_QUOTE_12', 'D19_WEIN_FEINKOST', 'HH_DELTA_FLAG',\n",
              "        'KBA05_ANTG4', 'KBA05_SEG6', 'KONSUMZELLE', 'SOHO_KZ', 'TITEL_KZ',\n",
              "        'UNGLEICHENN_FLAG'], dtype='<U26'), 46)"
            ]
          },
          "metadata": {},
          "execution_count": 107
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "yi7Pq2d4vF6m",
        "outputId": "a68f2a9a-34a9-4ae2-d74b-25d008ba4e15"
      },
      "source": [
        "# Create plot data, in a new df get 'Attributes', 'Entropy' and \"NaNportions\"\n",
        "plt_data = pd.DataFrame(df_train_var_nan80_ent.items(), columns=['Attributes', 'Entropy'])\n",
        "\n",
        "# Get NaN portions for the attributes with NaNs more than 80%\n",
        "plt_data[\"NaNportions\"] = df_train_nans.loc[df_train_nans[\"Columns\"]\n",
        "                                            .isin(df_train_nans_gt80),\"NaNportions\"].values\n",
        "\n",
        "# Sort by `Entropy` values\n",
        "plt_data.sort_values(\"Entropy\", ascending=False, inplace=True)\n",
        "\n",
        "plt_data.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Attributes</th>\n",
              "      <th>Entropy</th>\n",
              "      <th>NaNportions</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>36</th>\n",
              "      <td>D19_VERSAND_REST</td>\n",
              "      <td>1.0198</td>\n",
              "      <td>0.829644</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>D19_BANKEN_DIREKT</td>\n",
              "      <td>1.0002</td>\n",
              "      <td>0.835387</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>D19_KINDERARTIKEL</td>\n",
              "      <td>0.9998</td>\n",
              "      <td>0.828273</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>D19_DROGERIEARTIKEL</td>\n",
              "      <td>0.9767</td>\n",
              "      <td>0.849622</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32</th>\n",
              "      <td>D19_TELKO_MOBILE</td>\n",
              "      <td>0.9498</td>\n",
              "      <td>0.821413</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "             Attributes  Entropy  NaNportions\n",
              "36     D19_VERSAND_REST   1.0198     0.829644\n",
              "9     D19_BANKEN_DIREKT   1.0002     0.835387\n",
              "23    D19_KINDERARTIKEL   0.9998     0.828273\n",
              "18  D19_DROGERIEARTIKEL   0.9767     0.849622\n",
              "32     D19_TELKO_MOBILE   0.9498     0.821413"
            ]
          },
          "metadata": {},
          "execution_count": 109
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HxjUUuTbBamP"
      },
      "source": [
        "# Save image location\n",
        "nan_perc_entr_file = f'{colab_dir if IN_COLAB else remote_dir}nan_perc_entr_{ts}.png'\n",
        "\n",
        "# Let's visualize the results\n",
        "with sns.axes_style(\"whitegrid\"), sns.plotting_context(\"paper\", rc={\"font.size\": 8,}):\n",
        "    fig, ax = plt.subplots(1, 2, figsize=(12, 7), sharey=True)\n",
        "    sns.barplot(ax=ax[0], data=plt_data, x='Entropy', y='Attributes', color='royalblue').set(\n",
        "        title='[AZDIAS] Entropy for high percentage (> 0.8) NaN attributes')\n",
        "    sns.barplot(ax=ax[1], data=plt_data, x='NaNportions', y='Attributes', color='cornflowerblue').set(\n",
        "        title='[AZDIAS] Attributes NaN portions (0.8 min)', xlim=(0.8,1.0))\n",
        "    # copyright text\n",
        "    fig.suptitle(cprt, y=-.03, verticalalignment='bottom', bbox=bb_cprt_args)\n",
        "    fig.tight_layout()\n",
        "    fig.savefig(nan_perc_entr_file, transparent=False, bbox_inches=\"tight\")\n",
        "    plt.close(fig)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EajCR8mE2NBO"
      },
      "source": [
        "Plots show that some of the attributes found with the lowest entropy also have a high number of NaNs. Checking entropy as a measure of information per column will allow us to drop those columns with a high percentage of NaNs, with more confidence than just choosing to drop columns above a high NaN percentage threshold."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x53vxMzraOo3"
      },
      "source": [
        "###Drop previously identified attributes\n",
        "\n",
        "* Drop collinear attributes in `to_drop` array.\n",
        "* Drop attributes in `azd_ent_drop` with NaNs more than 80%, also identified as having the lowest entropy among the rest with such a high NaN percentage.\n",
        "* Drop `CUSTOMERS` specific attributes (`'CUSTOMER_GROUP', 'ONLINE_PURCHASE', and 'PRODUCT_GROUP'`).\n",
        "* Drop the datetime `EINGEFUEGT_AM`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aRzwWxQEwr4m",
        "outputId": "16edb8b4-17b9-4f58-9ccd-7f017fc8397b"
      },
      "source": [
        "df_train.drop(columns=df_train_ent_drop, inplace=True)\n",
        "\n",
        "df_train.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(450000, 377)"
            ]
          },
          "metadata": {},
          "execution_count": 111
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SMkE0JZEUu14"
      },
      "source": [
        "At this point all the previous steps can be aggregated in a single function to help us manipulate datasets in the following steps. We will include `zero_imp_attrs` as an optional parameter that will be used in the following analysis (in parts 2 and 3)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XiU8dBOGWgn3"
      },
      "source": [
        "def wrangle_df(df, val_pools, common_attrs, replace_nan, zero_imp_attrs=None):\n",
        "    \"\"\"Clean and fix an input demographics dataframe for this project.\n",
        "\n",
        "    This function is for cleaning and fixing datasets for this project. It\n",
        "    returns a clean dataframe with several columns dropped based on identified\n",
        "    NaNs, entropy and  collinearity. Additionally, values per column are checked\n",
        "    and replaced with NaNs if they fall outside an accepted values pool.\n",
        "\n",
        "    Args:\n",
        "        df (pd.Dataframe): Input demographics datafrme (The notebook crashes when\n",
        "            the entire `azdias` is loaded).\n",
        "        val_pools (dict): A dictionary with keys the attributes names and values\n",
        "            lists with the allowed values for each attribute. e.g.:\n",
        "            {'AGER_TYP': [nan, -1, 0, 1, 2, 3], ...}\n",
        "        common_attrs (np.array): Common attributes (attrs_common) between\n",
        "            demographic dataset and excel files. e.g.:\n",
        "            (array(['AKT_DAT_KL', 'ALTERSKATEGORIE_FEIN', ...)]\n",
        "        replace_nan (dict): Mapping of attributes and values to replace with NaNs\n",
        "            within each attribute. e.g. {'AGER_TYP': 9} will replace 9 in 'AGER_TYP'\n",
        "            with NaN.\n",
        "        zero_imp_attrs (array like or None): `zero_val_attrs` computed zero\n",
        "            importance attributes. If provided these will be dropped from the\n",
        "            input df.\n",
        "    \n",
        "    Returns:\n",
        "        df (pd.Dataframe): The cleaned and fixed df.\n",
        "    \"\"\"\n",
        "    # Keep \"LNR\" IDs\n",
        "    clientIDs = df[\"LNR\"]\n",
        "\n",
        "    # Drop `customer` specific attributes.\n",
        "    other_to_drop = np.array([\"CUSTOMER_GROUP\", \"ONLINE_PURCHASE\", \"PRODUCT_GROUP\"])\n",
        "\n",
        "    # Remove \"LNR\" and \"EINGEFUEGT_AM\" columns which are client IDs and entry datetimes\n",
        "    df.drop(columns=np.union1d([\"LNR\", \"EINGEFUEGT_AM\"], other_to_drop),\n",
        "            errors='ignore', inplace=True)\n",
        "\n",
        "    # find object columns\n",
        "    cat_cols = df.select_dtypes(include=object).columns.tolist()\n",
        "    # Convert to category columns\n",
        "    df[cat_cols] = df[cat_cols].astype('category')\n",
        "\n",
        "    # Replace unknown and specific values with NaNs\n",
        "    # for every key (attribute) in the dictionary created earlier\n",
        "    for k in val_pools:\n",
        "        # if the attribute is among the common ones\n",
        "        if k in common_attrs:\n",
        "            # select all those rows in the column that fall outside the attritbute's specified values pool and replace them with NaN.\n",
        "            df.loc[~df[k].isin(val_pools[k]), k] = np.NaN\n",
        "    # Use a dictionary mapping of attributes to replace value 9 (unknown) for NaN for specific attributes.\n",
        "    df.replace(replace_nan, np.NaN, inplace=True)\n",
        "\n",
        "    # NaNs\n",
        "    # Total\n",
        "    print(\"Total df NaNs portion: \", round(df.isna().sum().sum() / df.size, 4))\n",
        "    # Columns\n",
        "    print(\"Column NaNs:\")\n",
        "    df_cols_nans = cols_nans_info(df)\n",
        "    # Rows\n",
        "    print(\"Row NaNs:\")\n",
        "    df_rows_nans = rows_nans_info(df)\n",
        "\n",
        "    # Convert float columns to int.\n",
        "    df_int_cols = df.select_dtypes(include=['float64']).fillna(-1.0).astype(int)\n",
        "    df.loc[:, df_int_cols.columns] = df_int_cols\n",
        "\n",
        "    # One-hot encoding, for ['CAMEO_DEU_2015', 'D19_LETZTER_KAUF_BRANCHE', 'OST_WEST_KZ']\n",
        "    df = pd.get_dummies(df)\n",
        "\n",
        "    # Find attribute columns with correlation greater than 0.95 and drop them\n",
        "    corr_cols_to_drop = drop_collinear_cols(df)\n",
        "    df.drop(columns=corr_cols_to_drop, inplace=True)\n",
        "\n",
        "    # Select the columns with NaNs more than 80% and with entropy less than 1.0.\n",
        "    df_ent_drop, ent = drop_nan_cols(df, df_cols_nans)\n",
        "    \n",
        "    # Drop above selected NaN attributes and supplied zero importance attributes.\n",
        "    zero_imp_drop = zero_imp_attrs or []\n",
        "    df.drop(columns=np.union1d(df_ent_drop, zero_imp_drop),\n",
        "            errors='ignore', inplace=True)\n",
        "\n",
        "    return df.assign(LNR=clientIDs.values)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tPec3toZbUG4"
      },
      "source": [
        "## Part 1: Customer Segmentation Report\n",
        "\n",
        "The main bulk of your analysis will come in this part of the project. Here, you should use unsupervised learning techniques to describe the relationship between the demographics of the company's existing customers and the general population of Germany. By the end of this part, you should be able to describe parts of the general population that are more likely to be part of the mail-order company's main customer base, and which parts of the general population are less so."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LIwQiNEUfgV9"
      },
      "source": [
        "> The goal of unsupervised learning is to find the underlying structure of a dataset, group that data according to similarities, and represent that dataset in a compressed format.\n",
        "\n",
        "The unsupervised learning algorithm can be further categorized into two types of problems:\n",
        "\n",
        "* **Clustering**: Clustering is a method of grouping the objects into clusters such that objects with most similarities remain in a group and have less or no similarities with the objects of another group. Cluster analysis finds the commonalities between the data objects and categorizes them as per the presence and absence of those commonalities.\n",
        "\n",
        "* **Association**: An association rule is an unsupervised learning method which is used for finding the relationships between variables in the large database. It determines the set of items that occurs together in the dataset. Association rule makes marketing strategy more effective. For example, people who buy X item (suppose a bread) also tend to purchase Y (Butter/Jam) item. A typical example of Association rule is Market Basket Analysis.\n",
        "\n",
        "Ref: https://www.javatpoint.com/unsupervised-machine-learning\n",
        "\n",
        "Popular unsupervised learning algorithms:\n",
        "\n",
        "* K-means clustering\n",
        "* KNN (k-nearest neighbors)\n",
        "* Hierarchical clustering\n",
        "* Anomaly detection\n",
        "* Neural Networks\n",
        "* Principal Component Analysis\n",
        "* Independent Component Analysis\n",
        "* Apriori algorithm\n",
        "* Singular value decomposition"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f1LcAm_yEXSP"
      },
      "source": [
        "### Clustering with Mini Batch K-Means\n",
        "\n",
        "We will choose the KMeans algorithm for data clustering, because it is a general-purpose clustering algorithm with even cluster sizes and has good scalability for very large samples. We will specifically use the [MiniBatch code](https://scikit-learn.org/stable/modules/clustering.html#mini-batch-kmeans), a variant of the KMeans algorithm which uses mini-batches to reduce the computation time.\n",
        "\n",
        "The K-means algorithm aims to choose centroids that minimise the inertia, or within-cluster sum-of-squares criterion. However, inertia is not a normalized metric, becoming a drawback for the use of KMeans. To overcome this it is usual practice to run a dimensionality reduction algorithm such as Principal component analysis (PCA) prior to k-means clustering. K-means algorithm is very much susceptible to the range in which features are measured, we will therefore normalize our data in the process."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jzFJmjAlMiXm"
      },
      "source": [
        "PCA can be performed using the correlation matrix or the covariance matrix of the data. These will provide different results. In general the covariance matrix is used when the variable scales are similar and the correlation matrix when variables are on different scales ([Ref.](https://stats.stackexchange.com/a/78/336933)).\n",
        "\n",
        "For this data we would like to have a more informative PCA so we will go with  PCA on correlation, since the covariance matrix on our data will be primarily dominated by the amount of missing values. \n",
        "\n",
        "A pipeline that will initially impute missing values with `most_frequent` values, then standardize the data with `MinMaxScaler` as we cannot assume that the shape of all your features follows a normal distribution, and finally fit PCA will allow us to:\n",
        "\n",
        "* Initially fit the PCA with all components to acquire the full picture of  variability in the data.\n",
        "* Examine the `explained_variance_ratio_` and decide on the number of components to finally use. Typically, we want the explained variance to be at least 95%.\n",
        "* Fit PCA with the selected components (`n_components = .95`)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KVefbRlIuK1p"
      },
      "source": [
        "from sklearn.preprocessing import MinMaxScaler\n",
        "from sklearn.decomposition import PCA\n",
        "\n",
        "scaler = MinMaxScaler()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tT0bmUh6kZwD"
      },
      "source": [
        "#### Impute missing values with ***random selection***\n",
        "\n",
        "> The  Random  Selection  approach  is  a  method  based  on randomly assigning  a value to  the missing data.  The values randomly selected  are framed  in a specific  range of  values, which should  have the same  characteristics as the  values in the  variable  with  the  missingness (numerical  or  categorical data).  Each  number  in  between  the  range  has  the  same probability of being assigned to the missing data.\n",
        "\n",
        "\n",
        "\n",
        "Ref: https://www.researchgate.net/publication/326435546_Missing_Data_Imputation_for_Ordinal_Data\n",
        "\n",
        "We will create a function to apply to the dataframe for imputing missing values with random selection from the column's values range."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KMGORKvLd7hk"
      },
      "source": [
        "# Initialize random generator\n",
        "rng = np.random.default_rng()\n",
        "\n",
        "def imput_nan_rand(col, in_place=False):\n",
        "    \"\"\"Get a pd.Series with NaNs and impute missing values with random selection\n",
        "    from the column's values range.\n",
        "    \"\"\"\n",
        "    c = col if in_place else col.copy()\n",
        "    # get indices of `-1` occurrences\n",
        "    idxs = np.where(c == -1)[0]\n",
        "    # initialize boolean mask\n",
        "    m = np.ones(c.size, dtype=bool)\n",
        "    # change boolean mask at indices\n",
        "    m[idxs] = False\n",
        "    # create column unique values range\n",
        "    pool = np.unique(c[m]).astype(int)\n",
        "    # assign random values from the pool at `-1` locations\n",
        "    c[~m] = rng.choice(pool, size=idxs.size, shuffle=False)\n",
        "    return c.astype(int)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lj6vQNZQkg3K"
      },
      "source": [
        "# Create train and target sets.\n",
        "train = df_train.drop(columns=[\"TARGET\"])\n",
        "target = df_train[\"TARGET\"]\n",
        "\n",
        "# Impute missing values with Random Selection\n",
        "norm_train = train.apply(imput_nan_rand)\n",
        "\n",
        "# Normalization\n",
        "norm_train = pd.DataFrame(scaler.fit_transform(norm_train))\n",
        "norm_train.columns = train.columns\n",
        "norm_train.index = train.index"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ot7oyEieM-og"
      },
      "source": [
        "A view of our original cleaned data. Let's produce a scatter matrix for some pairs of attributes in the data to have a view of data before imputing, scaling."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i0wOFRqAL5bN"
      },
      "source": [
        "# Save image location\n",
        "pairplot1_file = f'{colab_dir if IN_COLAB else remote_dir}pairplot_orig_clean_{ts}.png'\n",
        "\n",
        "pr1plt = sns.pairplot(train.sample(100000).iloc[:, :4], diag_kind='kde')\n",
        "pr1plt.fig.suptitle(\"Sample data view BEFORE imputing, scaling\", y=1.02)\n",
        "# copyright text\n",
        "pr1plt.fig.text(.4 ,-.03, cprt, bbox=bb_cprt_args)\n",
        "plt.savefig(pairplot1_file, transparent=False, bbox_inches=\"tight\")\n",
        "plt.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hqr3BYlhNIs6"
      },
      "source": [
        "A view of the scaled data **after** imputing missing values with random selection and normalization."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zG-93SE0uj9y"
      },
      "source": [
        "# Save image location\n",
        "pairplot2_file = f'{colab_dir if IN_COLAB else remote_dir}pairplot_imputd_scld_{ts}.png'\n",
        "\n",
        "pr2plt = sns.pairplot(norm_train.sample(100000).iloc[:, :4], diag_kind='kde')\n",
        "pr2plt.fig.suptitle(\"Sample data view AFTER imputing, scaling\", y=1.02)\n",
        "# copyright text\n",
        "pr2plt.fig.text(.4 ,-.03, cprt, bbox=bb_cprt_args)\n",
        "plt.savefig(pairplot2_file, transparent=False, bbox_inches=\"tight\")\n",
        "plt.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Zt9ip_ofKaIQ",
        "outputId": "62614116-c8d3-42bb-8bed-87cddf0bb82d"
      },
      "source": [
        "# Initialize PCA model\n",
        "pca = PCA()\n",
        "\n",
        "# Fit the model with train data\n",
        "pca.fit(norm_train.to_numpy())\n",
        "\n",
        "# components\n",
        "pca.n_components_"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "376"
            ]
          },
          "metadata": {},
          "execution_count": 118
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MnGxJ51Lkvuq"
      },
      "source": [
        "# Let's visualize the results\n",
        "def plt_pca(pca_model, n_pcs=5, cumul_expl_var=.95, fig_size=(10, 5), save_file=None):\n",
        "    \"\"\"Plots first n_pcs principal components and the cumulative variance of\n",
        "    components.Plots cumulative variance ratio curve marks the cumulative\n",
        "    explained threshold if < 1.0 indicating the number of PCs required.\n",
        "    \n",
        "    Args:\n",
        "        pca_model: A fitted PCA model (sklearn.decomposition.PCA).\n",
        "        n_pcs (int): Number of PCs to plot.\n",
        "        cumul_expl_var (float): Plot information about cumulative variance ratio\n",
        "        below `cumul_expl_var` threshold.\n",
        "        fig_size (tuple of int): Figure size for the combined plot.\n",
        "        \n",
        "    Returns:\n",
        "        df (pd.DataFrame): A dataframe ordered by PCs variance ratios with a\n",
        "            cumulative variance ratio column (sums to 1).\n",
        "        thres_num_pcs: Number of PCs for `cumul_expl_var` threshold\n",
        "    \"\"\"\n",
        "    pca_comp = n_pcs if pca_model.n_components_ >= 25 else pca_model.n_components_\n",
        "    df = pd.DataFrame({'PC': np.arange(1, pca_model.n_components_ + 1),\n",
        "                           'var': pca_model.explained_variance_ratio_})\n",
        "    df[\"cumul_var\"] = df[\"var\"].cumsum()\n",
        "    thres_num_pcs = df[df[\"cumul_var\"] <= \\\n",
        "                           cumul_expl_var][\"cumul_var\"].count()\n",
        "    if cumul_expl_var < 1.0:\n",
        "        thres_num_pcs += 1\n",
        "    with sns.axes_style(\"whitegrid\"), sns.plotting_context(\"paper\"):\n",
        "        fig, ax = plt.subplots(1, 2, figsize=fig_size)\n",
        "        # Bar plot of explained variance per PC\n",
        "        sns.barplot(ax=ax[0], data=df.iloc[:pca_comp, :],\n",
        "                        x='PC', y='var', color='royalblue')\n",
        "        ax[0].set(title='Variance explained, first ' + str(pca_comp) + ' PCs',\n",
        "                xlabel='Principal components')\n",
        "        # Cumulative explained variance plot\n",
        "        sns.lineplot(ax=ax[1], data=df['var'].cumsum(), color='r', linewidth=3)\n",
        "        ax[1].set(title='Scree plot, mark: ' \\\n",
        "                    + str(cumul_expl_var) \\\n",
        "                    + ', ' + str(thres_num_pcs) + ' PCs',\n",
        "                xlabel='Number of components',\n",
        "                ylabel='Cumulative explained variance')\n",
        "        # if the number of PCs at cumul_expl_var is less than the\n",
        "        # total computed PCs mark it on the chart\n",
        "        if thres_num_pcs < df.shape[0]:\n",
        "            # horizontal line marker\n",
        "            ax[1].axhline(cumul_expl_var, linestyle='--',\n",
        "                          color='royalblue', lw=2, alpha=0.6)\n",
        "            # vertical line marker\n",
        "            ax[1].axvline(thres_num_pcs, linestyle='--',\n",
        "                          color='royalblue', lw=2, alpha=0.6)\n",
        "        # copyright text\n",
        "        fig.suptitle(cprt, y=-.03, verticalalignment='bottom', bbox=bb_cprt_args)\n",
        "        fig.tight_layout()\n",
        "        if save_file:\n",
        "            fig.savefig(save_file, transparent=False, bbox_inches=\"tight\")\n",
        "            plt.close(fig)\n",
        "    return df, thres_num_pcs\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "ck3AssWYGYUp",
        "outputId": "4b164f47-dd80-4c80-bcfa-0dd1189acde5"
      },
      "source": [
        "pca_scree_file = f'{colab_dir if IN_COLAB else remote_dir}pca_screeplt_{ts}.png'\n",
        "\n",
        "pca_df, pcs_num = plt_pca(pca, n_pcs=25, cumul_expl_var=.7, save_file=pca_scree_file)\n",
        "\n",
        "pca_df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>PC</th>\n",
              "      <th>var</th>\n",
              "      <th>cumul_var</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>0.068443</td>\n",
              "      <td>0.068443</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>0.044690</td>\n",
              "      <td>0.113133</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>0.035743</td>\n",
              "      <td>0.148876</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>0.032697</td>\n",
              "      <td>0.181572</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>0.023131</td>\n",
              "      <td>0.204703</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   PC       var  cumul_var\n",
              "0   1  0.068443   0.068443\n",
              "1   2  0.044690   0.113133\n",
              "2   3  0.035743   0.148876\n",
              "3   4  0.032697   0.181572\n",
              "4   5  0.023131   0.204703"
            ]
          },
          "metadata": {},
          "execution_count": 120
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IWjaMv-Ogazw"
      },
      "source": [
        "So, for approximately **0.7** cumulative explained variance ratio line marked components in the figure are required (amount of variance that needs to be explained is greater than the percentage specified by `n_components`).\n",
        "\n",
        "We will fit PCA again using the number of components for 0.7 cumulative explained variance ratio:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SaJWjWoWjy9S",
        "outputId": "d172e4d0-7f14-46f6-d4b1-af1a7a36516d"
      },
      "source": [
        "# Initialize PCA with the new parameter (could also use n_components=.75)\n",
        "pca = PCA(n_components=pcs_num)\n",
        "\n",
        "# Fit the model with the new parameter again and apply the dimensionality reduction.\n",
        "train_reduced = pca.fit_transform(norm_train.to_numpy())\n",
        "\n",
        "train_reduced.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(450000, 104)"
            ]
          },
          "metadata": {},
          "execution_count": 121
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0qm_w9DbTh5C",
        "outputId": "4a5927c3-aec3-44e1-bc6c-0bb69a2ec245"
      },
      "source": [
        "pca.components_.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(104, 376)"
            ]
          },
          "metadata": {},
          "execution_count": 122
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t9qon6uYXtZm"
      },
      "source": [
        "We have `n` `components_` representing the `p` principal axes in feature space. Each component is an array of loadings (weights) defining the direction of the principal component along which data varies the most. It results in a line in the `p` dimensional feature space.\n",
        "\n",
        "We can see how each attribute is represneted by these weights by creating a mapping:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0iB1ZcacgzYD",
        "outputId": "7c25c912-bf42-42e0-ce75-a9cb6231abdf"
      },
      "source": [
        "# Create the column names for each PC\n",
        "pc_names = ['PC' + str(i + 1) for i in range(pca.n_components_)]\n",
        "\n",
        "# Create a reduced attributes dataframe for later use\n",
        "train_reduced = pd.DataFrame(train_reduced, columns=pc_names, index=norm_train.index)\n",
        "\n",
        "# Create loadings - attributes mapping\n",
        "pca_wghts_df = pd.DataFrame(pca.components_,\n",
        "                            columns=df_train.drop(columns=[\"TARGET\"]).columns.tolist(),\n",
        "                            index=pc_names)\n",
        "\n",
        "pca_wghts_df.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(104, 376)"
            ]
          },
          "metadata": {},
          "execution_count": 123
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nLRSMHSvvD7R"
      },
      "source": [
        "If we transpose this we get:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 160
        },
        "id": "54xAK1FWpoTq",
        "outputId": "5faf582c-b122-4657-cf1b-6be7394c63a5"
      },
      "source": [
        "pca_wghts_df.T.head(2)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>PC1</th>\n",
              "      <th>PC2</th>\n",
              "      <th>PC3</th>\n",
              "      <th>PC4</th>\n",
              "      <th>PC5</th>\n",
              "      <th>PC6</th>\n",
              "      <th>PC7</th>\n",
              "      <th>PC8</th>\n",
              "      <th>PC9</th>\n",
              "      <th>PC10</th>\n",
              "      <th>PC11</th>\n",
              "      <th>PC12</th>\n",
              "      <th>PC13</th>\n",
              "      <th>PC14</th>\n",
              "      <th>PC15</th>\n",
              "      <th>PC16</th>\n",
              "      <th>PC17</th>\n",
              "      <th>PC18</th>\n",
              "      <th>PC19</th>\n",
              "      <th>PC20</th>\n",
              "      <th>PC21</th>\n",
              "      <th>PC22</th>\n",
              "      <th>PC23</th>\n",
              "      <th>PC24</th>\n",
              "      <th>PC25</th>\n",
              "      <th>PC26</th>\n",
              "      <th>PC27</th>\n",
              "      <th>PC28</th>\n",
              "      <th>PC29</th>\n",
              "      <th>PC30</th>\n",
              "      <th>PC31</th>\n",
              "      <th>PC32</th>\n",
              "      <th>PC33</th>\n",
              "      <th>PC34</th>\n",
              "      <th>PC35</th>\n",
              "      <th>PC36</th>\n",
              "      <th>PC37</th>\n",
              "      <th>PC38</th>\n",
              "      <th>PC39</th>\n",
              "      <th>PC40</th>\n",
              "      <th>...</th>\n",
              "      <th>PC65</th>\n",
              "      <th>PC66</th>\n",
              "      <th>PC67</th>\n",
              "      <th>PC68</th>\n",
              "      <th>PC69</th>\n",
              "      <th>PC70</th>\n",
              "      <th>PC71</th>\n",
              "      <th>PC72</th>\n",
              "      <th>PC73</th>\n",
              "      <th>PC74</th>\n",
              "      <th>PC75</th>\n",
              "      <th>PC76</th>\n",
              "      <th>PC77</th>\n",
              "      <th>PC78</th>\n",
              "      <th>PC79</th>\n",
              "      <th>PC80</th>\n",
              "      <th>PC81</th>\n",
              "      <th>PC82</th>\n",
              "      <th>PC83</th>\n",
              "      <th>PC84</th>\n",
              "      <th>PC85</th>\n",
              "      <th>PC86</th>\n",
              "      <th>PC87</th>\n",
              "      <th>PC88</th>\n",
              "      <th>PC89</th>\n",
              "      <th>PC90</th>\n",
              "      <th>PC91</th>\n",
              "      <th>PC92</th>\n",
              "      <th>PC93</th>\n",
              "      <th>PC94</th>\n",
              "      <th>PC95</th>\n",
              "      <th>PC96</th>\n",
              "      <th>PC97</th>\n",
              "      <th>PC98</th>\n",
              "      <th>PC99</th>\n",
              "      <th>PC100</th>\n",
              "      <th>PC101</th>\n",
              "      <th>PC102</th>\n",
              "      <th>PC103</th>\n",
              "      <th>PC104</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>AGER_TYP</th>\n",
              "      <td>0.040499</td>\n",
              "      <td>0.019273</td>\n",
              "      <td>0.017980</td>\n",
              "      <td>-0.004219</td>\n",
              "      <td>-0.001565</td>\n",
              "      <td>-0.000258</td>\n",
              "      <td>0.006022</td>\n",
              "      <td>-0.014194</td>\n",
              "      <td>0.005669</td>\n",
              "      <td>-0.010191</td>\n",
              "      <td>0.007729</td>\n",
              "      <td>-0.039241</td>\n",
              "      <td>0.042344</td>\n",
              "      <td>-0.014783</td>\n",
              "      <td>0.019302</td>\n",
              "      <td>0.031102</td>\n",
              "      <td>0.032487</td>\n",
              "      <td>-0.002453</td>\n",
              "      <td>0.020443</td>\n",
              "      <td>0.007879</td>\n",
              "      <td>-0.019790</td>\n",
              "      <td>0.032408</td>\n",
              "      <td>0.054422</td>\n",
              "      <td>-0.150948</td>\n",
              "      <td>-0.049295</td>\n",
              "      <td>-0.079318</td>\n",
              "      <td>-0.030557</td>\n",
              "      <td>-0.173139</td>\n",
              "      <td>0.047624</td>\n",
              "      <td>-0.002524</td>\n",
              "      <td>-0.082730</td>\n",
              "      <td>0.019034</td>\n",
              "      <td>0.106739</td>\n",
              "      <td>0.145640</td>\n",
              "      <td>-0.446251</td>\n",
              "      <td>0.221433</td>\n",
              "      <td>0.214156</td>\n",
              "      <td>0.371410</td>\n",
              "      <td>-0.151337</td>\n",
              "      <td>0.242150</td>\n",
              "      <td>...</td>\n",
              "      <td>-0.008568</td>\n",
              "      <td>0.020501</td>\n",
              "      <td>-0.019924</td>\n",
              "      <td>0.012193</td>\n",
              "      <td>-0.015130</td>\n",
              "      <td>-0.003970</td>\n",
              "      <td>-0.003982</td>\n",
              "      <td>0.005635</td>\n",
              "      <td>0.041221</td>\n",
              "      <td>0.012159</td>\n",
              "      <td>0.000564</td>\n",
              "      <td>-0.013774</td>\n",
              "      <td>0.013212</td>\n",
              "      <td>0.026896</td>\n",
              "      <td>-0.017190</td>\n",
              "      <td>0.027402</td>\n",
              "      <td>0.016086</td>\n",
              "      <td>-0.010980</td>\n",
              "      <td>-0.006706</td>\n",
              "      <td>0.017079</td>\n",
              "      <td>-0.001329</td>\n",
              "      <td>0.014007</td>\n",
              "      <td>0.007533</td>\n",
              "      <td>-0.013356</td>\n",
              "      <td>0.002867</td>\n",
              "      <td>0.007722</td>\n",
              "      <td>-0.038068</td>\n",
              "      <td>-0.017381</td>\n",
              "      <td>0.004318</td>\n",
              "      <td>-0.000098</td>\n",
              "      <td>-0.017653</td>\n",
              "      <td>-0.005291</td>\n",
              "      <td>-0.004985</td>\n",
              "      <td>-0.002139</td>\n",
              "      <td>-0.015422</td>\n",
              "      <td>-0.018455</td>\n",
              "      <td>0.006800</td>\n",
              "      <td>0.014299</td>\n",
              "      <td>0.003058</td>\n",
              "      <td>-0.021616</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>AKT_DAT_KL</th>\n",
              "      <td>0.152465</td>\n",
              "      <td>0.010389</td>\n",
              "      <td>-0.014757</td>\n",
              "      <td>-0.072644</td>\n",
              "      <td>-0.052135</td>\n",
              "      <td>-0.101453</td>\n",
              "      <td>0.071686</td>\n",
              "      <td>-0.016238</td>\n",
              "      <td>0.005246</td>\n",
              "      <td>0.012983</td>\n",
              "      <td>0.028318</td>\n",
              "      <td>-0.015893</td>\n",
              "      <td>0.020779</td>\n",
              "      <td>0.006200</td>\n",
              "      <td>-0.020310</td>\n",
              "      <td>0.021642</td>\n",
              "      <td>-0.023154</td>\n",
              "      <td>-0.022461</td>\n",
              "      <td>0.074776</td>\n",
              "      <td>0.047494</td>\n",
              "      <td>-0.002094</td>\n",
              "      <td>-0.039537</td>\n",
              "      <td>-0.013757</td>\n",
              "      <td>0.030089</td>\n",
              "      <td>0.045386</td>\n",
              "      <td>-0.074345</td>\n",
              "      <td>0.079225</td>\n",
              "      <td>0.048936</td>\n",
              "      <td>-0.035783</td>\n",
              "      <td>0.023037</td>\n",
              "      <td>-0.017126</td>\n",
              "      <td>-0.020888</td>\n",
              "      <td>-0.171201</td>\n",
              "      <td>-0.017245</td>\n",
              "      <td>0.153162</td>\n",
              "      <td>-0.174816</td>\n",
              "      <td>0.176846</td>\n",
              "      <td>-0.008283</td>\n",
              "      <td>-0.038711</td>\n",
              "      <td>0.000405</td>\n",
              "      <td>...</td>\n",
              "      <td>-0.068520</td>\n",
              "      <td>-0.094211</td>\n",
              "      <td>0.067700</td>\n",
              "      <td>0.080490</td>\n",
              "      <td>0.049474</td>\n",
              "      <td>-0.137913</td>\n",
              "      <td>-0.074241</td>\n",
              "      <td>0.139672</td>\n",
              "      <td>0.056110</td>\n",
              "      <td>0.089608</td>\n",
              "      <td>0.036279</td>\n",
              "      <td>0.067461</td>\n",
              "      <td>0.046748</td>\n",
              "      <td>-0.156053</td>\n",
              "      <td>-0.000363</td>\n",
              "      <td>-0.150017</td>\n",
              "      <td>0.050865</td>\n",
              "      <td>-0.042738</td>\n",
              "      <td>-0.014365</td>\n",
              "      <td>-0.005070</td>\n",
              "      <td>0.174441</td>\n",
              "      <td>0.115032</td>\n",
              "      <td>0.040947</td>\n",
              "      <td>-0.011871</td>\n",
              "      <td>0.029424</td>\n",
              "      <td>0.010613</td>\n",
              "      <td>0.042750</td>\n",
              "      <td>-0.019973</td>\n",
              "      <td>0.020907</td>\n",
              "      <td>-0.044007</td>\n",
              "      <td>0.019236</td>\n",
              "      <td>0.034034</td>\n",
              "      <td>-0.063241</td>\n",
              "      <td>-0.014246</td>\n",
              "      <td>-0.010589</td>\n",
              "      <td>-0.062912</td>\n",
              "      <td>0.034017</td>\n",
              "      <td>-0.125696</td>\n",
              "      <td>-0.079933</td>\n",
              "      <td>0.058497</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>2 rows × 104 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                 PC1       PC2       PC3  ...     PC102     PC103     PC104\n",
              "AGER_TYP    0.040499  0.019273  0.017980  ...  0.014299  0.003058 -0.021616\n",
              "AKT_DAT_KL  0.152465  0.010389 -0.014757  ... -0.125696 -0.079933  0.058497\n",
              "\n",
              "[2 rows x 104 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 124
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OEm0CLRnwR92"
      },
      "source": [
        "# Save image location\n",
        "pc1pc2_makeup_file = f'{colab_dir if IN_COLAB else remote_dir}PC1PC2_attr_makeup_{ts}.png'\n",
        "\n",
        "# Let's visualize the results sampling some attributes\n",
        "plt_attrs_wgts = pca_wghts_df.T[[\"PC1\", \"PC2\"]].sample(30).rename_axis(\"Attributes\").reset_index()\n",
        "# Combine the two PCs to create a type of category for plotting purposes.\n",
        "plt_attrs_wgts = plt_attrs_wgts.melt(id_vars='Attributes')\n",
        "# Sort by values\n",
        "plt_attrs_wgts = plt_attrs_wgts.sort_values(by=['value'], ascending=False)\n",
        "# Plot\n",
        "with sns.axes_style(\"whitegrid\"), sns.plotting_context(\"paper\", rc={\"font.size\": 8,}):\n",
        "    plt.figure(figsize=(10,6))\n",
        "    sns.barplot(data=plt_attrs_wgts, x='value', y='Attributes', hue='variable',\n",
        "                palette='Blues_d').set(\n",
        "                    title=\"Attributes' influences on first two PCs (sample of 30 attributes)\",\n",
        "                    xlabel=\"PCs weights\")\n",
        "    # copyright text\n",
        "    plt.suptitle(cprt, y=-.03, verticalalignment='bottom', bbox=bb_cprt_args)\n",
        "    plt.tight_layout()\n",
        "    plt.savefig(pc1pc2_makeup_file, transparent=False, bbox_inches=\"tight\")\n",
        "    plt.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RTC65vpf_HM0"
      },
      "source": [
        "We would like to examine the makeup of each PCA component based on the weightings of the original attributes that are included in each component. To do so will create a function to help us visually identify top `n` and bottom `n` attributes' influence on each principal component."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PqsQUQgQ6q9x"
      },
      "source": [
        "def plot_pc_attrs(df_wghts, pc_n, plot_n, fig_size=(8, 4), save_name=None):\n",
        "    \"\"\"Plot top `n` and bottom `n` attributes' influence on each principal component.\n",
        "\n",
        "    Args:\n",
        "        df_wghts (pd.DataFrame): Contains mapping of PCs' weights for each attribute\n",
        "        pc_n (int): Number of PCs to plot.\n",
        "        plot_n (int): Number of attributes to plot (will be used for top & bottom)\n",
        "        fig_size (tuple): Size of figure, e.g. (12, 8)\n",
        "        save_name (str or None): File name to save the figure to.\n",
        "            If `None` figure will not be saved.\n",
        "    \"\"\"\n",
        "    pc = 'PC' + str(pc_n)\n",
        "    pc_attrs_wgts = df_wghts.T[pc].sort_values(ascending=False)\\\n",
        "                    .rename_axis(\"Attributes\").reset_index()\n",
        "    # Plot top n and bottom n\n",
        "    with sns.axes_style(\"whitegrid\"), sns.plotting_context(\"paper\", rc={\"font.size\": 8,}):\n",
        "        plt.figure(figsize=fig_size)\n",
        "        sns.barplot(data=pc_attrs_wgts.iloc[np.r_[0:plot_n, -plot_n:0]], x=pc,\n",
        "                    y='Attributes', color='royalblue').set(\n",
        "                        title=\"Attributes' influences on {} (top & bottom {})\"\\\n",
        "                        .format(pc, plot_n),\n",
        "                        xlabel=\"{} weights\".format(pc))\n",
        "        # copyright text\n",
        "        plt.suptitle(cprt, y=-.03, verticalalignment='bottom', bbox=bb_cprt_args)\n",
        "        plt.tight_layout()\n",
        "        if save_name:\n",
        "            plt.savefig(save_name, transparent=False, bbox_inches=\"tight\")\n",
        "            plt.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wk5w4wtU9qG3"
      },
      "source": [
        "# Save image location\n",
        "pc1_makeup_file = f'{colab_dir if IN_COLAB else remote_dir}PC1_attr_makeup_{ts}.png'\n",
        "\n",
        "plot_pc_attrs(pca_wghts_df, 1, 10, save_name=pc1_makeup_file)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tpSv_HrEEzgO"
      },
      "source": [
        "# Save image location 'PC2'\n",
        "pc2_makeup_file = f'{colab_dir if IN_COLAB else remote_dir}PC2_attr_makeup_{ts}.png'\n",
        "\n",
        "plot_pc_attrs(pca_wghts_df, 2, 10, save_name=pc2_makeup_file)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hN238pI9JAYA"
      },
      "source": [
        "We can create a list of the **two** highest influencing attribute names in each component."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EfUktRgaI_De",
        "outputId": "5d04fa71-9aac-4fe1-a71c-074d6fc42271"
      },
      "source": [
        "# initialize list to hold attribute names\n",
        "pca_list = []\n",
        "# iterate through the PC columns, sort values and get the top two attributes with\n",
        "# higher weights. Join names in a single string representing each component.\n",
        "for i in pca_wghts_df.T:\n",
        "    pca_list.append('/'.join(list(pca_wghts_df.T[i].sort_values(ascending=False).index[:2])))\n",
        "\n",
        "pca_list[:4], len(pca_list)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(['CJT_TYP_2/FINANZ_SPARER',\n",
              "  'FINANZ_ANLEGER/FINANZ_UNAUFFAELLIGER',\n",
              "  'OST_WEST_KZ_W/STRUKTURTYP',\n",
              "  'HH_EINKOMMEN_SCORE/OST_WEST_KZ_O'],\n",
              " 104)"
            ]
          },
          "metadata": {},
          "execution_count": 129
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dyQRp_ZkhhNI"
      },
      "source": [
        "We can now create a dataframe where each individual (row) is described by the number of principal components that we chose previously. Each of these selected components is a linear combination of the original attribute space. We can interpret each of these selected principal components by analyzing the makeup of the component as shown earlier."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 253
        },
        "id": "Yj9RXj3tfbX7",
        "outputId": "489f9ab5-9da0-4392-eefe-4d8fe610a75d"
      },
      "source": [
        "population_transformed = train_reduced.copy()\n",
        "population_transformed.columns=pca_list\n",
        "population_transformed.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>CJT_TYP_2/FINANZ_SPARER</th>\n",
              "      <th>FINANZ_ANLEGER/FINANZ_UNAUFFAELLIGER</th>\n",
              "      <th>OST_WEST_KZ_W/STRUKTURTYP</th>\n",
              "      <th>HH_EINKOMMEN_SCORE/OST_WEST_KZ_O</th>\n",
              "      <th>ANREDE_KZ/SEMIO_KAEM</th>\n",
              "      <th>KBA13_BAUMAX/LP_FAMILIE_FEIN</th>\n",
              "      <th>OST_WEST_KZ_O/KBA13_KW_61_120</th>\n",
              "      <th>PLZ8_ANTG4/KBA13_ANTG4</th>\n",
              "      <th>KBA13_CCM_3001/KBA13_CCM_2501</th>\n",
              "      <th>KBA05_MAXSEG/KBA05_MOTOR</th>\n",
              "      <th>KBA13_ANTG4/PLZ8_ANTG4</th>\n",
              "      <th>KBA05_MAXBJ/SHOPPER_TYP</th>\n",
              "      <th>VERS_TYP/HEALTH_TYP</th>\n",
              "      <th>KBA05_MOD8/KBA05_SEG9</th>\n",
              "      <th>LP_FAMILIE_FEIN/LP_LEBENSPHASE_FEIN</th>\n",
              "      <th>KBA05_KW3/KBA05_CCM4</th>\n",
              "      <th>KBA13_ALTERHALTER_45/KBA13_HALTER_35</th>\n",
              "      <th>KBA13_VORB_1_2/KBA13_ALTERHALTER_61</th>\n",
              "      <th>D19_GESAMT_ANZ_24/D19_VERSAND_ANZ_24</th>\n",
              "      <th>D19_GESAMT_ANZ_24/D19_VERSAND_ANZ_24</th>\n",
              "      <th>KBA13_KRSZUL_NEU/KBA05_MOD8</th>\n",
              "      <th>KBA05_ANHANG/KBA05_ANTG3</th>\n",
              "      <th>KBA05_ANHANG/KBA05_BAUMAX</th>\n",
              "      <th>SHOPPER_TYP/KBA05_ANTG1</th>\n",
              "      <th>SHOPPER_TYP/KBA13_KRSSEG_VAN</th>\n",
              "      <th>KBA05_ANHANG/KBA05_MOTRAD</th>\n",
              "      <th>KBA05_MOTRAD/BALLRAUM</th>\n",
              "      <th>VHN/D19_SOZIALES</th>\n",
              "      <th>PLZ8_ANTG4/KBA05_ANHANG</th>\n",
              "      <th>PLZ8_ANTG4/KBA05_ANTG3</th>\n",
              "      <th>SHOPPER_TYP/KBA13_ANTG4</th>\n",
              "      <th>PLZ8_ANTG4/KBA13_KRSSEG_VAN</th>\n",
              "      <th>D19_LETZTER_KAUF_BRANCHE_D19_UNBEKANNT/D19_KONSUMTYP</th>\n",
              "      <th>KBA05_ANHANG/KBA05_SEG7</th>\n",
              "      <th>KBA05_MOTRAD/KBA05_ANTG3</th>\n",
              "      <th>KBA05_MOTRAD/AGER_TYP</th>\n",
              "      <th>VHN/KBA05_MOTRAD</th>\n",
              "      <th>AGER_TYP/KBA05_ZUL2</th>\n",
              "      <th>KBA05_SEG8/KBA05_SEG7</th>\n",
              "      <th>KBA05_SEG7/AGER_TYP</th>\n",
              "      <th>...</th>\n",
              "      <th>D19_BANKEN_DIREKT/D19_REISEN</th>\n",
              "      <th>D19_REISEN/KBA13_CCM_1500</th>\n",
              "      <th>D19_HAUS_DEKO/D19_BEKLEIDUNG_REST</th>\n",
              "      <th>D19_BEKLEIDUNG_REST/D19_BANKEN_DIREKT</th>\n",
              "      <th>KBA13_KW_70/KBA05_ANTG2</th>\n",
              "      <th>D19_BEKLEIDUNG_REST/D19_HAUS_DEKO</th>\n",
              "      <th>D19_VERSAND_REST/D19_BEKLEIDUNG_REST</th>\n",
              "      <th>KBA05_MOD1/KBA05_SEG10</th>\n",
              "      <th>KK_KUNDENTYP/D19_VERSAND_ANZ_24</th>\n",
              "      <th>KBA05_ANTG2/D19_VOLLSORTIMENT</th>\n",
              "      <th>KBA05_SEG9/D19_TECHNIK</th>\n",
              "      <th>KBA05_ALTER2/D19_VERSAND_ANZ_12</th>\n",
              "      <th>D19_SONSTIGE/D19_VERSAND_ANZ_12</th>\n",
              "      <th>KBA05_SEG9/KBA05_ALTER3</th>\n",
              "      <th>D19_VOLLSORTIMENT/KBA05_HERST5</th>\n",
              "      <th>KBA05_MOD8/D19_REISEN</th>\n",
              "      <th>D19_BUCH_CD/KBA05_KW3</th>\n",
              "      <th>D19_BUCH_CD/KBA05_MOD8</th>\n",
              "      <th>D19_BUCH_CD/KBA05_KRSVAN</th>\n",
              "      <th>D19_VOLLSORTIMENT/D19_GESAMT_ONLINE_QUOTE_12</th>\n",
              "      <th>RELAT_AB/KBA05_KW3</th>\n",
              "      <th>D19_SONSTIGE/EXTSEL992</th>\n",
              "      <th>KBA05_CCM4/D19_SONSTIGE</th>\n",
              "      <th>KBA05_DIESEL/UMFELD_JUNG</th>\n",
              "      <th>KBA05_DIESEL/RT_UEBERGROESSE</th>\n",
              "      <th>KBA13_VORB_2/KBA13_VORB_1_2</th>\n",
              "      <th>GFK_URLAUBERTYP/KBA05_KRSVAN</th>\n",
              "      <th>D19_SONSTIGE/D19_VERSAND_ANZ_24</th>\n",
              "      <th>KBA05_SEG5/KBA13_SEG_OBERKLASSE</th>\n",
              "      <th>KBA05_BAUMAX/UMFELD_ALT</th>\n",
              "      <th>KBA13_KW_90/KBA13_CCM_3001</th>\n",
              "      <th>KBA05_DIESEL/KBA05_HERST4</th>\n",
              "      <th>KBA13_SEG_OBERKLASSE/KBA13_KRSZUL_NEU</th>\n",
              "      <th>KBA05_MOD4/KBA05_SEG5</th>\n",
              "      <th>KBA13_BJ_2008/RETOURTYP_BK_S</th>\n",
              "      <th>KBA05_VORB2/EXTSEL992</th>\n",
              "      <th>KBA05_HERST4/KBA05_FRAU</th>\n",
              "      <th>D19_VERSAND_ONLINE_QUOTE_12/KBA05_KRSVAN</th>\n",
              "      <th>GFK_URLAUBERTYP/D19_VERSAND_ONLINE_QUOTE_12</th>\n",
              "      <th>KK_KUNDENTYP/D19_GESAMT_ANZ_12</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>-1.234629</td>\n",
              "      <td>-0.867582</td>\n",
              "      <td>-0.578983</td>\n",
              "      <td>-0.220230</td>\n",
              "      <td>-0.742176</td>\n",
              "      <td>0.211376</td>\n",
              "      <td>-0.222328</td>\n",
              "      <td>0.117435</td>\n",
              "      <td>0.625594</td>\n",
              "      <td>-0.276393</td>\n",
              "      <td>0.209671</td>\n",
              "      <td>1.080525</td>\n",
              "      <td>0.073092</td>\n",
              "      <td>-0.005344</td>\n",
              "      <td>-0.725521</td>\n",
              "      <td>0.534920</td>\n",
              "      <td>-0.816195</td>\n",
              "      <td>0.000018</td>\n",
              "      <td>-0.548220</td>\n",
              "      <td>-0.422507</td>\n",
              "      <td>0.358123</td>\n",
              "      <td>0.910408</td>\n",
              "      <td>-0.577757</td>\n",
              "      <td>-0.334013</td>\n",
              "      <td>-0.317721</td>\n",
              "      <td>-0.865876</td>\n",
              "      <td>-0.034087</td>\n",
              "      <td>0.055927</td>\n",
              "      <td>0.069684</td>\n",
              "      <td>0.000181</td>\n",
              "      <td>0.192486</td>\n",
              "      <td>0.326819</td>\n",
              "      <td>-0.207724</td>\n",
              "      <td>0.509368</td>\n",
              "      <td>-0.385418</td>\n",
              "      <td>0.339499</td>\n",
              "      <td>0.397615</td>\n",
              "      <td>0.350358</td>\n",
              "      <td>-0.024274</td>\n",
              "      <td>0.040678</td>\n",
              "      <td>...</td>\n",
              "      <td>-0.127198</td>\n",
              "      <td>0.040113</td>\n",
              "      <td>0.447853</td>\n",
              "      <td>0.491691</td>\n",
              "      <td>0.887629</td>\n",
              "      <td>0.179989</td>\n",
              "      <td>0.231284</td>\n",
              "      <td>0.223842</td>\n",
              "      <td>0.026079</td>\n",
              "      <td>-0.465061</td>\n",
              "      <td>-0.227642</td>\n",
              "      <td>-0.249795</td>\n",
              "      <td>-0.011948</td>\n",
              "      <td>0.248586</td>\n",
              "      <td>-0.388448</td>\n",
              "      <td>0.285329</td>\n",
              "      <td>-0.139759</td>\n",
              "      <td>-0.337783</td>\n",
              "      <td>-0.083996</td>\n",
              "      <td>-0.233873</td>\n",
              "      <td>-0.179353</td>\n",
              "      <td>-0.448052</td>\n",
              "      <td>-0.513697</td>\n",
              "      <td>-0.347848</td>\n",
              "      <td>-0.781476</td>\n",
              "      <td>0.001404</td>\n",
              "      <td>-0.283709</td>\n",
              "      <td>-0.331719</td>\n",
              "      <td>0.221276</td>\n",
              "      <td>-0.163244</td>\n",
              "      <td>0.291595</td>\n",
              "      <td>0.186463</td>\n",
              "      <td>-0.005497</td>\n",
              "      <td>-0.228021</td>\n",
              "      <td>0.307563</td>\n",
              "      <td>-0.713716</td>\n",
              "      <td>0.401752</td>\n",
              "      <td>0.477682</td>\n",
              "      <td>-0.182475</td>\n",
              "      <td>-0.062534</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.977389</td>\n",
              "      <td>0.134613</td>\n",
              "      <td>0.048990</td>\n",
              "      <td>-0.160303</td>\n",
              "      <td>-0.804427</td>\n",
              "      <td>-0.352177</td>\n",
              "      <td>0.670749</td>\n",
              "      <td>-1.172047</td>\n",
              "      <td>-1.055501</td>\n",
              "      <td>0.573792</td>\n",
              "      <td>0.661885</td>\n",
              "      <td>-1.608553</td>\n",
              "      <td>0.358878</td>\n",
              "      <td>0.913354</td>\n",
              "      <td>-1.173424</td>\n",
              "      <td>0.805668</td>\n",
              "      <td>-0.148719</td>\n",
              "      <td>0.882287</td>\n",
              "      <td>0.335386</td>\n",
              "      <td>0.451623</td>\n",
              "      <td>0.064425</td>\n",
              "      <td>1.197852</td>\n",
              "      <td>-0.666765</td>\n",
              "      <td>0.302235</td>\n",
              "      <td>0.656651</td>\n",
              "      <td>0.195785</td>\n",
              "      <td>-0.440172</td>\n",
              "      <td>0.129148</td>\n",
              "      <td>-0.404084</td>\n",
              "      <td>0.013804</td>\n",
              "      <td>-0.379258</td>\n",
              "      <td>0.256196</td>\n",
              "      <td>-0.816625</td>\n",
              "      <td>0.160029</td>\n",
              "      <td>0.401891</td>\n",
              "      <td>-0.057911</td>\n",
              "      <td>0.288800</td>\n",
              "      <td>0.054266</td>\n",
              "      <td>0.203565</td>\n",
              "      <td>-0.225481</td>\n",
              "      <td>...</td>\n",
              "      <td>-0.006713</td>\n",
              "      <td>-0.835488</td>\n",
              "      <td>-0.042651</td>\n",
              "      <td>0.597577</td>\n",
              "      <td>0.099311</td>\n",
              "      <td>0.298713</td>\n",
              "      <td>0.140128</td>\n",
              "      <td>0.083948</td>\n",
              "      <td>-0.230444</td>\n",
              "      <td>0.391376</td>\n",
              "      <td>0.510545</td>\n",
              "      <td>0.052131</td>\n",
              "      <td>0.096806</td>\n",
              "      <td>-0.743018</td>\n",
              "      <td>0.162313</td>\n",
              "      <td>-0.162888</td>\n",
              "      <td>-0.008984</td>\n",
              "      <td>0.063964</td>\n",
              "      <td>-0.152078</td>\n",
              "      <td>-0.096144</td>\n",
              "      <td>-0.237473</td>\n",
              "      <td>0.091904</td>\n",
              "      <td>0.133245</td>\n",
              "      <td>0.210970</td>\n",
              "      <td>-0.439297</td>\n",
              "      <td>0.643481</td>\n",
              "      <td>0.427124</td>\n",
              "      <td>0.061224</td>\n",
              "      <td>0.371090</td>\n",
              "      <td>-0.009132</td>\n",
              "      <td>-0.335295</td>\n",
              "      <td>-0.327904</td>\n",
              "      <td>0.320068</td>\n",
              "      <td>-0.003117</td>\n",
              "      <td>-0.307557</td>\n",
              "      <td>-0.310712</td>\n",
              "      <td>-0.153566</td>\n",
              "      <td>0.351870</td>\n",
              "      <td>-0.000796</td>\n",
              "      <td>0.130907</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.640617</td>\n",
              "      <td>0.397748</td>\n",
              "      <td>0.081203</td>\n",
              "      <td>-0.250016</td>\n",
              "      <td>0.403747</td>\n",
              "      <td>-1.531851</td>\n",
              "      <td>-0.125694</td>\n",
              "      <td>0.206615</td>\n",
              "      <td>1.062772</td>\n",
              "      <td>-0.607811</td>\n",
              "      <td>-0.043918</td>\n",
              "      <td>-0.782834</td>\n",
              "      <td>0.357914</td>\n",
              "      <td>0.292886</td>\n",
              "      <td>-0.547790</td>\n",
              "      <td>-0.100105</td>\n",
              "      <td>-0.214742</td>\n",
              "      <td>0.114630</td>\n",
              "      <td>-0.347420</td>\n",
              "      <td>-0.215213</td>\n",
              "      <td>0.445487</td>\n",
              "      <td>-0.195424</td>\n",
              "      <td>-0.093167</td>\n",
              "      <td>-0.405749</td>\n",
              "      <td>0.601990</td>\n",
              "      <td>-0.136346</td>\n",
              "      <td>-0.291381</td>\n",
              "      <td>0.456311</td>\n",
              "      <td>-0.518213</td>\n",
              "      <td>-0.715454</td>\n",
              "      <td>-0.119906</td>\n",
              "      <td>-0.011904</td>\n",
              "      <td>-0.492868</td>\n",
              "      <td>-0.152883</td>\n",
              "      <td>-0.046090</td>\n",
              "      <td>0.267519</td>\n",
              "      <td>-0.506884</td>\n",
              "      <td>-0.513384</td>\n",
              "      <td>-0.453967</td>\n",
              "      <td>0.361384</td>\n",
              "      <td>...</td>\n",
              "      <td>-0.265729</td>\n",
              "      <td>-0.401882</td>\n",
              "      <td>-0.338360</td>\n",
              "      <td>-0.159854</td>\n",
              "      <td>-0.263444</td>\n",
              "      <td>0.874286</td>\n",
              "      <td>0.041411</td>\n",
              "      <td>0.421441</td>\n",
              "      <td>-0.227531</td>\n",
              "      <td>-0.088987</td>\n",
              "      <td>0.038793</td>\n",
              "      <td>-0.179560</td>\n",
              "      <td>-0.421240</td>\n",
              "      <td>-0.329272</td>\n",
              "      <td>0.088884</td>\n",
              "      <td>0.130497</td>\n",
              "      <td>0.239671</td>\n",
              "      <td>0.161087</td>\n",
              "      <td>0.131114</td>\n",
              "      <td>-0.074357</td>\n",
              "      <td>-0.439820</td>\n",
              "      <td>-0.504712</td>\n",
              "      <td>-0.170880</td>\n",
              "      <td>-0.202946</td>\n",
              "      <td>-0.042772</td>\n",
              "      <td>0.007435</td>\n",
              "      <td>0.203907</td>\n",
              "      <td>-0.274071</td>\n",
              "      <td>0.202893</td>\n",
              "      <td>-0.240996</td>\n",
              "      <td>-0.050707</td>\n",
              "      <td>-0.097291</td>\n",
              "      <td>-0.152809</td>\n",
              "      <td>0.085488</td>\n",
              "      <td>0.211906</td>\n",
              "      <td>-0.143104</td>\n",
              "      <td>0.328685</td>\n",
              "      <td>-0.102155</td>\n",
              "      <td>0.400096</td>\n",
              "      <td>0.396859</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>-2.199834</td>\n",
              "      <td>-0.007616</td>\n",
              "      <td>1.126011</td>\n",
              "      <td>-1.318851</td>\n",
              "      <td>1.140490</td>\n",
              "      <td>0.057711</td>\n",
              "      <td>1.092936</td>\n",
              "      <td>0.061931</td>\n",
              "      <td>-0.510787</td>\n",
              "      <td>0.248087</td>\n",
              "      <td>1.407731</td>\n",
              "      <td>-0.546964</td>\n",
              "      <td>-1.101588</td>\n",
              "      <td>0.236766</td>\n",
              "      <td>0.230220</td>\n",
              "      <td>-0.052759</td>\n",
              "      <td>-0.764123</td>\n",
              "      <td>-0.253049</td>\n",
              "      <td>-0.406299</td>\n",
              "      <td>-0.314603</td>\n",
              "      <td>-0.711263</td>\n",
              "      <td>0.080031</td>\n",
              "      <td>-0.518777</td>\n",
              "      <td>-0.530842</td>\n",
              "      <td>-0.018792</td>\n",
              "      <td>0.094442</td>\n",
              "      <td>0.103434</td>\n",
              "      <td>-0.471235</td>\n",
              "      <td>-0.115663</td>\n",
              "      <td>0.119208</td>\n",
              "      <td>-0.258963</td>\n",
              "      <td>-0.118394</td>\n",
              "      <td>0.020771</td>\n",
              "      <td>0.095762</td>\n",
              "      <td>-0.096231</td>\n",
              "      <td>0.039596</td>\n",
              "      <td>0.050026</td>\n",
              "      <td>-0.136419</td>\n",
              "      <td>-0.129697</td>\n",
              "      <td>0.468808</td>\n",
              "      <td>...</td>\n",
              "      <td>-0.724067</td>\n",
              "      <td>-0.085839</td>\n",
              "      <td>0.197129</td>\n",
              "      <td>0.017804</td>\n",
              "      <td>0.069867</td>\n",
              "      <td>-0.359567</td>\n",
              "      <td>0.394705</td>\n",
              "      <td>-0.420672</td>\n",
              "      <td>0.274572</td>\n",
              "      <td>0.142926</td>\n",
              "      <td>-0.218485</td>\n",
              "      <td>0.258072</td>\n",
              "      <td>0.359043</td>\n",
              "      <td>0.245578</td>\n",
              "      <td>-0.307132</td>\n",
              "      <td>0.119405</td>\n",
              "      <td>0.432255</td>\n",
              "      <td>0.264530</td>\n",
              "      <td>-0.079858</td>\n",
              "      <td>0.020682</td>\n",
              "      <td>0.426232</td>\n",
              "      <td>-0.092529</td>\n",
              "      <td>-0.215258</td>\n",
              "      <td>0.162319</td>\n",
              "      <td>0.053370</td>\n",
              "      <td>0.113821</td>\n",
              "      <td>0.497985</td>\n",
              "      <td>-0.488428</td>\n",
              "      <td>-0.038022</td>\n",
              "      <td>0.106081</td>\n",
              "      <td>0.070431</td>\n",
              "      <td>-0.067926</td>\n",
              "      <td>0.348257</td>\n",
              "      <td>-0.685797</td>\n",
              "      <td>0.230047</td>\n",
              "      <td>-0.211343</td>\n",
              "      <td>-0.124144</td>\n",
              "      <td>0.192463</td>\n",
              "      <td>0.117103</td>\n",
              "      <td>-0.323287</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1.247861</td>\n",
              "      <td>0.700665</td>\n",
              "      <td>-0.974589</td>\n",
              "      <td>-1.270208</td>\n",
              "      <td>0.270992</td>\n",
              "      <td>0.698793</td>\n",
              "      <td>-0.026303</td>\n",
              "      <td>-0.420971</td>\n",
              "      <td>0.619710</td>\n",
              "      <td>0.289441</td>\n",
              "      <td>-0.748615</td>\n",
              "      <td>0.083695</td>\n",
              "      <td>-0.194543</td>\n",
              "      <td>-0.113975</td>\n",
              "      <td>-0.428714</td>\n",
              "      <td>0.272176</td>\n",
              "      <td>0.096895</td>\n",
              "      <td>0.909333</td>\n",
              "      <td>-0.025264</td>\n",
              "      <td>-0.212352</td>\n",
              "      <td>0.108476</td>\n",
              "      <td>-0.112836</td>\n",
              "      <td>-0.233279</td>\n",
              "      <td>-0.201998</td>\n",
              "      <td>-0.278923</td>\n",
              "      <td>0.223151</td>\n",
              "      <td>0.017902</td>\n",
              "      <td>-0.011831</td>\n",
              "      <td>-0.354386</td>\n",
              "      <td>0.508833</td>\n",
              "      <td>-0.120905</td>\n",
              "      <td>-0.948406</td>\n",
              "      <td>0.591599</td>\n",
              "      <td>-0.442476</td>\n",
              "      <td>0.251363</td>\n",
              "      <td>0.397072</td>\n",
              "      <td>0.023746</td>\n",
              "      <td>-0.288343</td>\n",
              "      <td>0.420796</td>\n",
              "      <td>-0.082198</td>\n",
              "      <td>...</td>\n",
              "      <td>0.447921</td>\n",
              "      <td>0.390193</td>\n",
              "      <td>-0.235996</td>\n",
              "      <td>-0.095490</td>\n",
              "      <td>-0.420967</td>\n",
              "      <td>0.463086</td>\n",
              "      <td>0.393737</td>\n",
              "      <td>0.393154</td>\n",
              "      <td>-0.283677</td>\n",
              "      <td>-0.621931</td>\n",
              "      <td>-0.360244</td>\n",
              "      <td>-0.102992</td>\n",
              "      <td>0.031882</td>\n",
              "      <td>-0.106679</td>\n",
              "      <td>0.716718</td>\n",
              "      <td>0.197329</td>\n",
              "      <td>-0.261676</td>\n",
              "      <td>-0.326734</td>\n",
              "      <td>-0.202511</td>\n",
              "      <td>0.366927</td>\n",
              "      <td>-0.062942</td>\n",
              "      <td>0.176530</td>\n",
              "      <td>0.060247</td>\n",
              "      <td>-0.298104</td>\n",
              "      <td>-0.551207</td>\n",
              "      <td>0.373706</td>\n",
              "      <td>-0.042755</td>\n",
              "      <td>0.481701</td>\n",
              "      <td>0.260109</td>\n",
              "      <td>0.584887</td>\n",
              "      <td>0.289511</td>\n",
              "      <td>-0.167374</td>\n",
              "      <td>0.010033</td>\n",
              "      <td>0.462439</td>\n",
              "      <td>-0.076547</td>\n",
              "      <td>-0.282838</td>\n",
              "      <td>-0.036585</td>\n",
              "      <td>-0.392120</td>\n",
              "      <td>-0.279264</td>\n",
              "      <td>0.283381</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 104 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "   CJT_TYP_2/FINANZ_SPARER  ...  KK_KUNDENTYP/D19_GESAMT_ANZ_12\n",
              "0                -1.234629  ...                       -0.062534\n",
              "1                 0.977389  ...                        0.130907\n",
              "2                 0.640617  ...                        0.396859\n",
              "3                -2.199834  ...                       -0.323287\n",
              "4                 1.247861  ...                        0.283381\n",
              "\n",
              "[5 rows x 104 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 130
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ewLu1T3SQnnU"
      },
      "source": [
        "Feature selection with PCA has left us with a transformed dataset `train` with less dimensions than before. We can proceed to clustering with this dataframe."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GKInlXKwZoyQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d9bd831f-c55c-44d6-b18e-a6cdf669e235"
      },
      "source": [
        "print(\"Elapsed time: \", str(timedelta(seconds=(timer() - start_run))))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Elapsed time:  0:04:30.844065\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3vcBlVOQWsEe"
      },
      "source": [
        "##Segmentation using unsupervised clustering"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0YkvysoiRXnr"
      },
      "source": [
        "#### Evaluating Clustering Performance - Clusters choice\n",
        "\n",
        "Our approach for determining the quality of the cluster assignments will be:\n",
        "\n",
        "* Randomly select k centroids, (k is equal to the number of clusters chosen)\n",
        "* Record computed sum of the squared error (**SSE**), defined as the sum of the squared Euclidean distances of each point to its closest centroid. (visual **elbow method**). Objective is to minimize SSE.\n",
        "* Record the **silhouette coefficient**, a measure of cluster cohesion and separation. The silhouette ranges from −1 to +1. A high number, close to 1, indicates that a sample is closer to its cluster than to other clusters. (In scikit-learn [`silhouette_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.silhouette_score.html) returns the mean Silhouette Coefficient over all samples in a cluster and is only defined if the number of labels (clusters) is greater than 2.)\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KpoF9-VLduzq"
      },
      "source": [
        "from sklearn.metrics import silhouette_score\n",
        "from sklearn.cluster import MiniBatchKMeans"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jsN-vi9qq8u6"
      },
      "source": [
        "#### Compute SSE and Silhouette coefficient\n",
        "\n",
        "Using the `MiniBatchKMeans` which is much faster compared to `KMeans` and a `sample_size = 10000` for `silhouette_score` we can achieve acceptable results relatively fast for these large datasets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 252
        },
        "id": "4t4q13wZHXHy",
        "outputId": "8a745275-aafd-437a-a0ab-e2bde3de584f"
      },
      "source": [
        "# Create a range of values to try for cluster numbers\n",
        "n_clusters_range = np.arange(2, 15)\n",
        "\n",
        "all_Kscores = []\n",
        "start = timer()\n",
        "# Run several k-means, incrementing k with each iteration, record the SSE and\n",
        "# silhouette_score\n",
        "train_vals = train_reduced.to_numpy()\n",
        "for k in n_clusters_range:\n",
        "    #Initialize k-means using the faster MiniBatchKMeans\n",
        "    clusterer = MiniBatchKMeans(n_clusters=k, random_state=0)\n",
        "    # Fit standardized data with reduced features\n",
        "    clusterer.fit(train_vals)\n",
        "    # Get computed scores, and save them for further reference.\n",
        "    k_scores = {\n",
        "        \"Clusters\": k,\n",
        "        \"SSE\": clusterer.inertia_,\n",
        "        \"Sil_coef\": silhouette_score(train_vals, clusterer.labels_,\n",
        "                                     sample_size=10000,\n",
        "                                     n_jobs=-1),\n",
        "    }\n",
        "    all_Kscores.append(k_scores)\n",
        "\n",
        "print(\"Time elapsed: \", str(timedelta(seconds=(timer() - start))))\n",
        "\n",
        "#Create a dataframe of clustering scores that we can use for plotting.\n",
        "all_Kscores = pd.DataFrame(all_Kscores)\n",
        "all_Kscores.set_index(\"Clusters\", inplace=True)\n",
        "\n",
        "all_Kscores.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Time elapsed:  0:01:33.096290\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>SSE</th>\n",
              "      <th>Sil_coef</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Clusters</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>8.814419e+06</td>\n",
              "      <td>0.070148</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>8.510458e+06</td>\n",
              "      <td>0.055318</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>8.330091e+06</td>\n",
              "      <td>0.056230</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>8.006994e+06</td>\n",
              "      <td>0.054746</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>7.869831e+06</td>\n",
              "      <td>0.053762</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                   SSE  Sil_coef\n",
              "Clusters                        \n",
              "2         8.814419e+06  0.070148\n",
              "3         8.510458e+06  0.055318\n",
              "4         8.330091e+06  0.056230\n",
              "5         8.006994e+06  0.054746\n",
              "6         7.869831e+06  0.053762"
            ]
          },
          "metadata": {},
          "execution_count": 133
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aYFkGlk7qB6b"
      },
      "source": [
        "#### Elbow method\n",
        "\n",
        "Determine the elbow point in the SSE curve using the [kneed](https://github.com/arvkevi/kneed) package:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dZ4NLEWAhOGx",
        "outputId": "84f8746c-688f-4994-e726-6a51574fb64e"
      },
      "source": [
        "# Elbow method knee locator\n",
        "from kneed import KneeLocator\n",
        "\n",
        "kl = KneeLocator(n_clusters_range, all_Kscores.SSE.to_list(),\n",
        "                 curve=\"convex\", direction=\"decreasing\")\n",
        "\n",
        "kl.elbow"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "8"
            ]
          },
          "metadata": {},
          "execution_count": 134
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r0-aq-mspfQN"
      },
      "source": [
        "Optimal number of clusters is found using the **elbow method**, at the iteration where the SSE curve starts to bend, known as the elbow point at `kl.elbow`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MbqByzWTa7TT"
      },
      "source": [
        "# Save image location\n",
        "elbow_sil_file = f'{colab_dir if IN_COLAB else remote_dir}Elbow_silcoef_{ts}.png'\n",
        "\n",
        "with sns.axes_style(\"whitegrid\"), sns.plotting_context(\"paper\"):\n",
        "    fig, ax = plt.subplots(1, 2, figsize=(10, 5))\n",
        "    # plot elbow\n",
        "    sns.lineplot(ax=ax[0], x='Clusters', y='SSE', data=all_Kscores, linewidth=3).set(\n",
        "        xticks=n_clusters_range,\n",
        "        title=\"Elbow method for optimal n_clusters\")\n",
        "    ax[0].axvline(kl.elbow, linestyle='--', color='r', lw=1, alpha=0.65)\n",
        "    # Plot average silhouette\n",
        "    sns.lineplot(ax=ax[1], x='Clusters', y='Sil_coef', data=all_Kscores, linewidth=3).set(\n",
        "        xticks=n_clusters_range,\n",
        "        ylabel='Average silhouette scores',\n",
        "        title=\"Silhouette scores for n_clusters\")\n",
        "    # copyright text\n",
        "    fig.suptitle(cprt, y=-.03, verticalalignment='bottom', bbox=bb_cprt_args)\n",
        "    fig.savefig(elbow_sil_file, transparent=False, bbox_inches=\"tight\")\n",
        "    fig.tight_layout()\n",
        "    plt.close(fig)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jVlPA-LBtrSB"
      },
      "source": [
        "Number of clusters will be chosen from the plot of SSE scores using the elbow method as the decision method. Average silhouette scores at the moment cannot be used as a cluster validation criterion since they are close to 0.0 indicating that the samples are on or very close to the decision boundary. We would have to perform further analysis with the silhouette coefficient to validate our `elbow` method choice with this criterion, but we will leave this for future work. For now, we will use silhouette scores as complementary information, providing a view on our clusters' performance with this criterion.\n",
        "\n",
        "We will fit `MiniBatchKMeans` with the decided clusters number:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SahpgyfNqOqX",
        "outputId": "22f90008-49b5-4884-905d-2823a13fbe3a"
      },
      "source": [
        "clusterer = MiniBatchKMeans(n_clusters=kl.elbow)\n",
        "# train_clusters = pd.Series(clusterer.fit_predict(norm_train.to_numpy()))\n",
        "train_cluster_labels = pd.Series(clusterer.fit_predict(train_vals))\n",
        "\n",
        "train_cluster_labels.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(450000,)"
            ]
          },
          "metadata": {},
          "execution_count": 136
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xXHoa_in28g-",
        "outputId": "f72c8317-5a15-47e4-f59d-79c7f7bb73ce"
      },
      "source": [
        "train_cluster_labels.value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1    64149\n",
              "4    63555\n",
              "0    62222\n",
              "2    61050\n",
              "6    59945\n",
              "3    53187\n",
              "5    50904\n",
              "7    34988\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 137
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "d6o9n5nQsdvE",
        "outputId": "b5e81c4f-4654-45b6-9f86-66a47db86c91"
      },
      "source": [
        "df_clusters = pd.DataFrame({\n",
        "    \"Clusters\": train_cluster_labels,\n",
        "    \"TARGET\": target,\n",
        "})\n",
        "\n",
        "df_clusters.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Clusters</th>\n",
              "      <th>TARGET</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>-1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>6</td>\n",
              "      <td>-1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>6</td>\n",
              "      <td>-1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>5</td>\n",
              "      <td>-1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Clusters  TARGET\n",
              "0         0      -1\n",
              "1         6      -1\n",
              "2         6      -1\n",
              "3         5      -1\n",
              "4         1       1"
            ]
          },
          "metadata": {},
          "execution_count": 138
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 545
        },
        "id": "u6w4CbwONflo",
        "outputId": "691495a4-5bc1-4e6b-d485-315d2d2583c9"
      },
      "source": [
        "# Compute population count per cluster for general population (-1) and customers (1)\n",
        "target_group = df_clusters.groupby(by=\"TARGET\")[\"Clusters\"].value_counts().to_frame(name='Counts').reset_index()\n",
        "# Calculate percentages per cluster per population\n",
        "target_group['Perc'] = target_group.groupby(\"TARGET\")['Counts'].apply(lambda x: round(100 * x.astype(float)/x.sum(), 2))\n",
        "\n",
        "target_group"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>TARGET</th>\n",
              "      <th>Clusters</th>\n",
              "      <th>Counts</th>\n",
              "      <th>Perc</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>-1</td>\n",
              "      <td>6</td>\n",
              "      <td>55418</td>\n",
              "      <td>18.47</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>-1</td>\n",
              "      <td>3</td>\n",
              "      <td>51266</td>\n",
              "      <td>17.09</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>-1</td>\n",
              "      <td>4</td>\n",
              "      <td>47796</td>\n",
              "      <td>15.93</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>-1</td>\n",
              "      <td>0</td>\n",
              "      <td>38008</td>\n",
              "      <td>12.67</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>-1</td>\n",
              "      <td>2</td>\n",
              "      <td>31227</td>\n",
              "      <td>10.41</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>-1</td>\n",
              "      <td>7</td>\n",
              "      <td>30985</td>\n",
              "      <td>10.33</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>-1</td>\n",
              "      <td>1</td>\n",
              "      <td>27521</td>\n",
              "      <td>9.17</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>-1</td>\n",
              "      <td>5</td>\n",
              "      <td>17779</td>\n",
              "      <td>5.93</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>36628</td>\n",
              "      <td>24.42</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>1</td>\n",
              "      <td>5</td>\n",
              "      <td>33125</td>\n",
              "      <td>22.08</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>29823</td>\n",
              "      <td>19.88</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>24214</td>\n",
              "      <td>16.14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>1</td>\n",
              "      <td>4</td>\n",
              "      <td>15759</td>\n",
              "      <td>10.51</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>1</td>\n",
              "      <td>6</td>\n",
              "      <td>4527</td>\n",
              "      <td>3.02</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>1</td>\n",
              "      <td>7</td>\n",
              "      <td>4003</td>\n",
              "      <td>2.67</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>1921</td>\n",
              "      <td>1.28</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    TARGET  Clusters  Counts   Perc\n",
              "0       -1         6   55418  18.47\n",
              "1       -1         3   51266  17.09\n",
              "2       -1         4   47796  15.93\n",
              "3       -1         0   38008  12.67\n",
              "4       -1         2   31227  10.41\n",
              "5       -1         7   30985  10.33\n",
              "6       -1         1   27521   9.17\n",
              "7       -1         5   17779   5.93\n",
              "8        1         1   36628  24.42\n",
              "9        1         5   33125  22.08\n",
              "10       1         2   29823  19.88\n",
              "11       1         0   24214  16.14\n",
              "12       1         4   15759  10.51\n",
              "13       1         6    4527   3.02\n",
              "14       1         7    4003   2.67\n",
              "15       1         3    1921   1.28"
            ]
          },
          "metadata": {},
          "execution_count": 139
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m0OiNO9R36-U"
      },
      "source": [
        "# Plot population per cluster for `AZDIAS` and `CUSTOMERS`\n",
        "popul_clust_file = f'{colab_dir if IN_COLAB else remote_dir}pop_cluster_bars_{ts}.png'\n",
        "\n",
        "with sns.axes_style(\"whitegrid\"):\n",
        "    plt.figure(figsize=(10, 6))\n",
        "    totals = target_group.groupby(\"TARGET\")['Counts'].sum()\n",
        "    ax = sns.barplot(x=\"Clusters\", y='Perc', hue=\"TARGET\", data=target_group, palette='Blues_d')\n",
        "    h, l = ax.get_legend_handles_labels()\n",
        "    ax.legend(h, ['General', 'Customers'], title=\"Population\")\n",
        "    ax.set(title='Population portion per cluster', ylabel='Population percentage (%)')\n",
        "    # add percentage annotations on top of bars\n",
        "    for tot, p in zip(totals, ax.containers):\n",
        "        labels = [str(rect.get_height()) + '%' for rect in p]\n",
        "        for rect, label in zip(p, labels):\n",
        "            ax.text(rect.get_x() + rect.get_width() / 2,\n",
        "                    rect.get_height() * 1.01,\n",
        "                    label,\n",
        "                    ha=\"center\",\n",
        "                    va=\"bottom\",\n",
        "                    fontsize=\"small\",\n",
        "                    fontweight=\"semibold\"\n",
        "            )\n",
        "    # copyright text\n",
        "    plt.suptitle(cprt, y=-.03, verticalalignment='bottom', bbox=bb_cprt_args)\n",
        "    plt.tight_layout()\n",
        "    plt.savefig(popul_clust_file, transparent=False, bbox_inches=\"tight\")\n",
        "    plt.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UmZI--G5XE8e"
      },
      "source": [
        "# Create plotting data combining PCA results with K-Means clustering\n",
        "pcadf = train_reduced.assign(Predicted_cluster=clusterer.labels_, Population=target)\n",
        "pcadf[\"Population\"] = pcadf[\"Population\"].replace(-1, 'General').replace(1, 'Customers')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eR8TgLKbqyeg"
      },
      "source": [
        "popul_clust_file = f'{colab_dir if IN_COLAB else remote_dir}cluster_results_{ts}.png'\n",
        "\n",
        "with sns.axes_style(\"whitegrid\"), sns.plotting_context(\"paper\"):\n",
        "    plt.figure(figsize=(10, 8))\n",
        "    scatr = sns.scatterplot(x=\"PC1\", y=\"PC2\", data=pcadf, hue=\"Predicted_cluster\", style=\"Population\", palette=\"Set2\", s=50,)\n",
        "    scatr.set(title='Clustering results from demographics data')\n",
        "    plt.legend(bbox_to_anchor=(1.01, .6), loc=2, borderaxespad=0.0)\n",
        "    # copyright text\n",
        "    plt.suptitle(cprt, y=-.03, verticalalignment='bottom', bbox=bb_cprt_args)\n",
        "    plt.tight_layout()\n",
        "    plt.savefig(popul_clust_file, transparent=False, bbox_inches=\"tight\")\n",
        "    plt.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4AJwykWHj25H",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 346
        },
        "outputId": "2f302ea1-21d8-4fbd-d976-47b379ae771c"
      },
      "source": [
        "# Clusters centroid locations on PCA space\n",
        "cluster_centroids = pd.DataFrame(clusterer.cluster_centers_, columns=pca_list)\n",
        "cluster_centroids"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>CJT_TYP_2/FINANZ_SPARER</th>\n",
              "      <th>FINANZ_ANLEGER/FINANZ_UNAUFFAELLIGER</th>\n",
              "      <th>OST_WEST_KZ_W/STRUKTURTYP</th>\n",
              "      <th>HH_EINKOMMEN_SCORE/OST_WEST_KZ_O</th>\n",
              "      <th>ANREDE_KZ/SEMIO_KAEM</th>\n",
              "      <th>KBA13_BAUMAX/LP_FAMILIE_FEIN</th>\n",
              "      <th>OST_WEST_KZ_O/KBA13_KW_61_120</th>\n",
              "      <th>PLZ8_ANTG4/KBA13_ANTG4</th>\n",
              "      <th>KBA13_CCM_3001/KBA13_CCM_2501</th>\n",
              "      <th>KBA05_MAXSEG/KBA05_MOTOR</th>\n",
              "      <th>KBA13_ANTG4/PLZ8_ANTG4</th>\n",
              "      <th>KBA05_MAXBJ/SHOPPER_TYP</th>\n",
              "      <th>VERS_TYP/HEALTH_TYP</th>\n",
              "      <th>KBA05_MOD8/KBA05_SEG9</th>\n",
              "      <th>LP_FAMILIE_FEIN/LP_LEBENSPHASE_FEIN</th>\n",
              "      <th>KBA05_KW3/KBA05_CCM4</th>\n",
              "      <th>KBA13_ALTERHALTER_45/KBA13_HALTER_35</th>\n",
              "      <th>KBA13_VORB_1_2/KBA13_ALTERHALTER_61</th>\n",
              "      <th>D19_GESAMT_ANZ_24/D19_VERSAND_ANZ_24</th>\n",
              "      <th>D19_GESAMT_ANZ_24/D19_VERSAND_ANZ_24</th>\n",
              "      <th>KBA13_KRSZUL_NEU/KBA05_MOD8</th>\n",
              "      <th>KBA05_ANHANG/KBA05_ANTG3</th>\n",
              "      <th>KBA05_ANHANG/KBA05_BAUMAX</th>\n",
              "      <th>SHOPPER_TYP/KBA05_ANTG1</th>\n",
              "      <th>SHOPPER_TYP/KBA13_KRSSEG_VAN</th>\n",
              "      <th>KBA05_ANHANG/KBA05_MOTRAD</th>\n",
              "      <th>KBA05_MOTRAD/BALLRAUM</th>\n",
              "      <th>VHN/D19_SOZIALES</th>\n",
              "      <th>PLZ8_ANTG4/KBA05_ANHANG</th>\n",
              "      <th>PLZ8_ANTG4/KBA05_ANTG3</th>\n",
              "      <th>SHOPPER_TYP/KBA13_ANTG4</th>\n",
              "      <th>PLZ8_ANTG4/KBA13_KRSSEG_VAN</th>\n",
              "      <th>D19_LETZTER_KAUF_BRANCHE_D19_UNBEKANNT/D19_KONSUMTYP</th>\n",
              "      <th>KBA05_ANHANG/KBA05_SEG7</th>\n",
              "      <th>KBA05_MOTRAD/KBA05_ANTG3</th>\n",
              "      <th>KBA05_MOTRAD/AGER_TYP</th>\n",
              "      <th>VHN/KBA05_MOTRAD</th>\n",
              "      <th>AGER_TYP/KBA05_ZUL2</th>\n",
              "      <th>KBA05_SEG8/KBA05_SEG7</th>\n",
              "      <th>KBA05_SEG7/AGER_TYP</th>\n",
              "      <th>...</th>\n",
              "      <th>D19_BANKEN_DIREKT/D19_REISEN</th>\n",
              "      <th>D19_REISEN/KBA13_CCM_1500</th>\n",
              "      <th>D19_HAUS_DEKO/D19_BEKLEIDUNG_REST</th>\n",
              "      <th>D19_BEKLEIDUNG_REST/D19_BANKEN_DIREKT</th>\n",
              "      <th>KBA13_KW_70/KBA05_ANTG2</th>\n",
              "      <th>D19_BEKLEIDUNG_REST/D19_HAUS_DEKO</th>\n",
              "      <th>D19_VERSAND_REST/D19_BEKLEIDUNG_REST</th>\n",
              "      <th>KBA05_MOD1/KBA05_SEG10</th>\n",
              "      <th>KK_KUNDENTYP/D19_VERSAND_ANZ_24</th>\n",
              "      <th>KBA05_ANTG2/D19_VOLLSORTIMENT</th>\n",
              "      <th>KBA05_SEG9/D19_TECHNIK</th>\n",
              "      <th>KBA05_ALTER2/D19_VERSAND_ANZ_12</th>\n",
              "      <th>D19_SONSTIGE/D19_VERSAND_ANZ_12</th>\n",
              "      <th>KBA05_SEG9/KBA05_ALTER3</th>\n",
              "      <th>D19_VOLLSORTIMENT/KBA05_HERST5</th>\n",
              "      <th>KBA05_MOD8/D19_REISEN</th>\n",
              "      <th>D19_BUCH_CD/KBA05_KW3</th>\n",
              "      <th>D19_BUCH_CD/KBA05_MOD8</th>\n",
              "      <th>D19_BUCH_CD/KBA05_KRSVAN</th>\n",
              "      <th>D19_VOLLSORTIMENT/D19_GESAMT_ONLINE_QUOTE_12</th>\n",
              "      <th>RELAT_AB/KBA05_KW3</th>\n",
              "      <th>D19_SONSTIGE/EXTSEL992</th>\n",
              "      <th>KBA05_CCM4/D19_SONSTIGE</th>\n",
              "      <th>KBA05_DIESEL/UMFELD_JUNG</th>\n",
              "      <th>KBA05_DIESEL/RT_UEBERGROESSE</th>\n",
              "      <th>KBA13_VORB_2/KBA13_VORB_1_2</th>\n",
              "      <th>GFK_URLAUBERTYP/KBA05_KRSVAN</th>\n",
              "      <th>D19_SONSTIGE/D19_VERSAND_ANZ_24</th>\n",
              "      <th>KBA05_SEG5/KBA13_SEG_OBERKLASSE</th>\n",
              "      <th>KBA05_BAUMAX/UMFELD_ALT</th>\n",
              "      <th>KBA13_KW_90/KBA13_CCM_3001</th>\n",
              "      <th>KBA05_DIESEL/KBA05_HERST4</th>\n",
              "      <th>KBA13_SEG_OBERKLASSE/KBA13_KRSZUL_NEU</th>\n",
              "      <th>KBA05_MOD4/KBA05_SEG5</th>\n",
              "      <th>KBA13_BJ_2008/RETOURTYP_BK_S</th>\n",
              "      <th>KBA05_VORB2/EXTSEL992</th>\n",
              "      <th>KBA05_HERST4/KBA05_FRAU</th>\n",
              "      <th>D19_VERSAND_ONLINE_QUOTE_12/KBA05_KRSVAN</th>\n",
              "      <th>GFK_URLAUBERTYP/D19_VERSAND_ONLINE_QUOTE_12</th>\n",
              "      <th>KK_KUNDENTYP/D19_GESAMT_ANZ_12</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>-1.203329</td>\n",
              "      <td>-0.223992</td>\n",
              "      <td>-0.743978</td>\n",
              "      <td>-0.358931</td>\n",
              "      <td>-0.215974</td>\n",
              "      <td>-0.670338</td>\n",
              "      <td>0.027057</td>\n",
              "      <td>0.045513</td>\n",
              "      <td>-0.050277</td>\n",
              "      <td>-0.068199</td>\n",
              "      <td>0.149190</td>\n",
              "      <td>0.059806</td>\n",
              "      <td>0.010193</td>\n",
              "      <td>0.049111</td>\n",
              "      <td>-0.018365</td>\n",
              "      <td>-0.017170</td>\n",
              "      <td>0.039205</td>\n",
              "      <td>-0.010736</td>\n",
              "      <td>-0.015513</td>\n",
              "      <td>-0.012801</td>\n",
              "      <td>-0.000073</td>\n",
              "      <td>-0.004348</td>\n",
              "      <td>-0.038466</td>\n",
              "      <td>0.002093</td>\n",
              "      <td>-0.035603</td>\n",
              "      <td>0.029497</td>\n",
              "      <td>0.003356</td>\n",
              "      <td>-0.010572</td>\n",
              "      <td>-0.002776</td>\n",
              "      <td>0.007718</td>\n",
              "      <td>0.030059</td>\n",
              "      <td>0.000504</td>\n",
              "      <td>0.040567</td>\n",
              "      <td>-0.015568</td>\n",
              "      <td>-0.025967</td>\n",
              "      <td>0.014422</td>\n",
              "      <td>-0.024002</td>\n",
              "      <td>0.004746</td>\n",
              "      <td>-0.000651</td>\n",
              "      <td>0.002267</td>\n",
              "      <td>...</td>\n",
              "      <td>0.012147</td>\n",
              "      <td>-0.006764</td>\n",
              "      <td>-0.007272</td>\n",
              "      <td>-0.015309</td>\n",
              "      <td>-0.001701</td>\n",
              "      <td>0.000734</td>\n",
              "      <td>-0.015052</td>\n",
              "      <td>-0.017821</td>\n",
              "      <td>-0.004639</td>\n",
              "      <td>-0.000819</td>\n",
              "      <td>-0.010061</td>\n",
              "      <td>-0.008805</td>\n",
              "      <td>0.019471</td>\n",
              "      <td>-0.014947</td>\n",
              "      <td>-0.010884</td>\n",
              "      <td>0.010046</td>\n",
              "      <td>0.002587</td>\n",
              "      <td>-0.004962</td>\n",
              "      <td>0.011280</td>\n",
              "      <td>0.001355</td>\n",
              "      <td>-0.003376</td>\n",
              "      <td>-0.002830</td>\n",
              "      <td>-0.000733</td>\n",
              "      <td>0.022321</td>\n",
              "      <td>-0.007416</td>\n",
              "      <td>-0.015653</td>\n",
              "      <td>0.010489</td>\n",
              "      <td>-0.017771</td>\n",
              "      <td>-0.001920</td>\n",
              "      <td>-0.019528</td>\n",
              "      <td>-0.006500</td>\n",
              "      <td>-0.010605</td>\n",
              "      <td>0.006411</td>\n",
              "      <td>-0.009840</td>\n",
              "      <td>-0.013895</td>\n",
              "      <td>0.006126</td>\n",
              "      <td>-0.005134</td>\n",
              "      <td>0.009664</td>\n",
              "      <td>0.004420</td>\n",
              "      <td>0.002309</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.189757</td>\n",
              "      <td>0.597621</td>\n",
              "      <td>-0.734094</td>\n",
              "      <td>-1.377590</td>\n",
              "      <td>0.097498</td>\n",
              "      <td>0.876267</td>\n",
              "      <td>-0.297897</td>\n",
              "      <td>-0.068185</td>\n",
              "      <td>-0.199166</td>\n",
              "      <td>-0.053083</td>\n",
              "      <td>-0.298689</td>\n",
              "      <td>-0.049058</td>\n",
              "      <td>0.019444</td>\n",
              "      <td>-0.050699</td>\n",
              "      <td>-0.014909</td>\n",
              "      <td>-0.057886</td>\n",
              "      <td>0.039705</td>\n",
              "      <td>-0.005297</td>\n",
              "      <td>0.032628</td>\n",
              "      <td>0.018790</td>\n",
              "      <td>0.027643</td>\n",
              "      <td>-0.017474</td>\n",
              "      <td>-0.004252</td>\n",
              "      <td>-0.011265</td>\n",
              "      <td>0.016628</td>\n",
              "      <td>-0.047244</td>\n",
              "      <td>0.017252</td>\n",
              "      <td>0.018758</td>\n",
              "      <td>-0.035465</td>\n",
              "      <td>0.008850</td>\n",
              "      <td>-0.010114</td>\n",
              "      <td>0.009972</td>\n",
              "      <td>-0.002060</td>\n",
              "      <td>0.031978</td>\n",
              "      <td>-0.014015</td>\n",
              "      <td>0.023253</td>\n",
              "      <td>-0.029281</td>\n",
              "      <td>0.003445</td>\n",
              "      <td>0.002080</td>\n",
              "      <td>0.012119</td>\n",
              "      <td>...</td>\n",
              "      <td>-0.006814</td>\n",
              "      <td>0.006297</td>\n",
              "      <td>0.004343</td>\n",
              "      <td>0.001312</td>\n",
              "      <td>-0.016860</td>\n",
              "      <td>0.003112</td>\n",
              "      <td>0.003790</td>\n",
              "      <td>0.000480</td>\n",
              "      <td>0.013166</td>\n",
              "      <td>0.011383</td>\n",
              "      <td>0.007280</td>\n",
              "      <td>-0.010061</td>\n",
              "      <td>-0.009563</td>\n",
              "      <td>-0.001066</td>\n",
              "      <td>0.000470</td>\n",
              "      <td>-0.000179</td>\n",
              "      <td>0.005277</td>\n",
              "      <td>0.000953</td>\n",
              "      <td>0.002640</td>\n",
              "      <td>-0.000785</td>\n",
              "      <td>0.006743</td>\n",
              "      <td>0.008656</td>\n",
              "      <td>-0.000214</td>\n",
              "      <td>-0.008452</td>\n",
              "      <td>0.016615</td>\n",
              "      <td>0.004329</td>\n",
              "      <td>0.005615</td>\n",
              "      <td>-0.004625</td>\n",
              "      <td>0.007773</td>\n",
              "      <td>-0.008486</td>\n",
              "      <td>0.001407</td>\n",
              "      <td>0.011943</td>\n",
              "      <td>-0.003393</td>\n",
              "      <td>0.023931</td>\n",
              "      <td>0.017953</td>\n",
              "      <td>-0.010767</td>\n",
              "      <td>0.018076</td>\n",
              "      <td>0.001721</td>\n",
              "      <td>0.004933</td>\n",
              "      <td>-0.026375</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>-1.543665</td>\n",
              "      <td>0.865122</td>\n",
              "      <td>0.069302</td>\n",
              "      <td>0.639530</td>\n",
              "      <td>0.278657</td>\n",
              "      <td>0.131343</td>\n",
              "      <td>-0.315198</td>\n",
              "      <td>0.126088</td>\n",
              "      <td>-0.006557</td>\n",
              "      <td>-0.094476</td>\n",
              "      <td>-0.007997</td>\n",
              "      <td>-0.027546</td>\n",
              "      <td>0.058509</td>\n",
              "      <td>0.107610</td>\n",
              "      <td>0.066454</td>\n",
              "      <td>0.002132</td>\n",
              "      <td>-0.009539</td>\n",
              "      <td>-0.022984</td>\n",
              "      <td>0.093150</td>\n",
              "      <td>0.070689</td>\n",
              "      <td>0.012023</td>\n",
              "      <td>0.005554</td>\n",
              "      <td>0.018877</td>\n",
              "      <td>-0.038802</td>\n",
              "      <td>-0.000952</td>\n",
              "      <td>-0.034635</td>\n",
              "      <td>0.042623</td>\n",
              "      <td>0.012724</td>\n",
              "      <td>-0.022934</td>\n",
              "      <td>0.005105</td>\n",
              "      <td>-0.016162</td>\n",
              "      <td>-0.005665</td>\n",
              "      <td>-0.022061</td>\n",
              "      <td>0.020415</td>\n",
              "      <td>-0.016118</td>\n",
              "      <td>0.001692</td>\n",
              "      <td>0.007787</td>\n",
              "      <td>-0.007930</td>\n",
              "      <td>0.001820</td>\n",
              "      <td>-0.002093</td>\n",
              "      <td>...</td>\n",
              "      <td>-0.001516</td>\n",
              "      <td>-0.002141</td>\n",
              "      <td>-0.000380</td>\n",
              "      <td>0.017295</td>\n",
              "      <td>-0.014654</td>\n",
              "      <td>-0.006622</td>\n",
              "      <td>0.002162</td>\n",
              "      <td>-0.001878</td>\n",
              "      <td>0.018022</td>\n",
              "      <td>0.016565</td>\n",
              "      <td>0.009550</td>\n",
              "      <td>-0.008625</td>\n",
              "      <td>-0.009203</td>\n",
              "      <td>-0.007255</td>\n",
              "      <td>0.006330</td>\n",
              "      <td>0.009126</td>\n",
              "      <td>-0.005499</td>\n",
              "      <td>-0.010491</td>\n",
              "      <td>0.003310</td>\n",
              "      <td>-0.016809</td>\n",
              "      <td>0.008243</td>\n",
              "      <td>0.002445</td>\n",
              "      <td>0.020301</td>\n",
              "      <td>0.001632</td>\n",
              "      <td>0.006899</td>\n",
              "      <td>-0.004566</td>\n",
              "      <td>0.001397</td>\n",
              "      <td>0.005374</td>\n",
              "      <td>-0.003155</td>\n",
              "      <td>0.017807</td>\n",
              "      <td>0.008674</td>\n",
              "      <td>0.001151</td>\n",
              "      <td>-0.004938</td>\n",
              "      <td>0.000438</td>\n",
              "      <td>-0.014015</td>\n",
              "      <td>-0.005141</td>\n",
              "      <td>-0.007483</td>\n",
              "      <td>0.005775</td>\n",
              "      <td>0.010885</td>\n",
              "      <td>0.001282</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2.071815</td>\n",
              "      <td>-0.155733</td>\n",
              "      <td>0.834181</td>\n",
              "      <td>0.476433</td>\n",
              "      <td>-0.177871</td>\n",
              "      <td>-0.317296</td>\n",
              "      <td>0.019073</td>\n",
              "      <td>0.046619</td>\n",
              "      <td>-0.026845</td>\n",
              "      <td>0.022611</td>\n",
              "      <td>0.060064</td>\n",
              "      <td>0.114290</td>\n",
              "      <td>-0.105528</td>\n",
              "      <td>-0.003786</td>\n",
              "      <td>0.016345</td>\n",
              "      <td>-0.059173</td>\n",
              "      <td>0.009870</td>\n",
              "      <td>-0.009190</td>\n",
              "      <td>-0.050169</td>\n",
              "      <td>-0.055538</td>\n",
              "      <td>-0.019654</td>\n",
              "      <td>-0.032503</td>\n",
              "      <td>-0.075851</td>\n",
              "      <td>0.067993</td>\n",
              "      <td>0.004569</td>\n",
              "      <td>0.081958</td>\n",
              "      <td>-0.049710</td>\n",
              "      <td>-0.016253</td>\n",
              "      <td>0.041038</td>\n",
              "      <td>-0.000519</td>\n",
              "      <td>0.020002</td>\n",
              "      <td>-0.040254</td>\n",
              "      <td>0.000431</td>\n",
              "      <td>-0.039105</td>\n",
              "      <td>0.009800</td>\n",
              "      <td>-0.018743</td>\n",
              "      <td>-0.022681</td>\n",
              "      <td>-0.004096</td>\n",
              "      <td>-0.002684</td>\n",
              "      <td>-0.008828</td>\n",
              "      <td>...</td>\n",
              "      <td>0.011081</td>\n",
              "      <td>0.001855</td>\n",
              "      <td>-0.010888</td>\n",
              "      <td>-0.001604</td>\n",
              "      <td>0.008896</td>\n",
              "      <td>-0.005247</td>\n",
              "      <td>-0.016757</td>\n",
              "      <td>0.001909</td>\n",
              "      <td>0.001327</td>\n",
              "      <td>-0.005987</td>\n",
              "      <td>-0.006772</td>\n",
              "      <td>-0.016502</td>\n",
              "      <td>0.012524</td>\n",
              "      <td>0.002153</td>\n",
              "      <td>-0.009752</td>\n",
              "      <td>0.013938</td>\n",
              "      <td>-0.005096</td>\n",
              "      <td>-0.010907</td>\n",
              "      <td>0.003215</td>\n",
              "      <td>-0.001776</td>\n",
              "      <td>-0.006953</td>\n",
              "      <td>0.008059</td>\n",
              "      <td>-0.005785</td>\n",
              "      <td>0.026969</td>\n",
              "      <td>-0.016062</td>\n",
              "      <td>-0.008047</td>\n",
              "      <td>-0.002058</td>\n",
              "      <td>-0.010984</td>\n",
              "      <td>-0.009587</td>\n",
              "      <td>0.016956</td>\n",
              "      <td>-0.007382</td>\n",
              "      <td>-0.011393</td>\n",
              "      <td>0.020234</td>\n",
              "      <td>-0.009559</td>\n",
              "      <td>-0.018920</td>\n",
              "      <td>0.014452</td>\n",
              "      <td>-0.026139</td>\n",
              "      <td>0.036937</td>\n",
              "      <td>0.004260</td>\n",
              "      <td>-0.001923</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.383392</td>\n",
              "      <td>-1.583445</td>\n",
              "      <td>0.045390</td>\n",
              "      <td>0.237026</td>\n",
              "      <td>0.166570</td>\n",
              "      <td>-0.333962</td>\n",
              "      <td>-0.279711</td>\n",
              "      <td>0.041165</td>\n",
              "      <td>0.023101</td>\n",
              "      <td>-0.079983</td>\n",
              "      <td>-0.071007</td>\n",
              "      <td>-0.002118</td>\n",
              "      <td>0.100736</td>\n",
              "      <td>0.046703</td>\n",
              "      <td>0.030631</td>\n",
              "      <td>-0.017294</td>\n",
              "      <td>-0.011982</td>\n",
              "      <td>0.006506</td>\n",
              "      <td>0.075630</td>\n",
              "      <td>0.060280</td>\n",
              "      <td>-0.009435</td>\n",
              "      <td>0.041408</td>\n",
              "      <td>0.088005</td>\n",
              "      <td>-0.064816</td>\n",
              "      <td>0.007708</td>\n",
              "      <td>-0.039396</td>\n",
              "      <td>0.026463</td>\n",
              "      <td>0.003002</td>\n",
              "      <td>-0.013578</td>\n",
              "      <td>-0.007715</td>\n",
              "      <td>-0.038570</td>\n",
              "      <td>0.008869</td>\n",
              "      <td>-0.026156</td>\n",
              "      <td>0.036357</td>\n",
              "      <td>-0.014320</td>\n",
              "      <td>0.002437</td>\n",
              "      <td>0.025971</td>\n",
              "      <td>-0.015670</td>\n",
              "      <td>0.000746</td>\n",
              "      <td>-0.007495</td>\n",
              "      <td>...</td>\n",
              "      <td>-0.012677</td>\n",
              "      <td>0.003486</td>\n",
              "      <td>0.009415</td>\n",
              "      <td>-0.001844</td>\n",
              "      <td>0.000302</td>\n",
              "      <td>0.004370</td>\n",
              "      <td>0.017891</td>\n",
              "      <td>-0.007643</td>\n",
              "      <td>0.001666</td>\n",
              "      <td>0.003844</td>\n",
              "      <td>0.005788</td>\n",
              "      <td>-0.000856</td>\n",
              "      <td>0.009435</td>\n",
              "      <td>0.011510</td>\n",
              "      <td>0.025418</td>\n",
              "      <td>-0.019478</td>\n",
              "      <td>0.003128</td>\n",
              "      <td>0.025867</td>\n",
              "      <td>0.004695</td>\n",
              "      <td>0.002093</td>\n",
              "      <td>0.012676</td>\n",
              "      <td>-0.009338</td>\n",
              "      <td>0.014635</td>\n",
              "      <td>-0.027723</td>\n",
              "      <td>0.004416</td>\n",
              "      <td>0.004574</td>\n",
              "      <td>0.000167</td>\n",
              "      <td>0.020463</td>\n",
              "      <td>0.001843</td>\n",
              "      <td>-0.004559</td>\n",
              "      <td>0.003456</td>\n",
              "      <td>0.004103</td>\n",
              "      <td>-0.013973</td>\n",
              "      <td>0.000054</td>\n",
              "      <td>0.009592</td>\n",
              "      <td>-0.003104</td>\n",
              "      <td>0.026791</td>\n",
              "      <td>-0.019342</td>\n",
              "      <td>-0.006984</td>\n",
              "      <td>0.011899</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>-1.607038</td>\n",
              "      <td>-0.772047</td>\n",
              "      <td>1.001595</td>\n",
              "      <td>-0.454125</td>\n",
              "      <td>-0.083941</td>\n",
              "      <td>0.376086</td>\n",
              "      <td>0.209281</td>\n",
              "      <td>-0.092562</td>\n",
              "      <td>-0.025321</td>\n",
              "      <td>0.136563</td>\n",
              "      <td>-0.020303</td>\n",
              "      <td>-0.079387</td>\n",
              "      <td>-0.088281</td>\n",
              "      <td>-0.100529</td>\n",
              "      <td>-0.029477</td>\n",
              "      <td>0.096597</td>\n",
              "      <td>-0.015817</td>\n",
              "      <td>-0.012033</td>\n",
              "      <td>-0.079284</td>\n",
              "      <td>-0.062103</td>\n",
              "      <td>0.019498</td>\n",
              "      <td>0.021582</td>\n",
              "      <td>-0.013966</td>\n",
              "      <td>-0.012060</td>\n",
              "      <td>0.029826</td>\n",
              "      <td>-0.000923</td>\n",
              "      <td>-0.049778</td>\n",
              "      <td>-0.032860</td>\n",
              "      <td>0.035909</td>\n",
              "      <td>-0.007440</td>\n",
              "      <td>0.021073</td>\n",
              "      <td>0.000576</td>\n",
              "      <td>-0.036172</td>\n",
              "      <td>-0.001540</td>\n",
              "      <td>0.031169</td>\n",
              "      <td>-0.004095</td>\n",
              "      <td>-0.002573</td>\n",
              "      <td>-0.006358</td>\n",
              "      <td>-0.004733</td>\n",
              "      <td>-0.010137</td>\n",
              "      <td>...</td>\n",
              "      <td>-0.014414</td>\n",
              "      <td>0.023307</td>\n",
              "      <td>0.006260</td>\n",
              "      <td>-0.001052</td>\n",
              "      <td>0.016425</td>\n",
              "      <td>0.005924</td>\n",
              "      <td>0.003374</td>\n",
              "      <td>0.027679</td>\n",
              "      <td>-0.007631</td>\n",
              "      <td>-0.015257</td>\n",
              "      <td>0.005794</td>\n",
              "      <td>0.013913</td>\n",
              "      <td>-0.020306</td>\n",
              "      <td>0.008545</td>\n",
              "      <td>-0.000377</td>\n",
              "      <td>-0.008332</td>\n",
              "      <td>0.001839</td>\n",
              "      <td>0.001901</td>\n",
              "      <td>-0.016482</td>\n",
              "      <td>0.001070</td>\n",
              "      <td>-0.007092</td>\n",
              "      <td>0.007975</td>\n",
              "      <td>-0.014949</td>\n",
              "      <td>0.000185</td>\n",
              "      <td>0.002291</td>\n",
              "      <td>0.011256</td>\n",
              "      <td>0.002081</td>\n",
              "      <td>0.001304</td>\n",
              "      <td>0.001137</td>\n",
              "      <td>0.019076</td>\n",
              "      <td>-0.001702</td>\n",
              "      <td>0.003213</td>\n",
              "      <td>0.004473</td>\n",
              "      <td>0.005102</td>\n",
              "      <td>0.006014</td>\n",
              "      <td>-0.000728</td>\n",
              "      <td>-0.001237</td>\n",
              "      <td>0.007103</td>\n",
              "      <td>0.001534</td>\n",
              "      <td>-0.013189</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>0.356377</td>\n",
              "      <td>1.419541</td>\n",
              "      <td>0.656143</td>\n",
              "      <td>0.187513</td>\n",
              "      <td>-0.183262</td>\n",
              "      <td>-0.411397</td>\n",
              "      <td>0.211959</td>\n",
              "      <td>-0.116999</td>\n",
              "      <td>0.046620</td>\n",
              "      <td>0.022663</td>\n",
              "      <td>0.073723</td>\n",
              "      <td>0.107387</td>\n",
              "      <td>-0.017441</td>\n",
              "      <td>-0.046547</td>\n",
              "      <td>-0.072299</td>\n",
              "      <td>-0.027761</td>\n",
              "      <td>-0.036820</td>\n",
              "      <td>0.017779</td>\n",
              "      <td>-0.036726</td>\n",
              "      <td>-0.036720</td>\n",
              "      <td>-0.023301</td>\n",
              "      <td>-0.012562</td>\n",
              "      <td>0.002959</td>\n",
              "      <td>0.052055</td>\n",
              "      <td>-0.005372</td>\n",
              "      <td>0.026846</td>\n",
              "      <td>-0.006625</td>\n",
              "      <td>0.021713</td>\n",
              "      <td>0.007520</td>\n",
              "      <td>-0.009137</td>\n",
              "      <td>-0.001551</td>\n",
              "      <td>0.007817</td>\n",
              "      <td>0.045002</td>\n",
              "      <td>-0.027059</td>\n",
              "      <td>0.016073</td>\n",
              "      <td>-0.010376</td>\n",
              "      <td>0.016260</td>\n",
              "      <td>0.018186</td>\n",
              "      <td>-0.009968</td>\n",
              "      <td>0.009241</td>\n",
              "      <td>...</td>\n",
              "      <td>0.004681</td>\n",
              "      <td>-0.014665</td>\n",
              "      <td>0.002061</td>\n",
              "      <td>0.004650</td>\n",
              "      <td>0.000720</td>\n",
              "      <td>-0.008986</td>\n",
              "      <td>-0.001800</td>\n",
              "      <td>-0.004639</td>\n",
              "      <td>-0.013885</td>\n",
              "      <td>-0.007760</td>\n",
              "      <td>-0.003791</td>\n",
              "      <td>0.013050</td>\n",
              "      <td>0.015975</td>\n",
              "      <td>-0.003688</td>\n",
              "      <td>0.003260</td>\n",
              "      <td>-0.008653</td>\n",
              "      <td>0.003287</td>\n",
              "      <td>0.015633</td>\n",
              "      <td>0.001504</td>\n",
              "      <td>0.001718</td>\n",
              "      <td>-0.002294</td>\n",
              "      <td>-0.008893</td>\n",
              "      <td>-0.001871</td>\n",
              "      <td>-0.017582</td>\n",
              "      <td>-0.008499</td>\n",
              "      <td>-0.002285</td>\n",
              "      <td>0.007688</td>\n",
              "      <td>0.005121</td>\n",
              "      <td>-0.000743</td>\n",
              "      <td>-0.018593</td>\n",
              "      <td>-0.004291</td>\n",
              "      <td>-0.003616</td>\n",
              "      <td>-0.001385</td>\n",
              "      <td>-0.017832</td>\n",
              "      <td>0.007758</td>\n",
              "      <td>0.007508</td>\n",
              "      <td>0.010655</td>\n",
              "      <td>-0.034398</td>\n",
              "      <td>-0.014357</td>\n",
              "      <td>0.021575</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>0.499745</td>\n",
              "      <td>-0.331242</td>\n",
              "      <td>-1.415351</td>\n",
              "      <td>1.212553</td>\n",
              "      <td>0.126358</td>\n",
              "      <td>0.574479</td>\n",
              "      <td>0.860138</td>\n",
              "      <td>0.023800</td>\n",
              "      <td>0.430271</td>\n",
              "      <td>0.258688</td>\n",
              "      <td>0.237820</td>\n",
              "      <td>-0.210620</td>\n",
              "      <td>-0.001895</td>\n",
              "      <td>-0.022048</td>\n",
              "      <td>0.012727</td>\n",
              "      <td>0.158163</td>\n",
              "      <td>-0.038242</td>\n",
              "      <td>0.071895</td>\n",
              "      <td>-0.066783</td>\n",
              "      <td>-0.000243</td>\n",
              "      <td>-0.013030</td>\n",
              "      <td>-0.010907</td>\n",
              "      <td>0.024794</td>\n",
              "      <td>0.032245</td>\n",
              "      <td>-0.011475</td>\n",
              "      <td>0.005190</td>\n",
              "      <td>-0.009434</td>\n",
              "      <td>-0.006140</td>\n",
              "      <td>-0.012387</td>\n",
              "      <td>-0.002516</td>\n",
              "      <td>-0.000688</td>\n",
              "      <td>0.010824</td>\n",
              "      <td>-0.015590</td>\n",
              "      <td>-0.024089</td>\n",
              "      <td>0.022528</td>\n",
              "      <td>-0.021848</td>\n",
              "      <td>0.036835</td>\n",
              "      <td>0.009844</td>\n",
              "      <td>0.016655</td>\n",
              "      <td>-0.001682</td>\n",
              "      <td>...</td>\n",
              "      <td>0.006454</td>\n",
              "      <td>-0.011276</td>\n",
              "      <td>-0.011746</td>\n",
              "      <td>0.000214</td>\n",
              "      <td>0.009906</td>\n",
              "      <td>0.004468</td>\n",
              "      <td>0.008192</td>\n",
              "      <td>0.013086</td>\n",
              "      <td>-0.018237</td>\n",
              "      <td>-0.007702</td>\n",
              "      <td>-0.018482</td>\n",
              "      <td>0.040959</td>\n",
              "      <td>-0.035576</td>\n",
              "      <td>0.009252</td>\n",
              "      <td>-0.030392</td>\n",
              "      <td>0.011468</td>\n",
              "      <td>-0.020099</td>\n",
              "      <td>-0.031462</td>\n",
              "      <td>-0.020549</td>\n",
              "      <td>0.008231</td>\n",
              "      <td>-0.017160</td>\n",
              "      <td>-0.015488</td>\n",
              "      <td>-0.007786</td>\n",
              "      <td>0.010213</td>\n",
              "      <td>0.007401</td>\n",
              "      <td>0.023361</td>\n",
              "      <td>-0.040144</td>\n",
              "      <td>0.005142</td>\n",
              "      <td>0.005635</td>\n",
              "      <td>0.003293</td>\n",
              "      <td>0.013838</td>\n",
              "      <td>0.005747</td>\n",
              "      <td>-0.001707</td>\n",
              "      <td>0.011066</td>\n",
              "      <td>0.012436</td>\n",
              "      <td>-0.007342</td>\n",
              "      <td>-0.032992</td>\n",
              "      <td>-0.007761</td>\n",
              "      <td>-0.005308</td>\n",
              "      <td>0.006786</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>8 rows × 104 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "   CJT_TYP_2/FINANZ_SPARER  ...  KK_KUNDENTYP/D19_GESAMT_ANZ_12\n",
              "0                -1.203329  ...                        0.002309\n",
              "1                 1.189757  ...                       -0.026375\n",
              "2                -1.543665  ...                        0.001282\n",
              "3                 2.071815  ...                       -0.001923\n",
              "4                 0.383392  ...                        0.011899\n",
              "5                -1.607038  ...                       -0.013189\n",
              "6                 0.356377  ...                        0.021575\n",
              "7                 0.499745  ...                        0.006786\n",
              "\n",
              "[8 rows x 104 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 143
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Doyq7VVZa7x5"
      },
      "source": [
        "centroid_heatmp_file = f'{colab_dir if IN_COLAB else remote_dir}centroids_heatmap_{ts}.png'\n",
        "\n",
        "# Plot a heatmap with centroid coordinates of the top 20 most important PCs\n",
        "with sns.plotting_context(\"paper\"):\n",
        "    plt.figure(figsize=(8, 6))\n",
        "    ax = sns.heatmap(cluster_centroids.T.iloc[:20, :], annot=True, cmap='YlGnBu')\n",
        "    ax.set(xlabel=\"Cluster\", title=\"Attribute Value by Centroid\")\n",
        "    # copyright text\n",
        "    plt.suptitle(cprt, y=-.03, verticalalignment='bottom', bbox=bb_cprt_args)\n",
        "    plt.tight_layout()\n",
        "    plt.savefig(centroid_heatmp_file, transparent=False, bbox_inches=\"tight\")\n",
        "    plt.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 253
        },
        "id": "F4dxhyYOfOMn",
        "outputId": "c6190336-32e9-4468-b005-838149564ae9"
      },
      "source": [
        "# map the cluster labels back to each individual person from the population data\n",
        "population_transformed = pd.concat([population_transformed, df_clusters], axis=1)\n",
        "# Add the LNR attribute to be able to associate with original data\n",
        "population_transformed = population_transformed.assign(LNR=clientIDs)\n",
        "population_transformed.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>CJT_TYP_2/FINANZ_SPARER</th>\n",
              "      <th>FINANZ_ANLEGER/FINANZ_UNAUFFAELLIGER</th>\n",
              "      <th>OST_WEST_KZ_W/STRUKTURTYP</th>\n",
              "      <th>HH_EINKOMMEN_SCORE/OST_WEST_KZ_O</th>\n",
              "      <th>ANREDE_KZ/SEMIO_KAEM</th>\n",
              "      <th>KBA13_BAUMAX/LP_FAMILIE_FEIN</th>\n",
              "      <th>OST_WEST_KZ_O/KBA13_KW_61_120</th>\n",
              "      <th>PLZ8_ANTG4/KBA13_ANTG4</th>\n",
              "      <th>KBA13_CCM_3001/KBA13_CCM_2501</th>\n",
              "      <th>KBA05_MAXSEG/KBA05_MOTOR</th>\n",
              "      <th>KBA13_ANTG4/PLZ8_ANTG4</th>\n",
              "      <th>KBA05_MAXBJ/SHOPPER_TYP</th>\n",
              "      <th>VERS_TYP/HEALTH_TYP</th>\n",
              "      <th>KBA05_MOD8/KBA05_SEG9</th>\n",
              "      <th>LP_FAMILIE_FEIN/LP_LEBENSPHASE_FEIN</th>\n",
              "      <th>KBA05_KW3/KBA05_CCM4</th>\n",
              "      <th>KBA13_ALTERHALTER_45/KBA13_HALTER_35</th>\n",
              "      <th>KBA13_VORB_1_2/KBA13_ALTERHALTER_61</th>\n",
              "      <th>D19_GESAMT_ANZ_24/D19_VERSAND_ANZ_24</th>\n",
              "      <th>D19_GESAMT_ANZ_24/D19_VERSAND_ANZ_24</th>\n",
              "      <th>KBA13_KRSZUL_NEU/KBA05_MOD8</th>\n",
              "      <th>KBA05_ANHANG/KBA05_ANTG3</th>\n",
              "      <th>KBA05_ANHANG/KBA05_BAUMAX</th>\n",
              "      <th>SHOPPER_TYP/KBA05_ANTG1</th>\n",
              "      <th>SHOPPER_TYP/KBA13_KRSSEG_VAN</th>\n",
              "      <th>KBA05_ANHANG/KBA05_MOTRAD</th>\n",
              "      <th>KBA05_MOTRAD/BALLRAUM</th>\n",
              "      <th>VHN/D19_SOZIALES</th>\n",
              "      <th>PLZ8_ANTG4/KBA05_ANHANG</th>\n",
              "      <th>PLZ8_ANTG4/KBA05_ANTG3</th>\n",
              "      <th>SHOPPER_TYP/KBA13_ANTG4</th>\n",
              "      <th>PLZ8_ANTG4/KBA13_KRSSEG_VAN</th>\n",
              "      <th>D19_LETZTER_KAUF_BRANCHE_D19_UNBEKANNT/D19_KONSUMTYP</th>\n",
              "      <th>KBA05_ANHANG/KBA05_SEG7</th>\n",
              "      <th>KBA05_MOTRAD/KBA05_ANTG3</th>\n",
              "      <th>KBA05_MOTRAD/AGER_TYP</th>\n",
              "      <th>VHN/KBA05_MOTRAD</th>\n",
              "      <th>AGER_TYP/KBA05_ZUL2</th>\n",
              "      <th>KBA05_SEG8/KBA05_SEG7</th>\n",
              "      <th>KBA05_SEG7/AGER_TYP</th>\n",
              "      <th>...</th>\n",
              "      <th>D19_BEKLEIDUNG_REST/D19_BANKEN_DIREKT</th>\n",
              "      <th>KBA13_KW_70/KBA05_ANTG2</th>\n",
              "      <th>D19_BEKLEIDUNG_REST/D19_HAUS_DEKO</th>\n",
              "      <th>D19_VERSAND_REST/D19_BEKLEIDUNG_REST</th>\n",
              "      <th>KBA05_MOD1/KBA05_SEG10</th>\n",
              "      <th>KK_KUNDENTYP/D19_VERSAND_ANZ_24</th>\n",
              "      <th>KBA05_ANTG2/D19_VOLLSORTIMENT</th>\n",
              "      <th>KBA05_SEG9/D19_TECHNIK</th>\n",
              "      <th>KBA05_ALTER2/D19_VERSAND_ANZ_12</th>\n",
              "      <th>D19_SONSTIGE/D19_VERSAND_ANZ_12</th>\n",
              "      <th>KBA05_SEG9/KBA05_ALTER3</th>\n",
              "      <th>D19_VOLLSORTIMENT/KBA05_HERST5</th>\n",
              "      <th>KBA05_MOD8/D19_REISEN</th>\n",
              "      <th>D19_BUCH_CD/KBA05_KW3</th>\n",
              "      <th>D19_BUCH_CD/KBA05_MOD8</th>\n",
              "      <th>D19_BUCH_CD/KBA05_KRSVAN</th>\n",
              "      <th>D19_VOLLSORTIMENT/D19_GESAMT_ONLINE_QUOTE_12</th>\n",
              "      <th>RELAT_AB/KBA05_KW3</th>\n",
              "      <th>D19_SONSTIGE/EXTSEL992</th>\n",
              "      <th>KBA05_CCM4/D19_SONSTIGE</th>\n",
              "      <th>KBA05_DIESEL/UMFELD_JUNG</th>\n",
              "      <th>KBA05_DIESEL/RT_UEBERGROESSE</th>\n",
              "      <th>KBA13_VORB_2/KBA13_VORB_1_2</th>\n",
              "      <th>GFK_URLAUBERTYP/KBA05_KRSVAN</th>\n",
              "      <th>D19_SONSTIGE/D19_VERSAND_ANZ_24</th>\n",
              "      <th>KBA05_SEG5/KBA13_SEG_OBERKLASSE</th>\n",
              "      <th>KBA05_BAUMAX/UMFELD_ALT</th>\n",
              "      <th>KBA13_KW_90/KBA13_CCM_3001</th>\n",
              "      <th>KBA05_DIESEL/KBA05_HERST4</th>\n",
              "      <th>KBA13_SEG_OBERKLASSE/KBA13_KRSZUL_NEU</th>\n",
              "      <th>KBA05_MOD4/KBA05_SEG5</th>\n",
              "      <th>KBA13_BJ_2008/RETOURTYP_BK_S</th>\n",
              "      <th>KBA05_VORB2/EXTSEL992</th>\n",
              "      <th>KBA05_HERST4/KBA05_FRAU</th>\n",
              "      <th>D19_VERSAND_ONLINE_QUOTE_12/KBA05_KRSVAN</th>\n",
              "      <th>GFK_URLAUBERTYP/D19_VERSAND_ONLINE_QUOTE_12</th>\n",
              "      <th>KK_KUNDENTYP/D19_GESAMT_ANZ_12</th>\n",
              "      <th>Clusters</th>\n",
              "      <th>TARGET</th>\n",
              "      <th>LNR</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>-1.234629</td>\n",
              "      <td>-0.867582</td>\n",
              "      <td>-0.578983</td>\n",
              "      <td>-0.220230</td>\n",
              "      <td>-0.742176</td>\n",
              "      <td>0.211376</td>\n",
              "      <td>-0.222328</td>\n",
              "      <td>0.117435</td>\n",
              "      <td>0.625594</td>\n",
              "      <td>-0.276393</td>\n",
              "      <td>0.209671</td>\n",
              "      <td>1.080525</td>\n",
              "      <td>0.073092</td>\n",
              "      <td>-0.005344</td>\n",
              "      <td>-0.725521</td>\n",
              "      <td>0.534920</td>\n",
              "      <td>-0.816195</td>\n",
              "      <td>0.000018</td>\n",
              "      <td>-0.548220</td>\n",
              "      <td>-0.422507</td>\n",
              "      <td>0.358123</td>\n",
              "      <td>0.910408</td>\n",
              "      <td>-0.577757</td>\n",
              "      <td>-0.334013</td>\n",
              "      <td>-0.317721</td>\n",
              "      <td>-0.865876</td>\n",
              "      <td>-0.034087</td>\n",
              "      <td>0.055927</td>\n",
              "      <td>0.069684</td>\n",
              "      <td>0.000181</td>\n",
              "      <td>0.192486</td>\n",
              "      <td>0.326819</td>\n",
              "      <td>-0.207724</td>\n",
              "      <td>0.509368</td>\n",
              "      <td>-0.385418</td>\n",
              "      <td>0.339499</td>\n",
              "      <td>0.397615</td>\n",
              "      <td>0.350358</td>\n",
              "      <td>-0.024274</td>\n",
              "      <td>0.040678</td>\n",
              "      <td>...</td>\n",
              "      <td>0.491691</td>\n",
              "      <td>0.887629</td>\n",
              "      <td>0.179989</td>\n",
              "      <td>0.231284</td>\n",
              "      <td>0.223842</td>\n",
              "      <td>0.026079</td>\n",
              "      <td>-0.465061</td>\n",
              "      <td>-0.227642</td>\n",
              "      <td>-0.249795</td>\n",
              "      <td>-0.011948</td>\n",
              "      <td>0.248586</td>\n",
              "      <td>-0.388448</td>\n",
              "      <td>0.285329</td>\n",
              "      <td>-0.139759</td>\n",
              "      <td>-0.337783</td>\n",
              "      <td>-0.083996</td>\n",
              "      <td>-0.233873</td>\n",
              "      <td>-0.179353</td>\n",
              "      <td>-0.448052</td>\n",
              "      <td>-0.513697</td>\n",
              "      <td>-0.347848</td>\n",
              "      <td>-0.781476</td>\n",
              "      <td>0.001404</td>\n",
              "      <td>-0.283709</td>\n",
              "      <td>-0.331719</td>\n",
              "      <td>0.221276</td>\n",
              "      <td>-0.163244</td>\n",
              "      <td>0.291595</td>\n",
              "      <td>0.186463</td>\n",
              "      <td>-0.005497</td>\n",
              "      <td>-0.228021</td>\n",
              "      <td>0.307563</td>\n",
              "      <td>-0.713716</td>\n",
              "      <td>0.401752</td>\n",
              "      <td>0.477682</td>\n",
              "      <td>-0.182475</td>\n",
              "      <td>-0.062534</td>\n",
              "      <td>0</td>\n",
              "      <td>-1</td>\n",
              "      <td>780657</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.977389</td>\n",
              "      <td>0.134613</td>\n",
              "      <td>0.048990</td>\n",
              "      <td>-0.160303</td>\n",
              "      <td>-0.804427</td>\n",
              "      <td>-0.352177</td>\n",
              "      <td>0.670749</td>\n",
              "      <td>-1.172047</td>\n",
              "      <td>-1.055501</td>\n",
              "      <td>0.573792</td>\n",
              "      <td>0.661885</td>\n",
              "      <td>-1.608553</td>\n",
              "      <td>0.358878</td>\n",
              "      <td>0.913354</td>\n",
              "      <td>-1.173424</td>\n",
              "      <td>0.805668</td>\n",
              "      <td>-0.148719</td>\n",
              "      <td>0.882287</td>\n",
              "      <td>0.335386</td>\n",
              "      <td>0.451623</td>\n",
              "      <td>0.064425</td>\n",
              "      <td>1.197852</td>\n",
              "      <td>-0.666765</td>\n",
              "      <td>0.302235</td>\n",
              "      <td>0.656651</td>\n",
              "      <td>0.195785</td>\n",
              "      <td>-0.440172</td>\n",
              "      <td>0.129148</td>\n",
              "      <td>-0.404084</td>\n",
              "      <td>0.013804</td>\n",
              "      <td>-0.379258</td>\n",
              "      <td>0.256196</td>\n",
              "      <td>-0.816625</td>\n",
              "      <td>0.160029</td>\n",
              "      <td>0.401891</td>\n",
              "      <td>-0.057911</td>\n",
              "      <td>0.288800</td>\n",
              "      <td>0.054266</td>\n",
              "      <td>0.203565</td>\n",
              "      <td>-0.225481</td>\n",
              "      <td>...</td>\n",
              "      <td>0.597577</td>\n",
              "      <td>0.099311</td>\n",
              "      <td>0.298713</td>\n",
              "      <td>0.140128</td>\n",
              "      <td>0.083948</td>\n",
              "      <td>-0.230444</td>\n",
              "      <td>0.391376</td>\n",
              "      <td>0.510545</td>\n",
              "      <td>0.052131</td>\n",
              "      <td>0.096806</td>\n",
              "      <td>-0.743018</td>\n",
              "      <td>0.162313</td>\n",
              "      <td>-0.162888</td>\n",
              "      <td>-0.008984</td>\n",
              "      <td>0.063964</td>\n",
              "      <td>-0.152078</td>\n",
              "      <td>-0.096144</td>\n",
              "      <td>-0.237473</td>\n",
              "      <td>0.091904</td>\n",
              "      <td>0.133245</td>\n",
              "      <td>0.210970</td>\n",
              "      <td>-0.439297</td>\n",
              "      <td>0.643481</td>\n",
              "      <td>0.427124</td>\n",
              "      <td>0.061224</td>\n",
              "      <td>0.371090</td>\n",
              "      <td>-0.009132</td>\n",
              "      <td>-0.335295</td>\n",
              "      <td>-0.327904</td>\n",
              "      <td>0.320068</td>\n",
              "      <td>-0.003117</td>\n",
              "      <td>-0.307557</td>\n",
              "      <td>-0.310712</td>\n",
              "      <td>-0.153566</td>\n",
              "      <td>0.351870</td>\n",
              "      <td>-0.000796</td>\n",
              "      <td>0.130907</td>\n",
              "      <td>6</td>\n",
              "      <td>-1</td>\n",
              "      <td>283232</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.640617</td>\n",
              "      <td>0.397748</td>\n",
              "      <td>0.081203</td>\n",
              "      <td>-0.250016</td>\n",
              "      <td>0.403747</td>\n",
              "      <td>-1.531851</td>\n",
              "      <td>-0.125694</td>\n",
              "      <td>0.206615</td>\n",
              "      <td>1.062772</td>\n",
              "      <td>-0.607811</td>\n",
              "      <td>-0.043918</td>\n",
              "      <td>-0.782834</td>\n",
              "      <td>0.357914</td>\n",
              "      <td>0.292886</td>\n",
              "      <td>-0.547790</td>\n",
              "      <td>-0.100105</td>\n",
              "      <td>-0.214742</td>\n",
              "      <td>0.114630</td>\n",
              "      <td>-0.347420</td>\n",
              "      <td>-0.215213</td>\n",
              "      <td>0.445487</td>\n",
              "      <td>-0.195424</td>\n",
              "      <td>-0.093167</td>\n",
              "      <td>-0.405749</td>\n",
              "      <td>0.601990</td>\n",
              "      <td>-0.136346</td>\n",
              "      <td>-0.291381</td>\n",
              "      <td>0.456311</td>\n",
              "      <td>-0.518213</td>\n",
              "      <td>-0.715454</td>\n",
              "      <td>-0.119906</td>\n",
              "      <td>-0.011904</td>\n",
              "      <td>-0.492868</td>\n",
              "      <td>-0.152883</td>\n",
              "      <td>-0.046090</td>\n",
              "      <td>0.267519</td>\n",
              "      <td>-0.506884</td>\n",
              "      <td>-0.513384</td>\n",
              "      <td>-0.453967</td>\n",
              "      <td>0.361384</td>\n",
              "      <td>...</td>\n",
              "      <td>-0.159854</td>\n",
              "      <td>-0.263444</td>\n",
              "      <td>0.874286</td>\n",
              "      <td>0.041411</td>\n",
              "      <td>0.421441</td>\n",
              "      <td>-0.227531</td>\n",
              "      <td>-0.088987</td>\n",
              "      <td>0.038793</td>\n",
              "      <td>-0.179560</td>\n",
              "      <td>-0.421240</td>\n",
              "      <td>-0.329272</td>\n",
              "      <td>0.088884</td>\n",
              "      <td>0.130497</td>\n",
              "      <td>0.239671</td>\n",
              "      <td>0.161087</td>\n",
              "      <td>0.131114</td>\n",
              "      <td>-0.074357</td>\n",
              "      <td>-0.439820</td>\n",
              "      <td>-0.504712</td>\n",
              "      <td>-0.170880</td>\n",
              "      <td>-0.202946</td>\n",
              "      <td>-0.042772</td>\n",
              "      <td>0.007435</td>\n",
              "      <td>0.203907</td>\n",
              "      <td>-0.274071</td>\n",
              "      <td>0.202893</td>\n",
              "      <td>-0.240996</td>\n",
              "      <td>-0.050707</td>\n",
              "      <td>-0.097291</td>\n",
              "      <td>-0.152809</td>\n",
              "      <td>0.085488</td>\n",
              "      <td>0.211906</td>\n",
              "      <td>-0.143104</td>\n",
              "      <td>0.328685</td>\n",
              "      <td>-0.102155</td>\n",
              "      <td>0.400096</td>\n",
              "      <td>0.396859</td>\n",
              "      <td>6</td>\n",
              "      <td>-1</td>\n",
              "      <td>728552</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>-2.199834</td>\n",
              "      <td>-0.007616</td>\n",
              "      <td>1.126011</td>\n",
              "      <td>-1.318851</td>\n",
              "      <td>1.140490</td>\n",
              "      <td>0.057711</td>\n",
              "      <td>1.092936</td>\n",
              "      <td>0.061931</td>\n",
              "      <td>-0.510787</td>\n",
              "      <td>0.248087</td>\n",
              "      <td>1.407731</td>\n",
              "      <td>-0.546964</td>\n",
              "      <td>-1.101588</td>\n",
              "      <td>0.236766</td>\n",
              "      <td>0.230220</td>\n",
              "      <td>-0.052759</td>\n",
              "      <td>-0.764123</td>\n",
              "      <td>-0.253049</td>\n",
              "      <td>-0.406299</td>\n",
              "      <td>-0.314603</td>\n",
              "      <td>-0.711263</td>\n",
              "      <td>0.080031</td>\n",
              "      <td>-0.518777</td>\n",
              "      <td>-0.530842</td>\n",
              "      <td>-0.018792</td>\n",
              "      <td>0.094442</td>\n",
              "      <td>0.103434</td>\n",
              "      <td>-0.471235</td>\n",
              "      <td>-0.115663</td>\n",
              "      <td>0.119208</td>\n",
              "      <td>-0.258963</td>\n",
              "      <td>-0.118394</td>\n",
              "      <td>0.020771</td>\n",
              "      <td>0.095762</td>\n",
              "      <td>-0.096231</td>\n",
              "      <td>0.039596</td>\n",
              "      <td>0.050026</td>\n",
              "      <td>-0.136419</td>\n",
              "      <td>-0.129697</td>\n",
              "      <td>0.468808</td>\n",
              "      <td>...</td>\n",
              "      <td>0.017804</td>\n",
              "      <td>0.069867</td>\n",
              "      <td>-0.359567</td>\n",
              "      <td>0.394705</td>\n",
              "      <td>-0.420672</td>\n",
              "      <td>0.274572</td>\n",
              "      <td>0.142926</td>\n",
              "      <td>-0.218485</td>\n",
              "      <td>0.258072</td>\n",
              "      <td>0.359043</td>\n",
              "      <td>0.245578</td>\n",
              "      <td>-0.307132</td>\n",
              "      <td>0.119405</td>\n",
              "      <td>0.432255</td>\n",
              "      <td>0.264530</td>\n",
              "      <td>-0.079858</td>\n",
              "      <td>0.020682</td>\n",
              "      <td>0.426232</td>\n",
              "      <td>-0.092529</td>\n",
              "      <td>-0.215258</td>\n",
              "      <td>0.162319</td>\n",
              "      <td>0.053370</td>\n",
              "      <td>0.113821</td>\n",
              "      <td>0.497985</td>\n",
              "      <td>-0.488428</td>\n",
              "      <td>-0.038022</td>\n",
              "      <td>0.106081</td>\n",
              "      <td>0.070431</td>\n",
              "      <td>-0.067926</td>\n",
              "      <td>0.348257</td>\n",
              "      <td>-0.685797</td>\n",
              "      <td>0.230047</td>\n",
              "      <td>-0.211343</td>\n",
              "      <td>-0.124144</td>\n",
              "      <td>0.192463</td>\n",
              "      <td>0.117103</td>\n",
              "      <td>-0.323287</td>\n",
              "      <td>5</td>\n",
              "      <td>-1</td>\n",
              "      <td>390673</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1.247861</td>\n",
              "      <td>0.700665</td>\n",
              "      <td>-0.974589</td>\n",
              "      <td>-1.270208</td>\n",
              "      <td>0.270992</td>\n",
              "      <td>0.698793</td>\n",
              "      <td>-0.026303</td>\n",
              "      <td>-0.420971</td>\n",
              "      <td>0.619710</td>\n",
              "      <td>0.289441</td>\n",
              "      <td>-0.748615</td>\n",
              "      <td>0.083695</td>\n",
              "      <td>-0.194543</td>\n",
              "      <td>-0.113975</td>\n",
              "      <td>-0.428714</td>\n",
              "      <td>0.272176</td>\n",
              "      <td>0.096895</td>\n",
              "      <td>0.909333</td>\n",
              "      <td>-0.025264</td>\n",
              "      <td>-0.212352</td>\n",
              "      <td>0.108476</td>\n",
              "      <td>-0.112836</td>\n",
              "      <td>-0.233279</td>\n",
              "      <td>-0.201998</td>\n",
              "      <td>-0.278923</td>\n",
              "      <td>0.223151</td>\n",
              "      <td>0.017902</td>\n",
              "      <td>-0.011831</td>\n",
              "      <td>-0.354386</td>\n",
              "      <td>0.508833</td>\n",
              "      <td>-0.120905</td>\n",
              "      <td>-0.948406</td>\n",
              "      <td>0.591599</td>\n",
              "      <td>-0.442476</td>\n",
              "      <td>0.251363</td>\n",
              "      <td>0.397072</td>\n",
              "      <td>0.023746</td>\n",
              "      <td>-0.288343</td>\n",
              "      <td>0.420796</td>\n",
              "      <td>-0.082198</td>\n",
              "      <td>...</td>\n",
              "      <td>-0.095490</td>\n",
              "      <td>-0.420967</td>\n",
              "      <td>0.463086</td>\n",
              "      <td>0.393737</td>\n",
              "      <td>0.393154</td>\n",
              "      <td>-0.283677</td>\n",
              "      <td>-0.621931</td>\n",
              "      <td>-0.360244</td>\n",
              "      <td>-0.102992</td>\n",
              "      <td>0.031882</td>\n",
              "      <td>-0.106679</td>\n",
              "      <td>0.716718</td>\n",
              "      <td>0.197329</td>\n",
              "      <td>-0.261676</td>\n",
              "      <td>-0.326734</td>\n",
              "      <td>-0.202511</td>\n",
              "      <td>0.366927</td>\n",
              "      <td>-0.062942</td>\n",
              "      <td>0.176530</td>\n",
              "      <td>0.060247</td>\n",
              "      <td>-0.298104</td>\n",
              "      <td>-0.551207</td>\n",
              "      <td>0.373706</td>\n",
              "      <td>-0.042755</td>\n",
              "      <td>0.481701</td>\n",
              "      <td>0.260109</td>\n",
              "      <td>0.584887</td>\n",
              "      <td>0.289511</td>\n",
              "      <td>-0.167374</td>\n",
              "      <td>0.010033</td>\n",
              "      <td>0.462439</td>\n",
              "      <td>-0.076547</td>\n",
              "      <td>-0.282838</td>\n",
              "      <td>-0.036585</td>\n",
              "      <td>-0.392120</td>\n",
              "      <td>-0.279264</td>\n",
              "      <td>0.283381</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>40377</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 107 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "   CJT_TYP_2/FINANZ_SPARER  ...     LNR\n",
              "0                -1.234629  ...  780657\n",
              "1                 0.977389  ...  283232\n",
              "2                 0.640617  ...  728552\n",
              "3                -2.199834  ...  390673\n",
              "4                 1.247861  ...   40377\n",
              "\n",
              "[5 rows x 107 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 145
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cfzoZG_jotoD"
      },
      "source": [
        "Now we can examine which individuals are grouped together in the same cluster by looking at one of the clusters in more detail. If we take cluster 2 for example and look at the location of the centroid in the heatmap above, it tells us that it has the highest value for the attributes:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "wQ20lJYQcsIG",
        "outputId": "86db8004-5774-4e1e-d76b-1397ce8d2f5f"
      },
      "source": [
        "cluster_centroids.T[1].nlargest(1).index[0]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'CJT_TYP_2/FINANZ_SPARER'"
            ]
          },
          "metadata": {},
          "execution_count": 146
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lhd0_tg5dyUE"
      },
      "source": [
        "In addition, we can check the individuals that fit that description:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 372
        },
        "id": "hlXD8rCdo3ZL",
        "outputId": "cbb5c34e-91e9-46ec-c00b-7dc1dcb71035"
      },
      "source": [
        "cluster_2 = population_transformed[population_transformed['Clusters'] == 2]\n",
        "print(\"Individuals in cluster: \", cluster_2.shape[0])\n",
        "print(\"\\nIndividuals `Customers`: \", cluster_2[cluster_2['TARGET'] == 1].shape[0])\n",
        "print(\"\\nIndividuals `General`: \", cluster_2[cluster_2['TARGET'] == -1].shape[0])\n",
        "print(30*'-' + '\\n')\n",
        "cluster_2.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Individuals in cluster:  61050\n",
            "\n",
            "Individuals `Customers`:  29823\n",
            "\n",
            "Individuals `General`:  31227\n",
            "------------------------------\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>CJT_TYP_2/FINANZ_SPARER</th>\n",
              "      <th>FINANZ_ANLEGER/FINANZ_UNAUFFAELLIGER</th>\n",
              "      <th>OST_WEST_KZ_W/STRUKTURTYP</th>\n",
              "      <th>HH_EINKOMMEN_SCORE/OST_WEST_KZ_O</th>\n",
              "      <th>ANREDE_KZ/SEMIO_KAEM</th>\n",
              "      <th>KBA13_BAUMAX/LP_FAMILIE_FEIN</th>\n",
              "      <th>OST_WEST_KZ_O/KBA13_KW_61_120</th>\n",
              "      <th>PLZ8_ANTG4/KBA13_ANTG4</th>\n",
              "      <th>KBA13_CCM_3001/KBA13_CCM_2501</th>\n",
              "      <th>KBA05_MAXSEG/KBA05_MOTOR</th>\n",
              "      <th>KBA13_ANTG4/PLZ8_ANTG4</th>\n",
              "      <th>KBA05_MAXBJ/SHOPPER_TYP</th>\n",
              "      <th>VERS_TYP/HEALTH_TYP</th>\n",
              "      <th>KBA05_MOD8/KBA05_SEG9</th>\n",
              "      <th>LP_FAMILIE_FEIN/LP_LEBENSPHASE_FEIN</th>\n",
              "      <th>KBA05_KW3/KBA05_CCM4</th>\n",
              "      <th>KBA13_ALTERHALTER_45/KBA13_HALTER_35</th>\n",
              "      <th>KBA13_VORB_1_2/KBA13_ALTERHALTER_61</th>\n",
              "      <th>D19_GESAMT_ANZ_24/D19_VERSAND_ANZ_24</th>\n",
              "      <th>D19_GESAMT_ANZ_24/D19_VERSAND_ANZ_24</th>\n",
              "      <th>KBA13_KRSZUL_NEU/KBA05_MOD8</th>\n",
              "      <th>KBA05_ANHANG/KBA05_ANTG3</th>\n",
              "      <th>KBA05_ANHANG/KBA05_BAUMAX</th>\n",
              "      <th>SHOPPER_TYP/KBA05_ANTG1</th>\n",
              "      <th>SHOPPER_TYP/KBA13_KRSSEG_VAN</th>\n",
              "      <th>KBA05_ANHANG/KBA05_MOTRAD</th>\n",
              "      <th>KBA05_MOTRAD/BALLRAUM</th>\n",
              "      <th>VHN/D19_SOZIALES</th>\n",
              "      <th>PLZ8_ANTG4/KBA05_ANHANG</th>\n",
              "      <th>PLZ8_ANTG4/KBA05_ANTG3</th>\n",
              "      <th>SHOPPER_TYP/KBA13_ANTG4</th>\n",
              "      <th>PLZ8_ANTG4/KBA13_KRSSEG_VAN</th>\n",
              "      <th>D19_LETZTER_KAUF_BRANCHE_D19_UNBEKANNT/D19_KONSUMTYP</th>\n",
              "      <th>KBA05_ANHANG/KBA05_SEG7</th>\n",
              "      <th>KBA05_MOTRAD/KBA05_ANTG3</th>\n",
              "      <th>KBA05_MOTRAD/AGER_TYP</th>\n",
              "      <th>VHN/KBA05_MOTRAD</th>\n",
              "      <th>AGER_TYP/KBA05_ZUL2</th>\n",
              "      <th>KBA05_SEG8/KBA05_SEG7</th>\n",
              "      <th>KBA05_SEG7/AGER_TYP</th>\n",
              "      <th>...</th>\n",
              "      <th>D19_BEKLEIDUNG_REST/D19_BANKEN_DIREKT</th>\n",
              "      <th>KBA13_KW_70/KBA05_ANTG2</th>\n",
              "      <th>D19_BEKLEIDUNG_REST/D19_HAUS_DEKO</th>\n",
              "      <th>D19_VERSAND_REST/D19_BEKLEIDUNG_REST</th>\n",
              "      <th>KBA05_MOD1/KBA05_SEG10</th>\n",
              "      <th>KK_KUNDENTYP/D19_VERSAND_ANZ_24</th>\n",
              "      <th>KBA05_ANTG2/D19_VOLLSORTIMENT</th>\n",
              "      <th>KBA05_SEG9/D19_TECHNIK</th>\n",
              "      <th>KBA05_ALTER2/D19_VERSAND_ANZ_12</th>\n",
              "      <th>D19_SONSTIGE/D19_VERSAND_ANZ_12</th>\n",
              "      <th>KBA05_SEG9/KBA05_ALTER3</th>\n",
              "      <th>D19_VOLLSORTIMENT/KBA05_HERST5</th>\n",
              "      <th>KBA05_MOD8/D19_REISEN</th>\n",
              "      <th>D19_BUCH_CD/KBA05_KW3</th>\n",
              "      <th>D19_BUCH_CD/KBA05_MOD8</th>\n",
              "      <th>D19_BUCH_CD/KBA05_KRSVAN</th>\n",
              "      <th>D19_VOLLSORTIMENT/D19_GESAMT_ONLINE_QUOTE_12</th>\n",
              "      <th>RELAT_AB/KBA05_KW3</th>\n",
              "      <th>D19_SONSTIGE/EXTSEL992</th>\n",
              "      <th>KBA05_CCM4/D19_SONSTIGE</th>\n",
              "      <th>KBA05_DIESEL/UMFELD_JUNG</th>\n",
              "      <th>KBA05_DIESEL/RT_UEBERGROESSE</th>\n",
              "      <th>KBA13_VORB_2/KBA13_VORB_1_2</th>\n",
              "      <th>GFK_URLAUBERTYP/KBA05_KRSVAN</th>\n",
              "      <th>D19_SONSTIGE/D19_VERSAND_ANZ_24</th>\n",
              "      <th>KBA05_SEG5/KBA13_SEG_OBERKLASSE</th>\n",
              "      <th>KBA05_BAUMAX/UMFELD_ALT</th>\n",
              "      <th>KBA13_KW_90/KBA13_CCM_3001</th>\n",
              "      <th>KBA05_DIESEL/KBA05_HERST4</th>\n",
              "      <th>KBA13_SEG_OBERKLASSE/KBA13_KRSZUL_NEU</th>\n",
              "      <th>KBA05_MOD4/KBA05_SEG5</th>\n",
              "      <th>KBA13_BJ_2008/RETOURTYP_BK_S</th>\n",
              "      <th>KBA05_VORB2/EXTSEL992</th>\n",
              "      <th>KBA05_HERST4/KBA05_FRAU</th>\n",
              "      <th>D19_VERSAND_ONLINE_QUOTE_12/KBA05_KRSVAN</th>\n",
              "      <th>GFK_URLAUBERTYP/D19_VERSAND_ONLINE_QUOTE_12</th>\n",
              "      <th>KK_KUNDENTYP/D19_GESAMT_ANZ_12</th>\n",
              "      <th>Clusters</th>\n",
              "      <th>TARGET</th>\n",
              "      <th>LNR</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>-2.640672</td>\n",
              "      <td>0.843719</td>\n",
              "      <td>0.102702</td>\n",
              "      <td>-0.075584</td>\n",
              "      <td>-0.362793</td>\n",
              "      <td>0.264781</td>\n",
              "      <td>-0.223328</td>\n",
              "      <td>-0.506855</td>\n",
              "      <td>0.951140</td>\n",
              "      <td>-0.468598</td>\n",
              "      <td>0.486646</td>\n",
              "      <td>0.660295</td>\n",
              "      <td>-0.003846</td>\n",
              "      <td>0.296077</td>\n",
              "      <td>-0.272760</td>\n",
              "      <td>0.113319</td>\n",
              "      <td>0.108795</td>\n",
              "      <td>0.099468</td>\n",
              "      <td>0.122817</td>\n",
              "      <td>0.083201</td>\n",
              "      <td>1.247350</td>\n",
              "      <td>-0.455726</td>\n",
              "      <td>-0.571825</td>\n",
              "      <td>-0.067285</td>\n",
              "      <td>0.560072</td>\n",
              "      <td>-0.089024</td>\n",
              "      <td>0.193925</td>\n",
              "      <td>-0.005272</td>\n",
              "      <td>0.283273</td>\n",
              "      <td>0.301201</td>\n",
              "      <td>-0.373616</td>\n",
              "      <td>0.631035</td>\n",
              "      <td>-0.102140</td>\n",
              "      <td>-0.161052</td>\n",
              "      <td>0.052280</td>\n",
              "      <td>0.024935</td>\n",
              "      <td>-0.390521</td>\n",
              "      <td>-0.242384</td>\n",
              "      <td>-0.069935</td>\n",
              "      <td>-0.120373</td>\n",
              "      <td>...</td>\n",
              "      <td>-0.093179</td>\n",
              "      <td>-0.266858</td>\n",
              "      <td>0.061900</td>\n",
              "      <td>-0.121609</td>\n",
              "      <td>-0.099128</td>\n",
              "      <td>1.126712</td>\n",
              "      <td>0.046679</td>\n",
              "      <td>-0.203011</td>\n",
              "      <td>-0.181079</td>\n",
              "      <td>0.299730</td>\n",
              "      <td>-0.202914</td>\n",
              "      <td>-0.123354</td>\n",
              "      <td>-0.024537</td>\n",
              "      <td>-0.002241</td>\n",
              "      <td>-0.569587</td>\n",
              "      <td>0.333229</td>\n",
              "      <td>0.299639</td>\n",
              "      <td>0.236294</td>\n",
              "      <td>0.160245</td>\n",
              "      <td>0.115646</td>\n",
              "      <td>0.066705</td>\n",
              "      <td>-0.318140</td>\n",
              "      <td>-0.009263</td>\n",
              "      <td>-0.165102</td>\n",
              "      <td>-0.293401</td>\n",
              "      <td>0.003036</td>\n",
              "      <td>0.051617</td>\n",
              "      <td>-0.015282</td>\n",
              "      <td>0.585075</td>\n",
              "      <td>0.076598</td>\n",
              "      <td>-0.012210</td>\n",
              "      <td>-0.207770</td>\n",
              "      <td>0.055532</td>\n",
              "      <td>0.163585</td>\n",
              "      <td>0.575585</td>\n",
              "      <td>-0.187314</td>\n",
              "      <td>-0.558813</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>24700</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>-2.063223</td>\n",
              "      <td>0.400269</td>\n",
              "      <td>0.857781</td>\n",
              "      <td>0.483734</td>\n",
              "      <td>-0.047520</td>\n",
              "      <td>0.626514</td>\n",
              "      <td>-0.435673</td>\n",
              "      <td>0.023308</td>\n",
              "      <td>0.416930</td>\n",
              "      <td>-0.389410</td>\n",
              "      <td>-0.945167</td>\n",
              "      <td>-0.911041</td>\n",
              "      <td>0.583819</td>\n",
              "      <td>1.262275</td>\n",
              "      <td>-0.467349</td>\n",
              "      <td>0.022939</td>\n",
              "      <td>-0.079326</td>\n",
              "      <td>0.396674</td>\n",
              "      <td>0.914962</td>\n",
              "      <td>0.051674</td>\n",
              "      <td>0.387914</td>\n",
              "      <td>0.044065</td>\n",
              "      <td>-0.259569</td>\n",
              "      <td>-0.227781</td>\n",
              "      <td>-0.473022</td>\n",
              "      <td>0.260998</td>\n",
              "      <td>-0.377862</td>\n",
              "      <td>0.094678</td>\n",
              "      <td>0.330117</td>\n",
              "      <td>-0.230632</td>\n",
              "      <td>-0.042768</td>\n",
              "      <td>-0.003732</td>\n",
              "      <td>-0.358954</td>\n",
              "      <td>0.082959</td>\n",
              "      <td>0.452334</td>\n",
              "      <td>-0.659340</td>\n",
              "      <td>-0.557442</td>\n",
              "      <td>0.425277</td>\n",
              "      <td>-0.677737</td>\n",
              "      <td>0.100660</td>\n",
              "      <td>...</td>\n",
              "      <td>0.190761</td>\n",
              "      <td>0.065015</td>\n",
              "      <td>0.571585</td>\n",
              "      <td>0.325712</td>\n",
              "      <td>-0.128486</td>\n",
              "      <td>-0.151182</td>\n",
              "      <td>0.555767</td>\n",
              "      <td>-0.171523</td>\n",
              "      <td>-0.063445</td>\n",
              "      <td>-0.067942</td>\n",
              "      <td>-0.042285</td>\n",
              "      <td>0.232594</td>\n",
              "      <td>-0.314477</td>\n",
              "      <td>-0.153548</td>\n",
              "      <td>0.298703</td>\n",
              "      <td>0.239172</td>\n",
              "      <td>-0.167129</td>\n",
              "      <td>-0.444540</td>\n",
              "      <td>0.722695</td>\n",
              "      <td>0.113475</td>\n",
              "      <td>0.369564</td>\n",
              "      <td>-0.260474</td>\n",
              "      <td>-0.217681</td>\n",
              "      <td>0.462359</td>\n",
              "      <td>-0.165925</td>\n",
              "      <td>-0.070592</td>\n",
              "      <td>-0.045608</td>\n",
              "      <td>0.159999</td>\n",
              "      <td>-0.279164</td>\n",
              "      <td>-0.239771</td>\n",
              "      <td>-0.236415</td>\n",
              "      <td>0.097393</td>\n",
              "      <td>0.292148</td>\n",
              "      <td>-0.015301</td>\n",
              "      <td>-0.459012</td>\n",
              "      <td>0.149128</td>\n",
              "      <td>-0.046471</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>164256</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>-1.372159</td>\n",
              "      <td>1.166249</td>\n",
              "      <td>-1.043671</td>\n",
              "      <td>0.247380</td>\n",
              "      <td>0.794591</td>\n",
              "      <td>-0.903142</td>\n",
              "      <td>-1.727027</td>\n",
              "      <td>1.351772</td>\n",
              "      <td>-0.244045</td>\n",
              "      <td>0.745800</td>\n",
              "      <td>0.651984</td>\n",
              "      <td>-0.782614</td>\n",
              "      <td>0.237077</td>\n",
              "      <td>-0.593261</td>\n",
              "      <td>-0.004930</td>\n",
              "      <td>-0.168611</td>\n",
              "      <td>0.727278</td>\n",
              "      <td>-0.001004</td>\n",
              "      <td>0.176355</td>\n",
              "      <td>-0.105044</td>\n",
              "      <td>-0.082041</td>\n",
              "      <td>1.114180</td>\n",
              "      <td>0.089635</td>\n",
              "      <td>-0.294440</td>\n",
              "      <td>-0.417220</td>\n",
              "      <td>0.283097</td>\n",
              "      <td>-0.283342</td>\n",
              "      <td>0.296308</td>\n",
              "      <td>0.666146</td>\n",
              "      <td>0.279036</td>\n",
              "      <td>-0.544161</td>\n",
              "      <td>-0.391942</td>\n",
              "      <td>-0.175988</td>\n",
              "      <td>-0.031520</td>\n",
              "      <td>0.484813</td>\n",
              "      <td>-0.143667</td>\n",
              "      <td>-0.349746</td>\n",
              "      <td>-0.503226</td>\n",
              "      <td>0.380985</td>\n",
              "      <td>0.011175</td>\n",
              "      <td>...</td>\n",
              "      <td>-0.119336</td>\n",
              "      <td>0.193120</td>\n",
              "      <td>0.052614</td>\n",
              "      <td>0.179707</td>\n",
              "      <td>-0.122782</td>\n",
              "      <td>0.123304</td>\n",
              "      <td>-0.334469</td>\n",
              "      <td>-0.284484</td>\n",
              "      <td>0.046115</td>\n",
              "      <td>0.357731</td>\n",
              "      <td>0.452035</td>\n",
              "      <td>-0.228360</td>\n",
              "      <td>-0.174317</td>\n",
              "      <td>-0.338441</td>\n",
              "      <td>0.085373</td>\n",
              "      <td>0.011598</td>\n",
              "      <td>-0.016361</td>\n",
              "      <td>0.025277</td>\n",
              "      <td>0.176017</td>\n",
              "      <td>0.400706</td>\n",
              "      <td>0.008046</td>\n",
              "      <td>-0.097396</td>\n",
              "      <td>-0.043868</td>\n",
              "      <td>0.388968</td>\n",
              "      <td>-0.325925</td>\n",
              "      <td>0.125511</td>\n",
              "      <td>0.470923</td>\n",
              "      <td>-0.150063</td>\n",
              "      <td>0.295416</td>\n",
              "      <td>-0.002793</td>\n",
              "      <td>0.061119</td>\n",
              "      <td>-0.483951</td>\n",
              "      <td>0.047001</td>\n",
              "      <td>-0.281829</td>\n",
              "      <td>-0.218722</td>\n",
              "      <td>-0.218273</td>\n",
              "      <td>0.061485</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>36904</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>-0.622610</td>\n",
              "      <td>-0.065053</td>\n",
              "      <td>0.821297</td>\n",
              "      <td>1.121812</td>\n",
              "      <td>-0.472048</td>\n",
              "      <td>0.010100</td>\n",
              "      <td>0.405544</td>\n",
              "      <td>-0.236830</td>\n",
              "      <td>0.075706</td>\n",
              "      <td>0.339974</td>\n",
              "      <td>-0.467588</td>\n",
              "      <td>0.036995</td>\n",
              "      <td>0.885449</td>\n",
              "      <td>-0.770944</td>\n",
              "      <td>0.195356</td>\n",
              "      <td>-0.290143</td>\n",
              "      <td>0.678201</td>\n",
              "      <td>-0.167262</td>\n",
              "      <td>-0.299035</td>\n",
              "      <td>0.604685</td>\n",
              "      <td>-0.019437</td>\n",
              "      <td>-0.188901</td>\n",
              "      <td>0.202551</td>\n",
              "      <td>-0.414020</td>\n",
              "      <td>-0.133697</td>\n",
              "      <td>-0.114712</td>\n",
              "      <td>0.191796</td>\n",
              "      <td>0.730658</td>\n",
              "      <td>-0.056620</td>\n",
              "      <td>0.068252</td>\n",
              "      <td>0.239848</td>\n",
              "      <td>-0.122530</td>\n",
              "      <td>0.428666</td>\n",
              "      <td>0.314938</td>\n",
              "      <td>-0.221472</td>\n",
              "      <td>0.098372</td>\n",
              "      <td>0.113208</td>\n",
              "      <td>0.329631</td>\n",
              "      <td>0.539758</td>\n",
              "      <td>-0.337404</td>\n",
              "      <td>...</td>\n",
              "      <td>0.100177</td>\n",
              "      <td>-0.326417</td>\n",
              "      <td>0.031772</td>\n",
              "      <td>-0.357952</td>\n",
              "      <td>0.420073</td>\n",
              "      <td>-0.266167</td>\n",
              "      <td>0.293056</td>\n",
              "      <td>0.326806</td>\n",
              "      <td>0.416744</td>\n",
              "      <td>-0.253868</td>\n",
              "      <td>0.345884</td>\n",
              "      <td>-0.303418</td>\n",
              "      <td>-0.182663</td>\n",
              "      <td>-0.047127</td>\n",
              "      <td>0.341984</td>\n",
              "      <td>-0.139045</td>\n",
              "      <td>-0.078901</td>\n",
              "      <td>-0.203956</td>\n",
              "      <td>-0.128059</td>\n",
              "      <td>-0.360531</td>\n",
              "      <td>0.050204</td>\n",
              "      <td>0.229974</td>\n",
              "      <td>-0.049812</td>\n",
              "      <td>0.199613</td>\n",
              "      <td>-0.257879</td>\n",
              "      <td>0.103803</td>\n",
              "      <td>-0.479614</td>\n",
              "      <td>0.107008</td>\n",
              "      <td>-0.119015</td>\n",
              "      <td>-0.006012</td>\n",
              "      <td>-0.046189</td>\n",
              "      <td>0.651832</td>\n",
              "      <td>0.154374</td>\n",
              "      <td>0.260852</td>\n",
              "      <td>-0.224118</td>\n",
              "      <td>0.267607</td>\n",
              "      <td>0.197021</td>\n",
              "      <td>2</td>\n",
              "      <td>-1</td>\n",
              "      <td>934184</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>35</th>\n",
              "      <td>-0.855624</td>\n",
              "      <td>3.098685</td>\n",
              "      <td>0.250012</td>\n",
              "      <td>0.230511</td>\n",
              "      <td>1.059720</td>\n",
              "      <td>-0.627812</td>\n",
              "      <td>-0.654421</td>\n",
              "      <td>0.456267</td>\n",
              "      <td>0.817005</td>\n",
              "      <td>-0.054278</td>\n",
              "      <td>0.489636</td>\n",
              "      <td>0.324316</td>\n",
              "      <td>-0.091211</td>\n",
              "      <td>-0.379414</td>\n",
              "      <td>0.160945</td>\n",
              "      <td>-0.880575</td>\n",
              "      <td>-0.171045</td>\n",
              "      <td>-0.484757</td>\n",
              "      <td>0.256562</td>\n",
              "      <td>1.462890</td>\n",
              "      <td>0.623760</td>\n",
              "      <td>0.816858</td>\n",
              "      <td>-0.089464</td>\n",
              "      <td>0.011598</td>\n",
              "      <td>0.301120</td>\n",
              "      <td>-0.383985</td>\n",
              "      <td>-0.613529</td>\n",
              "      <td>-0.452742</td>\n",
              "      <td>0.276201</td>\n",
              "      <td>-0.797456</td>\n",
              "      <td>0.020515</td>\n",
              "      <td>-0.274112</td>\n",
              "      <td>-0.259049</td>\n",
              "      <td>0.308186</td>\n",
              "      <td>-0.167700</td>\n",
              "      <td>0.536849</td>\n",
              "      <td>-0.091647</td>\n",
              "      <td>-0.242193</td>\n",
              "      <td>0.094677</td>\n",
              "      <td>0.090738</td>\n",
              "      <td>...</td>\n",
              "      <td>-0.127433</td>\n",
              "      <td>0.157835</td>\n",
              "      <td>-0.164282</td>\n",
              "      <td>0.369541</td>\n",
              "      <td>-0.160702</td>\n",
              "      <td>0.188433</td>\n",
              "      <td>0.781503</td>\n",
              "      <td>-0.340645</td>\n",
              "      <td>0.103674</td>\n",
              "      <td>-0.493021</td>\n",
              "      <td>0.704301</td>\n",
              "      <td>-0.193590</td>\n",
              "      <td>0.528758</td>\n",
              "      <td>0.176995</td>\n",
              "      <td>0.220275</td>\n",
              "      <td>-0.037410</td>\n",
              "      <td>0.473194</td>\n",
              "      <td>-0.042443</td>\n",
              "      <td>-0.218568</td>\n",
              "      <td>-0.033859</td>\n",
              "      <td>-0.108060</td>\n",
              "      <td>0.525622</td>\n",
              "      <td>0.196588</td>\n",
              "      <td>0.203919</td>\n",
              "      <td>-0.106998</td>\n",
              "      <td>-0.021664</td>\n",
              "      <td>-0.357374</td>\n",
              "      <td>-0.202134</td>\n",
              "      <td>-0.174691</td>\n",
              "      <td>0.514296</td>\n",
              "      <td>0.169565</td>\n",
              "      <td>0.182207</td>\n",
              "      <td>0.051376</td>\n",
              "      <td>0.251897</td>\n",
              "      <td>-0.302851</td>\n",
              "      <td>-0.024586</td>\n",
              "      <td>0.274944</td>\n",
              "      <td>2</td>\n",
              "      <td>-1</td>\n",
              "      <td>843518</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 107 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "    CJT_TYP_2/FINANZ_SPARER  ...     LNR\n",
              "10                -2.640672  ...   24700\n",
              "11                -2.063223  ...  164256\n",
              "18                -1.372159  ...   36904\n",
              "22                -0.622610  ...  934184\n",
              "35                -0.855624  ...  843518\n",
              "\n",
              "[5 rows x 107 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 147
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dcy4xFROSBg7"
      },
      "source": [
        "We have created a list of the two most influencing attributes characterising each principal component. We will create a dictionary mapping of these two attributes for each PC, extracting more available information from the excel files provided for the attributes' explanation. This will give us more information on the dominant context of each PC."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nGqK3ZZqTe-Y",
        "outputId": "77aa31d1-5245-4a60-e86b-2b14db80e33b"
      },
      "source": [
        "# For each name we created for PCs we check if the attribute exists in the excel file and extract descriptions\n",
        "pca_list_map = {pc: ', '.join([dfdias_attrib[dfdias_attrib[\"Attribute\"].isin([i, i+'_RZ'])][\"Description\"].iloc[0] \\\n",
        "                               if dfdias_attrib[\"Attribute\"].isin([i, i+'_RZ']).any() else 'UNKNOWN' for i in pc.split('/')]) \\\n",
        "                for pc in pca_list}\n",
        "\n",
        "list(pca_list_map.items())[:3]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('CJT_TYP_2/FINANZ_SPARER', 'UNKNOWN, financial typology: money saver'),\n",
              " ('FINANZ_ANLEGER/FINANZ_UNAUFFAELLIGER',\n",
              "  'financial typology: investor, financial typology: unremarkable'),\n",
              " ('OST_WEST_KZ_W/STRUKTURTYP', 'UNKNOWN, UNKNOWN')]"
            ]
          },
          "metadata": {},
          "execution_count": 148
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O1KuyXiO9A7t"
      },
      "source": [
        "The above mapping allows us to further look into the characteristics of a cluster. For example for cluster 2 we can look for the location of the highest (most dominant) centroid coordinates in the heatmap above. This tells us that it has the highest value in some principal components. We can extract these names and look at the dictionary we created for more info."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MsWa_LzV-xxl",
        "outputId": "20676716-f876-41b7-e1cc-be08cf3e6c52"
      },
      "source": [
        "# Cluster 2\n",
        "cluster2_top5_pc = cluster_centroids.T[1].sort_values(ascending=False).head().index.to_list()\n",
        "\n",
        "for pc in cluster2_top5_pc:\n",
        "    print(\"[Attributes]- {},\\t[Description]- {}\".format(pc, pca_list_map[pc]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[Attributes]- CJT_TYP_2/FINANZ_SPARER,\t[Descritpion]- UNKNOWN, financial typology: money saver\n",
            "[Attributes]- KBA13_BAUMAX/LP_FAMILIE_FEIN,\t[Descritpion]- UNKNOWN, familytyp fine\n",
            "[Attributes]- FINANZ_ANLEGER/FINANZ_UNAUFFAELLIGER,\t[Descritpion]- financial typology: investor, financial typology: unremarkable\n",
            "[Attributes]- ANREDE_KZ/SEMIO_KAEM,\t[Descritpion]- gender, affinity indicating in what way the person is of a fightfull attitude\n",
            "[Attributes]- KBA05_BAUMAX/KBA05_ANTG3,\t[Descritpion]- most common building-type within the cell, number of 6-10 family houses in the cell\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kl0NuTSNNtyU"
      },
      "source": [
        "From the above, on first look we can extract characteristics of **Cluster 2**  group such as:\n",
        "> *Individuals that are money savers and investors, are in a good financial status, living in the suburbs, or sparse areas*.\n",
        "\n",
        "If we further investigate the unknown attributes we find:\n",
        "\n",
        "* `CJT_TYP_2`: relating to the preferred information and buying channels of consumers\n",
        "* `KBA13_BAUMAX`: is the estimated share of >10 family houses in the PLZ8.\n",
        "\n",
        "With this we can add to the above cluster description:\n",
        "\n",
        "> *Individuals that are money savers and investors, are in a good financial status, living in the suburbs, or sparse areas …in common houses with buying preference channel 2 (Advertising- and Consumptiontraditionalist)*.\n",
        "\n",
        "Similarly for **Cluster 1**:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zTm-qTbfUsTk",
        "outputId": "56723e14-8b92-4e2c-81dd-619ce8feaf54"
      },
      "source": [
        "# Cluster 1\n",
        "cluster1_top5_pc = cluster_centroids.T[0].sort_values(ascending=False).head().index.to_list()\n",
        "\n",
        "for pc in cluster1_top5_pc:\n",
        "    print(\"[Attributes]- {},\\t[Description]- {}\".format(pc, pca_list_map[pc]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[Attributes]- KBA13_ANTG4/PLZ8_ANTG4,\t[Descritpion]- UNKNOWN, number of >10 family houses in the PLZ8\n",
            "[Attributes]- KBA05_MAXBJ/SHOPPER_TYP,\t[Descritpion]- most common age of the cars in the microcell, shopping typology\n",
            "[Attributes]- KBA05_MOD8/KBA05_SEG9,\t[Descritpion]- share of vans (in an AZ specific definition), share of vans in the microcell\n",
            "[Attributes]- PLZ8_ANTG4/KBA13_ANTG4,\t[Descritpion]- number of >10 family houses in the PLZ8, UNKNOWN\n",
            "[Attributes]- GEBAEUDETYP/VHN,\t[Descritpion]- type of building (residential or commercial), UNKNOWN\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fmPu7P6NVF1N"
      },
      "source": [
        "> *Individuals that live in not very dense areas, in residential or commercial buildings, with most common cars and vans*.\n",
        "\n",
        "We may also drill down to specifics of each attribute:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uHahSVdahnmk"
      },
      "source": [
        "train = train.assign(LNR=clientIDs, TARGET=target)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lOnW_6isfKFO",
        "outputId": "f33746d4-ce33-432d-a876-070fecb6aacf"
      },
      "source": [
        "# E.g. FINANZ_ANLEGER\n",
        "# Check the population in Cluster 1 for PC: CJT_TYP_2/FINANZ_ANLEGER\n",
        "cluster_1 = population_transformed[population_transformed['Clusters'] == 1]['LNR']\n",
        "# Then filter the original data by these ids and get percentages:\n",
        "train[train['LNR'].isin(cluster_1)]['FINANZ_ANLEGER'].value_counts(sort=False, normalize=True).mul(100).round(1)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1     0.0\n",
              "2     0.2\n",
              "3     0.4\n",
              "4     0.9\n",
              "5    98.5\n",
              "Name: FINANZ_ANLEGER, dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 152
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HORoZqJ-jjFy"
      },
      "source": [
        "So, in this cluster we have these kinds of `investor` in general. We could also examine subsets based on general population and customers"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dN_LqEcVmEVI",
        "outputId": "b5c5503d-24e0-43d8-ba47-aa7db2d1ac68"
      },
      "source": [
        "group_cl1 = train[train['LNR'].isin(cluster_1)].groupby('TARGET')\n",
        "# General population dataset\n",
        "group_cl1.get_group((list(group_cl1.groups)[0]))['FINANZ_ANLEGER'].value_counts(sort=False, normalize=True).mul(100).round(1)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1     0.1\n",
              "2     0.3\n",
              "3     0.9\n",
              "4     2.0\n",
              "5    96.7\n",
              "Name: FINANZ_ANLEGER, dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 153
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O1gfXf9fqBvy"
      },
      "source": [
        "For the **general population** the percentages of `investor` in cluster 1 are as above."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fqv7EZbBo-kU",
        "outputId": "ca069902-bf7d-43dd-f16e-d88dc8c46c7d"
      },
      "source": [
        "# Customers dataset\n",
        "group_cl1.get_group((list(group_cl1.groups)[1]))['FINANZ_ANLEGER'].value_counts(sort=False, normalize=True).mul(100).round(1)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1     0.0\n",
              "2     0.0\n",
              "3     0.1\n",
              "4     0.1\n",
              "5    99.8\n",
              "Name: FINANZ_ANLEGER, dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 154
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qqsnXFWnqPJT"
      },
      "source": [
        "For the **customers population** the percentages of `investor` in cluster 1 are as above."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JHMnYFTexuNd"
      },
      "source": [
        "### Segmentation conclusion\n",
        "\n",
        "The parts of the general population that may be considered more likely to be part of the mail-order company's main customer base are the ones in clusters:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 111
        },
        "id": "bXLBDu8QzDtL",
        "outputId": "93394d65-47d0-49a3-9c4c-ec78110c2b52"
      },
      "source": [
        "top_customer_clusters = target_group.loc[(target_group[\"TARGET\"] == 1), [\"Clusters\", \"Perc\"]].nlargest(2, \"Perc\").reset_index(drop=True)\n",
        "top_customer_clusters"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Clusters</th>\n",
              "      <th>Perc</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>24.42</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>5</td>\n",
              "      <td>22.08</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Clusters   Perc\n",
              "0         1  24.42\n",
              "1         5  22.08"
            ]
          },
          "metadata": {},
          "execution_count": 155
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nD6mMe2Y3qjH"
      },
      "source": [
        "These are the clusters with the highest customers population concentration. We may consider that these clusters in the corresponding general population will provide the highest chances for acquiring new customers. If we like to have a greater level of detail, we could transform the data to fit in more clusters. We could then target more specific segments of the general population."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "I1jnRlKxXOLX",
        "outputId": "6e782eaf-eec9-4de5-b322-b52211fa0fb4"
      },
      "source": [
        "print(\"Elapsed time: \", str(timedelta(seconds=(timer() - start_run))))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Elapsed time:  0:06:39.358728\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CPlq8v35SkpJ",
        "outputId": "2bb9dc67-0059-42e1-8f23-0b79f76cbbe9"
      },
      "source": [
        "del train, population_transformed, cluster_2, cluster_1, cluster_centroids, pcadf, pca_df, train_vals, \\\n",
        "clusterer, all_Kscores,plt_attrs_wgts, train_reduced, pca_wghts_df, norm_train, pca, azdias, custrs\n",
        "gc.collect()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "28066"
            ]
          },
          "metadata": {},
          "execution_count": 157
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "38O2YOgGbUG5"
      },
      "source": [
        "## Part 2: Supervised Learning Model\n",
        "\n",
        "Now that you've found which parts of the population are more likely to be customers of the mail-order company, it's time to build a prediction model. Each of the rows in the \"MAILOUT\" data files represents an individual that was targeted for a mailout campaign. Ideally, we should be able to use the demographic information from each individual to decide whether or not it will be worth it to include that person in the campaign.\n",
        "\n",
        "The \"MAILOUT\" data has been split into two approximately equal parts, each with almost 43 000 data rows. In this part, you can verify your model with the \"TRAIN\" partition, which includes a column, \"RESPONSE\", that states whether or not a person became a customer of the company following the campaign. In the next part, you'll need to create predictions on the \"TEST\" partition, where the \"RESPONSE\" column has been withheld."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pMAzv8ftZANv"
      },
      "source": [
        "###Feature selection using [**LightGBM**](https://lightgbm.readthedocs.io/en/latest/index.html), a gradient boosting framework that uses tree-based learning algorithms.\n",
        "\n",
        "We will use **feature importances** of a LightGBM model with a threshold value to remove attributes not considered important by the model. We will achieve this by removing all zero importance features (as computed by LightGBM) from the model.\n",
        "\n",
        "The evaluation metric we will use is ROC_AUC_SCORE:\n",
        "\n",
        "An ROC curve (receiver operating characteristic curve) is a graph showing the performance of a classification model at all classification thresholds. This curve plots two parameters:\n",
        "\n",
        "* True Positive Rate\n",
        "* False Positive Rate\n",
        "\n",
        "AUC measures the entire two-dimensional area underneath the entire ROC curve  from (0,0) to (1,1). AUC provides an aggregate measure of performance across all possible classification thresholds.\n",
        "\n",
        "* AUC is scale-invariant. It measures how well predictions are ranked, rather than their absolute values.\n",
        "* AUC is classification-threshold-invariant. It measures the quality of the model's predictions irrespective of what classification threshold is chosen.\n",
        "\n",
        "We will use [LGBMClassifier](https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMClassifier.html), a Gradient Boosted Model from the LightGBM library to assess feature importances."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hXtxtxpXPtHZ"
      },
      "source": [
        "LGB plotting and interpretation:\n",
        "* https://stackoverflow.com/questions/61862597/how-to-get-variance-explained-of-features-for-lightgbm-best_iter\n",
        "* https://stackoverflow.com/questions/53413701/feature-importance-using-lightgbm\n",
        "\n",
        "KFOLDS\n",
        "\n",
        "* https://scikit-learn.org/stable/modules/cross_validation.html#k-fold"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7tEsTrk2sQfW"
      },
      "source": [
        "# Add the `LNR` column.\n",
        "df_train = df_train.assign(LNR=clientIDs.values)\n",
        "df_train.to_csv(f'{colab_dir if IN_COLAB else remote_dir}df_train_{ts}.csv', index=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hbv9s_G6s_sz"
      },
      "source": [
        "# df_train = pd.read_csv(f'{colab_dir if IN_COLAB else remote_dir}' + 'df_train_<add ts>.csv')\n",
        "# df_train.shape"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aUXgIBU9zJCF",
        "outputId": "86427749-b777-4e5c-cd63-68541510303c"
      },
      "source": [
        "# Create train and target sets.\n",
        "train = df_train.drop(columns=[\"TARGET\"])\n",
        "target = df_train[\"TARGET\"]\n",
        "target[target == -1] = 0\n",
        "\n",
        "# Add the `LNR` column.\n",
        "train = train.assign(LNR=clientIDs.values)\n",
        "\n",
        "train.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(450000, 377)"
            ]
          },
          "metadata": {},
          "execution_count": 159
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jgkdFmhNSU4N"
      },
      "source": [
        "So, from feature selection using gradient boosting model we may keep:\n",
        "\n",
        "* the list that contains all attributes identified with zero importance.\n",
        "* a list that contains a smaller subset of non zero importance attributes reaching 0.95 cumulative importance.\n",
        "\n",
        "We may test the above to other sample subsets. If it turns out that the feature selection works well we can also use the model above for making predictions in later steps.\n",
        "\n",
        "For this we will create a couple more functions to help us through the steps."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iVQT-uvWeD4F"
      },
      "source": [
        "from sklearn.model_selection import train_test_split, KFold\n",
        "from sklearn.metrics import roc_auc_score, auc, RocCurveDisplay\n",
        "from sklearn.ensemble import RandomForestClassifier"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7WFqpOOe3X9d"
      },
      "source": [
        "def rf_model(features_labels_split, rf_params, n_folds=3):\n",
        "    \"\"\"Train and test a light gradient boosting model using\n",
        "    cross validation with KFold splits. \n",
        "    \n",
        "    Args:\n",
        "        features_labels_split (list): List containing train-test split of inputs.\n",
        "            Either [X_train, X_test, y_train, y_test] or [features, test_features]\n",
        "            features will be pd.DataFrame of training features to use for training\n",
        "            a model. Must include the `RESPONSE` column.\n",
        "            test_features will be pd.DataFrame of testing features to use for\n",
        "            making predictions with the model.\n",
        "        lgbm_params (dict): LightGBM classifier init parameters.\n",
        "        n_folds (int, default = 5): number of folds to use for cross validation.\n",
        "        \n",
        "    Returns:\n",
        "        submission (pd.DataFrame): dataframe with `LNR` and `RESPONSE`\n",
        "            probabilities predicted by the model.\n",
        "        attrib_imp (pd.DataFrame): dataframe with the feature\n",
        "            importances from the model.\n",
        "        valid_metrics (pd.DataFrame): dataframe with training and validation\n",
        "            metrics (ROC AUC) for each fold and overall.\n",
        "    \n",
        "    Ref: https://www.kaggle.com/willkoehrsen/introduction-to-feature-selection#Remove-Collinear-Variables\n",
        "    \"\"\"\n",
        "    rf_ts = datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n",
        "    # Differentiate assignments according to input (length=4 -> train_test_split)\n",
        "    if len(features_labels_split) == 4:\n",
        "        features, test_features, labels, test_labels = features_labels_split\n",
        "    else:\n",
        "        features, test_features = features_labels_split\n",
        "        # Extract the labels for training\n",
        "        labels = features['RESPONSE']\n",
        "        test_labels = None\n",
        "\n",
        "    # Extract the client IDs\n",
        "    train_ids = features['LNR']\n",
        "    test_ids = test_features['LNR']\n",
        "    # Drop the client IDs and datetimes\n",
        "    features.drop(columns=['LNR', 'EINGEFUEGT_AM', 'RESPONSE'], errors='ignore', inplace=True)\n",
        "    test_features.drop(columns=['LNR', 'EINGEFUEGT_AM'], errors='ignore', inplace=True)\n",
        "        \n",
        "    print('\\nTraining Data Shape: ', features.shape)\n",
        "    print('Testing Data Shape: ', test_features.shape)\n",
        "    \n",
        "    # Extract feature names\n",
        "    feature_names = features.columns.tolist()\n",
        "    \n",
        "    # Impute missing values with Random Selection\n",
        "    traintestX = [i.apply(imput_nan_rand) for i in (features, test_features)]\n",
        "    # Normalization\n",
        "    features, test_features = [scaler.fit_transform(i) for i in traintestX]\n",
        "\n",
        "    # Empty array for feature importances\n",
        "    feature_importance_values = np.zeros(len(feature_names))\n",
        "    # Empty array for test predictions\n",
        "    test_predictions = np.zeros(test_features.shape[0])\n",
        "    # Empty array for out of fold validation predictions\n",
        "    out_of_fold = np.zeros(features.shape[0])\n",
        "    \n",
        "    # Create the kfold object\n",
        "    print(\"Creating KFold object\")\n",
        "    folds = KFold(n_splits=n_folds)\n",
        "    \n",
        "    # Lists for recording validation and training scores\n",
        "    train_scores = []\n",
        "    test_scores = []\n",
        "    tprs = []\n",
        "    aucs = []\n",
        "    mean_fpr = np.linspace(0, 1, 100)\n",
        "\n",
        "    # save figure location\n",
        "    roc_plot_file = f'{colab_dir if IN_COLAB else remote_dir}rf_roc_{ts}_{rf_ts}.png'\n",
        "    print(\"\\nStarting fold iterations\")\n",
        "    print(25*'-')\n",
        "    fig, ax = plt.subplots(figsize=(8,5))\n",
        "    # Iterate through folds\n",
        "    for i, (train_idx, val_idx) in enumerate(folds.split(features)):\n",
        "        # Training data for the fold\n",
        "        fold_train_X, fold_train_y = features[train_idx], labels.iloc[train_idx]\n",
        "        # Validation data for the fold\n",
        "        fold_test_X, fold_test_y = features[val_idx], labels.iloc[val_idx]\n",
        "        # Create the model\n",
        "        rf_clf = RandomForestClassifier(**rf_params)\n",
        "        # Train the model\n",
        "        rf_clf.fit(fold_train_X, fold_train_y)\n",
        "        # Extract and keep feature importances\n",
        "        feature_importance_values += rf_clf.feature_importances_ / folds.n_splits\n",
        "        # Make predictions using these importances\n",
        "        test_predictions += rf_clf.predict_proba(test_features)[:, 1] / folds.n_splits\n",
        "        # Record the out of fold predictions\n",
        "        y_pred = rf_clf.predict(fold_test_X)\n",
        "        y_proba = rf_clf.predict_proba(fold_test_X)\n",
        "        out_of_fold[val_idx] = y_proba[:, 1]\n",
        "        # Record train and ROC AUC scores\n",
        "        train_scores.append(roc_auc_score(fold_train_y, rf_clf.predict_proba(fold_train_X)[:, 1]))\n",
        "        test_scores.append(roc_auc_score(fold_test_y, y_proba[:, 1]))\n",
        "        viz = RocCurveDisplay.from_predictions(\n",
        "            fold_test_y,\n",
        "            y_proba[:, 1],\n",
        "            name=\"ROC fold {}\".format(i),\n",
        "            alpha=0.3,\n",
        "            lw=1,\n",
        "            ax=ax,\n",
        "        )\n",
        "        interp_tpr = np.interp(mean_fpr, viz.fpr, viz.tpr)\n",
        "        interp_tpr[0] = 0.0\n",
        "        tprs.append(interp_tpr)\n",
        "        aucs.append(viz.roc_auc)\n",
        "        # Clean up memory\n",
        "        del rf_clf, fold_train_X, fold_test_X, fold_train_y, fold_test_y\n",
        "        gc.collect()\n",
        "    \n",
        "    ax.plot([0, 1], [0, 1], linestyle=\"--\", lw=2, color=\"r\", label=\"Chance\", alpha=0.8)\n",
        "\n",
        "    mean_tpr = np.mean(tprs, axis=0)\n",
        "    mean_tpr[-1] = 1.0\n",
        "    mean_auc = auc(mean_fpr, mean_tpr)\n",
        "    std_auc = np.std(aucs)\n",
        "    ax.plot(\n",
        "        mean_fpr,\n",
        "        mean_tpr,\n",
        "        color=\"b\",\n",
        "        label=r\"Mean ROC (AUC = %0.2f $\\pm$ %0.2f)\" % (mean_auc, std_auc),\n",
        "        lw=2,\n",
        "        alpha=0.8,\n",
        "    )\n",
        "    \n",
        "    std_tpr = np.std(tprs, axis=0)\n",
        "    tprs_upper = np.minimum(mean_tpr + std_tpr, 1)\n",
        "    tprs_lower = np.maximum(mean_tpr - std_tpr, 0)\n",
        "    ax.fill_between(\n",
        "        mean_fpr,\n",
        "        tprs_lower,\n",
        "        tprs_upper,\n",
        "        color=\"grey\",\n",
        "        alpha=0.2,\n",
        "        label=r\"$\\pm$ 1 std. dev.\",\n",
        "    )\n",
        "\n",
        "    ax.set(\n",
        "        xlim=[-0.05, 1.05],\n",
        "        ylim=[-0.05, 1.05],\n",
        "        title=\"RandomForest - Receiver operating characteristic (ROC)\",\n",
        "    )\n",
        "    ax.legend(loc=\"lower right\")\n",
        "    # copyright text\n",
        "    plt.suptitle(cprt, y=-.03, verticalalignment='bottom', bbox=bb_cprt_args)\n",
        "    plt.tight_layout()\n",
        "    plt.savefig(roc_plot_file, transparent=False, bbox_inches=\"tight\")\n",
        "    plt.close()\n",
        "\n",
        "    print(25*'-')\n",
        "    print(\"Creating output\")\n",
        "    # Make the submission dataframe\n",
        "    submission = pd.DataFrame({'LNR': test_ids, 'RESPONSE': test_predictions})\n",
        "\n",
        "    # Make the feature importance dataframe\n",
        "    attrib_imp = pd.DataFrame({'attribute': feature_names,\n",
        "                               'importance': feature_importance_values})\n",
        "    \n",
        "    # Overall validation score\n",
        "    test_auc = roc_auc_score(labels, out_of_fold)\n",
        "    \n",
        "    # Add the overall scores to the metrics\n",
        "    test_scores.append(test_auc)\n",
        "    train_scores.append(np.mean(train_scores))\n",
        "    \n",
        "    # Needed for creating dataframe of validation scores\n",
        "    fold_names = list(range(n_folds))\n",
        "    fold_names.append('overall')\n",
        "    \n",
        "    # Dataframe of validation scores\n",
        "    metrics = pd.DataFrame({'fold': fold_names,\n",
        "                            'train': train_scores,\n",
        "                            'test': test_scores})\n",
        "    \n",
        "    return submission, attrib_imp, metrics"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W81ioUX69zUn",
        "outputId": "717e4e92-41dd-44bb-d00a-c3d8619d2c08"
      },
      "source": [
        "rf_params = {\n",
        "    'class_weight': 'balanced',\n",
        "}\n",
        "\n",
        "# submit_i, attrib_imp_i, metrics_i = rf_model(train_test_split(train, target, stratify=target, test_size=.25), rf_params)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Training Data Shape:  (262500, 376)\n",
            "Testing Data Shape:  (87500, 376)\n",
            "Creating KFold object\n",
            "\n",
            "Starting fold iterations\n",
            "-------------------------\n",
            "-------------------------\n",
            "Creating output\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zd97F3Sqay3m"
      },
      "source": [
        "# LightGBM\n",
        "import lightgbm as lgb\n",
        "\n",
        "def lgb_model(features_labels_split, lgbm_params, n_folds=5):\n",
        "    \"\"\"Train and test a light gradient boosting model using\n",
        "    cross validation with KFold splits. \n",
        "    \n",
        "    Args:\n",
        "        features_labels_split (list): List containing train-test split of inputs.\n",
        "            Either [X_train, X_test, y_train, y_test] or [features, test_features]\n",
        "            features will be pd.DataFrame of training features to use for training\n",
        "            a model. Must include the `RESPONSE` column.\n",
        "            test_features will be pd.DataFrame of testing features to use for\n",
        "            making predictions with the model.\n",
        "        lgbm_params (dict): LightGBM classifier init parameters.\n",
        "        n_folds (int, default = 5): number of folds to use for cross validation.\n",
        "        \n",
        "    Returns:\n",
        "        submission (pd.DataFrame): dataframe with `LNR` and `RESPONSE`\n",
        "            probabilities predicted by the model.\n",
        "        attrib_imp (pd.DataFrame): dataframe with the feature\n",
        "            importances from the model.\n",
        "        valid_metrics (pd.DataFrame): dataframe with training and validation\n",
        "            metrics (ROC AUC) for each fold and overall.\n",
        "    \n",
        "    Ref: https://www.kaggle.com/willkoehrsen/introduction-to-feature-selection#Remove-Collinear-Variables\n",
        "    \"\"\"\n",
        "    lgbm_ts = datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n",
        "    # Differentiate assignments according to input (length=4 -> train_test_split)\n",
        "    if len(features_labels_split) == 4:\n",
        "        features, test_features, labels, test_labels = features_labels_split\n",
        "    else:\n",
        "        features, test_features = features_labels_split\n",
        "        # Extract the labels for training\n",
        "        labels = features['RESPONSE']\n",
        "\n",
        "    # Extract the client IDs\n",
        "    train_ids = features['LNR']\n",
        "    test_ids = test_features['LNR']\n",
        "    # Drop the client IDs and datetimes\n",
        "    features.drop(columns=['LNR', 'EINGEFUEGT_AM', 'RESPONSE'], errors='ignore', inplace=True)\n",
        "    test_features.drop(columns=['LNR', 'EINGEFUEGT_AM'], errors='ignore', inplace=True)\n",
        "        \n",
        "    print('\\nTraining Data Shape: ', features.shape)\n",
        "    print('Testing Data Shape: ', test_features.shape)\n",
        "    \n",
        "    # Extract feature names\n",
        "    feature_names = features.columns.tolist()\n",
        "    \n",
        "    # Impute missing values with Random Selection\n",
        "    traintestX = [i.apply(imput_nan_rand) for i in (features, test_features)]\n",
        "    # Normalization\n",
        "    features, test_features = [scaler.fit_transform(i) for i in traintestX]\n",
        "\n",
        "    # Empty array for feature importances\n",
        "    feature_importance_values = np.zeros(len(feature_names))\n",
        "    # Empty array for test predictions\n",
        "    test_predictions = np.zeros(test_features.shape[0])\n",
        "    # Empty array for out of fold validation predictions\n",
        "    out_of_fold = np.zeros(features.shape[0])\n",
        "    \n",
        "    # Create the kfold object\n",
        "    print(\"Creating KFold object\")\n",
        "    folds = KFold(n_splits=n_folds)\n",
        "    \n",
        "    # Lists for recording validation and training scores\n",
        "    train_scores = []\n",
        "    test_scores = []\n",
        "    tprs = []\n",
        "    aucs = []\n",
        "    mean_fpr = np.linspace(0, 1, 100)\n",
        "\n",
        "    # save figure location\n",
        "    lgb_roc_plt_file = f'{colab_dir if IN_COLAB else remote_dir}lgb_roc_{ts}_{lgbm_ts}.png'\n",
        "    print(\"\\nStarting fold iterations\")\n",
        "    print(25*'-')\n",
        "    fig, ax = plt.subplots(figsize=(8,5))\n",
        "    # Iterate through folds\n",
        "    for i, (train_idx, val_idx) in enumerate(folds.split(features)):\n",
        "        # Training data for the fold\n",
        "        fold_train_X, fold_train_y = features[train_idx], labels.iloc[train_idx]\n",
        "        # Validation data for the fold\n",
        "        fold_test_X, fold_test_y = features[val_idx], labels.iloc[val_idx]\n",
        "        # Create the model\n",
        "        lgbm_clf = lgb.LGBMClassifier(**lgbm_params)\n",
        "        # Train the model\n",
        "        lgbm_clf.fit(\n",
        "            fold_train_X, fold_train_y,\n",
        "            eval_metric='auc',\n",
        "            eval_set=[(fold_test_X, fold_test_y), (fold_train_X, fold_train_y)],\n",
        "            eval_names=['test', 'train'],\n",
        "            early_stopping_rounds=100,\n",
        "            verbose = 200\n",
        "        )\n",
        "        # Keep the best iteration\n",
        "        best_iter = lgbm_clf.best_iteration_\n",
        "        # Extract and keep feature importances\n",
        "        feature_importance_values += lgbm_clf.feature_importances_ / folds.n_splits\n",
        "        # Make predictions using these importances\n",
        "        test_predictions += lgbm_clf.predict_proba(test_features, num_iteration=best_iter)[:, 1] / folds.n_splits\n",
        "        # Record the out of fold predictions\n",
        "        y_proba = lgbm_clf.predict_proba(fold_test_X, num_iteration=best_iter)\n",
        "        out_of_fold[val_idx] = y_proba[:, 1]\n",
        "        # Record the best score\n",
        "        train_scores.append(lgbm_clf.best_score_['train']['auc'])\n",
        "        test_scores.append(lgbm_clf.best_score_['test']['auc'])\n",
        "        viz = RocCurveDisplay.from_predictions(\n",
        "            fold_test_y,\n",
        "            y_proba[:, 1],\n",
        "            name=\"ROC fold {}\".format(i),\n",
        "            alpha=0.3,\n",
        "            lw=1,\n",
        "            ax=ax,\n",
        "        )\n",
        "        interp_tpr = np.interp(mean_fpr, viz.fpr, viz.tpr)\n",
        "        interp_tpr[0] = 0.0\n",
        "        tprs.append(interp_tpr)\n",
        "        aucs.append(viz.roc_auc)\n",
        "        # Clean up memory\n",
        "        del lgbm_clf, fold_train_X, fold_test_X, fold_train_y, fold_test_y\n",
        "        gc.collect()\n",
        "\n",
        "    ax.plot([0, 1], [0, 1], linestyle=\"--\", lw=2, color=\"r\", label=\"Chance\", alpha=0.8)\n",
        "\n",
        "    mean_tpr = np.mean(tprs, axis=0)\n",
        "    mean_tpr[-1] = 1.0\n",
        "    mean_auc = auc(mean_fpr, mean_tpr)\n",
        "    std_auc = np.std(aucs)\n",
        "    ax.plot(\n",
        "        mean_fpr,\n",
        "        mean_tpr,\n",
        "        color=\"b\",\n",
        "        label=r\"Mean ROC (AUC = %0.2f $\\pm$ %0.2f)\" % (mean_auc, std_auc),\n",
        "        lw=2,\n",
        "        alpha=0.8,\n",
        "    )\n",
        "    \n",
        "    std_tpr = np.std(tprs, axis=0)\n",
        "    tprs_upper = np.minimum(mean_tpr + std_tpr, 1)\n",
        "    tprs_lower = np.maximum(mean_tpr - std_tpr, 0)\n",
        "    ax.fill_between(\n",
        "        mean_fpr,\n",
        "        tprs_lower,\n",
        "        tprs_upper,\n",
        "        color=\"grey\",\n",
        "        alpha=0.2,\n",
        "        label=r\"$\\pm$ 1 std. dev.\",\n",
        "    )\n",
        "\n",
        "    ax.set(\n",
        "        xlim=[-0.05, 1.05],\n",
        "        ylim=[-0.05, 1.05],\n",
        "        title=\"LightGBM - Receiver operating characteristic (ROC)\",\n",
        "    )\n",
        "    ax.legend(loc=\"lower right\")\n",
        "    # copyright text\n",
        "    plt.suptitle(cprt, y=-.03, verticalalignment='bottom', bbox=bb_cprt_args)\n",
        "    plt.tight_layout()\n",
        "    plt.savefig(lgb_roc_plt_file, transparent=False, bbox_inches=\"tight\")\n",
        "    plt.close()\n",
        "\n",
        "    print(25*'-')\n",
        "    print(\"Creating output\")    \n",
        "    # Make the submission dataframe\n",
        "    submission = pd.DataFrame({'LNR': test_ids, 'RESPONSE': test_predictions})\n",
        "    \n",
        "    # Make the feature importance dataframe\n",
        "    attrib_imp = pd.DataFrame({'attribute': feature_names,\n",
        "                               'importance': feature_importance_values})\n",
        "    \n",
        "    # Overall validation score\n",
        "    test_auc = roc_auc_score(labels, out_of_fold)\n",
        "    \n",
        "    # Add the overall scores to the metrics\n",
        "    test_scores.append(test_auc)\n",
        "    train_scores.append(np.mean(train_scores))\n",
        "    \n",
        "    # Needed for creating dataframe of validation scores\n",
        "    fold_names = list(range(n_folds))\n",
        "    fold_names.append('overall')\n",
        "    \n",
        "    # Dataframe of validation scores\n",
        "    metrics = pd.DataFrame({'fold': fold_names,\n",
        "                            'train': train_scores,\n",
        "                            'test': test_scores}) \n",
        "    \n",
        "    return submission, attrib_imp, metrics"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8FtR_dqmQzVz"
      },
      "source": [
        "\"Ensembles like LGBM build trees in iterations, and each new tree is used to correct the “errors” of the previous trees. This approach is fast and powerful, and prone to overfitting.\n",
        "That’s why gradient boosted ensembles have a learning_rate parameter that controls the learning speed. Typical values lie within 0.01 and 0.3, but it is possible to go beyond these, especially towards 0.\n",
        "So, the perfect setup for these 2 parameters (n_estimators and learning_rate) is to use many trees with early stopping and set a low value for learning_rate.\"\n",
        "\n",
        "https://towardsdatascience.com/kagglers-guide-to-lightgbm-hyperparameter-tuning-with-optuna-in-2021-ed048d9838b5"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Npns7YOxdb7J",
        "outputId": "ac6c05b9-8523-4ea2-c827-5b247afacdda"
      },
      "source": [
        "lgbm_params = {\n",
        "    'boosting_type': 'goss',\n",
        "    'learning_rate': 0.05,\n",
        "    'n_estimators': 10000,\n",
        "    'objective': 'binary',\n",
        "    'class_weight': 'balanced',\n",
        "    'reg_alpha': 0.1,\n",
        "    'reg_lambda': 0.1,\n",
        "    'random_state': 67,\n",
        "    'n_jobs': -1,\n",
        "}\n",
        "\n",
        "srch_params = {\n",
        "    'boosting_type': 'gbdt',\n",
        "    'num_leaves': 100,\n",
        "    'max_depth': 13,\n",
        "    'learning_rate': 0.3,\n",
        "    'n_estimators': 10000,\n",
        "    'objective': 'binary',\n",
        "    'is_unbalance': True,\n",
        "    'subsample': .8,\n",
        "    'subsample_freq': 1,\n",
        "    'reg_alpha': 0.1,\n",
        "    'reg_lambda': 0.1,\n",
        "    'random_state': 67,\n",
        "    'n_jobs': -1,\n",
        "}\n",
        "\n",
        "submit_i, attrib_imp_i, metrics_i = lgb_model(train_test_split(train, target, stratify=target, test_size=.25), lgbm_params)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Training Data Shape:  (337500, 376)\n",
            "Testing Data Shape:  (112500, 376)\n",
            "Creating KFold object\n",
            "\n",
            "Starting fold iterations\n",
            "-------------------------\n",
            "Training until validation scores don't improve for 100 rounds.\n",
            "[200]\ttrain's auc: 0.929145\ttrain's binary_logloss: 0.339655\ttest's auc: 0.922348\ttest's binary_logloss: 0.354317\n",
            "[400]\ttrain's auc: 0.936108\ttrain's binary_logloss: 0.324076\ttest's auc: 0.923682\ttest's binary_logloss: 0.350557\n",
            "Early stopping, best iteration is:\n",
            "[480]\ttrain's auc: 0.938517\ttrain's binary_logloss: 0.318646\ttest's auc: 0.923901\ttest's binary_logloss: 0.349637\n",
            "Training until validation scores don't improve for 100 rounds.\n",
            "[200]\ttrain's auc: 0.928541\ttrain's binary_logloss: 0.341116\ttest's auc: 0.925391\ttest's binary_logloss: 0.348965\n",
            "[400]\ttrain's auc: 0.935509\ttrain's binary_logloss: 0.325617\ttest's auc: 0.926226\ttest's binary_logloss: 0.34578\n",
            "Early stopping, best iteration is:\n",
            "[487]\ttrain's auc: 0.937958\ttrain's binary_logloss: 0.320037\ttest's auc: 0.926275\ttest's binary_logloss: 0.345141\n",
            "Training until validation scores don't improve for 100 rounds.\n",
            "[200]\ttrain's auc: 0.928999\ttrain's binary_logloss: 0.340123\ttest's auc: 0.923092\ttest's binary_logloss: 0.353408\n",
            "[400]\ttrain's auc: 0.935964\ttrain's binary_logloss: 0.324475\ttest's auc: 0.924266\ttest's binary_logloss: 0.349549\n",
            "[600]\ttrain's auc: 0.941494\ttrain's binary_logloss: 0.311896\ttest's auc: 0.9243\ttest's binary_logloss: 0.348578\n",
            "Early stopping, best iteration is:\n",
            "[501]\ttrain's auc: 0.938797\ttrain's binary_logloss: 0.317972\ttest's auc: 0.924384\ttest's binary_logloss: 0.348897\n",
            "Training until validation scores don't improve for 100 rounds.\n",
            "[200]\ttrain's auc: 0.928488\ttrain's binary_logloss: 0.341264\ttest's auc: 0.924988\ttest's binary_logloss: 0.350768\n",
            "[400]\ttrain's auc: 0.935481\ttrain's binary_logloss: 0.325642\ttest's auc: 0.926473\ttest's binary_logloss: 0.346014\n",
            "[600]\ttrain's auc: 0.94105\ttrain's binary_logloss: 0.313093\ttest's auc: 0.926545\ttest's binary_logloss: 0.344911\n",
            "Early stopping, best iteration is:\n",
            "[510]\ttrain's auc: 0.938604\ttrain's binary_logloss: 0.31862\ttest's auc: 0.926604\ttest's binary_logloss: 0.345286\n",
            "Training until validation scores don't improve for 100 rounds.\n",
            "[200]\ttrain's auc: 0.929051\ttrain's binary_logloss: 0.339852\ttest's auc: 0.923238\ttest's binary_logloss: 0.353566\n",
            "[400]\ttrain's auc: 0.936033\ttrain's binary_logloss: 0.324233\ttest's auc: 0.92427\ttest's binary_logloss: 0.350171\n",
            "[600]\ttrain's auc: 0.941581\ttrain's binary_logloss: 0.311634\ttest's auc: 0.924471\ttest's binary_logloss: 0.348543\n",
            "Early stopping, best iteration is:\n",
            "[641]\ttrain's auc: 0.942619\ttrain's binary_logloss: 0.309186\ttest's auc: 0.924531\ttest's binary_logloss: 0.348428\n",
            "-------------------------\n",
            "Creating output\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 235
        },
        "id": "lxLjBQ4UmCRX",
        "outputId": "90c5b9d5-1c3f-4227-ced9-c287fe9e18c8"
      },
      "source": [
        "# metrics_i.to_csv(f'{colab_dir if IN_COLAB else remote_dir}train_metricsLGBM_{ts}.csv', index=False)\n",
        "metrics_i"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>fold</th>\n",
              "      <th>train</th>\n",
              "      <th>test</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>0.938517</td>\n",
              "      <td>0.923901</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>0.937958</td>\n",
              "      <td>0.926275</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>0.938797</td>\n",
              "      <td>0.924384</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>0.938604</td>\n",
              "      <td>0.926604</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>0.942619</td>\n",
              "      <td>0.924531</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>overall</td>\n",
              "      <td>0.939299</td>\n",
              "      <td>0.925134</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "      fold     train      test\n",
              "0        0  0.938517  0.923901\n",
              "1        1  0.937958  0.926275\n",
              "2        2  0.938797  0.924384\n",
              "3        3  0.938604  0.926604\n",
              "4        4  0.942619  0.924531\n",
              "5  overall  0.939299  0.925134"
            ]
          },
          "metadata": {},
          "execution_count": 165
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D6SWKHEgl6d0"
      },
      "source": [
        "Our model has identified **zero importance** attributes that we may use to further reduce the number of attributes in our datasets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9i5Fp2aWGmWH"
      },
      "source": [
        "def plot_attr_imp(df, n_attrs=15, thres=.95, fig_size=(10, 5), save_file=None):\n",
        "    \"\"\"Plots n_attrs most important features and the cumulative importance of\n",
        "    attributes.Prints the number of features needed to reach the threshold of\n",
        "    cumulative importance.\n",
        "    \n",
        "    Args:\n",
        "        df (pd.DataFrame): A dataframe of attribute importances.\n",
        "            Columns must be `attribute` and `importance`.\n",
        "        n_attrs (int): Number of attributes to plot.\n",
        "        thres (float): Plot information about cumulative importances above `thres`.\n",
        "        fig_size (tuple of int): Figure size for the combined plot.\n",
        "        \n",
        "    Returns:\n",
        "        df (pd.DataFrame): A dataframe ordered by attribute importances with a\n",
        "            normalized column (sums to 1) and a cumulative importance column.\n",
        "    \"\"\"\n",
        "    # Sort features according to importance\n",
        "    df = df.sort_values('importance', ascending=False).reset_index()\n",
        "    # Normalize the feature importances to add up to one\n",
        "    df['imp_norm'] = df['importance'].div(df['importance'].sum())\n",
        "    df['cumul_imp'] = df['imp_norm'].cumsum()\n",
        "\n",
        "    imp_attrs = df[df['cumul_imp'] < thres]['cumul_imp'].count()\n",
        "    print(f'\\n{imp_attrs} attributes provide cumulative importance almost {thres:.2f}\\n')\n",
        "\n",
        "    # with sns.axes_style(\"whitegrid\"), sns.plotting_context(\"notebook\", font_scale=1.5):\n",
        "    with sns.axes_style(\"whitegrid\"), sns.plotting_context(\"paper\"):\n",
        "        fig, ax = plt.subplots(1, 2, figsize=fig_size)\n",
        "        # Horizontal bar plot of attribute importances\n",
        "        sns.barplot(ax=ax[0], data=df.head(n_attrs),\n",
        "                    x='imp_norm', y='attribute', color='royalblue')\n",
        "        ax[0].set(title=f'Attribute importances [first {str(n_attrs)}]',\n",
        "                  xlabel='Importance (normalized)')\n",
        "        # Cumulative importance plot\n",
        "        sns.lineplot(ax=ax[1], data=df['cumul_imp'], color='r', linewidth=3)\n",
        "        ax[1].set(title=f'Scree plot, mark: {str(thres)}, {str(imp_attrs)} attributes',\n",
        "              xlabel='Number of attributes',\n",
        "              ylabel='Cumulative importance (normalized)')\n",
        "        ax[1].axhline(thres, linestyle='--',\n",
        "                      color='royalblue', lw=2, alpha=0.6)\n",
        "        ax[1].axvline(imp_attrs, linestyle='--',\n",
        "                      color='royalblue', lw=2, alpha=0.6)\n",
        "        # copyright text\n",
        "        fig.suptitle(cprt, y=-.03, verticalalignment='bottom', bbox=bb_cprt_args)\n",
        "        fig.tight_layout()\n",
        "        if save_file:\n",
        "            fig.savefig(save_file, transparent=False, bbox_inches=\"tight\")\n",
        "            plt.close(fig)\n",
        "\n",
        "    return df"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mUvRLfvOK5rB"
      },
      "source": [
        "# save figure location\n",
        "attr_imp_plt_file = f'{colab_dir if IN_COLAB else remote_dir}tree_attr_imp_{ts}.png'\n",
        "\n",
        "norm_attrs_imp = plot_attr_imp(attrib_imp_i, save_file=attr_imp_plt_file)\n",
        "norm_attrs_imp.head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DFx47peYbR-u"
      },
      "source": [
        "zero_imp_drop = norm_attrs_imp.loc[norm_attrs_imp['cumul_imp'] > .95, 'attribute'].to_list()\n",
        "len(zero_imp_drop)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b33C4jsCdyGd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "373a97bc-9789-44c9-a91a-8df1ac4f772f"
      },
      "source": [
        "# Clean up memory\n",
        "del train, target, df_train\n",
        "gc.collect()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "5489"
            ]
          },
          "metadata": {},
          "execution_count": 167
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s61WnK25Gsmh"
      },
      "source": [
        "We can observe that using 0.95 cumulative feature importance can provide almost the same results as when testing with most attributes or 1.0 cumulative features importance. Usually 0.95 removes about 100 attributes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "elzZ61pJbUG5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7faa9336-c329-4c06-b4e9-2053b18ee3f6"
      },
      "source": [
        "# mailout_train = pd.read_csv(mailout_train_file, sep=';')\n",
        "mailout_train = pd.read_csv(mailout_train_file, sep=';', na_values=na_tupl, parse_dates=['EINGEFUEGT_AM'])\n",
        "mailout_train.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(42962, 367)"
            ]
          },
          "metadata": {},
          "execution_count": 168
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lRS3VIPoKeDY",
        "outputId": "6b56d554-c9d4-40e9-cb0e-98d29134e48e"
      },
      "source": [
        "# Hold RESPONSE values separately\n",
        "target_mailout = mailout_train['RESPONSE'].fillna(0)\n",
        "mailout_train.drop(columns=['RESPONSE'], inplace=True)\n",
        "\n",
        "mailout_train.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(42962, 366)"
            ]
          },
          "metadata": {},
          "execution_count": 169
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3uy3vUZUbUG6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "203233d0-16b6-45a1-dfbb-62e400dffe5a"
      },
      "source": [
        "target_mailout.value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.0    42430\n",
              "1.0      532\n",
              "Name: RESPONSE, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 170
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "guexXUC5wsnF"
      },
      "source": [
        "We see that there is a large class **imbalance** in the dataset. There are a lot of individuals that did not respond to the mail campaign."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3a2Ii-_zqCmx"
      },
      "source": [
        "# save figure location\n",
        "mailout_resp_distr_file = f'{colab_dir if IN_COLAB else remote_dir}mailout_resp_distr_{ts}.png'\n",
        "\n",
        "with sns.axes_style(\"whitegrid\"):\n",
        "    plt.figure(figsize=(4,5))\n",
        "    ax1 = sns.countplot(x=target_mailout)\n",
        "    ax1.set(xticks=range(2),\n",
        "            xticklabels=['Missing', 'Customers'],\n",
        "            title='Responses counts')\n",
        "    # copyright text\n",
        "    plt.suptitle(cprt, y=-.03, verticalalignment='bottom', bbox=bb_cprt_args)\n",
        "    plt.tight_layout()\n",
        "    plt.savefig(mailout_resp_distr_file, transparent=False, bbox_inches=\"tight\")\n",
        "    plt.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R6MKDkgFumT7"
      },
      "source": [
        "#### Clean - fix dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1s48lqowuJNJ",
        "outputId": "89a8b703-7b10-4552-92ec-25d77a8c72f8"
      },
      "source": [
        "# `wrangle_df` parameters tuple\n",
        "# including *** Drop zero importance attributes *** `zero_imp_drop`\n",
        "wrangle_args = [\n",
        "    attrs_val_pools,\n",
        "    attrs_common,\n",
        "    replace9_nan_dict,\n",
        "    zero_imp_drop\n",
        "]\n",
        "\n",
        "mailout_train = wrangle_df(mailout_train, *wrangle_args)\n",
        "mailout_train.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total df NaNs portion:  0.3064\n",
            "Column NaNs:\n",
            "Average NaNs per column:  0.3064\n",
            "Max NaNs per column:  0.9991\n",
            "Min NaNs per column:  0.0\n",
            "Row NaNs:\n",
            "Average NaNs per row:  0.3064\n",
            "Max NaNs per row:  0.8462\n",
            "Min NaNs per row:  0.0714\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(42962, 361)"
            ]
          },
          "metadata": {},
          "execution_count": 172
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ebBxq1lpvHZa",
        "outputId": "8d684f38-8fda-40ec-a346-120d3999bef2"
      },
      "source": [
        "# Run the LightGBM model\n",
        "submitlgb_ml, attrib_imp_ml, metrics_ml = lgb_model(train_test_split(mailout_train, target_mailout, stratify=target_mailout, test_size=.2), lgbm_params)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Training Data Shape:  (34369, 360)\n",
            "Testing Data Shape:  (8593, 360)\n",
            "Creating KFold object\n",
            "\n",
            "Starting fold iterations\n",
            "-------------------------\n",
            "Training until validation scores don't improve for 100 rounds.\n",
            "Early stopping, best iteration is:\n",
            "[33]\ttrain's auc: 0.974685\ttrain's binary_logloss: 0.367624\ttest's auc: 0.73954\ttest's binary_logloss: 0.40398\n",
            "Training until validation scores don't improve for 100 rounds.\n",
            "Early stopping, best iteration is:\n",
            "[64]\ttrain's auc: 0.999321\ttrain's binary_logloss: 0.246163\ttest's auc: 0.724219\ttest's binary_logloss: 0.28541\n",
            "Training until validation scores don't improve for 100 rounds.\n",
            "Early stopping, best iteration is:\n",
            "[39]\ttrain's auc: 0.984599\ttrain's binary_logloss: 0.343664\ttest's auc: 0.746461\ttest's binary_logloss: 0.391839\n",
            "Training until validation scores don't improve for 100 rounds.\n",
            "Early stopping, best iteration is:\n",
            "[49]\ttrain's auc: 0.993578\ttrain's binary_logloss: 0.305967\ttest's auc: 0.763851\ttest's binary_logloss: 0.346889\n",
            "Training until validation scores don't improve for 100 rounds.\n",
            "Early stopping, best iteration is:\n",
            "[40]\ttrain's auc: 0.984937\ttrain's binary_logloss: 0.338968\ttest's auc: 0.748874\ttest's binary_logloss: 0.376165\n",
            "-------------------------\n",
            "Creating output\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sZ67vHte6AUX",
        "outputId": "7aaa34ea-4c59-4072-aa16-56319307d807"
      },
      "source": [
        "# Run the RandomForest model\n",
        "# submitlgb_ml, attrib_imp_ml, metrics_ml = rf_model(train_test_split(mailout_train, target_mailout, stratify=target_mailout, test_size=.2), rf_params)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Training Data Shape:  (34369, 360)\n",
            "Testing Data Shape:  (8593, 360)\n",
            "Creating KFold object\n",
            "\n",
            "Starting fold iterations\n",
            "-------------------------\n",
            "-------------------------\n",
            "Creating output\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 235
        },
        "id": "376jjSdP6vFn",
        "outputId": "955d402f-514a-405e-b1ee-b51d0142f6bb"
      },
      "source": [
        "# metrics_ml.to_csv(f'{colab_dir if IN_COLAB else remote_dir}mailout_train_metrics_{ts}.csv', index=False)\n",
        "metrics_ml"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>fold</th>\n",
              "      <th>train</th>\n",
              "      <th>test</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>0.974685</td>\n",
              "      <td>0.739540</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>0.999321</td>\n",
              "      <td>0.724219</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>0.984599</td>\n",
              "      <td>0.746461</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>0.993578</td>\n",
              "      <td>0.763851</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>0.984937</td>\n",
              "      <td>0.748874</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>overall</td>\n",
              "      <td>0.987424</td>\n",
              "      <td>0.740067</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "      fold     train      test\n",
              "0        0  0.974685  0.739540\n",
              "1        1  0.999321  0.724219\n",
              "2        2  0.984599  0.746461\n",
              "3        3  0.993578  0.763851\n",
              "4        4  0.984937  0.748874\n",
              "5  overall  0.987424  0.740067"
            ]
          },
          "metadata": {},
          "execution_count": 176
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rycrvhFAbUG6"
      },
      "source": [
        "## Part 3: Kaggle Competition\n",
        "\n",
        "Now that you've created a model to predict which individuals are most likely to respond to a mailout campaign, it's time to test that model in competition through Kaggle. If you click on the link [here](http://www.kaggle.com/t/21e6d45d4c574c7fa2d868f0e8c83140), you'll be taken to the competition page where, if you have a Kaggle account, you can enter.\n",
        "\n",
        "Your entry to the competition should be a CSV file with two columns. The first column should be a copy of \"LNR\", which acts as an ID number for each individual in the \"TEST\" partition. The second column, \"RESPONSE\", should be some measure of how likely each individual became a customer – this might not be a straightforward probability. As you should have found in Part 2, there is a large output class imbalance, where most individuals did not respond to the mailout. Thus, predicting individual classes and using accuracy does not seem to be an appropriate performance evaluation method. Instead, the competition will be using AUC to evaluate performance. The exact values of the \"RESPONSE\" column do not matter as much: only that the higher values try to capture as many of the actual customers as possible, early in the ROC curve sweep."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yuGsuMO6bUG6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6083f7b4-5fda-4c09-d351-e5202fa66be8"
      },
      "source": [
        "#mailout_test = pd.read_csv(mailout_test_file, sep=';') \n",
        "mailout_test = pd.read_csv(mailout_test_file, sep=';', na_values=na_tupl, parse_dates=['EINGEFUEGT_AM'])\n",
        "mailout_test.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(42833, 366)"
            ]
          },
          "metadata": {},
          "execution_count": 177
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3DLSx-CGQpxa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0fb64ea4-0b72-42fd-f3cf-2b7cf7d50d07"
      },
      "source": [
        "# Clean - fix\n",
        "# including *** Drop zero importance attributes *** `zero_imp_drop`\n",
        "mailout_test = wrangle_df(mailout_test, *wrangle_args)\n",
        "mailout_test.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total df NaNs portion:  0.3048\n",
            "Column NaNs:\n",
            "Average NaNs per column:  0.3048\n",
            "Max NaNs per column:  0.9991\n",
            "Min NaNs per column:  0.0\n",
            "Row NaNs:\n",
            "Average NaNs per row:  0.3048\n",
            "Max NaNs per row:  0.8599\n",
            "Min NaNs per row:  0.0742\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(42833, 361)"
            ]
          },
          "metadata": {},
          "execution_count": 178
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ou2XknY6L40H",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2671fc9c-7f19-48d6-e0f5-e80105a1f03b"
      },
      "source": [
        "# Put the RESPONSE column back in the train set.\n",
        "mailout_train = mailout_train.assign(RESPONSE=target_mailout.values)\n",
        "mailout_train.shape, mailout_test.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((42962, 362), (42833, 361))"
            ]
          },
          "metadata": {},
          "execution_count": 179
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dvlV7gj7RzoD",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d41d48d2-1aee-4af5-cde6-8739bbdfc2f7"
      },
      "source": [
        "# Check there are no NaNs and no infinites\n",
        "np.any(np.isnan(mailout_test)), np.all(np.isfinite(mailout_test))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(False, True)"
            ]
          },
          "metadata": {},
          "execution_count": 191
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6OBqyD1alU--"
      },
      "source": [
        "Create a function to submit directly to Kaggle competition from within the notebook."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VXh1BEtebw5o"
      },
      "source": [
        "# When using Google Colab we want to copy the Kaggle API key and check kaggle version\n",
        "if IN_COLAB:\n",
        "    !cp -Rv /content/drive/MyDrive/workspace/arvato_data/.kaggle ~/ && chmod 600 ~/.kaggle/kaggle.json"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A60ehY5Xb341"
      },
      "source": [
        "kg_version = !pip show kaggle | grep Version\n",
        "kg_version = kg_version[0].split()[1].split('.')\n",
        "kg_version"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cbQ73nNFb-qL"
      },
      "source": [
        "# if kaggle API version is greater than 1.5.0 load kaggle API:\n",
        "if not (int(kg_version[1]) == 5) and not (int(kg_version[2]) > 0):\n",
        "    print(\"Kaggle module must be upgraded!\")\n",
        "    !pip install kaggle --upgrade"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SDhQXEzEHP-P"
      },
      "source": [
        "from kaggle.api.kaggle_api_extended import KaggleApi\n",
        "\n",
        "def kaggle_submit(filename, message, competition, data):\n",
        "    \"\"\"Prepare data, export to csv file and submit to kaggle competition.\n",
        "    Args:\n",
        "        filename (srt): name of the csv file to submit and write to\n",
        "        message (srt): message related to submission\n",
        "        competition (srt): the name of kaggle competition to submit \n",
        "        data (pd.DataFrame): submission dataframe of the form e.g.\n",
        "        {\"LNR\": pd.Series, \"RESPONSE\": pd.Series}\n",
        "    Returns:\n",
        "        (str): Submission request response\n",
        "    \"\"\"\n",
        "    # kaggle competition: udacity-arvato-identify-customers\n",
        "    # Kaggle competition invitation: http://www.kaggle.com/t/21e6d45d4c574c7fa2d868f0e8c83140\n",
        "    kg_api = KaggleApi()\n",
        "    kg_api.authenticate()\n",
        "\n",
        "    data.to_csv(filename, index=False)\n",
        "\n",
        "    return kg_api.competition_submit(filename, message, competition)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iau2mjtvT16h",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dd9eea9e-653a-4a3a-bf1f-8e3c952a06f1"
      },
      "source": [
        "# Run the model\n",
        "submit_kg, attrib_imp_kg, metrics_kg = lgb_model([mailout_train, mailout_test], lgbm_params)\n",
        "# submit_kg, attrib_imp_kg, metrics_kg = rf_model([mailout_train, mailout_test], rf_params)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Training Data Shape:  (42962, 360)\n",
            "Testing Data Shape:  (42833, 360)\n",
            "Creating KFold object\n",
            "\n",
            "Starting fold iterations\n",
            "-------------------------\n",
            "Training until validation scores don't improve for 100 rounds.\n",
            "Early stopping, best iteration is:\n",
            "[14]\ttrain's auc: 0.90073\ttrain's binary_logloss: 0.511441\ttest's auc: 0.739591\ttest's binary_logloss: 0.528152\n",
            "Training until validation scores don't improve for 100 rounds.\n",
            "Early stopping, best iteration is:\n",
            "[56]\ttrain's auc: 0.991289\ttrain's binary_logloss: 0.303036\ttest's auc: 0.76046\ttest's binary_logloss: 0.349822\n",
            "Training until validation scores don't improve for 100 rounds.\n",
            "Early stopping, best iteration is:\n",
            "[30]\ttrain's auc: 0.935044\ttrain's binary_logloss: 0.421667\ttest's auc: 0.789337\ttest's binary_logloss: 0.450539\n",
            "Training until validation scores don't improve for 100 rounds.\n",
            "Early stopping, best iteration is:\n",
            "[55]\ttrain's auc: 0.989218\ttrain's binary_logloss: 0.309941\ttest's auc: 0.744332\ttest's binary_logloss: 0.352998\n",
            "Training until validation scores don't improve for 100 rounds.\n",
            "Early stopping, best iteration is:\n",
            "[14]\ttrain's auc: 0.875167\ttrain's binary_logloss: 0.514108\ttest's auc: 0.74285\ttest's binary_logloss: 0.532049\n",
            "-------------------------\n",
            "Creating output\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fjxMsZF9iaR0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 235
        },
        "outputId": "a977750e-d189-4734-a96a-16d101cf835c"
      },
      "source": [
        "# metrics_kg.to_csv(f'{colab_dir if IN_COLAB else remote_dir}mailout_TEST_metrics_{ts}.csv', index=False)\n",
        "metrics_kg"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>fold</th>\n",
              "      <th>train</th>\n",
              "      <th>test</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>0.900730</td>\n",
              "      <td>0.739591</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>0.991289</td>\n",
              "      <td>0.760460</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>0.935044</td>\n",
              "      <td>0.789337</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>0.989218</td>\n",
              "      <td>0.744332</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>0.875167</td>\n",
              "      <td>0.742850</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>overall</td>\n",
              "      <td>0.938290</td>\n",
              "      <td>0.743078</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "      fold     train      test\n",
              "0        0  0.900730  0.739591\n",
              "1        1  0.991289  0.760460\n",
              "2        2  0.935044  0.789337\n",
              "3        3  0.989218  0.744332\n",
              "4        4  0.875167  0.742850\n",
              "5  overall  0.938290  0.743078"
            ]
          },
          "metadata": {},
          "execution_count": 184
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CeTpW-y_jHV6"
      },
      "source": [
        "# Submit to Kaggle competition\n",
        "msg = 'LightGBM [31 leaves, -1 depth] all mailout data'\n",
        "try:\n",
        "    submit_result = kaggle_submit(kg_submission + \"lgbm_test_{}.csv\".format(ts), kg_competition, msg, submit_kg)\n",
        "    print(submit_result)\n",
        "except Exception as ex:\n",
        "    print(\"Exception: \", str(ex))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LJwDJaTU1FMg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c804ddb4-6d37-49ef-cdf8-0d2e88c27e6c"
      },
      "source": [
        "print(\"Total run time: \", str(timedelta(seconds=(timer() - start_run))))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "metadata": {
            "tags": null
          },
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Total run time:  0:20:24.338065\n"
          ]
        }
      ]
    }
  ]
}