(2)automated_feature_engineering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (2) Automated Feature Engineering\n",
    "\n",
    "In this secondary notebook, we will apply automated **feature engineering** via `featuretools`. It is worthwile to mention that `h2o` contains two algorithms that are somewhat related to what we are doing here: [Aggregator](http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/aggregator.html), [Principal Component Analysis (PCP)](http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/pca.html), and [Generalized Low Rank Models (GLRM)](http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/glrm.html). While the functionalities of `featuretools` surpasses **Aggregator** algorithm of `h2o`, **PCA** and **GLRM** could be explored more for dimensionality reduction. This is especially relevant for the problem at hand, as we will be generating over hundred features in the end of this notebook, and training models based on these features will take a lot of time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "# !pip install featuretools\n",
    "# !pip install graphviz\n",
    "import featuretools as ft\n",
    "from utils import unknown_to_nan"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading Data\n",
    "\n",
    "### Important Notes\n",
    "* We don't want to use the parameter `@index_col` of `pandas.read_csv()` method. For establishing **one-to-many** relationships between different variables in different data frames with `featuretools`, we need to make sure to have them as separate columns rather than indices for access. Specifically, here we are referring to `entity_from_dataframe()` method of the `featuretools.EntitySet()` which accepts a parameter `@index`. We will cover these in the next notebook. \n",
    "* We will also not be utilizing `countries.csv` in this example as countries referred to here are the labels (to-be predicted values) for the problem at hand.\n",
    "* Altough countries in this file and the country destinations in `train_users.csv` constitute a **one-to-many** relationship, we will remove 'country_destinations' column from the training data frame as we don't want to synthesize any new features on it by transformations or aggregations via `featuretools`. The `dfs()` method, as we shall see later on this notebook, alternatively contains the argument `@ignore_variables` and we could have used this too, but this is probably a safer approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "date_variables = ['date_account_created', 'timestamp_first_active']\n",
    "users_df = pd.read_csv('(1)data_manual_ops/train_users.csv', parse_dates=date_variables)\n",
    "users_test_df = pd.read_csv('(1)data_manual_ops/test_users.csv', parse_dates=date_variables)\n",
    "buckets_df = pd.read_csv('(1)data_manual_ops/age_gender_bkts.csv')\n",
    "sessions_df = pd.read_csv('(0)data/sessions.csv')\n",
    "\n",
    "# Sort data frames by user ID to match the feature matrix featuretools will yield\n",
    "users_df.sort_values('id', inplace=True)\n",
    "users_test_df.sort_values('id', inplace=True)\n",
    "\n",
    "# Remove target variable from training set\n",
    "targets = users_df['country_destination'].values\n",
    "users_df.drop('country_destination', axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Extraction & Synthesis with `featuretools`\n",
    "\n",
    "### Entities & EntitySet Setup\n",
    "An `EntitySet` is a collection of entities and the relationships between them. They are useful for preparing raw, structured datasets for feature engineering. Here, each `Entity` can be roughly thought of as an independent data frame and the `EntitySet` wraps these data frames together via **one-to-many** relationships. \n",
    "\n",
    "After creating an `Entity` within an `EntitySet` as shown below, you can use `print(<EntitySet[<EntityID>].variable_types)` to see the mapping of string variable IDs (column names of the data frame) to the corresponding `ft.variable_types.variable` and check if there is any disparity OR something that could be improved. Some main variable types included are: *Index*, *TimeIndex*, *Datetime*, *Numeric*, *Categorical*, *Ordinal*, *Boolean*, *Text*, *LatLong*, *ZIPCode*, *IPAddress*, *FullName*, *Email Address*, *URL*, *PhoneNumber*, *DateOfBirth*, and *CountryCode*. Correctly mapping variables to variable types will enhance the utility of primitives as we will discuss soon. For example, in our problem, 'signup_flow' column will be converted to a categorical variable whereas it was taken as a numerical variable by `pandas` initially.\n",
    "\n",
    "In case a data frame that we are adding to the entity set as a new entity doesn't already have a unique ID column, we can create a new index by passing an arbitrary column name to `@index parameter` and suppress the warning by enabling parameter `@make_index`.\n",
    "\n",
    "In this example we will create seperate `EntitySet`s for training and testing users. This is the recommended solution as outlined by Max Kanter (creator of `featuretools`) [here](https://stackoverflow.com/questions/49711987/how-do-i-prevent-data-leakage-with-featuretools). Although other methods could also potentially work, my experience was that `featuretools` automatically sorts data frame rows during **DFS (Deep Feature Synthesis)** and extracting training and testing data frames through user IDs was a very messy method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Entityset: bookings\n",
      "  Entities:\n",
      "    users [Rows: 212593, Columns: 15]\n",
      "    sessions [Rows: 10567737, Columns: 7]\n",
      "    buckets [Rows: 42, Columns: 11]\n",
      "  Relationships:\n",
      "    No relationships\n",
      "{'id': <class 'featuretools.variable_types.variable.Index'>, 'date_account_created': <class 'featuretools.variable_types.variable.Datetime'>, 'timestamp_first_active': <class 'featuretools.variable_types.variable.Datetime'>, 'gender': <class 'featuretools.variable_types.variable.Categorical'>, 'age': <class 'featuretools.variable_types.variable.Numeric'>, 'signup_method': <class 'featuretools.variable_types.variable.Categorical'>, 'language': <class 'featuretools.variable_types.variable.Categorical'>, 'affiliate_channel': <class 'featuretools.variable_types.variable.Categorical'>, 'affiliate_provider': <class 'featuretools.variable_types.variable.Categorical'>, 'first_affiliate_tracked': <class 'featuretools.variable_types.variable.Categorical'>, 'signup_app': <class 'featuretools.variable_types.variable.Categorical'>, 'first_device_type': <class 'featuretools.variable_types.variable.Categorical'>, 'first_browser': <class 'featuretools.variable_types.variable.Categorical'>, 'age_gender_bucket': <class 'featuretools.variable_types.variable.Categorical'>, 'signup_flow': <class 'featuretools.variable_types.variable.Categorical'>}\n",
      "Entityset: bookings_test\n",
      "  Entities:\n",
      "    users [Rows: 62096, Columns: 15]\n",
      "    sessions [Rows: 10567737, Columns: 7]\n",
      "    buckets [Rows: 42, Columns: 11]\n",
      "  Relationships:\n",
      "    No relationships\n",
      "{'id': <class 'featuretools.variable_types.variable.Index'>, 'date_account_created': <class 'featuretools.variable_types.variable.Datetime'>, 'timestamp_first_active': <class 'featuretools.variable_types.variable.Datetime'>, 'gender': <class 'featuretools.variable_types.variable.Categorical'>, 'age': <class 'featuretools.variable_types.variable.Numeric'>, 'signup_method': <class 'featuretools.variable_types.variable.Categorical'>, 'language': <class 'featuretools.variable_types.variable.Categorical'>, 'affiliate_channel': <class 'featuretools.variable_types.variable.Categorical'>, 'affiliate_provider': <class 'featuretools.variable_types.variable.Categorical'>, 'first_affiliate_tracked': <class 'featuretools.variable_types.variable.Categorical'>, 'signup_app': <class 'featuretools.variable_types.variable.Categorical'>, 'first_device_type': <class 'featuretools.variable_types.variable.Categorical'>, 'first_browser': <class 'featuretools.variable_types.variable.Categorical'>, 'age_gender_bucket': <class 'featuretools.variable_types.variable.Categorical'>, 'signup_flow': <class 'featuretools.variable_types.variable.Categorical'>}\n"
     ]
    }
   ],
   "source": [
    "# Initialize the main training entity set\n",
    "es = ft.EntitySet(id='bookings')\n",
    "# Add training users to training entity set as a new entity\n",
    "es = es.entity_from_dataframe(entity_id='users',\n",
    "                              dataframe=users_df,\n",
    "                              index='id',\n",
    "                              variable_types={'signup_flow': ft.variable_types.Categorical})\n",
    "# Add sessions to training entity set as a new entity\n",
    "es = es.entity_from_dataframe(entity_id='sessions',\n",
    "                              dataframe=sessions_df,\n",
    "                              index='data_id',\n",
    "                              make_index=True)\n",
    "# Add age & gender buckets to training entity set as a new entity\n",
    "es = es.entity_from_dataframe(entity_id='buckets',\n",
    "                              dataframe=buckets_df,\n",
    "                              index='bucket_id')\n",
    "# es.plot()  # display training entity set\n",
    "print(es)\n",
    "print(es['users'].variable_types)\n",
    "\n",
    "# Initialize the main testing entity set\n",
    "es_test = ft.EntitySet(id='bookings_test')\n",
    "# Add testing users to testing entity set as a new entity\n",
    "es_test = es_test.entity_from_dataframe(entity_id='users',\n",
    "                                        dataframe=users_test_df,\n",
    "                                        index='id',\n",
    "                                        variable_types={'signup_flow': ft.variable_types.Categorical})\n",
    "# Add sessions to testing entity set as a new entity\n",
    "es_test = es_test.entity_from_dataframe(entity_id='sessions',\n",
    "                                        dataframe=sessions_df,\n",
    "                                        index='data_id')\n",
    "# Add age & gender buckets to testing entity set as a new entity\n",
    "es_test = es_test.entity_from_dataframe(entity_id='buckets',\n",
    "                                   dataframe=buckets_df,\n",
    "                                   index='bucket_id')\n",
    "# es_test.plot()  # display testing entity set\n",
    "print(es_test)\n",
    "print(es_test['users'].variable_types)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Relationships\n",
    "In our problem, each user (and associated ID) has multiple session records. Here, the users entity is called **parent entity**, and the sessions entity is known as the **child entity** in this relationship. Similarly, every gender bucket has multiple examples in the users entity. Hence, buckets entity is the parent entity and the users entity is the child entity in this relationship. When specifying relationships we list the variable in the parent entity first. Note that each `ft.Relationship` must denote a **one-to-many** relationship rather than a relationship which is *one-to-one* or *many-to-many*.\n",
    "\n",
    "Creating relationships between variables is pretty easy within `featuretools`, but it comes with two main problems:\n",
    "1. The only allowed relationships are **one-to-many**, and this means that there will be scenarios, like the one we exhibited in *Manual Feature Engineering*, where we will have to mutate parameters until we reach such a relationship.\n",
    "2. We can identify unique columns in our `pandas` data frame and these could potentially give us ideas of relationships where these unique columns constitute the *one* in the **one-to-many** relationships. The realization is that there is nothing automated here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<Relationship: sessions.user_id -> users.id>, <Relationship: users.age_gender_bucket -> buckets.bucket_id>]\n",
      "[<Relationship: sessions.user_id -> users.id>, <Relationship: users.age_gender_bucket -> buckets.bucket_id>]\n"
     ]
    }
   ],
   "source": [
    "# Initialize one-to-many (in this exact order) relationships & add to training entity set\n",
    "id_to_id = ft.Relationship(es['users']['id'], \n",
    "                           es['sessions']['user_id'])\n",
    "bucket_to_bucket = ft.Relationship(es['buckets']['bucket_id'], \n",
    "                                   es['users']['age_gender_bucket'])\n",
    "es.add_relationships([id_to_id, bucket_to_bucket])\n",
    "# Initialize one-to-many (in this exact order) relationships & add to testing entity set\n",
    "id_to_id_test = ft.Relationship(es_test['users']['id'], \n",
    "                                es_test['sessions']['user_id'])\n",
    "bucket_to_bucket_test = ft.Relationship(es_test['buckets']['bucket_id'],\n",
    "                                        es_test['users']['age_gender_bucket'])\n",
    "es_test.add_relationships([id_to_id_test, bucket_to_bucket_test])\n",
    "# Observe initialized relationships\n",
    "print(es.relationships)\n",
    "print(es_test.relationships)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature Primitives\n",
    "\n",
    "Feature primitives are the building blocks of Featuretools. They define individual computations that can be applied to raw datasets to create new features. There are two types of primitives supported by `featuretools`: i) **aggregation**, ii) **transform**. \n",
    "\n",
    "Aggregation primitives take related instances as an input and output a single value and they are applied across *multiple* entities which are described by a parent-child relationship in the entity set. Transform primitives each take one or more variables from a *single* entity as an input and output a new variable for that entity. Call `featuretools.list_primitives()` to see the full list of primitives, or check `featuretools_primitives.csv`. Some common aggregation primitives include `mean`, `median`, `count`, and `std`. Some common transformation primitives include `hour`, `day`, `month`, and `year`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Defining Custom Primitives\n",
    "\n",
    "We can also define our own primitives with `featuretools`. To define a primitive, a we must\n",
    "* specify the type of primitive Aggregation or Transform\n",
    "* define the input and output data types\n",
    "* write a function in python to do the calculation\n",
    "* annotate with attributes to constrain how it is applied\n",
    "\n",
    "Let's implement two examples:\n",
    "* Let's create a custom **transform** primitive and let's assume that we have a review (of the booking) of each user in text format in our data frames. Using [NRC Emotion Lexicon](http://web.stanford.edu/class/cs124/NRC-emotion-lexicon-wordlevel-alphabetized-v0.92.txt), we can count the number of occurrences words related *anger* for each user with the below function and calls. (NOTE: A few specific words are used here for reference.) The `AngerWordsCount()` class can now be used as any other transform primitive.\n",
    "* Let's implement a custom **aggregation** primitive, and this time let's apply this on the actual information we get from the session informations for each user. Specifically, we will be trying to compute the number of times each user has spent over 1 hour at a single session in all types (as a single feature) and two particular types of 'action_detail', viewing search results or updating their wishlists (as two features). The `CountLongDreamingSessions()` class can now be used as any other aggregation primitive.\n",
    "\n",
    "The second example also demonstrates the utilization of `interesting_values`, which allows us to specify to-be-focused values for a variable in an `Entity`, and the utilization of parameter `@where_primitives` aggregates on the to-be-focused values. Parameter `@agg_primitives`, on the other hand, ensures that we aggregate on all action detail types. Moreover, passing an empty list to this formerly mentioned parameter will **prevent** any type of aggregation to be performed with regards to this new primitive we defined. Finally, we are not *yet* calculating any actual feature matrix here as we will leave feature engineering to `featuretools` completely, which means that examples below will not be incorporated in the final data frame we pass to prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<Feature: gender>, <Feature: age>, <Feature: signup_method>, <Feature: language>, <Feature: affiliate_channel>, <Feature: affiliate_provider>, <Feature: first_affiliate_tracked>, <Feature: signup_app>, <Feature: first_device_type>, <Feature: first_browser>, <Feature: age_gender_bucket>, <Feature: signup_flow>, <Feature: COUNT_LONG_DREAMING_SESSIONS(sessions.secs_elapsed)>, <Feature: buckets.CA>, <Feature: buckets.DE>, <Feature: buckets.FR>, <Feature: buckets.GB>, <Feature: buckets.AU>, <Feature: buckets.NL>, <Feature: buckets.US>, <Feature: buckets.IT>, <Feature: buckets.PT>, <Feature: buckets.ES>, <Feature: COUNT_LONG_DREAMING_SESSIONS(sessions.secs_elapsed WHERE action_detail = wishlist_content_update)>, <Feature: COUNT_LONG_DREAMING_SESSIONS(sessions.secs_elapsed WHERE action_detail = view_search_results)>]\n"
     ]
    }
   ],
   "source": [
    "from featuretools.primitives import make_agg_primitive, make_trans_primitive\n",
    "from featuretools.variable_types import Text, Numeric, Categorical\n",
    "\n",
    "# Example (1)\n",
    "def anger_words_count(column, anger_words=['hate', 'furious', 'terrible', 'disgusting', 'shameful']):\n",
    "    counts = [np.sum([review.lower().count(anger_word) for anger_word in anger_words]) for review in column]\n",
    "    return counts\n",
    "\n",
    "AngerWordsCount = make_trans_primitive(function=anger_words_count,\n",
    "                                       input_types=[Text],\n",
    "                                       return_type=Numeric)\n",
    "\n",
    "# Example (2)\n",
    "def count_long_dreaming_sessions(seconds_elapsed):\n",
    "    df = pd.DataFrame({'seconds_elapsed': seconds_elapsed})\n",
    "    return len(df[df['seconds_elapsed'] >= 3600.0])\n",
    "                                    \n",
    "\n",
    "CountLongDreamingSessions = make_agg_primitive(function=count_long_dreaming_sessions,\n",
    "                                               input_types=[Numeric],\n",
    "                                               return_type=Numeric)\n",
    "\n",
    "es['sessions']['action_detail'].interesting_values = ['view_search_results', 'wishlist_content_update']\n",
    "\n",
    "feature_definitions = ft.dfs(entityset=es,\n",
    "                             target_entity=\"users\",\n",
    "                             agg_primitives=[CountLongDreamingSessions],\n",
    "                             trans_primitives=[],\n",
    "                             where_primitives=[CountLongDreamingSessions],\n",
    "                             max_depth=1,\n",
    "                             ignore_variables={'sessions': ['action', 'action_type', 'device_type']},\n",
    "                             features_only=True)\n",
    "print(feature_definitions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deep Feature Synthesis (DFS)\n",
    "\n",
    "**Deep Feature Synthesis (DFS)** is at the heart of `featuretools`, and is an automated method for performing feature engineering on relational and temporal data. A data scientist would write code to aggregate data for a customer, and apply different statistical functions resulting in features quantifying the customer’s behavior; DFS aims to overcome the limits of time and imagination of a data scientist to create many new features from multiple related tables of data. The idea, in essense, is to stack multiple feature primitives (both aggregations and transformations) to create new features. Then, these new features and the old original features are combined in a single data frame which will be used for training predictive models.This idea is based on the paper *Deep Feature Synthesis: Towards Automating Data Science Endeavors* by Max Kanter (creator of `featuretools` as well). \n",
    "\n",
    "Here, it may also be worthwile to mention that a method called **Evolutionary Feature Synthesis (EFS)** also exists in literature, but is more focused on generating features based on a single data frame rather than multiple, relational ones. The paper *Building Predictive Models via Feature Synthesis* by Ignacio Arnaldo et al. describes the algorithm behind in detail.\n",
    "\n",
    "#### Implementation Notes\n",
    "DFS can be performed with a simple call to `ft.dfs()`. First, pass parameter `@features_only=True` to only get feature definitions without the feature matrix (like we have done in the above section) and hence the time-consuming calculations themselves for debugging and printing the feature synthesis before hand. For example, if you see that `TIME_SINCE_FIRST` (aggregation primitive) feature hasn't been computed for the entity set at hand even though we specified it in the above list, this would mean that the entity containing the parent side of the relationship contains no Datetime variables. This increases the robustness of `dfs()`, we can list a large amount of primitives and try to generate the largest (hopefully most informative) feature matrix without getting runtime errors. Morever, we should note that aggregate primitives such as `avg_time_between`, `time_since_first`, and `time_since_last` can come in handy in any kind of problem where a track of sessions are kept for the users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Built 138 features\n",
      "Elapsed: 27:16 | Remaining: 00:00 | Progress: 100%|██████████| Calculated: 11/11 chunks\n",
      "[<Feature: gender>, <Feature: age>, <Feature: signup_method>, <Feature: language>, <Feature: affiliate_channel>, <Feature: affiliate_provider>, <Feature: first_affiliate_tracked>, <Feature: signup_app>, <Feature: first_device_type>, <Feature: first_browser>, <Feature: age_gender_bucket>, <Feature: signup_flow>, <Feature: LAST(sessions.action)>, <Feature: LAST(sessions.action_type)>, <Feature: LAST(sessions.action_detail)>, <Feature: LAST(sessions.device_type)>, <Feature: LAST(sessions.secs_elapsed)>, <Feature: NUM_UNIQUE(sessions.action)>, <Feature: NUM_UNIQUE(sessions.action_type)>, <Feature: NUM_UNIQUE(sessions.action_detail)>, <Feature: NUM_UNIQUE(sessions.device_type)>, <Feature: SKEW(sessions.secs_elapsed)>, <Feature: MIN(sessions.secs_elapsed)>, <Feature: MEAN(sessions.secs_elapsed)>, <Feature: STD(sessions.secs_elapsed)>, <Feature: MAX(sessions.secs_elapsed)>, <Feature: MEDIAN(sessions.secs_elapsed)>, <Feature: MODE(sessions.action)>, <Feature: MODE(sessions.action_type)>, <Feature: MODE(sessions.action_detail)>, <Feature: MODE(sessions.device_type)>, <Feature: HOUR(date_account_created)>, <Feature: HOUR(timestamp_first_active)>, <Feature: DAY(date_account_created)>, <Feature: DAY(timestamp_first_active)>, <Feature: WEEK(date_account_created)>, <Feature: WEEK(timestamp_first_active)>, <Feature: MONTH(date_account_created)>, <Feature: MONTH(timestamp_first_active)>, <Feature: YEAR(date_account_created)>, <Feature: YEAR(timestamp_first_active)>, <Feature: IS_WEEKEND(date_account_created)>, <Feature: IS_WEEKEND(timestamp_first_active)>, <Feature: CUM_SUM(age)>, <Feature: buckets.CA>, <Feature: buckets.DE>, <Feature: buckets.FR>, <Feature: buckets.GB>, <Feature: buckets.AU>, <Feature: buckets.NL>, <Feature: buckets.US>, <Feature: buckets.IT>, <Feature: buckets.PT>, <Feature: buckets.ES>, <Feature: LAST(sessions.CUM_SUM(secs_elapsed))>, <Feature: SKEW(sessions.CUM_SUM(secs_elapsed))>, <Feature: MIN(sessions.CUM_SUM(secs_elapsed))>, <Feature: MEAN(sessions.CUM_SUM(secs_elapsed))>, <Feature: STD(sessions.CUM_SUM(secs_elapsed))>, <Feature: MAX(sessions.CUM_SUM(secs_elapsed))>, <Feature: MEDIAN(sessions.CUM_SUM(secs_elapsed))>, <Feature: CUM_SUM(LAST(sessions.secs_elapsed))>, <Feature: CUM_SUM(NUM_UNIQUE(sessions.action))>, <Feature: CUM_SUM(NUM_UNIQUE(sessions.action_type))>, <Feature: CUM_SUM(NUM_UNIQUE(sessions.action_detail))>, <Feature: CUM_SUM(NUM_UNIQUE(sessions.device_type))>, <Feature: CUM_SUM(SKEW(sessions.secs_elapsed))>, <Feature: CUM_SUM(MIN(sessions.secs_elapsed))>, <Feature: CUM_SUM(MEAN(sessions.secs_elapsed))>, <Feature: CUM_SUM(STD(sessions.secs_elapsed))>, <Feature: CUM_SUM(MAX(sessions.secs_elapsed))>, <Feature: CUM_SUM(MEDIAN(sessions.secs_elapsed))>, <Feature: buckets.LAST(users.gender)>, <Feature: buckets.LAST(users.age)>, <Feature: buckets.LAST(users.signup_method)>, <Feature: buckets.LAST(users.language)>, <Feature: buckets.LAST(users.affiliate_channel)>, <Feature: buckets.LAST(users.affiliate_provider)>, <Feature: buckets.LAST(users.first_affiliate_tracked)>, <Feature: buckets.LAST(users.signup_app)>, <Feature: buckets.LAST(users.first_device_type)>, <Feature: buckets.LAST(users.first_browser)>, <Feature: buckets.LAST(users.signup_flow)>, <Feature: buckets.NUM_UNIQUE(users.gender)>, <Feature: buckets.NUM_UNIQUE(users.signup_method)>, <Feature: buckets.NUM_UNIQUE(users.language)>, <Feature: buckets.NUM_UNIQUE(users.affiliate_channel)>, <Feature: buckets.NUM_UNIQUE(users.affiliate_provider)>, <Feature: buckets.NUM_UNIQUE(users.first_affiliate_tracked)>, <Feature: buckets.NUM_UNIQUE(users.signup_app)>, <Feature: buckets.NUM_UNIQUE(users.first_device_type)>, <Feature: buckets.NUM_UNIQUE(users.first_browser)>, <Feature: buckets.NUM_UNIQUE(users.signup_flow)>, <Feature: buckets.SKEW(users.age)>, <Feature: buckets.MIN(users.age)>, <Feature: buckets.MEAN(users.age)>, <Feature: buckets.STD(users.age)>, <Feature: buckets.MAX(users.age)>, <Feature: buckets.MEDIAN(users.age)>, <Feature: buckets.MODE(users.gender)>, <Feature: buckets.MODE(users.signup_method)>, <Feature: buckets.MODE(users.language)>, <Feature: buckets.MODE(users.affiliate_channel)>, <Feature: buckets.MODE(users.affiliate_provider)>, <Feature: buckets.MODE(users.first_affiliate_tracked)>, <Feature: buckets.MODE(users.signup_app)>, <Feature: buckets.MODE(users.first_device_type)>, <Feature: buckets.MODE(users.first_browser)>, <Feature: buckets.MODE(users.signup_flow)>, <Feature: buckets.LAST(sessions.action)>, <Feature: buckets.LAST(sessions.action_type)>, <Feature: buckets.LAST(sessions.action_detail)>, <Feature: buckets.LAST(sessions.device_type)>, <Feature: buckets.LAST(sessions.secs_elapsed)>, <Feature: buckets.NUM_UNIQUE(sessions.action)>, <Feature: buckets.NUM_UNIQUE(sessions.action_type)>, <Feature: buckets.NUM_UNIQUE(sessions.action_detail)>, <Feature: buckets.NUM_UNIQUE(sessions.device_type)>, <Feature: buckets.SKEW(sessions.secs_elapsed)>, <Feature: buckets.MIN(sessions.secs_elapsed)>, <Feature: buckets.MEAN(sessions.secs_elapsed)>, <Feature: buckets.STD(sessions.secs_elapsed)>, <Feature: buckets.MAX(sessions.secs_elapsed)>, <Feature: buckets.MEDIAN(sessions.secs_elapsed)>, <Feature: buckets.MODE(sessions.action)>, <Feature: buckets.MODE(sessions.action_type)>, <Feature: buckets.MODE(sessions.action_detail)>, <Feature: buckets.MODE(sessions.device_type)>, <Feature: buckets.CUM_SUM(CA)>, <Feature: buckets.CUM_SUM(DE)>, <Feature: buckets.CUM_SUM(FR)>, <Feature: buckets.CUM_SUM(GB)>, <Feature: buckets.CUM_SUM(AU)>, <Feature: buckets.CUM_SUM(NL)>, <Feature: buckets.CUM_SUM(US)>, <Feature: buckets.CUM_SUM(IT)>, <Feature: buckets.CUM_SUM(PT)>, <Feature: buckets.CUM_SUM(ES)>]\n"
     ]
    }
   ],
   "source": [
    "# Specify aggregation primitives to be applied\n",
    "aggregations = ['last', 'num_unique', 'skew', 'min', 'mean', 'count', 'std', 'max', 'median', 'mode']\n",
    "# Specify transformation primitives to be applied\n",
    "transformations = ['hour', 'day', 'week', 'month', 'year', 'is_weekend', 'cum_sum']\n",
    "# Creating deep features and combining all primitive outputs & original features at a single training data frame\n",
    "feature_matrix, feature_definitions = ft.dfs(entityset=es,\n",
    "                                             target_entity='users',\n",
    "                                             agg_primitives=aggregations,\n",
    "                                             trans_primitives=transformations,\n",
    "                                             ignore_variables={'sessions': ['data_id'], 'users':['id']},\n",
    "                                             max_depth=2,\n",
    "                                             verbose=True)\n",
    "print(feature_definitions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Apply Equivalent Feature Extractions for Test Set\n",
    "The `calculate_feature_matrix()` method applies the same synthesized feature mappings to a different `EntitySet`, and in our case this set was seperated as test. The `encode_features()` allows us to easily one-hot encode the variables. However, we will deal with data encoding in the next notbeook. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Elapsed: 12:46 | Remaining: 00:00 | Progress: 100%|██████████| Calculated: 11/11 chunks\n"
     ]
    }
   ],
   "source": [
    "# Keep the extracted variables in their raw form (without any encoding) and do the same for test set\n",
    "feature_matrix_test = ft.calculate_feature_matrix(features=feature_definitions, entityset=es_test, verbose=True)\n",
    "\n",
    "# One-hot encode categorical variables and do the same for test set\n",
    "#feature_matrix_encoded, feature_encodings = ft.encode_features(feature_matrix, feature_definitions, verbose=True)\n",
    "#feature_matrix_encoded_test = ft.calculate_feature_matrix(features=feature_encodings, entityset=es_test, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Converting Unknowns to NaNs\n",
    "Although we spent some time imputing missing data values in the introductory notebook, we won't be imputing the missing values originating from the feature extractions we performed above. For example, users with no session information whatsoever will have a few columns that has the value NaN. Imputing for this would be not only inefficient, but it would also be illogical as the unknown values tell a very specific story here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_matrix = unknown_to_nan(df=feature_matrix)\n",
    "feature_matrix_test = unknown_to_nan(df=feature_matrix_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save Progress\n",
    "With this second notebook, our saved data is gone through automated feature engineering based on DFS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add back the response variables to sets & save progress\n",
    "feature_matrix['country_destination'] = targets\n",
    "feature_matrix.to_csv('(2)data_automated_ops/train_users.csv', index=None)\n",
    "feature_matrix_test.to_csv('(2)data_automated_ops/test_users.csv', index=None)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}