notebooks/introduction/2_data_handling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to choice-learn's data management"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "\n",
    "sys.path.append(\"../../\")\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## An introduction to ChoiceDataset\n",
    "\n",
    "Choice-Learn's ChoiceDataset aims at being able to handle large datasets, typically by limiting the usage of memory to store several times the same feature. Its structure is made to fit a choice modelling setup it is needed to estimate choice models weights. \n",
    "This notebook introduces how the package handles data. Here is a summary of the different points that will be adressed:\n",
    "\n",
    "## Summary\n",
    "\n",
    "- [**Introduction**](#an-introduction-to-choicedataset)\n",
    "    - [Example dataset: SwissMetro](#our-example-dataset-swissmetro)\n",
    "    - [The different types of data](#the-different-type-of-data)\n",
    "- [**ChoiceDataset's Instantiation from a single DataFrame**](#hands-on:-example-from-a-panda's-dataframe)\n",
    "    - [Wide format](#creating-a-choicedataset-from-a-wide-dataframe)\n",
    "    - [Long format](#creating-a-choicedataset-from-a-long-dataframe)\n",
    "- [**ChoiceDataset's Instantiation from separate objects**](#instantiation-from-different-objects)\n",
    "    - [Specifying each data type](#separating-data-types)\n",
    "    - [Stacking features](#stacking-features)\n",
    "- [**How to use the ChoiceDataset?**](#using-the-choicedataset-object)\n",
    "    - [Estimating choice models](#estimating-choice-models)\n",
    "    - [Slicing into batches](#slicing-and-batching)\n",
    "- [**Further optimizing RAM usage**](#more-advanced-use-the-featurestorage--ram-optimization)\n",
    "    - [The FeaturesStorage object](#featuresstorage-why-should-i-use-it)\n",
    "    - [Complete example of FeaturesStorage use](#example-on-the-swissmetro-dataset)\n",
    "    - [Specific example of the OneHotStorage](#specific-case-of-the-onehot-storage)\n",
    "- [**Additional Examples**](#additional-examples)\n",
    "    - [The ModeCanada Dataset](#the-modecanada-dataset)\n",
    "    - [A step-by-step FeaturesStorage use](#a-manual-example-of-featuresstorage)\n",
    "- [**List of Ready-To-Use datasets**](#ready-to-use-datasets)\n",
    "- [**References**](#references)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from choice_learn.data import ChoiceDataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Our example dataset: SwissMetro\n",
    "\n",
    "The SwissMetro[2] is a well-known dataset used to illustrate choice modelling. The dataset is provided with the Choice-Learn package and can be downloaded as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from choice_learn.datasets import load_swissmetro\n",
    "\n",
    "swissmetro_df = load_swissmetro(as_frame=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The SwissMetro is a collection of answers to a survey about mode transportation choice in Switzerland. Before building a costly new public transport line the government decided to better understand the needs of the future customers.\n",
    "A complete description of the dataset and the columns can be found [here](https://artefactory.github.io/choice-learn/references/data/references_choice_dataset/). We will use a subset of the information during the tutorial.\n",
    "\n",
    "<ins>**Available Modes:**</ins>\n",
    "- The current, existing train, 'TRAIN'\n",
    "- The potentially future SwissMetro, 'SM'\n",
    "- The customer car, 'CAR'\n",
    "\n",
    "<ins>**Columns:**</ins>\n",
    "- PURPOSE: What is the customer's travel purpose:\n",
    "- AGE: The customer's age category\n",
    "- mode_AV: Whether the mode is available (1) or not (0)\n",
    "- mode_TT: The mode travel time\n",
    "- mode_CO: The mode cost\n",
    "- CHOICE: the transport mode chosen by the customer\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kept_columns = [\"PURPOSE\", \"AGE\", \"ORIGIN\", \"CAR_AV\", \"TRAIN_AV\", \"SM_AV\", \"CAR_TT\",\n",
    "                \"TRAIN_TT\", \"SM_TT\", \"CAR_CO\", \"TRAIN_CO\", \"SM_CO\", \"CHOICE\"]\n",
    "swissmetro_df = swissmetro_df[kept_columns]\n",
    "swissmetro_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The different type of data\n",
    "\n",
    "We can split the columns into three distincts categories that are common to most choice modelling use-cases:\n",
    "\n",
    "- Choices - or outputs of our model: it's what we want to predict\n",
    "- Features - or inputs of our model\n",
    "- Availabilities - or the description of the set among which the customer chooses\n",
    "\n",
    "Going further, we have two types of features: the features describing the customer and the features describing the mean of transportation. Those are the four types of data that can be specified in a ChoiceDataset.\n",
    "\n",
    "\n",
    "**Vocabulary:**\n",
    "\n",
    "<ins>*Items*</ins> represent a product, an alternative that can be chosen by the customer at some point.\n",
    "\n",
    "\n",
    "Throughout Choice-Learn examples and code here is the naming of our four types of data:\n",
    "\n",
    "- **choices:** which item has been chosen among all availables\n",
    "\n",
    "- **shared_features_by_choice:** It represents all the features that might change from one choice to another and that are **common** to all items (e.g. day of week, customer features, etc...).\n",
    "  \n",
    "- **items_features_by_choice:** The features each of the available item for a choice (e.g. prices might change from one choice to another and are specific to each sold item).\n",
    "  \n",
    "- **available_items_by_choice:** For each choice it represents whether each item is proposed to the customer (1.) or not (0.).\n",
    "\n",
    "**Summary:**\n",
    "\n",
    "| index | feature  | typical shape  |  Example   | Taken Values |\n",
    "|---|---|---|---|---|\n",
    "| 1 | shared_features_by_choice | (n_choices, n_features) | customer age, day of week | float, int |\n",
    "| 2 | items_features_by_choice | (n_choices, n_items, n_items_features) | price | float, int |\n",
    "| 3 | available_items_by_choice | (n_choices, n_items) | | 1.(av) or 0. (not av.) |\n",
    "| 4 | choices | (n_choices,) | | int: index of chosen item |\n",
    "\n",
    "\n",
    "![DatasetDiagram](../../docs/illustrations/choice_learn_dataset.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hands-on: example from a pandas' DataFrame\n",
    "\n",
    "The easiest way create a ChoiceDataset is to use a pandas DataFrame.\n",
    "\n",
    "First, here is a small explanation about wide vs long format, in case you have never heard about it, from [Wikipedia](https://en.wikipedia.org/wiki/Wide_and_narrow_data).\n",
    "\n",
    "*Long (or narrow) Format:*  One column containing all the values and another column listing the context of the value\\\n",
    "*Wide Format:* Each different data variable in a separate column.\n",
    "\n",
    "<table>\n",
    "<tr><th>Example Long Format: </th><th>Example Wide Format:</th></tr>\n",
    "\n",
    "<tr><td>\n",
    "\n",
    "| choice id | item | price | availability | choice |\n",
    "|---|---|---|---|---|\n",
    "| 1 | A | 2.0 | 1 | 1 |\n",
    "| 1 | B | 6.0 | 1 | 0 |\n",
    "| 2 | A | 1.5 | 1 | 0 |\n",
    "| 2 | B | 5.5 | 1 | 1 |\n",
    "\n",
    "</td><td>\n",
    "\n",
    "| choice id | price_A | price_B | availability_A | availability_B | choice |\n",
    "|---|---|---|---|---|---|\n",
    "| 1 | 2.0 | 6.0 | 1 | 1 | A |\n",
    "| 2 | 1.5 | 5.5 | 1 | 1 | B |\n",
    "\n",
    "</td></tr> </table>\n",
    "\n",
    "Choice-Learn handles both formats, but slightly differently:\n",
    "- example for [wide](#creating-a-choicedataset-from-a-wide-dataframe) format\n",
    "- example for [long](#creating-a-choicedataset-from-a-long-dataframe) format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a ChoiceDataset from a wide DataFrame\n",
    "\n",
    "Our example dataframe on SwissMetro is on the wide format. Each row indicates a choice and each item has its specific features columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = ChoiceDataset.from_single_wide_df(\n",
    "    # The main DataFrame\n",
    "    df=swissmetro_df,\n",
    "    # The names of the items, will be used to find columns and organize them\n",
    "    items_id=[\"TRAIN\", \"SM\", \"CAR\"],\n",
    "\n",
    "    # The column containing the choices\n",
    "    choices_column=\"CHOICE\",\n",
    "    # How the choices are encoded: item_index means that the choice is the index of the item in the items_id list\n",
    "    choice_format=\"items_index\",\n",
    "\n",
    "    # Columns for shared_features_by_choice\n",
    "    shared_features_columns=[\"PURPOSE\", \"AGE\"],\n",
    "\n",
    "    # Columns for items_features_by_choice\n",
    "    # They will be reconstructed as item_id + delimiter + feature_suffix\n",
    "    items_features_suffixes=[\"CO\", \"TT\"],\n",
    "    # Same with availabilities\n",
    "    available_items_suffix=\"AV\",\n",
    "    delimiter=\"_\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Options**\n",
    "\n",
    "choice_format: \"item_index\" or \"item_id\"\n",
    "\n",
    "\n",
    "<table>\n",
    "<tr><th>\"item_index\" </th><th>\"item_id\"</th></tr>\n",
    "\n",
    "<tr><td>\n",
    "\n",
    "| choice_column|\n",
    "|---|\n",
    "| 0 |\n",
    "| 1 |\n",
    "| 0 |\n",
    "| 2 |\n",
    "\n",
    "</td><td>\n",
    "\n",
    "| choice_column|\n",
    "|---|\n",
    "| \"TRAIN\" |\n",
    "| \"SM\" |\n",
    "| \"TRAIN\" |\n",
    "| \"CAR\" |\n",
    "\n",
    "</td></tr> </table>\n",
    "\n",
    "items_features_by_choice and available_items_by_choice:\n",
    "\n",
    "It is possible to precise:\n",
    "- Suffixes: in this case the column used will be \"item_id\" + \"delimiter\" + \"suffix\"\n",
    "- Prefixes: in this case the column used will be \"prefix\" + \"delimiter\" + \"item_id\"\n",
    "- Columns: each item's features in list. In this case it is you duty to ensure coherence in terms of items and features orders. For our example it would be:\n",
    "\n",
    "    ```python\n",
    "    items_features_by_choice_columns=[[\"TRAIN_CO\", \"TRAIN_TT\"], [\"SM_CO\", \"SM_TT\"], [\"CAR_CO\", \"CAR_TT\"]],\n",
    "    available_items_by_choice_columns=[\"TRAIN_AV\", \"SM_AV\", \"CAR_AV\"],\n",
    "    ```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a ChoiceDataset from a long DataFrame\n",
    "The long format is also commonly used in which each row represents an alternative. One of its benefits is represent unavailability through missing rows - taking litteraly zero memory space. On the contrary the 'shared_features' such as customer features must be duplicated on each row.\\\n",
    "The ChoiceDataset object can be instantiated from a long DF. It will infer the availabilities from existing/missing rows, if it is not specified.\\\n",
    "It is needed to precise:\n",
    "- columns representing the features ('shared_features_columns' and 'items_features_columns')\n",
    "- the column in which the choice is given and how it is formatted ('choices_columns' and 'choice_format')\n",
    "- which column can identify the items ('items_id_column')\n",
    "- which column can identify all the rows corresponding to the same choice ('choices_id_column')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Transformation of our dataset on the long format\n",
    "long_df = load_swissmetro(preprocessing=\"long_format\")\n",
    "long_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example of the long format instantiation\n",
    "dataset = ChoiceDataset.from_single_long_df(\n",
    "    df=long_df,\n",
    "    items_id_column=\"item_id\",\n",
    "    choices_id_column=\"choice_id\",\n",
    "\n",
    "    shared_features_columns=[\"PURPOSE\", \"AGE\"],\n",
    "    items_features_columns=[\"TT\", \"CO\"],\n",
    "\n",
    "    choices_column=\"CHOICE\",\n",
    "    choice_format=\"one_zero\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Options**\n",
    "\n",
    "choice_format: \"one_zero\" or \"item_id\"\n",
    "\n",
    "\n",
    "<table>\n",
    "<tr><th>\"one_zero\" </th><th>\"item_id\"</th></tr>\n",
    "\n",
    "<tr><td>\n",
    "\n",
    "| choice_id_column | item_id_column | choice_column|\n",
    "|---|---|---|\n",
    "| 1 | \"CAR\" | 0 |\n",
    "| 1 | \"SM\" | 1 |\n",
    "| 2 | \"CAR\" | 1 |\n",
    "| 2 | \"SM\" | 0 |\n",
    "\n",
    "</td><td>\n",
    "\n",
    "| choice_id_column | item_id_column | choice_column|\n",
    "|---|---|---|\n",
    "| 1 | \"CAR\" | \"SM\" |\n",
    "| 1 | \"SM\" | \"SM\" |\n",
    "| 2 | \"CAR\" | \"CAR\" |\n",
    "| 2 | \"SM\" | \"CAR\" |\n",
    "\n",
    "</td></tr> </table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instantiation from different objects\n",
    "\n",
    "For RAM optimization purposes or just because of the format of the data source, it might happen that a dataset is split into separate files. You can instantiate a ChoiceDataset keeping this structure, saving time to concatenate everything.\\\n",
    "You can work either with pandas.DataFrames or numpy.ndarrays.\n",
    "\n",
    "### Separating data types\n",
    "The four distinct data types: choices, shared_features_by_choice, items_features_by_choice, available_items_by_choice can be manually given to the ChoiceDataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using pandas.DataFrames\n",
    "dataset = ChoiceDataset(\n",
    "    choices=swissmetro_df[\"CHOICE\"],\n",
    "    shared_features_by_choice=swissmetro_df[[\"PURPOSE\", \"AGE\"]],\n",
    "    items_features_by_choice=long_df[[\"choice_id\", \"item_id\", \"CO\", \"TT\"]]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that if you pass items_features_by_choice as a pandas.DataFrame, it needs to be in the long format and with the columns 'choice_id' and 'item_id'. They will be used to get the features in the right order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using numpy.ndarrays\n",
    "# Be aware of items_features_by_choices shape that is (n_choices, n_items, n_features)\n",
    "\n",
    "items_features_by_choice = np.stack([swissmetro_df[[\"TRAIN_CO\", \"TRAIN_TT\"]].to_numpy(),\n",
    "                                     swissmetro_df[[\"SM_CO\", \"SM_TT\"]].to_numpy(),\n",
    "                                     swissmetro_df[[\"CAR_CO\", \"CAR_TT\"]].to_numpy()],\n",
    "                                     axis=1)\n",
    "shared_features_by_choice = swissmetro_df[[\"PURPOSE\", \"AGE\"]].to_numpy()\n",
    "available_items_by_choice = swissmetro_df[[\"TRAIN_AV\", \"SM_AV\", \"CAR_AV\"]].to_numpy()\n",
    "\n",
    "print(\"The data shapes are:\")\n",
    "print(f\"choices: {swissmetro_df['CHOICE'].shape}\")\n",
    "print(f\"shared_features_by_choice: {shared_features_by_choice.shape}\")\n",
    "print(f\"items_features_by_choice: {items_features_by_choice.shape}\")\n",
    "print(f\"available_items_by_choice: {available_items_by_choice.shape}\")\n",
    "\n",
    "dataset = ChoiceDataset(\n",
    "    choices=swissmetro_df[\"CHOICE\"].to_numpy(),\n",
    "    shared_features_by_choice=shared_features_by_choice,\n",
    "    items_features_by_choice=items_features_by_choice,\n",
    "    available_items_by_choice=available_items_by_choice,\n",
    "\n",
    "    # Features names can optionally be provided\n",
    "    # the structure of data and names must match\n",
    "    shared_features_by_choice_names=[\"PURPOSE\", \"AGE\"],\n",
    "    items_features_by_choice_names=[\"CO\", \"TT\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stacking Features\n",
    "\n",
    "It is allowed to specify more than one features object by wrapping them in a tuple. This structure is kept inside the ChoiceDataset object as well as with the slicing into batches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using pandas.DataFrames - Similar with np.ndarrays\n",
    "dataset = ChoiceDataset(\n",
    "    choices=swissmetro_df[\"CHOICE\"],\n",
    "    shared_features_by_choice=(swissmetro_df[[\"PURPOSE\"]], swissmetro_df[[\"AGE\"]]),\n",
    "    items_features_by_choice=(long_df[[\"choice_id\", \"item_id\", \"CO\"]],\n",
    "                              long_df[[\"choice_id\", \"item_id\", \"TT\"]]),\n",
    "    available_items_by_choice=swissmetro_df[[\"TRAIN_AV\", \"SM_AV\", \"CAR_AV\"]],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other examples are provided [here](data/dataset_creation.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the ChoiceDataset object\n",
    "\n",
    "### Estimating choice models\n",
    "\n",
    "With your ChoiceDataset instantiated, it can be used as is to fit choice models. An illustration can be found in the conditional MNL introduction [notebook](./3_model_clogit.ipynb).\n",
    "\n",
    "### Slicing and batching\n",
    "\n",
    "ChoiceDatasets are indexed by choice, meaning that accessing the i-th index corresponds to the i-th choice. Differently said it is the i-th value of the object given as 'choices' in the ChoiceDataset instantiation.\n",
    "\n",
    "A ChoiceDataset can be sliced commonly using the [.] Python method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_dataset = dataset[[0, 2, 4]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "sub_dataset will be a ChoiceDataset containing only the 0th, 2nd and 4th choice of dataset. The other data (shared_featurs_by_choice, items_features_by_choice and available_items_by_choice) are also kept and sliced accordingly.\n",
    "\n",
    "In order to only get a chunk of data, it is possible to use .batch[.]. It will return the different data types sliced along choices in a raw np.ndarray format. Use .iter_batch() to iterate over all data in the ChoiceDataset by setting the batch_size argument to control the length of each chunk.\n",
    "\n",
    "Also note that ```batch_size=-1```returns the whole dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.choices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch = dataset.batch[[0, 2, 4]]\n",
    "print(\"\")\n",
    "for batch in dataset.iter_batch(batch_size=1024):\n",
    "    print(\"Num choices:\", len(batch[-1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More Advanced use: the FeatureStorage & RAM optimization\n",
    "\n",
    "In many use-cases we will see features or group of features values being repeated over the dataset. For example if one customer comes several times, its features will be repeated. With One-Hot representations, it can create memory-heavy repetitions.\\\n",
    "Choice-Learn introduces FeaturesStorage and FeaturesByIds in order to limit the memory usage before accessing a batch of data.\n",
    "\n",
    "### FeaturesStorage, why should I use it ?\n",
    "If you are not using a large dataset with many features you can pass this part. FeaturesStorage are here if you want to further optimize your memory consumption and if you take some time to understand it.\\\n",
    "It is mainly built to work well with ChoiceDataset, but here is a small introduction on how it works.\n",
    "\n",
    "![StorageDiagram](../../docs/illustrations/choice_learn_features_storage.png)\n",
    "\n",
    "### Example on the SwissMetro dataset\n",
    "\n",
    "**/!\\ Disclaimer**\n",
    "For the sake of the example, some features will be introduced and created. They are totally made up and do not exist in the original - and true - version of the SwissMetro Dataset.\n",
    "\n",
    "Let's consider the survey that happened in the three cantons: Geneva, Berne and Zürich. Now we want to integrate localization features.\n",
    "\n",
    "| Canton | Inhabitants (M) | Surface (km^2) | Origin Code |\n",
    "|---|---|---|---|\n",
    "| Geneva | 0.5 | 282 | 25 |\n",
    "| Zürich | 1.5 | 1729 | 1 |\n",
    "| Berne | 1.0 | 5959 | 2 |\n",
    "\n",
    "A naive way to integreate those features is to add them as 'shared_features_by_choice'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Filtering cantons\n",
    "swiss_df = swissmetro_df.loc[swissmetro_df.ORIGIN.isin([1, 2, 25])]\n",
    "\n",
    "# Adding features\n",
    "swiss_df.loc[:, \"CANTON_SURFACE\"] = swiss_df.apply(lambda row: {1: 1729, 2: 5959, 25: 282}[row.ORIGIN], axis=1)\n",
    "swiss_df[\"CANTON_INHAB\"] = swiss_df.apply(lambda row: {1: 1.5, 2: 1.0, 25: 0.5}[row.ORIGIN], axis=1)\n",
    "\n",
    "dataset = ChoiceDataset.from_single_wide_df(\n",
    "    df=swiss_df,\n",
    "    items_id=[\"TRAIN\", \"SM\", \"CAR\"],\n",
    "\n",
    "    choices_column=\"CHOICE\",\n",
    "    choice_format=\"items_index\",\n",
    "\n",
    "    # The new features are added here compared to example above\n",
    "    shared_features_columns=[\"PURPOSE\", \"AGE\", \"CANTON_SURFACE\", \"CANTON_INHAB\"],\n",
    "    items_features_suffixes=[\"CO\", \"TT\"],\n",
    "    available_items_suffix=\"AV\",\n",
    "    delimiter=\"_\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main caveat is that the same features are repeated over the rows of the dataset. If we consider hundreds of stores on several millions - or billions - of choices, it would become... unreasonable!\\\n",
    "One idea is to regroup the features behind an ID (the canton id for example) and to reconstruct the features only in batches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from choice_learn.data import FeaturesStorage\n",
    "\n",
    "origin_canton_features = {1: [1.5, 1729], 2: [1.0, 5959], 25: [0.5, 282]}\n",
    "canton_storage = FeaturesStorage(values=origin_canton_features, name=\"ORIGIN\") # Remark that the name matches the ID column name in the DF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The FeaturesStorage is basically a Python dictionnary with a wrap-up to easily get batches of data.\\\n",
    "You can ask for a sequence of features with .batch. It works with the keys of our dictionnary that can be int, float, str, etc..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Retrieving features of canton id 1:\")\n",
    "print(canton_storage.batch[1])\n",
    "print(\"Retrieving a batch of features:\")\n",
    "print(canton_storage.batch[[1, 25, 1]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The FeaturesStorage is handy for its transparent use with ChoiceDataset. For it to work well it is needed to:\n",
    "- specify a FeaturesStorage name that matches the feature names given to the ChoiceDataset\n",
    "- match FeaturesStorage ids with the sequence (types and values)\n",
    "- specify the FeaturesStorage objects listed with the features_by_ids argument\n",
    "\n",
    "In our case we call our FeaturesStorage \"canton_storage\", the ids are now strings, let's make the sequence match:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "storage_dataset = ChoiceDataset(choices=swiss_df[\"CHOICE\"],\n",
    "                                items_features_by_choice=np.stack([swiss_df[[\"TRAIN_CO\", \"TRAIN_TT\"]].to_numpy(),\n",
    "                                                                   swiss_df[[\"SM_CO\", \"SM_TT\"]].to_numpy(),\n",
    "                                                                   swiss_df[[\"CAR_CO\", \"CAR_TT\"]].to_numpy()],\n",
    "                                                                   axis=1),\n",
    "                                shared_features_by_choice=swiss_df[[\"AGE\", \"PURPOSE\", \"ORIGIN\"]].to_numpy(),\n",
    "                                features_by_ids=[canton_storage],\n",
    "                                items_features_by_choice_names=[\"CO\", \"TT\"],\n",
    "                                shared_features_by_choice_names=[\"AGE\", \"PURPOSE\", \"ORIGIN\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at a batch of data, here is how it looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# batching the dataset as before\n",
    "batch = storage_dataset.batch[[1, 2, 3]]\n",
    "print(\"Batch of shared_features_by_choice:\", batch[0])\n",
    "print(\"Batch of choices:\", batch[3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The features stored in the FeaturesStorage have been stacked with the usual 'shared_features_by_choice' !"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Specific case of the OneHot Storage\n",
    "Manually looking for canton features is quiet time consuming. Another idea is to represent each canton by a unique one-hot vector.\n",
    "A recurring usecase is the use of **OneHot** representation of features. The OneHotStorage is built specifically for one-hot encoded features and further improves memory consumption. The storage is to be used the same way as FeaturesStorage, but behind will only keep the index of the one of each element and will consitute the one-hot vector only when needed.\\\n",
    "In order terms it stores a sparse version of the vectors and returns a dense representation when batched."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from choice_learn.data import OneHotStorage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "storage = OneHotStorage(ids=swissmetro_df.ORIGIN.unique())\n",
    "\n",
    "print(\"RAM storage of the OneHotStore:\", storage.storage)\n",
    "# When indexing with .batch, we can access the one-hot encoding of the element using its id\n",
    "print(\"One-hot vector batch: storage.batch[2]\", storage.batch[2])\n",
    "print(\"One-hot vector batch: storage.batch[[5, 20, 18, 23, 25, 15,  5, 20]]\")\n",
    "print(storage.batch[[5, 20, 18, 23, 25, 15,  5, 20]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other examples of features_by_ids usage can be found [here](data/features_byID_examples.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The ModeCanada dataset\n",
    "\n",
    "We will use the ModeCanada [1] dataset for this example. The dataset is originally in the long format. It is provided with the choice-learn package and can loaded as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from choice_learn.datasets import load_modecanada\n",
    "\n",
    "canada_transport_df = load_modecanada(as_frame=True)\n",
    "canada_transport_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An extensive description of the dataset can be found [here](https://www.ssc.wisc.edu/~bhansen/econometrics/Koppelman_description.pdf). An extract indicates:\n",
    "\n",
    "\"The dataset was assembled in 1989 by VIA Rail (the Canadian national rail carrier) to estimate the demand for high-speed rail in the Toronto-Montreal corridor. The main information source was a Passenger Review administered to business travelers augmented by information about each trip. The observations consist of a choice between four modes of transportation (train, air, bus, car) with information about the travel mode and about the passenger. The posted dataset has been balanced to only include cases where all four travel modes are recorded. The file contains 11,116 observations on 2779 individuals. \"\n",
    "\n",
    "Alright ! If we go back to our dataframe, we can see the following columns:\n",
    "\n",
    "case: an ID of the traveler\n",
    "alt: the alternative concerned by the row\n",
    "choice: 1 if the alternative was chosen, 0 otherwise\n",
    "dist: trip distance\n",
    "cost: trip cost\n",
    "ivt: travel time in-vehicule (minutes)\n",
    "ovt: travel time out-vehicule (minutes)\n",
    "income: housold income of traveler ($)\n",
    "urban: 1 if origin or destination is a large city\n",
    "noalt: the number of alternative among which the traveler had to chose\n",
    "freq: the frequence of the alternative (0 for car) (e.g. how many train by hour)\n",
    "Following our specification, we can see that one case corresponds to one customer thus one choice. In our choice-learn language it corresponds to \"one context\": a set of available alternatives and their features/specificites resulting in one choice. Let's regroup our features:\n",
    "\n",
    "**choices:** Easy ! It is the alternative whenever the value is one.\n",
    "\n",
    "**shared_features_by_choice:** The income, urban and distance (also noalt which is not really a feature) features are the same for all the alternatives within a single choice. They are all constant with respect to (case=traveler_ID).\n",
    "\n",
    "**items_features_by_choice:** Ivt, Ovt, cost and freq depends on and describe each of the alternative.\n",
    "\n",
    "**available_items_by_choice:** It in not directly indicated, however it can be easily deduced. Whenever an alternative is not available, it is not precised for its case. For example for the case=1, our first choice, only train and car are given as alternatives, meaning that air and bus could not be chosen/were not available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = ChoiceDataset.from_single_long_df(\n",
    "    df=canada_transport_df,\n",
    "    choices_column=\"choice\",\n",
    "    items_id_column=\"alt\",\n",
    "    choices_id_column=\"case\",\n",
    "    shared_features_columns=[\"income\", \"urban\", \"dist\"],\n",
    "    items_features_columns=[\"cost\", \"freq\", \"ovt\", \"ivt\"],\n",
    "    choice_format=\"one_zero\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example the 'choice_format' is \"one_zero\" while it was \"item_id\" in our previous SwissMetro example. As a short memento it specifies how the chosen alternative is precised: with ones (chosen) and zeros (not chosen) or directlu with the item_id of the chosen item.\n",
    "\n",
    "\n",
    "\n",
    "<table>\n",
    "<tr><th>\"one_zero\" </th><th>\"item_id\"</th></tr>\n",
    "\n",
    "<tr><td>\n",
    "\n",
    "\n",
    "| | case | alt | choice | dist | cost | ivt | ovt | freq | \tincome |\n",
    "|---|---|---|---|---|---|---|---|---|---|\n",
    "| 1 | 1 | train | 0 | 83 | 28.25 | 50 | 66 | 4 | 45 |\n",
    "| 2 | 1 | car | 1 | 83 | 15.77 | 61 | 0 | 0 | 45 |\n",
    "| 3 | 2 | train | 0 | 83 | 28.25 | 50 | 66 | 4 | 25 |\n",
    "| 4 | 2 | car | 1 | 83 | 15.77 | 61 | 0 | 0 | 25 |\n",
    "| 5 | 3 | train | 0 | 83 | 28.25 | 50 | 66 | 4 | 70 |\n",
    "\n",
    "</td><td>\n",
    "\n",
    "\n",
    "| | case | alt | choice | dist | cost | ivt | ovt | freq | \tincome |\n",
    "|---|---|---|---|---|---|---|---|---|---|\n",
    "| 1 | 1 | train | car | 83 | 28.25 | 50 | 66 | 4 | 45 |\n",
    "| 2 | 1 | car | car | 83 | 15.77 | 61 | 0 | 0 | 45 |\n",
    "| 3 | 2 | train | car | 83 | 28.25 | 50 | 66 | 4 | 25 |\n",
    "| 4 | 2 | car | car | 83 | 15.77 | 61 | 0 | 0 | 25 |\n",
    "| 5 | 3 | train | car | 83 | 28.25 | 50 | 66 | 4 | 70 |\n",
    "\n",
    "</td></tr> </table>\n",
    "\n",
    "In the first 5 examples, the chosen transportation is always the car."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it !"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A manual example of FeaturesStorage\n",
    "\n",
    "Let's consider a case where we consider three supermarkets: \n",
    "- supermarket_1 with surface of 100 and 250 average nb of customers\n",
    "- supermarket_2 with surface of 150 and 500 average nb of customers\n",
    "- supermarket_3 with surface of 80 and 100 average nb of customers \n",
    "\n",
    "In each store, we have 4 available products for which we have little information. For the example'sake, let's consider the following utility:\n",
    "$$U(i) = u_i + \\beta_1 \\cdot S_s + \\beta_2 \\cdot C_s$$\n",
    "With $S_s$ the surface of the store and $C_s$ its average number of customers.\n",
    "\n",
    "We want to estimate the base utilities $u_i$ and the two coefficients: $\\beta_1$ and $\\beta_2$.\n",
    "\n",
    "Let's start with creating a ChoiceDataset without the FeaturesStorage:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's consider a case where we consider three supermarkets: \n",
    "- supermarket_1 with surface of 100 and 250 average nb of customers\n",
    "- supermarket_2 with surface of 150 and 500 average nb of customers\n",
    "- supermarket_3 with surface of 80 and 100 average nb of customers \n",
    "\n",
    "In each store, we have 4 available products for which we have little information. For the example'sake, let's consider the following utility:\n",
    "$$U(i) = u_i + \\beta_1 \\cdot S_s + \\beta_2 \\cdot C_s$$\n",
    "With $S_s$ the surface of the store and $C_s$ its average number of customers.\n",
    "\n",
    "We want to estimate the base utilities $u_i$ and the two coefficients: $\\beta_1$ and $\\beta_2$.\n",
    "\n",
    "Let's start with creating a ChoiceDataset without the FeaturesStorage:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "keep_output": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Usual Supermakerket Features Shape: (18, 2)\n"
     ]
    }
   ],
   "source": [
    "# Here are our choices:\n",
    "choices = [0, 1, 2, 0, 2, 1, 1, 0, 2, 1, 2, 0, 2, 0, 1, 2, 1, 0]\n",
    "supermarket_features = [[100, 250], [150, 500], [80, 100]]\n",
    "# Now our store sequence of supermarkets is:\n",
    "supermarkets_sequence = [1, 1, 2, 3, 2, 1, 2, 1, 1, 2, 3, 2, 1, 2, 2, 3, 1, 2]\n",
    "\n",
    "# The usual way to store the features would be to create the contexts_features array that contains\n",
    "# the right features:\n",
    "usual_supermarket_features = np.array([supermarket_features[supermarket_id - 1] for supermarket_id in supermarkets_sequence])\n",
    "print(\"Usual Supermakerket Features Shape:\", usual_supermarket_features.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Supermarket features being repeated several times, it's a great opportunity to use a FeaturesStorage !\\\n",
    "Let's see how to use strings as IDs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features_dict = {f\"supermarket_{i+1}\": supermarket_features[i] for i in range(3)}\n",
    "storage = FeaturesStorage(values=features_dict, name=\"supermarket_features\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "keep_output": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Retrieving features of first supermarket:\n",
      "[100 250]\n",
      "Retrieving a batch of features:\n",
      "[[100 250]\n",
      " [150 500]\n",
      " [100 250]]\n"
     ]
    }
   ],
   "source": [
    "print(\"Retrieving features of first supermarket:\")\n",
    "print(storage.batch[\"supermarket_1\"])\n",
    "print(\"Retrieving a batch of features:\")\n",
    "print(storage.batch[[\"supermarket_1\", \"supermarket_2\", \"supermarket_1\"]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ins>Reminder:</ins>\n",
    "\n",
    "It is needed to:\n",
    "- specify a FeaturesStorage name\n",
    "- match FeaturesStorage ids with the sequence\n",
    "\n",
    "In our case we call our FeaturesStorage \"supermarket_features\", the ids are now strings, let's make the sequence match:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "str_supermarkets_sequence = [[f\"supermarket_{i}\"] for i in supermarkets_sequence]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we can create our ChoiceDataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "storage_dataset = ChoiceDataset(choices=choices,\n",
    "                                shared_features_by_choice=str_supermarkets_sequence,\n",
    "                                shared_features_by_choice_names=[\"supermarket_features\"],\n",
    "                                available_items_by_choice=np.ones((len(choices), 3)),\n",
    "                                features_by_ids=[storage],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now let's see how batches work:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "keep_output": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Batch Shared Items Features: [100 250]\n",
      "Batch Items Features: None\n",
      "Batch Choice: 0\n",
      "%-------------------------%\n",
      "Batch Shared Items Features: [[100 250]\n",
      " [150 500]\n",
      " [ 80 100]]\n",
      "Batch Items Features: None\n",
      "Batch Choice: [1 2 0]\n",
      "%-------------------------%\n",
      "Batch Shared Items Features: [[100 250]\n",
      " [100 250]\n",
      " [100 250]]\n",
      "Batch Items Features: None\n",
      "Batch Choice: [0 1 1]\n"
     ]
    }
   ],
   "source": [
    "batch = storage_dataset.batch[0]\n",
    "print(\"Batch Shared Items Features:\", batch[0])\n",
    "print(\"Batch Items Features:\", batch[1])\n",
    "print(\"Batch Choice:\", batch[3])\n",
    "print(\"%-------------------------%\")\n",
    "batch = storage_dataset.batch[[1, 2, 3]]\n",
    "print(\"Batch Shared Items Features:\", batch[0])\n",
    "print(\"Batch Items Features:\", batch[1])\n",
    "print(\"Batch Choice:\", batch[3])\n",
    "print(\"%-------------------------%\")\n",
    "batch = storage_dataset.batch[[0, 1, 5]]\n",
    "print(\"Batch Shared Items Features:\", batch[0])\n",
    "print(\"Batch Items Features:\", batch[1])\n",
    "print(\"Batch Choice:\", batch[3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Everything is mapped as needed. And the great thing is that you can easily mix ''classical'' features with FeaturesStorages.\\\n",
    "Let's add a 'is_week_end' feature to our problem that will also be stored as a contexts_features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "keep_output": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>supermarket_features</th>\n",
       "      <th>is_week_end</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>supermarket_1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>supermarket_1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>supermarket_2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>supermarket_3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>supermarket_2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  supermarket_features  is_week_end\n",
       "0        supermarket_1            0\n",
       "1        supermarket_1            0\n",
       "2        supermarket_2            0\n",
       "3        supermarket_3            1\n",
       "4        supermarket_2            1"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "shared_features = pd.DataFrame({\"supermarket_features\": np.array(str_supermarkets_sequence).squeeze(),\n",
    "\"is_week_end\": [0, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0]})\n",
    "shared_features.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creation of the ChoiceDataset\n",
    "storage_dataset = ChoiceDataset(choices=choices,\n",
    "                                shared_features_by_choice=shared_features,\n",
    "                                available_items_by_choice=np.ones((len(choices), 3)),\n",
    "                                features_by_ids=[storage],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "keep_output": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Batch Shared Items Features: [[100 250]\n",
      " [150 500]\n",
      " [ 80 100]]\n",
      "Batch Items Features: None\n",
      "Batch Choice: [1 2 0]\n"
     ]
    }
   ],
   "source": [
    "# And now it's ready\n",
    "batch = storage_dataset.batch[[1, 2, 3]]\n",
    "print(\"Batch Shared Items Features:\", batch[0])\n",
    "print(\"Batch Items Features:\", batch[1])\n",
    "print(\"Batch Choice:\", batch[3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note that:**\n",
    "- We use strings as ids for the example, however we recommend to use integers.\n",
    "- FeaturesStorage can be instantiated from dict, np.ndarray, list, pandas.DataFrame, etc...\n",
    "- More in-depth examples and explanations can be found [here](./features_byID_example.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ready-to-use datasets\n",
    "A few well-known open source datasets are directly integrated and the package and can be downloaded in one line:\n",
    "- SwissMetro from Bierlaire et al (2001) [2]\n",
    "- ModeCanada from Koppleman et al. (1993) [1]\n",
    "- The Train dataset from Ben Akiva et al. (1993) [4]\n",
    "- The Heating & Electricity datasets from Kenneth Train [3]\n",
    "- The TaFeng dataset from Kaggle [5]\n",
    "\n",
    "If you feel like another open-source dataset could be included, reach out !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from choice_learn.datasets import (load_swissmetro,\n",
    "                                   load_modecanada,\n",
    "                                   load_train,\n",
    "                                   load_heating,\n",
    "                                   load_electricity,\n",
    "                                   load_tafeng\n",
    "                                   )\n",
    "\n",
    "canada_choice_dataset = load_modecanada()\n",
    "swissmetro_choice_dataset = load_swissmetro()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The datasets can also be downloaded as dataframes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "keep_output": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>GROUP</th>\n",
       "      <th>SURVEY</th>\n",
       "      <th>SP</th>\n",
       "      <th>ID</th>\n",
       "      <th>PURPOSE</th>\n",
       "      <th>FIRST</th>\n",
       "      <th>TICKET</th>\n",
       "      <th>WHO</th>\n",
       "      <th>LUGGAGE</th>\n",
       "      <th>AGE</th>\n",
       "      <th>...</th>\n",
       "      <th>TRAIN_CO</th>\n",
       "      <th>TRAIN_HE</th>\n",
       "      <th>SM_TT</th>\n",
       "      <th>SM_CO</th>\n",
       "      <th>SM_HE</th>\n",
       "      <th>SM_SEATS</th>\n",
       "      <th>CAR_TT</th>\n",
       "      <th>CAR_CO</th>\n",
       "      <th>CHOICE</th>\n",
       "      <th>CAR_HE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>48</td>\n",
       "      <td>120</td>\n",
       "      <td>63</td>\n",
       "      <td>52</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>117</td>\n",
       "      <td>65</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>48</td>\n",
       "      <td>30</td>\n",
       "      <td>60</td>\n",
       "      <td>49</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>117</td>\n",
       "      <td>84</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>48</td>\n",
       "      <td>60</td>\n",
       "      <td>67</td>\n",
       "      <td>58</td>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "      <td>117</td>\n",
       "      <td>52</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>40</td>\n",
       "      <td>30</td>\n",
       "      <td>63</td>\n",
       "      <td>52</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>72</td>\n",
       "      <td>52</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>36</td>\n",
       "      <td>60</td>\n",
       "      <td>63</td>\n",
       "      <td>42</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>90</td>\n",
       "      <td>84</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 29 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   GROUP  SURVEY  SP  ID  PURPOSE  FIRST  TICKET  WHO  LUGGAGE  AGE  ...  \\\n",
       "0      2       0   1   1        1      0       1    1        0    3  ...   \n",
       "1      2       0   1   1        1      0       1    1        0    3  ...   \n",
       "2      2       0   1   1        1      0       1    1        0    3  ...   \n",
       "3      2       0   1   1        1      0       1    1        0    3  ...   \n",
       "4      2       0   1   1        1      0       1    1        0    3  ...   \n",
       "\n",
       "   TRAIN_CO  TRAIN_HE  SM_TT  SM_CO  SM_HE  SM_SEATS  CAR_TT  CAR_CO  CHOICE  \\\n",
       "0        48       120     63     52     20         0     117      65       1   \n",
       "1        48        30     60     49     10         0     117      84       1   \n",
       "2        48        60     67     58     30         0     117      52       1   \n",
       "3        40        30     63     52     20         0      72      52       1   \n",
       "4        36        60     63     42     20         0      90      84       1   \n",
       "\n",
       "   CAR_HE  \n",
       "0     0.0  \n",
       "1     0.0  \n",
       "2     0.0  \n",
       "3     0.0  \n",
       "4     0.0  \n",
       "\n",
       "[5 rows x 29 columns]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "swissmetro_df = load_swissmetro(as_frame=True)\n",
    "swissmetro_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References\n",
    "[1] Koppelman et al. (1993), *Application and Interpretation of Nested Logit Models of Intercity Mode Choice*\\\n",
    "[2] Bierlaire, M., Axhausen, K. and Abay, G. (2001), *The Acceptance of Modal Innovation: The Case of SwissMetro*\\\n",
    "[3] Train, K.E. (2003) *Discrete Choice Methods with Simulation.* Cambridge University Press.\\\n",
    "[4] Ben-Akiva M.; Bolduc D.; Bradley M. (1993) *Estimation of Travel Choice Models with Randomly Distributed Values of Time*\\\n",
    "[5] The Ta Feng Grocery dataset on [Kaggle](https://www.kaggle.com/datasets/chiranjivdas09/ta-feng-grocery-dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}