tutorials/usage/tutorial02_imputation.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# User Guide Tutorial 02: Preprocessing › Imputation\n",
    "\n",
    "This tutorial shows how to use TemporAI `preprocessing.imputation` plugins."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Skip the below cell if you are not on Google Colab / already have TemporAI installed:*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install temporai\n",
    "\n",
    "# Or from the repo, for the latest version:\n",
    "# %pip install git+https://github.com/vanderschaarlab/temporai.git"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## All `preprocessing.imputation` plugins\n",
    "\n",
    "To see all the relevant plugins:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'static': ['static_tabular_imputer'],\n",
       " 'temporal': ['ffill', 'ts_tabular_imputer', 'bfill']}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tempor import plugin_loader\n",
    "\n",
    "plugin_loader.list()[\"preprocessing\"][\"imputation\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now also load data source(s) we will use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SineDataSource = plugin_loader.get_class(\"prediction.one_off.sine\", plugin_type=\"datasource\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using a static data imputation plugin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:42 | INFO     | hyperimpute.logger:log_and_print:65 | Iteration imputation: select_model_by_column: True, select_model_by_iteration: True\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OneOffPredictionDataset(\n",
      "    time_series=TimeSeriesSamples([100, *, 5]),\n",
      "    static=StaticSamples([100, 4]),\n",
      "    predictive=OneOffPredictionTaskData(targets=StaticSamples([100, 1]))\n",
      ")\n",
      "StaticTabularImputer(\n",
      "    name='static_tabular_imputer',\n",
      "    category='preprocessing.imputation.static',\n",
      "    plugin_type='method',\n",
      "    params={\n",
      "        'imputer': 'ice',\n",
      "        'random_state': 0,\n",
      "        'imputer_params': {'random_state': 0}\n",
      "    }\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "from tempor import plugin_loader\n",
    "\n",
    "dataset = SineDataSource(with_missing=True, random_state=42).load()\n",
    "print(dataset)\n",
    "\n",
    "model = plugin_loader.get(\"preprocessing.imputation.static.static_tabular_imputer\", static_imputer=\"mean\")\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Missing value count: 40\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<p><span style=\"font-family: monospace;\">StaticSamples</span> with data:</p><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sample_idx</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.374540</td>\n",
       "      <td>0.950714</td>\n",
       "      <td>0.731994</td>\n",
       "      <td>0.598658</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.156019</td>\n",
       "      <td>0.155995</td>\n",
       "      <td>0.058084</td>\n",
       "      <td>0.866176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.601115</td>\n",
       "      <td>0.708073</td>\n",
       "      <td>0.020584</td>\n",
       "      <td>0.969910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.832443</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.181825</td>\n",
       "      <td>0.183405</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.304242</td>\n",
       "      <td>0.524756</td>\n",
       "      <td>0.431945</td>\n",
       "      <td>0.291229</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>NaN</td>\n",
       "      <td>0.696737</td>\n",
       "      <td>0.628943</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>0.735071</td>\n",
       "      <td>0.803481</td>\n",
       "      <td>0.282035</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>0.750615</td>\n",
       "      <td>0.806835</td>\n",
       "      <td>0.990505</td>\n",
       "      <td>0.412618</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>0.372018</td>\n",
       "      <td>0.776413</td>\n",
       "      <td>0.340804</td>\n",
       "      <td>0.930757</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>0.858413</td>\n",
       "      <td>0.428994</td>\n",
       "      <td>0.750871</td>\n",
       "      <td>0.754543</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "StaticSamples with data:\n",
       "                   0         1         2         3\n",
       "sample_idx                                        \n",
       "0           0.374540  0.950714  0.731994  0.598658\n",
       "1           0.156019  0.155995  0.058084  0.866176\n",
       "2           0.601115  0.708073  0.020584  0.969910\n",
       "3           0.832443       NaN  0.181825  0.183405\n",
       "4           0.304242  0.524756  0.431945  0.291229\n",
       "...              ...       ...       ...       ...\n",
       "95               NaN  0.696737  0.628943       NaN\n",
       "96          0.735071  0.803481  0.282035       NaN\n",
       "97          0.750615  0.806835  0.990505  0.412618\n",
       "98          0.372018  0.776413  0.340804  0.930757\n",
       "99          0.858413  0.428994  0.750871  0.754543\n",
       "\n",
       "[100 rows x 4 columns]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Note missingness in static data.\n",
    "\n",
    "print(\"Missing value count:\", dataset.static.dataframe().isnull().sum().sum())  # type: ignore\n",
    "\n",
    "dataset.static"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:42 | INFO     | hyperimpute.logger:log_and_print:65 |   > HyperImpute using inner optimization\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:42 | INFO     | hyperimpute.logger:log_and_print:65 |   > Imputation iter 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:42 | INFO     | hyperimpute.logger:log_and_print:65 |      >>> Column 0 <-- score -0.23109777030617995 <-- Model linear_regression\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:42 | INFO     | hyperimpute.logger:log_and_print:65 |      >>> Column 1 <-- score -0.22005514968324613 <-- Model linear_regression\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:42 | INFO     | hyperimpute.logger:log_and_print:65 |   > Imputation iter 1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:42 | INFO     | hyperimpute.logger:log_and_print:65 |      >>> Column 0 <-- score -0.23109777030617995 <-- Model linear_regression\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:42 | INFO     | hyperimpute.logger:log_and_print:65 |      >>> Column 1 <-- score -0.21670750510884584 <-- Model linear_regression\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:42 | INFO     | hyperimpute.logger:log_and_print:65 |   > Imputation iter 2\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:42 | INFO     | hyperimpute.logger:log_and_print:65 |      >>> Column 0 <-- score -0.23109777030617995 <-- Model linear_regression\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:42 | INFO     | hyperimpute.logger:log_and_print:65 |      >>> Column 1 <-- score -0.2166465658117811 <-- Model linear_regression\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:42 | INFO     | hyperimpute.logger:log_and_print:65 |   > Imputation iter 3\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:42 | INFO     | hyperimpute.logger:log_and_print:65 |      >>> Column 0 <-- score -0.23109777030617995 <-- Model linear_regression\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:42 | INFO     | hyperimpute.logger:log_and_print:65 |      >>> Column 1 <-- score -0.2166456843686172 <-- Model linear_regression\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:42 | INFO     | hyperimpute.logger:log_and_print:65 |   > Imputation iter 4\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:42 | INFO     | hyperimpute.logger:log_and_print:65 |      >>> Column 0 <-- score -0.23109777030617995 <-- Model linear_regression\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:43 | INFO     | hyperimpute.logger:log_and_print:65 |      >>> Column 1 <-- score -0.21664567863448503 <-- Model linear_regression\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:43 | INFO     | hyperimpute.logger:log_and_print:65 |   > Imputation iter 5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:43 | INFO     | hyperimpute.logger:log_and_print:65 |      >>> Column 0 <-- score -0.23109777030617995 <-- Model linear_regression\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:43 | INFO     | hyperimpute.logger:log_and_print:65 |      >>> Column 1 <-- score -0.21664567863448503 <-- Model linear_regression\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:43 | INFO     | hyperimpute.logger:log_and_print:65 |   > Imputation iter 6\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:43 | INFO     | hyperimpute.logger:log_and_print:65 |      >>> Column 0 <-- score -0.23109777030617995 <-- Model linear_regression\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:43 | INFO     | hyperimpute.logger:log_and_print:65 |      >>> Column 1 <-- score -0.21664567863448503 <-- Model linear_regression\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:43 | INFO     | hyperimpute.logger:log_and_print:65 |   > Imputation iter 7\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:43 | INFO     | hyperimpute.logger:log_and_print:65 |      >>> Column 0 <-- score -0.23109777030617995 <-- Model linear_regression\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:43 | INFO     | hyperimpute.logger:log_and_print:65 |      >>> Column 1 <-- score -0.21664567863448503 <-- Model linear_regression\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:43 | INFO     | hyperimpute.logger:log_and_print:65 |   > Imputation iter 8\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:43 | INFO     | hyperimpute.logger:log_and_print:65 |      >>> Column 0 <-- score -0.23109777030617995 <-- Model linear_regression\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:43 | INFO     | hyperimpute.logger:log_and_print:65 |      >>> Column 1 <-- score -0.21664567863448503 <-- Model linear_regression\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:43 | INFO     | hyperimpute.logger:log_and_print:65 |   > Imputation iter 9\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:43 | INFO     | hyperimpute.logger:log_and_print:65 |      >>> Column 0 <-- score -0.23109777030617995 <-- Model linear_regression\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:43 | INFO     | hyperimpute.logger:log_and_print:65 |      >>> Column 1 <-- score -0.21664567863448503 <-- Model linear_regression\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:43 | INFO     | hyperimpute.logger:log_and_print:65 |   > Imputation iter 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:43 | INFO     | hyperimpute.logger:log_and_print:65 |      >>> Column 0 <-- score -0.23109777030617995 <-- Model linear_regression\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:43 | INFO     | hyperimpute.logger:log_and_print:65 |      >>> Column 1 <-- score -0.21664567863448503 <-- Model linear_regression\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-16 22:18:43 | INFO     | hyperimpute.logger:log_and_print:65 |      >>>> Early stopping on objective diff iteration\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Missing value count: 0\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<p><span style=\"font-family: monospace;\">StaticSamples</span> with data:</p><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sample_idx</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.374540</td>\n",
       "      <td>0.950714</td>\n",
       "      <td>0.731994</td>\n",
       "      <td>0.598658</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.156019</td>\n",
       "      <td>0.155995</td>\n",
       "      <td>0.058084</td>\n",
       "      <td>0.866176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.601115</td>\n",
       "      <td>0.708073</td>\n",
       "      <td>0.020584</td>\n",
       "      <td>0.969910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.832443</td>\n",
       "      <td>0.450438</td>\n",
       "      <td>0.181825</td>\n",
       "      <td>0.183405</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.304242</td>\n",
       "      <td>0.524756</td>\n",
       "      <td>0.431945</td>\n",
       "      <td>0.291229</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>0.498806</td>\n",
       "      <td>0.696737</td>\n",
       "      <td>0.628943</td>\n",
       "      <td>0.509994</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>0.735071</td>\n",
       "      <td>0.803481</td>\n",
       "      <td>0.282035</td>\n",
       "      <td>0.503886</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>0.750615</td>\n",
       "      <td>0.806835</td>\n",
       "      <td>0.990505</td>\n",
       "      <td>0.412618</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>0.372018</td>\n",
       "      <td>0.776413</td>\n",
       "      <td>0.340804</td>\n",
       "      <td>0.930757</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>0.858413</td>\n",
       "      <td>0.428994</td>\n",
       "      <td>0.750871</td>\n",
       "      <td>0.754543</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "StaticSamples with data:\n",
       "                   0         1         2         3\n",
       "sample_idx                                        \n",
       "0           0.374540  0.950714  0.731994  0.598658\n",
       "1           0.156019  0.155995  0.058084  0.866176\n",
       "2           0.601115  0.708073  0.020584  0.969910\n",
       "3           0.832443  0.450438  0.181825  0.183405\n",
       "4           0.304242  0.524756  0.431945  0.291229\n",
       "...              ...       ...       ...       ...\n",
       "95          0.498806  0.696737  0.628943  0.509994\n",
       "96          0.735071  0.803481  0.282035  0.503886\n",
       "97          0.750615  0.806835  0.990505  0.412618\n",
       "98          0.372018  0.776413  0.340804  0.930757\n",
       "99          0.858413  0.428994  0.750871  0.754543\n",
       "\n",
       "[100 rows x 4 columns]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Note no more missingness in static data.\n",
    "\n",
    "dataset = model.fit_transform(dataset)  # Or call fit() then transform().\n",
    "\n",
    "print(\"Missing value count:\", dataset.static.dataframe().isnull().sum().sum())  # type: ignore\n",
    "\n",
    "dataset.static"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using a temporal data imputation plugin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OneOffPredictionDataset(\n",
      "    time_series=TimeSeriesSamples([100, *, 5]),\n",
      "    static=StaticSamples([100, 4]),\n",
      "    predictive=OneOffPredictionTaskData(targets=StaticSamples([100, 1]))\n",
      ")\n",
      "BFillImputer(\n",
      "    name='bfill',\n",
      "    category='preprocessing.imputation.temporal',\n",
      "    plugin_type='method',\n",
      "    params={}\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "from tempor import plugin_loader\n",
    "\n",
    "dataset = SineDataSource(with_missing=True, random_state=42).load()\n",
    "print(dataset)\n",
    "\n",
    "model = plugin_loader.get(\"preprocessing.imputation.temporal.bfill\")\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Missing value count: 500\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<p><span style=\"font-family: monospace;\">TimeSeriesSamples</span> with data:</p><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sample_idx</th>\n",
       "      <th>time_idx</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">0</th>\n",
       "      <th>0</th>\n",
       "      <td>-0.955338</td>\n",
       "      <td>0.016053</td>\n",
       "      <td>-0.995752</td>\n",
       "      <td>0.948138</td>\n",
       "      <td>0.738158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.896718</td>\n",
       "      <td>0.717189</td>\n",
       "      <td>-0.497625</td>\n",
       "      <td>0.962001</td>\n",
       "      <td>0.968258</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.346466</td>\n",
       "      <td>0.999920</td>\n",
       "      <td>0.423104</td>\n",
       "      <td>0.639780</td>\n",
       "      <td>0.972469</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.393737</td>\n",
       "      <td>0.699299</td>\n",
       "      <td>0.984517</td>\n",
       "      <td>0.094046</td>\n",
       "      <td>0.749807</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.918072</td>\n",
       "      <td>-0.009290</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">99</th>\n",
       "      <th>5</th>\n",
       "      <td>0.904284</td>\n",
       "      <td>-0.939985</td>\n",
       "      <td>0.994099</td>\n",
       "      <td>-0.984349</td>\n",
       "      <td>0.688521</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.990911</td>\n",
       "      <td>-0.518593</td>\n",
       "      <td>0.908681</td>\n",
       "      <td>-0.801263</td>\n",
       "      <td>0.813486</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.757745</td>\n",
       "      <td>0.131791</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-0.110629</td>\n",
       "      <td>0.908965</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>NaN</td>\n",
       "      <td>0.723981</td>\n",
       "      <td>0.476023</td>\n",
       "      <td>0.650082</td>\n",
       "      <td>0.971498</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>-0.288052</td>\n",
       "      <td>0.996486</td>\n",
       "      <td>0.173255</td>\n",
       "      <td>0.999008</td>\n",
       "      <td>0.998817</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1000 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "TimeSeriesSamples with data:\n",
       "                            0         1         2         3         4\n",
       "sample_idx time_idx                                                  \n",
       "0          0        -0.955338  0.016053 -0.995752  0.948138  0.738158\n",
       "           1        -0.896718  0.717189 -0.497625  0.962001  0.968258\n",
       "           2        -0.346466  0.999920  0.423104  0.639780  0.972469\n",
       "           3         0.393737  0.699299  0.984517  0.094046  0.749807\n",
       "           4         0.918072 -0.009290       NaN       NaN       NaN\n",
       "...                       ...       ...       ...       ...       ...\n",
       "99         5         0.904284 -0.939985  0.994099 -0.984349  0.688521\n",
       "           6         0.990911 -0.518593  0.908681 -0.801263  0.813486\n",
       "           7         0.757745  0.131791       NaN -0.110629  0.908965\n",
       "           8              NaN  0.723981  0.476023  0.650082  0.971498\n",
       "           9        -0.288052  0.996486  0.173255  0.999008  0.998817\n",
       "\n",
       "[1000 rows x 5 columns]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Note missingness in temporal data.\n",
    "\n",
    "print(\"Missing value count:\", dataset.time_series.dataframe().isnull().sum().sum())\n",
    "\n",
    "dataset.time_series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Missing value count: 0\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<p><span style=\"font-family: monospace;\">TimeSeriesSamples</span> with data:</p><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sample_idx</th>\n",
       "      <th>time_idx</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">0</th>\n",
       "      <th>0</th>\n",
       "      <td>-0.955338</td>\n",
       "      <td>0.016053</td>\n",
       "      <td>-0.995752</td>\n",
       "      <td>0.948138</td>\n",
       "      <td>0.738158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.896718</td>\n",
       "      <td>0.717189</td>\n",
       "      <td>-0.497625</td>\n",
       "      <td>0.962001</td>\n",
       "      <td>0.968258</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.346466</td>\n",
       "      <td>0.999920</td>\n",
       "      <td>0.423104</td>\n",
       "      <td>0.639780</td>\n",
       "      <td>0.972469</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.393737</td>\n",
       "      <td>0.699299</td>\n",
       "      <td>0.984517</td>\n",
       "      <td>0.094046</td>\n",
       "      <td>0.749807</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.918072</td>\n",
       "      <td>-0.009290</td>\n",
       "      <td>-0.167662</td>\n",
       "      <td>-0.893854</td>\n",
       "      <td>-0.127538</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">99</th>\n",
       "      <th>5</th>\n",
       "      <td>0.904284</td>\n",
       "      <td>-0.939985</td>\n",
       "      <td>0.994099</td>\n",
       "      <td>-0.984349</td>\n",
       "      <td>0.688521</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.990911</td>\n",
       "      <td>-0.518593</td>\n",
       "      <td>0.908681</td>\n",
       "      <td>-0.801263</td>\n",
       "      <td>0.813486</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.757745</td>\n",
       "      <td>0.131791</td>\n",
       "      <td>0.476023</td>\n",
       "      <td>-0.110629</td>\n",
       "      <td>0.908965</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-0.288052</td>\n",
       "      <td>0.723981</td>\n",
       "      <td>0.476023</td>\n",
       "      <td>0.650082</td>\n",
       "      <td>0.971498</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>-0.288052</td>\n",
       "      <td>0.996486</td>\n",
       "      <td>0.173255</td>\n",
       "      <td>0.999008</td>\n",
       "      <td>0.998817</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1000 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "TimeSeriesSamples with data:\n",
       "                            0         1         2         3         4\n",
       "sample_idx time_idx                                                  \n",
       "0          0        -0.955338  0.016053 -0.995752  0.948138  0.738158\n",
       "           1        -0.896718  0.717189 -0.497625  0.962001  0.968258\n",
       "           2        -0.346466  0.999920  0.423104  0.639780  0.972469\n",
       "           3         0.393737  0.699299  0.984517  0.094046  0.749807\n",
       "           4         0.918072 -0.009290 -0.167662 -0.893854 -0.127538\n",
       "...                       ...       ...       ...       ...       ...\n",
       "99         5         0.904284 -0.939985  0.994099 -0.984349  0.688521\n",
       "           6         0.990911 -0.518593  0.908681 -0.801263  0.813486\n",
       "           7         0.757745  0.131791  0.476023 -0.110629  0.908965\n",
       "           8        -0.288052  0.723981  0.476023  0.650082  0.971498\n",
       "           9        -0.288052  0.996486  0.173255  0.999008  0.998817\n",
       "\n",
       "[1000 rows x 5 columns]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Note no more missingness in temporal data.\n",
    "\n",
    "dataset = model.fit_transform(dataset)  # Or call fit() then transform().\n",
    "\n",
    "print(\"Missing value count:\", dataset.time_series.dataframe().isnull().sum().sum())\n",
    "\n",
    "dataset.time_series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 🎉 Congratulations!\n",
    "\n",
    "Congratulations on completing this notebook tutorial! If you enjoyed this and would like to join the movement towards *Machine learning and AI for Medicine*, you can do so in the following ways!\n",
    "\n",
    "\n",
    "\n",
    "### ⭐ Star [TemporAI](https://github.com/vanderschaarlab/temporai) on GitHub\n",
    "\n",
    "- The easiest way to help our community is by just starring the repos! This helps raise awareness of the tools we're building.\n",
    "\n",
    "\n",
    "\n",
    "### Check out other projects from [vanderschaarlab](https://github.com/vanderschaarlab)\n",
    "- 📝 [HyperImpute](https://github.com/vanderschaarlab/hyperimpute)\n",
    "- 📊 [AutoPrognosis](https://github.com/vanderschaarlab/autoprognosis)\n",
    "- 🤖 [SynthCity](https://github.com/vanderschaarlab/synthcity)\n",
    " "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "RE_py38_temporai",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}