tutorials/extending/tutorial02_testing_custom_method.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extending TemporAI Tutorial 02: Testing a Custom Method Plugin\n",
    "\n",
    "This tutorial gives a brief overview of how to test your custom method plugin.\n",
    "\n",
    "For the basics of writing a custom plugin, see the *\"Writing Custom Method Plugin\" tutorial* first. This tutorial assumes you have already written a custom plugin."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Skip the below cell if you are not on Google Colab / already have TemporAI installed:*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install temporai[dev]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**⚙️ Installation for testing**\n",
    "\n",
    "You should install the `[dev]` TemporAI to run tests:\n",
    "```bash\n",
    "pip install temporai[dev]\n",
    "```\n",
    "\n",
    "Alternatively, the best way to install TemporAI for development is to clone the repo, checkout a branch, and install in editable mode:\n",
    "```bash\n",
    "git clone https://github.com/vanderschaarlab/temporai.git\n",
    "cd temporai\n",
    "git checkout -b my-branch-name\n",
    "pip install -e .[dev]\n",
    "```\n",
    "\n",
    "📘 See also the [contribution guide](https://github.com/vanderschaarlab/temporai/blob/main/CONTRIBUTING.md)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Find and adapt suitable tests\n",
    "\n",
    "The simplest way to run a set of tests on your custom method plugin is to find and adapt existing tests for plugins of the same category.\n",
    "\n",
    "Method plugin categories in TemporAI can be found as below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Method plugin categories:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">[</span>\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'prediction.one_off.classification'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'prediction.one_off.regression'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'prediction.temporal.classification'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'prediction.temporal.regression'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'preprocessing.encoding.static'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'preprocessing.encoding.temporal'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'preprocessing.imputation.static'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'preprocessing.imputation.temporal'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'preprocessing.nop'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'preprocessing.scaling.static'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'preprocessing.scaling.temporal'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'time_to_event'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'treatments.one_off.regression'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'treatments.temporal.classification'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'treatments.temporal.regression'</span>\n",
       "<span style=\"font-weight: bold\">]</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m[\u001b[0m\n",
       "    \u001b[32m'prediction.one_off.classification'\u001b[0m,\n",
       "    \u001b[32m'prediction.one_off.regression'\u001b[0m,\n",
       "    \u001b[32m'prediction.temporal.classification'\u001b[0m,\n",
       "    \u001b[32m'prediction.temporal.regression'\u001b[0m,\n",
       "    \u001b[32m'preprocessing.encoding.static'\u001b[0m,\n",
       "    \u001b[32m'preprocessing.encoding.temporal'\u001b[0m,\n",
       "    \u001b[32m'preprocessing.imputation.static'\u001b[0m,\n",
       "    \u001b[32m'preprocessing.imputation.temporal'\u001b[0m,\n",
       "    \u001b[32m'preprocessing.nop'\u001b[0m,\n",
       "    \u001b[32m'preprocessing.scaling.static'\u001b[0m,\n",
       "    \u001b[32m'preprocessing.scaling.temporal'\u001b[0m,\n",
       "    \u001b[32m'time_to_event'\u001b[0m,\n",
       "    \u001b[32m'treatments.one_off.regression'\u001b[0m,\n",
       "    \u001b[32m'treatments.temporal.classification'\u001b[0m,\n",
       "    \u001b[32m'treatments.temporal.regression'\u001b[0m\n",
       "\u001b[1m]\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All method plugins:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">{</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008000; text-decoration-color: #008000\">'prediction'</span>: <span style=\"font-weight: bold\">{</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #008000; text-decoration-color: #008000\">'one_off'</span>: <span style=\"font-weight: bold\">{</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #008000; text-decoration-color: #008000\">'classification'</span>: <span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'cde_classifier'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'ode_classifier'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'nn_classifier'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'laplace_ode_classifier'</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #008000; text-decoration-color: #008000\">'regression'</span>: <span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'laplace_ode_regressor'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'nn_regressor'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'ode_regressor'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'cde_regressor'</span><span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"font-weight: bold\">}</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #008000; text-decoration-color: #008000\">'temporal'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'classification'</span>: <span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'seq2seq_classifier'</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'regression'</span>: <span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'seq2seq_regressor'</span><span style=\"font-weight: bold\">]}</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">}</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008000; text-decoration-color: #008000\">'preprocessing'</span>: <span style=\"font-weight: bold\">{</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #008000; text-decoration-color: #008000\">'encoding'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'static'</span>: <span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'static_onehot_encoder'</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'temporal'</span>: <span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'ts_onehot_encoder'</span><span style=\"font-weight: bold\">]}</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #008000; text-decoration-color: #008000\">'imputation'</span>: <span style=\"font-weight: bold\">{</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #008000; text-decoration-color: #008000\">'static'</span>: <span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'static_tabular_imputer'</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #008000; text-decoration-color: #008000\">'temporal'</span>: <span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'ffill'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'ts_tabular_imputer'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'bfill'</span><span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"font-weight: bold\">}</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #008000; text-decoration-color: #008000\">'nop'</span>: <span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'nop_transformer'</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #008000; text-decoration-color: #008000\">'scaling'</span>: <span style=\"font-weight: bold\">{</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #008000; text-decoration-color: #008000\">'static'</span>: <span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'static_minmax_scaler'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'static_standard_scaler'</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #008000; text-decoration-color: #008000\">'temporal'</span>: <span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'ts_minmax_scaler'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'ts_standard_scaler'</span><span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"font-weight: bold\">}</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">}</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008000; text-decoration-color: #008000\">'time_to_event'</span>: <span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'ts_coxph'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'ts_xgb'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'dynamic_deephit'</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008000; text-decoration-color: #008000\">'treatments'</span>: <span style=\"font-weight: bold\">{</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #008000; text-decoration-color: #008000\">'one_off'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'regression'</span>: <span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'synctwin_regressor'</span><span style=\"font-weight: bold\">]}</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #008000; text-decoration-color: #008000\">'temporal'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'classification'</span>: <span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'crn_classifier'</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'regression'</span>: <span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'crn_regressor'</span><span style=\"font-weight: bold\">]}</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">}</span>\n",
       "<span style=\"font-weight: bold\">}</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[32m'prediction'\u001b[0m: \u001b[1m{\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[32m'one_off'\u001b[0m: \u001b[1m{\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[32m'classification'\u001b[0m: \u001b[1m[\u001b[0m\u001b[32m'cde_classifier'\u001b[0m, \u001b[32m'ode_classifier'\u001b[0m, \u001b[32m'nn_classifier'\u001b[0m, \u001b[32m'laplace_ode_classifier'\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[32m'regression'\u001b[0m: \u001b[1m[\u001b[0m\u001b[32m'laplace_ode_regressor'\u001b[0m, \u001b[32m'nn_regressor'\u001b[0m, \u001b[32m'ode_regressor'\u001b[0m, \u001b[32m'cde_regressor'\u001b[0m\u001b[1m]\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1m}\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[32m'temporal'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'classification'\u001b[0m: \u001b[1m[\u001b[0m\u001b[32m'seq2seq_classifier'\u001b[0m\u001b[1m]\u001b[0m, \u001b[32m'regression'\u001b[0m: \u001b[1m[\u001b[0m\u001b[32m'seq2seq_regressor'\u001b[0m\u001b[1m]\u001b[0m\u001b[1m}\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m}\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[32m'preprocessing'\u001b[0m: \u001b[1m{\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[32m'encoding'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'static'\u001b[0m: \u001b[1m[\u001b[0m\u001b[32m'static_onehot_encoder'\u001b[0m\u001b[1m]\u001b[0m, \u001b[32m'temporal'\u001b[0m: \u001b[1m[\u001b[0m\u001b[32m'ts_onehot_encoder'\u001b[0m\u001b[1m]\u001b[0m\u001b[1m}\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[32m'imputation'\u001b[0m: \u001b[1m{\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[32m'static'\u001b[0m: \u001b[1m[\u001b[0m\u001b[32m'static_tabular_imputer'\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[32m'temporal'\u001b[0m: \u001b[1m[\u001b[0m\u001b[32m'ffill'\u001b[0m, \u001b[32m'ts_tabular_imputer'\u001b[0m, \u001b[32m'bfill'\u001b[0m\u001b[1m]\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1m}\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[32m'nop'\u001b[0m: \u001b[1m[\u001b[0m\u001b[32m'nop_transformer'\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[32m'scaling'\u001b[0m: \u001b[1m{\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[32m'static'\u001b[0m: \u001b[1m[\u001b[0m\u001b[32m'static_minmax_scaler'\u001b[0m, \u001b[32m'static_standard_scaler'\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[32m'temporal'\u001b[0m: \u001b[1m[\u001b[0m\u001b[32m'ts_minmax_scaler'\u001b[0m, \u001b[32m'ts_standard_scaler'\u001b[0m\u001b[1m]\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1m}\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m}\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[32m'time_to_event'\u001b[0m: \u001b[1m[\u001b[0m\u001b[32m'ts_coxph'\u001b[0m, \u001b[32m'ts_xgb'\u001b[0m, \u001b[32m'dynamic_deephit'\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[32m'treatments'\u001b[0m: \u001b[1m{\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[32m'one_off'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'regression'\u001b[0m: \u001b[1m[\u001b[0m\u001b[32m'synctwin_regressor'\u001b[0m\u001b[1m]\u001b[0m\u001b[1m}\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[32m'temporal'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'classification'\u001b[0m: \u001b[1m[\u001b[0m\u001b[32m'crn_classifier'\u001b[0m\u001b[1m]\u001b[0m, \u001b[32m'regression'\u001b[0m: \u001b[1m[\u001b[0m\u001b[32m'crn_regressor'\u001b[0m\u001b[1m]\u001b[0m\u001b[1m}\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m}\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from tempor import plugin_loader\n",
    "from rich.pretty import pprint  # For prettifying the print output only.\n",
    "\n",
    "plugin_categories = plugin_loader.list_categories(plugin_type=\"method\")\n",
    "\n",
    "print(\"Method plugin categories:\")\n",
    "pprint(list(plugin_categories.keys()), indent_guides=False)\n",
    "\n",
    "print(\"All method plugins:\")\n",
    "pprint(plugin_loader.list(plugin_type=\"method\"), indent_guides=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suitable tests can be found in TemporAI source code (`tests/methods/...`), organized hierarchically by plugin category.\n",
    "\n",
    "For example, tests for `prediction/one_off/classification` will be located under:\n",
    "* [`tests/methods/prediction/one_off/classification`](https://github.com/vanderschaarlab/temporai/tree/main/tests/methods/prediction/one_off/classification).\n",
    "\n",
    "Find an example test file in this category, and adapt the tests from it to your custom plugin. For instance, we could look at:\n",
    "* [test file for `nn_classifier` plugin](https://github.com/vanderschaarlab/temporai/blob/main/tests/methods/prediction/one_off/classification/test_nn_classifier.py).\n",
    "\n",
    "The test suite differs by plugin category, but in general, the following points are worth noting.\n",
    "\n",
    "* [`pytest`](https://docs.pytest.org/) is used for testing.\n",
    "* [Parametrization](https://docs.pytest.org/en/7.3.x/how-to/parametrize.html) is used to run similar tests with different parameters.\n",
    "* [Fixtures](https://docs.pytest.org/en/7.3.x/how-to/fixtures.html) are used for reusable test elements, e.g. datasets.\n",
    "* Plugins are tested with two methods of importing them (`PLUGIN_FROM_OPTIONS`): `\"from_api\"` and `\"from_module\"`, this is handled by the helper fixture `get_test_plugin`.\n",
    "* Typically, the set of test functions needed looks something like this:\n",
    "    * `test_sanity`: the very basics, check loading of your plugin works.\n",
    "    * `test_fit`: test your plugin's `fit` method.\n",
    "    * `test_predict, test_transform, ...`: test the additional methods relevant to your plugin's category.\n",
    "    * `test_serde`: test serialization and deserialization works.\n",
    "* The below constants are set and reused as parameters in the test suite. Adapt these as necessary. The various dataset fixtures can be found in [`conftest.py`](https://github.com/vanderschaarlab/temporai/blob/main/tests/conftest.py).\n",
    "    ```python\n",
    "    INIT_KWARGS = {\"random_state\": 123, \"n_iter\": 5}  # Input parameters for the plugin.\n",
    "    TEST_ON_DATASETS = [\"sine_data_small\"]  # A list of dataset fixtures to test the plugin on.\n",
    "    ```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An example test file would looks something like below.\n",
    "\n",
    "In this example, it is assumed that the plugin is of the `prediction/one_off/classification` category, and is named `\"my_classifier\"`.\n",
    "\n",
    "```python\n",
    "# test_my_classifier.py\n",
    "# Example for illustration only - adapt to your own plugin as needed.\n",
    "\n",
    "from typing import Callable, Dict\n",
    "\n",
    "import pytest\n",
    "\n",
    "from tempor.methods.prediction.one_off.classification import BaseOneOffClassifier\n",
    "from tempor.methods.prediction.one_off.classification.plugin_my_classifier import MyClassifier\n",
    "from tempor.utils.serialization import load, save\n",
    "\n",
    "INIT_KWARGS = {\"random_state\": 123, \"n_iter\": 5}\n",
    "PLUGIN_FROM_OPTIONS = [\"from_api\", pytest.param(\"from_module\", marks=pytest.mark.extra)]\n",
    "TEST_ON_DATASETS = [\"sine_data_small\"]\n",
    "\n",
    "\n",
    "@pytest.fixture\n",
    "def get_test_plugin(get_plugin: Callable):\n",
    "    def func(plugin_from: str, base_kwargs: Dict):\n",
    "        return get_plugin(\n",
    "            plugin_from,\n",
    "            fqn=\"prediction.one_off.classification.my_classifier\",\n",
    "            cls=MyClassifier,\n",
    "            kwargs=base_kwargs,\n",
    "        )\n",
    "\n",
    "    return func\n",
    "\n",
    "\n",
    "@pytest.mark.parametrize(\"plugin_from\", PLUGIN_FROM_OPTIONS)\n",
    "def test_sanity(get_test_plugin: Callable, plugin_from: str) -> None:\n",
    "    test_plugin = get_test_plugin(plugin_from, INIT_KWARGS)\n",
    "    assert test_plugin is not None\n",
    "    assert test_plugin.name == \"nn_classifier\"\n",
    "    assert test_plugin.full_name() == \"prediction.one_off.classification.my_classifier\"\n",
    "    assert len(test_plugin.hyperparameter_space()) == 9\n",
    "\n",
    "\n",
    "@pytest.mark.parametrize(\"plugin_from\", PLUGIN_FROM_OPTIONS)\n",
    "@pytest.mark.parametrize(\"data\", TEST_ON_DATASETS)\n",
    "def test_fit(plugin_from: str, data: str, get_test_plugin: Callable, get_dataset: Callable) -> None:\n",
    "    test_plugin: BaseOneOffClassifier = get_test_plugin(plugin_from, INIT_KWARGS)\n",
    "    dataset = get_dataset(data)\n",
    "    test_plugin.fit(dataset)\n",
    "\n",
    "\n",
    "@pytest.mark.parametrize(\"plugin_from\", PLUGIN_FROM_OPTIONS)\n",
    "@pytest.mark.parametrize(\"data\", TEST_ON_DATASETS)\n",
    "def test_predict(\n",
    "    plugin_from: str, data: str, no_targets: bool, get_test_plugin: Callable, get_dataset: Callable\n",
    ") -> None:\n",
    "    test_plugin: BaseOneOffClassifier = get_test_plugin(plugin_from, INIT_KWARGS)\n",
    "    dataset = get_dataset(data)\n",
    "    test_plugin.fit(dataset)\n",
    "    output = test_plugin.predict(dataset)\n",
    "    assert output.numpy().shape == (len(dataset.time_series), 1)\n",
    "\n",
    "\n",
    "# Other categories of plugins would have more / different methods to test.\n",
    "\n",
    "\n",
    "@pytest.mark.parametrize(\"data\", TEST_ON_DATASETS)\n",
    "def test_serde(data: str, get_test_plugin: Callable, get_dataset: Callable) -> None:\n",
    "    test_plugin: BaseOneOffClassifier = get_test_plugin(\"from_api\", INIT_KWARGS)\n",
    "    dataset = get_dataset(data)\n",
    "\n",
    "    dump = save(test_plugin)\n",
    "    reloaded1 = load(dump)\n",
    "\n",
    "    reloaded1.fit(dataset)\n",
    "\n",
    "    dump = save(reloaded1)\n",
    "    reloaded2 = load(dump)\n",
    "\n",
    "    reloaded2.predict(dataset)\n",
    "\n",
    "```\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to run the tests, the test file (e.g. `test_my_classifier.py`) should be placed into the appropriate test directory, e.g. in this example under:\n",
    "\n",
    "[`tests/methods/prediction/one_off/classification`](https://github.com/vanderschaarlab/temporai/tree/main/tests/methods/prediction/one_off/classification).\n",
    "\n",
    "The tests can the be run like so:\n",
    "```bash\n",
    "pytest -x tests/methods/prediction/one_off/classification/test_my_classifier.py\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Plugin loader and all method plugin tests\n",
    "\n",
    "Some additional tests that apply to your plugin are found in:\n",
    "* [`tests/test_plugin_loader.py`](https://github.com/vanderschaarlab/temporai/blob/main/tests/test_plugin_loader.py): The plugin registry checks.\n",
    "* [`tests/methods/test_all_plugins.py`](https://github.com/vanderschaarlab/temporai/blob/main/tests/methods/test_all_method_plugins.py): The common automatic basic tests for all plugins.\n",
    "\n",
    "In `test_plugin_loader.py`, your custom plugin should be added to `test_methods_contents`, e.g.\n",
    "```python\n",
    "assert \"my_classifier\" in all_plugins[\"prediction\"][\"one_off\"][\"classification\"]\n",
    "```\n",
    "\n",
    "`test_all_method_plugins.py` tests will run automatically, no changes here are needed.\n",
    "\n",
    "To check that these two test files pass:\n",
    "```bash\n",
    "pytest -x tests/test_plugin_loader.py\n",
    "pytest -x tests/methods/test_all_method_plugins.py\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Finally...\n",
    "\n",
    "Now is a perfect time to contribute your awesome plugin to the open sources eco-system by submitting a PR to TemporAI!\n",
    "\n",
    "Please follow the [contribution guide](https://github.com/vanderschaarlab/temporai/blob/main/CONTRIBUTING.md)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 🎉 Congratulations!\n",
    "\n",
    "Congratulations on completing this notebook tutorial! If you enjoyed this and would like to join the movement towards *Machine learning and AI for Medicine*, you can do so in the following ways!\n",
    "\n",
    "\n",
    "\n",
    "### ⭐ Star [TemporAI](https://github.com/vanderschaarlab/temporai) on GitHub\n",
    "\n",
    "- The easiest way to help our community is by just starring the repos! This helps raise awareness of the tools we're building.\n",
    "\n",
    "\n",
    "\n",
    "### Check out other projects from [vanderschaarlab](https://github.com/vanderschaarlab)\n",
    "- 📝 [HyperImpute](https://github.com/vanderschaarlab/hyperimpute)\n",
    "- 📊 [AutoPrognosis](https://github.com/vanderschaarlab/autoprognosis)\n",
    "- 🤖 [SynthCity](https://github.com/vanderschaarlab/synthcity)\n",
    " "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "RE_py38_temporai",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}