label_extraction.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment the next lines if running in Google Colab\n",
    "# !pip install clinicadl==0.2.1\n",
    "# !curl -k https://aramislab.paris.inria.fr/files/data/databases/tuto/dataOasis.tar.gz -o dataOasis.tar.gz\n",
    "# !tar xf dataOasis.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Define your population\n",
    "\n",
    "This notebook is an introduction to tools that can be used to identify relevant samples and split them between training, validation and test cohorts. **This step is mandatory preliminary to training to avoid issues such as lack of clinical meaning or data leakage**. \n",
    "\n",
    "In the following, we will see how to\n",
    "- extract samples corresponding to labels of interest from a BIDS hierarchy,\n",
    "- split these samples between training, validation and test sets\n",
    "\n",
    "using tools available in `clinica` and `clinicadl`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Merge metadata from a BIDS hierarchy with `clinica iotools`\n",
    "\n",
    "In a BIDS hierarchy, demographic, clinical and imaging metadata are stored in TSV files located at different levels of the hierarchy depending on whether they are specific to a subject (e.g. gender), a session (e.g. diagnosis) or a scan (e.g. acquisitions parameters).\n",
    "\n",
    "The following command line can be used to merge all the metadata in a single TSV file:\n",
    "```bash\n",
    "clinica iotools merge-tsv <bids_directory> <output_tsv>\n",
    "```\n",
    "where:\n",
    "- `bids_directory` is the input folder containing the dataset in a BIDS hierarchy.\n",
    "- `output_tsv` is the path of the output tsv. If a directory is specified instead of a file name, the default name for the file created will be `merge-tsv.tsv`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the [preprocessing section](./preprocessing.ipynb) an example BIDS of 4 subjects from OASIS-1 was generated (if you did not run interactively that section, download the dataset by uncomment the next cell):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !curl -k https://aramislab.paris.inria.fr/files/data/databases/tuto/OasisBids.tar.gz -o OasisBids.tar.gz\n",
    "# !tar xf OasisBids.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Execute the following command to gather metadata included in this BIDS:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge meta-data information\n",
    "!clinica iotools merge-tsv OasisBids_example example_merged.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to restrict the list of the sessions used to those including a T1-MR image. Then the following command is needed to identify which modalities are present for each session:\n",
    "```bash\n",
    "clinica iotools check-missing-modalities <bids_directory> <output_directory>\n",
    "```\n",
    "where:\n",
    "- `bids_directory` is the input folder of a BIDS compliant dataset.\n",
    "- `output_directory` is the output folder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Execute the following command to find which sessions include a T1-MR image on the example BIDS of OASIS:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find missing modalities\n",
    "!clinica iotools check-missing-modalities OasisBids_example example_missing_mods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of this command, `data/example_missing_mods`, is a folder in which a series of tsv files is written (one file per session label containing one row per subject and one column per modality)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get labels with `clinicadl tsvtool` on OASIS\n",
    "\n",
    "In this section we will now get the labels from **the whole OASIS dataset** on which `clinica iotools merge-tsv` and `clinica iotools check-missing-modalities` were already performed. \n",
    "\n",
    "The outputs of the corresponding pipelines can be found respectively in `data/OASIS_BIDS.tsv` and `data/OASIS_missing_mods`, and are provided on GitHub with the notebooks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Restrict the OASIS dataset to older adults\n",
    "\n",
    "OASIS-1 consists of 416 subjects aged 18 to 96: \n",
    "\n",
    "|      | subjects   | age                      | sex         | MMSE                | CDR                        |\n",
    "|------|------------|--------------------------|-------------|---------------------|----------------------------|\n",
    "|  AD  | 100        | 76.8 ± 7.1 [62.0, 96.0]  | 59F / 41M   | 24.3 ± 4.1 [14, 30] | 0: 0, 0.5: 70, 1: 28, 2: 2 |\n",
    "|  CN  | 316        | <font color=\"red\">45.1 ± 23.9 [18.0, 94.0] </font>| 197F / 119M | 29.6 ± 0.9 [25, 30] | 0: 316                     |\n",
    "\n",
    "As you can see, CN participants are on average younger than AD participants, which makes sense as AD mainly affects older adults. However, this is an issue as aging also causes brain atrophy. The classifier may mix the signal due to healthy brain aging and Alzheimer's disease on such dataset, which could lead to an over-estimation of the performance. \n",
    "\n",
    "To avoid this bias, CN participants younger than the youngest AD patient were removed. This restriction can be run with the following command line:\n",
    "\n",
    "```bash\n",
    "clinicadl tsvtool restrict <dataset> <merged_tsv> <results_path>\n",
    "```\n",
    "where:\n",
    "\n",
    "- `dataset` (str) is the name of the dataset. Choices are `OASIS` or `AIBL`.\n",
    "- `merged_tsv` (str) is the output file of the `clinica iotools merge-tsv` command.\n",
    "- `results_path` (str) is the path to the output tsv file (filename included). This tsv file comprises the same columns as `merged_tsv`.\n",
    "\n",
    "Execute the following cell to apply the restriction to OASIS-1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!clinicadl tsvtool restrict OASIS data/OASIS_BIDS.tsv data/OASIS_restricted_BIDS.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some other sessions were also excluded because the preprocessing operations failed. The list of the images that were kept after the preprocessing is stored in `data/OASIS_qc_output.tsv`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the labels\n",
    "\n",
    "The 5 labels described in the [first part of the course](./clinical) can be extracted with clinicadl using the command:\n",
    "\n",
    "```bash\n",
    "clinicadl tsvtool getlabels merged_tsv missing_mods results_path\n",
    "```\n",
    "where:\n",
    "- `merged_tsv` is the output file of the `clinica iotools merge-tsv` or `clinicadl tsvtool restrict` commands.\n",
    "- `missing_mods` is the folder containing the outputs of the `clinica iotools missing-mods` command.\n",
    "- `results_path` is the path to the folder where output tsv files will be written.\n",
    "\n",
    "```{tip}\n",
    "You can increase the verbosity of the command by adding -v flag(s).\n",
    "```\n",
    "\n",
    "By default the pipeline only extracts the AD and CN labels, which corresponds to the only available labels in OASIS. Run the following cell to extract them in a new folder `labels_lists` from the restricted version of OASIS:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!clinicadl tsvtool getlabels data/OASIS_restricted_BIDS.tsv data/OASIS_missing_mods data/labels_lists --restriction_path data/OASIS_after_qc.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each diagnostic label, a file has been created comprising all the sessions that can be included in the classification task.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Restriction path:</b><p>\n",
    "    At the end of the command line another restriction was given to extract the labels only from sessions in <code>data/OASIS_after_qc.tsv</code>. This tsv file corresponds to the output of the <a href=\"./preprocessing.ipynb\">quality check procedure</a> that was manually cut to only keep the sessions passing the quality check. It depends on the preprocessing: here it concerns a run of <code>t1-linear</code>.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyze the population\n",
    "\n",
    "The age bias in OASIS is well known and this is why the youngest CN participants were previously excluded. However, other biases may exist, especially after the quality check of the preprocessing which removed sessions from the dataset. Thus it is crucial to check before going further if there are other biases in the dataset.\n",
    "\n",
    "`clinicadl` implements a tool to perform a demographic and clinical analysis of the population:\n",
    "```bash\n",
    "clinicadl tsvtool analysis <merged_tsv> <formatted_data_path> <results_path>\n",
    "```\n",
    "where:\n",
    "- `merged_tsv` is the output file of the `clinica iotools merge-tsv` or `clinicadl tsvtool restrict` commands.\n",
    "- `formatted_data_path` is a folder containing one tsv file per label (output of `clinicadl tsvtool getlabels|split|kfold`).\n",
    "- `results_path` is the path to the tsv file that will be written (filename included).\n",
    "\n",
    "\n",
    "The following command will extract statistical values on the populations for each diagnostic label. Based on those it is possible to check that the dataset is suitable for the classification task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the analysis\n",
    "!clinicadl tsvtool analysis data/OASIS_BIDS.tsv data/labels_lists data/OASIS_analysis.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "def display_table(table_path):\n",
    "    \"\"\"Custom function to display the clinicadl tsvtool analysis output\"\"\"\n",
    "    import pandas as pd\n",
    "\n",
    "    OASIS_analysis_df = pd.read_csv(table_path, sep='\\t')\n",
    "    OASIS_analysis_df.set_index(\"diagnosis\", drop=True, inplace=True)\n",
    "    columns = [\"n_subjects\", \"n_scans\",\n",
    "               \"mean_age\", \"std_age\", \"min_age\", \"max_age\",\n",
    "               \"sexF\", \"sexM\",\n",
    "               \"mean_MMSE\", \"std_MMSE\", \"min_MMSE\", \"max_MMSE\",\n",
    "               \"CDR_0\", \"CDR_0.5\", \"CDR_1\", \"CDR_2\", \"CDR_3\"]\n",
    "\n",
    "    # Print formatted table\n",
    "    format_columns = [\"subjects\", \"scans\", \"age\", \"sex\", \"MMSE\", \"CDR\"]\n",
    "    format_df = pd.DataFrame(index=OASIS_analysis_df.index, columns=format_columns)\n",
    "    for idx in OASIS_analysis_df.index.values:    \n",
    "        row_str = \"%i; %i; %.1f ± %.1f [%.1f, %.1f]; %iF / %iM; %.1f ± %.1f [%.1f, %.1f]; 0: %i, 0.5: %i, 1: %i, 2:%i, 3:%i\" % tuple([OASIS_analysis_df.loc[idx, col] for col in columns])\n",
    "        row_list = row_str.split(';')\n",
    "        format_df.loc[idx] = row_list\n",
    "\n",
    "    format_df.index.name = None\n",
    "    display(format_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>subjects</th>\n",
       "      <th>scans</th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>MMSE</th>\n",
       "      <th>CDR</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>AD</th>\n",
       "      <td>78</td>\n",
       "      <td>78</td>\n",
       "      <td>75.6 ± 7.0 [62.0, 96.0]</td>\n",
       "      <td>43F / 35M</td>\n",
       "      <td>24.4 ± 4.3 [14.0, 30.0]</td>\n",
       "      <td>0: 0, 0.5: 56, 1: 20, 2:2, 3:0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CN</th>\n",
       "      <td>76</td>\n",
       "      <td>76</td>\n",
       "      <td>76.5 ± 8.4 [62.0, 94.0]</td>\n",
       "      <td>62F / 14M</td>\n",
       "      <td>29.0 ± 1.2 [25.0, 30.0]</td>\n",
       "      <td>0: 76, 0.5: 0, 1: 0, 2:0, 3:0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   subjects scans                       age         sex  \\\n",
       "AD       78    78   75.6 ± 7.0 [62.0, 96.0]   43F / 35M   \n",
       "CN       76    76   76.5 ± 8.4 [62.0, 94.0]   62F / 14M   \n",
       "\n",
       "                        MMSE                              CDR  \n",
       "AD   24.4 ± 4.3 [14.0, 30.0]   0: 0, 0.5: 56, 1: 20, 2:2, 3:0  \n",
       "CN   29.0 ± 1.2 [25.0, 30.0]    0: 76, 0.5: 0, 1: 0, 2:0, 3:0  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display_table(\"data/OASIS_analysis.tsv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is no significant bias on age anymore, but do you notice any other problems? \n",
    "\n",
    "<div class=\"alert alert-block alert-warning\">\n",
    "<b>Demographic bias:</b>\n",
    "    <p>There is still a difference in sex distribution and the network could learn a bias on sex such as \"women are cognitively normal\" and \"men are demented\". However, there are too few images in OASIS to continue removing sessions to equilibrate the groups.\n",
    "\n",
    "To check that such bias is not learnt, it is possible to run a logistic regression after training between sex and the predicted label to check if they are correlated.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split the data samples into training, validation and test sets\n",
    "\n",
    "Now that the labels have been extracted and possible biases have been identified, data have to be split in different sets. This step is essential to guarantee the independence of the final evaluation. \n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Definition of sets:</b><p>\n",
    "    In this notebook, data samples are divided between train, validation and test sets:\n",
    "<ul>\n",
    "    <li> The <b>train set</b> is used to update the weights, </li>\n",
    "    <li> The <b>validation set</b> is used to stop the training process and select the best model, </li>\n",
    "    <li> The <b>test set</b> is used after the end of the training process to perform an unbiased evaluation of the performance. </li>\n",
    "</ul>\n",
    "    <img src=\"./images/split.png\">\n",
    "    <p>Due to the k-fold validation procedure, k trainings are conducted according to the k training/validation pairs generated. This leads to k different models that are evaluated on the test set at the end. The final test performance is then the mean value of these k models.</p>\n",
    "</div>\n",
    "\n",
    "Tools that have been developed for this part are based on the guidelines of ([Varoquaux et al., 2017](https://doi.org/10.1016/j.neuroimage.2016.10.038))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the test set\n",
    "\n",
    "The test set is obtained by performing a single split obtained with `clinicadl tsvtool split`:\n",
    "```bash\n",
    "clinicadl tsvtool split <merged_tsv> <formatted_data_path>\n",
    "```\n",
    "where:\n",
    "- `merged_tsv` is the output file of the `clinica iotools merge-tsv` or `clinicadl tsvtool restrict` commands.\n",
    "- `formatted_data_path` is a folder containing one tsv file per label (output of `clinicadl tsvtool getlabels|split|kfold`).\n",
    "\n",
    "Each diagnostic label is split independently. Random splits are generated until there are non-significant differences between age and sex distributions between the test set and the train + validation set. Then three TSV files are written per label:\n",
    "\n",
    "- the baseline sessions of the test set,\n",
    "- the baseline sessions of the train + validation set,\n",
    "- the longitudinal sessions of the train + validation set.\n",
    "\n",
    "In OASIS there is no longitudinal follow-up, hence the last two TSV files are identical.\n",
    "\n",
    "Let's create a test set including 20 subjects per label:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!clinicadl tsvtool split data/labels_lists --n_test 20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The differences between populations of the train + validation and test sets can be assessed to check that there is no discrepancies between the two sets.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!clinicadl tsvtool analysis data/OASIS_BIDS.tsv data/labels_lists/train data/OASIS_trainval_analysis.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!clinicadl tsvtool analysis data/OASIS_BIDS.tsv data/labels_lists/test data/OASIS_test_analysis.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train + validation set\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>subjects</th>\n",
       "      <th>scans</th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>MMSE</th>\n",
       "      <th>CDR</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>AD</th>\n",
       "      <td>58</td>\n",
       "      <td>58</td>\n",
       "      <td>75.5 ± 7.1 [62.0, 96.0]</td>\n",
       "      <td>31F / 27M</td>\n",
       "      <td>24.3 ± 4.3 [14.0, 30.0]</td>\n",
       "      <td>0: 0, 0.5: 41, 1: 16, 2:1, 3:0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CN</th>\n",
       "      <td>56</td>\n",
       "      <td>56</td>\n",
       "      <td>76.4 ± 8.4 [62.0, 94.0]</td>\n",
       "      <td>46F / 10M</td>\n",
       "      <td>29.0 ± 1.2 [25.0, 30.0]</td>\n",
       "      <td>0: 56, 0.5: 0, 1: 0, 2:0, 3:0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   subjects scans                       age         sex  \\\n",
       "AD       58    58   75.5 ± 7.1 [62.0, 96.0]   31F / 27M   \n",
       "CN       56    56   76.4 ± 8.4 [62.0, 94.0]   46F / 10M   \n",
       "\n",
       "                        MMSE                              CDR  \n",
       "AD   24.3 ± 4.3 [14.0, 30.0]   0: 0, 0.5: 41, 1: 16, 2:1, 3:0  \n",
       "CN   29.0 ± 1.2 [25.0, 30.0]    0: 56, 0.5: 0, 1: 0, 2:0, 3:0  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test set\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>subjects</th>\n",
       "      <th>scans</th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>MMSE</th>\n",
       "      <th>CDR</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>AD</th>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "      <td>76.0 ± 6.8 [64.0, 90.0]</td>\n",
       "      <td>12F / 8M</td>\n",
       "      <td>24.6 ± 4.3 [15.0, 30.0]</td>\n",
       "      <td>0: 0, 0.5: 15, 1: 4, 2:1, 3:0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CN</th>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "      <td>76.8 ± 8.4 [64.0, 90.0]</td>\n",
       "      <td>16F / 4M</td>\n",
       "      <td>28.8 ± 1.3 [26.0, 30.0]</td>\n",
       "      <td>0: 20, 0.5: 0, 1: 0, 2:0, 3:0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   subjects scans                       age        sex  \\\n",
       "AD       20    20   76.0 ± 6.8 [64.0, 90.0]   12F / 8M   \n",
       "CN       20    20   76.8 ± 8.4 [64.0, 90.0]   16F / 4M   \n",
       "\n",
       "                        MMSE                             CDR  \n",
       "AD   24.6 ± 4.3 [15.0, 30.0]   0: 0, 0.5: 15, 1: 4, 2:1, 3:0  \n",
       "CN   28.8 ± 1.3 [26.0, 30.0]   0: 20, 0.5: 0, 1: 0, 2:0, 3:0  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"Train + validation set\")\n",
    "display_table(\"data/OASIS_trainval_analysis.tsv\")\n",
    "print(\"Test set\")\n",
    "display_table(\"data/OASIS_test_analysis.tsv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are not satisfied with these populations, you can relaunch the test or change the parameters used to evaluate the difference between the distributions: `p_val_threshold` and `t_val_threshold`.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Unique test set:</b>\n",
    "    <p>Only one test set was created in (<a href=\"https://www.sciencedirect.com/science/article/abs/pii/S1361841520300591\">Wen et al., 2020</a>) to evaluate the final performance of one model. This is because architecture search was performed on the training + validation sets. As this operation is very costly and/or is done mostly manually, it was not possible to do it several times.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the validation sets\n",
    "\n",
    "To better estimate the performance of the network, it is trained 5 times using a 5-fold cross-validation procedure. In this procedure, each sample is used once to validate and the other times to train the network. In the same way as for the single split, the TSV files can be processed with `clinicadl`:\n",
    "\n",
    "```bash\n",
    "clinicadl tsvtool kfold <formatted_data_path>\n",
    "```\n",
    "where `formatted_data_path` is a folder containing one tsv file per label (output of `clinicadl tsvtool getlabels|split|kfold`).\n",
    "\n",
    "In a similar way than for the test split, three tsv files are written per label **per split** for each set:\n",
    "\n",
    "- the baseline sessions of the validation set,\n",
    "- the baseline sessions of the train set,\n",
    "- the longitudinal sessions of the train set.\n",
    "\n",
    "Contrary to the test split, there is no attempt to control the similarity between the age and sex distributions. Indeed here we consider that averaging across the results of the 5 folds already reduces bias compared to a single data split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!clinicadl tsvtool kfold data/labels_lists/train --n_splits 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check the absence of data leakage\n",
    "\n",
    "In OASIS-1 there is no risk of data leakage due to the data split itself as there is only one session per subject. Also there is no MCI patients, hence there is no risk of data leakage during a transfer learning between a source task involving the MCI set and a target task involving at least one of its subsets (sMCI or pMCI). However for other datasets, it might be useful to check that there is no correlated data spread between the train and test sets.\n",
    "\n",
    "A script in `clinicadl` has been created to check that there was no data leakage after the split steps. More specifically it checks that:\n",
    "\n",
    "1. baseline datasets contain only one scan per subject,\n",
    "2. no intersection exists between train and test sets,\n",
    "3. MCI train subjects are absent from test sets of subcategories of MCI.\n",
    "    \n",
    "As it is not a common function, it has not been integrated to the general commandline. The next cell executes it on the splits generated in the previous sections."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from clinicadl.tools.tsv.test import run_test_suite\n",
    "\n",
    "# Run check for train+val / test split\n",
    "run_test_suite(\"./data/labels_lists\", n_splits=0, subset_name=\"test\")\n",
    "\n",
    "# Run check for train / validation splits\n",
    "run_test_suite(\"./data/labels_lists/train\", n_splits=5, subset_name=\"validation\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If no Error was raised then none of the three conditions was broken. It is now possible to use the train and the validation sets to perform a classification task, and then to evaluate correctly the performance of the classifier on the test set.\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\">\n",
    "<b>Data leakage:</b>\n",
    "    <p>Many procedures can cause data leakage and thus bias the performance, leading to impossible claims. It is crucial to check that the test set has not been contaminated by data that is correlated to the train and/or validation sets. You will find below examples of procedures that can lead to data leakage.</p>\n",
    "    <img src=\"./images/data_leakage.png\">\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}