stroke_starting_kit.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1><center>\n",
    "    <img src=\"https://raw.githubusercontent.com/ramp-kits/stroke/master/figs/logo_inria.png\" width=\"150px\"/><img src=\"https://raw.githubusercontent.com/ramp-kits/stroke/master/figs/logo_cds.png\" width=\"150px\" />\n",
    "</center>\n",
    "</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: center\">\n",
    "<img src=\"https://raw.githubusercontent.com/ramp-kits/stroke/master/figs/stroke_logo.png\" width=\"400px\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <center> RAMP: segmentation of the brain lesions </center>\n",
    "\n",
    "<b>Authors</b>: Alexandre Hutton$^{1}$, Maria Teleńczuk$^{2}$, Swetha Shanker$^{2}$, Guillaume Lemaitre$^{2}$, Alexandre Gramfort$^{2}$, Sook-Lei Liew$^{1}$\n",
    "\n",
    "$^{1}$: <a href=\"https://chan.usc.edu/npnl/\">Neural Plasticity & Neurorehabilitation Lab</a>  \n",
    "$^{2}$: <a href=\"http://www.datascience-paris-saclay.fr\">Paris-Saclay Center for Data Science</a> (Inria), France"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table of contents\n",
    "1. [Introduction](#Introduction)\n",
    "    1. [Motivation](#Motivation)\n",
    "    2. [Setup](#Setup)\n",
    "2. [Data Exploration](#data_exp)\n",
    "    1. [Visualize Data](#visualize)\n",
    "    1. [Paired Loading](#paired)\n",
    "    2. [Lesion Analysis](#lesion)\n",
    "3. [Creating Estimators](#prediction)\n",
    "4. [Scores](#scores)\n",
    "3. [Submission](#Submission)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Introduction <a class=\"anchor\" id=\"Introduction\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clinical/research motivation <a name=Motivation></a>\n",
    "\n",
    "Stroke is the leading cause of adult disability worldwide, with up to two-thirds of affected individuals experiencing long-term disabilities. Large-scale neuroimaging studies have shown promise in identifying robust biomarkers (e.g. measures of brain structure) of long-term stroke recovery following rehabilitation. However, analyzing large rehabilitation-related datasets is problematic due to barriers in accurate stroke lesion segmentation. Manually-traced lesions are currently the gold standard for lesion segmentation on T1-weighted MRIs, but require anatomical expertise and are labor-intensive. Further, manual segmentation is subjective with raters producing different results. While algorithms have been developed to automate this process, the resulting lesion masks often lack the accuracy needed to make them reliable information. Newer algorithms that employ machine-learning and deep learning techniques are promising avenues, but they require large, diverse datasets for training and testing and developing generalizable models. In this challenge, training can be performed on our public ATLAS 2.0 dataset, and testing is done with a private dataset comprised of multi-site data from the same sites as ATLAS 2.0. <BR>\n",
    "\n",
    "For more information refer to:\n",
    "<a href=\"http://fcon_1000.projects.nitrc.org/indi/retro/atlas.html\">Anatomical Tracings of Lesions After Stroke</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://www.sciencedirect.com/science/article/abs/pii/S1361841516301268\">1. Maier, Oskar, et al. \"ISLES 2015-A public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI.\" Medical image analysis 35 (2017): 250-269.</a><br>\n",
    "\n",
    "<a href=\"https://www.nature.com/articles/sdata201811\">2. Liew, Sook-Lei, et al. \"A large, open source dataset of stroke anatomical brain images and manual lesion segmentations.\" Scientific data 5 (2018): 180011.</a><br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objective of the challenge\n",
    "\n",
    "In this challenge you will be given 3D medical images (T1 MRI scans in `nii.gz` format) of stroke patients and the files with the corresponding lesions (binary masks) traced manually by experts. Your algorithm will be scored on the overlap between your prediction and the segmentation mask."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup <a name=Setup></a>\n",
    "### Prerequisites\n",
    "The following cell will install the required package dependencies, if necessary. You can examine the file, `requirements.txt`, included in the repo to view the list of dependencies.\n",
    "NOTE: Due to the structure of the challenge, libraries not included in `requirements.txt` will need to be added to be added via a pull request to the [GitHub repo](https://github.com/ramp-kits/stroke)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get this notebook running and test your models locally using `ramp-test` (from ramp-workflow), we recommend that you use the Anaconda or Miniconda Python distribution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Data exploration <a name=data_exp></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data download and reformat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will need four things to be able to get a local copy of the data:\n",
    "1. Downloaded archive.\n",
    "2. Encryption password.\n",
    "3. Decryption + tar extraction.\n",
    "4. Data reformatting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Archive download\n",
    "\n",
    "You can get a local (encrypted) copy of the data by going to the [NITRC website](http://fcon_1000.projects.nitrc.org/indi/retro/atlas_download.html) or running the following cell. The relevant code is commented to prevent accidentally downloading the (large) data. Make sure to uncomment the last line.\n",
    "\n",
    "Note that the data is approx. 15GB; ensure that you have enough space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import download_data\n",
    "\n",
    "# This will take some time; you can skip data validation by setting check_hash=False, but that is not recommended.\n",
    "### Uncomment this line to download the data:\n",
    "# download_data.data_fetch(check_hash=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Encryption password\n",
    "\n",
    "The data is encrypted; to obtain the password to decrypt the data, you'll need to agree to the terms of use [found on this form](https://docs.google.com/forms/d/e/1FAIpQLSclH8padHr9zwdQVx9YY_yeM_4OqD1OQFvYcYpAQKaqC6Vscg/viewform)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. OpenSSL + tar extraction\n",
    "\n",
    "If you are running either __Linux or MacOS__, you can run the following cell. You will be prompted for the password obtained in step 2; you can copy+paste it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import getpass, subprocess\n",
    "## Decrypt the data.\n",
    "# subprocess.call(['openssl', 'aes-256-cbc', '-md', 'sha256', \n",
    "#                  '-d', '-a', '-in',\n",
    "#                  'ATLAS_R2.0_encrypted.tar.gz', '-out', 'ATLAS_R2.0.tar.gz',\n",
    "#                  '-pass', f'pass:{getpass.getpass(\"Enter password\")}'])\n",
    " \n",
    "# subprocess.call(['tar', '-xzf', 'ATLAS_R2.0.tar.gz'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For __Windows__, you'll need to install OpenSSL, open the terminal, then run the following from that window:  \n",
    "`openssl aes-256-cbc -md sha256 -d -a -in ATLAS_R2.0_encrypted.tar.gz -out ATLAS_R2.0.tar.gz`  \n",
    "Next, extract the .tar.gz archive. You should be left with a directory called ATLAS_2 once it is extracted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Data reformatting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, the data provided via INDI is not compatible with PyBIDS, which is used throughout the challenge. You can run the following cell to correct the formatting; update the values as appropriate.\n",
    "WARNING: This will move the data files on your disk. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from stroke import indi_reformat\n",
    "## Uncomment the following lines to correct the formatting.\n",
    "#atlas_2_path = 'ATLAS_2'  # Assumes that it is in the current directory. Set this to the location on your drive.\n",
    "#destination = 'data'  # Assumes that we're in the stroke/ directory.\n",
    "#indi_reformat.bidsify_indi_atlas(atlas_path=atlas_2_path, destination_path=destination)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Viewing the data files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In your `data` folder you should now be able to find the `test` and `train` directories, each containing BIDS directories. Subject directories contain two files:\n",
    " - `sub-X_ses-Y_space-MNI152NLin2009aSym_T1w.nii.gz`\n",
    "     - Structural T1 that has been defaced, resampled, and normalized to MNI152NLin2009\n",
    " - `sub-X_ses-Y_space-MNI152NLin2009aSym_label-L_desc-T1lesion_mask.nii.gz`\n",
    "     - Lesion mask. '1' indicates the presence of a lesion, '0' otherwise.\n",
    "\n",
    "If you wish to view any of those files from outside of Python, you can use applications like [ITK-snap](http://www.itksnap.org/pmwiki/pmwiki.php). There, you can load the structural file as the main image and the lesion mask as a segmentation to overlap the two."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To load the images to Python, we will be using [PyBIDS](https://github.com/bids-standard/pybids). If you want to use a libarary which is not listed in `requirements.txt` or `extra_libraries.txt`, please make a pull request to the the [Stroke Lesion Segmentation Challenge repository on GitHub](https://github.com/ramp-kits/stroke) by adding the required library to [extra_libraries.txt](https://github.com/ramp-kits/stroke/blob/master/extra_libraries.txt) file)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize data <a name=visualize></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data provided with this challenge are 3D images. In this section, we'll show you how to view slices of the data.\n",
    "Since the data is in [BIDS](https://bids-specification.readthedocs.io/en/stable/), we can access the data fairly easily using [PyBIDS](https://github.com/bids-standard/pybids)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import bids\n",
    "from bids import BIDSLayout\n",
    "bids.config.set_option('extension_initial_dot', True)  # Warning suppression\n",
    "data = BIDSLayout('data/train', derivatives=True).derivatives['ATLAS']\n",
    "\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that there are 436 subjects in the training set with a total of 436 sessions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pick random sample\n",
    "import random\n",
    "subjects = data.entities['subject'].unique()\n",
    "sample = random.choice(subjects)\n",
    "\n",
    "structural = data.get(subject=sample, suffix='T1w')[0]\n",
    "mask = data.get(subject=sample, suffix='mask')[0]\n",
    "print(structural.filename)\n",
    "print(mask.filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from stroke.nii_slice import qc_slice\n",
    "from matplotlib import pyplot as plt\n",
    "plt.rcParams['figure.figsize'] = [14, 6]\n",
    "qc_slice(structural.path, mask_path=mask.path, nslices=5)\n",
    "\n",
    "# Beautify plot\n",
    "plt.text(0,0, f'{sample}', bbox={'facecolor': 'white'});  # Label which subject\n",
    "plt.tick_params(axis='both', labelbottom=False, labelleft=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above image, we see slices for each plane. The masked stroke is denoted in red, although the slices may not intersect the lesion mask for every subject (re-run the last two cells to display a new subject). Some preprocessing has already been done on the images; you can review the preprocessing steps [here](https://www.nature.com/articles/sdata201811).\n",
    "\n",
    "Due to the unreliable performance of non-linear registration with subjects with pathologies, the data was registered linearly. If you examine multiple subjects, you may notice that although the image dimensions are the same and the brains are roughly in the right area, there are differences between subjects (e.g., dimensions of the head, location of specific structures). This is typical of neuroimaging datasets.\n",
    "\n",
    "Before we look closer at the lesions, let's look at some of the image properties:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'Shape of structural scan: {structural.get_image().shape}')\n",
    "print(structural.get_image().header)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The previous cell displays some of the metadata about the file. You can [read here](https://brainder.org/2012/09/23/the-nifti-file-format/) for more information, but the shape of the data is most relevant.\n",
    "Every image contains 197x233x189 (8.6M) voxels. Consequently, it is not usually possible to load the entire dataset into memory at once. We will see the effects of this in later sections."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Paired Loading <a name=paired></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading individual files is fine for examining the data, but we need to ensure that our data and masks are paired, even if new sessions or modalities are added later. We've included a loader for this in `bids_loader.py`, as `BIDSLoader`. We can look at its docstrings to see the expected input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from stroke.bids_loader import BIDSLoader\n",
    "help(BIDSLoader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from stroke.bids_loader import BIDSLoader\n",
    "training_set = BIDSLoader(root_dir='data/train/',\n",
    "                          data_entities={'subject': '',\n",
    "                                         'session': '',\n",
    "                                         'suffix': 'T1w'},\n",
    "                          target_entities={'label': 'L',\n",
    "                                          'desc': 'T1lesion',\n",
    "                                          'suffix': 'mask'},\n",
    "                          data_derivatives_names=['ATLAS'],\n",
    "                          target_derivatives_names=['ATLAS'],\n",
    "                          label_names=['not lesion', 'lesion'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The object, `training_set` now has the data list paired with the target list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'Image: {training_set.data_list[0]}')\n",
    "print(f'Mask: {training_set.target_list[0]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can load the data/target into separate arrays:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data, target = training_set.load_sample(0)\n",
    "print(f'data shape: {data.shape}')\n",
    "print(f'target shape: {target.shape}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also load multiple images as a batch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data, target = training_set.load_batch([0,1,2])\n",
    "print(f'data shape: {data.shape}')\n",
    "print(f'target shape: {target.shape}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Along with `load_image_tuple` and `load_image_tuple_list`, these methods are used during RAMP's training to feed your estimator."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lesion analysis <a name=lesion></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, that we understand better the anatomical T1 images that we will work with, let's look at the lesion masks. We will visualize overlap of the two:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nilearn.image import load_img\n",
    "from nilearn import plotting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_overlap(path_to_t1, path_to_lesion, title='overlap'):\n",
    "    fig = plt.figure(1, figsize=(14, 5), frameon=False, dpi=50)\n",
    "    ax = plt.gca()\n",
    "    plotting.plot_roi(path_to_lesion, path_to_t1, title=title,\n",
    "                       axes=ax,\n",
    "                       draw_cross=False,\n",
    "                       annotate=False, \n",
    "                       cmap='autumn',  # the lesions will be shown in red\n",
    "                       cut_coords=(0, 0, 0)\n",
    "                       )\n",
    "plot_overlap(path_to_t1=structural.path,\n",
    "             path_to_lesion=mask.path,\n",
    "             title=f'overlap, {sample}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that lesions are heterogeneous: they appear in different places, have different sizes, and are different shapes.\n",
    "\n",
    "We can examine a number of subjects and get the overall lesions for each subject. Note that some subjects have multiple lesions, so the following value is the _lesion load_ rather than the lesion size. Even using the aggregate measure, we'll see a wide distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from stroke.stroke_config import bids_loader_train, bids_loader_test  # Default loaders for starting kit\n",
    "import numpy as np\n",
    "\n",
    "train_fraction = np.floor(len(bids_loader_train.target_list)/10)\n",
    "test_fraction = np.floor(len(bids_loader_train.target_list))\n",
    "subset_size = np.min([50, train_fraction, test_fraction])  # number of subjects to examine; reduce if it takes too long to load\n",
    "subset_size = int(subset_size)\n",
    "lesion_load_train = np.zeros((subset_size,))  # lesion load per subject\n",
    "lesion_load_test = np.zeros((subset_size,))\n",
    "\n",
    "# Load the data, compute the lesion load, store\n",
    "for i in range(subset_size):\n",
    "    target_tuple = bids_loader_train.target_list[i]\n",
    "    target = bids_loader_train.load_image_tuple(target_tuple)\n",
    "    lesion_load_train[i] = np.sum(target)\n",
    "for i in range(subset_size):\n",
    "    target_tuple = bids_loader_test.target_list[i]\n",
    "    target = bids_loader_test.load_image_tuple(target_tuple)\n",
    "    lesion_load_test[i] = np.sum(target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the distribution!\n",
    "from matplotlib import pyplot as plt\n",
    "plt.hist(np.log10(lesion_load_train), bins=10)  # Note the log scale\n",
    "plt.hist(np.log10(lesion_load_test), bins=10, alpha=0.5)\n",
    "plt.xlabel('log10 Lesion Load')\n",
    "plt.ylabel('Frequency of lesion load')\n",
    "plt.title('Distribution of lesion load in sets')\n",
    "plt.legend(['Training set', 'Testing set'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see in the distribution that even among the 100 subjects, the lesion load varies quite a bit across patients. The test set is a random subset from the same sites as the training set, but we can (qualitatively) observe different distribution between the two sets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sample prediction algorithms <a name=prediction></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This section will show you the expected format for your estimators. We'll first go through a sample estimator that defines the required properties and methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dummy solution (predict only 1s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The starter kit has a sample estimator in `submissions/sample/`. There, you'll find a file, `estimator.py`, which RAMP expects to have BIDSEstimator defined. The class is intentionally barebones, and is a good starting place for your own estimators.\n",
    "Let's take a look at the sample class and its methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from submissions.sample.estimator import BIDSEstimator  # Load estimator\n",
    "methods = [d for d in dir(BIDSEstimator) if not d.startswith('_')]  # Get methods; remove private ones.\n",
    "print(methods)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two of these methods, `get_params` and `set_params` are from `sklearn`'s `BIDSEstimator`. You don't need to implement these.\n",
    "  \n",
    "The fitting methods, `fit` and `fit_partial` are related to fitting your estimator to the data.  \n",
    "`fit` expects to be given the entirety of the data to fit. You won't be able to fit all the data in memory, and as such `fit` is not required.  \n",
    "`fit_partial` behaves similarly, but instead assumes that not all data is being presented at once. It is used in iterative fitting (e.g. stochastic gradient descent). This method is required.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(BIDSEstimator.fit_partial)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, the prediction methods, `predict` and `predict_proba` use input data and return a prediction.  \n",
    "Broadly, `predict` is expected to return the predicted class (i.e. argmax instead of softmax). In the case of our binary masks, we would expect predictions of either '0' or '1'.\n",
    "`predict_proba` is expected to return continuous values (i.e. softmax instead of argmax), which can be useful for model evaluation and calibration.\n",
    "\n",
    "Scoring functions will call the relevant prediction method for their requirements, but will cast the predictions into the right type (e.g. binary) before scoring. You can implement only `predict` and then have `predict_proba` simply return the results of `predict` without issue. Having your own implementation allows you to customize parameters such as thresholds, but is not required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(BIDSEstimator.predict_proba)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submitting to RAMP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before submitting to RAMP, you can test your solution locally to ensure that trivial errors (e.g. typos, path issues, etc.) are resolved. We can test a given submission using the `ramp` command that was installed in the virtual environment.  \n",
    "We'll use the following command:  \n",
    "`!ramp test --submission sample --quick-test`  \n",
    "The `!` signals that the command should be run on the command line instead of this notebook.  \n",
    "`ramp` is the command to be executed.  \n",
    "`test` is the first argument; it signals `ramp` to perform a local test.  \n",
    "`--submission sample` specifies which submission to run. You can have multiple potential submissions in the `submissions/` directory; this prevents `ramp` from running all of them.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ramp test --quick-test --submission sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the results are not very good, but that is expected: our estimator completely ignores the data!\n",
    "\n",
    "RAMP will automatically perform 5-fold cross-validation and report the Sørensen–Dice for each of the folds and report the mean across the folds. Bagging of the results has been disabled; the output can be ignored."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Scores used <a name=scores></a>\n",
    "- [__Sørensen–Dice__ coefficient](https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient): measures the overlap between the prediction and the ground truth. The measure is bounded between 0 and 1, with 0 indicating no overlap and 1 indicating perfect overlap.\n",
    "\n",
    "Other scores will be added as the challenge is released, and will likely include precision and recall.\n",
    "You can find the scoring metrics in the `scoring.py` file. If you want to use the score to evaluate your own results, use the `calc_score` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from stroke.scoring import DiceCoeff\n",
    "help(DiceCoeff)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(DiceCoeff.calc_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the reference, if you wish to view previous medical image segmentation challenges:\n",
    "    \n",
    "- [Cada (Cerebral Aneurysm Detection) grand-challenge](https://cada.grand-challenge.org/)\n",
    "- [Augmented Segmentation of Coronary Arteries](https://asoca.grand-challenge.org/)\n",
    "- [TN (Thyroid Nodule Segmentation and Classification) SCUI2020](https://tn-scui2020.grand-challenge.org/)\n",
    "- [Pneumothorax Segmentation SIIM-ACR](https://www.kaggle.com/c/siim-acr-pneumothorax-segmentation)\n",
    "- [Brats brain tumor segmentation Challenge 2020](https://www.med.upenn.edu/cbica/brats2020/data.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submission"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When your estimator in `estimator.py` is ready, place it in the `submissions/` directory in its own, unique directory (e.g., the path should be `submissions/my_estimator/estimator.py`).\n",
    "\n",
    "You can then test your submission locally using the command:\n",
    "\n",
    "`ramp test --submission <your submission folder name>`  \n",
    "Example:\n",
    "`ramp test --submission my_estimator`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more information on how to submit your code on [ramp.studio](https://ramp.studio/), refer to the [online documentation](https://paris-saclay-cds.github.io/ramp-docs/ramp-workflow/stable/using_kits.html)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "stroke_venv",
   "language": "python",
   "name": "stroke_venv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}