notebooks/submission_tutorial/2_behind_the_scenes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# How we evaluate the competition entries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### This notebook will show how we calculate the competition scores on our website\n",
    "\n",
    "Here, we illustrate in detail how we use the submission file to compute the scores of the competition. We do this by taking one of the datasets that is not part of the competition (i.e. one of the \"pre-training\" datasets). All of the \"pre-training\" datasets also have a test set (i.e. the `live_test` set), and the responses of all neurons to these test set images are present. Thus, you can test your model by training on one or all of the \"pre-training\" recordings and see how the model performs on the test set.\n",
    "\n",
    "Here, we use one of the sets to show how we extract the responses from the test set into a `ground_truth` file. And how we use the submitted files and the ground truth file to calculate the scores.\n",
    "\n",
    "In detail, this notebooks includes these steps:\n",
    "- we first load a pretrained model\n",
    "- we select a pre-training dataset, which is not part of the competition, and treat the \"test\" set in it as if it was part of the competition track\n",
    "- this example then illustrates the complete process how the ground truth responses are extracted, and how the scores are getting calculated between ground truth and the submitted responses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import collections.abc\n",
    "#hyper needs the four following aliases to be done manually.\n",
    "collections.Iterable = collections.abc.Iterable\n",
    "collections.Mapping = collections.abc.Mapping\n",
    "collections.MutableSet = collections.abc.MutableSet\n",
    "collections.MutableMapping = collections.abc.MutableMapping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import os\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "from nnfabrik.builder import get_data, get_model, get_trainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c:\\Users\\hp\\sensorium\n"
     ]
    }
   ],
   "source": [
    "%cd ../../"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Get dataloader and model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[1;32mc:\\Users\\hp\\sensorium\\notebooks\\submission_tutorial\\2_behind_the_scenes.ipynb Cell 8\u001b[0m in \u001b[0;36m<cell line: 14>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      <a href='vscode-notebook-cell:/c%3A/Users/hp/sensorium/notebooks/submission_tutorial/2_behind_the_scenes.ipynb#W5sZmlsZQ%3D%3D?line=4'>5</a>\u001b[0m dataset_fn \u001b[39m=\u001b[39m \u001b[39m'\u001b[39m\u001b[39msensorium.datasets.static_loaders\u001b[39m\u001b[39m'\u001b[39m\n\u001b[0;32m      <a href='vscode-notebook-cell:/c%3A/Users/hp/sensorium/notebooks/submission_tutorial/2_behind_the_scenes.ipynb#W5sZmlsZQ%3D%3D?line=5'>6</a>\u001b[0m dataset_config \u001b[39m=\u001b[39m {\u001b[39m'\u001b[39m\u001b[39mpaths\u001b[39m\u001b[39m'\u001b[39m: filenames,\n\u001b[0;32m      <a href='vscode-notebook-cell:/c%3A/Users/hp/sensorium/notebooks/submission_tutorial/2_behind_the_scenes.ipynb#W5sZmlsZQ%3D%3D?line=6'>7</a>\u001b[0m                  \u001b[39m'\u001b[39m\u001b[39mnormalize\u001b[39m\u001b[39m'\u001b[39m: \u001b[39mTrue\u001b[39;00m,\n\u001b[0;32m      <a href='vscode-notebook-cell:/c%3A/Users/hp/sensorium/notebooks/submission_tutorial/2_behind_the_scenes.ipynb#W5sZmlsZQ%3D%3D?line=7'>8</a>\u001b[0m                  \u001b[39m'\u001b[39m\u001b[39minclude_behavior\u001b[39m\u001b[39m'\u001b[39m: \u001b[39mFalse\u001b[39;00m,\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m     <a href='vscode-notebook-cell:/c%3A/Users/hp/sensorium/notebooks/submission_tutorial/2_behind_the_scenes.ipynb#W5sZmlsZQ%3D%3D?line=10'>11</a>\u001b[0m                  \u001b[39m'\u001b[39m\u001b[39mscale\u001b[39m\u001b[39m'\u001b[39m:\u001b[39m.25\u001b[39m,\n\u001b[0;32m     <a href='vscode-notebook-cell:/c%3A/Users/hp/sensorium/notebooks/submission_tutorial/2_behind_the_scenes.ipynb#W5sZmlsZQ%3D%3D?line=11'>12</a>\u001b[0m                  }\n\u001b[1;32m---> <a href='vscode-notebook-cell:/c%3A/Users/hp/sensorium/notebooks/submission_tutorial/2_behind_the_scenes.ipynb#W5sZmlsZQ%3D%3D?line=13'>14</a>\u001b[0m dataloaders \u001b[39m=\u001b[39m get_data(dataset_fn, dataset_config)\n\u001b[0;32m     <a href='vscode-notebook-cell:/c%3A/Users/hp/sensorium/notebooks/submission_tutorial/2_behind_the_scenes.ipynb#W5sZmlsZQ%3D%3D?line=15'>16</a>\u001b[0m \u001b[39m# get model\u001b[39;00m\n\u001b[0;32m     <a href='vscode-notebook-cell:/c%3A/Users/hp/sensorium/notebooks/submission_tutorial/2_behind_the_scenes.ipynb#W5sZmlsZQ%3D%3D?line=16'>17</a>\u001b[0m model_fn \u001b[39m=\u001b[39m \u001b[39m'\u001b[39m\u001b[39msensorium.models.stacked_core_full_gauss_readout\u001b[39m\u001b[39m'\u001b[39m\n",
      "File \u001b[1;32m~\\AppData\\Roaming\\Python\\Python310\\site-packages\\nnfabrik\\builder.py:102\u001b[0m, in \u001b[0;36mget_data\u001b[1;34m(dataset_fn, dataset_config)\u001b[0m\n\u001b[0;32m     99\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39misinstance\u001b[39m(dataset_fn, \u001b[39mstr\u001b[39m):\n\u001b[0;32m    100\u001b[0m     dataset_fn \u001b[39m=\u001b[39m resolve_data(dataset_fn)\n\u001b[1;32m--> 102\u001b[0m \u001b[39mreturn\u001b[39;00m dataset_fn(\u001b[39m*\u001b[39m\u001b[39m*\u001b[39mdataset_config)\n",
      "File \u001b[1;32mc:\\Users\\hp\\sensorium\\sensorium\\datasets\\mouse_loaders.py:362\u001b[0m, in \u001b[0;36mstatic_loaders\u001b[1;34m(paths, batch_size, seed, areas, layers, tier, neuron_ids, neuron_n, exclude_neuron_n, neuron_base_seed, image_ids, image_n, image_base_seed, cuda, normalize, include_behavior, add_behavior_as_channels, exclude, select_input_channel, file_tree, image_condition, inputs_mean, inputs_std, scale, include_eye_position, add_eye_pos_as_channels, include_trial_info_keys, overwrite_data_path, include_px_position, image_reshape_list, trial_idx_selection)\u001b[0m\n\u001b[0;32m    359\u001b[0m \u001b[39mif\u001b[39;00m (overwrite_data_path) \u001b[39mand\u001b[39;00m (os\u001b[39m.\u001b[39mpath\u001b[39m.\u001b[39mexists(basepath)):\n\u001b[0;32m    360\u001b[0m     path \u001b[39m=\u001b[39m os\u001b[39m.\u001b[39mpath\u001b[39m.\u001b[39mjoin(basepath, path)\n\u001b[1;32m--> 362\u001b[0m out \u001b[39m=\u001b[39m static_loader(\n\u001b[0;32m    363\u001b[0m     path,\n\u001b[0;32m    364\u001b[0m     batch_size,\n\u001b[0;32m    365\u001b[0m     areas\u001b[39m=\u001b[39;49mareas,\n\u001b[0;32m    366\u001b[0m     layers\u001b[39m=\u001b[39;49mlayers,\n\u001b[0;32m    367\u001b[0m     cuda\u001b[39m=\u001b[39;49mcuda,\n\u001b[0;32m    368\u001b[0m     tier\u001b[39m=\u001b[39;49mtier,\n\u001b[0;32m    369\u001b[0m     get_key\u001b[39m=\u001b[39;49m\u001b[39mTrue\u001b[39;49;00m,\n\u001b[0;32m    370\u001b[0m     neuron_ids\u001b[39m=\u001b[39;49mneuron_id,\n\u001b[0;32m    371\u001b[0m     neuron_n\u001b[39m=\u001b[39;49mneuron_n,\n\u001b[0;32m    372\u001b[0m     exclude_neuron_n\u001b[39m=\u001b[39;49mexclude_neuron_n,\n\u001b[0;32m    373\u001b[0m     neuron_base_seed\u001b[39m=\u001b[39;49mneuron_base_seed,\n\u001b[0;32m    374\u001b[0m     image_ids\u001b[39m=\u001b[39;49mimage_id,\n\u001b[0;32m    375\u001b[0m     image_n\u001b[39m=\u001b[39;49mimage_n,\n\u001b[0;32m    376\u001b[0m     image_base_seed\u001b[39m=\u001b[39;49mimage_base_seed,\n\u001b[0;32m    377\u001b[0m     normalize\u001b[39m=\u001b[39;49mnormalize,\n\u001b[0;32m    378\u001b[0m     include_behavior\u001b[39m=\u001b[39;49minclude_behavior,\n\u001b[0;32m    379\u001b[0m     add_behavior_as_channels\u001b[39m=\u001b[39;49madd_behavior_as_channels,\n\u001b[0;32m    380\u001b[0m     exclude\u001b[39m=\u001b[39;49mexclude,\n\u001b[0;32m    381\u001b[0m     select_input_channel\u001b[39m=\u001b[39;49mselect_input_channel,\n\u001b[0;32m    382\u001b[0m     file_tree\u001b[39m=\u001b[39;49mfile_tree,\n\u001b[0;32m    383\u001b[0m     image_condition\u001b[39m=\u001b[39;49mimage_condition,\n\u001b[0;32m    384\u001b[0m     inputs_mean\u001b[39m=\u001b[39;49minputs_mean,\n\u001b[0;32m    385\u001b[0m     inputs_std\u001b[39m=\u001b[39;49minputs_std,\n\u001b[0;32m    386\u001b[0m     scale\u001b[39m=\u001b[39;49mscale,\n\u001b[0;32m    387\u001b[0m     include_eye_position\u001b[39m=\u001b[39;49minclude_eye_position,\n\u001b[0;32m    388\u001b[0m     add_eye_pos_as_channels\u001b[39m=\u001b[39;49madd_eye_pos_as_channels,\n\u001b[0;32m    389\u001b[0m     include_trial_info_keys\u001b[39m=\u001b[39;49minclude_trial_info_keys,\n\u001b[0;32m    390\u001b[0m     include_px_position\u001b[39m=\u001b[39;49minclude_px_position,\n\u001b[0;32m    391\u001b[0m     image_reshape_list\u001b[39m=\u001b[39;49mimage_reshape_list,\n\u001b[0;32m    392\u001b[0m     trial_idx_selection\u001b[39m=\u001b[39;49mtrial_idx_selection,\n\u001b[0;32m    393\u001b[0m )\n\u001b[0;32m    394\u001b[0m \u001b[39mfor\u001b[39;00m k \u001b[39min\u001b[39;00m dls:\n\u001b[0;32m    395\u001b[0m     dls[k][out[\u001b[39m0\u001b[39m]] \u001b[39m=\u001b[39m out[\u001b[39m1\u001b[39m][k]\n",
      "File \u001b[1;32mc:\\Users\\hp\\sensorium\\sensorium\\datasets\\mouse_loaders.py:122\u001b[0m, in \u001b[0;36mstatic_loader\u001b[1;34m(path, batch_size, areas, layers, tier, neuron_ids, neuron_n, exclude_neuron_n, neuron_base_seed, image_ids, image_n, image_base_seed, get_key, cuda, normalize, exclude, include_behavior, add_behavior_as_channels, select_input_channel, file_tree, image_condition, inputs_mean, inputs_std, scale, include_eye_position, add_eye_pos_as_channels, include_trial_info_keys, include_px_position, image_reshape_list, trial_idx_selection)\u001b[0m\n\u001b[0;32m    119\u001b[0m     data_keys\u001b[39m.\u001b[39mextend(include_trial_info_keys)\n\u001b[0;32m    121\u001b[0m \u001b[39mif\u001b[39;00m file_tree:\n\u001b[1;32m--> 122\u001b[0m     dat \u001b[39m=\u001b[39m FileTreeDataset(path, \u001b[39m*\u001b[39;49mdata_keys)\n\u001b[0;32m    123\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[0;32m    124\u001b[0m     dat \u001b[39m=\u001b[39m StaticImageSet(path, \u001b[39m*\u001b[39mdata_keys)\n",
      "File \u001b[1;32m~\\AppData\\Roaming\\Python\\Python310\\site-packages\\neuralpredictors\\data\\datasets\\base.py:317\u001b[0m, in \u001b[0;36mFileTreeDatasetBase.__init__\u001b[1;34m(self, dirname, transforms, use_cache, output_rename, output_dict, *data_keys)\u001b[0m\n\u001b[0;32m    315\u001b[0m \u001b[39mif\u001b[39;00m dirname\u001b[39m.\u001b[39mendswith(\u001b[39m\"\u001b[39m\u001b[39m.zip\u001b[39m\u001b[39m\"\u001b[39m):\n\u001b[0;32m    316\u001b[0m     \u001b[39mif\u001b[39;00m \u001b[39mnot\u001b[39;00m Path(dirname[:\u001b[39m-\u001b[39m\u001b[39m4\u001b[39m])\u001b[39m.\u001b[39mexists():\n\u001b[1;32m--> 317\u001b[0m         \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49munzip(dirname, Path(dirname)\u001b[39m.\u001b[39;49mabsolute()\u001b[39m.\u001b[39;49mparent)\n\u001b[0;32m    318\u001b[0m     \u001b[39melse\u001b[39;00m:\n\u001b[0;32m    319\u001b[0m         logger\u001b[39m.\u001b[39minfo(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39m{\u001b[39;00mdirname[:\u001b[39m-\u001b[39m\u001b[39m4\u001b[39m]\u001b[39m}\u001b[39;00m\u001b[39m exists already. Not unpacking \u001b[39m\u001b[39m{\u001b[39;00mdirname\u001b[39m}\u001b[39;00m\u001b[39m\"\u001b[39m)\n",
      "File \u001b[1;32m~\\AppData\\Roaming\\Python\\Python310\\site-packages\\neuralpredictors\\data\\datasets\\statics\\filetree.py:55\u001b[0m, in \u001b[0;36mFileTreeDataset.unzip\u001b[1;34m(self, filename, path)\u001b[0m\n\u001b[0;32m     53\u001b[0m logger\u001b[39m.\u001b[39minfo(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mUnzipping \u001b[39m\u001b[39m{\u001b[39;00mfilename\u001b[39m}\u001b[39;00m\u001b[39m into \u001b[39m\u001b[39m{\u001b[39;00mpath\u001b[39m}\u001b[39;00m\u001b[39m\"\u001b[39m)\n\u001b[0;32m     54\u001b[0m \u001b[39mwith\u001b[39;00m ZipFile(filename, \u001b[39m\"\u001b[39m\u001b[39mr\u001b[39m\u001b[39m\"\u001b[39m) \u001b[39mas\u001b[39;00m zip_obj:\n\u001b[1;32m---> 55\u001b[0m     zip_obj\u001b[39m.\u001b[39;49mextractall(path)\n",
      "File \u001b[1;32mc:\\Program Files\\Python310\\lib\\zipfile.py:1634\u001b[0m, in \u001b[0;36mZipFile.extractall\u001b[1;34m(self, path, members, pwd)\u001b[0m\n\u001b[0;32m   1631\u001b[0m     path \u001b[39m=\u001b[39m os\u001b[39m.\u001b[39mfspath(path)\n\u001b[0;32m   1633\u001b[0m \u001b[39mfor\u001b[39;00m zipinfo \u001b[39min\u001b[39;00m members:\n\u001b[1;32m-> 1634\u001b[0m     \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_extract_member(zipinfo, path, pwd)\n",
      "File \u001b[1;32mc:\\Program Files\\Python310\\lib\\zipfile.py:1687\u001b[0m, in \u001b[0;36mZipFile._extract_member\u001b[1;34m(self, member, targetpath, pwd)\u001b[0m\n\u001b[0;32m   1684\u001b[0m         os\u001b[39m.\u001b[39mmkdir(targetpath)\n\u001b[0;32m   1685\u001b[0m     \u001b[39mreturn\u001b[39;00m targetpath\n\u001b[1;32m-> 1687\u001b[0m \u001b[39mwith\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mopen(member, pwd\u001b[39m=\u001b[39mpwd) \u001b[39mas\u001b[39;00m source, \\\n\u001b[0;32m   1688\u001b[0m      \u001b[39mopen\u001b[39m(targetpath, \u001b[39m\"\u001b[39m\u001b[39mwb\u001b[39m\u001b[39m\"\u001b[39m) \u001b[39mas\u001b[39;00m target:\n\u001b[0;32m   1689\u001b[0m     shutil\u001b[39m.\u001b[39mcopyfileobj(source, target)\n\u001b[0;32m   1691\u001b[0m \u001b[39mreturn\u001b[39;00m targetpath\n",
      "\u001b[1;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "#get dataloader\n",
    "basepath = \"notebooks/data/\"\n",
    "filenames = [os.path.join(basepath, file) for file in os.listdir(basepath) if \".zip\" in file ]\n",
    "\n",
    "dataset_fn = 'sensorium.datasets.static_loaders'\n",
    "dataset_config = {'paths': filenames,\n",
    "                 'normalize': True,\n",
    "                 'include_behavior': False,\n",
    "                 'include_eye_position': False,\n",
    "                 'batch_size': 128,\n",
    "                 'scale':.25,\n",
    "                 }\n",
    "\n",
    "dataloaders = get_data(dataset_fn, dataset_config)\n",
    "\n",
    "# get model\n",
    "model_fn = 'sensorium.models.stacked_core_full_gauss_readout'\n",
    "model_config = {'pad_input': False,\n",
    "  'layers': 4,\n",
    "  'input_kern': 9,\n",
    "  'gamma_input': 6.3831,\n",
    "  'gamma_readout': 0.0076,\n",
    "  'hidden_kern': 7,\n",
    "  'hidden_channels': 64,\n",
    "  'depth_separable': True,\n",
    "  'grid_mean_predictor': {'type': 'cortex',\n",
    "   'input_dimensions': 2,\n",
    "   'hidden_layers': 1,\n",
    "   'hidden_features': 30,\n",
    "   'final_tanh': True},\n",
    "  'init_sigma': 0.1,\n",
    "  'init_mu_range': 0.3,\n",
    "  'gauss_type': 'full',\n",
    "  'shifter': False,\n",
    "  'stack': -1,\n",
    "}\n",
    "\n",
    "model = get_model(model_fn=model_fn,\n",
    "                  model_config=model_config,\n",
    "                  dataloaders=dataloaders,\n",
    "                  seed=42,)\n",
    "\n",
    "# load model weights\n",
    "model.load_state_dict(torch.load(\"notebooks/model_tutorial/model_checkpoints/pretrained/generalization_model.pth\"));\n",
    "model.eval();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# How we calculate the competition scores behind the scenes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "- we are withholding the ground truth neuronal responses to the test set images in the actual competition\n",
    "- here we show \n",
    " - how we extract the ground truth responses from the demo dataset (where the test set responses are present)\n",
    " - how the metrics of the competition are calculated\n",
    "\n",
    "The following steps are necessary:\n",
    "1. pick a dataset\n",
    "2. generate a file that contains the ground truth responses to the test set\n",
    "3. generate a submission file that contains the predictions\n",
    "4. Calculate the performance metrics based on these 2 files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# !! Important !!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Our grund truth file is storing **standardized responses**, meaning the responses of each neuron normalized by its own STD. Our dataloader is automatically normalizing the images and responses, and we encourage you to use our DataLoader and our submission API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# import the API from the competition repo\n",
    "from sensorium.utility import submission"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 1) Example Dataset:'21067-10-18' from the pre-training recordings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "filename = ['../data/static21067-10-18-GrayImageNet-94c6ff995dac583098847cfecd43e7b6.zip',]\n",
    "\n",
    "dataset_name = \"21067-10-18\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 2) Generate Ground Truth File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'submission' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32mc:\\Users\\hp\\sensorium\\notebooks\\submission_tutorial\\2_behind_the_scenes.ipynb Cell 18\u001b[0m in \u001b[0;36m<cell line: 4>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      <a href='vscode-notebook-cell:/c%3A/Users/hp/sensorium/notebooks/submission_tutorial/2_behind_the_scenes.ipynb#X21sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m \u001b[39m# we load the dataset which contains the held-out \"test\" responses, and save them in the .csv format\u001b[39;00m\n\u001b[0;32m      <a href='vscode-notebook-cell:/c%3A/Users/hp/sensorium/notebooks/submission_tutorial/2_behind_the_scenes.ipynb#X21sZmlsZQ%3D%3D?line=1'>2</a>\u001b[0m \u001b[39m# for the demo dataset that we provide here, these \"test\" responses are present\u001b[39;00m\n\u001b[1;32m----> <a href='vscode-notebook-cell:/c%3A/Users/hp/sensorium/notebooks/submission_tutorial/2_behind_the_scenes.ipynb#X21sZmlsZQ%3D%3D?line=3'>4</a>\u001b[0m submission\u001b[39m.\u001b[39mgenerate_ground_truth_file(filename\u001b[39m=\u001b[39mfilename,\n\u001b[0;32m      <a href='vscode-notebook-cell:/c%3A/Users/hp/sensorium/notebooks/submission_tutorial/2_behind_the_scenes.ipynb#X21sZmlsZQ%3D%3D?line=4'>5</a>\u001b[0m                                       path\u001b[39m=\u001b[39m\u001b[39m'\u001b[39m\u001b[39m./ground_truth_files/\u001b[39m\u001b[39m'\u001b[39m,\n\u001b[0;32m      <a href='vscode-notebook-cell:/c%3A/Users/hp/sensorium/notebooks/submission_tutorial/2_behind_the_scenes.ipynb#X21sZmlsZQ%3D%3D?line=5'>6</a>\u001b[0m                                       tier\u001b[39m=\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mtest\u001b[39m\u001b[39m\"\u001b[39m)\n",
      "\u001b[1;31mNameError\u001b[0m: name 'submission' is not defined"
     ]
    }
   ],
   "source": [
    "# we load the dataset which contains the held-out \"test\" responses, and save them in the .csv format\n",
    "# for the demo dataset that we provide here, these \"test\" responses are present\n",
    "\n",
    "submission.generate_ground_truth_file(filename=filename,\n",
    "                                      path='./ground_truth_files/',\n",
    "                                      tier=\"test\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "##### Inspect the Ground Truth File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c:\\Users\\hp\\sensorium\n"
     ]
    }
   ],
   "source": [
    "%cd ./hp/sensorium"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Volume in drive C is Windows\n",
      " Volume Serial Number is 46C8-E9DC\n",
      "\n",
      " Directory of c:\\Users\n",
      "\n",
      "18-08-2022  03:15    <DIR>          .\n",
      "23-08-2022  14:51    <DIR>          hp\n",
      "18-08-2022  16:43    <DIR>          Public\n",
      "               0 File(s)              0 bytes\n",
      "               3 Dir(s)  36,525,690,880 bytes free\n"
     ]
    }
   ],
   "source": [
    "%ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: 'notebooks/submission_tutorial/ground_truth_files/ground_truth_file_test.csv'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[1;32mc:\\Users\\hp\\sensorium\\notebooks\\submission_tutorial\\2_behind_the_scenes.ipynb Cell 21\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> <a href='vscode-notebook-cell:/c%3A/Users/hp/sensorium/notebooks/submission_tutorial/2_behind_the_scenes.ipynb#X23sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m data \u001b[39m=\u001b[39m pd\u001b[39m.\u001b[39;49mread_csv(\u001b[39m'\u001b[39;49m\u001b[39mnotebooks/submission_tutorial/ground_truth_files/ground_truth_file_test.csv\u001b[39;49m\u001b[39m'\u001b[39;49m)\n\u001b[0;32m      <a href='vscode-notebook-cell:/c%3A/Users/hp/sensorium/notebooks/submission_tutorial/2_behind_the_scenes.ipynb#X23sZmlsZQ%3D%3D?line=1'>2</a>\u001b[0m \u001b[39mlen\u001b[39m(data\u001b[39m.\u001b[39mloc[\u001b[39m0\u001b[39m]\u001b[39m.\u001b[39mat[\u001b[39m\"\u001b[39m\u001b[39mresponses\u001b[39m\u001b[39m\"\u001b[39m])\n",
      "File \u001b[1;32m~\\AppData\\Roaming\\Python\\Python310\\site-packages\\pandas\\util\\_decorators.py:311\u001b[0m, in \u001b[0;36mdeprecate_nonkeyword_arguments.<locals>.decorate.<locals>.wrapper\u001b[1;34m(*args, **kwargs)\u001b[0m\n\u001b[0;32m    305\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mlen\u001b[39m(args) \u001b[39m>\u001b[39m num_allow_args:\n\u001b[0;32m    306\u001b[0m     warnings\u001b[39m.\u001b[39mwarn(\n\u001b[0;32m    307\u001b[0m         msg\u001b[39m.\u001b[39mformat(arguments\u001b[39m=\u001b[39marguments),\n\u001b[0;32m    308\u001b[0m         \u001b[39mFutureWarning\u001b[39;00m,\n\u001b[0;32m    309\u001b[0m         stacklevel\u001b[39m=\u001b[39mstacklevel,\n\u001b[0;32m    310\u001b[0m     )\n\u001b[1;32m--> 311\u001b[0m \u001b[39mreturn\u001b[39;00m func(\u001b[39m*\u001b[39margs, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs)\n",
      "File \u001b[1;32m~\\AppData\\Roaming\\Python\\Python310\\site-packages\\pandas\\io\\parsers\\readers.py:680\u001b[0m, in \u001b[0;36mread_csv\u001b[1;34m(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, encoding_errors, dialect, error_bad_lines, warn_bad_lines, on_bad_lines, delim_whitespace, low_memory, memory_map, float_precision, storage_options)\u001b[0m\n\u001b[0;32m    665\u001b[0m kwds_defaults \u001b[39m=\u001b[39m _refine_defaults_read(\n\u001b[0;32m    666\u001b[0m     dialect,\n\u001b[0;32m    667\u001b[0m     delimiter,\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m    676\u001b[0m     defaults\u001b[39m=\u001b[39m{\u001b[39m\"\u001b[39m\u001b[39mdelimiter\u001b[39m\u001b[39m\"\u001b[39m: \u001b[39m\"\u001b[39m\u001b[39m,\u001b[39m\u001b[39m\"\u001b[39m},\n\u001b[0;32m    677\u001b[0m )\n\u001b[0;32m    678\u001b[0m kwds\u001b[39m.\u001b[39mupdate(kwds_defaults)\n\u001b[1;32m--> 680\u001b[0m \u001b[39mreturn\u001b[39;00m _read(filepath_or_buffer, kwds)\n",
      "File \u001b[1;32m~\\AppData\\Roaming\\Python\\Python310\\site-packages\\pandas\\io\\parsers\\readers.py:575\u001b[0m, in \u001b[0;36m_read\u001b[1;34m(filepath_or_buffer, kwds)\u001b[0m\n\u001b[0;32m    572\u001b[0m _validate_names(kwds\u001b[39m.\u001b[39mget(\u001b[39m\"\u001b[39m\u001b[39mnames\u001b[39m\u001b[39m\"\u001b[39m, \u001b[39mNone\u001b[39;00m))\n\u001b[0;32m    574\u001b[0m \u001b[39m# Create the parser.\u001b[39;00m\n\u001b[1;32m--> 575\u001b[0m parser \u001b[39m=\u001b[39m TextFileReader(filepath_or_buffer, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwds)\n\u001b[0;32m    577\u001b[0m \u001b[39mif\u001b[39;00m chunksize \u001b[39mor\u001b[39;00m iterator:\n\u001b[0;32m    578\u001b[0m     \u001b[39mreturn\u001b[39;00m parser\n",
      "File \u001b[1;32m~\\AppData\\Roaming\\Python\\Python310\\site-packages\\pandas\\io\\parsers\\readers.py:934\u001b[0m, in \u001b[0;36mTextFileReader.__init__\u001b[1;34m(self, f, engine, **kwds)\u001b[0m\n\u001b[0;32m    931\u001b[0m     \u001b[39mself\u001b[39m\u001b[39m.\u001b[39moptions[\u001b[39m\"\u001b[39m\u001b[39mhas_index_names\u001b[39m\u001b[39m\"\u001b[39m] \u001b[39m=\u001b[39m kwds[\u001b[39m\"\u001b[39m\u001b[39mhas_index_names\u001b[39m\u001b[39m\"\u001b[39m]\n\u001b[0;32m    933\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mhandles: IOHandles \u001b[39m|\u001b[39m \u001b[39mNone\u001b[39;00m \u001b[39m=\u001b[39m \u001b[39mNone\u001b[39;00m\n\u001b[1;32m--> 934\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_engine \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_make_engine(f, \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mengine)\n",
      "File \u001b[1;32m~\\AppData\\Roaming\\Python\\Python310\\site-packages\\pandas\\io\\parsers\\readers.py:1218\u001b[0m, in \u001b[0;36mTextFileReader._make_engine\u001b[1;34m(self, f, engine)\u001b[0m\n\u001b[0;32m   1214\u001b[0m     mode \u001b[39m=\u001b[39m \u001b[39m\"\u001b[39m\u001b[39mrb\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[0;32m   1215\u001b[0m \u001b[39m# error: No overload variant of \"get_handle\" matches argument types\u001b[39;00m\n\u001b[0;32m   1216\u001b[0m \u001b[39m# \"Union[str, PathLike[str], ReadCsvBuffer[bytes], ReadCsvBuffer[str]]\"\u001b[39;00m\n\u001b[0;32m   1217\u001b[0m \u001b[39m# , \"str\", \"bool\", \"Any\", \"Any\", \"Any\", \"Any\", \"Any\"\u001b[39;00m\n\u001b[1;32m-> 1218\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mhandles \u001b[39m=\u001b[39m get_handle(  \u001b[39m# type: ignore[call-overload]\u001b[39;49;00m\n\u001b[0;32m   1219\u001b[0m     f,\n\u001b[0;32m   1220\u001b[0m     mode,\n\u001b[0;32m   1221\u001b[0m     encoding\u001b[39m=\u001b[39;49m\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49moptions\u001b[39m.\u001b[39;49mget(\u001b[39m\"\u001b[39;49m\u001b[39mencoding\u001b[39;49m\u001b[39m\"\u001b[39;49m, \u001b[39mNone\u001b[39;49;00m),\n\u001b[0;32m   1222\u001b[0m     compression\u001b[39m=\u001b[39;49m\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49moptions\u001b[39m.\u001b[39;49mget(\u001b[39m\"\u001b[39;49m\u001b[39mcompression\u001b[39;49m\u001b[39m\"\u001b[39;49m, \u001b[39mNone\u001b[39;49;00m),\n\u001b[0;32m   1223\u001b[0m     memory_map\u001b[39m=\u001b[39;49m\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49moptions\u001b[39m.\u001b[39;49mget(\u001b[39m\"\u001b[39;49m\u001b[39mmemory_map\u001b[39;49m\u001b[39m\"\u001b[39;49m, \u001b[39mFalse\u001b[39;49;00m),\n\u001b[0;32m   1224\u001b[0m     is_text\u001b[39m=\u001b[39;49mis_text,\n\u001b[0;32m   1225\u001b[0m     errors\u001b[39m=\u001b[39;49m\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49moptions\u001b[39m.\u001b[39;49mget(\u001b[39m\"\u001b[39;49m\u001b[39mencoding_errors\u001b[39;49m\u001b[39m\"\u001b[39;49m, \u001b[39m\"\u001b[39;49m\u001b[39mstrict\u001b[39;49m\u001b[39m\"\u001b[39;49m),\n\u001b[0;32m   1226\u001b[0m     storage_options\u001b[39m=\u001b[39;49m\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49moptions\u001b[39m.\u001b[39;49mget(\u001b[39m\"\u001b[39;49m\u001b[39mstorage_options\u001b[39;49m\u001b[39m\"\u001b[39;49m, \u001b[39mNone\u001b[39;49;00m),\n\u001b[0;32m   1227\u001b[0m )\n\u001b[0;32m   1228\u001b[0m \u001b[39massert\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mhandles \u001b[39mis\u001b[39;00m \u001b[39mnot\u001b[39;00m \u001b[39mNone\u001b[39;00m\n\u001b[0;32m   1229\u001b[0m f \u001b[39m=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mhandles\u001b[39m.\u001b[39mhandle\n",
      "File \u001b[1;32m~\\AppData\\Roaming\\Python\\Python310\\site-packages\\pandas\\io\\common.py:786\u001b[0m, in \u001b[0;36mget_handle\u001b[1;34m(path_or_buf, mode, encoding, compression, memory_map, is_text, errors, storage_options)\u001b[0m\n\u001b[0;32m    781\u001b[0m \u001b[39melif\u001b[39;00m \u001b[39misinstance\u001b[39m(handle, \u001b[39mstr\u001b[39m):\n\u001b[0;32m    782\u001b[0m     \u001b[39m# Check whether the filename is to be opened in binary mode.\u001b[39;00m\n\u001b[0;32m    783\u001b[0m     \u001b[39m# Binary mode does not support 'encoding' and 'newline'.\u001b[39;00m\n\u001b[0;32m    784\u001b[0m     \u001b[39mif\u001b[39;00m ioargs\u001b[39m.\u001b[39mencoding \u001b[39mand\u001b[39;00m \u001b[39m\"\u001b[39m\u001b[39mb\u001b[39m\u001b[39m\"\u001b[39m \u001b[39mnot\u001b[39;00m \u001b[39min\u001b[39;00m ioargs\u001b[39m.\u001b[39mmode:\n\u001b[0;32m    785\u001b[0m         \u001b[39m# Encoding\u001b[39;00m\n\u001b[1;32m--> 786\u001b[0m         handle \u001b[39m=\u001b[39m \u001b[39mopen\u001b[39;49m(\n\u001b[0;32m    787\u001b[0m             handle,\n\u001b[0;32m    788\u001b[0m             ioargs\u001b[39m.\u001b[39;49mmode,\n\u001b[0;32m    789\u001b[0m             encoding\u001b[39m=\u001b[39;49mioargs\u001b[39m.\u001b[39;49mencoding,\n\u001b[0;32m    790\u001b[0m             errors\u001b[39m=\u001b[39;49merrors,\n\u001b[0;32m    791\u001b[0m             newline\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39m\"\u001b[39;49m,\n\u001b[0;32m    792\u001b[0m         )\n\u001b[0;32m    793\u001b[0m     \u001b[39melse\u001b[39;00m:\n\u001b[0;32m    794\u001b[0m         \u001b[39m# Binary mode\u001b[39;00m\n\u001b[0;32m    795\u001b[0m         handle \u001b[39m=\u001b[39m \u001b[39mopen\u001b[39m(handle, ioargs\u001b[39m.\u001b[39mmode)\n",
      "\u001b[1;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'notebooks/submission_tutorial/ground_truth_files/ground_truth_file_test.csv'"
     ]
    }
   ],
   "source": [
    "data = pd.read_csv('notebooks/submission_tutorial/ground_truth_files/ground_truth_file_test.csv')\n",
    "len(data.loc[0].at[\"responses\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 3) Generate Submission file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submission file saved for tier: live_test. Saved in: ./submission_files/submission_file_live_test.csv\n"
     ]
    }
   ],
   "source": [
    "# generate the submission file\n",
    "submission.generate_submission_file(trained_model=model, \n",
    "                                    dataloaders=dataloaders,\n",
    "                                    data_key=dataset_name,\n",
    "                                    path='./submission_files/',\n",
    "                                    device=\"cuda\",\n",
    "                                    tier=\"test\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "##### Inspect content of submission file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>trial_indices</th>\n",
       "      <th>image_ids</th>\n",
       "      <th>prediction</th>\n",
       "      <th>neuron_ids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>126</td>\n",
       "      <td>2214</td>\n",
       "      <td>[0.24674898386001587, 0.23912948369979858, 0.4...</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>297</td>\n",
       "      <td>2214</td>\n",
       "      <td>[0.24674898386001587, 0.23912948369979858, 0.4...</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>597</td>\n",
       "      <td>2214</td>\n",
       "      <td>[0.24674898386001587, 0.23912948369979858, 0.4...</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>852</td>\n",
       "      <td>2214</td>\n",
       "      <td>[0.24674898386001587, 0.23912948369979858, 0.4...</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>908</td>\n",
       "      <td>2214</td>\n",
       "      <td>[0.24674898386001587, 0.23912948369979858, 0.4...</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>993</th>\n",
       "      <td>2752</td>\n",
       "      <td>3487</td>\n",
       "      <td>[0.040638625621795654, 0.11597681045532227, 0....</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>994</th>\n",
       "      <td>3039</td>\n",
       "      <td>3487</td>\n",
       "      <td>[0.040638625621795654, 0.11597681045532227, 0....</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>995</th>\n",
       "      <td>4312</td>\n",
       "      <td>3487</td>\n",
       "      <td>[0.040638625621795654, 0.11597681045532227, 0....</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>996</th>\n",
       "      <td>4380</td>\n",
       "      <td>3487</td>\n",
       "      <td>[0.040638625621795654, 0.11597681045532227, 0....</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>997</th>\n",
       "      <td>4571</td>\n",
       "      <td>3487</td>\n",
       "      <td>[0.040638625621795654, 0.11597681045532227, 0....</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>998 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     trial_indices  image_ids  \\\n",
       "0              126       2214   \n",
       "1              297       2214   \n",
       "2              597       2214   \n",
       "3              852       2214   \n",
       "4              908       2214   \n",
       "..             ...        ...   \n",
       "993           2752       3487   \n",
       "994           3039       3487   \n",
       "995           4312       3487   \n",
       "996           4380       3487   \n",
       "997           4571       3487   \n",
       "\n",
       "                                            prediction  \\\n",
       "0    [0.24674898386001587, 0.23912948369979858, 0.4...   \n",
       "1    [0.24674898386001587, 0.23912948369979858, 0.4...   \n",
       "2    [0.24674898386001587, 0.23912948369979858, 0.4...   \n",
       "3    [0.24674898386001587, 0.23912948369979858, 0.4...   \n",
       "4    [0.24674898386001587, 0.23912948369979858, 0.4...   \n",
       "..                                                 ...   \n",
       "993  [0.040638625621795654, 0.11597681045532227, 0....   \n",
       "994  [0.040638625621795654, 0.11597681045532227, 0....   \n",
       "995  [0.040638625621795654, 0.11597681045532227, 0....   \n",
       "996  [0.040638625621795654, 0.11597681045532227, 0....   \n",
       "997  [0.040638625621795654, 0.11597681045532227, 0....   \n",
       "\n",
       "                                            neuron_ids  \n",
       "0    [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "1    [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "2    [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "3    [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "4    [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "..                                                 ...  \n",
       "993  [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "994  [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "995  [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "996  [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "997  [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "\n",
       "[998 rows x 4 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "pd.read_csv('./submission_files/submission_file_live_test.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 4) Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "This is what is happening in the backend of our competition website"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from sensorium import evaluate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# specify submission and ground truth file\n",
    "ground_truth_file = './ground_truth_files/ground_truth_file_test.csv'\n",
    "submission_file = './submission_files/submission_file_live_test.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from sensorium import evaluate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# specify submission and ground truth file\n",
    "ground_truth_file = './ground_truth_files/ground_truth_file_test.csv'\n",
    "submission_file = './submission_files/submission_file_live_test.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "out = evaluate(submission_file, ground_truth_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results for the SOTA model:\n",
      "Single Trial Correlation: 0.286\n",
      "Correlation to Average: 0.542\n",
      "FEVE: 0.452\n"
     ]
    }
   ],
   "source": [
    "print(\"Results for the SOTA model:\")\n",
    "for metric, value in out.items():\n",
    "    print(f\"{metric}: {np.round(value, 3)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### These scores are calcualted in the backend of our website\n",
    "- we have two test sets, so these scores will be computed for our **live** test set, and our **final** test set\n",
    "- the **live** scores will get published on the live leaderboard\n",
    "- the **final** scores will be hidden, and we will release them to the public on Oct 22, after checking the scores carefully\n",
    "- the **final** scores will then determine the winner of the competition"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.2 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.2"
  },
  "vscode": {
   "interpreter": {
    "hash": "26de051ba29f2982a8de78e945f0abaf191376122a1563185a90213a26c5da77"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}