Tutorial_051_Pandas_DataFrames.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial 05, Part 1: Pandas DataFrames \n",
    "[The official project homepage](https://pandas.pydata.org)\n",
    "\n",
    "* Goal\n",
    "    * Extend what we learned about Series objects in the previous tutorial to their 2D counterpart - DataFrames\n",
    "    * Develop some tools for dealing with missing data (not exhaustive, but a good start)\n",
    "    * Take this chance to also learn a bit about file input/output (I/O) and some other more advanced coding techniques"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DataFrames\n",
    "\n",
    "[Pandas quick start guide for DataFrames](https://pandas.pydata.org/pandas-docs/stable/dsintro.html#dataframe)\n",
    "\n",
    "* A DataFrame (DF) is a labeled data struture that can be thought of as a 2D extension of the Series objects that we discussed in the first part of the tutorial\n",
    "* A DF can accept many types of input, from a 2D ndarray, multiple Series, a dict of 1D arrays, another DF, etc\n",
    "* Like a Series, DFs contain data values and their labels. Because we're now dealing with a 2D structure, we call the **row labels the index argument** and the **column labels the column argument**. \n",
    "    * Like a Series, if you don't explicitly assign row and column labels, then they will be auto-generated (but not as useful as specifying the labels yourself!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "Much of what we learned about Series objects will generalize to DFs, so here we'll focus on some of key functionality that might not be obvious based on the previous tutorial.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "One more quick note: if using an older version of Python (earlier than 3.6) and Pandas (earlier than 0.23) and you create a DF from a dict without explicitly specifying column names, then the column names will be entered into the DF based on lexical order\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import libs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# standard numpy module\n",
    "import numpy as np\n",
    "\n",
    "# import a generic pandas object and also a few specific functions that we'll use\n",
    "import pandas as pd \n",
    "from pandas import DataFrame, read_csv\n",
    "\n",
    "# new - get and store current file path for file i/o later on in tutorial\n",
    "import os\n",
    "cwd = os.getcwd()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make up a data set to demonstrate functionality, will import some real data later on\n",
    "* Here we'll pretend that we did a unit recording experiment \n",
    "    * There are two stimulus conditions\n",
    "    * And we are recording from 10 different neurons "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mtrand.RandomState at 0x240215a5ee8>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# seed random number generator so that we're all seeing the same thing in class\n",
    "np.random.RandomState(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# index lables for our 10 neurons...see previous tutorial for more elegant ways of generating\n",
    "# index labels, here we're just going to write them out\n",
    "neuron_labels = ['Nrn0', 'Nrn1','Nrn2','Nrn3','Nrn4','Nrn5','Nrn6','Nrn7','Nrn8','Nrn9']  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# generate response in each neuron to stimulus 1...\n",
    "resp1_hz = [14, 27, 62, 88, 45, 56, 75, 63, 33, 46]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# generate a response to stimulus 2...use random.randint just for practice/fun\n",
    "min_resp = 0  # inclusive\n",
    "max_resp = 90 # exclusive\n",
    "resp2_hz = np.random.randint(min_resp, max_resp, len(resp1_hz))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## New - use 'zip' function to wrap up the data from each list into one list\n",
    "[reference page for zip](https://www.w3schools.com/python/ref_func_zip.asp)\n",
    "\n",
    "* Operates just like it sounds  - takes a set of iterators and groups them together into a single iterator with the 1st element in the resultant iterator comprised of the first element of each iterator 'zipped' together, then the second element from each iterator zipped together, etc. \n",
    "* Length of resulting iterator limited by the length of the shortest input iterator!\n",
    "\n",
    "<div class=\"alert alert-warning\">\n",
    "Because the length of the resulting iterator is limited by length of shortest input iterator, you can sometimes not get an error if you try to zip together iterators with unequal lengths - this is fine if intentional, but if the unequal length was caused by a bug, then you may not find it when using zip!\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(14, 35), (27, 53), (62, 1), (88, 86), (45, 28), (56, 58), (75, 20), (63, 11), (33, 19), (46, 62)]\n",
      "Grab one index to see the two response arrays zipped together:\n",
      "(46, 62)\n"
     ]
    }
   ],
   "source": [
    "neuron_data = list(zip(resp1_hz, resp2_hz))\n",
    "print(neuron_data)\n",
    "\n",
    "print('Grab one index to see the two response arrays zipped together:')\n",
    "print(neuron_data[9])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(35, 53, 1, 86, 28, 58, 20, 11, 19, 62)\n"
     ]
    }
   ],
   "source": [
    "# note: to unzip go like this and you'll get back the original\n",
    "uz_data1, uz_data2 = zip(*neuron_data)\n",
    "uz_data1\n",
    "print(uz_data2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make a DataFrame object to hold the contents of the data set\n",
    "[DataFrame help page](https://pandas.pydata.org/pandas-docs/version/0.21/generated/pandas.DataFrame.html)\n",
    "\n",
    "* Just like with the pd.Series call, you specify the data, index labels (row labels in this case)\n",
    "* In addition to row labels, you can also specify column labels (with 'columns')\n",
    "* Can also specify data type (default is inferred)\n",
    "* Like pd.Series you can ask for an independent copy of the data (copy=True) or you will get a view by default (i.e. copy=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>resp1</th>\n",
       "      <th>resp2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Nrn0</th>\n",
       "      <td>14</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn1</th>\n",
       "      <td>27</td>\n",
       "      <td>53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn2</th>\n",
       "      <td>62</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn3</th>\n",
       "      <td>88</td>\n",
       "      <td>86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn4</th>\n",
       "      <td>45</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn5</th>\n",
       "      <td>56</td>\n",
       "      <td>58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn6</th>\n",
       "      <td>75</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn7</th>\n",
       "      <td>63</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn8</th>\n",
       "      <td>33</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn9</th>\n",
       "      <td>46</td>\n",
       "      <td>62</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      resp1  resp2\n",
       "Nrn0     14     35\n",
       "Nrn1     27     53\n",
       "Nrn2     62      1\n",
       "Nrn3     88     86\n",
       "Nrn4     45     28\n",
       "Nrn5     56     58\n",
       "Nrn6     75     20\n",
       "Nrn7     63     11\n",
       "Nrn8     33     19\n",
       "Nrn9     46     62"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# make the call to pd.DataFrames to create the DF - usage much like pd.Series\n",
    "df = pd.DataFrame(data = neuron_data, index=neuron_labels, columns = ['resp1', 'resp2'])\n",
    "\n",
    "# take a look at the output...\n",
    "display(df)   # compare to print(df) - looks nicer with display thanks to iPython backend "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>resp1</th>\n",
       "      <th>resp2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Nrn5</th>\n",
       "      <td>56</td>\n",
       "      <td>58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn6</th>\n",
       "      <td>75</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn7</th>\n",
       "      <td>63</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn8</th>\n",
       "      <td>33</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn9</th>\n",
       "      <td>46</td>\n",
       "      <td>62</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      resp1  resp2\n",
       "Nrn5     56     58\n",
       "Nrn6     75     20\n",
       "Nrn7     63     11\n",
       "Nrn8     33     19\n",
       "Nrn9     46     62"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# another handy display function...good for large dfs that are too big to fit - at least you can get an idea of \n",
    "# the overall structure\n",
    "df.head()\n",
    "df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get a high-level summary of the data using built-in functionality of DataFrame object\n",
    "[API reference page](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.describe.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>resp1</th>\n",
       "      <th>resp2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>10.000000</td>\n",
       "      <td>10.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>50.900000</td>\n",
       "      <td>37.300000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>22.521348</td>\n",
       "      <td>26.650203</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>14.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>36.000000</td>\n",
       "      <td>19.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>51.000000</td>\n",
       "      <td>31.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>62.750000</td>\n",
       "      <td>56.750000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>88.000000</td>\n",
       "      <td>86.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           resp1      resp2\n",
       "count  10.000000  10.000000\n",
       "mean   50.900000  37.300000\n",
       "std    22.521348  26.650203\n",
       "min    14.000000   1.000000\n",
       "25%    36.000000  19.250000\n",
       "50%    51.000000  31.500000\n",
       "75%    62.750000  56.750000\n",
       "max    88.000000  86.000000"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# first call this using the defaults\n",
    "desc = df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>resp1</th>\n",
       "      <th>resp2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>10.000000</td>\n",
       "      <td>10.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>50.900000</td>\n",
       "      <td>37.300000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>22.521348</td>\n",
       "      <td>26.650203</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>14.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0%</th>\n",
       "      <td>14.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10%</th>\n",
       "      <td>25.700000</td>\n",
       "      <td>10.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20%</th>\n",
       "      <td>31.800000</td>\n",
       "      <td>17.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30.0%</th>\n",
       "      <td>41.400000</td>\n",
       "      <td>19.700000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40%</th>\n",
       "      <td>45.600000</td>\n",
       "      <td>24.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>51.000000</td>\n",
       "      <td>31.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60.0%</th>\n",
       "      <td>58.400000</td>\n",
       "      <td>42.200000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70%</th>\n",
       "      <td>62.300000</td>\n",
       "      <td>54.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80%</th>\n",
       "      <td>65.400000</td>\n",
       "      <td>58.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90%</th>\n",
       "      <td>76.300000</td>\n",
       "      <td>64.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100%</th>\n",
       "      <td>88.000000</td>\n",
       "      <td>86.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>88.000000</td>\n",
       "      <td>86.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           resp1      resp2\n",
       "count  10.000000  10.000000\n",
       "mean   50.900000  37.300000\n",
       "std    22.521348  26.650203\n",
       "min    14.000000   1.000000\n",
       "0%     14.000000   1.000000\n",
       "10%    25.700000  10.000000\n",
       "20%    31.800000  17.400000\n",
       "30.0%  41.400000  19.700000\n",
       "40%    45.600000  24.800000\n",
       "50%    51.000000  31.500000\n",
       "60.0%  58.400000  42.200000\n",
       "70%    62.300000  54.500000\n",
       "80%    65.400000  58.800000\n",
       "90%    76.300000  64.400000\n",
       "100%   88.000000  86.000000\n",
       "max    88.000000  86.000000"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# can specify different values to change behavior...\n",
    "df.describe(percentiles=np.linspace(0,1,11))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "\n",
    "BEGIN SIDEBAR - Common source of NaNs in data and how to deal with them in general\n",
    "\n",
    "Important bit of info for avoiding a common source of confusion (and potential bugs!!!)\n",
    "\n",
    "\n",
    "* Note that if you make a DF out of a set of Series (e.g. a dict of Series), then the resulting DF index labels will be the union of the index labels in each Series\n",
    "* This can be confusing because the DF will still be formed even if you have mismatching labels or even if you have two series of different sizes..\n",
    "* Fortunately, the misaligned (or missing) values will be filled in with NaNs ('Not-a-Number') to serve as a placeholder for the misaligned or missing info\n",
    "* Quick demo below before continuing on with our sample neuron data from above \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dict0</th>\n",
       "      <th>dict1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.723944</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.283484</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.894407</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-2.067083</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      dict0  dict1\n",
       "0  1.723944      2\n",
       "1 -0.283484      0\n",
       "2  1.894407      4\n",
       "3 -2.067083      0\n",
       "4       NaN      1"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# make a set of two Series with unequal lengths stored in a dict, \n",
    "# with each Series having data and index labels\n",
    "\n",
    "# Note that Series 1 has 4 elements, but Series 2 has 5 elements!\n",
    "\n",
    "data_dict = {'dict0' : pd.Series(data = np.random.randn(4), index=['0','1','2','3']), \n",
    "            'dict1' : pd.Series(data = np.random.randint(0,5,5), index=['0','1','2','3','4'])}\n",
    "\n",
    "# make a data frame\n",
    "weird_df = pd.DataFrame(data_dict)\n",
    "\n",
    "# take a look - notice that pd.DataFrame did not throw an error even though\n",
    "# the input Series are different sizes...however, it did mark the missing value \n",
    "# with a NaN\n",
    "display(weird_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class = \"alert alert-info\">\n",
    "Because of the above behavior, often good to frequently check for NaNs in your data to identify processing steps that might have gone awry...(unless you are expecting NaNs as part of routine processing, in which case this might not be very helpful). \n",
    "Many ways to do this, but here is one way that works pretty well with identifying the presence of a NaN anywhere in the DataFrame (which is handy beacuse often times the DataFrames are too large to easily see in their entirety) \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dict0</th>\n",
       "      <th>dict1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   dict0  dict1\n",
       "0  False  False\n",
       "1  False  False\n",
       "2  False  False\n",
       "3  False  False\n",
       "4   True  False"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "weird...you have NaNs in your data\n"
     ]
    }
   ],
   "source": [
    "# show True/False for each element in DF\n",
    "# first do it the NumPy way\n",
    "display(np.isnan(weird_df))\n",
    "\n",
    "# use the 'any' method to figure out if any entries are NaN\n",
    "if np.isnan(weird_df).any:\n",
    "    print('weird...you have NaNs in your data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pandas also provides functions for dealing with NaNs (or missing values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dict0</th>\n",
       "      <th>dict1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   dict0  dict1\n",
       "0  False  False\n",
       "1  False  False\n",
       "2  False  False\n",
       "3  False  False\n",
       "4   True  False"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# return true for NaNs\n",
    "pd.isna(weird_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dict0</th>\n",
       "      <th>dict1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   dict0  dict1\n",
       "0   True   True\n",
       "1   True   True\n",
       "2   True   True\n",
       "3   True   True\n",
       "4  False   True"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Or return true for non NaNs\n",
    "pd.notna(weird_df)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     True\n",
       "1     True\n",
       "2     True\n",
       "3     True\n",
       "4    False\n",
       "Name: dict0, dtype: bool"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# apply to just one column at a time\n",
    "# note that you can call the function from the object\n",
    "# directly (instead of using the pd object)\n",
    "weird_df['dict0'].notna() #check the first column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpolate across missing values...\n",
    "* Default is linear, but can do all sorts of other interpolations using scipy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dict0</th>\n",
       "      <th>dict1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.723944</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.283484</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.894407</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-2.067083</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      dict0  dict1\n",
       "0  1.723944      2\n",
       "1 -0.283484      0\n",
       "2  1.894407      4\n",
       "3 -2.067083      0\n",
       "4       NaN      1"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dict0</th>\n",
       "      <th>dict1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.723944</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.283484</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.894407</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-2.067083</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-2.067083</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      dict0  dict1\n",
       "0  1.723944      2\n",
       "1 -0.283484      0\n",
       "2  1.894407      4\n",
       "3 -2.067083      0\n",
       "4 -2.067083      1"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "norm_df = weird_df.interpolate()\n",
    "display(weird_df)\n",
    "display(norm_df)\n",
    "# note that since the missing value is the last in the series\n",
    "# it just takes the value of the 2nd to last entry\n",
    "# otherwise it would be linearly interpolated by default"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Can also fill NaNs lots of other ways...like assigning the mean of all or some of the data to NaNs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dict0</th>\n",
       "      <th>dict1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.723944</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.283484</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.894407</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-2.067083</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.316946</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      dict0  dict1\n",
       "0  1.723944      2\n",
       "1 -0.283484      0\n",
       "2  1.894407      4\n",
       "3 -2.067083      0\n",
       "4  0.316946      1"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# e.g. fill the NaN with the mean of the first column!\n",
    "weird_df.fillna(weird_df.mean()['dict0'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class = \"alert alert-info\">\n",
    "\n",
    "This is just a small subset of methods for finding and dealing with missing data. See this link for a more complete set of possibilities:\n",
    "\n",
    "[Pandas missing data](https://pandas.pydata.org/pandas-docs/stable/missing_data.html)\n",
    "\n",
    "Be careful rooting out NaNs using equivalence testing - NaNs don't compare equal (cause they are missing or unknown values). Better to use boolean indexing based on the return vector from np.isnan or pd.isna, etc\n",
    "\n",
    "\n",
    "Remember that in most cases:\n",
    "* When summing data, NaNs will be treated as zero.\n",
    "* Functions like cumsum() and cumprod() ignore NaNs by default, but will propogate them to returned arrays. \n",
    "    * To change behaviour to include NaNs, often use skipna=False or similar.\n",
    "* If multiplying by NaN, the result is generally NaN\n",
    "    * Except if computing the prod of all NaNs, in which case the answer is 1!\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nan\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# make an array of nans\n",
    "x = np.repeat(np.nan, 10)\n",
    "print(x[0]*x[1])  # NaN\n",
    "\n",
    "# convert x to a series\n",
    "s = pd.Series(x)\n",
    "# compute the prod of elements in series\n",
    "pd.Series.prod(s)\n",
    "np.sum(s)\n",
    "np.prod(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "END SIDEBAR...now back to the main thread and our 10 neuron experimental data\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indexing, adding, deleting entire columns\n",
    "* Think of the DF as a dict of Series objects with common labels - much of the syntax is the same as for dicts (and for Series)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Nrn0    35\n",
       "Nrn1    53\n",
       "Nrn2     1\n",
       "Nrn3    86\n",
       "Nrn4    28\n",
       "Nrn5    58\n",
       "Nrn6    20\n",
       "Nrn7    11\n",
       "Nrn8    19\n",
       "Nrn9    62\n",
       "Name: resp2, dtype: int64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# grab the second column from our DF\n",
    "display(df['resp2'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding a column is easy and can be done dynamically (on the fly)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>resp1</th>\n",
       "      <th>resp2</th>\n",
       "      <th>resp3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Nrn0</th>\n",
       "      <td>14</td>\n",
       "      <td>35</td>\n",
       "      <td>490</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn1</th>\n",
       "      <td>27</td>\n",
       "      <td>53</td>\n",
       "      <td>1431</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn2</th>\n",
       "      <td>62</td>\n",
       "      <td>1</td>\n",
       "      <td>62</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn3</th>\n",
       "      <td>88</td>\n",
       "      <td>86</td>\n",
       "      <td>7568</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn4</th>\n",
       "      <td>45</td>\n",
       "      <td>28</td>\n",
       "      <td>1260</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn5</th>\n",
       "      <td>56</td>\n",
       "      <td>58</td>\n",
       "      <td>3248</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn6</th>\n",
       "      <td>75</td>\n",
       "      <td>20</td>\n",
       "      <td>1500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn7</th>\n",
       "      <td>63</td>\n",
       "      <td>11</td>\n",
       "      <td>693</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn8</th>\n",
       "      <td>33</td>\n",
       "      <td>19</td>\n",
       "      <td>627</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn9</th>\n",
       "      <td>46</td>\n",
       "      <td>62</td>\n",
       "      <td>2852</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      resp1  resp2  resp3\n",
       "Nrn0     14     35    490\n",
       "Nrn1     27     53   1431\n",
       "Nrn2     62      1     62\n",
       "Nrn3     88     86   7568\n",
       "Nrn4     45     28   1260\n",
       "Nrn5     56     58   3248\n",
       "Nrn6     75     20   1500\n",
       "Nrn7     63     11    693\n",
       "Nrn8     33     19    627\n",
       "Nrn9     46     62   2852"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# define a third response column as the product of the first two columns\n",
    "df['resp3'] = df.resp1 * df.resp2\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing columns is also easy and done on the fly... "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>resp1</th>\n",
       "      <th>resp2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Nrn0</th>\n",
       "      <td>14</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn1</th>\n",
       "      <td>27</td>\n",
       "      <td>53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn2</th>\n",
       "      <td>62</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn3</th>\n",
       "      <td>88</td>\n",
       "      <td>86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn4</th>\n",
       "      <td>45</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn5</th>\n",
       "      <td>56</td>\n",
       "      <td>58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn6</th>\n",
       "      <td>75</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn7</th>\n",
       "      <td>63</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn8</th>\n",
       "      <td>33</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn9</th>\n",
       "      <td>46</td>\n",
       "      <td>62</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      resp1  resp2\n",
       "Nrn0     14     35\n",
       "Nrn1     27     53\n",
       "Nrn2     62      1\n",
       "Nrn3     88     86\n",
       "Nrn4     45     28\n",
       "Nrn5     56     58\n",
       "Nrn6     75     20\n",
       "Nrn7     63     11\n",
       "Nrn8     33     19\n",
       "Nrn9     46     62"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# using the del command will delete a column from the DF\n",
    "# note that here you have to use the df['resp3'] notation\n",
    "# the df.resp3 notation will not work.\n",
    "del df['resp3']\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Instead of deleting outright, you can also \"pop\" a column out and asign it to another variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Nrn0     490\n",
       "Nrn1    1431\n",
       "Nrn2      62\n",
       "Nrn3    7568\n",
       "Nrn4    1260\n",
       "Nrn5    3248\n",
       "Nrn6    1500\n",
       "Nrn7     693\n",
       "Nrn8     627\n",
       "Nrn9    2852\n",
       "Name: resp3, dtype: int64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>resp1</th>\n",
       "      <th>resp2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Nrn0</th>\n",
       "      <td>14</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn1</th>\n",
       "      <td>27</td>\n",
       "      <td>53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn2</th>\n",
       "      <td>62</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn3</th>\n",
       "      <td>88</td>\n",
       "      <td>86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn4</th>\n",
       "      <td>45</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn5</th>\n",
       "      <td>56</td>\n",
       "      <td>58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn6</th>\n",
       "      <td>75</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn7</th>\n",
       "      <td>63</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn8</th>\n",
       "      <td>33</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn9</th>\n",
       "      <td>46</td>\n",
       "      <td>62</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      resp1  resp2\n",
       "Nrn0     14     35\n",
       "Nrn1     27     53\n",
       "Nrn2     62      1\n",
       "Nrn3     88     86\n",
       "Nrn4     45     28\n",
       "Nrn5     56     58\n",
       "Nrn6     75     20\n",
       "Nrn7     63     11\n",
       "Nrn8     33     19\n",
       "Nrn9     46     62"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# define a third response column as the product of the first two columns\n",
    "df['resp3'] = df.resp1 * df.resp2\n",
    "\n",
    "# then pop it out\n",
    "resp3 = df.pop('resp3')\n",
    "display(resp3)\n",
    "\n",
    "# now df is back down to just 2 columns\n",
    "print('\\n')\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More on indexing and selection of specific coordinates in a DF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Row selection - this is a bit more confusing as there are many methods\n",
    "* You can use df.loc to select a row by its label name\n",
    "* You can use df.iloc to select a row by its integer location \n",
    "* You can use boolean vectors to select a set of rows that satisfy some condition\n",
    "* You can slice rows using standard notation e.g. df[1:3] for rows 1-3\n",
    "* You can also isolate a particular row/column using a combo of column indexing (see above) and standard slicing notation\n",
    "\n",
    "<div class=\"alert alert-danger\">\n",
    "Contrary to usual slicing conventions, both the start and the stop indices are included when using the DF.LOC function...see below for demo. This makes sense (sort of) because you're indexing by label name, not by a zero-based integer index. \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "resp1    27\n",
       "resp2    53\n",
       "Name: Nrn1, dtype: int64"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# data from 2nd neuron across both stimulus conditions\n",
    "df.loc['Nrn1']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>resp1</th>\n",
       "      <th>resp2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Nrn1</th>\n",
       "      <td>27</td>\n",
       "      <td>53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn2</th>\n",
       "      <td>62</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn3</th>\n",
       "      <td>88</td>\n",
       "      <td>86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn4</th>\n",
       "      <td>45</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn5</th>\n",
       "      <td>56</td>\n",
       "      <td>58</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      resp1  resp2\n",
       "Nrn1     27     53\n",
       "Nrn2     62      1\n",
       "Nrn3     88     86\n",
       "Nrn4     45     28\n",
       "Nrn5     56     58"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# CAREFUL!\n",
    "# data from 2nd-6th neuron inclusive across both stimulus conditions\n",
    "df.loc['Nrn1':'Nrn5']\n",
    "\n",
    "# again, just need to be careful but this makes sense given that you're indexing based on label name (not 0-based counting)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "resp1    45\n",
       "resp2    28\n",
       "Name: Nrn4, dtype: int64"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# data from 5th neuron across both stimulus conditions\n",
    "df.iloc[4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>resp1</th>\n",
       "      <th>resp2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Nrn2</th>\n",
       "      <td>62</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn3</th>\n",
       "      <td>88</td>\n",
       "      <td>86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn4</th>\n",
       "      <td>45</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      resp1  resp2\n",
       "Nrn2     62      1\n",
       "Nrn3     88     86\n",
       "Nrn4     45     28"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# data from 2nd-5th neuron across both stimulus conditions\n",
    "df[2:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2nd column, 2nd entry\n",
      "53\n",
      "2nd column, 4th entry\n",
      "86\n"
     ]
    }
   ],
   "source": [
    "# can use the trick for returning only a subset of values from a function that we discussed in\n",
    "# the randomization/bootstrapping lecture:\n",
    "# here grab the 2nd entry from the 2nd column\n",
    "print('2nd column, 2nd entry')\n",
    "print(df['resp2'][1])\n",
    "\n",
    "# can also go like this\n",
    "print('2nd column, 4th entry')\n",
    "print(df.resp2[3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data alignment - like with the Series object, but accounts for a union of both column and index(row) labels\n",
    "* Recall from the tutorial on Series objects that if you have mismatched or different length Series, you can still combine them and do operations on them.\n",
    "* The missing/mismatched elements in the resulting Series was marked with a NaN, and the same will happen with a DF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>r1</th>\n",
       "      <td>0.144893</td>\n",
       "      <td>0.540631</td>\n",
       "      <td>0.903394</td>\n",
       "      <td>0.596385</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>r2</th>\n",
       "      <td>0.370876</td>\n",
       "      <td>0.145731</td>\n",
       "      <td>0.665880</td>\n",
       "      <td>0.168836</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>r3</th>\n",
       "      <td>0.305509</td>\n",
       "      <td>0.040215</td>\n",
       "      <td>0.421145</td>\n",
       "      <td>0.024843</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>r4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           a         b         c         d   e\n",
       "r1  0.144893  0.540631  0.903394  0.596385 NaN\n",
       "r2  0.370876  0.145731  0.665880  0.168836 NaN\n",
       "r3  0.305509  0.040215  0.421145  0.024843 NaN\n",
       "r4       NaN       NaN       NaN       NaN NaN"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# make two dfs and then combine them...\n",
    "\n",
    "# first df will be 3 x 4\n",
    "r0 = 3\n",
    "c0 = 4\n",
    "\n",
    "df0 = pd.DataFrame(np.random.rand(r0,c0), index=['r1', 'r2', 'r3'], columns=['a', 'b', 'c', 'd'])\n",
    "\n",
    "# second df will be 4 x 5\n",
    "r1 = 4\n",
    "c1 = 5\n",
    "\n",
    "df1 = pd.DataFrame(np.random.rand(r1,c1), index=['r1', 'r2', 'r3','r4'], columns=['a', 'b', 'c', 'd','e'])\n",
    "\n",
    "# now combine using a unary operation (lets do element by element multiplication)\n",
    "\n",
    "df_prod = df0 * df1\n",
    "\n",
    "# note the NaNs in all the places where there is a mismatch between the two original dfs\n",
    "display(df_prod)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Similar to above example, but now make two dfs that are the same size, but only a subset of the labels match\n",
    "* To demonstrate that combining dfs relies on a union of both row (index) and column labels, have one of each mismatch \n",
    "* Write this as if we made a common coding error and mislabeled some of index/column labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>cv</th>\n",
       "      <th>d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>r1</th>\n",
       "      <td>0.080427</td>\n",
       "      <td>0.289458</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.406319</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>r2</th>\n",
       "      <td>0.106269</td>\n",
       "      <td>0.731222</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.027500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>r3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>r4</th>\n",
       "      <td>0.251952</td>\n",
       "      <td>0.080080</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.377724</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rr3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            a         b   c  cv         d\n",
       "r1   0.080427  0.289458 NaN NaN  0.406319\n",
       "r2   0.106269  0.731222 NaN NaN  0.027500\n",
       "r3        NaN       NaN NaN NaN       NaN\n",
       "r4   0.251952  0.080080 NaN NaN  0.377724\n",
       "rr3       NaN       NaN NaN NaN       NaN"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# first df will be 4 x 4\n",
    "r0 = 4\n",
    "c0 = 4\n",
    "\n",
    "df0 = pd.DataFrame(np.random.rand(r0,c0), index=['r1', 'r2', 'rr3','r4'], columns=['a', 'b', 'c', 'd'])\n",
    "\n",
    "# second df will be 4 x 5\n",
    "r1 = 4\n",
    "c1 = 4\n",
    "\n",
    "df1 = pd.DataFrame(np.random.rand(r1,c1), index=['r1', 'r2', 'r3','r4'], columns=['a', 'b', 'cv', 'd'])\n",
    "\n",
    "# now combine using a unary operation (lets do element by element multiplication)\n",
    "\n",
    "df_prod = df0 * df1\n",
    "\n",
    "# note the NaNs in all the places where there is a mismatch \n",
    "# between the index/column labels in two original dfs\n",
    "display(df_prod)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Just as with the Series objects, we can use most NumPy operations on DFs without any trouble "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.89867799, 1.23548929, 0.85451936, 1.27908341],\n",
       "       [0.69436071, 1.07575098, 0.8222312 , 0.85047405],\n",
       "       [1.32818877, 1.26888245, 0.64834019, 2.34094615],\n",
       "       [0.79035077, 1.05117948, 0.76121311, 1.04038978]])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# true matrix mulitplication (not element by element)\n",
    "np.dot(df0, df1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get a high-level summary of the data using built-in functionality of DataFrame object\n",
    "[API reference page](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.describe.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>resp1</th>\n",
       "      <th>resp2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>10.000000</td>\n",
       "      <td>10.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>50.900000</td>\n",
       "      <td>37.300000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>22.521348</td>\n",
       "      <td>26.650203</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>14.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>36.000000</td>\n",
       "      <td>19.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>51.000000</td>\n",
       "      <td>31.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>62.750000</td>\n",
       "      <td>56.750000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>88.000000</td>\n",
       "      <td>86.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           resp1      resp2\n",
       "count  10.000000  10.000000\n",
       "mean   50.900000  37.300000\n",
       "std    22.521348  26.650203\n",
       "min    14.000000   1.000000\n",
       "25%    36.000000  19.250000\n",
       "50%    51.000000  31.500000\n",
       "75%    62.750000  56.750000\n",
       "max    88.000000  86.000000"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# first call this using the defaults\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>resp1</th>\n",
       "      <th>resp2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>10.000000</td>\n",
       "      <td>10.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>50.900000</td>\n",
       "      <td>37.300000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>22.521348</td>\n",
       "      <td>26.650203</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>14.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0%</th>\n",
       "      <td>14.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10%</th>\n",
       "      <td>25.700000</td>\n",
       "      <td>10.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20%</th>\n",
       "      <td>31.800000</td>\n",
       "      <td>17.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30.0%</th>\n",
       "      <td>41.400000</td>\n",
       "      <td>19.700000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40%</th>\n",
       "      <td>45.600000</td>\n",
       "      <td>24.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>51.000000</td>\n",
       "      <td>31.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60.0%</th>\n",
       "      <td>58.400000</td>\n",
       "      <td>42.200000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70%</th>\n",
       "      <td>62.300000</td>\n",
       "      <td>54.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80%</th>\n",
       "      <td>65.400000</td>\n",
       "      <td>58.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90%</th>\n",
       "      <td>76.300000</td>\n",
       "      <td>64.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100%</th>\n",
       "      <td>88.000000</td>\n",
       "      <td>86.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>88.000000</td>\n",
       "      <td>86.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           resp1      resp2\n",
       "count  10.000000  10.000000\n",
       "mean   50.900000  37.300000\n",
       "std    22.521348  26.650203\n",
       "min    14.000000   1.000000\n",
       "0%     14.000000   1.000000\n",
       "10%    25.700000  10.000000\n",
       "20%    31.800000  17.400000\n",
       "30.0%  41.400000  19.700000\n",
       "40%    45.600000  24.800000\n",
       "50%    51.000000  31.500000\n",
       "60.0%  58.400000  42.200000\n",
       "70%    62.300000  54.500000\n",
       "80%    65.400000  58.800000\n",
       "90%    76.300000  64.400000\n",
       "100%   88.000000  86.000000\n",
       "max    88.000000  86.000000"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# can specify different values to change behavior...\n",
    "df.describe(percentiles=np.linspace(0,1,11))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## And also just like the Series object, you can also apply many other standard analyses using the DF object directly\n",
    "\n",
    "* [Pandas doc for all functions](https://pandas.pydata.org/pandas-docs/stable/api.html#api-dataframe-stats)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "resp1    50.9\n",
       "resp2    37.3\n",
       "dtype: float64"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.mean(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Given our main DF with the data from the simulated unit recording study, demonstrate some simple file I/O\n",
    "* Start by saving the data in a csv (comma separated values) file, which is a common format that is useful for small-to-medium sized data and can easily store mixed data types (strs, ints, floats, etc)\n",
    "\n",
    "[more than you ever wanted to know about the CSV format](https://www.loc.gov/preservation/digital/formats/fdd/fdd000323.shtml)\n",
    "\n",
    "[API reference page for csv read/write in Pandas](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_csv.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check out API reference page for all the flags, but for the moment\n",
    "# the we'll save our header in the file so that reading it back in is easier...\n",
    "# can do this either way, you just need to know how a file was written when \n",
    "# go to read it \n",
    "df.to_csv('spike_rates.csv',index=True,header=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Have a look at the csv file in a text editor..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Yaqian\\Study\\PhD\\2018 Fall\\NEU231\\PSYC-NEU-231\n",
      "C:\\Users\\Yaqian\\Study\\PhD\\2018 Fall\\NEU231\\PSYC-NEU-231/spike_rates.csv\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>resp1</th>\n",
       "      <th>resp2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Nrn0</th>\n",
       "      <td>14</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn1</th>\n",
       "      <td>27</td>\n",
       "      <td>53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn2</th>\n",
       "      <td>62</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn3</th>\n",
       "      <td>88</td>\n",
       "      <td>86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn4</th>\n",
       "      <td>45</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn5</th>\n",
       "      <td>56</td>\n",
       "      <td>58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn6</th>\n",
       "      <td>75</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn7</th>\n",
       "      <td>63</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn8</th>\n",
       "      <td>33</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nrn9</th>\n",
       "      <td>46</td>\n",
       "      <td>62</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      resp1  resp2\n",
       "Nrn0     14     35\n",
       "Nrn1     27     53\n",
       "Nrn2     62      1\n",
       "Nrn3     88     86\n",
       "Nrn4     45     28\n",
       "Nrn5     56     58\n",
       "Nrn6     75     20\n",
       "Nrn7     63     11\n",
       "Nrn8     33     19\n",
       "Nrn9     46     62"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# use our current working directory to build a path to the file\n",
    "print(cwd)\n",
    "file_name = cwd + '/spike_rates.csv'\n",
    "print(file_name)\n",
    "\n",
    "# read back in the .csv file that we made above. \n",
    "# by default read_csv will try to infer the column headers but since\n",
    "# we explicitly wrote them out (header=True on write) we can tell \n",
    "# read_csv that row 0 in the file has the header info...\n",
    "# same for the row labels (index labels)...tell\n",
    "# read_csv that our row labels are in the first column (0) \n",
    "# of the file\n",
    "df = pd.read_csv(file_name, index_col=0, header=0)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making cooler DataFrame styles (and more useful...although that should take a backseat to coolness)\n",
    "[Check here for a bunch of neat style options](https://pandas.pydata.org/pandas-docs/stable/style.html)\n",
    "* Simple demo - can write custom functions that highlight specific aspects of your data - can be very useful for more clearly highlighting/communicating key points in the data within a notebook  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# highlight the max value in each column in yellow...\n",
    "def highlight_max_value_in_columns(data_frame):\n",
    "    ind_max = data_frame == data_frame.max()\n",
    "    return ['background-color: yellow' if i else '' for i in ind_max]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "    #T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5row3_col0 {\n",
       "            background-color:  yellow;\n",
       "        }    #T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5row3_col1 {\n",
       "            background-color:  yellow;\n",
       "        }</style>  \n",
       "<table id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5\" > \n",
       "<thead>    <tr> \n",
       "        <th class=\"blank level0\" ></th> \n",
       "        <th class=\"col_heading level0 col0\" >resp1</th> \n",
       "        <th class=\"col_heading level0 col1\" >resp2</th> \n",
       "    </tr></thead> \n",
       "<tbody>    <tr> \n",
       "        <th id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5level0_row0\" class=\"row_heading level0 row0\" >Nrn0</th> \n",
       "        <td id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5row0_col0\" class=\"data row0 col0\" >14</td> \n",
       "        <td id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5row0_col1\" class=\"data row0 col1\" >35</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5level0_row1\" class=\"row_heading level0 row1\" >Nrn1</th> \n",
       "        <td id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5row1_col0\" class=\"data row1 col0\" >27</td> \n",
       "        <td id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5row1_col1\" class=\"data row1 col1\" >53</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5level0_row2\" class=\"row_heading level0 row2\" >Nrn2</th> \n",
       "        <td id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5row2_col0\" class=\"data row2 col0\" >62</td> \n",
       "        <td id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5row2_col1\" class=\"data row2 col1\" >1</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5level0_row3\" class=\"row_heading level0 row3\" >Nrn3</th> \n",
       "        <td id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5row3_col0\" class=\"data row3 col0\" >88</td> \n",
       "        <td id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5row3_col1\" class=\"data row3 col1\" >86</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5level0_row4\" class=\"row_heading level0 row4\" >Nrn4</th> \n",
       "        <td id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5row4_col0\" class=\"data row4 col0\" >45</td> \n",
       "        <td id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5row4_col1\" class=\"data row4 col1\" >28</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5level0_row5\" class=\"row_heading level0 row5\" >Nrn5</th> \n",
       "        <td id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5row5_col0\" class=\"data row5 col0\" >56</td> \n",
       "        <td id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5row5_col1\" class=\"data row5 col1\" >58</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5level0_row6\" class=\"row_heading level0 row6\" >Nrn6</th> \n",
       "        <td id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5row6_col0\" class=\"data row6 col0\" >75</td> \n",
       "        <td id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5row6_col1\" class=\"data row6 col1\" >20</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5level0_row7\" class=\"row_heading level0 row7\" >Nrn7</th> \n",
       "        <td id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5row7_col0\" class=\"data row7 col0\" >63</td> \n",
       "        <td id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5row7_col1\" class=\"data row7 col1\" >11</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5level0_row8\" class=\"row_heading level0 row8\" >Nrn8</th> \n",
       "        <td id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5row8_col0\" class=\"data row8 col0\" >33</td> \n",
       "        <td id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5row8_col1\" class=\"data row8 col1\" >19</td> \n",
       "    </tr>    <tr> \n",
       "        <th id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5level0_row9\" class=\"row_heading level0 row9\" >Nrn9</th> \n",
       "        <td id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5row9_col0\" class=\"data row9 col0\" >46</td> \n",
       "        <td id=\"T_80ac7f0a_e83e_11e8_b4ea_54ee75bab6f5row9_col1\" class=\"data row9 col1\" >62</td> \n",
       "    </tr></tbody> \n",
       "</table> "
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x24023c8e320>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# apply it here!\n",
    "df.style.apply(highlight_max_value_in_columns)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}