Tutorial_052_InClassExercises_SomeStarterAnswers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# In class exercises - Intro to Pandas Series and DataFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import libs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# get and store current file path for file i/o later on in tutorial\n",
    "import os\n",
    "cwd = os.getcwd()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## First import 'response_time_data.csv' data file\n",
    "* Contains RTs from 800 trials of a simple detection task from each of 20 subjects\n",
    "* Organizing into a DataFrame and then saved out in csv format\n",
    "* The index (row) and column labels are encoded in the csv file, so you'll need to read those in explcitly\n",
    "* Make sure to have a look at the DataFrame - use the df.head() function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_name = cwd + '/response_time_data.csv'\n",
    "\n",
    "df = pd.read_csv(file_name, index_col=0, header=0)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now have a look at the data using built in Padas functionality\n",
    "* Check out the max/min of each row, standard deviation, percentiles, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d = df.describe()\n",
    "d['Sub0'][1]\n",
    "# etc..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Are there missing values (NaNs) in the data?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum(np.isnan(df), axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What about outliers? Lets define outliers here as > 2 * std away from the mean for each subject\n",
    "* After you've found the outliers for each subject, replace those values with a np.nan (NaN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d = df.describe()\n",
    "\n",
    "# then use the std and mean fields to define the outlier limit for each subject\n",
    "# then replace with NaNs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## After you've found the outliers and replaced with NaNs for each subject, check out this function:\n",
    "[pandas.DataFrame.interpolate](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.interpolate.html#pandas.DataFrame.interpolate)\n",
    "\n",
    "* Use this function to interpolate the missing values for each subject (do not interpolate across subjects!)\n",
    "* Just use linear interpolation..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.interpolate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## You can explore the \"Missing Values\" page for Pandas to figure out other ways of filling in missing values and outliers\n",
    "\n",
    "[page is here](https://pandas.pydata.org/pandas-docs/stable/missing_data.html#missing-data)\n",
    "\n",
    "* Figure out how to replace the outliers with the mean of each subject"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use the Pandas.DataFrame.Sample function to generate bootstrapped confidence intervals for the data from subject 11\n",
    "\n",
    "[see this page for Samples](https://pandas.pydata.org/pandas-docs/version/0.22/generated/pandas.DataFrame.sample.html)\n",
    "\n",
    "\n",
    "* Resample Sub11's data with replacement, each time pulling N samples (800 in this case)\n",
    "* Generate a distribution of means across all resamples\n",
    "* Compute 95% confidence intervals using:\n",
    "\n",
    "[this page for quantiles](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.quantile.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}