notebooks/code001_dataImport.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><h1>Mapping the emotional face. How individual face parts contribute to successful emotion recognition.</h1></center>\n",
    "\n",
    "# 1. Reading Logfiles  \n",
    "\n",
    "This notebook takes all logfiles from the experiment and transforms them into [pandas](http://pandas.pydata.org/) dataframes. Then these tables are saved as csv files for later re-use in other notebooks where the actual analysis happens.\n",
    "\n",
    "<p>All aquired data (i.e. the logfiles that are imported in this notebook, located in the folder ./experiment/app/static/logfiles/) are licensed under a creative commons Public Domain Dedication (No Rights Reserved).<br><a rel=\"license\" href=\"http://creativecommons.org/publicdomain/zero/1.0/\"><img src=\"http://i.creativecommons.org/p/zero/1.0/88x31.png\" alt=\"CC0\" /></a></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import modules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Modules and global variables needed in most or all notebooks are stored in the file myBasics.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from myBasics import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Versions of modules used in this and the following notebooks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "last updated: 2016-03-11 \n",
      "\n",
      "CPython 2.7.11\n",
      "IPython 4.0.3\n",
      "\n",
      "python 2.7.11\n",
      "numpy 1.10.4\n",
      "scipy 0.17.0\n",
      "pandas 0.17.1\n",
      "scikit-learn 0.17\n",
      "matplotlib 1.5.1\n",
      "seaborn 0.7.0\n",
      "pillow 3.1.1\n",
      "\n",
      "compiler   : GCC 4.4.7 20120313 (Red Hat 4.4.7-1)\n",
      "system     : Linux\n",
      "release    : 3.13.0-79-generic\n",
      "machine    : x86_64\n",
      "processor  : x86_64\n",
      "CPU cores  : 8\n",
      "interpreter: 64bit\n",
      "Git hash   : 40611ed2048f6dce426ab1d22dd072e6049cf005\n"
     ]
    }
   ],
   "source": [
    "# Sebastian Raschka; https://github.com/rasbt/watermark/blob/master/docs/watermark.ipynb\n",
    "%load_ext watermark\n",
    "%watermark -v -m -d -u -p python,numpy,scipy,pandas,scikit-learn,matplotlib,seaborn,pillow -g"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Logfiles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def getLogfile(whichFolder, whichExperiment):\n",
    "    logList = []\n",
    "    for filename in os.listdir(whichFolder):\n",
    "        if fnmatch.fnmatch(filename, whichExperiment):\n",
    "            logList.append(whichFolder+filename)\n",
    "    return logList"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "myFolder = '../experiment/app/static/logfiles/'\n",
    "myExperiment = 'log*.txt'\n",
    "logList = getLogfile(myFolder,myExperiment)\n",
    "logList.sort()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are all the logfiles we have. Note that \\#10 and \\#67 are missing, because they were not started (e.g. participant did not come to the appointment). But these are not dropouts and there are no data that were unusable or had to be removed from the analyses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 :\t../experiment/app/static/logfiles/logfile1.txt\n",
      "1 :\t../experiment/app/static/logfiles/logfile11.txt\n",
      "2 :\t../experiment/app/static/logfiles/logfile12.txt\n",
      "3 :\t../experiment/app/static/logfiles/logfile13.txt\n",
      "4 :\t../experiment/app/static/logfiles/logfile14.txt\n",
      "5 :\t../experiment/app/static/logfiles/logfile15.txt\n",
      "6 :\t../experiment/app/static/logfiles/logfile16.txt\n",
      "7 :\t../experiment/app/static/logfiles/logfile17.txt\n",
      "8 :\t../experiment/app/static/logfiles/logfile18.txt\n",
      "9 :\t../experiment/app/static/logfiles/logfile19.txt\n",
      "10 :\t../experiment/app/static/logfiles/logfile2.txt\n",
      "11 :\t../experiment/app/static/logfiles/logfile20.txt\n",
      "12 :\t../experiment/app/static/logfiles/logfile21.txt\n",
      "13 :\t../experiment/app/static/logfiles/logfile22.txt\n",
      "14 :\t../experiment/app/static/logfiles/logfile23.txt\n",
      "15 :\t../experiment/app/static/logfiles/logfile24.txt\n",
      "16 :\t../experiment/app/static/logfiles/logfile25.txt\n",
      "17 :\t../experiment/app/static/logfiles/logfile26.txt\n",
      "18 :\t../experiment/app/static/logfiles/logfile27.txt\n",
      "19 :\t../experiment/app/static/logfiles/logfile28.txt\n",
      "20 :\t../experiment/app/static/logfiles/logfile29.txt\n",
      "21 :\t../experiment/app/static/logfiles/logfile3.txt\n",
      "22 :\t../experiment/app/static/logfiles/logfile30.txt\n",
      "23 :\t../experiment/app/static/logfiles/logfile31.txt\n",
      "24 :\t../experiment/app/static/logfiles/logfile32.txt\n",
      "25 :\t../experiment/app/static/logfiles/logfile33.txt\n",
      "26 :\t../experiment/app/static/logfiles/logfile34.txt\n",
      "27 :\t../experiment/app/static/logfiles/logfile35.txt\n",
      "28 :\t../experiment/app/static/logfiles/logfile36.txt\n",
      "29 :\t../experiment/app/static/logfiles/logfile37.txt\n",
      "30 :\t../experiment/app/static/logfiles/logfile38.txt\n",
      "31 :\t../experiment/app/static/logfiles/logfile39.txt\n",
      "32 :\t../experiment/app/static/logfiles/logfile4.txt\n",
      "33 :\t../experiment/app/static/logfiles/logfile40.txt\n",
      "34 :\t../experiment/app/static/logfiles/logfile41.txt\n",
      "35 :\t../experiment/app/static/logfiles/logfile42.txt\n",
      "36 :\t../experiment/app/static/logfiles/logfile43.txt\n",
      "37 :\t../experiment/app/static/logfiles/logfile44.txt\n",
      "38 :\t../experiment/app/static/logfiles/logfile45.txt\n",
      "39 :\t../experiment/app/static/logfiles/logfile46.txt\n",
      "40 :\t../experiment/app/static/logfiles/logfile47.txt\n",
      "41 :\t../experiment/app/static/logfiles/logfile48.txt\n",
      "42 :\t../experiment/app/static/logfiles/logfile49.txt\n",
      "43 :\t../experiment/app/static/logfiles/logfile5.txt\n",
      "44 :\t../experiment/app/static/logfiles/logfile50.txt\n",
      "45 :\t../experiment/app/static/logfiles/logfile51.txt\n",
      "46 :\t../experiment/app/static/logfiles/logfile52.txt\n",
      "47 :\t../experiment/app/static/logfiles/logfile53.txt\n",
      "48 :\t../experiment/app/static/logfiles/logfile54.txt\n",
      "49 :\t../experiment/app/static/logfiles/logfile55.txt\n",
      "50 :\t../experiment/app/static/logfiles/logfile56.txt\n",
      "51 :\t../experiment/app/static/logfiles/logfile57.txt\n",
      "52 :\t../experiment/app/static/logfiles/logfile58.txt\n",
      "53 :\t../experiment/app/static/logfiles/logfile59.txt\n",
      "54 :\t../experiment/app/static/logfiles/logfile6.txt\n",
      "55 :\t../experiment/app/static/logfiles/logfile60.txt\n",
      "56 :\t../experiment/app/static/logfiles/logfile61.txt\n",
      "57 :\t../experiment/app/static/logfiles/logfile62.txt\n",
      "58 :\t../experiment/app/static/logfiles/logfile63.txt\n",
      "59 :\t../experiment/app/static/logfiles/logfile64.txt\n",
      "60 :\t../experiment/app/static/logfiles/logfile65.txt\n",
      "61 :\t../experiment/app/static/logfiles/logfile66.txt\n",
      "62 :\t../experiment/app/static/logfiles/logfile68.txt\n",
      "63 :\t../experiment/app/static/logfiles/logfile69.txt\n",
      "64 :\t../experiment/app/static/logfiles/logfile7.txt\n",
      "65 :\t../experiment/app/static/logfiles/logfile70.txt\n",
      "66 :\t../experiment/app/static/logfiles/logfile71.txt\n",
      "67 :\t../experiment/app/static/logfiles/logfile72.txt\n",
      "68 :\t../experiment/app/static/logfiles/logfile73.txt\n",
      "69 :\t../experiment/app/static/logfiles/logfile74.txt\n",
      "70 :\t../experiment/app/static/logfiles/logfile75.txt\n",
      "71 :\t../experiment/app/static/logfiles/logfile76.txt\n",
      "72 :\t../experiment/app/static/logfiles/logfile77.txt\n",
      "73 :\t../experiment/app/static/logfiles/logfile78.txt\n",
      "74 :\t../experiment/app/static/logfiles/logfile79.txt\n",
      "75 :\t../experiment/app/static/logfiles/logfile8.txt\n",
      "76 :\t../experiment/app/static/logfiles/logfile80.txt\n",
      "77 :\t../experiment/app/static/logfiles/logfile81.txt\n",
      "78 :\t../experiment/app/static/logfiles/logfile82.txt\n",
      "79 :\t../experiment/app/static/logfiles/logfile83.txt\n",
      "80 :\t../experiment/app/static/logfiles/logfile84.txt\n",
      "81 :\t../experiment/app/static/logfiles/logfile85.txt\n",
      "82 :\t../experiment/app/static/logfiles/logfile86.txt\n",
      "83 :\t../experiment/app/static/logfiles/logfile87.txt\n",
      "84 :\t../experiment/app/static/logfiles/logfile88.txt\n",
      "85 :\t../experiment/app/static/logfiles/logfile89.txt\n",
      "86 :\t../experiment/app/static/logfiles/logfile9.txt\n",
      "87 :\t../experiment/app/static/logfiles/logfile90.txt\n",
      "88 :\t../experiment/app/static/logfiles/logfile91.txt\n",
      "89 :\t../experiment/app/static/logfiles/logfile92.txt\n",
      "90 :\t../experiment/app/static/logfiles/logfile93.txt\n",
      "91 :\t../experiment/app/static/logfiles/logfile94.txt\n",
      "92 :\t../experiment/app/static/logfiles/logfile95.txt\n",
      "93 :\t../experiment/app/static/logfiles/logfile96.txt\n"
     ]
    }
   ],
   "source": [
    "i = 0\n",
    "for log in logList:\n",
    "    print i,':\\t',log\n",
    "    i+=1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example of how the head of the logfile looks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "####### THIS IS A LOGFILE FOR THE DYNAMIC MASKING FACE EXPERIMENT ######\n",
      "\n",
      "Participant Number: 96\n",
      "\n",
      "Date and Time: 2016-03-10 15:04:08.090637\n",
      "\n",
      "age: 20 ,gender: 1 ,environ: 2 ,occup: 0 ,advert: 0\n",
      "\n",
      "###################################################################\n",
      "\n",
      "\n",
      "\n",
      "time\tcumtime\texpress\tident\tbutton\tfilename\tevaluation\tstopRT\tchoiceRT\tmaskNum\tmaskList\n",
      "\n",
      "2016-03-10 15:04:53\t0.0\t2\t1\tang\timg/m_ang_cut.png\tHIT\t25384.0\t34289.0\t26\t33-43-3-21-44-34-8-10-36-20-15-1-29-19-38-9-25-40-47-16-32-4-13-23-12-0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for index,entry in enumerate(open(logList[-1],'r')):\n",
    "    print entry\n",
    "    if index > 6:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get information about age and gender"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def getDemographics(logList):\n",
    "    #empty dict to write to\n",
    "    d = {}\n",
    "    # loop through the logfiles of all participants\n",
    "    for log in logList:\n",
    "        # get the filename of the logfile (clean)\n",
    "        logName = log[log.rfind('/logfile')+len('/logfile'):log.rfind('.')]\n",
    "        # loop throught the logfiles content\n",
    "        for index,entry in enumerate(open(log,'r')):\n",
    "            # the demographics are stored in the 3rd row\n",
    "            if index ==3:\n",
    "                # we get the conents of that row, split it and put in in a list\n",
    "                thisEntry = entry.split()\n",
    "                \n",
    "                d['p'+('000'+logName)[-3:]] = {'age':int(thisEntry[1]) , 'gender':int(thisEntry[3])}\n",
    "    \n",
    "    # make a dataframe\n",
    "    demographicsDf = pd.DataFrame(d).T\n",
    "    # missing values (99) are turned into nans\n",
    "    demographicsDf = demographicsDf.replace(99,np.nan)\n",
    "    return demographicsDf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "demoDf = getDemographics(logList)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>gender</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>p001</th>\n",
       "      <td>24</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>p002</th>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>p003</th>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>p004</th>\n",
       "      <td>26</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>p005</th>\n",
       "      <td>28</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      age  gender\n",
       "p001   24       0\n",
       "p002   30       0\n",
       "p003   25       0\n",
       "p004   26       0\n",
       "p005   28       1"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "demoDf.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>gender</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>94.000000</td>\n",
       "      <td>94.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>23.595745</td>\n",
       "      <td>0.361702</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>3.370712</td>\n",
       "      <td>0.483070</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>18.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>21.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>24.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>25.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>36.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             age     gender\n",
       "count  94.000000  94.000000\n",
       "mean   23.595745   0.361702\n",
       "std     3.370712   0.483070\n",
       "min    18.000000   0.000000\n",
       "25%    21.000000   0.000000\n",
       "50%    24.000000   0.000000\n",
       "75%    25.000000   1.000000\n",
       "max    36.000000   1.000000"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "demoDf.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    60\n",
       "1    34\n",
       "Name: gender, dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 0 == female\n",
    "# 1 == male\n",
    "# 2 == other\n",
    "# 99/NaN == no information\n",
    "demoDf['gender'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Some meta-information and sanity checks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the times when the first and last trials were shown"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def getClocktimes(logFile):\n",
    "    timing = []\n",
    "    for entry in open(logFile,'r'):\n",
    "        if 'img' in entry:\n",
    "            timing.append(entry.split()[1])\n",
    "    print logFile, '; started at %s ; finished at: %s' %(timing[0], timing[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "../experiment/app/static/logfiles/logfile1.txt ; started at 15:01:20 ; finished at: 16:02:43\n",
      "../experiment/app/static/logfiles/logfile11.txt ; started at 11:52:16 ; finished at: 12:35:53\n",
      "../experiment/app/static/logfiles/logfile12.txt ; started at 11:52:55 ; finished at: 13:05:16\n",
      "../experiment/app/static/logfiles/logfile13.txt ; started at 14:08:20 ; finished at: 14:56:36\n",
      "../experiment/app/static/logfiles/logfile14.txt ; started at 15:31:56 ; finished at: 16:26:33\n",
      "../experiment/app/static/logfiles/logfile15.txt ; started at 15:31:41 ; finished at: 16:38:39\n",
      "../experiment/app/static/logfiles/logfile16.txt ; started at 11:57:53 ; finished at: 12:44:36\n",
      "../experiment/app/static/logfiles/logfile17.txt ; started at 12:00:49 ; finished at: 12:56:46\n",
      "../experiment/app/static/logfiles/logfile18.txt ; started at 14:14:05 ; finished at: 15:25:08\n",
      "../experiment/app/static/logfiles/logfile19.txt ; started at 14:32:17 ; finished at: 15:22:41\n",
      "../experiment/app/static/logfiles/logfile2.txt ; started at 17:06:50 ; finished at: 17:44:52\n",
      "../experiment/app/static/logfiles/logfile20.txt ; started at 16:13:00 ; finished at: 16:58:07\n",
      "../experiment/app/static/logfiles/logfile21.txt ; started at 16:13:32 ; finished at: 16:56:18\n",
      "../experiment/app/static/logfiles/logfile22.txt ; started at 09:46:44 ; finished at: 10:47:41\n",
      "../experiment/app/static/logfiles/logfile23.txt ; started at 09:47:30 ; finished at: 10:57:35\n",
      "../experiment/app/static/logfiles/logfile24.txt ; started at 10:56:47 ; finished at: 11:59:55\n",
      "../experiment/app/static/logfiles/logfile25.txt ; started at 10:57:48 ; finished at: 12:02:43\n",
      "../experiment/app/static/logfiles/logfile26.txt ; started at 11:01:32 ; finished at: 11:48:03\n",
      "../experiment/app/static/logfiles/logfile27.txt ; started at 15:25:41 ; finished at: 16:04:10\n",
      "../experiment/app/static/logfiles/logfile28.txt ; started at 15:25:34 ; finished at: 16:18:21\n",
      "../experiment/app/static/logfiles/logfile29.txt ; started at 10:11:30 ; finished at: 11:02:26\n",
      "../experiment/app/static/logfiles/logfile3.txt ; started at 14:59:48 ; finished at: 15:48:42\n",
      "../experiment/app/static/logfiles/logfile30.txt ; started at 12:02:14 ; finished at: 12:35:14\n",
      "../experiment/app/static/logfiles/logfile31.txt ; started at 15:28:40 ; finished at: 16:06:19\n",
      "../experiment/app/static/logfiles/logfile32.txt ; started at 08:27:30 ; finished at: 09:13:17\n",
      "../experiment/app/static/logfiles/logfile33.txt ; started at 10:17:16 ; finished at: 11:03:39\n",
      "../experiment/app/static/logfiles/logfile34.txt ; started at 10:05:19 ; finished at: 10:58:23\n",
      "../experiment/app/static/logfiles/logfile35.txt ; started at 12:59:58 ; finished at: 14:12:35\n",
      "../experiment/app/static/logfiles/logfile36.txt ; started at 13:01:14 ; finished at: 13:50:17\n",
      "../experiment/app/static/logfiles/logfile37.txt ; started at 13:11:14 ; finished at: 13:53:53\n",
      "../experiment/app/static/logfiles/logfile38.txt ; started at 08:06:03 ; finished at: 09:03:04\n",
      "../experiment/app/static/logfiles/logfile39.txt ; started at 11:53:23 ; finished at: 12:48:32\n",
      "../experiment/app/static/logfiles/logfile4.txt ; started at 16:26:17 ; finished at: 17:21:07\n",
      "../experiment/app/static/logfiles/logfile40.txt ; started at 12:57:26 ; finished at: 13:48:29\n",
      "../experiment/app/static/logfiles/logfile41.txt ; started at 12:57:44 ; finished at: 13:49:12\n",
      "../experiment/app/static/logfiles/logfile42.txt ; started at 16:02:41 ; finished at: 16:54:38\n",
      "../experiment/app/static/logfiles/logfile43.txt ; started at 12:59:15 ; finished at: 14:06:48\n",
      "../experiment/app/static/logfiles/logfile44.txt ; started at 13:08:10 ; finished at: 13:57:16\n",
      "../experiment/app/static/logfiles/logfile45.txt ; started at 16:04:44 ; finished at: 16:58:07\n",
      "../experiment/app/static/logfiles/logfile46.txt ; started at 10:57:53 ; finished at: 11:51:38\n",
      "../experiment/app/static/logfiles/logfile47.txt ; started at 12:57:12 ; finished at: 13:36:45\n",
      "../experiment/app/static/logfiles/logfile48.txt ; started at 12:57:12 ; finished at: 13:38:29\n",
      "../experiment/app/static/logfiles/logfile49.txt ; started at 13:02:01 ; finished at: 13:52:45\n",
      "../experiment/app/static/logfiles/logfile5.txt ; started at 15:53:34 ; finished at: 17:01:05\n",
      "../experiment/app/static/logfiles/logfile50.txt ; started at 17:30:51 ; finished at: 18:21:46\n",
      "../experiment/app/static/logfiles/logfile51.txt ; started at 17:32:31 ; finished at: 18:29:58\n",
      "../experiment/app/static/logfiles/logfile52.txt ; started at 11:54:58 ; finished at: 12:32:25\n",
      "../experiment/app/static/logfiles/logfile53.txt ; started at 11:55:13 ; finished at: 13:06:03\n",
      "../experiment/app/static/logfiles/logfile54.txt ; started at 16:04:43 ; finished at: 16:49:29\n",
      "../experiment/app/static/logfiles/logfile55.txt ; started at 17:43:09 ; finished at: 18:32:49\n",
      "../experiment/app/static/logfiles/logfile56.txt ; started at 12:03:49 ; finished at: 12:49:50\n",
      "../experiment/app/static/logfiles/logfile57.txt ; started at 12:03:47 ; finished at: 12:47:25\n",
      "../experiment/app/static/logfiles/logfile58.txt ; started at 12:00:17 ; finished at: 12:36:42\n",
      "../experiment/app/static/logfiles/logfile59.txt ; started at 12:00:29 ; finished at: 12:42:33\n",
      "../experiment/app/static/logfiles/logfile6.txt ; started at 10:10:08 ; finished at: 11:06:53\n",
      "../experiment/app/static/logfiles/logfile60.txt ; started at 12:03:49 ; finished at: 12:41:30\n",
      "../experiment/app/static/logfiles/logfile61.txt ; started at 12:53:21 ; finished at: 13:40:56\n",
      "../experiment/app/static/logfiles/logfile62.txt ; started at 13:01:51 ; finished at: 13:58:32\n",
      "../experiment/app/static/logfiles/logfile63.txt ; started at 11:58:08 ; finished at: 12:38:07\n",
      "../experiment/app/static/logfiles/logfile64.txt ; started at 10:33:38 ; finished at: 11:25:00\n",
      "../experiment/app/static/logfiles/logfile65.txt ; started at 13:27:19 ; finished at: 14:10:20\n",
      "../experiment/app/static/logfiles/logfile66.txt ; started at 11:59:49 ; finished at: 12:53:26\n",
      "../experiment/app/static/logfiles/logfile68.txt ; started at 16:05:33 ; finished at: 16:49:42\n",
      "../experiment/app/static/logfiles/logfile69.txt ; started at 10:03:35 ; finished at: 10:45:23\n",
      "../experiment/app/static/logfiles/logfile7.txt ; started at 10:10:52 ; finished at: 11:12:10\n",
      "../experiment/app/static/logfiles/logfile70.txt ; started at 11:02:43 ; finished at: 11:59:17\n",
      "../experiment/app/static/logfiles/logfile71.txt ; started at 14:01:51 ; finished at: 14:50:36\n",
      "../experiment/app/static/logfiles/logfile72.txt ; started at 10:03:20 ; finished at: 10:47:48\n",
      "../experiment/app/static/logfiles/logfile73.txt ; started at 12:08:23 ; finished at: 12:52:02\n",
      "../experiment/app/static/logfiles/logfile74.txt ; started at 10:00:46 ; finished at: 10:57:17\n",
      "../experiment/app/static/logfiles/logfile75.txt ; started at 10:59:32 ; finished at: 11:45:42\n",
      "../experiment/app/static/logfiles/logfile76.txt ; started at 11:08:09 ; finished at: 12:03:12\n",
      "../experiment/app/static/logfiles/logfile77.txt ; started at 10:10:57 ; finished at: 11:18:42\n",
      "../experiment/app/static/logfiles/logfile78.txt ; started at 15:31:44 ; finished at: 16:12:14\n",
      "../experiment/app/static/logfiles/logfile79.txt ; started at 15:33:53 ; finished at: 16:36:56\n",
      "../experiment/app/static/logfiles/logfile8.txt ; started at 10:11:07 ; finished at: 10:59:00\n",
      "../experiment/app/static/logfiles/logfile80.txt ; started at 15:45:27 ; finished at: 16:32:56\n",
      "../experiment/app/static/logfiles/logfile81.txt ; started at 11:00:51 ; finished at: 11:56:47\n",
      "../experiment/app/static/logfiles/logfile82.txt ; started at 10:56:01 ; finished at: 11:33:05\n",
      "../experiment/app/static/logfiles/logfile83.txt ; started at 10:57:34 ; finished at: 11:37:47\n",
      "../experiment/app/static/logfiles/logfile84.txt ; started at 11:11:45 ; finished at: 11:46:29\n",
      "../experiment/app/static/logfiles/logfile85.txt ; started at 11:03:59 ; finished at: 12:00:17\n",
      "../experiment/app/static/logfiles/logfile86.txt ; started at 13:50:46 ; finished at: 14:38:59\n",
      "../experiment/app/static/logfiles/logfile87.txt ; started at 13:51:04 ; finished at: 14:41:06\n",
      "../experiment/app/static/logfiles/logfile88.txt ; started at 13:59:35 ; finished at: 14:41:03\n",
      "../experiment/app/static/logfiles/logfile89.txt ; started at 10:06:02 ; finished at: 11:08:51\n",
      "../experiment/app/static/logfiles/logfile9.txt ; started at 11:07:58 ; finished at: 11:56:50\n",
      "../experiment/app/static/logfiles/logfile90.txt ; started at 13:04:22 ; finished at: 13:48:56\n",
      "../experiment/app/static/logfiles/logfile91.txt ; started at 09:55:27 ; finished at: 10:55:31\n",
      "../experiment/app/static/logfiles/logfile92.txt ; started at 11:00:37 ; finished at: 11:49:11\n",
      "../experiment/app/static/logfiles/logfile93.txt ; started at 11:01:08 ; finished at: 11:49:51\n",
      "../experiment/app/static/logfiles/logfile94.txt ; started at 13:57:31 ; finished at: 14:52:09\n",
      "../experiment/app/static/logfiles/logfile95.txt ; started at 14:10:28 ; finished at: 15:17:39\n",
      "../experiment/app/static/logfiles/logfile96.txt ; started at 15:04:53 ; finished at: 15:52:53\n"
     ]
    }
   ],
   "source": [
    "for logFile in logList:\n",
    "    getClocktimes(logFile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the length of each block and the cumulated time of blocks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def getBlocktimes(logfile):\n",
    "    cumtime = []\n",
    "    blocktime = []\n",
    "    for entry in open(logfile,'r'):\n",
    "        if 'img' in entry:\n",
    "            cumtime.append( float(entry.split()[2]) )\n",
    "            try:\n",
    "                if cumtime[-2] > cumtime[-1]:\n",
    "                    blocktime.append(round(cumtime[-2]/60,2))\n",
    "            except:\n",
    "                pass\n",
    "    blocktime.append(round(cumtime[-1]/60,2))\n",
    "    print logfile[logfile.rfind('/')+1:], '; block lengths: %s ; total length: %smin' %(blocktime, sum(blocktime))\n",
    "    return sum(blocktime)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logfile1.txt ; block lengths: [35.99, 24.64] ; total length: 60.63min\n",
      "logfile11.txt ; block lengths: [20.44, 22.21] ; total length: 42.65min\n",
      "logfile12.txt ; block lengths: [38.39, 33.4] ; total length: 71.79min\n",
      "logfile13.txt ; block lengths: [23.76, 23.57] ; total length: 47.33min\n",
      "logfile14.txt ; block lengths: [31.64, 21.84] ; total length: 53.48min\n",
      "logfile15.txt ; block lengths: [36.64, 28.59] ; total length: 65.23min\n",
      "logfile16.txt ; block lengths: [25.37, 20.7] ; total length: 46.07min\n",
      "logfile17.txt ; block lengths: [30.37, 24.59] ; total length: 54.96min\n",
      "logfile18.txt ; block lengths: [36.83, 33.6] ; total length: 70.43min\n",
      "logfile19.txt ; block lengths: [26.08, 23.18] ; total length: 49.26min\n",
      "logfile2.txt ; block lengths: [20.65, 16.21] ; total length: 36.86min\n",
      "logfile20.txt ; block lengths: [25.32, 19.44] ; total length: 44.76min\n",
      "logfile21.txt ; block lengths: [24.72, 17.07] ; total length: 41.79min\n",
      "logfile22.txt ; block lengths: [43.75, 15.84] ; total length: 59.59min\n",
      "logfile23.txt ; block lengths: [36.26, 31.1] ; total length: 67.36min\n",
      "logfile24.txt ; block lengths: [35.06, 26.7] ; total length: 61.76min\n",
      "logfile25.txt ; block lengths: [34.7, 28.88] ; total length: 63.58min\n",
      "logfile26.txt ; block lengths: [25.91, 20.03] ; total length: 45.94min\n",
      "logfile27.txt ; block lengths: [19.39, 17.67] ; total length: 37.06min\n",
      "logfile28.txt ; block lengths: [29.73, 22.17] ; total length: 51.9min\n",
      "logfile29.txt ; block lengths: [25.35, 24.67] ; total length: 50.02min\n",
      "logfile3.txt ; block lengths: [23.66, 21.72] ; total length: 45.38min\n",
      "logfile30.txt ; block lengths: [18.88, 13.68] ; total length: 32.56min\n",
      "logfile31.txt ; block lengths: [20.62, 16.44] ; total length: 37.06min\n",
      "logfile32.txt ; block lengths: [26.04, 19.24] ; total length: 45.28min\n",
      "logfile33.txt ; block lengths: [25.63, 19.65] ; total length: 45.28min\n",
      "logfile34.txt ; block lengths: [27.27, 24.98] ; total length: 52.25min\n",
      "logfile35.txt ; block lengths: [36.24, 35.45] ; total length: 71.69min\n",
      "logfile36.txt ; block lengths: [24.98, 23.47] ; total length: 48.45min\n",
      "logfile37.txt ; block lengths: [23.99, 17.94] ; total length: 41.93min\n",
      "logfile38.txt ; block lengths: [31.46, 24.56] ; total length: 56.02min\n",
      "logfile39.txt ; block lengths: [28.36, 26.32] ; total length: 54.68min\n",
      "logfile4.txt ; block lengths: [30.99, 22.16] ; total length: 53.15min\n",
      "logfile40.txt ; block lengths: [28.93, 21.38] ; total length: 50.31min\n",
      "logfile41.txt ; block lengths: [24.11, 26.46] ; total length: 50.57min\n",
      "logfile42.txt ; block lengths: [31.44, 19.6] ; total length: 51.04min\n",
      "logfile43.txt ; block lengths: [34.79, 30.31] ; total length: 65.1min\n",
      "logfile44.txt ; block lengths: [28.14, 18.82] ; total length: 46.96min\n",
      "logfile45.txt ; block lengths: [26.02, 26.49] ; total length: 52.51min\n",
      "logfile46.txt ; block lengths: [32.81, 20.29] ; total length: 53.1min\n",
      "logfile47.txt ; block lengths: [21.63, 16.61] ; total length: 38.24min\n",
      "logfile48.txt ; block lengths: [18.97, 21.55] ; total length: 40.52min\n",
      "logfile49.txt ; block lengths: [27.79, 22.39] ; total length: 50.18min\n",
      "logfile5.txt ; block lengths: [38.57, 25.73] ; total length: 64.3min\n",
      "logfile50.txt ; block lengths: [23.36, 25.76] ; total length: 49.12min\n",
      "logfile51.txt ; block lengths: [29.07, 28.0] ; total length: 57.07min\n",
      "logfile52.txt ; block lengths: [23.91, 12.88] ; total length: 36.79min\n",
      "logfile53.txt ; block lengths: [40.12, 30.07] ; total length: 70.19min\n",
      "logfile54.txt ; block lengths: [24.39, 19.34] ; total length: 43.73min\n",
      "logfile55.txt ; block lengths: [27.05, 22.01] ; total length: 49.06min\n",
      "logfile56.txt ; block lengths: [26.77, 18.54] ; total length: 45.31min\n",
      "logfile57.txt ; block lengths: [24.97, 17.76] ; total length: 42.73min\n",
      "logfile58.txt ; block lengths: [20.53, 14.56] ; total length: 35.09min\n",
      "logfile59.txt ; block lengths: [19.32, 22.05] ; total length: 41.37min\n",
      "logfile6.txt ; block lengths: [24.63, 30.6] ; total length: 55.23min\n",
      "logfile60.txt ; block lengths: [16.24, 20.87] ; total length: 37.11min\n",
      "logfile61.txt ; block lengths: [24.94, 21.8] ; total length: 46.74min\n",
      "logfile62.txt ; block lengths: [29.89, 25.95] ; total length: 55.84min\n",
      "logfile63.txt ; block lengths: [21.04, 18.01] ; total length: 39.05min\n",
      "logfile64.txt ; block lengths: [27.4, 23.46] ; total length: 50.86min\n",
      "logfile65.txt ; block lengths: [24.71, 17.66] ; total length: 42.37min\n",
      "logfile66.txt ; block lengths: [32.39, 19.87] ; total length: 52.26min\n",
      "logfile68.txt ; block lengths: [28.1, 15.57] ; total length: 43.67min\n",
      "logfile69.txt ; block lengths: [23.05, 18.05] ; total length: 41.1min\n",
      "logfile7.txt ; block lengths: [33.6, 26.51] ; total length: 60.11min\n",
      "logfile70.txt ; block lengths: [31.48, 24.41] ; total length: 55.89min\n",
      "logfile71.txt ; block lengths: [27.58, 20.44] ; total length: 48.02min\n",
      "logfile72.txt ; block lengths: [25.69, 17.81] ; total length: 43.5min\n",
      "logfile73.txt ; block lengths: [25.91, 16.93] ; total length: 42.84min\n",
      "logfile74.txt ; block lengths: [35.31, 20.77] ; total length: 56.08min\n",
      "logfile75.txt ; block lengths: [26.3, 18.94] ; total length: 45.24min\n",
      "logfile76.txt ; block lengths: [33.0, 20.69] ; total length: 53.69min\n",
      "logfile77.txt ; block lengths: [35.01, 31.8] ; total length: 66.81min\n",
      "logfile78.txt ; block lengths: [22.86, 17.28] ; total length: 40.14min\n",
      "logfile79.txt ; block lengths: [34.0, 27.21] ; total length: 61.21min\n",
      "logfile8.txt ; block lengths: [24.59, 21.61] ; total length: 46.2min\n",
      "logfile80.txt ; block lengths: [27.22, 18.28] ; total length: 45.5min\n",
      "logfile81.txt ; block lengths: [30.36, 23.79] ; total length: 54.15min\n",
      "logfile82.txt ; block lengths: [20.87, 15.9] ; total length: 36.77min\n",
      "logfile83.txt ; block lengths: [21.49, 17.85] ; total length: 39.34min\n",
      "logfile84.txt ; block lengths: [20.34, 13.55] ; total length: 33.89min\n",
      "logfile85.txt ; block lengths: [29.28, 26.23] ; total length: 55.51min\n",
      "logfile86.txt ; block lengths: [34.36, 12.9] ; total length: 47.26min\n",
      "logfile87.txt ; block lengths: [26.84, 22.74] ; total length: 49.58min\n",
      "logfile88.txt ; block lengths: [20.99, 19.57] ; total length: 40.56min\n",
      "logfile89.txt ; block lengths: [36.87, 24.96] ; total length: 61.83min\n",
      "logfile9.txt ; block lengths: [27.05, 20.64] ; total length: 47.69min\n",
      "logfile90.txt ; block lengths: [26.37, 16.9] ; total length: 43.27min\n",
      "logfile91.txt ; block lengths: [31.81, 26.46] ; total length: 58.27min\n",
      "logfile92.txt ; block lengths: [26.69, 21.28] ; total length: 47.97min\n",
      "logfile93.txt ; block lengths: [23.13, 24.44] ; total length: 47.57min\n",
      "logfile94.txt ; block lengths: [28.5, 25.29] ; total length: 53.79min\n",
      "logfile95.txt ; block lengths: [36.09, 23.99] ; total length: 60.08min\n",
      "logfile96.txt ; block lengths: [28.45, 19.04] ; total length: 47.49min\n"
     ]
    }
   ],
   "source": [
    "allTimes = []\n",
    "for logFile in logList:\n",
    "    allTimes.append( getBlocktimes(logFile) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary statistics for time taken"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>94.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>49.935638</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>9.202309</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>32.560000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>43.327500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>48.755000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>55.440000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>71.790000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               0\n",
       "count  94.000000\n",
       "mean   49.935638\n",
       "std     9.202309\n",
       "min    32.560000\n",
       "25%    43.327500\n",
       "50%    48.755000\n",
       "75%    55.440000\n",
       "max    71.790000"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame( allTimes ).describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transform to Pandas dataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def makePandas(filename):\n",
    "    # we load the csv into pandas\n",
    "    df = pd.read_csv(open(filename,'r'),\n",
    "                skiprows=6,\n",
    "                header=0,\n",
    "                sep='\\t')\n",
    "    \n",
    "    # the index in passed into a column, so we do not loose it when reindexing\n",
    "    df['id'] = df.index\n",
    "    # we sort the data frame by the values we want to use for the (mulit)index\n",
    "    df = df.sort_values(by='express')\n",
    "    df = df.sort_values(by='ident')\n",
    "    # we set which variables are the new multi-index\n",
    "    df = df.set_index(['ident','express','id'],drop=False)\n",
    "    # we rename the variables because to avoid ambiguity\n",
    "    df.rename(columns={'ident': 'i', 'express': 'e','id':'#'}, inplace=True)\n",
    "    ### ugly hack to make index hierarchical\n",
    "    df = df.unstack(0).stack(1).unstack(0).stack(1).unstack(0).stack(1)\n",
    "    \n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df = makePandas(logList[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have a multi-index with three variables: the identity of the face (male/female), the facial expression and the trial number (id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>cumtime</th>\n",
       "      <th>e</th>\n",
       "      <th>i</th>\n",
       "      <th>button</th>\n",
       "      <th>filename</th>\n",
       "      <th>evaluation</th>\n",
       "      <th>stopRT</th>\n",
       "      <th>choiceRT</th>\n",
       "      <th>maskNum</th>\n",
       "      <th>maskList</th>\n",
       "      <th>#</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ident</th>\n",
       "      <th>express</th>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">0</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">0</th>\n",
       "      <th>9</th>\n",
       "      <td>2016-03-10 15:08:11</td>\n",
       "      <td>198.14952</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>hap</td>\n",
       "      <td>img/f_hap_cut.png</td>\n",
       "      <td>HIT</td>\n",
       "      <td>12275</td>\n",
       "      <td>16115</td>\n",
       "      <td>13</td>\n",
       "      <td>3-28-29-7-24-44-45-22-10-43-30-18-34</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>2016-03-10 15:15:00</td>\n",
       "      <td>607.15364</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>hap</td>\n",
       "      <td>img/f_hap_cut.png</td>\n",
       "      <td>HIT</td>\n",
       "      <td>19242</td>\n",
       "      <td>20994</td>\n",
       "      <td>20</td>\n",
       "      <td>40-15-0-37-1-27-45-46-44-23-18-43-22-3-41-29-1...</td>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>2016-03-10 15:18:36</td>\n",
       "      <td>823.46331</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>hap</td>\n",
       "      <td>img/f_hap_cut.png</td>\n",
       "      <td>HIT</td>\n",
       "      <td>16303</td>\n",
       "      <td>17847</td>\n",
       "      <td>17</td>\n",
       "      <td>9-44-23-35-39-14-1-40-46-19-28-20-26-37-17-21-30</td>\n",
       "      <td>37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>2016-03-10 15:20:31</td>\n",
       "      <td>937.80766</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>hap</td>\n",
       "      <td>img/f_hap_cut.png</td>\n",
       "      <td>HIT</td>\n",
       "      <td>5062</td>\n",
       "      <td>7262</td>\n",
       "      <td>6</td>\n",
       "      <td>36-29-47-3-43-20</td>\n",
       "      <td>43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>2016-03-10 15:23:36</td>\n",
       "      <td>1123.17332</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>hap</td>\n",
       "      <td>img/f_hap_cut.png</td>\n",
       "      <td>HIT</td>\n",
       "      <td>5377</td>\n",
       "      <td>6801</td>\n",
       "      <td>6</td>\n",
       "      <td>19-20-16-39-13-44</td>\n",
       "      <td>58</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 time     cumtime  e  i button  \\\n",
       "ident express id                                                 \n",
       "0     0       9   2016-03-10 15:08:11   198.14952  0  0    hap   \n",
       "              27  2016-03-10 15:15:00   607.15364  0  0    hap   \n",
       "              37  2016-03-10 15:18:36   823.46331  0  0    hap   \n",
       "              43  2016-03-10 15:20:31   937.80766  0  0    hap   \n",
       "              58  2016-03-10 15:23:36  1123.17332  0  0    hap   \n",
       "\n",
       "                           filename evaluation  stopRT  choiceRT  maskNum  \\\n",
       "ident express id                                                            \n",
       "0     0       9   img/f_hap_cut.png        HIT   12275     16115       13   \n",
       "              27  img/f_hap_cut.png        HIT   19242     20994       20   \n",
       "              37  img/f_hap_cut.png        HIT   16303     17847       17   \n",
       "              43  img/f_hap_cut.png        HIT    5062      7262        6   \n",
       "              58  img/f_hap_cut.png        HIT    5377      6801        6   \n",
       "\n",
       "                                                           maskList   #  \n",
       "ident express id                                                         \n",
       "0     0       9                3-28-29-7-24-44-45-22-10-43-30-18-34   9  \n",
       "              27  40-15-0-37-1-27-45-46-44-23-18-43-22-3-41-29-1...  27  \n",
       "              37   9-44-23-35-39-14-1-40-46-19-28-20-26-37-17-21-30  37  \n",
       "              43                                   36-29-47-3-43-20  43  \n",
       "              58                                  19-20-16-39-13-44  58  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given that there are 2 faces times 7 expressions times 16 repetitions, there should be 2x7x16=224 entries in each DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "../experiment/app/static/logfiles/logfile1.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile11.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile12.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile13.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile14.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile15.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile16.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile17.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile18.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile19.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile2.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile20.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile21.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile22.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile23.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile24.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile25.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile26.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile27.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile28.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile29.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile3.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile30.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile31.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile32.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile33.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile34.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile35.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile36.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile37.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile38.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile39.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile4.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile40.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile41.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile42.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile43.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile44.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile45.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile46.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile47.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile48.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile49.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile5.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile50.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile51.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile52.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile53.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile54.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile55.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile56.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile57.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile58.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile59.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile6.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile60.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile61.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile62.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile63.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile64.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile65.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile66.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile68.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile69.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile7.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile70.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile71.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile72.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile73.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile74.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile75.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile76.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile77.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile78.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile79.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile8.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile80.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile81.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile82.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile83.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile84.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile85.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile86.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile87.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile88.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile89.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile9.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile90.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile91.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile92.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile93.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile94.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile95.txt \tnumber of entries:  224\n",
      "../experiment/app/static/logfiles/logfile96.txt \tnumber of entries:  224\n"
     ]
    }
   ],
   "source": [
    "for logFile in logList:\n",
    "    thisDf = makePandas(logFile)\n",
    "    assert len(list(thisDf.index)) == 2*7*16, \"wrong number of entries in df %s\" % logFile\n",
    "    print logFile, '\\tnumber of entries: ', len(list(thisDf.index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save as csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def saveCsv(filelist):\n",
    "    for filename in filelist:\n",
    "        df = makePandas(filename)\n",
    "                \n",
    "        # define a name for the csv file that is created\n",
    "        pName = filename[filename.rfind('/logfile')+len('/logfile'):filename.rfind('.')]\n",
    "        csvName = 'pandas_logfile'+ ('000'+pName)[-3:] + '.csv'\n",
    "        \n",
    "        # check if the number of trials is correct\n",
    "        correctNumber = 7*2*8*2 # there are 7 emotions, 2 identities, 8 repetitions per block and 2 blocks (=224)\n",
    "        \n",
    "        if len(df.index.levels[-1]) != correctNumber:\n",
    "            csvName = 'invalid_'+csvName\n",
    "        else: \n",
    "            pass\n",
    "        \n",
    "        print \"...saving\", csvName\n",
    "        \n",
    "        df.to_csv('../rawTables/'+csvName)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "...saving pandas_logfile001.csv\n",
      "...saving pandas_logfile011.csv\n",
      "...saving pandas_logfile012.csv\n",
      "...saving pandas_logfile013.csv\n",
      "...saving pandas_logfile014.csv\n",
      "...saving pandas_logfile015.csv\n",
      "...saving pandas_logfile016.csv\n",
      "...saving pandas_logfile017.csv\n",
      "...saving pandas_logfile018.csv\n",
      "...saving pandas_logfile019.csv\n",
      "...saving pandas_logfile002.csv\n",
      "...saving pandas_logfile020.csv\n",
      "...saving pandas_logfile021.csv\n",
      "...saving pandas_logfile022.csv\n",
      "...saving pandas_logfile023.csv\n",
      "...saving pandas_logfile024.csv\n",
      "...saving pandas_logfile025.csv\n",
      "...saving pandas_logfile026.csv\n",
      "...saving pandas_logfile027.csv\n",
      "...saving pandas_logfile028.csv\n",
      "...saving pandas_logfile029.csv\n",
      "...saving pandas_logfile003.csv\n",
      "...saving pandas_logfile030.csv\n",
      "...saving pandas_logfile031.csv\n",
      "...saving pandas_logfile032.csv\n",
      "...saving pandas_logfile033.csv\n",
      "...saving pandas_logfile034.csv\n",
      "...saving pandas_logfile035.csv\n",
      "...saving pandas_logfile036.csv\n",
      "...saving pandas_logfile037.csv\n",
      "...saving pandas_logfile038.csv\n",
      "...saving pandas_logfile039.csv\n",
      "...saving pandas_logfile004.csv\n",
      "...saving pandas_logfile040.csv\n",
      "...saving pandas_logfile041.csv\n",
      "...saving pandas_logfile042.csv\n",
      "...saving pandas_logfile043.csv\n",
      "...saving pandas_logfile044.csv\n",
      "...saving pandas_logfile045.csv\n",
      "...saving pandas_logfile046.csv\n",
      "...saving pandas_logfile047.csv\n",
      "...saving pandas_logfile048.csv\n",
      "...saving pandas_logfile049.csv\n",
      "...saving pandas_logfile005.csv\n",
      "...saving pandas_logfile050.csv\n",
      "...saving pandas_logfile051.csv\n",
      "...saving pandas_logfile052.csv\n",
      "...saving pandas_logfile053.csv\n",
      "...saving pandas_logfile054.csv\n",
      "...saving pandas_logfile055.csv\n",
      "...saving pandas_logfile056.csv\n",
      "...saving pandas_logfile057.csv\n",
      "...saving pandas_logfile058.csv\n",
      "...saving pandas_logfile059.csv\n",
      "...saving pandas_logfile006.csv\n",
      "...saving pandas_logfile060.csv\n",
      "...saving pandas_logfile061.csv\n",
      "...saving pandas_logfile062.csv\n",
      "...saving pandas_logfile063.csv\n",
      "...saving pandas_logfile064.csv\n",
      "...saving pandas_logfile065.csv\n",
      "...saving pandas_logfile066.csv\n",
      "...saving pandas_logfile068.csv\n",
      "...saving pandas_logfile069.csv\n",
      "...saving pandas_logfile007.csv\n",
      "...saving pandas_logfile070.csv\n",
      "...saving pandas_logfile071.csv\n",
      "...saving pandas_logfile072.csv\n",
      "...saving pandas_logfile073.csv\n",
      "...saving pandas_logfile074.csv\n",
      "...saving pandas_logfile075.csv\n",
      "...saving pandas_logfile076.csv\n",
      "...saving pandas_logfile077.csv\n",
      "...saving pandas_logfile078.csv\n",
      "...saving pandas_logfile079.csv\n",
      "...saving pandas_logfile008.csv\n",
      "...saving pandas_logfile080.csv\n",
      "...saving pandas_logfile081.csv\n",
      "...saving pandas_logfile082.csv\n",
      "...saving pandas_logfile083.csv\n",
      "...saving pandas_logfile084.csv\n",
      "...saving pandas_logfile085.csv\n",
      "...saving pandas_logfile086.csv\n",
      "...saving pandas_logfile087.csv\n",
      "...saving pandas_logfile088.csv\n",
      "...saving pandas_logfile089.csv\n",
      "...saving pandas_logfile009.csv\n",
      "...saving pandas_logfile090.csv\n",
      "...saving pandas_logfile091.csv\n",
      "...saving pandas_logfile092.csv\n",
      "...saving pandas_logfile093.csv\n",
      "...saving pandas_logfile094.csv\n",
      "...saving pandas_logfile095.csv\n",
      "...saving pandas_logfile096.csv\n"
     ]
    }
   ],
   "source": [
    "saveCsv(logList)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sort the list of logfiles in ascending order"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pandasList = getLogfile('../rawTables/','pandas_*')\n",
    "pandasList.sort()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we have a df with the following multi-index:\n",
    "- identity (0=female, 1=male)\n",
    "- expression (0=happy, ... 6=neutral)\n",
    "- id (ascending number as experiment progresses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('../rawTables/pandas_logfile069.csv',index_col=[0,1,2],header=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>cumtime</th>\n",
       "      <th>e</th>\n",
       "      <th>i</th>\n",
       "      <th>button</th>\n",
       "      <th>filename</th>\n",
       "      <th>evaluation</th>\n",
       "      <th>stopRT</th>\n",
       "      <th>choiceRT</th>\n",
       "      <th>maskNum</th>\n",
       "      <th>maskList</th>\n",
       "      <th>#</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ident</th>\n",
       "      <th>express</th>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">0</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">0</th>\n",
       "      <th>13</th>\n",
       "      <td>2016-01-19 10:07:51</td>\n",
       "      <td>255.73029</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>hap</td>\n",
       "      <td>img/f_hap_cut.png</td>\n",
       "      <td>HIT</td>\n",
       "      <td>5152</td>\n",
       "      <td>7296</td>\n",
       "      <td>6</td>\n",
       "      <td>42-44-21-13-33-15</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>2016-01-19 10:10:36</td>\n",
       "      <td>420.75442</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>hap</td>\n",
       "      <td>img/f_hap_cut.png</td>\n",
       "      <td>HIT</td>\n",
       "      <td>2742</td>\n",
       "      <td>4079</td>\n",
       "      <td>3</td>\n",
       "      <td>13-30-41</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>2016-01-19 10:12:38</td>\n",
       "      <td>542.74912</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>hap</td>\n",
       "      <td>img/f_hap_cut.png</td>\n",
       "      <td>HIT</td>\n",
       "      <td>17895</td>\n",
       "      <td>19008</td>\n",
       "      <td>18</td>\n",
       "      <td>40-46-3-47-32-5-41-0-12-17-19-14-22-34-24-8-2-21</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>2016-01-19 10:14:27</td>\n",
       "      <td>652.15954</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>hap</td>\n",
       "      <td>img/f_hap_cut.png</td>\n",
       "      <td>HIT</td>\n",
       "      <td>5905</td>\n",
       "      <td>7025</td>\n",
       "      <td>6</td>\n",
       "      <td>8-12-4-30-40-13</td>\n",
       "      <td>44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>2016-01-19 10:18:11</td>\n",
       "      <td>875.79968</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>hap</td>\n",
       "      <td>img/f_hap_cut.png</td>\n",
       "      <td>HIT</td>\n",
       "      <td>7089</td>\n",
       "      <td>8969</td>\n",
       "      <td>8</td>\n",
       "      <td>22-36-35-25-39-2-30-15</td>\n",
       "      <td>62</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 time    cumtime  e  i button  \\\n",
       "ident express id                                                \n",
       "0     0       13  2016-01-19 10:07:51  255.73029  0  0    hap   \n",
       "              26  2016-01-19 10:10:36  420.75442  0  0    hap   \n",
       "              35  2016-01-19 10:12:38  542.74912  0  0    hap   \n",
       "              44  2016-01-19 10:14:27  652.15954  0  0    hap   \n",
       "              62  2016-01-19 10:18:11  875.79968  0  0    hap   \n",
       "\n",
       "                           filename evaluation  stopRT  choiceRT  maskNum  \\\n",
       "ident express id                                                            \n",
       "0     0       13  img/f_hap_cut.png        HIT    5152      7296        6   \n",
       "              26  img/f_hap_cut.png        HIT    2742      4079        3   \n",
       "              35  img/f_hap_cut.png        HIT   17895     19008       18   \n",
       "              44  img/f_hap_cut.png        HIT    5905      7025        6   \n",
       "              62  img/f_hap_cut.png        HIT    7089      8969        8   \n",
       "\n",
       "                                                          maskList   #  \n",
       "ident express id                                                        \n",
       "0     0       13                                 42-44-21-13-33-15  13  \n",
       "              26                                          13-30-41  26  \n",
       "              35  40-46-3-47-32-5-41-0-12-17-19-14-22-34-24-8-2-21  35  \n",
       "              44                                   8-12-4-30-40-13  44  \n",
       "              62                            22-36-35-25-39-2-30-15  62  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check if logfiles are not corrupted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "checking logfile ../rawTables/pandas_logfile001.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile002.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile003.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile004.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile005.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile006.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile007.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile008.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile009.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile011.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile012.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile013.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile014.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile015.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile016.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile017.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile018.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile019.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile020.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile021.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile022.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile023.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile024.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile025.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile026.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile027.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile028.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile029.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile030.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile031.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile032.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile033.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile034.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile035.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile036.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile037.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile038.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile039.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile040.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile041.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile042.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile043.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile044.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile045.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile046.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile047.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile048.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile049.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile050.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile051.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile052.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile053.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile054.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile055.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile056.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile057.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile058.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile059.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile060.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile061.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile062.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile063.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile064.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile065.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile066.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile068.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile069.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile070.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile071.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile072.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile073.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile074.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile075.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile076.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile077.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile078.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile079.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile080.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile081.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile082.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile083.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile084.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile085.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile086.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile087.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile088.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile089.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile090.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile091.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile092.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile093.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile094.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile095.csv ...\n",
      "checking logfile ../rawTables/pandas_logfile096.csv ...\n"
     ]
    }
   ],
   "source": [
    "for pandasFile in pandasList:\n",
    "    print \"checking logfile %s ...\" % pandasFile\n",
    "    # load each stored df into pandas\n",
    "    thisDf = pd.read_csv(pandasFile,index_col=[0,1,2],header=0)\n",
    "    #loop through identities\n",
    "    for ident in thisDf.index.levels[0]:\n",
    "        # loop through expressions\n",
    "        for express in thisDf.index.levels[1]:\n",
    "\n",
    "            # This is to double-check if each condition (expression of a particular face)\n",
    "            # has always exactly 16 trials (there a two blocks a 8 trials per condition)\n",
    "            assert len(thisDf.ix[ident].ix[express]) ==16 ,'trial numbers corrupted'\n",
    "\n",
    "            # This is to double-check whether the number of revealed tiles in the\n",
    "            # variable maskNum is equal to the number of items in the maskList.\n",
    "            for entry in thisDf.ix[ident].ix[express].index:\n",
    "                #print entry,df.ix[ident].ix[express].ix[entry]['maskNum'],len(df.ix[ident].ix[express].ix[entry]['maskList'].split('-'))\n",
    "                assert thisDf.ix[ident].ix[express].ix[entry]['maskNum']==len(thisDf.ix[ident].ix[express].ix[entry]['maskList'].split('-')),'mask numbers corrupted'"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}