power.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from statsmodels.stats import power\n",
    "from math import ceil"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The interpretation of the p-value, and the “p-value fallacy”\n",
    "\n",
    "---\n",
    "\n",
    "A value of p < 0.05 for the null hypothesis has to be interpreted as follows: If the null hypothesis is true, the chance that we find a test statistic as extreme or more extreme than the one observed is less than 5%. This is not the same as saying that the null hypothesis is false, and even less so, that an alternative hypothesis is true! Stating a p-value alone is no longer state-of-the-art for the statistical analysis of data. You should \n",
    "also state the the confidence intervals for the parameter that you investigate.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Types of Error\n",
    "\n",
    "In hypothesis testing, two types of errors can occur:\n",
    "![foo](images/type12errors.png)\n",
    "\n",
    "## Type I errors\n",
    " * your incorrect rejection of a true null hypothesis (a \"false positive\")\n",
    " * likelihood of a Type I error is commonly indicated with α, and is set before you start the data analysis.\n",
    "\n",
    "## Type II errors and Test Power\n",
    "* Incorrectly retaining a false null hypothesis (a \"false negative\").\n",
    "* probability for this type of error is commonly indicated with β."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sample Size\n",
    "The power of a statistical test depends on four factors:\n",
    "\n",
    "* α, the probability for Type I errors\n",
    "* β, the probability for Type II errors. Power = the probability of correctly rejecting a false null hypothesis = 1−β\n",
    "* d, the effect size, i.e. the magnitude of the investigated effect relative to σ (the standard deviation of the sample)\n",
    "* n, the sample size\n",
    "\n",
    "Only 3 of these 4 parameters can be chosen, the 4th is then automatically fixed.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## For 2 groups t-test use:\n",
    "power.tt_ind_solve_power\n",
    "## For 1 group t-test use:\n",
    "power.tt_solve_power"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tells us that if we compare two groups with the same number of subjects and the same standard deviation, require an α=0.05 a test power of 80%, and we want to detect a difference between the groups that is half the standard deviation, we need to test 64 subjects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "63.765611775409525"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "power.tt_ind_solve_power(effect_size=0.5, alpha=0.05, power=0.8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tells us that if we have an α=0.05, a test power of 80%, and 25 subjects in each group, then the smallest difference between the groups is 81% of the sample standard deviation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9357585233711874"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "power.tt_ind_solve_power(nobs1=25, alpha=0.05, power=0.8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example (one group t-test)\n",
    "Suppose a researcher wants to design a new study with a power of 0.8 and a significance of 0.05 to test whether the caffeine content for a brand of coffee is really 100mg. A previous study gave a mean caffeine level for this brand of 110 mg and a standard deviation of 7 mg.  Use PROC POWER to determine how many cups of coffee need testing.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nobs: 5 actual power: 0.670814322908\n"
     ]
    }
   ],
   "source": [
    "effect_size = (110 - 100) / 7\n",
    "nobs = power.tt_solve_power(effect_size=effect_size, alpha=0.05, power=0.8)\n",
    "nobs = ceil(nobs)\n",
    "actual_power = power.tt_solve_power(effect_size=effect_size, alpha=0.05, nobs=nobs)\n",
    "print('nobs:', nobs, 'actual power:', actual_power)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Credits:\n",
    "* https://en.wikipedia.org/wiki/Type_I_and_type_II_errors\n",
    "* http://work.thaslwanter.at/Stats/html/statsAnalysis.html"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}