DangerZone.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "acb89f3d-1ed7-4050-8193-55087a4d3557",
   "metadata": {},
   "source": [
    "# __*Czym jest ‘Danger Zone’ w Data Science*__\n",
    "\n",
    "### dr Michał Korzycki \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8159eb27-16c8-4ba4-967d-aa35ecedcf54",
   "metadata": {},
   "source": [
    "Jak uruchomić tą prezentację:\n",
    "\n",
    "- Zainstaluj https://jupyter.org/\n",
    "- Ściągnij ją z: https://github.com/MichalKorzycki/BEST"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2dfb587-0eac-4aae-b07f-ab4b5c176ec5",
   "metadata": {},
   "source": [
    "Albo skorzystaj z Google Colab\n",
    "\n",
    "https://colab.research.google.com/\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2266ab35-ca1f-4bfd-803b-029fdb9defde",
   "metadata": {},
   "source": [
    "---\n",
    "<br>\n",
    "\n",
    "![](img\\danger.png)\n",
    "\n",
    "\n",
    "http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram\n",
    "<br>\n",
    "---\n",
    "\n",
    "# <font color='red'>__*DANGER ZONE 1*__</font>\n",
    "\n",
    "# <font color='cyan'>__*Correlation is not Causation*__</font>\n",
    "\n",
    "# __*Post hoc ergo propter hoc*__ (z łac., po tym, więc wskutek tego)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "547be225-3b1a-4aa6-806f-af589e003155",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "margarine_consumption = [5,4.7,4.6,4.4,4.3,4.1,4.2,4.2,4.2,4.1]\n",
    "divorce_rate = [8.2,7,6.5,5.3,5.2,4,4.6,4.5,4.2,3.7]\n",
    "years = [2000,2001,2002,2003,2004,2005,2006,2007,2008,2009]\n",
    "\n",
    "df = pd.DataFrame({'years': years, 'margarine_consumption': margarine_consumption, 'divorce_rate': divorce_rate})\n",
    "df.set_index('years', drop=True, inplace=True)\n",
    "df.columns = [['Margarine Consumption','Divorce rate in Maine'], ['(lbs per year)','(per 1000 inhabitants)']]\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "307d73cb-2948-4403-b74d-88524f0af6d3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.dates as mdates\n",
    "from matplotlib.ticker import FormatStrFormatter\n",
    "\n",
    "\n",
    "plt.figure(figsize=(20,10))\n",
    "\n",
    "\n",
    "ax = plt.gca()\n",
    "plt.style.use(\"dark_background\")\n",
    "\n",
    "chart = sns.lineplot(x=years,\n",
    "                     y=divorce_rate,\n",
    "                     linestyle='-', marker='o', color='red',\n",
    "                     palette='bright',  markeredgecolor=\"black\",   \n",
    "                     ax=ax\n",
    "                    )\n",
    "ax.yaxis.set_major_formatter(FormatStrFormatter('%.2f per 1000'))\n",
    "\n",
    "ax2 = chart.twinx()\n",
    "ax2.yaxis.set_major_formatter(FormatStrFormatter('%.2f lbs'))\n",
    "chart = sns.lineplot(x=years,\n",
    "                     y=margarine_consumption,\n",
    "                     linestyle='-', marker='o', color=\"yellow\",\n",
    "                     palette='bright',  markeredgecolor=\"black\",   \n",
    "                    ax=ax2\n",
    "                    )\n",
    "ax.set_title(\"Margarine Consumption\", loc='right', y=0.3, x=0.9 , color = 'yellow', size=25)\n",
    "ax.set_title(\"Divorce rate in Maine\", loc='center', y=0.9, x=0.2, color = 'red', size=25)\n",
    "handles, labels = chart.get_legend_handles_labels()\n",
    "plt.legend(handles, labels, frameon=False, loc=\"best\")\n",
    "plt.xticks(years)\n",
    "plt.title(\"Divorce Rate in Maine vs Margarine Consumption\")\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "831f0bbb-e417-4abd-a812-da0dc9bc8dd5",
   "metadata": {},
   "outputs": [],
   "source": [
    "r = np.corrcoef(margarine_consumption, divorce_rate)\n",
    "r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c03d6a00-7254-414e-b072-daa8c1b36fab",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'Korelacja Pearsona: {r[0][1]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f39297c-e495-477d-89e7-a75ec9a1f348",
   "metadata": {},
   "source": [
    "# http://www.tylervigen.com/spurious-correlations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e11a0ba-0379-4604-9c68-7b8bfc85d639",
   "metadata": {},
   "source": [
    "\n",
    "---\n",
    "\n",
    "# <font color='red'>__*DANGER ZONE 2*__</font>\n",
    "\n",
    "# <font color='cyan'>__*Hipoteza zerowa JEST ważna*__</font>\n",
    "\n",
    "## Za optimizely:\n",
    "\n",
    "### __*A/B testing*__ (also known as split testing or bucket testing) is a method of <font color='cyan'>__*comparing two versions*__</font> of a webpage or app against each other to determine which one performs <font color='cyan'>__*better*__</font>.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b79b17f-c5fd-40cb-920d-905d193cb541",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Co to znaczy __*better*__\n",
    "- Bounce rate\n",
    "- Conversion\n",
    "- Click-Through Rate\n",
    "- Etc. \n",
    "<br>\n",
    "\n",
    "\n",
    "\n",
    "### <font color='cyan'>__*better*__</font> zakłada, że jedno jest lepsze od drugiego.\n",
    "\n",
    "### A tak naprawdę są <font color='red'>3</font> możliwe odpowiedzi:\n",
    "- #### __*A*__ jest lepsze od __*B*__\n",
    "- #### __*B*__ jest lepsze od __*A*__\n",
    "- #### <font color='cyan'>Wybór wersji __*A*__ czy __*B*__ __NIE MA WPŁYWU NA WYNIKI__</font>\n",
    "\n",
    "---\n",
    "\n",
    "# Dlatego liczymy poziom __*istotności statystycznej*__\n",
    "\n",
    "## $H_0$ - hipoteza zerowa - oba zbiory mają tą samą średnią"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c73884d1-209b-4472-81d4-a93cd806e936",
   "metadata": {},
   "source": [
    "### `scipy.stats.ttest_ind(a, b, axis=0, equal_var=True, nan_policy='propagate', alternative='two-sided')[source]`\n",
    "### Calculate the T-test for the means of two independent samples of scores.\n",
    "\n",
    "#### This is a two-sided test for the null hypothesis that 2 independent samples have identical average (expected) values. This test assumes that the populations have identical variances by default.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c9af1a8-3b6b-488f-a18c-c9205ebef064",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "import numpy as np\n",
    "from scipy import stats as st\n",
    "import random\n",
    "\n",
    "N=1000\n",
    "\n",
    "a = random.choices([0, 1],  weights=[980, 20], k=N)\n",
    "print(\"A :\", Counter(a))\n",
    "b = random.choices([0, 1],  weights=[980, 20], k=N)\n",
    "print(\"B :\", Counter(b))\n",
    "\n",
    "t, p = st.ttest_ind(a,b)\n",
    "print(f't={t} p={p}')\n",
    "print()\n",
    "\n",
    "if p>=0.05:\n",
    "    print(\"Nie można odrzucić hipotezy zerowej, że zbiory mają tą samą średnią\")\n",
    "else:\n",
    "    print(\"Należy odrzucić hipotezę zerową, że zbiory mają tą samą średnią\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17314c9a-1bd6-4fba-bdd2-16770bf402b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from scipy import stats as st\n",
    "import random\n",
    "\n",
    "N=1000\n",
    "\n",
    "a = random.choices([0, 1],  weights=[980, 20], k=N)\n",
    "print(\"A :\", Counter(a))\n",
    "b = random.choices([0, 1],  weights=[975, 25], k=N)\n",
    "print(\"B :\", Counter(b))\n",
    "\n",
    "t, p = st.ttest_ind(a,b)\n",
    "print(f't={t} p={p}')\n",
    "print()\n",
    "\n",
    "if p>=0.05:\n",
    "    print(\"Nie można odrzucić hipotezy zerowej, że zbiory mają tą samą średnią\")\n",
    "else:\n",
    "    print(\"Należy odrzucić hipotezę zerową, że zbiory mają tą samą średnią\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c11f8cce-aadc-4537-8668-bca3d93c0082",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from scipy import stats as st\n",
    "import random\n",
    "\n",
    "N=10000\n",
    "\n",
    "a = random.choices([0, 1],  weights=[980, 20], k=N)\n",
    "print(\"A :\", Counter(a))\n",
    "b = random.choices([0, 1],  weights=[975, 25], k=N)\n",
    "print(\"B :\", Counter(b))\n",
    "\n",
    "t, p = st.ttest_ind(a,b)\n",
    "print(f't={t} p={p}')\n",
    "print()\n",
    "\n",
    "if p>=0.05:\n",
    "    print(\"Nie można odrzucić hipotezy zerowej, że zbiory mają tą samą średnią\")\n",
    "else:\n",
    "    print(\"Należy odrzucić hipotezę zerową, że zbiory mają tą samą średnią\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f52a5869-51c9-4f8d-a2a4-0d532a990b3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from scipy import stats as st\n",
    "import random\n",
    "\n",
    "plt.style.use(\"dark_background\")\n",
    "\n",
    "def ab(N, delta, w1=980, w2=20):\n",
    "    a = random.choices([0, 1],  weights=[w1, w2], k=N)\n",
    "    b = random.choices([0, 1],  weights=[w1-delta, w2+delta], k=N)\n",
    "    t, p = st.ttest_ind(a,b)\n",
    "    return p+0.000000001\n",
    "    \n",
    "\n",
    "experiments = range(200, 25_000, 200)\n",
    "tstats5 = [ ab(N, 5) for N in experiments]\n",
    "tstats10 = [ ab(N,10) for N in experiments]\n",
    "flatp5 = [ 0.05 for N in experiments ]    \n",
    "\n",
    "fig = plt.figure();\n",
    "plt.figure(figsize=(20,10));\n",
    "plt.yscale(\"log\")\n",
    "\n",
    "\n",
    "plt.title('Poziom istotności statystycznej eksperymentu');\n",
    "t5, = plt.plot(experiments, tstats5, label=\"Eksperyment 2.0% vs 2.5%\");\n",
    "t10, = plt.plot(experiments, tstats10, label=\"Eksperyment 2.0% vs 3.0%\");\n",
    "p5, = plt.plot(experiments, flatp5, label=\"Poziom istotności 0.05\");\n",
    "plt.legend(handles=[t5, t10, p5], frameon=True, loc=\"best\",  fontsize=16);\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "313fae04-e080-431b-85a5-26b752ad9442",
   "metadata": {},
   "source": [
    "---\n",
    "# <font color='red'>__*DANGER ZONE 3*__</font>\n",
    "\n",
    "\n",
    "# Poziom istotności `p` ma sens _TYLKO_ w odniesieniu do _ZAKOŃCZONEGO_ eksperymentu\n",
    "\n",
    "# Przerwanie eksperymentu przed ustalonym a priori czasem jest nazywane \n",
    "- # `p-value fishing`\n",
    "- # `p-hacking`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdf30cbd-3344-4d6c-bdd4-cee9b740db77",
   "metadata": {},
   "source": [
    "---\n",
    "# <font color='cyan'>Moja checklista do testów __A/B__</font>\n",
    "\n",
    " - ### Testuj tylko jedną zmienną naraz (np. _CTR_)\n",
    " - ### Z góry ustal cel\n",
    " - ### <font color='yellow'>__Z góry ustal wielkość/granice testu__</font>\n",
    " - ### Z góry ustal poziom istotności i <font color='yellow'>NIE PRZERYWAJ WCZEŚNIEJ TESTU</font> po osiągnięciu tego poziomu\n",
    " - ### Ustal grupę kontrolną i grupę eksperymentalną\n",
    " - ### Podziel (w najprostszym przypadku) ruch na dwie części równe i <font color='yellow'>__LOSOWE__</font> \n",
    " - ### Testuj obie grupy <font color='yellow'>__RÓWNOCZESNIE__</font>\n",
    " - ### Jeśli tylko to możliwe, testuj wiele opcji:  *A/A*, *A/A/B*, *A/A/B/C*, *A/B/C/D* \n",
    "\n",
    "---\n",
    "\n",
    "## <font color='cyan'>__A/A__</font> albo <font color='cyan'>__A/A/B__</font> \n",
    "- ###  pozwalają sprawdzić czy system działa i stanowią dobrą podstawę do tłumaczenia się z poziomów istotności"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c1ff14b-9497-4237-9d39-6c0124c2687e",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# __*\"Human problems won't be  solved by root mean square error.\"*__ \n",
    "## <div style=\"text-align: right\">— @drewconway</div>\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aafc2f2c-b8dc-4b9a-8e43-8d9df548bdce",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}