Analyze_ab_test_results_notebook_AhmedHassan.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analyze A/B Test Results \n",
    "\n",
    "This project will assure you have mastered the subjects covered in the statistics lessons. We have organized the current notebook into the following sections: \n",
    "\n",
    "- [Introduction](#intro)\n",
    "- [Part I - Probability](#probability)\n",
    "- [Part II - A/B Test](#ab_test)\n",
    "- [Part III - Regression](#regression)\n",
    "- [Final Check](#finalcheck)\n",
    "- [Submission](#submission)\n",
    "\n",
    "Specific programming tasks are marked with a **ToDo** tag. \n",
    "\n",
    "<a id='intro'></a>\n",
    "## Introduction\n",
    "\n",
    "<a id='probability'></a>\n",
    "## Part I - Probability\n",
    "\n",
    "To get started, let's import our libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "#We are setting the seed to assure you get the same answers on quizzes as we set up\n",
    "random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ToDo 1.1\n",
    "\n",
    "\n",
    "<center>\n",
    "\n",
    "|Data columns|Purpose|Valid values|\n",
    "| ------------- |:-------------| -----:|\n",
    "|user_id|Unique ID|Int64 values|\n",
    "|timestamp|Time stamp when the user visited the webpage|-|\n",
    "|group|In the current A/B experiment, the users are categorized into two broad groups. <br>The `control` group users are expected to be served with `old_page`; and `treatment` group users are matched with the `new_page`. <br>However, **some inaccurate rows** are present in the initial data, such as a `control` group user is matched with a `new_page`. |`['control', 'treatment']`|\n",
    "|landing_page|It denotes whether the user visited the old or new webpage.|`['old_page', 'new_page']`|\n",
    "|converted|It denotes whether the user decided to pay for the company's product. Here, `1` means yes, the user bought the product.|`[0, 1]`|\n",
    "</center>\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "**a.** Read in the dataset from the `ab_data.csv` file and take a look at the top few rows here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0\n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0\n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0\n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0\n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('ab_data.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 294478 entries, 0 to 294477\n",
      "Data columns (total 5 columns):\n",
      "user_id         294478 non-null int64\n",
      "timestamp       294478 non-null object\n",
      "group           294478 non-null object\n",
      "landing_page    294478 non-null object\n",
      "converted       294478 non-null int64\n",
      "dtypes: int64(2), object(3)\n",
      "memory usage: 11.2+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b.** Use the cell below to find the number of rows in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "294478"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c.** The number of unique users in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "290584"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.user_id.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**d.** The proportion of users converted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11965919355605512"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.converted.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**e.** The number of times when the \"group\" is `treatment` but \"landing_page\" is not a `new_page`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3893"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[(df['group'] == 'treatment') & ~(df['landing_page'] == 'new_page')].shape[0] + df[(df['group'] == 'control') & ~(df['landing_page'] == 'old_page')].shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>308</th>\n",
       "      <td>857184</td>\n",
       "      <td>2017-01-20 07:34:59.832626</td>\n",
       "      <td>treatment</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>327</th>\n",
       "      <td>686623</td>\n",
       "      <td>2017-01-09 14:26:40.734775</td>\n",
       "      <td>treatment</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>357</th>\n",
       "      <td>856078</td>\n",
       "      <td>2017-01-12 12:29:30.354835</td>\n",
       "      <td>treatment</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>685</th>\n",
       "      <td>666385</td>\n",
       "      <td>2017-01-23 08:11:54.823806</td>\n",
       "      <td>treatment</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>713</th>\n",
       "      <td>748761</td>\n",
       "      <td>2017-01-10 15:47:44.445196</td>\n",
       "      <td>treatment</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     user_id                   timestamp      group landing_page  converted\n",
       "308   857184  2017-01-20 07:34:59.832626  treatment     old_page          0\n",
       "327   686623  2017-01-09 14:26:40.734775  treatment     old_page          0\n",
       "357   856078  2017-01-12 12:29:30.354835  treatment     old_page          0\n",
       "685   666385  2017-01-23 08:11:54.823806  treatment     old_page          0\n",
       "713   748761  2017-01-10 15:47:44.445196  treatment     old_page          0"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.query('group == \"treatment\" and landing_page != \"new_page\"').head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**f.** Do any of the rows have missing values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id         0\n",
       "timestamp       0\n",
       "group           0\n",
       "landing_page    0\n",
       "converted       0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ToDo 1.2  \n",
    "In a particular row, the **group** and **landing_page** columns should have either of the following acceptable values:\n",
    "\n",
    "|user_id| timestamp|group|landing_page|converted|\n",
    "|---|---|---|---|---|\n",
    "|XXXX|XXXX|`control`| `old_page`|X |\n",
    "|XXXX|XXXX|`treatment`|`new_page`|X |\n",
    "\n",
    "\n",
    "It means, the `control` group users should match with `old_page`; and `treatment` group users should matched with the `new_page`. \n",
    "\n",
    "However, for the rows where `treatment` does not match with `new_page` or `control` does not match with `old_page`, we cannot be sure if such rows truly received the new or old wepage.  \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "Store your new dataframe in **df2**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "list_to_drop_1 = list(df[(df['group'] == 'treatment') & ~(df['landing_page'] == 'new_page')].index)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "list_to_drop_2 = list(df[(df['group'] == 'control') & ~(df['landing_page'] == 'old_page')].index)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3893"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_index_drop = list_to_drop_1 + list_to_drop_2\n",
    "len(final_index_drop)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2 = df.drop(final_index_drop)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2[((df2['group'] == 'treatment') == (df2['landing_page'] == 'new_page')) == False].shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ToDo 1.3  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a.** How many unique **user_id**s are in **df2**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "290585"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "290584"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.user_id.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**b.** There is one **user_id** repeated in **df2**.  What is it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "773192"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "duplicate = list(df2[df2.user_id.duplicated()].user_id)[0]\n",
    "duplicate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c.** Display the rows for the duplicate **user_id**? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1899</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-09 05:37:58.781806</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2893</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-14 02:55:59.590927</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      user_id                   timestamp      group landing_page  converted\n",
       "1899   773192  2017-01-09 05:37:58.781806  treatment     new_page          0\n",
       "2893   773192  2017-01-14 02:55:59.590927  treatment     new_page          0"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display_dup = df2[df2['user_id'] == duplicate]\n",
    "display_dup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**d.** Remove **one** of the rows with a duplicate **user_id**, from the **df2** dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0\n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0\n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0\n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0\n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2 = df2.drop(display_dup.index[0])\n",
    "df2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [user_id, timestamp, group, landing_page, converted]\n",
       "Index: []"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2[df2.user_id.duplicated()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ToDo 1.4  \n",
    "\n",
    "**a.** What is the probability of an individual converting regardless of the page they receive?<br><br>\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_population = df2.converted.mean()\n",
    "p_population"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b.** Given that an individual was in the `control` group, what is the probability they converted?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1203863045004612"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "conversion_rate_control = df2.query('group == \"control\" and converted == 1').shape[0] / df2.query('group == \"control\"').shape[0]\n",
    "conversion_rate_control"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c.** Given that an individual was in the `treatment` group, what is the probability they converted?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11880806551510564"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "conversion_rate_treatment = df2.query('group == \"treatment\" and converted == 1').shape[0] / df2.query('group == \"treatment\"').shape[0]\n",
    "conversion_rate_treatment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.0015782389853555567"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obs_diff = df2['converted'][df2['group'] == 'treatment'].mean() - df2['converted'][df2['group'] == 'control'].mean()\n",
    "obs_diff"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**d.** What is the probability that an individual received the new page?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5000619442226688"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_new_page = df2.query('landing_page == \"new_page\"').shape[0] / df2.shape[0]\n",
    "p_new_page"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**e.** Consider your results from parts (a) through (d) above, and explain below whether the new `treatment` group users lead to more conversions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> as it is seen from parts (b), part (c), and (d), the probability of an individual in the control group to convert, 0.1204, is greater than the probability of an individual in the treatment group, 0.1188, to convert by about 0.0015, and the probability of an individual recieving a new page is approximately 0.5 (the half of the set which compose of only two outputs). So, treatment group did not show to leadd to more conversions initially. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='ab_test'></a>\n",
    "## Part II - A/B Test\n",
    "\n",
    "Since a timestamp is associated with each event, you could run a hypothesis test continuously as long as you observe the events. \n",
    "\n",
    "However, then the hard questions would be: \n",
    "- Do you stop as soon as one page is considered significantly better than another or does it need to happen consistently for a certain amount of time?  \n",
    "- How long do you run to render a decision that neither page is better than another?  \n",
    "\n",
    "These questions are the difficult parts associated with A/B tests in general.  \n",
    "\n",
    "\n",
    "### ToDo 2.1\n",
    "For now, consider you need to make the decision just based on all the data provided.  \n",
    "\n",
    "\n",
    "You can state your hypothesis in terms of words or in terms of **$ p_{old}$** and **$ p_{new}$**, which are the \"converted\" probability (or rate) for the old and new pages respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ H_0: p_{new} - p_{old} \\leq 0$$\n",
    "$$ H_1: p_{new} - p_{old} > 0$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ToDo 2.2 - Null Hypothesis $ H_0$ Testing\n",
    "Under the null hypothesis $ H_0$, assume that $ p_{new}$ and $ p_{old}$ are equal. Furthermore, assume that $ p_{new}$ and $ p_{old}$ both are equal to the **converted** success rate in the `df2` data regardless of the page. So, our assumption is: <br><br>\n",
    "<center>\n",
    "$ p_{new}$ = $ p_{old}$ = $ p_{population}$\n",
    "</center>\n",
    "\n",
    "In this section, I will: \n",
    "\n",
    "- Simulate (bootstrap) sample data set for both groups, and compute the  \"converted\" probability $p$ for those samples. \n",
    "\n",
    "\n",
    "- Use a sample size for each group equal to the ones in the `df2` data.\n",
    "\n",
    "\n",
    "- Compute the difference in the \"converted\" probability for the two samples above. \n",
    "\n",
    "\n",
    "- Perform the sampling distribution for the \"difference in the converted probability\" between the two simulated-samples over 10,000 iterations; and calculate an estimate. \n",
    "\n",
    "\n",
    "\n",
    "Use the cells below to provide the necessary parts of this simulation.  You can use **Quiz 5** in the classroom to make sure you are on the right track."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a.** What is the **conversion rate** for $ p_{new}$ under the null hypothesis? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "P_new_h_0 = p_population\n",
    "P_new_h_0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b.** What is the **conversion rate** for $ p_{old}$ under the null hypothesis? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "P_old_h_0 = p_population\n",
    "P_old_h_0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c.** What is $ n_{new}$, the number of individuals in the treatment group? <br><br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "145310"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_new = df2[(df2['group'] == 'treatment')].shape[0]\n",
    "n_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**d.** What is $ n_{old}$, the number of individuals in the control group?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "145274"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_old = df2[(df2['group'] == 'control')].shape[0]\n",
    "n_old"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**e. Simulate Sample for the `treatment` Group**<br> \n",
    "Simulate $ n_{new}$ transactions with a conversion rate of $ p_{new}$ under the null hypothesis.  <br><br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, ..., 0, 0, 0])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_page_converted = np.random.choice([1,0], size=n_new, p=[P_new_h_0, 1-P_new_h_0])\n",
    "new_page_converted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.12063863464317666"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_sample_new_h_0 = new_page_converted.mean()\n",
    "p_sample_new_h_0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**f. Simulate Sample for the `control` Group** <br>\n",
    "Simulate $ n_{old}$ transactions with a conversion rate of $ p_{old} $ under the null hypothesis. <br> Store these $ n_{old}$ 1's and 0's in the `old_page_converted` numpy array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 0, ..., 0, 0, 0])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "old_page_converted = np.random.choice([1,0], size=n_old, p=[P_old_h_0, 1-P_old_h_0])\n",
    "old_page_converted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11977366906672908"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_sample_old_h_0 = old_page_converted.mean()\n",
    "p_sample_old_h_0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**g.** Find the difference in the \"converted\" probability $ (p{'}_{new}$ - $p{'}_{old}) $ for your simulated samples from the parts (e) and (f) above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.00086496557644757943"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_sample_new_h_0 - p_sample_old_h_0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "**h. Sampling distribution** <br>\n",
    "Re-create `new_page_converted` and `old_page_converted` and find the $ (p{'}_{new}$ - $p{'}_{old}) $ value 10,000 times using the same simulation process I used in parts (a) through (g) above. \n",
    "\n",
    "<br>\n",
    "Store all  $ (p{'}_{new}$ - $p{'}_{old}) $  values in a NumPy array called `p_diffs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sampling distribution \n",
    "p_diffs = []\n",
    "for _ in range(10000):\n",
    "    sample_new_page_converted = np.random.choice([1,0], size=n_new, p=[P_new_h_0, 1-P_new_h_0])\n",
    "    sample_p_sample_new_h_0 = sample_new_page_converted.mean()\n",
    "    sample_old_page_converted = np.random.choice([1,0], size=n_old, p=[P_old_h_0, 1-P_old_h_0])\n",
    "    sample_p_sample_old_h_0 = sample_old_page_converted.mean()\n",
    "    sample_diff = sample_p_sample_new_h_0 - sample_p_sample_old_h_0\n",
    "    p_diffs.append(sample_diff)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.00149144, -0.00080053,  0.00063109, ..., -0.00086239,\n",
       "        0.00068609, -0.00137869])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_diffs = np.array(p_diffs)\n",
    "p_diffs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**i. Histogram**<br> \n",
    "Plot a histogram of the **p_diffs**.  Does this plot look like what you expected?  Use the matching problem in the classroom to assure you fully understand what was computed here.<br><br>\n",
    "\n",
    "Also, use `plt.axvline()` method to mark the actual difference observed  in the `df2` data (recall `obs_diff`), in the chart.  \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAEkZJREFUeJzt3X+s3fV93/HnqyaQbUmLKRfGbGd2M68q/FGSWYQp+4OVFgyJYio1kiOtsVIkVxpIidZpcpo/6NIhQbuWKlpKRYtVp0vrsCZRrMQrdVmiqtICNikhGNfzDdBwYw+7NSWpojGZvPfH+XgczPG95/449zh8ng/p6Ps97+/n+/1+Phjd1/1+P99zbqoKSVJ/fmjaHZAkTYcBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASerURdPuwHwuv/zy2rhx47S7oZVw9Ohg+eM/Pt1+SB14/PHH/6aqZhZqd0EHwMaNGzl06NC0u6GVcMMNg+VXvjLNXkhdSPLX47TzFpAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXqgv4ksLSQjbu+NLVzP3fPe6Z2bmkleAUgSZ0yACSpUwaAJHXKAJCkTi0YAEnenOSxJF9PcjjJf2z1TUkeTXIsyWeSXNzql7T3s237xqFjfbTVjya5eVKDkiQtbJwrgJeBn6qqnwSuBbYmuR64F7ivqjYDLwK3t/a3Ay9W1T8D7mvtSHI1sB24BtgK/HaSNSs5GEnS+BYMgBr4+/b2Te1VwE8Bf9zqe4Db2vq29p62/cYkafW9VfVyVT0LzALXrcgoJEmLNtYcQJI1SZ4ATgIHgG8Cf1dVZ1qTOWBdW18HPA/Qtr8E/OhwfcQ+kqRVNlYAVNUrVXUtsJ7Bb+0/MapZW+Y8285Xf40kO5McSnLo1KlT43RPkrQEi3oKqKr+DvgKcD1waZKznyReDxxv63PABoC2/UeA08P1EfsMn+OBqtpSVVtmZhb8m8aSpCUa5ymgmSSXtvV/APw0cAT4MvBzrdkO4AttfV97T9v+P6qqWn17e0poE7AZeGylBiJJWpxxvgvoKmBPe2Lnh4CHquqLSZ4G9ib5T8BfAg+29g8Cf5BklsFv/tsBqupwkoeAp4EzwB1V9crKDkeSNK4FA6CqngTeMaL+DCOe4qmq/wO8/zzHuhu4e/HdlCStND8JLEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjrl3wSWlmhaf4/Yv0WsleIVgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHVqwQBIsiHJl5McSXI4yYdb/VeSfDvJE+1169A+H00ym+RokpuH6ltbbTbJrskMSZI0jnH+JOQZ4Jeq6mtJ3go8nuRA23ZfVf3n4cZJrga2A9cA/wT4syT/vG3+JPAzwBxwMMm+qnp6JQYiSVqcBQOgqk4AJ9r6d5McAdbNs8s2YG9VvQw8m2QWuK5tm62qZwCS7G1tDQBJmoJFzQEk2Qi8A3i0le5M8mSS3UnWtto64Pmh3eZa7Xx1SdIUjB0ASd4CfBb4SFV9B7gfeDtwLYMrhN8423TE7jVP/dzz7ExyKMmhU6dOjds9SdIijRUASd7E4If/p6vqcwBV9UJVvVJV3wd+l1dv88wBG4Z2Xw8cn6f+GlX1QFVtqaotMzMzix2PJGlM4zwFFOBB4EhV/eZQ/aqhZj8LPNXW9wHbk1ySZBOwGXgMOAhsTrIpycUMJor3rcwwJEmLNc5TQO8Gfh74RpInWu2XgQ8kuZbBbZzngF8EqKrDSR5iMLl7Brijql4BSHIn8DCwBthdVYdXcCySpEUY5ymgv2D0/fv98+xzN3D3iPr++faTJK0ePwksSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnq1IIBkGRDki8nOZLkcJIPt/plSQ4kOdaWa1s9ST6RZDbJk0neOXSsHa39sSQ7JjcsSdJCxrkCOAP8UlX9BHA9cEeSq4FdwCNVtRl4pL0HuAXY3F47gfthEBjAXcC7gOuAu86GhiRp9S0YAFV1oqq+1ta/CxwB1gHbgD2t2R7gtra+DfhUDXwVuDTJVcDNwIGqOl1VLwIHgK0rOhpJ0tgWNQeQZCPwDuBR4MqqOgGDkACuaM3WAc8P7TbXauerS5KmYOwASPIW4LPAR6rqO/M1HVGreernnmdnkkNJDp06dWrc7kmSFmmsAEjyJgY//D9dVZ9r5RfarR3a8mSrzwEbhnZfDxyfp/4aVfVAVW2pqi0zMzOLGYskaRHGeQoowIPAkar6zaFN+4CzT/LsAL4wVP9gexroeuCldovoYeCmJGvb5O9NrSZJmoKLxmjzbuDngW8keaLVfhm4B3goye3At4D3t237gVuBWeB7wIcAqup0kl8FDrZ2H6+q0ysyCknSoi0YAFX1F4y+fw9w44j2BdxxnmPtBnYvpoOSpMnwk8CS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOjXOdwFJC9q460vzbt/7zN8CsH2BdpJWj1cAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnVowAJLsTnIyyVNDtV9J8u0kT7TXrUPbPppkNsnRJDcP1be22mySXSs/FEnSYoxzBfD7wNYR9fuq6tr22g+Q5GpgO3BN2+e3k6xJsgb4JHALcDXwgdZWkjQlC/49gKr68yQbxzzeNmBvVb0MPJtkFriubZutqmcAkuxtbZ9edI8lSStiOXMAdyZ5st0iWttq64Dnh9rMtdr56pKkKVlqANwPvB24FjgB/EarZ0Tbmqf+Okl2JjmU5NCpU6eW2D1J0kKWFABV9UJVvVJV3wd+l1dv88wBG4aargeOz1MfdewHqmpLVW2ZmZlZSvckSWNYUgAkuWro7c8CZ58Q2gdsT3JJkk3AZuAx4CCwOcmmJBczmCjet/RuS5KWa8FJ4CR/BNwAXJ5kDrgLuCHJtQxu4zwH/CJAVR1O8hCDyd0zwB1V9Uo7zp3Aw8AaYHdVHV7x0UiSxjbOU0AfGFF+cJ72dwN3j6jvB/YvqneSpInxk8CS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUws+BirpwrJx15emdu7n7nnP1M6tlecVgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHVqwQBIsjvJySRPDdUuS3IgybG2XNvqSfKJJLNJnkzyzqF9drT2x5LsmMxwJEnjGucK4PeBrefUdgGPVNVm4JH2HuAWYHN77QTuh0FgAHcB7wKuA+46GxqSpOlYMACq6s+B0+eUtwF72voe4Lah+qdq4KvApUmuAm4GDlTV6ap6ETjA60NFkrSKljoHcGVVnQBoyytafR3w/FC7uVY7X12SNCUrPQmcEbWap/76AyQ7kxxKcujUqVMr2jlJ0quWGgAvtFs7tOXJVp8DNgy1Ww8cn6f+OlX1QFVtqaotMzMzS+yeJGkhSw2AfcDZJ3l2AF8Yqn+wPQ10PfBSu0X0MHBTkrVt8vemVpMkTclFCzVI8kfADcDlSeYYPM1zD/BQktuBbwHvb833A7cCs8D3gA8BVNXpJL8KHGztPl5V504sS5JW0YIBUFUfOM+mG0e0LeCO8xxnN7B7Ub2TJE2MnwSWpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1alkBkOS5JN9I8kSSQ612WZIDSY615dpWT5JPJJlN8mSSd67EACRJS7MSVwD/uqquraot7f0u4JGq2gw80t4D3AJsbq+dwP0rcG5J0hJN4hbQNmBPW98D3DZU/1QNfBW4NMlVEzi/JGkMyw2AAv40yeNJdrbalVV1AqAtr2j1dcDzQ/vOtZokaQouWub+766q40muAA4k+at52mZErV7XaBAkOwHe9ra3LbN7/dm460vT7oKkHxDLugKoquNteRL4PHAd8MLZWzttebI1nwM2DO2+Hjg+4pgPVNWWqtoyMzOznO5Jkuax5ABI8o+SvPXsOnAT8BSwD9jRmu0AvtDW9wEfbE8DXQ+8dPZWkSRp9S3nFtCVwOeTnD3OH1bVnyQ5CDyU5HbgW8D7W/v9wK3ALPA94EPLOLckaZmWHABV9QzwkyPqfwvcOKJewB1LPZ8kaWX5SWBJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSp5b7ZXCSOjKtLxt87p73TOW8b3ReAUhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkTvl10BMwra/MlaTFWPUrgCRbkxxNMptk12qfX5I0sKpXAEnWAJ8EfgaYAw4m2VdVT69mPyT9YJnmVfUb+Y/RrPYVwHXAbFU9U1X/F9gLbFvlPkiSWP05gHXA80Pv54B3Tepk3ouXpPNb7QDIiFq9pkGyE9jZ3v59kqMT79XyXQ78zbQ7scoWNeZ/eXbl3vdOpDOrwH/jN76R4829U+jJ8v3TcRqtdgDMARuG3q8Hjg83qKoHgAdWs1PLleRQVW2Zdj9WU29j7m280N+YexsvrP4cwEFgc5JNSS4GtgP7VrkPkiRW+Qqgqs4kuRN4GFgD7K6qw6vZB0nSwKp/EKyq9gP7V/u8E/YDdctqhfQ25t7GC/2NubfxkqpauJUk6Q3H7wKSpE4ZAPNIclmSA0mOteXa87Tb0docS7JjqP4vknyjfe3FJ5LknP3+fZJKcvmkxzKuSY05ya8n+askTyb5fJJLV2tMoyz0lSRJLknymbb90SQbh7Z9tNWPJrl53GNO00qPN8mGJF9OciTJ4SQfXr3RjGcS/8Zt25okf5nki5MfxYRVla/zvIBfA3a19V3AvSPaXAY805Zr2/ratu0xBo/AB/jvwC1D+21gMBn+18Dl0x7rpMcM3ARc1NbvHXXcVRzjGuCbwI8BFwNfB64+p82/BX6nrW8HPtPWr27tLwE2teOsGeeYb7DxXgW8s7V5K/C/LpTxTmrMQ/v9O+APgS9Oe5zLfXkFML9twJ62vge4bUSbm4EDVXW6ql4EDgBbk1wF/HBV/c8a/F/zqXP2vw/4D5zzQbgLwETGXFV/WlVn2v5fZfAZkGkZ5ytJhv87/DFwY7ua2QbsraqXq+pZYLYd70L+mpMVH29VnaiqrwFU1XeBIww+6X+hmMS/MUnWA+8Bfm8VxjBxBsD8rqyqEwBtecWINqO+3mJde82NqJPkfcC3q+rrk+j0Mk1kzOf4BQZXB9Nyvv6PbNOC6yXgR+fZd5xjTsskxvv/tVsn7wAeXcE+L9ekxvxbDH5x+/7Kd3n1df/3AJL8GfCPR2z62LiHGFGr89WT/MN27JvGPP6KW+0xn3PujwFngE+Pea5JWLCf87Q5X33UL1MXytXdJMY72Cl5C/BZ4CNV9Z0l93DlrfiYk7wXOFlVjye5YZn9uyB0HwBV9dPn25bkhSRXVdWJdnvj5Ihmc8ANQ+/XA19p9fXn1I8Db2dwX/HrbX50PfC1JNdV1f9exlDGNoUxnz32DuC9wI3tFtG0LPiVJENt5pJcBPwIcHqBfRc65rRMZLxJ3sTgh/+nq+pzk+n6kk1izO8D3pfkVuDNwA8n+a9V9W8mM4RVMO1JiAv5Bfw6r50Q/bURbS4DnmUwGbq2rV/Wth0ErufVCdFbR+z/HBfWJPBExgxsBZ4GZi6AMV7EYOJ6E69OEF5zTps7eO0E4UNt/RpeO0H4DIMJxwWP+QYbbxjM8fzWtMe3WmM+Z98beANMAk+9Axfyi8H9wEeAY2159ofcFuD3htr9AoOJolngQ0P1LcBTDJ4i+C+0D96dc44LLQAmMubW7nngifb6nSmP81YGT658E/hYq30ceF9bfzPw31q/HwN+bGjfj7X9jvLaJ7ted8wL5bXS4wX+FYPbJU8O/Zu+7hecN9KYzzn2GyIA/CSwJHXKp4AkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnfp/zpOf+QcBNfcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe9d7ddbcc0>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(p_diffs)\n",
    "plt.axvline(obs_diff, color = 'r');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**j.** What proportion of the **p_diffs** are greater than the actual difference observed in the `df2` data?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.90680000000000005"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_value = (p_diffs > obs_diff).mean()\n",
    "p_value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**k.** Please explain in words what you have just computed in part **j** above.  \n",
    " - What is this value called in scientific studies?  \n",
    " - What does this value signify in terms of whether or not there is a difference between the new and old pages? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">it is the p-value, which is the probability of observing your data or more extreme data, if the null is true. \n",
    "\n",
    ">the value of the P-value is greater than the threshold of 0.05, as it is 0.906, so it is statistically significant because we fail to reject the null hypothesis of that $ H_0: p_{new} - p_{old} \\leq 0$ or we may produce a type I error."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "**l. Using Built-in Methods for Hypothesis Testing**<br>\n",
    "We could also use a built-in to achieve similar results.  Though using the built-in might be easier to code, the above portions are a walkthrough of the ideas that are critical to correctly thinking about statistical significance. \n",
    "\n",
    "Fill in the statements below to calculate the:\n",
    "- `convert_old`: number of conversions with the old_page\n",
    "- `convert_new`: number of conversions with the new_page\n",
    "- `n_old`: number of individuals who were shown the old_page\n",
    "- `n_new`: number of individuals who were shown the new_page\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0\n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0\n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/statsmodels/compat/pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.\n",
      "  from pandas.core import datetools\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "old converts is 17489, new converts is 17264\n",
      "old page set is 145274, new page set is 145310 \n"
     ]
    }
   ],
   "source": [
    "import statsmodels.api as sm\n",
    "\n",
    "# number of conversions with the old_page\n",
    "convert_old = df2[(df2['landing_page'] == 'old_page') & (df2['converted'] == 1)].shape[0]\n",
    "\n",
    "# number of conversions with the new_page\n",
    "convert_new = df2[(df2['landing_page'] == 'new_page') & (df2['converted'] == 1)].shape[0]\n",
    "\n",
    "# number of individuals who were shown the old_page\n",
    "n_old = df2[(df2['landing_page'] == 'old_page')].shape[0]\n",
    "\n",
    "# number of individuals who received new_page\n",
    "n_new = df2[(df2['landing_page'] == 'new_page')].shape[0]\n",
    "\n",
    "print(f\"old converts is {convert_old}, new converts is {convert_new}\\nold page set is {n_old}, new page set is {n_new} \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "count_array = np.array([convert_new, convert_old])\n",
    "nobs_array = np.array([n_new, n_old])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**m.** Now use `sm.stats.proportions_ztest()` to compute your test statistic and p-value.  \n",
    "\n",
    "The syntax is: \n",
    "```bash\n",
    "proportions_ztest(count_array, nobs_array, alternative='larger')\n",
    "```\n",
    "where, \n",
    "- `count_array` = represents the number of \"converted\" for each group\n",
    "- `nobs_array` = represents the total number of observations (rows) in each group\n",
    "- `alternative` = choose one of the values from `[‘two-sided’, ‘smaller’, ‘larger’]` depending upon two-tailed, left-tailed, or right-tailed respectively. \n",
    "\n",
    "The built-in function above will return the z_score, p_value. \n",
    "\n",
    "---\n",
    "### About the two-sample z-test\n",
    "Recall that you have plotted a distribution `p_diffs` representing the\n",
    "difference in the \"converted\" probability  $ (p{'}_{new}-p{'}_{old}) $  for your two simulated samples 10,000 times. \n",
    "\n",
    "Another way for comparing the mean of two independent and normal distribution is a **two-sample z-test**. You can perform the Z-test to calculate the Z_score, as shown in the equation below:\n",
    "\n",
    "$$\n",
    " Z_{score} = \\frac{ (p{'}_{new}-p{'}_{old}) - (p_{new}  -  p_{old})}{ \\sqrt{ \\frac{\\sigma^{2}_{new} }{n_{new}} + \\frac{\\sigma^{2}_{old} }{n_{old}}  } }\n",
    "$$\n",
    "\n",
    "where,\n",
    "- $p{'}$ is the \"converted\" success rate in the sample\n",
    "- $p_{new}$ and $p_{old}$ are the \"converted\" success rate for the two groups in the population. \n",
    "- $\\sigma_{new}$ and $\\sigma_{new}$ are the standard deviation for the two groups in the population. \n",
    "- $n_{new}$ and $n_{old}$ represent the size of the two groups or samples (it's same in our case)\n",
    "\n",
    "\n",
    ">Z-test is performed when the sample size is large, and the population variance is known. The z-score represents the distance between the two \"converted\" success rates in terms of the standard error. \n",
    "\n",
    "Next step is to make a decision to reject or fail to reject the null hypothesis based on comparing these two values: \n",
    "- $Z_{score}$\n",
    "- $Z_{\\alpha}$ or $Z_{0.05}$, also known as critical value at 95% confidence interval.  $Z_{0.05}$ is 1.645 for one-tailed tests,  and 1.960 for two-tailed test. You can determine the $Z_{\\alpha}$ from the z-table manually. \n",
    "\n",
    "Decide if your hypothesis is either a two-tailed, left-tailed, or right-tailed test. Accordingly, reject OR fail to reject the  null based on the comparison between $ Z_{score}$ and $ Z_{\\alpha}$. \n",
    "\n",
    "In other words, we determine whether or not the $ Z_{score}$ lies in the \"rejection region\" in the distribution. A \"rejection region\" is an interval where the null hypothesis is rejected iff the $ Z_{score}$ lies in that region.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-1.31092419842 0.905058312759\n"
     ]
    }
   ],
   "source": [
    "import statsmodels.api as sm\n",
    "# ToDo: Complete the sm.stats.proportions_ztest() method arguments\n",
    "z_score, p_value = sm.stats.proportions_ztest(count_array, nobs_array, alternative='larger')\n",
    "print(z_score, p_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**n.** What do the z-score and p-value you computed in the previous question mean for the conversion rates of the old and new pages?  Do they agree with the findings in parts **j.** and **k.**?<br><br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">the value of the z-score is -1.3109 which is less than the on-tailed $ Z_{\\alpha}$, which is 1.645, so it is $ Z_{score}$ < $Z_{\\alpha}$; thus we fail to reject the right-tailed hypothesis $ H_0$.\n",
    "\n",
    "> the p-value is vey similar, agrees with the findings in parts (j) and (k), to the sampling distribution that was done earlier, and that confirms the results produced.\n",
    "\n",
    ">these results confirm and agrees with the conversion rates in the population of the control group is greater than the treatment group."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='regression'></a>\n",
    "### Part III - A regression approach\n",
    "\n",
    "### ToDo 3.1 \n",
    "\n",
    "\n",
    "**a.** Since each row in the `df2` data is either a conversion or no conversion, what type of regression should you be performing in this case?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">logestic regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b.** The goal is to use **statsmodels** library to fit the regression model you specified in part **a.** above to see if there is a significant difference in conversion based on the page-type a customer receives. However, you first need to create the following two columns in the `df2` dataframe:\n",
    " 1. `intercept` - It should be `1` in the entire column. \n",
    " 2. `ab_page` - It's a dummy variable column, having a value `1` when an individual receives the **treatment**, otherwise `0`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2['intersept'] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intersept</th>\n",
       "      <th>ab_page</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   intersept  ab_page  \n",
       "0          1        0  \n",
       "1          1        0  \n",
       "2          1        1  \n",
       "3          1        1  \n",
       "4          1        0  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ab_page = pd.get_dummies(df2.group)['treatment']\n",
    "df2['ab_page'] = ab_page\n",
    "df2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c.** Use **statsmodels** to instantiate your regression model on the two columns you created in part (b). above, then fit the model to predict whether or not an individual converts. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366118\n",
      "         Iterations 6\n"
     ]
    }
   ],
   "source": [
    "x = df2[['intersept','ab_page']]\n",
    "y = df2['converted']\n",
    "log_mod = sm.Logit(y,x)\n",
    "results = log_mod.fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**d.** Provide the summary of your model below, and use it as necessary to answer the following questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "        <td>Model:</td>              <td>Logit</td>       <td>No. Iterations:</td>    <td>6.0000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <td>Dependent Variable:</td>     <td>converted</td>    <td>Pseudo R-squared:</td>    <td>0.000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "         <td>Date:</td>        <td>2022-09-26 10:57</td>       <td>AIC:</td>        <td>212780.3502</td>\n",
       "</tr>\n",
       "<tr>\n",
       "   <td>No. Observations:</td>       <td>290584</td>            <td>BIC:</td>        <td>212801.5095</td>\n",
       "</tr>\n",
       "<tr>\n",
       "       <td>Df Model:</td>              <td>1</td>         <td>Log-Likelihood:</td>  <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>Df Residuals:</td>         <td>290582</td>          <td>LL-Null:</td>      <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "      <td>Converged:</td>           <td>1.0000</td>           <td>Scale:</td>         <td>1.0000</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>       <th>Coef.</th>  <th>Std.Err.</th>     <th>z</th>      <th>P>|z|</th> <th>[0.025</th>  <th>0.975]</th> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intersept</th> <td>-1.9888</td>  <td>0.0081</td>  <td>-246.6690</td> <td>0.0000</td> <td>-2.0046</td> <td>-1.9730</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>-0.0150</td>  <td>0.0114</td>   <td>-1.3109</td>  <td>0.1899</td> <td>-0.0374</td> <td>0.0074</td> \n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary2.Summary'>\n",
       "\"\"\"\n",
       "                          Results: Logit\n",
       "==================================================================\n",
       "Model:              Logit            No. Iterations:   6.0000     \n",
       "Dependent Variable: converted        Pseudo R-squared: 0.000      \n",
       "Date:               2022-09-26 10:57 AIC:              212780.3502\n",
       "No. Observations:   290584           BIC:              212801.5095\n",
       "Df Model:           1                Log-Likelihood:   -1.0639e+05\n",
       "Df Residuals:       290582           LL-Null:          -1.0639e+05\n",
       "Converged:          1.0000           Scale:            1.0000     \n",
       "-------------------------------------------------------------------\n",
       "              Coef.   Std.Err.      z      P>|z|    [0.025   0.975]\n",
       "-------------------------------------------------------------------\n",
       "intersept    -1.9888    0.0081  -246.6690  0.0000  -2.0046  -1.9730\n",
       "ab_page      -0.0150    0.0114    -1.3109  0.1899  -0.0374   0.0074\n",
       "==================================================================\n",
       "\n",
       "\"\"\""
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.summary2()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**e.** What is the p-value associated with **ab_page**? Why does it differ from the value you found in **Part II**?<br><br>  \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">the p-value for the ab_page is 0.1899\tand it is different from the one in the A/B test section because the null and alternative hypotheses assiciated with the A/B section, which is $$H_0: p_{new} - p_{old} \\leq 0$$\n",
    "$$H_1: p_{new} - p_{old} > 0$$, is different from the hypotheses assiciated with the regression section, which is $$H_0: slope_{population} = 0$$ $$H_1:slope_{population}  ≠ 0$$\n",
    "\n",
    ">besides the hypotheses assiciated with the A/B section were one-tailed (one sided), but in the regression section it is two-tailed (two sided)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**f.** Now, you are considering other things that might influence whether or not an individual converts.  Discuss why it is a good idea to consider other factors to add into your regression model.  Are there any disadvantages to adding additional terms into your regression model?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">it is good to add more variables to the regression model as the time stamp. As we could see which period of time is assiciated with more converts, and we could use them catagorically as day, morning, evening ...etc; however, this will rise couple of issues: 1- it will make the results interpetation be harder, 2- it may lead to Multicollinearity if one or more variable are corelated to each other.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**g. Adding countries**<br> \n",
    "Now along with testing if the conversion rate changes for different pages, also add an effect based on which country a user lives in. \n",
    "\n",
    "\n",
    "1. Does it appear that country had an impact on conversion?  To answer this question, consider the three unique values, `['UK', 'US', 'CA']`, in the `country` column. Create dummy variables for these country columns. \n",
    "\n",
    "\n",
    " Provide the statistical output as well as a written response to answer this question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>834778</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>928468</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>822059</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>711597</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>710616</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id country\n",
       "0   834778      UK\n",
       "1   928468      US\n",
       "2   822059      UK\n",
       "3   711597      UK\n",
       "4   710616      UK"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3 = pd.read_csv('countries.csv')\n",
    "df3.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intersept</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   intersept  ab_page country  \n",
       "0          1        0      US  \n",
       "1          1        0      US  \n",
       "2          1        1      US  \n",
       "3          1        1      US  \n",
       "4          1        0      US  "
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_merged = df2.merge(df3, on ='user_id', how='left')\n",
    "df_merged.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intersept</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>country</th>\n",
       "      <th>CA</th>\n",
       "      <th>UK</th>\n",
       "      <th>US</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   intersept  ab_page country  CA  UK  US  \n",
       "0          1        0      US   0   0   1  \n",
       "1          1        0      US   0   0   1  \n",
       "2          1        1      US   0   0   1  \n",
       "3          1        1      US   0   0   1  \n",
       "4          1        0      US   0   0   1  "
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_merged[['CA','UK','US']] = pd.get_dummies(df_merged.country)\n",
    "df_merged.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**h. Fit your model and obtain the results**<br> \n",
    "Though you have now looked at the individual factors of country and page on conversion, we would now like to look at an interaction between page and country to see if are there significant effects on conversion.  **Create the necessary additional columns, and fit the new model.** \n",
    "\n",
    "\n",
    "Provide the summary results (statistical output), and your conclusions (written response) based on the results. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366113\n",
      "         Iterations 6\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "        <td>Model:</td>              <td>Logit</td>       <td>No. Iterations:</td>    <td>6.0000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <td>Dependent Variable:</td>     <td>converted</td>    <td>Pseudo R-squared:</td>    <td>0.000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "         <td>Date:</td>        <td>2022-09-26 10:57</td>       <td>AIC:</td>        <td>212781.1253</td>\n",
       "</tr>\n",
       "<tr>\n",
       "   <td>No. Observations:</td>       <td>290584</td>            <td>BIC:</td>        <td>212823.4439</td>\n",
       "</tr>\n",
       "<tr>\n",
       "       <td>Df Model:</td>              <td>3</td>         <td>Log-Likelihood:</td>  <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>Df Residuals:</td>         <td>290580</td>          <td>LL-Null:</td>      <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "      <td>Converged:</td>           <td>1.0000</td>           <td>Scale:</td>         <td>1.0000</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>       <th>Coef.</th>  <th>Std.Err.</th>     <th>z</th>     <th>P>|z|</th> <th>[0.025</th>  <th>0.975]</th> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intersept</th> <td>-2.0300</td>  <td>0.0266</td>  <td>-76.2488</td> <td>0.0000</td> <td>-2.0822</td> <td>-1.9778</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>-0.0149</td>  <td>0.0114</td>   <td>-1.3069</td> <td>0.1912</td> <td>-0.0374</td> <td>0.0075</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>UK</th>        <td>0.0506</td>   <td>0.0284</td>   <td>1.7835</td>  <td>0.0745</td> <td>-0.0050</td> <td>0.1063</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>US</th>        <td>0.0408</td>   <td>0.0269</td>   <td>1.5161</td>  <td>0.1295</td> <td>-0.0119</td> <td>0.0934</td> \n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary2.Summary'>\n",
       "\"\"\"\n",
       "                          Results: Logit\n",
       "==================================================================\n",
       "Model:              Logit            No. Iterations:   6.0000     \n",
       "Dependent Variable: converted        Pseudo R-squared: 0.000      \n",
       "Date:               2022-09-26 10:57 AIC:              212781.1253\n",
       "No. Observations:   290584           BIC:              212823.4439\n",
       "Df Model:           3                Log-Likelihood:   -1.0639e+05\n",
       "Df Residuals:       290580           LL-Null:          -1.0639e+05\n",
       "Converged:          1.0000           Scale:            1.0000     \n",
       "-------------------------------------------------------------------\n",
       "               Coef.   Std.Err.     z      P>|z|    [0.025   0.975]\n",
       "-------------------------------------------------------------------\n",
       "intersept     -2.0300    0.0266  -76.2488  0.0000  -2.0822  -1.9778\n",
       "ab_page       -0.0149    0.0114   -1.3069  0.1912  -0.0374   0.0075\n",
       "UK             0.0506    0.0284    1.7835  0.0745  -0.0050   0.1063\n",
       "US             0.0408    0.0269    1.5161  0.1295  -0.0119   0.0934\n",
       "==================================================================\n",
       "\n",
       "\"\"\""
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = df_merged[['intersept','ab_page','UK','US']]\n",
    "y = df_merged['converted']\n",
    "log_mod = sm.Logit(y,x)\n",
    "results = log_mod.fit()\n",
    "results.summary2()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "intersept    0.131332\n",
       "ab_page      0.985168\n",
       "UK           1.051944\n",
       "US           1.041599\n",
       "dtype: float64"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.exp(results.params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0150553002127556"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1/0.985168"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">*interpreting results:*\n",
    "\n",
    ">For every 100% decrease in new_page, convert will be about 1.5% more likely to happen, holding all other varible constant.\n",
    "\n",
    ">For every 100% increase in UK, convert is 5.2% more likely to happen, holding all other varible constant.\n",
    "\n",
    ">For every 100% increase in US, convert is 4.2% more likely to happen, holding all other varible constant.\n",
    "\n",
    ">under these results we fail to reject the null hypothesis and the p-value of the intersept is the most statistically significant factor in dertermining the converts rate.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366109\n",
      "         Iterations 6\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "        <td>Model:</td>              <td>Logit</td>       <td>No. Iterations:</td>    <td>6.0000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <td>Dependent Variable:</td>     <td>converted</td>    <td>Pseudo R-squared:</td>    <td>0.000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "         <td>Date:</td>        <td>2022-09-26 11:02</td>       <td>AIC:</td>        <td>212782.6602</td>\n",
       "</tr>\n",
       "<tr>\n",
       "   <td>No. Observations:</td>       <td>290584</td>            <td>BIC:</td>        <td>212846.1381</td>\n",
       "</tr>\n",
       "<tr>\n",
       "       <td>Df Model:</td>              <td>5</td>         <td>Log-Likelihood:</td>  <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>Df Residuals:</td>         <td>290578</td>          <td>LL-Null:</td>      <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "      <td>Converged:</td>           <td>1.0000</td>           <td>Scale:</td>         <td>1.0000</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "       <td></td>       <th>Coef.</th>  <th>Std.Err.</th>     <th>z</th>     <th>P>|z|</th> <th>[0.025</th>  <th>0.975]</th> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intersept</th>  <td>-2.0040</td>  <td>0.0364</td>  <td>-55.0077</td> <td>0.0000</td> <td>-2.0754</td> <td>-1.9326</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>    <td>-0.0674</td>  <td>0.0520</td>   <td>-1.2967</td> <td>0.1947</td> <td>-0.1694</td> <td>0.0345</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>UK</th>         <td>0.0118</td>   <td>0.0398</td>   <td>0.2957</td>  <td>0.7674</td> <td>-0.0663</td> <td>0.0899</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>US</th>         <td>0.0175</td>   <td>0.0377</td>   <td>0.4652</td>  <td>0.6418</td> <td>-0.0563</td> <td>0.0914</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>UK_ab_page</th> <td>0.0783</td>   <td>0.0568</td>   <td>1.3783</td>  <td>0.1681</td> <td>-0.0330</td> <td>0.1896</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>US_ab_page</th> <td>0.0469</td>   <td>0.0538</td>   <td>0.8718</td>  <td>0.3833</td> <td>-0.0585</td> <td>0.1523</td> \n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary2.Summary'>\n",
       "\"\"\"\n",
       "                          Results: Logit\n",
       "==================================================================\n",
       "Model:              Logit            No. Iterations:   6.0000     \n",
       "Dependent Variable: converted        Pseudo R-squared: 0.000      \n",
       "Date:               2022-09-26 11:02 AIC:              212782.6602\n",
       "No. Observations:   290584           BIC:              212846.1381\n",
       "Df Model:           5                Log-Likelihood:   -1.0639e+05\n",
       "Df Residuals:       290578           LL-Null:          -1.0639e+05\n",
       "Converged:          1.0000           Scale:            1.0000     \n",
       "-------------------------------------------------------------------\n",
       "               Coef.   Std.Err.     z      P>|z|    [0.025   0.975]\n",
       "-------------------------------------------------------------------\n",
       "intersept     -2.0040    0.0364  -55.0077  0.0000  -2.0754  -1.9326\n",
       "ab_page       -0.0674    0.0520   -1.2967  0.1947  -0.1694   0.0345\n",
       "UK             0.0118    0.0398    0.2957  0.7674  -0.0663   0.0899\n",
       "US             0.0175    0.0377    0.4652  0.6418  -0.0563   0.0914\n",
       "UK_ab_page     0.0783    0.0568    1.3783  0.1681  -0.0330   0.1896\n",
       "US_ab_page     0.0469    0.0538    0.8718  0.3833  -0.0585   0.1523\n",
       "==================================================================\n",
       "\n",
       "\"\"\""
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_merged['UK_ab_page'] = df_merged['UK']*df_merged['ab_page']\n",
    "df_merged['US_ab_page'] = df_merged['US']*df_merged['ab_page']\n",
    "x = df_merged[['intersept', 'ab_page', 'UK', 'US', 'UK_ab_page', 'US_ab_page']]\n",
    "y = df_merged['converted']\n",
    "logit3 = sm.Logit(y,x)\n",
    "\n",
    "results = logit3.fit()\n",
    "results.summary2()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "intersept     0.134794\n",
       "ab_page       0.934776\n",
       "UK            1.011854\n",
       "US            1.017682\n",
       "UK_ab_page    1.081428\n",
       "US_ab_page    1.048001\n",
       "dtype: float64"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.exp(results.params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.069775004920965"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1/0.934776"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">*interpreting results:*\n",
    "\n",
    "> only intercept's p-value is less than 0.05, which is statistically significant. Other varable in the summary are not statistically significant.\n",
    "\n",
    ">So, the country a user lives is not significant for the converted rate considering the page.\n",
    "\n",
    ">Convert is about 8.1% more likely to happen for UK and new page users more than CA, holding all other varible constant.\n",
    "\n",
    ">Convert is about 4.8% more likely to happen for US and new page users more than CA, holding all other varible constant.\n",
    "\n",
    ">Convert is about 6.9% more likely to happen for every 100% decrease in the new page, holding all other varible constant."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}