01_ex_AQM_2021_stud.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "AQM 2021 - Ex1-stud.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyOSNsr2huBGtYrJ/e7q/pp0",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/RJuro/Africalics-PhD-Academy-2018/blob/master/01_ex_AQM_2021_stud.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1CeV09vEmrLW"
      },
      "source": [
        "# Sampling\n",
        "\n",
        "We will start with a well known dataset - AirBnb (from the statistics exam E2020).\n",
        "Here I just copy-paste first the loading and preprocessing bits. This includes:\n",
        "- removing outliers (remove everything over 99. percentile)\n",
        "- removing missing data\n",
        "- some cleaning up of the price-column and recoding into floats"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JWfQj4mOmVoE"
      },
      "source": [
        "# loading packages for the day\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "sns.set()\n",
        "\n",
        "# Indlæs data\n",
        "df = pd.read_csv('https://sds-aau.github.io/IntroStat/Data/airbnb.csv.gz', compression='gzip')\n",
        "\n",
        "#Drop variable alle host_ variable på nær host_id (den skal bruges til analyse)\n",
        "#Drop også \"neighbourhood\", \"city\", 'square_feet'\n",
        "\n",
        "# du kan bruge df.columns komando for nemmere copy-paste af variablenavne \n",
        "\n",
        "\n",
        "df.drop(['host_name', 'host_since', 'host_location',\n",
        "       'host_acceptance_rate', 'host_is_superhost', 'host_listings_count', 'neighbourhood', 'city','square_feet'], axis = 'columns', inplace=True)\n",
        "\n",
        "# Udsmid manglende værdier i variable hvor der mangler op til og med 103 observationer \n",
        "# dvs. variable med flere manglende observationer end 103 skal ikke renses for manglende observationer\n",
        "\n",
        "df.dropna(subset=['name', 'bathrooms', 'bedrooms', 'beds'], inplace=True)\n",
        "\n",
        "# Her omdannes \"price\" til en float (numerisk variabel) og får fjernet \"$\" og \",\" - det har vi gjort for jer.\n",
        "df.price = df.price.replace('[\\$,]', '', regex=True).astype(float)\n",
        "\n",
        "# Med udgangspunkt i \"price\", beskær datasættet til 99. percentil (95 inkl.)\n",
        "df = ..."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2i04kcGicjCy"
      },
      "source": [
        "Let's look at some ways to visualize the distribution of the price"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4P2NELmyOYSc"
      },
      "source": [
        "# histogram\n",
        "..."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WmWTn0OwcokF"
      },
      "source": [
        "# density plot\n",
        "..."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_vmzxPo6csWY"
      },
      "source": [
        "# empirical cumulative distribution function\n",
        "..."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "su66BmSMOtur"
      },
      "source": [
        "# summary statistics\n",
        "..."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S8ZDIhhjOnAN"
      },
      "source": [
        "# taking a sample of 100 observations\n",
        "sample = ...\n",
        "sample.describe()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YbxrWiAnegEM"
      },
      "source": [
        "# taking a sample of 1000 observations\n",
        "sample = ...\n",
        "sample.describe()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GNylsWLegB-O"
      },
      "source": [
        "### Let's experiment"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AFAEc5xpej_f"
      },
      "source": [
        "# let's try taking a small sample many times, where we change the number of draw and number of samples\n",
        "\n",
        "\n",
        "..."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fJPqk9FIOf6b"
      },
      "source": [
        "# comparing sample to population\n",
        "\n",
        "fig, ax = plt.subplots(1,2)\n",
        "sns.ecdfplot(df.price, ax=ax[0])\n",
        "sns.ecdfplot(sample, ax=ax[1])\n",
        "fig.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qdDFBbHtqd9p"
      },
      "source": [
        "# Let's put more samples on the plot\n",
        "fig, ax = plt.subplots()\n",
        "\n",
        "...\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GqIQafpghZF5"
      },
      "source": [
        "One of the core ideas here is to figure out how much data do I need to know:\n",
        "- what's the average, meadian room cost / night\n",
        "- what's the probability of finding an affordable room given that I want to spend less than 1000 / night."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vYMKySXmm4ZX"
      },
      "source": [
        "# Random number generation and Bernoulli trials\n",
        "\n",
        "Let's return to random number generation from the lecture. We will use numpy's random number generation tools here rather than the more basic python random module that you have seen in the lecture.\n",
        "\n",
        "#### Simple random number generation between 0-1 and distribution"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ab4uuL_-lZnz"
      },
      "source": [
        "# Let's try\n",
        "np.random.random()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lpFux7ezlfd_"
      },
      "source": [
        "# We get values between 0 and 1\n",
        "np.random.random(...)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q6dPtBX0lntH"
      },
      "source": [
        "Let's explore the distribution of what the random number generator is producing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nb8Mk0qsnCFB"
      },
      "source": [
        "# Seed the random number generator - that's for reproduciability. Remember these generators are \"pseudo-random\"\n",
        "np.random.seed(21)\n",
        "\n",
        "# Instead of a list, we will use an empty numpy array here\n",
        "random_numbers = ...\n",
        "# Generate random numbers by looping over range(100000)\n",
        "...\n",
        "\n",
        "# Plot a histogram\n",
        "..."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G2Ft2kACjbTh"
      },
      "source": [
        "# and here is a ecdf version of that\n",
        "..."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BI80uA-elvgp"
      },
      "source": [
        "Now, let's extend that to something where we include a probability parameter $p$ that determines the probability of sucess of a \"Trial\". The parameter $n$ is the number of trials. Imagine flipping a coin $n$ times knowing that it is fair $p = 0.5$ or biased/manipulated $p=0.3$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ghKdM4Qqq5v3"
      },
      "source": [
        "# We will now define a function that takes in n and p and returns the number of \"sucess\" cases\n",
        "\n",
        "def perform_bernoulli_trials(n, p):\n",
        "    \"\"\"Perform n Bernoulli trials with success probability p\n",
        "    and return number of successes.\"\"\"\n",
        "\n",
        "    # Initialize number of successes: n_success\n",
        "    n_success = ... #we start at 0 and increment by one every time there is success\n",
        "\n",
        "    # Perform trials\n",
        "    for i in range(n):\n",
        "\n",
        "        # Generate random number between zero and one: random_number\n",
        "        random_number = ... # we know that already\n",
        "\n",
        "        # If less than p, it's a success so add one to n_success\n",
        "        \n",
        "    return n_success"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JYu8yx90ouvV"
      },
      "source": [
        "# Let's try\n",
        "\n",
        "perform_bernoulli_trials(100, 0.5)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yvDxotPpqDok"
      },
      "source": [
        "### Exercise: Crowdlending and defaults\n",
        "A quick-loan company on a crowdlending platform has 1000 loans. On average 3% of all loans have been defaulting per year.\n",
        "What's the probability that in any given year no loan defaults?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2l7gStN3rNp4"
      },
      "source": [
        "# Seed random number generator\n",
        "np.random.seed(21)\n",
        "\n",
        "# Initialize the number of defaults: n_defaults\n",
        "n_defaults = np.empty(10000)\n",
        "\n",
        "# Compute the number of defaults\n",
        "..."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0AmsDLfWdiuP"
      },
      "source": [
        "# Let's use seaborn to visualise that\n",
        "ax = sns.displot(n_defaults, kind='hist', stat='probability', bins=10)\n",
        "ax.set(xlabel='number of defaults out of 1000 loans', ylabel='probability')\n",
        "\n",
        "\n",
        "# Show the plot\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3_t3JVMFDJzX"
      },
      "source": [
        "# Let's use seaborn to visualise that\n",
        "ax = sns.displot(n_defaults, kind='ecdf')\n",
        "ax.set(xlabel='number of defaults out of 1000 loans', ylabel='probability')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oazASfl-DZoE"
      },
      "source": [
        "What's the probability of no loans defualt?\n",
        "We know that we the company is going to lose money when more than 50 loans default. What's the probability of losing money?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U_pKkLgMsxJQ"
      },
      "source": [
        "# Compute the number of 10000-loan simulations with 0 defaults\n",
        "no_defaults = ...\n",
        "\n",
        "# Compute the number of 10000-loan simulations with 50 or more defaults: n_lose_money\n",
        "n_lose_money = ...\n",
        "\n",
        "# Compute and print probability of having a great year!\n",
        "print('Probability of no default =', ...)\n",
        "\n",
        "# Compute and print probability of losing money\n",
        "print('Probability of losing money =', ...)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nmB1tCoSnwm7"
      },
      "source": [
        "# Binomial Distribution\n",
        "\n",
        "What we have done so far is working with a binomial distribution. We can do the same exercise without our \"self-made\" fuction above.\n",
        "\n",
        "Let's try another example:\n",
        "\n",
        "### Car Insurance\n",
        "\n",
        "A car insurance has 500 policies. Per year, there will be on average 1 fatal crash. The we assume that the number of crashes has a binomial distribution. What's the probability that no fatal crashes will happen in 1 year."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2In-ZbxZn1ts"
      },
      "source": [
        "# Take 10,000 samples out of the binomial distribution: n_defaults\n",
        "n_crashes = ...\n",
        "\n",
        "# Plot the histogram with default number of bins; label your axes\n",
        "ax = sns.displot(n_defaults, kind='ecdf')\n",
        "ax.set(xlabel='number of fatal crashes out of 500', ylabel='CDF')\n",
        "\n",
        "# Show the plot\n",
        "plt.show()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QQGFgQByFis-"
      },
      "source": [
        "# Compute the number of 10000-insurance simulations with 0 fatalities\n",
        "no_crashes = ...\n",
        "\n",
        "# Compute and print probability of not crashing!\n",
        "print('Probability of no fatality =', ...)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ncUEvVFoGkSG"
      },
      "source": [
        "#### Formal add on:\n",
        "\n",
        "Binomial coefficient (n choose k)\n",
        "$\n",
        "    \\binom{n}{k} = \\frac{n!}{k!(n-k)!}\n",
        "$\n",
        "\n",
        "The probability mass function is here the following\n",
        "\n",
        "$ f(k, n, p) = Pr(k;n,p) = Pr(X=k) = \\binom{n}{k} p^k(1-p)^{n-k}$\n",
        "\n",
        "$p$: average probability\n",
        "$k$: number of successes\n",
        "$n$: number of trials"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dNWHm0YRGnmJ"
      },
      "source": [
        "from sympy import *\n",
        "\n",
        "k, n, p = symbols('k, n, p')\n",
        "\n",
        "(factorial(n)/(factorial(k)*factorial(n-k)))*p**k*(1-p)**(n-k)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-N0d9gCnIjaV"
      },
      "source": [
        "# calculating the expected value using math...\n",
        "(factorial(500)/(factorial(0)*factorial(500-0)))*1/500**0*(1-1/500)**(500-0)"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}