01-descriptive-statistics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 01 Descriptive statistics - analysis of data set\n",
    "\n",
    "This notebook is part of a series for the analysis of data sets in the class \"multivariate statistics\". Note that running this notebook requires you to have the downloaded dataset, ideally in the same folder (otherwise, you'll need to adjust the path to the dataset below). \n",
    "\n",
    "(c) Florian Wellmann, CGRE, 2021\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparation steps\n",
    "\n",
    "Before we can perform any type of statistical analysis, we first need to load Python modules that enable us to load and analyse data (the basic Python installation is, well, basic... for any reasonable type of work, you need to add functionality by loading modules).\n",
    "\n",
    "For the descriptive and exploratory data analysis performed here, it is sufficient to load `pandas`, a Python module for data analysis (https://pandas.pydata.org):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now load the Excel table directly into a pandas object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = pd.read_excel(\"HyChemDaten_bearbeitet_2021_22.xls\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Viewing the data\n",
    "\n",
    "As a first simple step, we can execute the variable name to get a quick overview:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you see, the data set does not make any sense - what went wrong?\n",
    "\n",
    "If you go back to the dataset and open the table in Excel, you will observe that we have multiple Excel sheets - and by defaul, Pandas opens the first sheet. However: our (cleaned) data set is actually in the sheet called \"SPSS\". \n",
    "\n",
    "In order to load this specific sheet, we have to provide the sheet name as argument to the `pandas.read_excel()` function:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = pd.read_excel(\"HyChemDaten_bearbeitet_2021_22.xls\", sheet_name=\"SPSS\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, this data makes a lot more sense now - let's start to analyse it!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Descriptive statistics\n",
    "\n",
    "Performing a simple descriptive analysis with Pandas is, well, simple:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This table directly combines estimates of mean and standard devitation (`std`), as well as the percentile measures `min, 25%, 50% (median), 75%, max`.\n",
    "\n",
    "If required, we can also extract these measures directly (but do not get the nice table view back):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.median()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The full descriptive table is also quite long - it may well be the case that we only want to analyse a subset of the data. We have two options:\n",
    "\n",
    "1. We could create a new table with only a subset of the data;\n",
    "2. We can adjust the command to only provide results for a subset of the entire table.\n",
    "\n",
    "Let's assume we want to analyse the values of `K, Na, CA2`, as well as `pH` and `Redox`. We can create a table with only these columns extracted:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_sub = dataset[['K', 'Na', 'Ca2', 'pH', 'Redox']].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_sub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_sub.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the second option, we can extract the relevant columns from the dataframe using square brackets (note the double-brackets):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset[['K', 'Na']].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploratory data analysis - first steps\n",
    "\n",
    "Looking at tables is nice - but especially for large tables as the one analysed here, it quickly becomes overwhelming. In addition to the descriptive (numeric) analysis, we often create overview plots. Here some examples:\n",
    "\n",
    "### Histogram\n",
    "\n",
    "A histogram is a quick and easy way to visualise the distribution of data in a data set. It can easily be created directly from `pandas`.\n",
    "\n",
    "You _can_ create a histogram for all the columns in your entire data set at once - but beware, this can result in a big plot (in the case of our entire dataset):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.hist(); # Nopte: the semicolon at the end surpresses the text output - try removing it to see the effect"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It may make more sense to look at one column at a time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.hist(column='K')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or maybe at multiple columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.hist(column=['K', 'Na']);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many options to adjust these diagrams - for example, we can make the entire figure bigger so that we can actually see something:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.hist(column=['K', 'Na'], figsize=(12,6));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And it may be desirable to keep the y-axis the same for a better comparison, rotate axis labels, etc. etc.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = dataset.hist(column=['K', 'Na'], figsize=(12,6), sharey=True, rwidth=0.9,\n",
    "                   color='#607c8e', alpha=0.8, xrot=45);\n",
    "ax[0,0].set_xlabel('mg/l')\n",
    "ax[0,1].set_xlabel('mg/l')\n",
    "ax[0,0].set_ylabel('count')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Your task**: in class, we discussed the relevance of the choice of bin numbers - try adjusting the number of bins with the additional keyword (`bins=N`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.hist(column='Na', bins=20);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try this out for a couple of data columns. When do you reach a limit where increasing the number of bins leads to non-sensical results?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Your Answer here:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We talked about two ways to circumvent the problem of having to choose the number of bins - one is the use of a smoothed function (e.g., Kernel density), and the other option is to use cumulative plots.\n",
    "\n",
    "Both types of plots can also easily be created with `pandas`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset['K'].plot.kde();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset['K'].plot.hist(cumulative=True, bins=10, alpha=0.6);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Your task**: In the case of the cumulative plot: play around with the number of bins (just as above in the case of the normal histogram)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}