analysis/data_analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Analysis - [`predict-actions-price`]\n",
    "\n",
    "> This document contains the main findings regarding the behavior of the target dataset, which contains features and data descriptive of your population and with potential predictive capabilities.\n",
    "\n",
    "> All the bellow analysis are **suggestions** to help guide you in your analysis. If a heading or sub-heading does not apply, feel free to not include in your notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Useful packages \n",
    "\n",
    "* Pandas profiling\n",
    "* Seaborn\n",
    "* Pandas Plot\n",
    "* Altair\n",
    "* Plotly\n",
    "* Scipy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "Provide a description of your target dataset, including its purpose, max 3 lines.\n",
    "\n",
    "\"This dataset contains information about a company's operation, growth, financial health and location in order to predict its revenue.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature (Xs Variables) Analysis\n",
    "\n",
    "Run a univariate analysis suit, such as the [Pandas Profiling](https://github.com/pandas-profiling/pandas-profiling), on a sample and talk about the main insights from the data gathered.\n",
    "\n",
    "Talk about interesting findings and show the workspace bellow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Correlogram\n",
    "Include a correlogram with the main features of the target dataset. Include observed outputs in the last column.\n",
    "\n",
    "A correlogram describes correlations between variables. The variables grouped in this case form correlation clusters. The above example indicates that the main relationships can be grouped in 4 variables within each cluster.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### QQ plots - How big is your data?\n",
    "\n",
    "According to the [law of large numbers](https://en.wikipedia.org/wiki/Law_of_large_numbers), features in very large datasets will behave according to a normal distribution. Many models used for big data are optimal only for normally distributed variables, e.g. PCA, so it can be a good idea to verify that beforehand.\n",
    "\n",
    "Include a graph with normal QQ-plots of your main numeric variables. \n",
    "This graph allows you to verify whether your data is distributed according to a Normal/Gaussian variable. The redline is the baseline model, Gaussian in this case. An alignment of samples with the baseline indicates an accordance of the distributions.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Target (Y) variable analysis \n",
    "\n",
    "Check for skewedness of data, imbalaced labels, correlation with the calculated features and any other problem it might arrise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Representativeness\n",
    "\n",
    "This is one of the most important steps of your data quality report. It describes whether the domains of your datasets allow for the specifications to be met.\n",
    "\n",
    "Describe how the Features selected and created answer the problem in hand."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Coverage of problem's scope\n",
    "\n",
    "Describe whether your  datasets contain enough samples in regions of the space that can cover the entire problem scope.\n",
    "\n",
    "\"Our problem aims at predicting every state of Brazil for every cnae. So there are two potential [strata](https://en.wikipedia.org/wiki/Stratified_sampling): state and CNAE.\"\n",
    "\n",
    "\"Analysing how the variable `state` is covered by our datasets.\"\n",
    "\n",
    "State | Samples in target set | Samples in observed set | Ratio observed/target \n",
    "------|--------------------|-------------------|-----\n",
    "`SP` | 80M (80%) | 2K (20%) |  0.0001 \n",
    "`SC` | 10K (<1%) | 8K (80%) | 0.8\n",
    "`AM` | 0 (0%) | 0 (0%) | ind\n",
    "`Other` | ~20M (15%) | 0 (0%) | 0 \n",
    "\n",
    "\"A few takeways:\n",
    "\n",
    "1. You can see that the state `AM` is not covered by the datasets. \n",
    "2. You also notice that the observed set contains most of its samples in SC, i.e. there is an imbalance of your observed dataset. \n",
    "\n",
    "There is not much you can do about Item 1, except re-specify the problem or collect more data. Item 2 will affect how well your model can reproduce targets from states other than `SC`, this might lead to a poor model outside that region. \n",
    "\n",
    "Note that such coverage analysis can be applied for every feature. It is good practice to check this for the most important features and strata."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}