wrangle_report.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reporting: wrangle_report\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## WeRateDogs\n",
    "\n",
    "![](dog-rates-social.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction\n",
    "\n",
    "This project involves the wrangling, analyzing, and visualizing of datasets associated with the tweet archive of Twitter user [@dog_rates](https://twitter.com/dog_rates), also known as [WeRateDogs](https://en.wikipedia.org/wiki/WeRateDogs). From the documentation given, WeRateDogs is a Twitter account that rates people's dogs with a humorous comment about the dog. These ratings almost always have a denominator of 10 with over 4 million followers and ample international media coverage. The datasets used in this project will be gathered from various sources, assessed, cleaned, and insights will be drawn from them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Gathering\n",
    "Data was gathered from various sources and by various means:\n",
    "\n",
    "1. \"WeRateDogs\" twitter data('csv file') was accessed using the pandas read csv method\n",
    "2. Image prediction data('tsv file') was also accessed via same method with slight modification.\n",
    "3. Extraneous data was accessed using twitter's(API) and \"tweepy\" library(though issues with candidates twitter account, meant alternative was used)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Assessing\n",
    "\n",
    "Assesment was done both visually and programatically and the following issues were highlighted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Quality issues\n",
    "1. Retweets have to be removed.\n",
    "\n",
    "2. All relevant variables in datasets should have the correct data type.\n",
    "\n",
    "3. Twitter archives dataset contains columns with missing data.\n",
    "\n",
    "4. \"expanded_urls\" column had missing values which can be generated.\n",
    "\n",
    "5. There were quite a few names registered as \"none\".\n",
    "\n",
    "6. There are entries with numerators which are not reasonable.\n",
    "\n",
    "7. Entries with inconsistent denominators also exist.\n",
    "\n",
    "8. Less relevant variables have to be removed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "extensions": {
     "jupyter_dashboards": {
      "version": 1,
      "views": {
       "grid_default": {
        "col": 0,
        "height": 7,
        "hidden": false,
        "row": 40,
        "width": 12
       },
       "report_default": {
        "hidden": false
       }
      }
     }
    }
   },
   "source": [
    "##### Tidiness issues\n",
    "1. The various dog categories can be collapsed into one column.\n",
    "\n",
    "2. The rating_denominator column is pretty much redundant."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Cleaning\n",
    "The cleaning process consisted of both manual and programatic means where all issues identified were fixed using the define-code-test framework."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is to be noted though that while the wrangling process was relatively sucessful, the data set contained myriads of unresolved issues which could constitute errors in subsequent analytical sessions if left as is."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data wrangling process was carried out successfully for the relevant data. Different quality and tidiness issues were highlighted, cleaned, and tested using the relevant framework.\n",
    "Insights were then drawn from the cleaned data which coupled with some drawn up visualizations showed some pretty interesting details. However, it should be noted that the datasets in uestion still had some issues to be corrected, which coupled with appropriate domain knowledge should result in even more impressive and accurate insights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}