scraper-checkpoint-1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>THE WHAT INFLUENCES ARE IN THE AUTO INDUSTRY</h1>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src =https://n6-img-fp.akamaized.net/free-vector/petrol-station-design_1454-41.jpg?size=338&ext=jpg>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>MOTIVATION</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>When thinking about what type of final project we wanted to do, we wanted a topic that we can extract years of data from as well as be a topic that applies to current times. What we have selected was the Fuel economy data that contained fuel economy data from as far as 1978 but for this project we'll be focusing on years 2001 - 2017. The reason we used this time range is due to only wanting to use data within the 2000s while having enough data for formulate more accurate findings. With this data, we also wanted to have a correlations with the different categories like manufacturer, classification of the cars in the dataset as well as fuel cost and such. By the end of this tutorial, you'll find out what type of influence fuel pays a part in the auto industry while using past data to make a educated guess about the future for the current year.\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> THE CHALLENGES AHEAD </h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>We have chosen a website such that data is vast and filled with subjects that we can use. This was a double edged sword however because even though we had a lot of data to work with, we had to decide with data to not use and which data to we can use to benefit us. One of the biggest problems we ran into was that for our data set, the classification would change in spelling despite representing the same thing as pervious years or for the more recent datasets that we scrapped new classifications were created. With such a mix /mash of data varying in time periods, we had to find the similairities and use the common data in favor of us for this project.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>*\\BE SURE TO HAVE LXML INSTALLED ON YOUR NOTEBOOK OR ELSE THE READIND TIDYING OF THE DATA WON'T WORK*/</h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: lxml in /opt/conda/lib/python3.6/site-packages\r\n"
     ]
    }
   ],
   "source": [
    "!pip install lxml\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd\n",
    "import requests\n",
    "import re\n",
    "import numpy as np\n",
    "import zipfile\n",
    "import os\n",
    "import shutil\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<p>Open url for scraping. All data sources are stored within a table. All target data files have the\n",
    "same naming scheme, we will take advantage of this</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "url = 'http://www.fueleconomy.gov/feg/download.shtml'\n",
    "page = requests.get(url)\n",
    "soup = BeautifulSoup(page.content, 'lxml')\n",
    "\n",
    "\n",
    "reg = 'epadata/\\d{2}data.zip' ##luckily our target has a unique identifier for href\n",
    "suf = 'http://www.fueleconomy.gov/feg/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<p>Find target data sources within html page</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "links = soup.find_all('a')\n",
    "zlist = []\n",
    "for tag in links:\n",
    "    link = tag.get('href',None)\n",
    "    if link is not None:\n",
    "        if re.match(reg, link):\n",
    "            zlist.append(link)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<p>Download data to current working directory. These files are zipped. Unzip files</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "filelst = []\n",
    "for x in zlist:\n",
    "    url = suf+x\n",
    "    localfn = url.split('/')[-1]\n",
    "    yearno = int(localfn[:2])\n",
    "    if yearno < 50 and yearno > 00:\n",
    "        r = requests.get(url, stream=True)\n",
    "        with open(localfn, 'wb') as f:\n",
    "            for chunk in r.iter_content(chunk_size=1024): \n",
    "                if chunk: # filter out keep-alive new chunks\n",
    "                    f.write(chunk)\n",
    "        zip_ref = zipfile.ZipFile(localfn, 'r')\n",
    "        lcl = \"dldir\"+localfn\n",
    "        filelst.append(lcl)\n",
    "        zip_ref.extractall(lcl)\n",
    "        zip_ref.close()\n",
    "       \n",
    "        \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<p>create directory to store data</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "datdir = \"shibe_data\"\n",
    "if not os.path.exists(datdir):\n",
    "    os.makedirs(datdir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<p>extract downloaded data to single directory. Rename files so that they have uniform names</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: 'dldir18data.zip\\\\2018 FE Guide for DOE-release dates before 12-14-2017-no-sales-12-12-2017public.xlsx' -> 'shibe_data\\\\dldir18data.xlsx'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-61-259b66af861f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      8\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mm\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      9\u001b[0m             \u001b[0mnewname\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdatdir\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m\"\\\\\"\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mitem\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m4\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mm\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgroup\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 10\u001b[0;31m             \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrename\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcurname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnewname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'dldir18data.zip\\\\2018 FE Guide for DOE-release dates before 12-14-2017-no-sales-12-12-2017public.xlsx' -> 'shibe_data\\\\dldir18data.xlsx'"
     ]
    }
   ],
   "source": [
    "reg = \"(.xlsx|.xls|.csv)$\"\n",
    "for item in filelst:\n",
    "    for filename in os.listdir(item):\n",
    "        curname =  item + \"\\\\\" + filename \n",
    "        \n",
    "        \n",
    "        m = re.search(reg, filename)\n",
    "        if m:\n",
    "            newname = datdir + \"\\\\\" + item[:-4] + m.group(0)\n",
    "            os.rename(curname, newname)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<p>clean up files and directories created while scraping data</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "lst = os.listdir(os.getcwd())\n",
    "for x in lst:\n",
    "    m = re.search(\".zip$\", x)\n",
    "    if m:\n",
    "        n = re.search(\"^dldir\",x)\n",
    "        if n:\n",
    "            shutil.rmtree(x)\n",
    "        else:\n",
    "            os.remove(x)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<p> Reads data files into Pandas Dataframes according to file type (.csv .xlx .xlsx). The df are stored in a temporary dict according to their respective years. The original zip file from the EPA site does not give year in YYYY format, so we have chosen not to use the YYYY format until this point where it becomes necessary and after we have gleaned out only the years prefixed by \"20-\"Some lines of the original data files are formed incorrectly. Adding the option \"error_bad_lines=False\" skips these lines<p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "datfiles = os.listdir(datdir)\n",
    "pdfiles = {}\n",
    "for x in datfiles:\n",
    "    y = re.search(\"[0-9]{2}\",x)\n",
    "    year = \"20\" + y.group(0)\n",
    "    cs = re.search(\".csv$\", x)\n",
    "    if cs:\n",
    "        newp = pd.read_csv(datdir + \"\\\\\" + x, error_bad_lines=False)\n",
    "    else:\n",
    "        newp = pd.read_excel(datdir + \"\\\\\" + x)\n",
    "        \n",
    "    pdfiles[year] = newp\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>Went through all the data files from 2001-2018 and found all the persisting variables and listed their aliases in a dict, vars</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "varss = {}\n",
    "varss['Class'] = ['CLASS','Carline Class Desc', 'Class']\n",
    "varss['Manufacturer'] = ['MFR', 'Manufacturer', 'Mfr Name', 'Mfr Name ']\n",
    "varss['carline name'] = ['CAR LINE', 'Carline', 'carline name']\n",
    "varss['cmb'] = ['COMB MPG (GUIDE)', 'Comb FE (Guide) - Conventional Fuel', 'cmb']\n",
    "varss['cty'] = ['CITY MPG (GUIDE)', 'City FE (Guide) - Conventional Fuel', 'cty']\n",
    "varss['cyl'] = ['# Cyl', 'NUMB CYL', 'cyl']\n",
    "varss['displ'] = ['DISPLACEMENT', 'Eng Displ', 'displ']\n",
    "varss['fcost'] = ['ANL FL CST', 'Annual Fuel1 Cost - Conventional Fuel', 'fcost']\n",
    "varss['fl'] = ['FUEL TYPE', 'Fuel Usage Desc - Conventional Fuel', 'fl']\n",
    "varss['hwy'] = ['HWY MPG (GUIDE)', 'Hwy FE (Guide) - Conventional Fuel', 'hwy']\n",
    "varss['trans'] = ['TRANS', 'Trans as listed in FE Guide (derived from col AA thru AF)', 'Trans in FE Guide (MFR entered for data entered after May 13 2011)', 'Transmission', 'trans']\n",
    "varss['ucmb'] = ['Comb Unadj FE - Conventional Fuel', 'UNRND COMP (EPA)', 'ucmb']\n",
    "varss['ucty'] = ['City Unadj FE - Conventional Fuel', 'UNRND CITY (EPA)', 'ucty']\n",
    "varss['uhwy'] = ['Hwy Unadj FE - Conventional Fuel', 'UNRND HWY (EPA)', 'uhwy']\n",
    "varss['drv'] = ['DRIVE SYS', 'Drive Sys', 'drv']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p> Finds all the column names which are included inside of the shared variable map and replaces the dataframe \n",
    "with only those columns. Renames them to the shared variable map's key names, so that column names can be \n",
    "standardized. The columns are reordered before replacing the dataframes inside the original master map of df</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "varlst = list(varss.keys())\n",
    "yearlst = list(pdfiles.keys())\n",
    "\n",
    "\n",
    "for z in yearlst:\n",
    "    curpd = pdfiles[z]\n",
    "    cols = list(curpd.columns.values)\n",
    "    checklst = []\n",
    "    rename = []\n",
    "    \n",
    "    for x in cols:\n",
    "        for y in varlst:\n",
    "            if x in varss[y]:\n",
    "                checklst.append(x)\n",
    "                rename.append(y)\n",
    "    curpd = curpd[checklst]\n",
    "    curpd.columns = rename\n",
    "    curpd = curpd[varlst]\n",
    "    curpd['year'] = z\n",
    "    pdfiles[z] = curpd\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<p>Finds all car classes from over the years. Aliases for car classes are \n",
    "combined in the carclass dict. Some types (small SUV and standard SUV are \n",
    "combined since they were originally the same category in previous years).<p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cartypes = []\n",
    "for x in yearlst:\n",
    "    curpd = pdfiles[x]\n",
    "    temp = set(curpd['Class'].values)\n",
    "    cartypes.extend(list(temp))\n",
    "cartypes = list(set(cartypes))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "carclass = {}\n",
    "\n",
    "carclass['COM'] = ['COMPACT CARS','Compact Cars','SUBCOMPACT CARS','Subcompact Cars','MINICOMPACT CARS','Minicompact Cars']\n",
    "carclass['LAR'] = ['LARGE CARS','Large Cars']\n",
    "carclass['MID'] = [ 'MIDSIZE CARS','Midsize Cars']\n",
    "carclass['MINIV'] = ['MINIVAN - 2WD','Special Purpose Vehicle, minivan 2WD','SPEC PURP VEH - MINIVAN - 2WD','SPEC PURP VEH - MINIVAN - 4WD','Special Purpose Vehicle, minivan 4WD','MINIVAN - 4WD']\n",
    "carclass['PU'] = ['SMALL PICKUP TRUCKS 4WD','Small Pick-up Trucks 4WD', 'Small Pick-up Trucks 2WD','STANDARD PICKUP TRUCKS 2WD','Standard Pick-up Trucks 2WD','Standard Pick-up Trucks 4WD']\n",
    "\n",
    "carclass['SW'] = ['MIDSIZE STATION WAGONS','SMALL STATION WAGONS','Small Station Wagons','Midsize Station Wagons']\n",
    "carclass['SUV'] = ['SPEC PURP VEH - S.U.V. - 4WD','S.U.V. - 4WD','Small SUV 4WD','Special Purpose Vehicle, SUV 4WD','Standard SUV 4WD','SPEC PURP VEH - S.U.V. - 2WD','S.U.V. - 2WD','Small SUV 2WD','Special Purpose Vehicle, SUV 2WD','Standard SUV 2WD']\n",
    "\n",
    "carclass['2S'] = ['TWO SEATERS','Two Seaters']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pig = sns.light_palette(\"green\",18)\n",
    "piv = sns.light_palette(\"purple\",18)\n",
    "pir= sns.light_palette(\"red\",18)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<p> You can are going to get retrieve any 10 manufacturers and store them in an array that can be later used for accessing a subset of the data given to us. By doing this you can start being to isolate the data per manufacturer and to see if there's any evident correlation or more data to be found. The same applies to the class of the car but we'll use ,4 unlike the manufacturer. There are more than 10 manufacturers and 4 class of cars in the data set but we want to zoom in to certain classifications and see if the correlation is manufacturer dependent or if the data is the same or similar across the board.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "manufacturer = ['toyota', 'bmw', 'fca us llc','general motors', 'kia', 'nissan', 'honda', 'mercedes-benz', 'volkswagen']\n",
    "classs = ['COM','SUV','PU','MINIV']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<p>The code below is a loop that prints a scatter plot to represent each manufacturer (that was randomly selected due to their popularity).and the correlation that each manufacturer has between the annual fuel cost the the City (MPG)</p>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for brands in manufacturer:\n",
    "    fig = plt.figure()\n",
    "    plt.title(brands)\n",
    "    plt.xlabel('Annual Fuel Cost')\n",
    "    plt.ylabel('CITY (MPG)')\n",
    "    for x in yearlst:\n",
    "        curpd = pdfiles[x]\n",
    "        th = curpd[curpd['Manufacturer'].str.lower() == brands]\n",
    "        i = th['fcost']\n",
    "        j = th['cty']\n",
    "        plt.scatter(i,j, color = pin[int(x)-2001],label = x)\n",
    "        \n",
    "    plt.legend()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>The code below does the same as the code above, the only difference is that now we're seeing a correlation between annual fuel cost and the Highway (MPG)'</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for brands in manufacturer:\n",
    "    fig = plt.figure()\n",
    "    plt.title(brands)\n",
    "    plt.xlabel('Annual Fuel Cost')\n",
    "    plt.ylabel('HIGHWAY (MPG)')\n",
    "    for x in yearlst:\n",
    "        curpd = pdfiles[x]\n",
    "        th = curpd[curpd['Manufacturer'].str.lower() == brands]\n",
    "        i = th['fcost']\n",
    "        j = th['hwy']\n",
    "        plt.scatter(i,j, color = pin[int(x)-2001],label = x)\n",
    "        \n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " <p>The code below does the same as the code above, the only difference is that now we're seeing a correlation between annual fuel cost and the combined MPG of the Highway and the City'</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for brands in manufacturer:\n",
    "    fig = plt.figure()\n",
    "    plt.title(brands)\n",
    "    plt.xlabel('Annual Fuel Cost')\n",
    "    plt.ylabel('HIGHWAY (MPG)')\n",
    "    for x in yearlst:\n",
    "        curpd = pdfiles[x]\n",
    "        th = curpd[curpd['Manufacturer'].str.lower() == brands]\n",
    "        i = th['fcost']\n",
    "        j = th['cmb']\n",
    "        plt.scatter(i,j, color = pin[int(x)-2001],label = x)\n",
    "        \n",
    "    plt.legend()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>You are now going to use the same logic as before but now you'll be using the deriving data from the class of cars. We are testing things out and trying to find a relation between any of the data given to us. As you walk through this, even you yourself will see a pattern in which you'll start putting the data together to make a deduction of what the data is telling us.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlabel('Annual Fuel Cost')\n",
    "plt.ylabel('highway (MPG)')\n",
    "plt.title('Compact car analysis')\n",
    "for x in yearlst:\n",
    "    curpd = pdfiles[x]\n",
    "    s = set(curpd['Class'].values)\n",
    "    for y in s:\n",
    "        if y in carclass['COM']:\n",
    "            th = curpd[curpd['Class'] == y]\n",
    "            i = th['fcost']\n",
    "            j = th['hwy']\n",
    "            plt.scatter(i,j, color = pig[int(x)-2001])\n",
    "\n",
    "plt.legend()\n",
    "plt.show()    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlabel('Annual Fuel Cost')\n",
    "plt.ylabel('city (MPG)')\n",
    "plt.title('Compact car analysis')\n",
    "for x in yearlst:\n",
    "    curpd = pdfiles[x]\n",
    "    s = set(curpd['Class'].values)\n",
    "    for y in s:\n",
    "        if y in carclass['COM']:\n",
    "            th = curpd[curpd['Class'] == y]\n",
    "            i = th['fcost']\n",
    "            j = th['cty']\n",
    "            plt.scatter(i,j, color = piv[int(x)-2001],label = x)\n",
    "\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlabel('Annual Fuel Cost')\n",
    "plt.ylabel('combined (MPG)')\n",
    "plt.title('Compact car analysis')\n",
    "for x in yearlst:\n",
    "    curpd = pdfiles[x]\n",
    "    s = set(curpd['Class'].values)\n",
    "    for y in s:\n",
    "        if y in carclass['COM']:\n",
    "            th = curpd[curpd['Class'] == y]\n",
    "            i = th['fcost']\n",
    "            j = th['cmb']\n",
    "            plt.scatter(i,j, color = pir[int(x)-2001],label = x)\n",
    "\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<p>From the above plots, annual fuel cost has a clear relationship with \n",
    "mileage per gallon. Lower fuel cost corresponds to higher mpg, while the \n",
    "reverse happens for high annual fuel cost. Another observation is that \n",
    "annual fuel cost increases over the late 00s and then decreases over the years.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlabel('year')\n",
    "plt.ylabel('Annual fuel cost')\n",
    "plt.title('Annual Fuel Cost over the years')\n",
    "byear = []\n",
    "bcost = []\n",
    "for x in yearlst:\n",
    "    curpd = pdfiles[x]\n",
    "    byear.extend(curpd['year'])\n",
    "    bcost.extend(curpd['fcost'])\n",
    "    \n",
    "    plt.scatter(curpd['year'],curpd['fcost'])\n",
    "b = pd.DataFrame(byear,columns=['year'])\n",
    "b[\"cost\"] = bcost\n",
    "\n",
    "avg_lifeExp_per_interval = b.groupby('year').mean()\n",
    "avg_lifeExp_per_interval.cost.plot()\n",
    "plt.show()\n",
    "\n",
    "sns.violinplot(y=bcost, x=byear)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>The above graph and violin plot shows the change in annual fuel cost over the years across all car \n",
    " classes and car manufacturers. This may point to the fact that the annual fuel cost may have \n",
    " underlying factors besides the car maker and car types. One potential source could be the inflation of \n",
    " gas prices centered around 2009.</p>  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tested and Approved by: Chris Chan and Edward N Mancho"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}