2. Scraping a real site.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Gathering some reactor data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import modules to facilitate the scrape\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fetch HTML with `requests`\n",
    "\n",
    "`requests` is great at playing web browser. For more information, check out the [full documentation](http://docs.python-requests.org/en/master/).\n",
    "\n",
    "```python\n",
    "requests.get('some URL')\n",
    "# navigates to a site and sends you the response\n",
    "\n",
    "response.content\n",
    "# a way requests serves up the page's HTML source code\n",
    "```\n",
    "\n",
    "We are going to be getting data on nuclear reactors operating in the U.S.: http://www.nrc.gov/reactors/operating/list-power-reactor-units.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fetch the contents of webpage with requests\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parse HTML with `BeautifulSoup`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let BeautifulSoup parse the content of that page\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Target the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# snip out the table and pass it to a new variable\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print reactor_table to verify we have the right thing\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use .find_all to create a list of rows in the table\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# isolate the fourth row and print it\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of our table's rows, with a little shading and indentation:\n",
    "\n",
    "```html\n",
    "<tr valign=\"top\">\n",
    "    <td scope=\"row\"><a href=\"/info-finder/reactors/ano1.html\">Arkansas Nuclear 1</a><br/>05000313</td>\n",
    "    <td align=\"center\">DPR-51</td>\n",
    "    <td>PWR</td>\n",
    "    <td>6 miles WNW of Russellville,  AR</td>\n",
    "    <td>Entergy Nuclear Operations, Inc.</td>\n",
    "    <td align=\"middle\">4</td>\n",
    "</tr>```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use .find_all again to generate a list of the row's cells and return it\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "BeautifulSoup has a couple other methods we haven't discussed yet that are helpful for extracting the information _inside_ of tags:\n",
    "```python\n",
    "soup.contents\n",
    "# breaks up everything in a tag into a fresh list (useful when you have more than text in a cell)\n",
    "\n",
    "soup.get('some attribute')\n",
    "# returns the attribute (useful for getting URLs from <a> tags, for example)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# examine the \"contents\" of the first item in cells\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# isolate and print the name, the link and the docket number\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extract the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make an empty list to hold the data\n",
    "\n",
    "# a for loop is going to take us through every row in the table EXCEPT the header\n",
    "# combining two steps: the list it pulls from will be greated by a .find_all for 'tr' tags\n",
    "    \n",
    "    # .find_all 'td' tags in the row and put them into a variable\n",
    "    \n",
    "    # extract the cell contents\n",
    "    \n",
    "    # append the collected data to the empty list\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Write the data to CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# open a file and write our data to it\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### If you want only a simple HTML table on a page:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.read_html('https://en.wikipedia.org/wiki/List_of_United_States_cities_by_population', header=0)[3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.read_clipboard()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more on pandas and parsing tabular data, I recommend the CAR class and/or [this book](http://shop.oreilly.com/product/0636920023784.do)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}