web-scraping/Conceptual Introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Transforming the Web into Data (with Python)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Agenda\n",
    "\n",
    "- conceptual introduction to web scraping\n",
    "- tools for non-programmers\n",
    "- tools for python programmers\n",
    "- code tour\n",
    "- break\n",
    "- *scrape from scratch* exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# why scrape the web\n",
    "\n",
    "- there is a lot of human activity on the web, which produces\n",
    "- new and unique data/traces, that can lead to\n",
    "- insight & understanding for data science, the social sciences, and the humanities.\n",
    "- Ground Truthiness - *remember the web is only a particular representation of human behavior*\n",
    "- You can also scrape for [fun & profit](https://blog.hartleybrody.com/guide-to-web-scraping/) 💰"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# so what is scraping the web?\n",
    "\n",
    "Web scraping (web harvesting or web data extraction) is a computer software technique of **extracting information from websites**. Usually, such software programs **simulate human exploration** of the World Wide Web by either implementing low-level Hypertext Transfer Protocol (HTTP), or embedding a fully-fledged web browser, such as Internet Explorer or Mozilla Firefox. - [Wikipedia](http://en.wikipedia.org/wiki/Web_scraping)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# conceptual introduction to web scraping\n",
    "\n",
    "- there roughly three steps - *results may vary*\n",
    "    1. fetching resources - *asking a computer \"hey, can you send me `http://google.com`?\"*\n",
    "    2. parsing documents - *creating a **machine readable** representation of a web page*\n",
    "    3. extracting data - *pulling out just the information of interest*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![Scraping for fun & profit](scraping.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# fetching resources\n",
    "\n",
    "\n",
    "- [Hyper Text Transfer Protocol](http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol) (HTTP)\n",
    "    - fundamentally about **requests** & **responses**\n",
    "    - the language of the web\n",
    "    - four request methods: **GET, POST, PUT, DELETE**\n",
    "    - **URLs** point to **resources**\n",
    "- verbs & nouns\n",
    "    - request methods are the verbs\n",
    "    - resouces are the nouns\n",
    "    - URLs are the proper nouns\n",
    "- stateless\n",
    "    - doesn't have a good memory\n",
    "    - [sessions](http://en.wikipedia.org/wiki/Session_%28computer_science%29#HTTP_session_token) - *how HTTP servers remember \"state\"*\n",
    "    - [cookies](http://en.wikipedia.org/wiki/HTTP_cookie) - *the token passed in HTTP requests & responses*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# fetching resources\n",
    "\n",
    "- web pages - *made for humans*\n",
    "    - [HyperText Markup Language](http://en.wikipedia.org/wiki/HTML)(HTML) - *defines document structure*\n",
    "    - [Cascading Syle Sheets](http://en.wikipedia.org/wiki/Cascading_Style_Sheets)(CSS) - *makes web pages pretty*\n",
    "    - [JavaScript](http://en.wikipedia.org/wiki/JavaScript) - *makes web pages interactive*\n",
    "    - so many more standards... [W3C](http://www.w3.org/standards/)\n",
    "- APIs - *made for machines*\n",
    "    - [Application Programming Interface](http://en.wikipedia.org/wiki/Application_programming_interface) - *fancy name for how computer machines connect with each other*\n",
    "    - how to get data from the *social web* (i.e. Twitter, Facebook, etc.)\n",
    "    - related, but distinct from web scraping (more structured, access control)\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# parsing documents\n",
    "\n",
    "- HTML documents are composed of elements or **tags**\n",
    "    - the `<html>` tag is the **root** of the tree\n",
    "- the [HTML specification defines a bunch of tags](http://www.w3schools.com/tags/)\n",
    "    - `<p>this is a paragraph tag with text <em>inside</em> of it</p>`\n",
    "    - `<a href=\"http://pitt.edu\">This is an anchor tag, basically a link</a>`\n",
    "    - not enough time to review all of them\n",
    "- parsing transforms the barf into a tree of elements\n",
    "    - also called the the [Document Object Model](http://en.wikipedia.org/wiki/Document_Object_Model) or DOM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "```html\n",
    "<!DOCTYPE html>\n",
    "<html>\n",
    "  <head>\n",
    "    <title>A basic webpage</title>\n",
    "  <body>\n",
    "    <h1>My First Heading</h1>\n",
    "    <p>My first paragraph.</p>\n",
    "    <ul>\n",
    "      <li>First item in a unordered list</li>\n",
    "      <li>Second item in an unordered list</li>\n",
    "      <div class=\"stuff\">\n",
    "        <p>Another paragraph separated by a div element.</p>\n",
    "      </div>\n",
    "      <table>\n",
    "  </body>\n",
    "</html>\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE html>\r\n",
      "<!--[if IEMobile 7]><html class=\"iem7\"  lang=\"en\" dir=\"ltr\"><![endif]-->\r\n",
      "<!--[if lte IE 6]><html class=\"lt-ie9 lt-ie8 lt-ie7\"  lang=\"en\" dir=\"ltr\"><![endif]-->\r\n",
      "<!--[if (IE 7)&(!IEMobile)]><html class=\"lt-ie9 lt-ie8\"  lang=\"en\" dir=\"ltr\"><![endif]-->\r\n",
      "<!--[if IE 8]><html class=\"lt-ie9\"  lang=\"en\" dir=\"ltr\"><![endif]-->\r\n",
      "<!--[if (gte IE 9)|(gt IEMobile 7)]><!--><html  lang=\"en\" dir=\"ltr\" prefix=\"content: http://purl.org/rss/1.0/modules/content/ dc: http://purl.org/dc/terms/ foaf: http://xmlns.com/foaf/0.1/ og: http://ogp.me/ns# rdfs: http://www.w3.org/2000/01/rdf-schema# sioc: http://rdfs.org/sioc/ns# sioct: http://rdfs.org/sioc/types# skos: http://www.w3.org/2004/02/skos/core# xsd: http://www.w3.org/2001/XMLSchema#\"><!--<![endif]-->\r\n",
      "\r\n",
      "<head profile=\"http://www.w3.org/1999/xhtml/vocab\">\r\n",
      "  <meta charset=\"utf-8\" />\r\n",
      "<link rel=\"shortcut icon\" href=\"http://www.pitt.edu/sites/default/files/pitt_favicon_0.ico\" type=\"image/vnd.microsoft.icon\" />\r\n",
      "<link rel=\"shortlink\" href=\"/node/62\" />\r\n",
      "<link rel=\"canonical\" href=\"/home\" />\r\n",
      "<meta name=\"Generator\" content=\"Drupal 7 (http://drupal.org)\" />\r\n",
      "  <title>Home | University of Pittsburgh</title>\r\n",
      "  <meta name=\"description\" content=\"The University of Pittsburgh is among the nation's most distinguished comprehensive universities, with a wide variety of high-quality programs in both the arts and sciences and professional fields.\" />\r\n",
      "  <meta name=\"Keywords\" content=\"University, Pittsburgh, Pitt, College, Learning, Research, Students, Undergraduate, Graduate\" />\r\n",
      "    \r\n",
      "      <meta name=\"MobileOptimized\" content=\"width\">\r\n",
      "    <meta name=\"HandheldFriendly\" content=\"true\">\r\n",
      "    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\r\n",
      "    <meta http-equiv=\"cleartype\" content=\"on\">\r\n",
      "\t\r\n",
      "  <link type=\"text/css\" rel=\"stylesheet\" href=\"http://www.pitt.edu/sites/default/files/css/css_kShW4RPmRstZ3SpIC-ZvVGNFVAi0WEMuCnI0ZkYIaFw.css\" media=\"all\" />\r\n",
      "<link type=\"text/css\" rel=\"stylesheet\" href=\"http://www.pitt.edu/sites/default/files/css/css_vZ_wrMQ9Og-YPPxa1q4us3N7DsZMJa-14jShHgRoRNo.css\" media=\"screen\" />\r\n",
      "<link type=\"text/css\" rel=\"stylesheet\" href=\"http://www.pitt.edu/sites/default/files/css/css_4mmZo2I5oU53mjQh0UjgygKazedTCqZXNvrxFyYrT-g.css\" media=\"all\" />\r\n",
      "<link type=\"text/css\" rel=\"stylesheet\" href=\"http://www.pitt.edu/sites/default/files/css/css_hME6weH8liYUm6qr-IDiSXVwXgjKndoDaEQ2Jq3-W10.css\" media=\"all\" />\r\n",
      "<link type=\"text/css\" rel=\"stylesheet\" href=\"http://www.pitt.edu/sites/default/files/css/css_LwvUaww9zeUDxZ1r2K4dHcSbAEEzbSNA-5Zz2KIgwD4.css\" media=\"all\" />\r\n",
      "   \r\n",
      "  <script src=\"http://www.pitt.edu/sites/all/modules/jquery_update/replace/jquery/1.5/jquery.js?v=1.5.2\"></script>\r\n",
      "<script src=\"http://www.pitt.edu/misc/jquery.once.js?v=1.2\"></script>\r\n"
     ]
    }
   ],
   "source": [
    "! curl -s http://pitt.edu | head -n30"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# extracting data\n",
    "\n",
    "- ok, now we are going to get *really* technical\n",
    "- pull information out of the tree and *push it somewhere else*\n",
    "\n",
    "![push somewhere else](http://i3.kym-cdn.com/entries/icons/original/000/003/356/test.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# extracting data\n",
    "\n",
    "- how?\n",
    "    - copy & paste\n",
    "    - automated scripts\n",
    "- if you have a lot of data, copy & paste probably won't work for you\n",
    "- if the data are on multiple pages, you will need to **crawl** with a **spider**\n",
    "- **web crawlers** extract the links from a web page, fetch those pages, extract links, fetch, extract, fetch...\n",
    "- scripts and tools help automate this process"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# extracting data\n",
    "\n",
    "- first step: *where are the data* in the HTML tree?\n",
    "- right-click & select \"inspect element\" - *works in Firefox, Chrome, & if developer mode enabled in Safari*\n",
    "    - THE MATRIX \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# extracting data\n",
    "\n",
    "- *selection* is the key\n",
    "- many different ways to select HTML tags\n",
    "    - library APIs - *python code for navigating & searching parsed HTML documents*\n",
    "    - [CSS selectors](http://www.w3schools.com/cssref/css_selectors.asp) - *used by Cascading Style Sheets, good to know* \n",
    "    - [XPATH](http://en.wikipedia.org/wiki/XPath) - *a query language for XML documents, works for HTML too!*\n",
    "    - REGEX? - [*DON'T EVER USE REGEX TO PARSE HTML!*](http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# basic workflow\n",
    "\n",
    "1. fetch pages\n",
    "2. extract data\n",
    "3. extract links\n",
    "4. fetch more pages\n",
    "5. ...\n",
    "6. profit?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# DATA CLEANING!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# challenges in web scraping\n",
    "\n",
    "- logins, paywalls, and access control\n",
    "    - these are not impossible, tools support HTTP sessions & cookies\n",
    "    - throttling - *fine line between scraping & denial of service attack*\n",
    "    - THE LAW - *read the terms of service, copyright? FAIR USE!* \n",
    "- dynamic websites\n",
    "    - javascript - *hard to scrape because the DOM changes*\n",
    "    - AJAX or XMLHttpRequest - *pages can asyncronously fetch data & update themselves*\n",
    "- the document vs. application centric web\n",
    "    - scraping gmail?\n",
    "    - APIs help, if they exist\n",
    "- mobile web / apps????\n",
    "    -  ¯\\\\_(ツ)_/¯"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# tools for non-programmers\n",
    "\n",
    "- [Import.io](http://import.io) - *A commercial product, not sure how expensive*\n",
    "- [Wget](http://en.wikipedia.org/wiki/Wget) - *Swiss army knife of web scraping tools, command line*\n",
    "- [HTTrack](http://www.httrack.com/) - *windows tool for copying websites, GUI*\n",
    "- [ScraperWiki](https://scraperwiki.com/) - *a service that costs money, if you have a grant...*\n",
    "- [Scraper Plugin](https://chrome.google.com/webstore/detail/scraper/mbigbapnjcgaffohmbkdlecaccepngjd) – *chrome plugin instad of a service, looks pretty easy to use*\n",
    "- [Diffbot](http://diffbot.com/) – *more advanced extraction, really nice guys, costs money but they support research if you ask them*\n",
    "- **EMAIL** – *sometimes it doesn't hurt to ask!*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# tools for python programmers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## fetching\n",
    "\n",
    "- [`urllib2`](https://docs.python.org/2/library/urllib2.html) - *batteries included*\n",
    "- [Requests: HTTP for Humans](http://docs.python-requests.org/en/latest/) - *much nicer than urllib2*\n",
    "- [Scrapy](http://scrapy.org/) - *a complete framework for web crawling and data extraction*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## parsing & extracting\n",
    "\n",
    "- [Beautiful Soup](http://www.crummy.com/software/BeautifulSoup/) - *the most popular library*\n",
    "- [Soupy](http://soupy.readthedocs.org/en/stable/) - *a wrapper around BS to make life easier*\n",
    "- [Scrapely](https://github.com/scrapy/scrapely) - *another tool for extracting structured data from web pages*\n",
    "- [lxml](http://lxml.de/) - *a bit lower level, supports XPATH, which I prefer*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## data management\n",
    "\n",
    "- [Pandas Data Analysis Library](http://pandas.pydata.org/) - *think R dataframes*\n",
    "- [`sqlite`](https://docs.python.org/2/library/sqlite3.html) - *nice lightweight relational database*\n",
    "- [`json`](https://docs.python.org/2/library/json.html) or [`csv`](https://docs.python.org/2/library/csv.html?highlight=csv) - *serializing data to a file*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Where to go next?\n",
    "\n",
    "\n",
    "## Learning Python\n",
    "\n",
    "- [Python for Informatics](http://pythonlearn.com/book.php) - *A book as well as [online course materials](http://pythonlearn.com/) and even a [MOOC on Coursera](https://www.coursera.org/course/pythonlearn). Everything is FREE and OPEN SOURCE.*\n",
    "- [Learn Python the Hard Way](http://learnpythonthehardway.org/) - *Another book & companion website that is NOT FREE.*\n",
    "- [Codecademy Python Track](http://www.codecademy.com/tracks/python) - *A set of online interactive tutorials for learning python.*\n",
    "- [Google](http://google.com) - *Seriously, there are a million resources online for leaning Python. Try a few of them out and see which ones work best for you.*\n",
    "\n",
    "\n",
    "\n",
    "## Web Scraping\n",
    "\n",
    "- [The Ultimate Guide to Web Scraping](https://blog.hartleybrody.com/guide-to-web-scraping/) - *A short book that provides a conceptual introduction to web scraping*\n",
    "    - [\"I Don't Need No Stinking API\"](https://blog.hartleybrody.com/web-scraping/) - *A popular blog post written by the author of the aformentioned book.*\n",
    "- [Mining the Social Web, 2nd Edition](http://shop.oreilly.com/product/0636920030195.do) - *An excellent book for more advanced programmers who are interested in collecting and analyzing data from social websites like Twitter, Facebook, and Github.*\n",
    "- [Web Scraping with Python](http://shop.oreilly.com/product/0636920034391.do) - *A new book coming summer of 2015 that appears to cover the more technical aspects of scraping the web with python.*\n",
    "- [Google](http://google.com) - *Again, seriously, there are a million tutorials on the web. Some are more technical than others.*"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}