notes/openapi.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a019db0d-8ae6-4355-ae4f-f09ce534a880",
   "metadata": {},
   "source": [
    "# Pulling data from (open) REST APIs\n",
    "\n",
    "[Big source of public APIs](https://rapidapi.com/collection/list-of-free-apis)\n",
    "\n",
    "We have already seen how to use `requests` to fetch a webpage:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6038ee45-f4b9-410f-b534-23db4726108e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<!DOCTYPE html><html class=\"no-js\"><head><meta content=\"IE=edge,chrome=1\" http-equiv=\"X-UA-Compatible\"><meta charset=\"utf-8\"><meta content=\"text/html\" http-equiv=\"Content-Type\"><meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0, minimum-scale=1.0\"><link rel=\"dns-prefetch\" href=\"/op'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import requests\n",
    "r = requests.get('https://www.cnn.com')\n",
    "r.text[0:300]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37092277-5f3e-4441-9dc3-620d97758ee9",
   "metadata": {},
   "source": [
    "If the URL is to a page that gives you HTML, we would say that we are fetching a webpage. On the other hand, if the URL is returning data in some form, we would say that we are accessing a *REST* api.\n",
    " \n",
    "**REST** is an acronym for *REpresentational State Transfer* and is a very handy way to make something trivial sound very complicated.  Anytime you see the word REST, just think \"webpage that gives me data not HTML.\" There is a massive industry and giant following behind this term but I don't see anything beyond \"fetch data from webpage\".\n",
    "\n",
    "Anyway, we are going to pull data from web servers that intentionally provide nice data spigot URLs. Information you need in order to get data is typically:\n",
    "\n",
    "* Base URL, including machine name, port number, and \"file\" path\n",
    "* The names and values of parameters\n",
    "* What data comes back and in what format (XML, JSON, CSV, ...)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61e4eeef-9fde-47ec-b6e0-cd2fc811f565",
   "metadata": {},
   "source": [
    "## Looking up word definitions\n",
    "\n",
    "The [dictionaryapi.dev](https://dictionaryapi.dev/) API lets us look up words in various languages and get the definitions. The format of the URL to access the API is just:\n",
    "\n",
    "```\n",
    "https://api.dictionaryapi.dev/api/v2/entries/<language_code>/<word>\n",
    "```\n",
    "\n",
    "So, we can get the English definition for *science* like this (and parse the json result):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "88d62125-f57e-4ba4-ac9f-39fa45eeb67e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[{\"word\":\"science\",\"phonetic\":\"ˈsʌɪəns\",\"phonetics\":[{\"text\":\"ˈsʌɪəns\",\"audio\":\"//ssl.gstatic.com/dictionary/static/sounds/20200429/science--_gb_1.mp3\"}],\"origin\":\"Middle English (denoting knowledge):'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import requests\n",
    "import json\n",
    "\n",
    "r = requests.get('https://api.dictionaryapi.dev/api/v2/entries/en_US/science')\n",
    "r.text[0:200]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "c3f5128d-8386-40e0-922d-1dcde203b886",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "science ˈsʌɪəns\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'word': 'science',\n",
       "  'phonetic': 'ˈsʌɪəns',\n",
       "  'phonetics': [{'text': 'ˈsʌɪəns',\n",
       "    'audio': '//ssl.gstatic.com/dictionary/static/sounds/20200429/science--_gb_1.mp3'}],\n",
       "  'origin': 'Middle English (denoting knowledge): from Old French, from Latin scientia, from scire ‘know’.',\n",
       "  'meanings': [{'partOfSpeech': 'noun',\n",
       "    'definitions': [{'definition': 'the intellectual and practical activity encompassing the systematic study of the structure and behaviour of the physical and natural world through observation and experiment.',\n",
       "      'example': 'the world of science and technology',\n",
       "      'synonyms': ['branch of knowledge',\n",
       "       'area of study',\n",
       "       'discipline',\n",
       "       'field'],\n",
       "      'antonyms': []}]}]}]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = json.loads(r.text)\n",
    "print(data[0]['word'], data[0]['phonetic'])\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6dce09a-bc3d-4193-992d-86c38f455701",
   "metadata": {},
   "source": [
    "The JSON looks like the following when formatted in the browser (I think I have a JSON viewer plug-in).\n",
    "\n",
    "<img src=\"figures/dictionary-science-json.png\" width=\"400\">\n",
    "\n",
    "That looks like there is a list with one element, which is the actual dictionary of stuff we want so `data[0]` is the dictionary of stuff. This lets us get access to the definition and phonetics if we dig down."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "61d2e2bc-ab3b-48d0-87df-190004df0d94",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "e6d6ae49-68d9-4cb7-95b4-226b4d9c4a35",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ˈsʌɪəns\n",
      "the intellectual and practical activity encompassing the systematic study of the structure and behaviour of the physical and natural world through observation and experiment.\n"
     ]
    }
   ],
   "source": [
    "phonetic = data['phonetic']\n",
    "sciencedef = data['meanings'][0]['definitions'][0]['definition']\n",
    "print(phonetic)\n",
    "print(sciencedef)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc2a0f2a-d1fe-493b-85fe-0311d2bb4b20",
   "metadata": {},
   "source": [
    "**Exercise**:  Print out the origin of the word science from that JSON."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2dac8c14-3f73-4ddc-bfb8-dc9985585c4c",
   "metadata": {},
   "source": [
    "**Exercise**: Use the API to fetch and print out the definition of *Merhaba* (a greeting) in the Turkish language. The result should be *karşılaşıldığında söylenilen bir selamlaşma sözü.* (*a word of greeting when encountered.*)  The Turkish language code is `tr`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16acb0bd-85df-489d-84e5-1a9ae6f6775a",
   "metadata": {},
   "source": [
    "## JSON from openpayments.us\n",
    "\n",
    "(This site seems to go down a lot when they reboot our computer science machine so forgive me if it's not up...)\n",
    "\n",
    "Now, let's look at a website that will give us JSON data: [www.openpayments.us](http://www.openpayments.us).\n",
    " \n",
    "There is a REST data API available at URL template:\n",
    "\n",
    "```\n",
    "URL = f\"http://openpayments.us/data?query={q}\" # for some q\n",
    "```\n",
    "**Exercise**: Use `curl` to fetch data about a doctor.\n",
    "\n",
    "Here's how to fetch the data for a doctor's name, such as `John Chan`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "54a6b2c1-e999-4dda-9d06-7ef4dc0827b8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"query\": \"John AND Chan\", \"lucenequery\": \"+_composite:john +_composite:chan\", \"hits\": 10, \"page\": 1, \"pagesize\": 50, \"sortby\": null, \"reverse\": false, \"cols\": [\"Payment ID\", \"Vendor\", \"Date\", \"Amount\", \"Dr Name\", \"Nature of Payment\", \"Payment Type\", \"Dr Addr\", \"Dr Type\", \"Dr License\", \"Hospital\", \"Re Drug\", \"Re: Device\", \"Travel\", \"# Payments\", \"Submitter\", \"Year\", \"Publ Date\", \"Dr or Hosp\", \"Prod. Indicator\", \"Disputed?\", \"Dr Ownership\", \"3rd Party Recipient\", \"3rd Party\", \"Charity?\", \"3rd Party is Recipient\", \"Context\", \"Publ Delay\"], \"numresults\": 10, \"results\": [{\"Payment ID\": \"42285\", \"Vendor\": \"Shire US Holdings PA\", \"Date\": \"2013/08/26\", \"Amount\": \"124.99\", \"Dr Name\": \"JOHN CHAN\", \"Nature of Payment\": \"Food and Beverage\", \"Payment Type\": \"In-kind items and services\", \"Dr Addr\": \"8700 BEVERLY BLVD RM 5512 LOS ANGELES CA 90048 United States\", \"Dr Type\": \"DPM Internal Medicine\", \"Dr License\": \"CA\", \"Re: Device\": \"DERMAGRAFT\", \"# Payments\": \"1\", \"Submitter\": \"Shire US Holdings\", \"Y\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import json\n",
    "import sys\n",
    "\n",
    "name = \"John Chan\"\n",
    "URL = f\"http://openpayments.us/data?query={name}\"\n",
    "    \n",
    "r = requests.get(URL)\n",
    "data = json.loads(r.text)\n",
    "\n",
    "print(json.dumps(data)[0:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "997ed205-ba2f-4799-8710-481a0b37957a",
   "metadata": {},
   "source": [
    "This website gives you JSON, which is very easy to load and dump using the default `json` package as you can see from that code snippet. As before, you can grab one of the elements using dictionary like indexing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "b4eccf1b-ce73-4748-8ce0-8d6b2019ed7a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'Payment ID': '42285',\n",
       "  'Vendor': 'Shire US Holdings PA',\n",
       "  'Date': '2013/08/26',\n",
       "  'Amount': '124.99',\n",
       "  'Dr Name': 'JOHN CHAN',\n",
       "  'Nature of Payment': 'Food and Beverage',\n",
       "  'Payment Type': 'In-kind items and services',\n",
       "  'Dr Addr': '8700 BEVERLY BLVD RM 5512 LOS ANGELES CA 90048 United States',\n",
       "  'Dr Type': 'DPM Internal Medicine',\n",
       "  'Dr License': 'CA',\n",
       "  'Re: Device': 'DERMAGRAFT',\n",
       "  '# Payments': '1',\n",
       "  'Submitter': 'Shire US Holdings',\n",
       "  'Year': '2013',\n",
       "  'Publ Date': '2014/09/29',\n",
       "  'Dr or Hosp': 'Dr',\n",
       "  'Prod. Indicator': 'Covered',\n",
       "  'Disputed?': 'No',\n",
       "  'Dr Ownership': 'No',\n",
       "  '3rd Party Recipient': 'No',\n",
       "  'Publ Delay': 'No'},\n",
       " {'Payment ID': '683301',\n",
       "  'Vendor': 'AstraZeneca Pharmaceuticals LP DE',\n",
       "  'Date': '2013/09/10',\n",
       "  'Amount': '12.1',\n",
       "  'Dr Name': 'John Chan',\n",
       "  'Nature of Payment': 'Food and Beverage',\n",
       "  'Payment Type': 'In-kind items and services',\n",
       "  'Dr Addr': '2111 Geer Rd Suite 500 Turlock CA 95382-2458 United States',\n",
       "  'Dr Type': 'DDS Dental Providers/ Dentist',\n",
       "  'Dr License': 'CA',\n",
       "  '# Payments': '1',\n",
       "  'Submitter': 'AstraZeneca Pharmaceuticals LP',\n",
       "  'Year': '2013',\n",
       "  'Publ Date': '2014/09/29',\n",
       "  'Dr or Hosp': 'Dr',\n",
       "  'Prod. Indicator': 'Covered',\n",
       "  'Disputed?': 'No',\n",
       "  'Dr Ownership': 'No',\n",
       "  '3rd Party Recipient': 'No',\n",
       "  'Charity?': 'No',\n",
       "  'Context': 'Informational Meal',\n",
       "  'Publ Delay': 'No'}]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = data['results']\n",
    "results[0:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8511bf6-473e-4b8d-96ff-061cece2f0a9",
   "metadata": {},
   "source": [
    "It is convenient to look at the records in a data frame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "95401b96-98c8-4553-911b-b31e86b1271c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Payment ID</th>\n",
       "      <th>Vendor</th>\n",
       "      <th>Date</th>\n",
       "      <th>Amount</th>\n",
       "      <th>Dr Name</th>\n",
       "      <th>Nature of Payment</th>\n",
       "      <th>Payment Type</th>\n",
       "      <th>Dr Addr</th>\n",
       "      <th>Dr Type</th>\n",
       "      <th>Dr License</th>\n",
       "      <th>Re: Device</th>\n",
       "      <th># Payments</th>\n",
       "      <th>Submitter</th>\n",
       "      <th>Year</th>\n",
       "      <th>Publ Date</th>\n",
       "      <th>Dr or Hosp</th>\n",
       "      <th>Prod. Indicator</th>\n",
       "      <th>Disputed?</th>\n",
       "      <th>Dr Ownership</th>\n",
       "      <th>3rd Party Recipient</th>\n",
       "      <th>Publ Delay</th>\n",
       "      <th>Charity?</th>\n",
       "      <th>Context</th>\n",
       "      <th>3rd Party</th>\n",
       "      <th>3rd Party is Recipient</th>\n",
       "      <th>Travel</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>42285</td>\n",
       "      <td>Shire US Holdings PA</td>\n",
       "      <td>2013/08/26</td>\n",
       "      <td>124.99</td>\n",
       "      <td>JOHN CHAN</td>\n",
       "      <td>Food and Beverage</td>\n",
       "      <td>In-kind items and services</td>\n",
       "      <td>8700 BEVERLY BLVD RM 5512 LOS ANGELES CA 90048...</td>\n",
       "      <td>DPM Internal Medicine</td>\n",
       "      <td>CA</td>\n",
       "      <td>DERMAGRAFT</td>\n",
       "      <td>1</td>\n",
       "      <td>Shire US Holdings</td>\n",
       "      <td>2013</td>\n",
       "      <td>2014/09/29</td>\n",
       "      <td>Dr</td>\n",
       "      <td>Covered</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>683301</td>\n",
       "      <td>AstraZeneca Pharmaceuticals LP DE</td>\n",
       "      <td>2013/09/10</td>\n",
       "      <td>12.1</td>\n",
       "      <td>John Chan</td>\n",
       "      <td>Food and Beverage</td>\n",
       "      <td>In-kind items and services</td>\n",
       "      <td>2111 Geer Rd Suite 500 Turlock CA 95382-2458 U...</td>\n",
       "      <td>DDS Dental Providers/ Dentist</td>\n",
       "      <td>CA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>AstraZeneca Pharmaceuticals LP</td>\n",
       "      <td>2013</td>\n",
       "      <td>2014/09/29</td>\n",
       "      <td>Dr</td>\n",
       "      <td>Covered</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Informational Meal</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5406946</td>\n",
       "      <td>Otsuka America Pharmaceutical, Inc. MD</td>\n",
       "      <td>2013/11/12</td>\n",
       "      <td>23.04</td>\n",
       "      <td>JOHN CHAN</td>\n",
       "      <td>Food and Beverage</td>\n",
       "      <td>In-kind items and services</td>\n",
       "      <td>8700 BEVERLY BLVD RM 5512 LOS ANGELES CA 90048...</td>\n",
       "      <td>MD Internal Medicine</td>\n",
       "      <td>CA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>Otsuka America Pharmaceutical, Inc.</td>\n",
       "      <td>2013</td>\n",
       "      <td>2014/09/29</td>\n",
       "      <td>Dr</td>\n",
       "      <td>Covered</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Payment ID  ... Travel\n",
       "0      42285  ...    NaN\n",
       "1     683301  ...    NaN\n",
       "2    5406946  ...    NaN\n",
       "\n",
       "[3 rows x 26 columns]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "pd.DataFrame.from_dict(results).head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c9f79d1-7913-4470-8411-2dee327bb4e5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "a8b0230b-170a-4341-aaf7-94fe49e27606",
   "metadata": {},
   "source": [
    "A **technical detail** related to valid strings you can include as part of a URL.  Spaces are not allowed so `John Chan` has to be encoded or \"quoted\".  Fortunately, `requests` does this automatically for us. If you ever need to quote parameter values in URLs, you can do this:\n",
    "\n",
    "```python\n",
    "from urllib.parse import quote\n",
    "value = quote(value)\n",
    "```\n",
    "\n",
    "Because `&` is the separator between parameters, it is also invalid in a parameter name or value. Here are some example conversions:\n",
    "\n",
    "```python\n",
    ">>> quote(\"john chan\")\n",
    "'john%20chan'\n",
    ">>> quote(\"john&chan\")\n",
    "'john%26chan'\n",
    "```\n",
    "\n",
    "The conversion uses the ASCII character code (in 2-digit hexadecimal) for space and ampersand. Sometimes you will see the space converted to a `+`, which also works: `John+Chan`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}