notebooks/021_URLs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img alt='UCL' src=\"images/ucl_logo.png\" align='center'>\n",
    "\n",
    "\n",
    "[<img src=\"images/noun_post_2109127.svg\" width=\"50\" align='right'>](022_Pandas.ipynb)\n",
    "\n",
    "[<img src=\"images/noun_pre_2109128.svg\" width=\"50\" align='right'>](020_Python_files.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 021 URLs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Introduction\n",
    "\n",
    "\n",
    "\n",
    "### Purpose\n",
    "\n",
    "In this session, we will learn about files and similar resources. We will introduce the standard Python library [`pathlib`](https://docs.python.org/3/library/pathlib.html) which is how we deal with file paths. For URLs, we will use the Python packages: [urlpath](https://github.com/chrono-meter/urlpath).\n",
    "\n",
    "### Prerequisites\n",
    "\n",
    "You will need some understanding of the following:\n",
    "\n",
    "\n",
    "* [001 Using Notebooks](001_Notebook_use.ipynb)\n",
    "* [002 Unix](002_Unix.ipynb) with a good familiarity with the UNIX commands we have been through.\n",
    "* [003 Getting help](003_Help.ipynb)\n",
    "* [004_Accounts](004_Accounts.ipynb)\n",
    "* [005_Packages](005_Packages.ipynb)\n",
    "* [010 Variables, comments and print()](010_Python_Introduction.ipynb)\n",
    "* [011 Data types](011_Python_data_types.ipynb) \n",
    "* [012 String formatting](012_Python_strings.ipynb)\n",
    "* [013_Python_string_methods](013_Python_string_methods.ipynb)\n",
    "\n",
    "You will need a detailed understanding and familiarity with the `Path` package for dealing with files, as well asd the underlying concepts covered there.\n",
    "\n",
    "* [020_Python_files](020_Python_files.ipynb)\n",
    "\n",
    "### Test\n",
    "\n",
    "In any public information (like these notebooks) we do not want to expose sensitive information such as usernames and passwords.\n",
    "\n",
    "So here we will make use of stored passwords and usernames using the local [cylog](geog0111/cylog.py) package that was covered in [004_Accounts](004_Accounts.ipynb). \n",
    "\n",
    "We will be using the site [`https://e4ftl01.cr.usgs.gov`](https://e4ftl01.cr.usgs.gov) today. You should have already tested that your NASA Earthdata login works for files on that site. \n",
    "\n",
    "If you unsure about your login and/or password, test them on [the Earthdata login page](https://urs.earthdata.nasa.gov/home).\n",
    "\n",
    "We should see that a call to `Cylog(url).login()` returns the username and password.\n",
    "\n",
    "So, if you like, you can check what you have stored in [cylog](geog0111/cylog.py) by using:\n",
    "\n",
    "    url='https://e4ftl01.cr.usgs.gov'\n",
    "    print(Cylog(url).login())\n",
    "    \n",
    "If you want to check it, run those commands in the cell below (uncomment the print line)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from geog0111.cylog import Cylog\n",
    "url='https://e4ftl01.cr.usgs.gov'\n",
    "#print(Cylog(url).login())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are prompted for a login and password, this means that you haven't previously entered one for this site (just enter them now, or go back and look at [004_Accounts](004_Accounts.ipynb). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resources from a URL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `urlpath`\n",
    "\n",
    "The library [`urlpath`](https://github.com/chrono-meter/urlpath) is designed to operate in a similar manner to `pathlib` for reading data from URLs. It is based on [urllib.parse](https://docs.python.org/3/library/urllib.parse.html) and [requests](https://requests.readthedocs.io/en/master/). It doesn't quite have all of the functionality we will need, but it goes a long way to an object-oriented URL library that is similar to Pathlib. \n",
    "\n",
    "The object in `urlpath` corresponding to `Path` is `URL`.\n",
    "\n",
    "A text file example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "remote file https://www.metoffice.gov.uk/hadobs/hadukp/data/monthly/HadSEEP_monthly_totals.txt\n"
     ]
    }
   ],
   "source": [
    "from urlpath import URL\n",
    "site = 'https://www.metoffice.gov.uk/'\n",
    "site_dir = 'hadobs/hadukp/data/monthly'\n",
    "site_file = 'HadSEEP_monthly_totals.txt'\n",
    " \n",
    "url = URL(site,site_dir,site_file)\n",
    "print(f'remote file {url}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A binary file example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "remote file https://e4ftl01.cr.usgs.gov/MOTA/MCD15A3H.006/2020.01.01/MCD15A3H.A2020001.h08v06.006.2020006032951.hdf\n"
     ]
    }
   ],
   "source": [
    "from urlpath import URL\n",
    "\n",
    "site = 'https://e4ftl01.cr.usgs.gov'\n",
    "site_dir = 'MOTA/MCD15A3H.006/2020.01.01'\n",
    "site_file = 'MCD15A3H.A2020001.h08v06.006.2020006032951.hdf'\n",
    "\n",
    "url = URL(site,site_dir,site_file)\n",
    "print(f'remote file {url}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have similar functionality in `URL` to `Path` for manipulating filenames, but more limited file information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "URL    : https://e4ftl01.cr.usgs.gov/MOTA/MCD15A3H.006/2020.01.01/MCD15A3H.A2020001.h08v06.006.2020006032951.hdf\n",
      "name   : MCD15A3H.A2020001.h08v06.006.2020006032951.hdf\n",
      "parent : https://e4ftl01.cr.usgs.gov/MOTA/MCD15A3H.006/2020.01.01\n"
     ]
    }
   ],
   "source": [
    "print(f'URL    : {url}')\n",
    "print(f'name   : {url.name}')\n",
    "print(f'parent : {url.parent}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "but also some other helpful ones on the URL:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "anchor   : https://e4ftl01.cr.usgs.gov\n",
      "hostname : e4ftl01.cr.usgs.gov\n",
      "scheme   : https\n",
      "name     : MCD15A3H.A2020001.h08v06.006.2020006032951.hdf\n",
      "netloc   : e4ftl01.cr.usgs.gov\n",
      "parts    : ('https://e4ftl01.cr.usgs.gov', 'MOTA', 'MCD15A3H.006', '2020.01.01')\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "print(f'anchor   : {url.anchor}')\n",
    "print(f'hostname : {url.hostinfo}')\n",
    "print(f'scheme   : {url.scheme}')\n",
    "print(f'name     : {url.name}')\n",
    "print(f'netloc   : {url.netloc}')\n",
    "print(f'parts    : {url.parent.parts}') # parent parts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "#### Exercise 1\n",
    "\n",
    "* create a `URL` object for the file `table.html` in the directory `psd/enso/mei/` on the site `http://www.esrl.noaa.gov/`.\n",
    "* print out the url name and check it is `table.html`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://www.esrl.noaa.gov/psd/enso/mei/table.html\n",
      "passed\n"
     ]
    }
   ],
   "source": [
    "# ANSWER\n",
    "\n",
    "# create a URL object for the file table.html \n",
    "# in the directory psd/enso/mei/ on the site \n",
    "# http://www.esrl.noaa.gov/.\n",
    "\n",
    "site = 'http://www.esrl.noaa.gov/'\n",
    "site_dir = 'psd/enso/mei'\n",
    "site_file = 'table.html'\n",
    "url = URL(site,site_dir,site_file)\n",
    "\n",
    "# print out the url and check it is table.html\n",
    "print(url)\n",
    "assert url.name == site_file\n",
    "print('passed')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For accessing URLs, will mostly make use of the following functions in `URL` that you will see are very similar to those for `Path`:\n",
    "\n",
    "\n",
    "|function| purpose|\n",
    "|---|---|\n",
    "|`URL.name`|  filename |\n",
    "|`URL.parent`|  parent |\n",
    "|`URL.parts`|  parts |\n",
    "| `URL.as_posix()` | return URL as posix string |\n",
    "|`URL.with_userinfo()`|  add in username and password |\n",
    "|`URL.get()`|  URL get. Returns `requests.models.Response`|\n",
    "| `URL.netloc` | network location e.g. `www.google.com`|\n",
    "| `URL.path` | full pathname on server (including filename)|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `URL.get()`\n",
    "\n",
    "We can use `URL.get()` to access data. This returns a response (more formally, a type [`requests.models.Response`](https://docs.python-requests.org/en/latest/)):\n",
    "\n",
    "    r = URL.get()\n",
    "\n",
    "`requests.models.Response` information table:\n",
    "\n",
    "|function| purpose|\n",
    "|---|---|\n",
    "| `r.status_code` | return code after request. You need to check this in case the call fails. It should be `200` if ok.|\n",
    "| `r.text` | text content returned |\n",
    "| `r.json` | text content interpreted as json |\n",
    "| `r.content` | binary content returned |\n",
    "\n",
    "If the access has been successful, then `r.status_code` should return `200` (or `requests.codes.ok`). If it is `401` then there has been an access error. You can find the fuller list of possible [http status codes](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status) on the web. \n",
    "\n",
    "If the data type to be accessed is raw (ASCII) text, then you can access the data with `r.text`. [Similarly](https://docs.python-requests.org/en/latest/), if it is [json](https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Objects/JSON), `r.json`, or binary `r.content`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data access good for https://covid.ourworldindata.org/data/archived/ecdc/locations.csv\n",
      "\n",
      "==== data as text ====\n",
      "countriesAndTerritories,location,continent,population_year,population\n",
      "Afghanistan,Afghanistan,Asia,2020,38928341\n",
      "Albania,Albania,Europe,2020,2877800\n",
      "Algeria,Algeria,Africa,2020,43851043\n",
      "Andorra,Andorra,Europe,2020,77265\n",
      "Angola,Angola,Africa,2020,32866268\n",
      "Anguilla,Anguilla,North America,2020,15002\n",
      "Antigua_and_Barbuda,Antigua and Barbuda,North America,2020,97928\n",
      "Argentina,Argentina,South America,2020,45195777\n",
      "Armenia,Armenia,Asia,2020,2963234\n",
      "Aruba,Aruba,North America,2020,106766\n",
      "Australia,Austral\n",
      "\n",
      "==== data as binary ====\n",
      "b'countriesAndTerritories,location,continent,population_year,population\\nAfghanistan,Afghanistan,Asia,2020,38928341\\nAlbania,Albania,Europe,2020,2877800\\nAlgeria,Algeria,Africa,2020,43851043\\nAndorra,Andorra,Europe,2020,77265\\nAngola,Angola,Africa,2020,32866268\\nAnguilla,Anguilla,North America,2020,15002\\nAntigua_and_Barbuda,Antigua and Barbuda,North America,2020,97928\\nArgentina,Argentina,South America,2020,45195777\\nArmenia,Armenia,Asia,2020,2963234\\nAruba,Aruba,North America,2020,106766\\nAustralia,Austral'\n"
     ]
    }
   ],
   "source": [
    "from urlpath import URL\n",
    "\n",
    "site = 'https://covid.ourworldindata.org'\n",
    "site_dir = 'data/archived/ecdc'\n",
    "site_file = 'full_data.csv'\n",
    "site_file = 'locations.csv'\n",
    "\n",
    "url = URL(site,site_dir,site_file)\n",
    "r = url.get()\n",
    "\n",
    "# check error code: 200 is good\n",
    "# \n",
    "if r.status_code == 200:\n",
    "    print(f'data access good for {url.as_posix()}')\n",
    "    \n",
    "    # get data as text\n",
    "    data = r.text\n",
    "    # print first 500 characters\n",
    "    print('\\n==== data as text ====')\n",
    "    print(data[:500])\n",
    "    \n",
    "    # get data as binary\n",
    "    data = r.content\n",
    "    # print first 500 characters\n",
    "    print('\\n==== data as binary ====')\n",
    "    print(data[:500])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "#### Exercise 1\n",
    "\n",
    "* Use the `URL.get()` method to pull data from `https://covid.ourworldindata.org/data/ecdc/archived/full_data.csv` and store in a file called `work/full_data.csv`.\n",
    "* check the file size\n",
    "* show the first few lines of data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "work/full_data.csv size: 3215657 bytes\n",
      "0: date,location,new_cases,new_deaths,total_cases,total_deaths,weekly_cases,weekly_deaths,biweekly_cases,biweekly_deaths\n",
      "1: 2019-12-31,Afghanistan,0,0,,,,,,\n",
      "2: 2020-01-01,Afghanistan,0,0,,,,,,\n",
      "3: 2020-01-02,Afghanistan,0,0,,,,,,\n",
      "4: 2020-01-03,Afghanistan,0,0,,,,,,\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "from urlpath import URL\n",
    "# ANSWER\n",
    "\n",
    "# Use the URL.get() method to pull data from \n",
    "# https://covid.ourworldindata.org/data/ecdc/full_data.csv \n",
    "# and store in a file called work/full_data.csv\n",
    "\n",
    "# set up URL object\n",
    "url = URL('https://covid.ourworldindata.org/data/archived/ecdc/full_data.csv')\n",
    "# set up file for data as Path\n",
    "ofile = Path('work',url.name)\n",
    "\n",
    "# get data \n",
    "r = url.get()\n",
    "# check response:\n",
    "if r.status_code == 200:\n",
    "    # ok\n",
    "    data = r.text\n",
    "    # remember how to write to a file from \n",
    "    # previous session on Path\n",
    "    ofile.write_text(data)\n",
    "    # check the file size\n",
    "    print(f'{ofile} size: {ofile.stat().st_size} bytes')\n",
    "    # show the first few lines of data\n",
    "    # NB data is a big long string, so split it on \\n\n",
    "    # into lines\n",
    "    data_lines = data.split('\\n')\n",
    "    for i in range(5):\n",
    "        print(f'{i}: {data_lines[i]}')\n",
    "else:\n",
    "    print(f'failed to pull data from {url}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MODIS data|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Getting a MODIS product URL\n",
    "\n",
    "If we need to add a username and password to access a URL, we can add this to the URL object with `URL.with_userinfo(USERNAME,PASSWORD)`. We will demonstrate this with accessing data from NASA servers.\n",
    "\n",
    "One of the deepest sources of geospatial information over the last two decades is that obtained from the NASA [MODIS](https://modis.gsfc.nasa.gov/data/dataprod/) products. We will make use of various MODIS datasets in this course. We can access these via a URL.\n",
    "\n",
    "The encoding for this is not very complex, but is a bit beyond what we have time to go through at this stage of the course. For that reason, we will be using the local utility [`geog0111.modisUtils.modisURL`](geog0111/modisUtils.py) to access MODIS URLs. Further, accessing the URLs for these data needs a data file to vbe downloaded from the web. This can take some time, so in this code, we provide a cache of files that we have already downloaded. If you access a data product/date for the first time, it will take a minute to access the information. The next time you access, it will be immediate as you will use the cached version. See the help information for more details.\n",
    "\n",
    "We use the example of the product [`MCD15A3H`](https://lpdaac.usgs.gov/products/mcd15a3hv006/) below. This is the MODIS 4-day Leaf Area Index product."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from geog0111.modisUtils import modisURL\n",
    "\n",
    "# uncomment this to look at the help for this function\n",
    "#help(modisURL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-> https://e4ftl01.cr.usgs.gov/MOTA/MCD15A3H.006/2020.01.05/MCD15A3H.A2020005.h08v06.006.2020010210940.hdf\n"
     ]
    }
   ],
   "source": [
    "from geog0111.modisUtils import modisURL\n",
    "\n",
    "# set up control for which MODIS product/date/tile\n",
    "\n",
    "modinfo = {  \n",
    "    'product'  : 'MCD15A3H',\n",
    "    'year'     : 2020,\n",
    "    'month'    : 1,\n",
    "    'day'      : 5,\n",
    "    'tile'     : 'h08v06'\n",
    "}\n",
    "\n",
    "url = modisURL(**modinfo,verbose=False)\n",
    "print(f'-> {url}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pulling MODIS data with password \n",
    "\n",
    "We add the username and password with `URL.with_userinfo(USERNAME,PASSWORD)`.\n",
    "\n",
    "Rather than type this information in here or expose it in stored notebooks, we retrieve it using Cylog: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "anchor: https://e4ftl01.cr.usgs.gov\n",
      "-> https://e4ftl01.cr.usgs.gov/MOTA/MCD15A3H.006/2020.01.01/MCD15A3H.A2020001.h08v06.006.2020006032951.hdf\n"
     ]
    }
   ],
   "source": [
    "from geog0111.modisUtils import modisURL\n",
    "from geog0111.cylog import Cylog\n",
    "from urlpath import URL\n",
    "from pathlib import Path\n",
    "\n",
    "# set up control for which MODIS product/date/tile\n",
    "\n",
    "modinfo = {  \n",
    "    'product'  : 'MCD15A3H',\n",
    "    'year'     : 2020,\n",
    "    'month'    : 1,\n",
    "    'day'      : 1,\n",
    "    'tile'     : 'h08v06'\n",
    "}\n",
    "\n",
    "url = modisURL(**modinfo,verbose=False)\n",
    "if (url):\n",
    "    print(f'anchor: {url.anchor}')\n",
    "    print(f'-> {url}')\n",
    "\n",
    "    # use url.anchor to get server name for Cylog\n",
    "    # add the username and password\n",
    "    username,password = Cylog(url.anchor).login()\n",
    "    url = url.with_userinfo(username,password)\n",
    "\n",
    "    # uncomment this line to see how username and password are inserted\n",
    "    #print(f'-> {url}')\n",
    "else:\n",
    "    print(f'error with data request')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because of the NASA login, we need to make two calls to the server. The first will get a new URL with encoded authentification information. \n",
    "\n",
    "`url.get()` returns an object of type `requests.models.Response`. The new URL will be `r.url`, so we take that and add the username and password to it again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# first call to URL to get auth\n",
    "r = url.get()\n",
    "url2 = URL(r.url).with_userinfo(username,password)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can make a call to this with `get()` to retrieve the data from the URL. We need to check that the `status_code` returned is 200 to see if the call worked:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "response: 200\n"
     ]
    }
   ],
   "source": [
    "r2 = url2.get()\n",
    "print(f'response: {r2.status_code}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the response is *not* `200`, then you have a problem of some sort. This could be that you don't have a correct password stored via `cylog`. If you suspect that might be the case, **go back and check the test on [`004_Accounts.ipynb`](004_Accounts.ipynb)**. If it is *Wednesday afternoon*, it is probably just that the NASA servers are down for maintenance. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If its ok (`200`), we can get the binary data as `r2.content` and use `Path().write_bytes()` to write this to a file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "file work/MCD15A3H.A2020001.h08v06.006.2020006032951.hdf written: 8.6 MB\n"
     ]
    }
   ],
   "source": [
    "if r2.status_code == 200:\n",
    "    \n",
    "    # setup Path object for output file\n",
    "    filename = Path('work',url.name)\n",
    "    \n",
    "    # write binary data\n",
    "    filename.write_bytes(r2.content)\n",
    "    \n",
    "    # check size:\n",
    "    size_MB = filename.stat().st_size/(1024**2)\n",
    "    \n",
    "    # report\n",
    "    print(f'file {filename} written: {size_MB :.1f} MB')\n",
    "else:\n",
    "    print(f'{url} status code {r2.status_code}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Occasionally, you will find that the calls to get data from the server will not work, and may produce 401 errors. \n",
    "\n",
    "This is typically because:\n",
    "* it is Wednesday, when the servers are down for maintenance\n",
    "* the load is too high on the servers, in which case, wait a minute and try again or set the timeout option `URL.get(timeout=200)`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "#### Exercise 2\n",
    "\n",
    "* pull the MODIS dataset `MCD15A3H` for 9 January 2019 for tile `h08v06` and confirm that the dataset size is 8.5 MB\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "anchor: https://e4ftl01.cr.usgs.gov\n",
      "-> https://e4ftl01.cr.usgs.gov/MOTA/MCD15A3H.006/2020.01.09/MCD15A3H.A2020009.h08v06.006.2020014204616.hdf\n",
      "getting MODIS URL\n",
      "MOTA/MCD15A3H.006/2020.01.09/MCD15A3H.A2020009.h08v06.006.2020014204616.hdf\n",
      "getting Auth\n",
      "response: 200\n",
      "file work/MCD15A3H.A2020009.h08v06.006.2020014204616.hdf written: 8.5 MB\n"
     ]
    }
   ],
   "source": [
    "from geog0111.modisUtils import modisURL\n",
    "from geog0111.cylog import Cylog\n",
    "from urlpath import URL\n",
    "from pathlib import Path\n",
    "# ANSWER\n",
    "\n",
    "# set up control for which MODIS product/date/tile\n",
    "\n",
    "modinfo = {  \n",
    "    'product'  : 'MCD15A3H',\n",
    "    'year'     : 2020,\n",
    "    'month'    : 1,\n",
    "    'day'      : 9,\n",
    "    'tile'     : 'h08v06'\n",
    "}\n",
    "\n",
    "# get the URL\n",
    "url = modisURL(**modinfo,verbose=False)\n",
    "if url:\n",
    "    print(f'anchor: {url.anchor}')\n",
    "    print(f'-> {url}')\n",
    "\n",
    "    # use url.anchor to get server name for Cylog\n",
    "    # add the username and password\n",
    "    print('getting MODIS URL')\n",
    "    username,password = Cylog(url.anchor).login()\n",
    "    url = url.with_userinfo(username,password)\n",
    "    # dont print out the username and password!\n",
    "    print('/'.join(url.parts[1:]))\n",
    "\n",
    "    # first call to URL to get auth\n",
    "    print('getting Auth')\n",
    "    r = url.get()\n",
    "    url2 = URL(r.url).with_userinfo(username,password)\n",
    "\n",
    "    if url2:\n",
    "    # net call to get data\n",
    "        r2 = url2.get()\n",
    "        print(f'response: {r2.status_code}')\n",
    "\n",
    "        if r2.status_code == 200:\n",
    "\n",
    "            # setup Path object for output file\n",
    "            filename = Path('work',url.name)\n",
    "\n",
    "            # write binary data\n",
    "            filename.write_bytes(r2.content)\n",
    "\n",
    "            # check size:\n",
    "            size_MB = filename.stat().st_size/(1024**2)\n",
    "\n",
    "            # report\n",
    "            print(f'file {filename} written: {size_MB :.1f} MB')\n",
    "        else:\n",
    "            print(f'{\"/\".join(url.parts[1:])} status code {r2.status_code}')\n",
    "    else:\n",
    "        print('error getting url')\n",
    "else:\n",
    "    print('error in data request')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Easier MODIS file access\n",
    "\n",
    "In fact, we have a related MODIS utility `modisFile` that does all of this for you, and uses the cache to store files we have previously downloaded. You will generally be using that then, but you should know how to get the binary data from the URL directly as above to be a competent programmer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "file /Users/philiplewis/Documents/GitHub/geog0111/notebooks/.modis_cache/e4ftl01.cr.usgs.gov/MOTA/MCD15A3H.006/2020.01.05/MCD15A3H.A2020005.h08v06.006.2020010210940.hdf is: 8.9 MB\n"
     ]
    }
   ],
   "source": [
    "from geog0111.modisUtils import modisFile\n",
    "\n",
    "# set up control for which MODIS product/date/tile\n",
    "\n",
    "modinfo = {  \n",
    "    'product'  : 'MCD15A3H',\n",
    "    'year'     : 2020,\n",
    "    'month'    : 1,\n",
    "    'day'      : 5,\n",
    "    'tile'     : 'h08v06'\n",
    "}\n",
    "\n",
    "filename = modisFile(**modinfo,verbose=False)\n",
    "\n",
    "if filename:\n",
    "    size_MB = filename.stat().st_size/(1024**2)\n",
    "    \n",
    "    # report\n",
    "    print(f'file {filename} is: {size_MB :.1f} MB')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "#### Exercise 3\n",
    "\n",
    "* pull the MODIS dataset `MCD15A3H` for 13 January 2019 for tile `h08v06` using `modisFile` and confirm that the dataset size is 8.5 MB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "file /Users/philiplewis/Documents/GitHub/geog0111/notebooks/.modis_cache/e4ftl01.cr.usgs.gov/MOTA/MCD15A3H.006/2020.01.13/MCD15A3H.A2020013.h08v06.006.2020018030252.hdf is: 8.4 MB\n"
     ]
    }
   ],
   "source": [
    "from geog0111.modisUtils import modisFile\n",
    "\n",
    "# set up control for which MODIS product/date/tile\n",
    "\n",
    "modinfo = {  \n",
    "    'product'  : 'MCD15A3H',\n",
    "    'year'     : 2020,\n",
    "    'month'    : 1,\n",
    "    'day'      : 13,\n",
    "    'tile'     : 'h08v06'\n",
    "}\n",
    "\n",
    "filename = modisFile(**modinfo,verbose=False)\n",
    "\n",
    "if filename:\n",
    "    size_MB = filename.stat().st_size/(1024**2)\n",
    "    \n",
    "    # report\n",
    "    print(f'file {filename} is: {size_MB :.1f} MB')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `gdal`\n",
    "\n",
    "\n",
    "The MODIS files are in `hdf` format, and as we have noted, we do not generally want direct access to the raw (byte) information. Instead we must use some package to interpret the data. \n",
    "\n",
    "We can use the package [`gdal`](https://gdal.org/python/) to access information from these and other geospatial files. We will explore the contents of MODIS files in a later session, but for now, we can note that each MODIS file contains a set of sub datasets.\n",
    "\n",
    "Basic use of `gdal` in this context is:\n",
    "\n",
    "    g = gdal.Open(filename.as_posix())\n",
    "    \n",
    "where `filename.as_posix()` is a string of the filename we want open the file in `gdal`. If this returns None, there has been a problem opening the file, so we might check that.\n",
    "\n",
    "Then\n",
    "\n",
    "    g.GetSubDatasets()\n",
    "   \n",
    "returns a list of sub-dataset information. Each item in the list is a tuple of two strings. In each, the first is the full name of the sub-dataset, and the second a text descriptor of the dataset. We call these `filename,name` below.\n",
    "\n",
    "We read the dataset with:\n",
    "\n",
    "    gsub = gdal.Open(filename)\n",
    "    data = gsub.ReadAsArray()\n",
    "    \n",
    "In the illustration below, we will examine only the first sub-dataset `g.GetSubDatasets()[0]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dataset info is: [2400x2400] Fpar_500m MOD_Grid_MCD15A3H (8-bit unsigned integer)\n",
      "dataset read is shape (2400, 2400) and type <class 'numpy.ndarray'>\n"
     ]
    }
   ],
   "source": [
    "from osgeo import gdal\n",
    "from geog0111.modisUtils import modisFile\n",
    "# settings\n",
    "\n",
    "modinfo = {  \n",
    "    'product'  : 'MCD15A3H',\n",
    "    'year'     : 2019,\n",
    "    'month'    : 2,\n",
    "    'day'      : 10,\n",
    "    'tile'     : 'h08v06'\n",
    "}\n",
    "\n",
    "filename = modisFile(**modinfo,verbose=False)\n",
    "\n",
    "if filename:\n",
    "# open the local file associated with the dataset\n",
    "    g = gdal.Open(filename.as_posix())\n",
    "    if g:\n",
    "        # get the first SDS only for illustration\n",
    "        filename,name = g.GetSubDatasets()[0]\n",
    "        print(f'dataset info is: {name}')\n",
    "        # read the dataset\n",
    "        gsub = gdal.Open(filename)\n",
    "        if gsub:\n",
    "            data = gsub.ReadAsArray()\n",
    "        print(f'dataset read is shape {data.shape} and type {type(data)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "#### Exercise 4\n",
    "\n",
    "    name = '[2400x2400] Fpar_500m MOD_Grid_MCD15A3H (8-bit unsigned integer)'\n",
    "\n",
    "* Take the string variable `name` above, split it to obtain the second field (`Fpar_500m` here) and store this in a variable `sds_name`\n",
    "* Write a function called `getModisTiledata` that reads an HDF (MODIS) filename, and returns a dictionary of all of the sub-datasets in the file, using `ReadAsArray()`. The dictionary keys should correspond to the items in  `sds_name` above.\n",
    "* test the code by showing the keys in the dictionary returned and the shape of their dataset\n",
    "\n",
    "You will need to recall how to split a string, that was covered in [013 Python string methods](013_Python_string_methods.ipynb#split()-and-join()). You will also need to recall how to [loop over a dictionary](016_Python_for.ipynb#looping-over-dictionaries,-and-assert). We saw how to find the shape of the dataset returned above (`.shape`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fpar_500m\n"
     ]
    }
   ],
   "source": [
    "# ANSWER \n",
    "name = '[2400x2400] Fpar_500m MOD_Grid_MCD15A3H (8-bit unsigned integer)'\n",
    "\n",
    "# Take the string variable name above, split it to obtain the \n",
    "# second field (Fpar_500m here) and store this in a variable sds_name\n",
    "\n",
    "# use str.split() and take item 1 from the list\n",
    "sds_name = name.split()[1]\n",
    "print(sds_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fpar_500m Lai_500m FparLai_QC FparExtra_QC FparStdDev_500m LaiStdDev_500m\n"
     ]
    }
   ],
   "source": [
    "# ANSWER \n",
    "from osgeo import gdal\n",
    "from geog0111.modisUtils import modisFile\n",
    "\n",
    "# set up control for which MODIS product/date/tile\n",
    "\n",
    "modinfo = {  \n",
    "    'product'  : 'MCD15A3H',\n",
    "    'year'     : 2020,\n",
    "    'month'    : 1,\n",
    "    'day'      : 13,\n",
    "    'tile'     : 'h08v06'\n",
    "}\n",
    "\n",
    "\n",
    "def getModisTiledata(**modinfo):\n",
    "    '''\n",
    "    get MODIS data dictionary\n",
    "    '''\n",
    "    # set up blank dictionary for output\n",
    "    odata = {}\n",
    "    filename = modisFile(**modinfo)\n",
    "    # error checking\n",
    "    if not filename:\n",
    "        return odata\n",
    "    g = gdal.Open(filename.as_posix())\n",
    "    if g:\n",
    "        for filename,name in g.GetSubDatasets():\n",
    "            # get the SDS\n",
    "            #print(f'dataset info is: {name}')\n",
    "            # read the dataset\n",
    "            gsub = gdal.Open(filename)\n",
    "            if gsub:\n",
    "                data = gsub.ReadAsArray()\n",
    "                sds_name = name.split()[1]\n",
    "                # load into dictionary\n",
    "                odata[sds_name] = data\n",
    "    return odata\n",
    "    \n",
    "data = getModisTiledata(**modinfo)\n",
    "print(*data.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "\n",
    "In this section, we have considered URLs in some detail in the same way as we did filenames, and made use of functions from [urlpath](https://github.com/chrono-meter/urlpath) to access them. \n",
    "\n",
    "|function| purpose|\n",
    "|---|---|\n",
    "|`URL.name`|  filename |\n",
    "|`URL.parent`|  parent |\n",
    "|`URL.parts`|  parts |\n",
    "| `URL.as_posix()` | return URL as posix string |\n",
    "|`URL.with_userinfo()`|  add in username and password |\n",
    "|`URL.get()`|  URL get. Returns `requests.models.Response`|\n",
    "| `URL.netloc` | network location e.g. `www.google.com`|\n",
    "| `URL.path` | full pathname on server (including filename)|\n",
    "\n",
    "We have seen how to access and download both text and binary files from the web using `URL.get()`. We have seen how to add username and password information to this, and have used that to access MODIS binary datasets. We know how to save the files that we download.\n",
    "\n",
    "We have seen a simple MODIS file library to make downloading datasets easier. This includes caching of datasets that otherwise can take a long time to download. It also allows for sharing of downloaded datasets through a system-wide cache. \n",
    "\n",
    "There will be times when a call to a URL doesn't return as expected. There can be several reasons for this, so you should check that you have formed the URL correctly and that the data you expect to download exists. \n",
    "\n",
    "\n",
    "Functions in `modisUtils`:\n",
    "\n",
    "|function| purpose|\n",
    "|---|---|\n",
    "|`modisFile`| get a Path object for a file of the requested MODIS dataset, either from cache, or by downloading |\n",
    "|`modisURL`| get the URL of a MODIS dataset, possibly using a cache for the filename |\n",
    "\n",
    "Even then, there can be times when the server load or network traffic mean that your request times out. In such cases, you should try again a short while afterwards, and consider setting a timeout on the `URL.get()` call. In any case, this point illustrates the need for comprehensive error checking: you cannot just assume that you download of data has worked, you must check it. This should also apply when you are using any files: check that any file you want to read exists and perhaps see if it is non-zero sized. You can trap errors, but it is best to try foresee them.\n",
    "\n",
    "We have learned a little of how to use `gdal` to look at the sub-datasets in an HDF file and also how to read them.\n",
    "\n",
    "\n",
    "`gdal`\n",
    "\n",
    "| Command | Comment |\n",
    "|---|---|\n",
    "|`g = gdal.Open(filename)` | Open geospatial file `filename` and return `gdal` object `g` (`None` if file not opened correctly)|\n",
    "|`g.GetSubDatasets()` | Get list of sub-datasets from `gdal` object `g`| \n",
    "|`g.ReadAsArray()` | Read dataset from `gdal` object `g` into array |\n",
    "\n",
    "You should now have some confidence in these matters, so that if you were set a task of downloading and saving datasets, as well as other tasks such as finding their size, whether the exists or not, you could do this. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "[<img src=\"images/noun_post_2109127.svg\" width=\"50\" align='right'>](022_Pandas.ipynb)[<img src=\"images/noun_pre_2109128.svg\" width=\"50\" align='right'>](018_Running_Python.ipynb)\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda env:geog0111-geog0111",
   "language": "python",
   "name": "conda-env-geog0111-geog0111-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": false,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "207px"
   },
   "toc_section_display": false,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}