1_rest_demo.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZbvD2qFGSVxc"
      },
      "source": [
        "Let's start with a review of the URI (Uniform Resource Identifier) syntax.\n",
        "\n",
        "## Components of a URI (Uniform Resource Identifier)\n",
        "\n",
        "A URI is a string of characters used to identify a resource over a network. A typical URI consists of the following components:\n",
        "\n",
        "### 1. Scheme\n",
        "The **scheme** defines the protocol or method used to access the resource. Examples include `http`, `https`, `ftp`, etc. It is followed by a colon (`:`).\n",
        "\n",
        "- Example: `http` in `http://example.com`\n",
        "\n",
        "### 2. Authority\n",
        "The **authority** section usually contains the hostname and often the port number. It is prefixed by `//`.\n",
        "\n",
        "- Example: `example.com:8080` in `http://example.com:8080/path`\n",
        "\n",
        "### 3. Path\n",
        "The **path** specifies the resource or the specific page that you want to access. It comes after the authority and is separated by slashes (`/`).\n",
        "\n",
        "- Example: `/path/to/resource` in `http://example.com/path/to/resource`\n",
        "\n",
        "### 4. Query\n",
        "The **query** starts with a question mark (`?`) and is used to pass parameters to the resource. Multiple query parameters are separated by ampersands (`&`).\n",
        "\n",
        "- Example: `?key1=value1&key2=value2` in `http://example.com/path?key1=value1&key2=value2`\n",
        "\n",
        "### 5. Fragment\n",
        "The **fragment** is an optional part that starts with a hash (`#`) and is used to identify a specific part of the resource, like a section within a page.\n",
        "\n",
        "- Example: `#section1` in `http://example.com/path#section1`\n",
        "\n",
        "Here's how these components fit together to form a complete URI:\n",
        "\n",
        "\n",
        "- Complete Example: `http://example.com:8080/path/to/resource?key1=value1&key2=value2#section1`\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "id": "Of7fZ5VcSVxg"
      },
      "source": [
        "  ### Retrieving Data from the API\n",
        "\n",
        "Next we need a way to retrieve the data from the API. To achieve this, we first import Python's `requests` library, which allows us to make HTTP requests. Additionally, we import the `pprint` function from the `pprint` library to make the output more readable. Finally, we define the API endpoint we'll be interacting with, setting `API_ENDPOINT` to `https://testing-api.water.noaa.gov/hefs/`. Here's how you can do it:\n",
        "\n",
        "```python\n",
        "# start by importing the requests library\n",
        "import requests\n",
        "# import the pprint library to make the output more readable\n",
        "from pprint import pprint\n",
        "# define the api-endpoint\n",
        "# Note: API Endpoint will change to https://api.water.noaa.gov/hefs/ after testing is complete\n",
        "API_ENDPOINT = \"https://testing-api.water.noaa.gov/hefs\"\n",
        "```\n",
        "\n",
        "**Please change the endpoint to match the server you are interacting with.**\n",
        "\n",
        "From here  you will only need to set the uri, fetch the data, and print the results.  Here is a template for the code:\n",
        "\n",
        "```python\n",
        "# set the uri\n",
        "uri = \"/v1/locations\"\n",
        "# fetch the data\n",
        "r = requests.get(API_ENDPOINT + uri)\n",
        "# print the results\n",
        "pprint(r.json())\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [],
        "id": "yhYCb2CsSVxg"
      },
      "outputs": [],
      "source": [
        "# start by importing the requests library\n",
        "import requests\n",
        "# import the pprint library to make the output more readable\n",
        "from pprint import pprint\n",
        "# define the api-endpoint\n",
        "API_ENDPOINT = \"https://testing-api.water.noaa.gov/hefs\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "id": "Yu4zlMtiSVxh"
      },
      "source": [
        "## The Header Endpoint: `/v1/headers/`\n",
        "\n",
        "The `/v1/headers/` endpoint is a part of the RESTful API that provides access to various series data. By making HTTP requests to this URI, you can retrieve, update, or delete information related to data series stored in the backend. This endpoint by default returns time series sequentially.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [],
        "id": "rCa8HsA1SVxh"
      },
      "outputs": [],
      "source": [
        "# create a series request\n",
        "uri = API_ENDPOINT + \"/v1/headers/\"\n",
        "\n",
        "# get the response\n",
        "response = requests.request(\"GET\", uri, headers=None)\n",
        "# print the response\n",
        "print(response.headers)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "id": "KDwQ1yRwSVxi"
      },
      "source": [
        "## Pagination\n",
        "\n",
        "The `/v1/headers/?limit=10&offset=10` endpoint is a specialized version of the `/v1/headers/` URI, which includes query parameters for pagination. In this case, the `limit` parameter is set to 10, which means the API will return a maximum of 10 records in the response.\n",
        "The offset parameter is set to 10, which means the API will skip the first 10 records in the response. This is useful for retrieving large amounts of data, as it allows you to retrieve data in chunks."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "J-AlBYK4SVxi"
      },
      "outputs": [],
      "source": [
        "# create a series request returning the second tranche of 10 series\n",
        "uri = API_ENDPOINT + \"/v1/headers/?limit=10&offset=10\"\n",
        "# get the response\n",
        "response = requests.request(\"GET\", uri, headers=None)\n",
        "# print the response\n",
        "pprint(response.content)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "id": "Iiux7e0YSVxj"
      },
      "source": [
        "## Filtering Data with the API\n",
        "\n",
        "The API endpoint `/v1/headers/?limit=10&offset=10` allows you to retrieve a subset of data series.\n",
        "\n",
        "It uses query parameters to offer flexibility in filtering results. The `limit` parameter controls the number of records returned, while the `offset` parameter lets you skip a certain number of records from the beginning.\n",
        "\n",
        "You can also filter results based on specific fields like `location_id`, `parameter_id`, and many more. The API filters on all available fields by appending the field name and value to the URI as query parameters.\n",
        "\n",
        "For instance, if you're interested in series that have a specific `parameter_id` and `location_id`, the URI would look like this:\n",
        "\n",
        "/v1/headers/?limit=10&offset=10&parameter_id=QINE&location_id=BLUO2\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e-G9MxBKSVxj"
      },
      "outputs": [],
      "source": [
        "# create a series request returning the second tranche of 10 series, with location_id=BLUO2 and parameter_id=QINE filters\n",
        "uri = API_ENDPOINT + \"/v1/headers/?limit=10&offset=10&parameter_id=QINE&location_id=BLUO2\"\n",
        "# get the response\n",
        "response = requests.request(\"GET\", uri, headers=None)\n",
        "# print the response\n",
        "pprint(response.json())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ayr_AQZ1SVxj"
      },
      "source": [
        "### Filtering Data Based on Multiple Parameters\n",
        "\n",
        "In some use cases, you might want to retrieve data that matches more than one parameter value. The API allows filtering by multiple values for the same parameter. This is achieved by appending `&` to the parameter name and specifying the different values.\n",
        "\n",
        "For example, to retrieve data series that belong to either `location_id=BLUO2` or `location_id=ABEC2`, you can format the URI like this:\n",
        "\n",
        "`/v1/headers/?location_id=BLUO2&location_id=ABEC2`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "JqmsGpOcSVxj"
      },
      "outputs": [],
      "source": [
        "# create a series request with location_id=ABEC2 and location_id=BLUO2 filters\n",
        "uri = API_ENDPOINT + \"/v1/headers/?location_id=ABEC2&location_id=BLUO2\"\n",
        "# get the response\n",
        "response = requests.request(\"GET\", uri)\n",
        "# print the response\n",
        "pprint(response.json())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lWlt2sycS05l"
      },
      "source": [
        "### Most Recent Ensemble Forecast for a Location and plotting\n",
        "\n",
        "In other cases, you might want to retrieve and plot recent ensemble data. The API allows ordering by a certain value of a parameter. This is achieved by adding `ordering=` to a parameter name.\n",
        "\n",
        "For example, to retrieve the newest data series that belongs to `location_id=PGRC2` ordered by start_date_date, you can format the URI like this:\n",
        "\n",
        "`/v1/headers/?location_id=PGRC2&parameter_id=QINE&ordering=-start_date_date`\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from datetime import datetime, timedelta\n",
        "# create a series request with location_id, parameter_id, and ordering filters\n",
        "uri = API_ENDPOINT + f\"/v1/headers/?location_id=PGRC2&parameter_id=QINE&ordering=-start_date_date\"\n",
        "# get the response\n",
        "response = requests.request(\"GET\", uri)\n",
        "# get most recent start date\n",
        "startDate = response.json()['results'][0]['start_date_date']\n",
        "# print the response\n",
        "pprint(response.json())"
      ],
      "metadata": {
        "id": "d7mnUmBtEguU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now, with the most recent forecast startDate, we will retrieve all of the headers of that date. Then, get the total number of forecasts to be used in the graph process later.\n",
        "\n",
        "For example, to retrieve the newest data series headers, you can format the URI like this:\n",
        "\n",
        "`/v1/headers/?location_id=PGRC2&parameter_id=QINE&forecast_date_date={startDate}`\n",
        "\n",
        "Where the startDate is the value of the variable found in the previous section"
      ],
      "metadata": {
        "id": "eHADDfB6pJCJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from datetime import datetime, timedelta\n",
        "# create a series request with location_id, parameter_id, and start_date_date filters\n",
        "uri = API_ENDPOINT + f\"/v1/headers/?location_id=PGRC2&parameter_id=QINE&forecast_date_date={startDate}\"\n",
        "# get the response\n",
        "response = requests.request(\"GET\", uri)\n",
        "\n",
        "count = response.json()[\"count\"]\n",
        "# print the response\n",
        "pprint(response.json())"
      ],
      "metadata": {
        "id": "xYAhy4Pkl4RF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, with the most recent forecast startDate and total number of forecasts available, we will retrieve all of the ensemble data for that start date. This is the data that we will plot next.\n",
        "\n",
        "We will use the term `limit` to set the total series output to the count set above. To retrieve the newest data series ensembles, you can format the URI like this:\n",
        "\n",
        "`/v1/ensembles/?location_id=PGRC2&parameter_id=QINE&forecast_date_date={startDate}&limit={count}`\n",
        "\n",
        "Where the count is the total number of forecasts for a given ensemble"
      ],
      "metadata": {
        "id": "rhgeI53Gq4k7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create a series request with location_id, parameter_id, start_date_date, and limit filters\n",
        "uri = API_ENDPOINT + f\"/v1/ensembles/?location_id=PGRC2&parameter_id=QINE&fstart_date_date={startDate}&limit={count}\"\n",
        "# get the response\n",
        "response = requests.request(\"GET\", uri)\n",
        "# print the response\n",
        "pprint(response.json())"
      ],
      "metadata": {
        "id": "l9Z2jU100sM3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Finally, we take the response data to plot the ensemble."
      ],
      "metadata": {
        "id": "GolV7Z0Etds7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import datetime\n",
        "\n",
        "def graph_ensemble_data(response_data):\n",
        "  \"\"\"Graphs ensemble data from a DRF response.\n",
        "\n",
        "  Args:\n",
        "    response_data: A dictionary containing the DRF response data.\n",
        "  \"\"\"\n",
        "\n",
        "  if 'results' in response_data and response_data['results']:\n",
        "\n",
        "    for result in response_data['results']:\n",
        "\n",
        "      if 'events' in result and result['events']:\n",
        "        for event in result['events']:\n",
        "          times = []\n",
        "          values = []\n",
        "          for value in result['events']:\n",
        "              datetime_str = f\"{value['date']} {value['time']}\"\n",
        "              datetime_obj = datetime.datetime.strptime(datetime_str, \"%Y-%m-%d %H:%M:%S\")\n",
        "              times.append(datetime_obj)\n",
        "              values.append(value['value'])\n",
        "          plt.plot(times, values)\n",
        "    plt.xlabel('Time')\n",
        "    plt.ylabel('Value')\n",
        "    plt.title('Ensemble Data')\n",
        "    plt.xticks(rotation='vertical')\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "#pprint(response.json())\n",
        "graph_ensemble_data(response.json())\n"
      ],
      "metadata": {
        "id": "IPSP1PFqX-pi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kn31tBc-SVxj"
      },
      "source": [
        "## Summary\n",
        "\n",
        "In this notebook, we learned how to use the HEFS API to retrieve data from the HEFS data store. We also learned how to use the API to filter results based on specific fields like `location_id`, `parameter_id`, and many more. The API filters on all available fields by appending the field name and value to the URI as query parameters.\n",
        "\n",
        "These techniques can be used to retrieve, filter, and paginate data from all resources the HEFS data store.  With one exception, the GraphQL interface, we cover that next."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "conda_python3",
      "language": "python",
      "name": "conda_python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.13"
    },
    "vscode": {
      "interpreter": {
        "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
      }
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}