01-webscraping-FINAL.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "43ecd911-91e7-4332-9503-2156e2cd978e",
   "metadata": {},
   "source": [
    "# Fire Extent Prediction from Historical Weather Data\n",
    "\n",
    "###### Adam Swan, Jeff Warchall, Omar Younis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97d5ae5d-b6a8-4aab-8e0e-1712c24b6698",
   "metadata": {},
   "source": [
    "## Problem Statement"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fac1f30-fc69-4e6c-8bbd-5750707c7aec",
   "metadata": {},
   "source": [
    "To help the California Department of Forestry and Fire Protection allocate resources, can we predict the likelihood of fires utilizing historical weather and wildfire data?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e01be75-07aa-46eb-a027-b7b27a4a959e",
   "metadata": {},
   "source": [
    "### County Location Data\n",
    "\n",
    "From the California [Open Data Portal](https://data.ca.gov/dataset/ca-geographic-boundaries/resource/b0007416-a325-4777-9295-368ea6b710e6) we obtained a .SHP file of the geographic locations of the counties of California.  A .csv file containing the latitude and longitude of the centers of the counties was then extracted using [MyGeodata Converter](https://mygeodata.cloud/converter/shp-to-csv) stored in the data folder then placed into the ```location``` variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "047ea73b-c311-47d7-a8f9-961e180116a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import pandas as pd\n",
    "import time\n",
    "from IPython.display import clear_output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a77f461-bb43-4a18-a3cc-6682cce07494",
   "metadata": {},
   "outputs": [],
   "source": [
    "counties = pd.read_csv('data/CA_Counties_Location.csv')\n",
    "location = counties[['NAMELSAD', 'INTPTLAT', 'INTPTLON']]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1368fa70-2a5e-4cac-a58c-f654d67e1568",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Fire Data\n",
    "\n",
    "The fire data was obtained from the State of California [Geoportal](https://gis.data.ca.gov/), which generated the url used to fetch the data in the cell below.  We obtained all of the fire from 1952 to the present.  These data are described in the data dictionary at the end of Section 02 - Preprocessing.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "476eae40-2b09-47f5-87d0-f5baa2fcf8a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'https://egis.fire.ca.gov/arcgis/rest/services/FRAP/FirePerimeters_FS/FeatureServer/0/query?where=1%3D1&outFields=*&outSR=4326&f=json'\n",
    "res = requests.get(url)\n",
    "data = res.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfe62259-1ec8-44f1-a39d-67f809776813",
   "metadata": {},
   "source": [
    "The data is presented in .json format and the features are stored in a dataframe named ```df``` and then exported to a .csv file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39f5fa09-dec3-4121-8d33-4bfeffe3cf0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "list_of_records = []\n",
    "list_of_features = data['features'][0]['attributes'].keys()\n",
    "for index, value in enumerate(data['features']):\n",
    "    list_of_records.append(data['features'][index]['attributes'].values())\n",
    "    \n",
    "df = pd.DataFrame(list_of_records, columns = list_of_features)\n",
    "df.to_csv('./data/raw_fires.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9d386dc-ec2c-4787-a0e6-d077cb1a764a",
   "metadata": {},
   "source": [
    "### California Weather Data\n",
    "\n",
    "Weather data was obtained from [World Weather Online](https://www.worldweatheronline.com/developer/) (WWO).  First, using the county latitude and longitude data a list of the centers of each county in California is created.  Then, since the data available on WWO only goes back to July 1, 2008, list containing every year, month, and day in the range 07/01/2008 to 12/31/2020 are created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e7acc6e-eb54-4ca1-842b-180be715b063",
   "metadata": {},
   "outputs": [],
   "source": [
    "lats = list(location['INTPTLAT'])\n",
    "longs = list(location['INTPTLON'])\n",
    "years = list(range(2008, 2021))\n",
    "months = ['01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12']\n",
    "days = [31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d822ce5b-218c-4326-86dd-61fe5087a7c3",
   "metadata": {},
   "source": [
    "The WWO API requires a query string to be passed.  These queries are assembled in the following loop to create a unique query for every day in our range of interest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "986f8e26-63d4-472c-8126-de869c837516",
   "metadata": {},
   "outputs": [],
   "source": [
    "urls = []\n",
    "for year in years:\n",
    "    for i, month in enumerate(months):\n",
    "        if month > '06' or year > 2008:\n",
    "            for (j, lat) in enumerate(lats): \n",
    "                start_date = str(year)+ '-' + month + '-' + '01'\n",
    "                end_date = str(year)+ '-' + month + '-' + str(days[i])\n",
    "                long = longs[j]\n",
    "                url = 'http://api.worldweatheronline.com/premium/v1/past-weather.ashx?key=xxxxxxxxxxxxxxxxxxxxx'\n",
    "                url = url + '&format=json'\n",
    "                url = url + '&q=' + str(lat) + ',' + str(long)\n",
    "                url = url + '&tp=24'\n",
    "                url = url + '&date=' + start_date\n",
    "                url = url + '&enddate=' + end_date\n",
    "                urls.append(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95781258-dff1-49ac-acb4-f99e6c27b4f6",
   "metadata": {},
   "source": [
    "The list of urls is then passed to the API and the returned .json documents are stored in a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "96261aaa-6128-41b6-8b4e-958d64b02f48",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = []\n",
    "for i, url in enumerate(urls):\n",
    "    time.sleep(1)\n",
    "    res = requests.get(url)\n",
    "    data.append(res.json())\n",
    "    clear_output()\n",
    "    print(f'{i+1} of 3480 requests complete')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13d98696-666d-4827-ba91-5ddd3e8de6ee",
   "metadata": {},
   "source": [
    "Finally, we iterate through the .json documents to compile an array of records and save them to a data frame.  This data frame is then exported to the data folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d511233-1a1f-4178-ac21-e9245cce3aca",
   "metadata": {},
   "outputs": [],
   "source": [
    "list_of_features = list(data[0]['data']['weather'][0].keys())\n",
    "list_of_features.append('Lat and Long')\n",
    "\n",
    "list_of_records = []\n",
    "for i in range(len(data)):\n",
    "    for dictionary in data[i]['data']['weather']:\n",
    "        list_of_values = []\n",
    "        for value in dictionary.values():\n",
    "            list_of_values.append(value)\n",
    "        list_of_values.append(data[i]['data']['request'][0]['query'])\n",
    "        list_of_records.append(list_of_values)\n",
    "\n",
    "df = pd.DataFrame(list_of_records, columns = list_of_features)\n",
    "df.to_csv('./data/clean_daily_weather.csv')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}