03-resampling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "efec857a-42dd-4d39-9784-6140ea12982a",
   "metadata": {},
   "source": [
    "## Resampling Monthly Data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "43b96b2f-2df8-44cf-a624-dd5c071b15c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from IPython.display import clear_output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2b04310a-e841-47fd-be64-eef59bb111d9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 8700 entries, 0 to 8699\n",
      "Data columns (total 12 columns):\n",
      " #   Column        Non-Null Count  Dtype         \n",
      "---  ------        --------------  -----         \n",
      " 0   date          8700 non-null   datetime64[ns]\n",
      " 1   county        8700 non-null   object        \n",
      " 2   maxtempF      8700 non-null   float64       \n",
      " 3   mintempF      8700 non-null   float64       \n",
      " 4   avgtempF      8700 non-null   float64       \n",
      " 5   totalSnow_cm  8700 non-null   float64       \n",
      " 6   humid         8700 non-null   float64       \n",
      " 7   wind          8700 non-null   float64       \n",
      " 8   precip        8700 non-null   float64       \n",
      " 9   sunHour       8700 non-null   float64       \n",
      " 10  lat           8700 non-null   float64       \n",
      " 11  long          8700 non-null   float64       \n",
      "dtypes: datetime64[ns](1), float64(10), object(1)\n",
      "memory usage: 815.8+ KB\n"
     ]
    }
   ],
   "source": [
    "weather = pd.read_csv('./data/weather_data.csv')\n",
    "weather['date'] = pd.to_datetime(weather.date)\n",
    "weather.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a95fb8ea-0157-4487-b8c5-7c6d6794c516",
   "metadata": {},
   "source": [
    "At this point the weather data is aggregated on a monthly basis.  Since we are predicting the effect that weather might have on forest fires it may be beneficial to look at what the weather has been like in a given county for the months leading up to a potential fire.  If it has not rained in months, for example, this might be a valuable predictive feature.  Therefore we resample some selected features on a 3 month basis and add these new features to our data frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b53a303e-4cba-4b8f-a37d-de63f52d4701",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "58\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 8700 entries, 0 to 8699\n",
      "Data columns (total 15 columns):\n",
      " #   Column        Non-Null Count  Dtype  \n",
      "---  ------        --------------  -----  \n",
      " 0   date          8700 non-null   object \n",
      " 1   county        8700 non-null   object \n",
      " 2   maxtempF      8700 non-null   float64\n",
      " 3   mintempF      8700 non-null   float64\n",
      " 4   avgtempF      8700 non-null   float64\n",
      " 5   totalSnow_cm  8700 non-null   float64\n",
      " 6   humid         8700 non-null   float64\n",
      " 7   wind          8700 non-null   float64\n",
      " 8   precip        8700 non-null   float64\n",
      " 9   sunHour       8700 non-null   float64\n",
      " 10  lat           8700 non-null   float64\n",
      " 11  long          8700 non-null   float64\n",
      " 12  q_avgtempF    8584 non-null   float64\n",
      " 13  q_avghumid    8584 non-null   float64\n",
      " 14  q_sumprecip   8584 non-null   float64\n",
      "dtypes: float64(13), object(2)\n",
      "memory usage: 1019.7+ KB\n"
     ]
    }
   ],
   "source": [
    "county_list = list(weather['county'].unique())\n",
    "\n",
    "county_dfs = []\n",
    "for county in county_list:\n",
    "    clear_output()\n",
    "    print(county)\n",
    "    x = weather[weather['county'] == county]\n",
    "    x.set_index('date', inplace=True)\n",
    "    county_dfs.append(x)\n",
    "\n",
    "iters = 0\n",
    "for df in county_dfs:\n",
    "    county = df['county'][0]\n",
    "    trans_df = df[['precip']].rolling(3).sum()\n",
    "    temp_df = df[['avgtempF', 'humid']].rolling(3).mean()\n",
    "    temp_df['county'] = county\n",
    "    temp_df['precip'] = trans_df['precip']\n",
    "    temp_df.reset_index(inplace = True)\n",
    "    if iters == 0:\n",
    "        out = temp_df.copy()\n",
    "    else:\n",
    "        out = pd.concat([out, temp_df], ignore_index = True)\n",
    "    iters += 1\n",
    "    clear_output()\n",
    "    print(iters)\n",
    "    \n",
    "\n",
    "\n",
    "out['date'] = out['date'].astype(str)\n",
    "out['date'] = out['date'].map(lambda s: s[0:-3])\n",
    "\n",
    "weather['date'] = weather['date'].astype(str)\n",
    "weather['date'] = weather['date'].map(lambda s: s[0:-3])\n",
    "\n",
    "weather['q_avgtempF'] = out['avgtempF']\n",
    "weather['q_avghumid'] = out['humid']\n",
    "weather['q_sumprecip'] = out['precip']\n",
    "\n",
    "\n",
    "weather.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d95609fd-397a-400c-8230-a1ee574cfc36",
   "metadata": {},
   "source": [
    "Next we need to merge the fire data to the completed weather data.  Remember from before that the ```UNIT_ID``` feature from the fire data contains some cases where one unit id corresponded to two counties.  To correct for this we assign those fires to only one of the counties so that we can complete the merge."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ed52da71-8975-4be7-b6b1-742b2ac146b0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>county</th>\n",
       "      <th>maxtempF</th>\n",
       "      <th>mintempF</th>\n",
       "      <th>avgtempF</th>\n",
       "      <th>totalSnow_cm</th>\n",
       "      <th>humid</th>\n",
       "      <th>wind</th>\n",
       "      <th>precip</th>\n",
       "      <th>sunHour</th>\n",
       "      <th>...</th>\n",
       "      <th>q_sumprecip</th>\n",
       "      <th>UNIT_ID</th>\n",
       "      <th>FIRE_NAME</th>\n",
       "      <th>ALARM_DATE</th>\n",
       "      <th>CONT_DATE</th>\n",
       "      <th>CAUSE</th>\n",
       "      <th>REPORT_AC</th>\n",
       "      <th>GIS_ACRES</th>\n",
       "      <th>SHAPE_Length</th>\n",
       "      <th>SHAPE_Area</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2008-07</td>\n",
       "      <td>Sierra County</td>\n",
       "      <td>86.290323</td>\n",
       "      <td>46.645161</td>\n",
       "      <td>76.709677</td>\n",
       "      <td>0.0</td>\n",
       "      <td>32.709677</td>\n",
       "      <td>5.451613</td>\n",
       "      <td>0.0</td>\n",
       "      <td>14.396774</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2008-07</td>\n",
       "      <td>Sacramento County</td>\n",
       "      <td>97.290323</td>\n",
       "      <td>63.419355</td>\n",
       "      <td>86.516129</td>\n",
       "      <td>0.0</td>\n",
       "      <td>39.838710</td>\n",
       "      <td>4.741935</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13.741935</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2008-07</td>\n",
       "      <td>Santa Barbara County</td>\n",
       "      <td>89.129032</td>\n",
       "      <td>59.709677</td>\n",
       "      <td>80.548387</td>\n",
       "      <td>0.0</td>\n",
       "      <td>41.451613</td>\n",
       "      <td>7.354839</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13.164516</td>\n",
       "      <td>...</td>\n",
       "      <td>0.010000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2008-07</td>\n",
       "      <td>Calaveras County</td>\n",
       "      <td>96.419355</td>\n",
       "      <td>51.290323</td>\n",
       "      <td>87.032258</td>\n",
       "      <td>0.0</td>\n",
       "      <td>33.580645</td>\n",
       "      <td>5.387097</td>\n",
       "      <td>0.0</td>\n",
       "      <td>14.022581</td>\n",
       "      <td>...</td>\n",
       "      <td>0.068065</td>\n",
       "      <td>Calaveras County</td>\n",
       "      <td>SERPENTINE</td>\n",
       "      <td>2008-07</td>\n",
       "      <td>2008/07/29 00:00:00+00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>177.0</td>\n",
       "      <td>176.763400</td>\n",
       "      <td>7368.548447</td>\n",
       "      <td>1.146732e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2008-07</td>\n",
       "      <td>Calaveras County</td>\n",
       "      <td>96.419355</td>\n",
       "      <td>51.290323</td>\n",
       "      <td>87.032258</td>\n",
       "      <td>0.0</td>\n",
       "      <td>33.580645</td>\n",
       "      <td>5.387097</td>\n",
       "      <td>0.0</td>\n",
       "      <td>14.022581</td>\n",
       "      <td>...</td>\n",
       "      <td>0.068065</td>\n",
       "      <td>Calaveras County</td>\n",
       "      <td>POOL 2 - MAIN</td>\n",
       "      <td>2008-07</td>\n",
       "      <td>2008/07/29 00:00:00+00</td>\n",
       "      <td>10.0</td>\n",
       "      <td>73.5</td>\n",
       "      <td>73.444778</td>\n",
       "      <td>3200.154767</td>\n",
       "      <td>4.800265e+05</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 24 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      date                county   maxtempF   mintempF   avgtempF  \\\n",
       "0  2008-07         Sierra County  86.290323  46.645161  76.709677   \n",
       "1  2008-07     Sacramento County  97.290323  63.419355  86.516129   \n",
       "2  2008-07  Santa Barbara County  89.129032  59.709677  80.548387   \n",
       "3  2008-07      Calaveras County  96.419355  51.290323  87.032258   \n",
       "4  2008-07      Calaveras County  96.419355  51.290323  87.032258   \n",
       "\n",
       "   totalSnow_cm      humid      wind  precip    sunHour  ...  q_sumprecip  \\\n",
       "0           0.0  32.709677  5.451613     0.0  14.396774  ...          NaN   \n",
       "1           0.0  39.838710  4.741935     0.0  13.741935  ...          NaN   \n",
       "2           0.0  41.451613  7.354839     0.0  13.164516  ...     0.010000   \n",
       "3           0.0  33.580645  5.387097     0.0  14.022581  ...     0.068065   \n",
       "4           0.0  33.580645  5.387097     0.0  14.022581  ...     0.068065   \n",
       "\n",
       "            UNIT_ID       FIRE_NAME  ALARM_DATE               CONT_DATE CAUSE  \\\n",
       "0               NaN             NaN         NaN                     NaN   NaN   \n",
       "1               NaN             NaN         NaN                     NaN   NaN   \n",
       "2               NaN             NaN         NaN                     NaN   NaN   \n",
       "3  Calaveras County      SERPENTINE     2008-07  2008/07/29 00:00:00+00   1.0   \n",
       "4  Calaveras County  POOL 2 - MAIN      2008-07  2008/07/29 00:00:00+00  10.0   \n",
       "\n",
       "  REPORT_AC   GIS_ACRES SHAPE_Length    SHAPE_Area  \n",
       "0       NaN         NaN          NaN           NaN  \n",
       "1       NaN         NaN          NaN           NaN  \n",
       "2       NaN         NaN          NaN           NaN  \n",
       "3     177.0  176.763400  7368.548447  1.146732e+06  \n",
       "4      73.5   73.444778  3200.154767  4.800265e+05  \n",
       "\n",
       "[5 rows x 24 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fire = pd.read_csv('./data/fire_data.csv')\n",
    "\n",
    "def remove_dash(x):\n",
    "    if '-' in x:\n",
    "        county = x.split('- ')\n",
    "        return county[1]\n",
    "    else:\n",
    "        return x\n",
    "        \n",
    "fire['UNIT_ID'] = fire['UNIT_ID'].apply(remove_dash)\n",
    "final = pd.merge(weather, fire, how = 'left', left_on = ['county', 'date'], right_on = ['UNIT_ID', 'ALARM_DATE'])\n",
    "final.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f025010d-88b3-4b23-a60a-61e008caa30e",
   "metadata": {},
   "source": [
    "Next we drop columns that will not be used for modeling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3f6bdff4-44a3-4e7a-a8e4-5e7a047cf4a4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>county</th>\n",
       "      <th>maxtempF</th>\n",
       "      <th>mintempF</th>\n",
       "      <th>avgtempF</th>\n",
       "      <th>totalSnow_cm</th>\n",
       "      <th>humid</th>\n",
       "      <th>wind</th>\n",
       "      <th>precip</th>\n",
       "      <th>sunHour</th>\n",
       "      <th>lat</th>\n",
       "      <th>long</th>\n",
       "      <th>q_avgtempF</th>\n",
       "      <th>q_avghumid</th>\n",
       "      <th>q_sumprecip</th>\n",
       "      <th>FIRE_NAME</th>\n",
       "      <th>CAUSE</th>\n",
       "      <th>GIS_ACRES</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2008-07</td>\n",
       "      <td>Sierra County</td>\n",
       "      <td>86.290323</td>\n",
       "      <td>46.645161</td>\n",
       "      <td>76.709677</td>\n",
       "      <td>0.0</td>\n",
       "      <td>32.709677</td>\n",
       "      <td>5.451613</td>\n",
       "      <td>0.0</td>\n",
       "      <td>14.396774</td>\n",
       "      <td>39.58</td>\n",
       "      <td>-120.52</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2008-07</td>\n",
       "      <td>Sacramento County</td>\n",
       "      <td>97.290323</td>\n",
       "      <td>63.419355</td>\n",
       "      <td>86.516129</td>\n",
       "      <td>0.0</td>\n",
       "      <td>39.838710</td>\n",
       "      <td>4.741935</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13.741935</td>\n",
       "      <td>38.45</td>\n",
       "      <td>-121.34</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2008-07</td>\n",
       "      <td>Santa Barbara County</td>\n",
       "      <td>89.129032</td>\n",
       "      <td>59.709677</td>\n",
       "      <td>80.548387</td>\n",
       "      <td>0.0</td>\n",
       "      <td>41.451613</td>\n",
       "      <td>7.354839</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13.164516</td>\n",
       "      <td>34.54</td>\n",
       "      <td>-120.04</td>\n",
       "      <td>72.937634</td>\n",
       "      <td>34.079570</td>\n",
       "      <td>0.010000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2008-07</td>\n",
       "      <td>Calaveras County</td>\n",
       "      <td>96.419355</td>\n",
       "      <td>51.290323</td>\n",
       "      <td>87.032258</td>\n",
       "      <td>0.0</td>\n",
       "      <td>33.580645</td>\n",
       "      <td>5.387097</td>\n",
       "      <td>0.0</td>\n",
       "      <td>14.022581</td>\n",
       "      <td>38.18</td>\n",
       "      <td>-120.56</td>\n",
       "      <td>65.217204</td>\n",
       "      <td>38.434409</td>\n",
       "      <td>0.068065</td>\n",
       "      <td>SERPENTINE</td>\n",
       "      <td>1.0</td>\n",
       "      <td>176.763400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2008-07</td>\n",
       "      <td>Calaveras County</td>\n",
       "      <td>96.419355</td>\n",
       "      <td>51.290323</td>\n",
       "      <td>87.032258</td>\n",
       "      <td>0.0</td>\n",
       "      <td>33.580645</td>\n",
       "      <td>5.387097</td>\n",
       "      <td>0.0</td>\n",
       "      <td>14.022581</td>\n",
       "      <td>38.18</td>\n",
       "      <td>-120.56</td>\n",
       "      <td>65.217204</td>\n",
       "      <td>38.434409</td>\n",
       "      <td>0.068065</td>\n",
       "      <td>POOL 2 - MAIN</td>\n",
       "      <td>10.0</td>\n",
       "      <td>73.444778</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      date                county   maxtempF   mintempF   avgtempF  \\\n",
       "0  2008-07         Sierra County  86.290323  46.645161  76.709677   \n",
       "1  2008-07     Sacramento County  97.290323  63.419355  86.516129   \n",
       "2  2008-07  Santa Barbara County  89.129032  59.709677  80.548387   \n",
       "3  2008-07      Calaveras County  96.419355  51.290323  87.032258   \n",
       "4  2008-07      Calaveras County  96.419355  51.290323  87.032258   \n",
       "\n",
       "   totalSnow_cm      humid      wind  precip    sunHour    lat    long  \\\n",
       "0           0.0  32.709677  5.451613     0.0  14.396774  39.58 -120.52   \n",
       "1           0.0  39.838710  4.741935     0.0  13.741935  38.45 -121.34   \n",
       "2           0.0  41.451613  7.354839     0.0  13.164516  34.54 -120.04   \n",
       "3           0.0  33.580645  5.387097     0.0  14.022581  38.18 -120.56   \n",
       "4           0.0  33.580645  5.387097     0.0  14.022581  38.18 -120.56   \n",
       "\n",
       "   q_avgtempF  q_avghumid  q_sumprecip       FIRE_NAME  CAUSE   GIS_ACRES  \n",
       "0         NaN         NaN          NaN             NaN    NaN         NaN  \n",
       "1         NaN         NaN          NaN             NaN    NaN         NaN  \n",
       "2   72.937634   34.079570     0.010000             NaN    NaN         NaN  \n",
       "3   65.217204   38.434409     0.068065      SERPENTINE    1.0  176.763400  \n",
       "4   65.217204   38.434409     0.068065  POOL 2 - MAIN    10.0   73.444778  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final.drop(columns = ['UNIT_ID', 'ALARM_DATE', 'CONT_DATE', 'SHAPE_Length', 'SHAPE_Area', 'REPORT_AC'], inplace = True)\n",
    "\n",
    "final.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b85b1cfe-475f-4ca4-ad32-4a574c137a85",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 11149 entries, 0 to 11148\n",
      "Data columns (total 18 columns):\n",
      " #   Column        Non-Null Count  Dtype  \n",
      "---  ------        --------------  -----  \n",
      " 0   date          11149 non-null  object \n",
      " 1   county        11149 non-null  object \n",
      " 2   maxtempF      11149 non-null  float64\n",
      " 3   mintempF      11149 non-null  float64\n",
      " 4   avgtempF      11149 non-null  float64\n",
      " 5   totalSnow_cm  11149 non-null  float64\n",
      " 6   humid         11149 non-null  float64\n",
      " 7   wind          11149 non-null  float64\n",
      " 8   precip        11149 non-null  float64\n",
      " 9   sunHour       11149 non-null  float64\n",
      " 10  lat           11149 non-null  float64\n",
      " 11  long          11149 non-null  float64\n",
      " 12  q_avgtempF    10988 non-null  float64\n",
      " 13  q_avghumid    10988 non-null  float64\n",
      " 14  q_sumprecip   10988 non-null  float64\n",
      " 15  FIRE_NAME     4027 non-null   object \n",
      " 16  CAUSE         4015 non-null   float64\n",
      " 17  GIS_ACRES     4040 non-null   float64\n",
      "dtypes: float64(15), object(3)\n",
      "memory usage: 1.6+ MB\n"
     ]
    }
   ],
   "source": [
    "final.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "968709fc-6307-42ce-91fc-26d5441e7a68",
   "metadata": {},
   "source": [
    "There are some missing values that need to be imputed.  Some of these, such as where the ```GIS_ACRES``` is null we can just change that to 0 since the null represents that there was no fire.  For the missing causes we will assign a 0 if there was no fire, but will assign the most common cause for the instances where we do have a fire, but a null in the cause."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b4cb527e-c30f-403f-b56e-f314ee38f3ca",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11148 of 11148\n"
     ]
    }
   ],
   "source": [
    "final['FIRE_NAME'].fillna('no_fire', inplace = True)\n",
    "\n",
    "final['GIS_ACRES'].fillna(0, inplace = True)\n",
    "\n",
    "mode_cause = final['CAUSE'].value_counts().index[0]\n",
    "\n",
    "x , y = final.shape\n",
    "for i in range(x):\n",
    "    clear_output()\n",
    "    print(f\"{i} of {x-1}\")\n",
    "    if final.iloc[i,-1] != 0 and pd.isnull(final.iloc[i,-2]):\n",
    "        final.iloc[i,-2] = mode_cause\n",
    "\n",
    "final['CAUSE'].fillna(0, inplace = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97cb94ef-a197-4f99-97d0-c118aaf34c59",
   "metadata": {},
   "source": [
    "Since we used a 3 month average for some of the resampled fields there are two records for each county that contain null weather data corresponding to the first two months in our time interval.  These can be dropped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "24879c04-7b3b-4f16-a6b4-b67bacc0d00c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 10988 entries, 2 to 11148\n",
      "Data columns (total 18 columns):\n",
      " #   Column        Non-Null Count  Dtype  \n",
      "---  ------        --------------  -----  \n",
      " 0   date          10988 non-null  object \n",
      " 1   county        10988 non-null  object \n",
      " 2   maxtempF      10988 non-null  float64\n",
      " 3   mintempF      10988 non-null  float64\n",
      " 4   avgtempF      10988 non-null  float64\n",
      " 5   totalSnow_cm  10988 non-null  float64\n",
      " 6   humid         10988 non-null  float64\n",
      " 7   wind          10988 non-null  float64\n",
      " 8   precip        10988 non-null  float64\n",
      " 9   sunHour       10988 non-null  float64\n",
      " 10  lat           10988 non-null  float64\n",
      " 11  long          10988 non-null  float64\n",
      " 12  q_avgtempF    10988 non-null  float64\n",
      " 13  q_avghumid    10988 non-null  float64\n",
      " 14  q_sumprecip   10988 non-null  float64\n",
      " 15  FIRE_NAME     10988 non-null  object \n",
      " 16  CAUSE         10988 non-null  float64\n",
      " 17  GIS_ACRES     10988 non-null  float64\n",
      "dtypes: float64(15), object(3)\n",
      "memory usage: 1.6+ MB\n"
     ]
    }
   ],
   "source": [
    "final.dropna(inplace = True)\n",
    "\n",
    "final.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f36d6549-7c9f-4420-9617-dd646f963a47",
   "metadata": {},
   "source": [
    "All of our data was provided in imperial units with the exception of snow, which was given in centimeters.  These units are converted next."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1ec16d63-3097-4dcb-8972-92d704932c4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "final['totalSnow'] = final['totalSnow_cm'] / 2.54\n",
    "final.drop(columns = ['totalSnow_cm'], inplace = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0389f36f-a54e-47ad-84b7-91ecf03d19d5",
   "metadata": {},
   "source": [
    "Finally, the columns of our data frame are reordered to collect all of the weather data in contiguous columns simply for ease of readability.  The dataset is now complete and saved as ```combined.csv``` for use in eda and modeling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c8d932fd-b030-4e6e-8d4d-5899dc9cd1ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "final = final[['date', 'county', 'maxtempF', 'mintempF', 'avgtempF', 'totalSnow', 'humid', 'wind', 'precip', \n",
    "               'q_avgtempF', 'q_avghumid', 'q_sumprecip', 'sunHour', 'FIRE_NAME', 'CAUSE', 'lat', 'long', 'GIS_ACRES']]\n",
    "\n",
    "final.to_csv('./data/combined.csv', index = False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}