notes/sfpd-wordcloud.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Crime data visualization in San Francisco\n",
    "\n",
    "San Francisco has one of the most \"open data\" policies of any large city. In this lab, we are going to download about 85M of data (238,456) describing all police incidents since 2018 (I'm grabbing data on August 5, 2019).\n",
    "\n",
    "## Getting started\n",
    "\n",
    "Download [Police Department Incident Reports 2018 to present](https://data.sfgov.org/Public-Safety/Police-Department-Incident-Reports-2018-to-Present/wg3w-h783) or, if you want, all [San Francisco police department incident since 1 January 2003](https://data.sfgov.org/Public-Safety/SFPD-Incidents-from-1-January-2003/tmnf-yvry). \n",
    "\n",
    "*That turns out to be a slow link...try downloading from my S3 bucket*:\n",
    "\n",
    "`https://msan692.s3.us-west-1.amazonaws.com/SFPD.csv`\n",
    "\n",
    "Or, click the \"Export\" button and then save in \"CSV for Excel\" format. (It's fairly large at all 140MB so it could take a while if you have a slow connection.) You can name that data file whatever you want but I will call it `SFPD.csv` for these exercises and save it in `/tmp`.\n",
    "\n",
    "You can use the command line to directly access and download the link if you want:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100  173M    0  173M    0     0  5061k      0 --:--:--  0:00:35 --:--:-- 5212k-:-- 5238k     0 --:--:--  0:00:22 --:--:-- 5257k\n"
     ]
    }
   ],
   "source": [
    "! curl 'https://data.sfgov.org/api/views/wg3w-h783/rows.csv?accessType=DOWNLOAD&bom=true&format=true' > /tmp/SFPD.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can easily figure out how many records there are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  499423 /tmp/SFPD.csv\n"
     ]
    }
   ],
   "source": [
    "! wc -l /tmp/SFPD.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So that's currently about 500,000 records."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sniffing the data\n",
    "\n",
    "Let's assume the file you downloaded and is in `/tmp`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Incident Datetime</th>\n",
       "      <td>2019/03/05 09:00:00 PM</td>\n",
       "      <td>2018/12/22 12:00:00 PM</td>\n",
       "      <td>2018/08/24 05:30:00 PM</td>\n",
       "      <td>2018/10/20 03:10:00 PM</td>\n",
       "      <td>2019/06/05 12:36:00 AM</td>\n",
       "      <td>2019/02/12 07:00:00 AM</td>\n",
       "      <td>2018/12/07 12:30:00 PM</td>\n",
       "      <td>2018/10/04 09:00:00 AM</td>\n",
       "      <td>2019/05/17 05:00:00 PM</td>\n",
       "      <td>2019/01/31 01:24:00 PM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Incident Date</th>\n",
       "      <td>2019/03/05</td>\n",
       "      <td>2018/12/22</td>\n",
       "      <td>2018/08/24</td>\n",
       "      <td>2018/10/20</td>\n",
       "      <td>2019/06/05</td>\n",
       "      <td>2019/02/12</td>\n",
       "      <td>2018/12/07</td>\n",
       "      <td>2018/10/04</td>\n",
       "      <td>2019/05/17</td>\n",
       "      <td>2019/01/31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Incident Time</th>\n",
       "      <td>21:00</td>\n",
       "      <td>12:00</td>\n",
       "      <td>17:30</td>\n",
       "      <td>15:10</td>\n",
       "      <td>00:36</td>\n",
       "      <td>07:00</td>\n",
       "      <td>12:30</td>\n",
       "      <td>09:00</td>\n",
       "      <td>17:00</td>\n",
       "      <td>13:24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Incident Year</th>\n",
       "      <td>2019</td>\n",
       "      <td>2018</td>\n",
       "      <td>2018</td>\n",
       "      <td>2018</td>\n",
       "      <td>2019</td>\n",
       "      <td>2019</td>\n",
       "      <td>2018</td>\n",
       "      <td>2018</td>\n",
       "      <td>2019</td>\n",
       "      <td>2019</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Incident Day of Week</th>\n",
       "      <td>Tuesday</td>\n",
       "      <td>Saturday</td>\n",
       "      <td>Friday</td>\n",
       "      <td>Saturday</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>Tuesday</td>\n",
       "      <td>Friday</td>\n",
       "      <td>Thursday</td>\n",
       "      <td>Friday</td>\n",
       "      <td>Thursday</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Report Datetime</th>\n",
       "      <td>2019/03/07 12:54:00 PM</td>\n",
       "      <td>2018/12/26 02:29:00 PM</td>\n",
       "      <td>2018/08/24 07:35:00 PM</td>\n",
       "      <td>2018/10/20 03:10:00 PM</td>\n",
       "      <td>2019/06/05 12:37:00 AM</td>\n",
       "      <td>2019/02/12 09:20:00 AM</td>\n",
       "      <td>2018/12/07 12:35:00 PM</td>\n",
       "      <td>2018/10/04 09:00:00 AM</td>\n",
       "      <td>2019/05/18 01:49:00 PM</td>\n",
       "      <td>2019/01/31 01:24:00 PM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Row ID</th>\n",
       "      <td>77808706372</td>\n",
       "      <td>75246907023</td>\n",
       "      <td>70675306243</td>\n",
       "      <td>72826462071</td>\n",
       "      <td>80823030140</td>\n",
       "      <td>76962406304</td>\n",
       "      <td>74573526200</td>\n",
       "      <td>72557805053</td>\n",
       "      <td>80221615161</td>\n",
       "      <td>76530103034</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Incident ID</th>\n",
       "      <td>778087</td>\n",
       "      <td>752469</td>\n",
       "      <td>706753</td>\n",
       "      <td>728264</td>\n",
       "      <td>808230</td>\n",
       "      <td>769624</td>\n",
       "      <td>745735</td>\n",
       "      <td>725578</td>\n",
       "      <td>802216</td>\n",
       "      <td>765301</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Incident Number</th>\n",
       "      <td>196048512</td>\n",
       "      <td>180971777</td>\n",
       "      <td>180639339</td>\n",
       "      <td>180624631</td>\n",
       "      <td>190403077</td>\n",
       "      <td>190106328</td>\n",
       "      <td>180922946</td>\n",
       "      <td>180561958</td>\n",
       "      <td>190356272</td>\n",
       "      <td>190045641</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CAD Number</th>\n",
       "      <td>NaN</td>\n",
       "      <td>183601903.0</td>\n",
       "      <td>182363732.0</td>\n",
       "      <td>182933454.0</td>\n",
       "      <td>191560061.0</td>\n",
       "      <td>190430997.0</td>\n",
       "      <td>183411784.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>191381894.0</td>\n",
       "      <td>190312032.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Report Type Code</th>\n",
       "      <td>II</td>\n",
       "      <td>VI</td>\n",
       "      <td>II</td>\n",
       "      <td>IS</td>\n",
       "      <td>II</td>\n",
       "      <td>II</td>\n",
       "      <td>II</td>\n",
       "      <td>IS</td>\n",
       "      <td>II</td>\n",
       "      <td>IS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Report Type Description</th>\n",
       "      <td>Coplogic Initial</td>\n",
       "      <td>Vehicle Initial</td>\n",
       "      <td>Initial</td>\n",
       "      <td>Initial Supplement</td>\n",
       "      <td>Initial</td>\n",
       "      <td>Initial</td>\n",
       "      <td>Initial</td>\n",
       "      <td>Initial Supplement</td>\n",
       "      <td>Initial</td>\n",
       "      <td>Initial Supplement</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Filed Online</th>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Incident Code</th>\n",
       "      <td>6372</td>\n",
       "      <td>7023</td>\n",
       "      <td>6243</td>\n",
       "      <td>62071</td>\n",
       "      <td>30140</td>\n",
       "      <td>6304</td>\n",
       "      <td>26200</td>\n",
       "      <td>5053</td>\n",
       "      <td>15161</td>\n",
       "      <td>3034</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Incident Category</th>\n",
       "      <td>Larceny Theft</td>\n",
       "      <td>Motor Vehicle Theft</td>\n",
       "      <td>Larceny Theft</td>\n",
       "      <td>Warrant</td>\n",
       "      <td>Other Offenses</td>\n",
       "      <td>Larceny Theft</td>\n",
       "      <td>Other Miscellaneous</td>\n",
       "      <td>Burglary</td>\n",
       "      <td>Offences Against The Family And Children</td>\n",
       "      <td>Robbery</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Incident Subcategory</th>\n",
       "      <td>Larceny Theft - Other</td>\n",
       "      <td>Motor Vehicle Theft</td>\n",
       "      <td>Larceny - From Vehicle</td>\n",
       "      <td>Other</td>\n",
       "      <td>Other</td>\n",
       "      <td>Larceny Theft - From Building</td>\n",
       "      <td>Other</td>\n",
       "      <td>Burglary - Commercial</td>\n",
       "      <td>Other</td>\n",
       "      <td>Robbery - Commercial</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Incident Description</th>\n",
       "      <td>Theft, Other Property, $50-$200</td>\n",
       "      <td>Vehicle, Stolen, Motorcycle</td>\n",
       "      <td>Theft, From Locked Vehicle, $200-$950</td>\n",
       "      <td>Probation Search</td>\n",
       "      <td>Air Gun, Possession</td>\n",
       "      <td>Theft, From Building, &gt;$950</td>\n",
       "      <td>State Code Felony (general)</td>\n",
       "      <td>Burglary, Store, Unlawful Entry</td>\n",
       "      <td>Violation of Restraining Order</td>\n",
       "      <td>Robbery, Service Station, W/ Force</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Resolution</th>\n",
       "      <td>Open or Active</td>\n",
       "      <td>Open or Active</td>\n",
       "      <td>Open or Active</td>\n",
       "      <td>Cite or Arrest Adult</td>\n",
       "      <td>Cite or Arrest Adult</td>\n",
       "      <td>Open or Active</td>\n",
       "      <td>Open or Active</td>\n",
       "      <td>Cite or Arrest Adult</td>\n",
       "      <td>Open or Active</td>\n",
       "      <td>Cite or Arrest Adult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Intersection</th>\n",
       "      <td>UNITED NATIONS PLZ \\ LEAVENWORTH ST</td>\n",
       "      <td>LINDEN ST \\ BUCHANAN ST</td>\n",
       "      <td>PERU AVE \\ BURROWS ST</td>\n",
       "      <td>BOARDMAN PL \\ BRYANT ST</td>\n",
       "      <td>LOMBARD ST \\ NEWELL ST</td>\n",
       "      <td>NEW MONTGOMERY ST \\ JESSIE ST</td>\n",
       "      <td>03RD ST \\ LA SALLE AVE</td>\n",
       "      <td>SAINT GEORGE ALY \\ PINE ST</td>\n",
       "      <td>SHAFTER AVE \\ KEITH ST</td>\n",
       "      <td>PIERCE ST \\ POST ST</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CNN</th>\n",
       "      <td>30044000.0</td>\n",
       "      <td>25931000.0</td>\n",
       "      <td>21610000.0</td>\n",
       "      <td>23914000.0</td>\n",
       "      <td>30035000.0</td>\n",
       "      <td>24620000.0</td>\n",
       "      <td>20224000.0</td>\n",
       "      <td>24739000.0</td>\n",
       "      <td>20158000.0</td>\n",
       "      <td>26615000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Police District</th>\n",
       "      <td>Tenderloin</td>\n",
       "      <td>Northern</td>\n",
       "      <td>Ingleside</td>\n",
       "      <td>Southern</td>\n",
       "      <td>Central</td>\n",
       "      <td>Southern</td>\n",
       "      <td>Bayview</td>\n",
       "      <td>Central</td>\n",
       "      <td>Bayview</td>\n",
       "      <td>Northern</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Analysis Neighborhood</th>\n",
       "      <td>Tenderloin</td>\n",
       "      <td>Hayes Valley</td>\n",
       "      <td>Excelsior</td>\n",
       "      <td>South of Market</td>\n",
       "      <td>North Beach</td>\n",
       "      <td>Financial District/South Beach</td>\n",
       "      <td>Bayview Hunters Point</td>\n",
       "      <td>Financial District/South Beach</td>\n",
       "      <td>Bayview Hunters Point</td>\n",
       "      <td>Japantown</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Supervisor District</th>\n",
       "      <td>6.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Latitude</th>\n",
       "      <td>37.779992</td>\n",
       "      <td>37.775787</td>\n",
       "      <td>37.723944</td>\n",
       "      <td>37.775161</td>\n",
       "      <td>37.802755</td>\n",
       "      <td>37.78789</td>\n",
       "      <td>37.73717</td>\n",
       "      <td>37.791652</td>\n",
       "      <td>37.72998</td>\n",
       "      <td>37.784736</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Longitude</th>\n",
       "      <td>-122.413487</td>\n",
       "      <td>-122.427819</td>\n",
       "      <td>-122.42283</td>\n",
       "      <td>-122.403636</td>\n",
       "      <td>-122.413623</td>\n",
       "      <td>-122.401043</td>\n",
       "      <td>-122.389864</td>\n",
       "      <td>-122.404525</td>\n",
       "      <td>-122.388855</td>\n",
       "      <td>-122.436368</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Point</th>\n",
       "      <td>POINT (-122.41348740024354 37.77999173926721)</td>\n",
       "      <td>POINT (-122.42781910073226 37.77578657487239)</td>\n",
       "      <td>POINT (-122.42283006837721 37.72394419349765)</td>\n",
       "      <td>POINT (-122.40363551943442 37.7751608100771)</td>\n",
       "      <td>POINT (-122.4136227157606 37.80275505740129)</td>\n",
       "      <td>POINT (-122.40104304688674 37.787889612676835)</td>\n",
       "      <td>POINT (-122.38986446334408 37.73716953069777)</td>\n",
       "      <td>POINT (-122.40452450386199 37.79165205510238)</td>\n",
       "      <td>POINT (-122.38885511785887 37.72998041447379)</td>\n",
       "      <td>POINT (-122.43636836004865 37.78473606531529)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Neighborhoods</th>\n",
       "      <td>21.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>90.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>106.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>86.0</td>\n",
       "      <td>104.0</td>\n",
       "      <td>86.0</td>\n",
       "      <td>103.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ESNCAG - Boundary File</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Central Market/Tenderloin Boundary Polygon - Updated</th>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Civic Center Harm Reduction Project Boundary</th>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HSOC Zones as of 2018-06-05</th>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Invest In Neighborhoods (IIN) Areas</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Current Supervisor Districts</th>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Current Police Districts</th>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                0  \\\n",
       "Incident Datetime                                                          2019/03/05 09:00:00 PM   \n",
       "Incident Date                                                                          2019/03/05   \n",
       "Incident Time                                                                               21:00   \n",
       "Incident Year                                                                                2019   \n",
       "Incident Day of Week                                                                      Tuesday   \n",
       "Report Datetime                                                            2019/03/07 12:54:00 PM   \n",
       "Row ID                                                                                77808706372   \n",
       "Incident ID                                                                                778087   \n",
       "Incident Number                                                                         196048512   \n",
       "CAD Number                                                                                    NaN   \n",
       "Report Type Code                                                                               II   \n",
       "Report Type Description                                                          Coplogic Initial   \n",
       "Filed Online                                                                                 True   \n",
       "Incident Code                                                                                6372   \n",
       "Incident Category                                                                   Larceny Theft   \n",
       "Incident Subcategory                                                        Larceny Theft - Other   \n",
       "Incident Description                                              Theft, Other Property, $50-$200   \n",
       "Resolution                                                                         Open or Active   \n",
       "Intersection                                                  UNITED NATIONS PLZ \\ LEAVENWORTH ST   \n",
       "CNN                                                                                    30044000.0   \n",
       "Police District                                                                        Tenderloin   \n",
       "Analysis Neighborhood                                                                  Tenderloin   \n",
       "Supervisor District                                                                           6.0   \n",
       "Latitude                                                                                37.779992   \n",
       "Longitude                                                                             -122.413487   \n",
       "Point                                               POINT (-122.41348740024354 37.77999173926721)   \n",
       "Neighborhoods                                                                                21.0   \n",
       "ESNCAG - Boundary File                                                                        NaN   \n",
       "Central Market/Tenderloin Boundary Polygon - Up...                                            1.0   \n",
       "Civic Center Harm Reduction Project Boundary                                                  1.0   \n",
       "HSOC Zones as of 2018-06-05                                                                   1.0   \n",
       "Invest In Neighborhoods (IIN) Areas                                                           NaN   \n",
       "Current Supervisor Districts                                                                 10.0   \n",
       "Current Police Districts                                                                      5.0   \n",
       "\n",
       "                                                                                                1  \\\n",
       "Incident Datetime                                                          2018/12/22 12:00:00 PM   \n",
       "Incident Date                                                                          2018/12/22   \n",
       "Incident Time                                                                               12:00   \n",
       "Incident Year                                                                                2018   \n",
       "Incident Day of Week                                                                     Saturday   \n",
       "Report Datetime                                                            2018/12/26 02:29:00 PM   \n",
       "Row ID                                                                                75246907023   \n",
       "Incident ID                                                                                752469   \n",
       "Incident Number                                                                         180971777   \n",
       "CAD Number                                                                            183601903.0   \n",
       "Report Type Code                                                                               VI   \n",
       "Report Type Description                                                           Vehicle Initial   \n",
       "Filed Online                                                                                  NaN   \n",
       "Incident Code                                                                                7023   \n",
       "Incident Category                                                             Motor Vehicle Theft   \n",
       "Incident Subcategory                                                          Motor Vehicle Theft   \n",
       "Incident Description                                                  Vehicle, Stolen, Motorcycle   \n",
       "Resolution                                                                         Open or Active   \n",
       "Intersection                                                              LINDEN ST \\ BUCHANAN ST   \n",
       "CNN                                                                                    25931000.0   \n",
       "Police District                                                                          Northern   \n",
       "Analysis Neighborhood                                                                Hayes Valley   \n",
       "Supervisor District                                                                           5.0   \n",
       "Latitude                                                                                37.775787   \n",
       "Longitude                                                                             -122.427819   \n",
       "Point                                               POINT (-122.42781910073226 37.77578657487239)   \n",
       "Neighborhoods                                                                                22.0   \n",
       "ESNCAG - Boundary File                                                                        NaN   \n",
       "Central Market/Tenderloin Boundary Polygon - Up...                                            NaN   \n",
       "Civic Center Harm Reduction Project Boundary                                                  NaN   \n",
       "HSOC Zones as of 2018-06-05                                                                   NaN   \n",
       "Invest In Neighborhoods (IIN) Areas                                                           NaN   \n",
       "Current Supervisor Districts                                                                 11.0   \n",
       "Current Police Districts                                                                      4.0   \n",
       "\n",
       "                                                                                                2  \\\n",
       "Incident Datetime                                                          2018/08/24 05:30:00 PM   \n",
       "Incident Date                                                                          2018/08/24   \n",
       "Incident Time                                                                               17:30   \n",
       "Incident Year                                                                                2018   \n",
       "Incident Day of Week                                                                       Friday   \n",
       "Report Datetime                                                            2018/08/24 07:35:00 PM   \n",
       "Row ID                                                                                70675306243   \n",
       "Incident ID                                                                                706753   \n",
       "Incident Number                                                                         180639339   \n",
       "CAD Number                                                                            182363732.0   \n",
       "Report Type Code                                                                               II   \n",
       "Report Type Description                                                                   Initial   \n",
       "Filed Online                                                                                  NaN   \n",
       "Incident Code                                                                                6243   \n",
       "Incident Category                                                                   Larceny Theft   \n",
       "Incident Subcategory                                                       Larceny - From Vehicle   \n",
       "Incident Description                                        Theft, From Locked Vehicle, $200-$950   \n",
       "Resolution                                                                         Open or Active   \n",
       "Intersection                                                                PERU AVE \\ BURROWS ST   \n",
       "CNN                                                                                    21610000.0   \n",
       "Police District                                                                         Ingleside   \n",
       "Analysis Neighborhood                                                                   Excelsior   \n",
       "Supervisor District                                                                           9.0   \n",
       "Latitude                                                                                37.723944   \n",
       "Longitude                                                                              -122.42283   \n",
       "Point                                               POINT (-122.42283006837721 37.72394419349765)   \n",
       "Neighborhoods                                                                                90.0   \n",
       "ESNCAG - Boundary File                                                                        NaN   \n",
       "Central Market/Tenderloin Boundary Polygon - Up...                                            NaN   \n",
       "Civic Center Harm Reduction Project Boundary                                                  NaN   \n",
       "HSOC Zones as of 2018-06-05                                                                   NaN   \n",
       "Invest In Neighborhoods (IIN) Areas                                                           NaN   \n",
       "Current Supervisor Districts                                                                  2.0   \n",
       "Current Police Districts                                                                      9.0   \n",
       "\n",
       "                                                                                               3  \\\n",
       "Incident Datetime                                                         2018/10/20 03:10:00 PM   \n",
       "Incident Date                                                                         2018/10/20   \n",
       "Incident Time                                                                              15:10   \n",
       "Incident Year                                                                               2018   \n",
       "Incident Day of Week                                                                    Saturday   \n",
       "Report Datetime                                                           2018/10/20 03:10:00 PM   \n",
       "Row ID                                                                               72826462071   \n",
       "Incident ID                                                                               728264   \n",
       "Incident Number                                                                        180624631   \n",
       "CAD Number                                                                           182933454.0   \n",
       "Report Type Code                                                                              IS   \n",
       "Report Type Description                                                       Initial Supplement   \n",
       "Filed Online                                                                                 NaN   \n",
       "Incident Code                                                                              62071   \n",
       "Incident Category                                                                        Warrant   \n",
       "Incident Subcategory                                                                       Other   \n",
       "Incident Description                                                            Probation Search   \n",
       "Resolution                                                                  Cite or Arrest Adult   \n",
       "Intersection                                                             BOARDMAN PL \\ BRYANT ST   \n",
       "CNN                                                                                   23914000.0   \n",
       "Police District                                                                         Southern   \n",
       "Analysis Neighborhood                                                            South of Market   \n",
       "Supervisor District                                                                          6.0   \n",
       "Latitude                                                                               37.775161   \n",
       "Longitude                                                                            -122.403636   \n",
       "Point                                               POINT (-122.40363551943442 37.7751608100771)   \n",
       "Neighborhoods                                                                               32.0   \n",
       "ESNCAG - Boundary File                                                                       NaN   \n",
       "Central Market/Tenderloin Boundary Polygon - Up...                                           NaN   \n",
       "Civic Center Harm Reduction Project Boundary                                                 NaN   \n",
       "HSOC Zones as of 2018-06-05                                                                  NaN   \n",
       "Invest In Neighborhoods (IIN) Areas                                                          NaN   \n",
       "Current Supervisor Districts                                                                10.0   \n",
       "Current Police Districts                                                                     1.0   \n",
       "\n",
       "                                                                                               4  \\\n",
       "Incident Datetime                                                         2019/06/05 12:36:00 AM   \n",
       "Incident Date                                                                         2019/06/05   \n",
       "Incident Time                                                                              00:36   \n",
       "Incident Year                                                                               2019   \n",
       "Incident Day of Week                                                                   Wednesday   \n",
       "Report Datetime                                                           2019/06/05 12:37:00 AM   \n",
       "Row ID                                                                               80823030140   \n",
       "Incident ID                                                                               808230   \n",
       "Incident Number                                                                        190403077   \n",
       "CAD Number                                                                           191560061.0   \n",
       "Report Type Code                                                                              II   \n",
       "Report Type Description                                                                  Initial   \n",
       "Filed Online                                                                                 NaN   \n",
       "Incident Code                                                                              30140   \n",
       "Incident Category                                                                 Other Offenses   \n",
       "Incident Subcategory                                                                       Other   \n",
       "Incident Description                                                         Air Gun, Possession   \n",
       "Resolution                                                                  Cite or Arrest Adult   \n",
       "Intersection                                                              LOMBARD ST \\ NEWELL ST   \n",
       "CNN                                                                                   30035000.0   \n",
       "Police District                                                                          Central   \n",
       "Analysis Neighborhood                                                                North Beach   \n",
       "Supervisor District                                                                          3.0   \n",
       "Latitude                                                                               37.802755   \n",
       "Longitude                                                                            -122.413623   \n",
       "Point                                               POINT (-122.4136227157606 37.80275505740129)   \n",
       "Neighborhoods                                                                              106.0   \n",
       "ESNCAG - Boundary File                                                                       NaN   \n",
       "Central Market/Tenderloin Boundary Polygon - Up...                                           NaN   \n",
       "Civic Center Harm Reduction Project Boundary                                                 NaN   \n",
       "HSOC Zones as of 2018-06-05                                                                  NaN   \n",
       "Invest In Neighborhoods (IIN) Areas                                                          NaN   \n",
       "Current Supervisor Districts                                                                 3.0   \n",
       "Current Police Districts                                                                     6.0   \n",
       "\n",
       "                                                                                                 5  \\\n",
       "Incident Datetime                                                           2019/02/12 07:00:00 AM   \n",
       "Incident Date                                                                           2019/02/12   \n",
       "Incident Time                                                                                07:00   \n",
       "Incident Year                                                                                 2019   \n",
       "Incident Day of Week                                                                       Tuesday   \n",
       "Report Datetime                                                             2019/02/12 09:20:00 AM   \n",
       "Row ID                                                                                 76962406304   \n",
       "Incident ID                                                                                 769624   \n",
       "Incident Number                                                                          190106328   \n",
       "CAD Number                                                                             190430997.0   \n",
       "Report Type Code                                                                                II   \n",
       "Report Type Description                                                                    Initial   \n",
       "Filed Online                                                                                   NaN   \n",
       "Incident Code                                                                                 6304   \n",
       "Incident Category                                                                    Larceny Theft   \n",
       "Incident Subcategory                                                 Larceny Theft - From Building   \n",
       "Incident Description                                                   Theft, From Building, >$950   \n",
       "Resolution                                                                          Open or Active   \n",
       "Intersection                                                         NEW MONTGOMERY ST \\ JESSIE ST   \n",
       "CNN                                                                                     24620000.0   \n",
       "Police District                                                                           Southern   \n",
       "Analysis Neighborhood                                               Financial District/South Beach   \n",
       "Supervisor District                                                                            6.0   \n",
       "Latitude                                                                                  37.78789   \n",
       "Longitude                                                                              -122.401043   \n",
       "Point                                               POINT (-122.40104304688674 37.787889612676835)   \n",
       "Neighborhoods                                                                                  NaN   \n",
       "ESNCAG - Boundary File                                                                         NaN   \n",
       "Central Market/Tenderloin Boundary Polygon - Up...                                             NaN   \n",
       "Civic Center Harm Reduction Project Boundary                                                   NaN   \n",
       "HSOC Zones as of 2018-06-05                                                                    NaN   \n",
       "Invest In Neighborhoods (IIN) Areas                                                            NaN   \n",
       "Current Supervisor Districts                                                                  10.0   \n",
       "Current Police Districts                                                                       6.0   \n",
       "\n",
       "                                                                                                6  \\\n",
       "Incident Datetime                                                          2018/12/07 12:30:00 PM   \n",
       "Incident Date                                                                          2018/12/07   \n",
       "Incident Time                                                                               12:30   \n",
       "Incident Year                                                                                2018   \n",
       "Incident Day of Week                                                                       Friday   \n",
       "Report Datetime                                                            2018/12/07 12:35:00 PM   \n",
       "Row ID                                                                                74573526200   \n",
       "Incident ID                                                                                745735   \n",
       "Incident Number                                                                         180922946   \n",
       "CAD Number                                                                            183411784.0   \n",
       "Report Type Code                                                                               II   \n",
       "Report Type Description                                                                   Initial   \n",
       "Filed Online                                                                                  NaN   \n",
       "Incident Code                                                                               26200   \n",
       "Incident Category                                                             Other Miscellaneous   \n",
       "Incident Subcategory                                                                        Other   \n",
       "Incident Description                                                  State Code Felony (general)   \n",
       "Resolution                                                                         Open or Active   \n",
       "Intersection                                                               03RD ST \\ LA SALLE AVE   \n",
       "CNN                                                                                    20224000.0   \n",
       "Police District                                                                           Bayview   \n",
       "Analysis Neighborhood                                                       Bayview Hunters Point   \n",
       "Supervisor District                                                                          10.0   \n",
       "Latitude                                                                                 37.73717   \n",
       "Longitude                                                                             -122.389864   \n",
       "Point                                               POINT (-122.38986446334408 37.73716953069777)   \n",
       "Neighborhoods                                                                                86.0   \n",
       "ESNCAG - Boundary File                                                                        NaN   \n",
       "Central Market/Tenderloin Boundary Polygon - Up...                                            NaN   \n",
       "Civic Center Harm Reduction Project Boundary                                                  NaN   \n",
       "HSOC Zones as of 2018-06-05                                                                   NaN   \n",
       "Invest In Neighborhoods (IIN) Areas                                                           NaN   \n",
       "Current Supervisor Districts                                                                  9.0   \n",
       "Current Police Districts                                                                      2.0   \n",
       "\n",
       "                                                                                                7  \\\n",
       "Incident Datetime                                                          2018/10/04 09:00:00 AM   \n",
       "Incident Date                                                                          2018/10/04   \n",
       "Incident Time                                                                               09:00   \n",
       "Incident Year                                                                                2018   \n",
       "Incident Day of Week                                                                     Thursday   \n",
       "Report Datetime                                                            2018/10/04 09:00:00 AM   \n",
       "Row ID                                                                                72557805053   \n",
       "Incident ID                                                                                725578   \n",
       "Incident Number                                                                         180561958   \n",
       "CAD Number                                                                                    NaN   \n",
       "Report Type Code                                                                               IS   \n",
       "Report Type Description                                                        Initial Supplement   \n",
       "Filed Online                                                                                  NaN   \n",
       "Incident Code                                                                                5053   \n",
       "Incident Category                                                                        Burglary   \n",
       "Incident Subcategory                                                        Burglary - Commercial   \n",
       "Incident Description                                              Burglary, Store, Unlawful Entry   \n",
       "Resolution                                                                   Cite or Arrest Adult   \n",
       "Intersection                                                           SAINT GEORGE ALY \\ PINE ST   \n",
       "CNN                                                                                    24739000.0   \n",
       "Police District                                                                           Central   \n",
       "Analysis Neighborhood                                              Financial District/South Beach   \n",
       "Supervisor District                                                                           3.0   \n",
       "Latitude                                                                                37.791652   \n",
       "Longitude                                                                             -122.404525   \n",
       "Point                                               POINT (-122.40452450386199 37.79165205510238)   \n",
       "Neighborhoods                                                                               104.0   \n",
       "ESNCAG - Boundary File                                                                        NaN   \n",
       "Central Market/Tenderloin Boundary Polygon - Up...                                            NaN   \n",
       "Civic Center Harm Reduction Project Boundary                                                  NaN   \n",
       "HSOC Zones as of 2018-06-05                                                                   NaN   \n",
       "Invest In Neighborhoods (IIN) Areas                                                           NaN   \n",
       "Current Supervisor Districts                                                                  3.0   \n",
       "Current Police Districts                                                                      6.0   \n",
       "\n",
       "                                                                                                8  \\\n",
       "Incident Datetime                                                          2019/05/17 05:00:00 PM   \n",
       "Incident Date                                                                          2019/05/17   \n",
       "Incident Time                                                                               17:00   \n",
       "Incident Year                                                                                2019   \n",
       "Incident Day of Week                                                                       Friday   \n",
       "Report Datetime                                                            2019/05/18 01:49:00 PM   \n",
       "Row ID                                                                                80221615161   \n",
       "Incident ID                                                                                802216   \n",
       "Incident Number                                                                         190356272   \n",
       "CAD Number                                                                            191381894.0   \n",
       "Report Type Code                                                                               II   \n",
       "Report Type Description                                                                   Initial   \n",
       "Filed Online                                                                                  NaN   \n",
       "Incident Code                                                                               15161   \n",
       "Incident Category                                        Offences Against The Family And Children   \n",
       "Incident Subcategory                                                                        Other   \n",
       "Incident Description                                               Violation of Restraining Order   \n",
       "Resolution                                                                         Open or Active   \n",
       "Intersection                                                               SHAFTER AVE \\ KEITH ST   \n",
       "CNN                                                                                    20158000.0   \n",
       "Police District                                                                           Bayview   \n",
       "Analysis Neighborhood                                                       Bayview Hunters Point   \n",
       "Supervisor District                                                                          10.0   \n",
       "Latitude                                                                                 37.72998   \n",
       "Longitude                                                                             -122.388855   \n",
       "Point                                               POINT (-122.38885511785887 37.72998041447379)   \n",
       "Neighborhoods                                                                                86.0   \n",
       "ESNCAG - Boundary File                                                                        NaN   \n",
       "Central Market/Tenderloin Boundary Polygon - Up...                                            NaN   \n",
       "Civic Center Harm Reduction Project Boundary                                                  NaN   \n",
       "HSOC Zones as of 2018-06-05                                                                   NaN   \n",
       "Invest In Neighborhoods (IIN) Areas                                                           NaN   \n",
       "Current Supervisor Districts                                                                  9.0   \n",
       "Current Police Districts                                                                      2.0   \n",
       "\n",
       "                                                                                                9  \n",
       "Incident Datetime                                                          2019/01/31 01:24:00 PM  \n",
       "Incident Date                                                                          2019/01/31  \n",
       "Incident Time                                                                               13:24  \n",
       "Incident Year                                                                                2019  \n",
       "Incident Day of Week                                                                     Thursday  \n",
       "Report Datetime                                                            2019/01/31 01:24:00 PM  \n",
       "Row ID                                                                                76530103034  \n",
       "Incident ID                                                                                765301  \n",
       "Incident Number                                                                         190045641  \n",
       "CAD Number                                                                            190312032.0  \n",
       "Report Type Code                                                                               IS  \n",
       "Report Type Description                                                        Initial Supplement  \n",
       "Filed Online                                                                                  NaN  \n",
       "Incident Code                                                                                3034  \n",
       "Incident Category                                                                         Robbery  \n",
       "Incident Subcategory                                                         Robbery - Commercial  \n",
       "Incident Description                                           Robbery, Service Station, W/ Force  \n",
       "Resolution                                                                   Cite or Arrest Adult  \n",
       "Intersection                                                                  PIERCE ST \\ POST ST  \n",
       "CNN                                                                                    26615000.0  \n",
       "Police District                                                                          Northern  \n",
       "Analysis Neighborhood                                                                   Japantown  \n",
       "Supervisor District                                                                           5.0  \n",
       "Latitude                                                                                37.784736  \n",
       "Longitude                                                                             -122.436368  \n",
       "Point                                               POINT (-122.43636836004865 37.78473606531529)  \n",
       "Neighborhoods                                                                               103.0  \n",
       "ESNCAG - Boundary File                                                                        NaN  \n",
       "Central Market/Tenderloin Boundary Polygon - Up...                                            NaN  \n",
       "Civic Center Harm Reduction Project Boundary                                                  NaN  \n",
       "HSOC Zones as of 2018-06-05                                                                   NaN  \n",
       "Invest In Neighborhoods (IIN) Areas                                                           NaN  \n",
       "Current Supervisor Districts                                                                 11.0  \n",
       "Current Police Districts                                                                      4.0  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df_sfpd = pd.read_csv('/tmp/SFPD.csv')\n",
    "df_sfpd.head(10).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get a better idea of what the data looks like, let's do a simple histogram of the categories and crime descriptions.  Here is the category histogram:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Arson', 'Lost Property', 'Larceny Theft', 'Suspicious Occ',\n",
       "       'Other Miscellaneous', 'Motor Vehicle Theft', 'Non-Criminal',\n",
       "       'Burglary', 'Malicious Mischief', 'Assault',\n",
       "       'Weapons Carrying Etc', 'Weapons Offense', 'Recovered Vehicle',\n",
       "       'Warrant', 'Fraud', 'Drug Offense',\n",
       "       'Offences Against The Family And Children', 'Disorderly Conduct',\n",
       "       'Other Offenses', 'Miscellaneous Investigation', 'Missing Person',\n",
       "       'Suspicious', 'Traffic Violation Arrest', 'Robbery', 'Other',\n",
       "       'Traffic Collision', 'Drug Violation', 'Stolen Property',\n",
       "       'Courtesy Report', 'Case Closure', 'Fire Report', nan, 'Vandalism',\n",
       "       'Forgery And Counterfeiting', 'Sex Offense', 'Vehicle Impounded',\n",
       "       'Vehicle Misplaced', 'Civil Sidewalks', 'Homicide', 'Suicide',\n",
       "       'Embezzlement', 'Rape', 'Prostitution',\n",
       "       'Human Trafficking (A), Commercial Sex Acts', 'Weapons Offence',\n",
       "       'Motor Vehicle Theft?', 'Gambling', 'Liquor Laws',\n",
       "       'Human Trafficking, Commercial Sex Acts',\n",
       "       'Human Trafficking (B), Involuntary Servitude'], dtype=object)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_sfpd['Incident Category'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Larceny Theft', 150235),\n",
       " ('Other Miscellaneous', 36629),\n",
       " ('Malicious Mischief', 33244),\n",
       " ('Non-Criminal', 30669),\n",
       " ('Assault', 29969),\n",
       " ('Burglary', 28358),\n",
       " ('Motor Vehicle Theft', 23464),\n",
       " ('Recovered Vehicle', 18221),\n",
       " ('Warrant', 16050),\n",
       " ('Lost Property', 15282)]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import Counter\n",
    "counter = Counter(df_sfpd['Incident Category'])\n",
    "counter.most_common(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Theft, From Locked Vehicle, >$950', 62097),\n",
       " ('Malicious Mischief, Vandalism to Property', 16362),\n",
       " ('Lost Property', 15282),\n",
       " ('Theft, Other Property, $50-$200', 14678),\n",
       " ('Battery', 14529),\n",
       " ('Vehicle, Recovered, Auto', 13279),\n",
       " ('Vehicle, Stolen, Auto', 12873),\n",
       " ('Mental Health Detention', 11969),\n",
       " ('Theft, Other Property, >$950', 11621),\n",
       " ('Suspicious Occurrence', 9069)]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import Counter\n",
    "counter = Counter(df_sfpd['Incident Description'])\n",
    "counter.most_common(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word clouds\n",
    "\n",
    "A more interesting way to visualize differences in term frequency is using a so-called word cloud.  For example, here is a word cloud showing the categories from 2003 to the present.\n",
    "\n",
    "<img src=\"figures/SFPD-wordcloud.png\" width=\"400\">\n",
    "\n",
    "Python has a nice library you can use:\n",
    "\n",
    "```bash\n",
    "$ pip install wordcloud\n",
    "```\n",
    "\n",
    "**Exercise**: In a file called `catcloud.py`, once again get the categories and then create a word cloud object and display it:\n",
    "\n",
    "```python\n",
    "from wordcloud import WordCloud\n",
    "import matplotlib.pyplot as plt\n",
    "from collections import Counter\n",
    "import pandas as pd\n",
    "import sys\n",
    "\n",
    "df_sfpd = pd.read_csv(sys.argv[1])\n",
    "\n",
    "... delete Incident Categories with nan ...\n",
    "categories = ... create Counter object on column 'Incident Category' ...\n",
    "\n",
    "wordcloud = WordCloud(width=1800,\n",
    "                      height=1400,\n",
    "                      max_words=10000,\n",
    "                      random_state=1,\n",
    "                      relative_scaling=0.25)\n",
    "\n",
    "wordcloud.fit_words(categories)\n",
    "\n",
    "plt.imshow(wordcloud)\n",
    "plt.axis(\"off\")\n",
    "plt.show()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Which neighborhood is the \"worst\"?\n",
    "\n",
    "**Exercise**: Now, pullout the neighborthood and do a word cloud on that in `hoodcloud.py` (it's ok to cut/paste):\n",
    "\n",
    "<img src=\"figures/SFPD-hood-wordcloud.png\" width=\"400\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Crimes per neighborhood\n",
    "\n",
    "\n",
    "**Exercise**: Filter the data using pandas from a particular precinct or neighborhood, such as Mission and South of Market.  Modify `catcloud.py` to use a pandas query to filter for those records.  Pass the hood as an argument (`sys.argv[2]`):\n",
    "\n",
    "```bash\n",
    "$ python catcloud.py /tmp/SFPD.csv Mission\n",
    "```\n",
    "\n",
    "Run the `catcloud.py` script to get an idea of the types of crimes per those two neighborhoods. Here are the mission and SOMA districts crime category clouds:\n",
    "\n",
    "<table>\n",
    "    <tr>\n",
    "        <td><b>Mission</b></td><td><b>South of Market</b></td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td><img src=\"figures/SFPD-mission-wordcloud.png\" width=\"300\"></td><td><img src=\"figures/SFPD-soma-wordcloud.png\" width=\"300\"></td>\n",
    "    </tr>\n",
    " </table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Which neighborhood has most car thefts?\n",
    "\n",
    "**Exercise**: Modify `hoodcloud.py` to filter for `Motor Vehicle Theft`. Pass the hood as an argument (`sys.argv[2]`):\n",
    "\n",
    "```bash\n",
    "$ python hoodcloud.py /tmp/SFPD.csv 'Motor Vehicle Theft'\n",
    "```\n",
    "\n",
    "<img src=\"figures/SFPD-car-theft-hood-wordcloud.png\" width=\"300\">\n",
    "\n",
    "Hmm..ok, so parking in the Presidio Heights is ok, but SOMA, BayView/Hunters point are bad news.\n",
    "\n",
    "If you get stuck in any of these exercises, you can look at the [code associated with this notes](https://github.com/parrt/msds692/tree/master/notes/code/sfpd)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}