04_01_cultivating_concentration_camp_dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <center>Cultivating Good Datasets for Entities</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>Dr. W.J.B. Mattingly</center>\n",
    "\n",
    "<center>Smithsonian Data Science Lab and United States Holocaust Memorial Museum</center>\n",
    "\n",
    "<center>January 2021</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Key Concepts in this Notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1) Where to find good datasets<br>\n",
    "2) How to clean datasets<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The previous notebooks have covered the basics of named entity recognition and provided lessons on how to work with spaCy. With all of this in place, it is time to shift this series towards gathering and cleaning the data. This notebook, 09.01, will be on acquiring the requisite data for named entities relevant to the Holocaust. In Notebook 09.02 will be dedicated to cultivating a corpus to automate an the creation of a training set via the spaCy EntityRuler.\n",
    "\n",
    "In this notebook we will cover where to find data for good datasets. How to clean that data. And how to run some preliminary tests to make sure you have optimized the data for the purpose of creating a rules-based NER via spaCy's EntityRuler. This will allow you to have a good rules-based NER that can either be used on its own or leveraged to cultivate a good training set for a machine learning model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Acquiring the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the first questions you will ask yourself in this pursuit, is \"where can I acquire lists?\" The answer is, unfortunately, \"it depends.\" Sometimes good datasets exist. There are a few good places to look, such as GitHub, Wikipedia, and academic digital projects. For each project, you must don your detective goggles and explore the web to find these datasets. Most times, it will take a bit of work (and some Python code) to get the data into a structured form. In this notebook, we will try to cultivate a good dataset for concentration camps.\n",
    "\n",
    "If we are looking to generate entities for concentration camps, we have a wealth of data, but this data is not necessary cleaned or structured. Let's examine three different locations where we can collate a list of concentration camps from the web and the strengths and weaknesses of those sources."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wikipedia"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Chances are, you will often begin your search by simply typing into Google, \"List of [X]\". If you do this with concentration camp, then you will likely first encounter Wikipedia (https://en.wikipedia.org/wiki/List_of_Nazi_concentration_camps).\n",
    "\n",
    "This page is particularly useful for a few reasons. First, you have a list of the major camps, such as Auschwitz and Dachau. More importantly, however, you also have access to linked pages of subcamps, i.e. for Auchwitz, one can acess this page: https://en.wikipedia.org/wiki/List_of_subcamps_of_Auschwitz. With a little bit of Python, requests, and BeautifulSoup, we can easily compile a quick list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Harmense', 'Budy', 'Babitz', 'Birkenau', 'Raisko', 'Plawy', 'Golleschau', 'Jawischowitz', 'Chelmek', 'Monowitz Buna-Werke [6]', 'Eintrachthütte', 'Neu-Dachs', 'Fürstengrube', 'Janinagrube', 'Lagischa', 'Günthergrube', 'Gleiwitz I', 'Laurahütte', 'Blechhammer', 'Bobrek', 'Gleiwitz II', 'Sosnowitz II', 'Gleiwitz III', 'Hindenburg', 'Trzebinia', 'Tschechowitz I Bombensucherkommando[9]', 'Althammer', 'Bismarckhütte', 'Charlottengrube', 'Neustadt', 'Tschechowitz II Vacuum', 'Hubertshütte', 'Freudenthal', 'Lichtewerden', 'Sosnitz', 'Porombka', 'Altdorf', 'Radostowitz', 'Kobier', 'Brünn', 'Sosnowitz', 'Gleiwitz IV', 'Kattowitz', 'Bauzug']\n",
      "\n",
      "['Aachen', 'Berka', 'Berka', 'Stadtallendorf', 'Altenburg', 'Bad Arolsen', 'Aschersleben', 'Augustdorf', 'Bad Berka', 'Bad Gandersheim', 'Bad Salzungen', 'Bad Salzungen', 'Bergisch Gladbach', 'Berga', 'Berlstedt', 'Bernburg', 'Billroda', 'Blankenhain', 'Bochum', 'Bochum', 'Bochum', 'Böhlen', 'Braunschweig', 'Colditz', 'Crawinkel', 'Dessau', 'Dessau', 'Dortmund', 'Dortmund', 'Duisburg', 'Düsseldorf', 'Düsseldorf', 'Düsseldorf', 'Düsseldorf', 'Eisenach', 'Elsnig', 'Ohrdruf', 'Essen', 'Essen', 'Frohburg', 'Gelsenkirchen', 'Giessen', 'Goslar', 'Göttingen', 'Hadmersleben', 'Halberstadt', 'Halberstadt', 'Halberstadt', 'Halberstadt', 'Halle', 'Hessisch Lichtenau', 'Holzen', 'Jena', 'Kassel', 'Kassel', 'Cologne', 'Cologne', 'Cologne', 'Cologne', 'Kranichfeld', 'Bad Langensalza', 'Leipzig', 'Leipzig', 'Leipzig', 'Leipzig', 'Staßfurt', 'Prettin', 'Lippstadt', 'Lippstadt', 'near Mücheln', 'Magdeburg', 'Markkleeberg', 'Meuselwitz', 'Nordhausen', 'Mühlhausen', 'Mühlhausen', 'Niederorschel', 'Nordhausen', 'Kraftsdorf', 'Ohrdruf', 'Penig', 'Raguhn', 'Rothenburg', 'near Wurzbach', 'Schlieben', 'Schönebeck', 'Schönebeck', 'Schwerte', 'Sömmerda', 'Sonneberg', 'Staßfurt', 'Suhl', 'Bad Berka', 'Taucha', 'Tonndorf', 'Torgau', 'Elsteraue', 'Unna', 'Usingen', 'Weferlingen', 'Weimar', 'Wernigerode', 'Westeregeln', 'Witten-Annen', 'Wolfen']\n"
     ]
    }
   ],
   "source": [
    "#Code for gathering subcamps\n",
    "\n",
    "#import web scraping libraries\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "def grab_subcamps(url, start_row, cell, t_class=False):\n",
    "    '''\n",
    "    This function will grab table data from Wikipedia.\n",
    "    It allows you to grab specific rows and specific cells of data to cultivate lists of entities\n",
    "    \n",
    "    url       >> the Wikipedia url for grabbing data\n",
    "    start_row >> Where the data starts in the table\n",
    "    cell      >> The cell that you desire to grab from the table\n",
    "    t_class   >> Some Wikipedia tables have a specific class of table that you need to grab, i.e. \"wikipedia sortable\".\n",
    "    '''\n",
    "    #url of the page\n",
    "    \n",
    "    #create the html object\n",
    "    s = requests.get(url).content\n",
    "\n",
    "    #set up the soup so that the html object can be parsed\n",
    "    soup = BeautifulSoup(s)\n",
    "\n",
    "    #grab the first table\n",
    "    if t_class == True:\n",
    "        table = soup.find(\"table\", {\"class\": \"wikitable sortable\"})\n",
    "    else:\n",
    "         table = soup.find(\"table\")\n",
    "\n",
    "    #create a blank list to append to for gathering camps\n",
    "    camps = []\n",
    "\n",
    "    #iterate over rows in table, beginning with row 3\n",
    "    for row in table.find_all(\"tr\")[start_row:]:\n",
    "        #one of the rows has only one cell, so we set up an index exception\n",
    "        try:\n",
    "            #grabs the 2nd cell in each row and cleans the data, splits off the cases of parentheses\n",
    "            #and grabs the first index from that split list\n",
    "            camp = row.find_all(\"td\")[cell].text.strip().split(\"(\")[0].split(\"/\")[0].strip()\n",
    "            camps.append(camp)\n",
    "        except:\n",
    "            IndexError\n",
    "    return (camps)\n",
    "        \n",
    "#print off the subcamps of Auschwitz\n",
    "ausch_subcamps = grab_subcamps(\"https://en.wikipedia.org/wiki/List_of_subcamps_of_Auschwitz\", 2, 1)\n",
    "buch_subcamps = grab_subcamps(\"https://en.wikipedia.org/wiki/List_of_subcamps_of_Buchenwald\", 1, 1, t_class=True)\n",
    "\n",
    "print (ausch_subcamps)\n",
    "print (\"\")\n",
    "print (buch_subcamps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above list is a cleaned list of the subcamps of Auschwitz and Buchenwald. We gathered it and cleaned it with minimal code from our first Google search. The benefits of acquring the data in this manner is that we have some sense of structure as well. We know that these are the subcamps of Auschwitz, meaning we can link them in our data structure to their corresponding main camp.\n",
    "\n",
    "This data does, however, come at a cost. It is data gathered from Wikipedia. In order to ensure that this data is proper, a content expert should be consulted. If a content expert is not a hand, I would recommend acquiring data from more reputable sites, such as nationally-funded museums or academic institutions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## United States Holocaust Memorial Museum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One such institution is the United States Holocaust Memorial Museum (USHMM), located in Washington, D.C. in the United States. When searching the USHMM collections, one way to limit your search is by Key Camps (https://www.ushmm.org/)\n",
    "\n",
    "Which looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Alderney', 'Amersfoort', 'Auschwitz', 'Banjica', 'Bełżec', 'Bergen-Belsen,', 'Bernburg', 'Bogdanovka', 'Bolzano', 'Bor', 'Breendonk', 'Breitenau', 'Buchenwald,', 'Chełmno', 'Dachau', 'Drancy', 'Falstad', 'Flossenbürg', 'Fort VII', 'Fossoli', 'Grini', 'Gross-Rosen', 'Herzogenbusch', 'Hinzert', 'Janowska', 'Jasenovac', 'Kaiserwald', 'Kaunas', 'Kemna', 'Klooga', 'Le Vernet', 'Majdanek', 'Malchow', 'Maly Trostenets', 'Mechelen', 'Mittelbau-Dora', 'Natzweiler-Struthof', 'Neuengamme', 'Niederhagen', 'Oberer Kuhberg', 'Oranienburg', 'Osthofen', 'Płaszów', 'Ravensbruck', 'Risiera di San Sabba', 'Sachsenhausen', 'Sajmište', 'Salaspils', 'Sobibór', 'Soldau', 'Stutthof', 'Theresienstadt,', 'Trawniki', 'Treblinka', 'Vaivara']\n"
     ]
    }
   ],
   "source": [
    "ushmm_camps = ['Alderney', 'Amersfoort', 'Auschwitz', 'Banjica', 'Bełżec', 'Bergen-Belsen,', 'Bernburg', 'Bogdanovka', 'Bolzano', 'Bor', 'Breendonk',\n",
    "         'Breitenau', 'Buchenwald,', 'Chełmno', 'Dachau', 'Drancy', 'Falstad', 'Flossenbürg', 'Fort VII', 'Fossoli', 'Grini', 'Gross-Rosen',\n",
    "         'Herzogenbusch', 'Hinzert', 'Janowska', 'Jasenovac', 'Kaiserwald', 'Kaunas', 'Kemna', 'Klooga', 'Le Vernet', 'Majdanek', 'Malchow',\n",
    "         'Maly Trostenets', 'Mechelen', 'Mittelbau-Dora', 'Natzweiler-Struthof', 'Neuengamme', 'Niederhagen', 'Oberer Kuhberg', 'Oranienburg',\n",
    "         'Osthofen', 'Płaszów', 'Ravensbruck', 'Risiera di San Sabba', 'Sachsenhausen', 'Sajmište', 'Salaspils', 'Sobibór', 'Soldau', 'Stutthof',\n",
    "         'Theresienstadt,', 'Trawniki', 'Treblinka', 'Vaivara']\n",
    "print (ushmm_camps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While this dataset is cleaned and good, it has certain limitations. First, it is not complete. This is a list of *key* camps, not all camps. Note that subcamps are left off the list. The second problem we have is that these camps of certain characters in their names that reflect the accent marks or letters that are not in the English alphabet. Some Holocaust texts, however, use only English letters and characters. Therefore, searches for certain words, such as Płaszów will not return a hit in a search for Plaszow. It is important, therefore, to make sure both forms of the word are represented in a rules-based search.\n",
    "\n",
    "With the function below, we can produce copies of each word with and without these characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Alderney', 'Amersfoort', 'Auschwitz', 'Banjica', 'Belzec', 'Bergen-Belsen,', 'Bernburg', 'Bełżec', 'Bogdanovka', 'Bolzano', 'Bor', 'Breendonk', 'Breitenau', 'Buchenwald,', 'Chelmno', 'Chełmno', 'Dachau', 'Drancy', 'Falstad', 'Flossenburg', 'Flossenbürg', 'Fort VII', 'Fossoli', 'Grini', 'Gross-Rosen', 'Herzogenbusch', 'Hinzert', 'Janowska', 'Jasenovac', 'Kaiserwald', 'Kaunas', 'Kemna', 'Klooga', 'Le Vernet', 'Majdanek', 'Malchow', 'Maly Trostenets', 'Mechelen', 'Mittelbau-Dora', 'Natzweiler-Struthof', 'Neuengamme', 'Niederhagen', 'Oberer Kuhberg', 'Oranienburg', 'Osthofen', 'Plaszow', 'Płaszów', 'Ravensbruck', 'Risiera di San Sabba', 'Sachsenhausen', 'Sajmište', 'Salaspils', 'Sobibor', 'Sobibór', 'Soldau', 'Stutthof', 'Theresienstadt,', 'Trawniki', 'Treblinka', 'Vaivara']\n"
     ]
    }
   ],
   "source": [
    "def remove_accents(text):\n",
    "    \n",
    "    #Polish letters\n",
    "    letters= {\n",
    "    'ł':'l', 'ą':'a', 'ń':'n', 'ć':'c', 'ó':'o', 'ę':'e', 'ś':'s', 'ź':'z', 'ż':'z',\n",
    "    'Ł':'L', 'Ą':'A', 'Ń':'N', 'Ć':'C', 'Ó':'O', 'Ę':'E', 'Ś':'S', 'Ź':'Z', 'Ż':'Z',\n",
    "\n",
    "    #Accent Vowels\n",
    "    \"à\":\"a\", \"á\":\"a\", \"â\":\"a\", \"ã\":\"a\", \"ä\":\"a\", \"å\":\"a\", \"æ\": \"ae\",\n",
    "    \"À\":\"A\", \"Á\":\"A\", \"Â\":\"A\", \"Ã\":\"A\", \"Ä\":\"A\", \"Å\":\"A\", \"Æ\": \"ae\",\n",
    "\n",
    "    \"è\":\"e\", \"é\":\"e\", \"ê\":\"e\", \"ë\":\"e\",\n",
    "    \"È\":\"E\", \"É\":\"E\", \"Ê\":\"E\", \"Ë\":\"E\",\n",
    "\n",
    "    \"ì\":\"i\", \"í\":\"i\", \"î\":\"i\", \"ï\":\"i\",\n",
    "    \"Ì\":\"I\", \"Í\":\"I\", \"Î\":\"I\", \"Ï\":\"I\",\n",
    "\n",
    "    \"ò\": \"o\", \"ó\": \"o\", \"ô\": \"o\",  \"õ\": \"o\",  \"ö\": \"o\", \"ø\": \"o\",\n",
    "    \"Ò\": \"O\", \"Ó\": \"O\", \"Ô\": \"O\",  \"Õ\": \"O\",  \"Ö\": \"O\", \"Ø\": \"O\",\n",
    "\n",
    "    \"ù\": \"u\", \"ú\": \"u\",  \"û\": \"u\",  \"ü\": \"u\",\n",
    "    \"Ù\": \"U\", \"Ú\": \"U\",  \"Û\": \"U\",  \"Ü\": \"U\",\n",
    "\n",
    "    \"ý\": \"y\", \"ÿ\": \"y\",\n",
    "    \"Ý\": \"Y\", \"Ÿ\": \"Y\",\n",
    "\n",
    "    #Accent Cononants\n",
    "    \"ç\": \"c\", \"Ç\": \"C\",\n",
    "    \"ß\": \"ss\"\n",
    "    }\n",
    "    \n",
    "    trans=str.maketrans(letters)\n",
    "    result=text.translate(trans)\n",
    "    return (result)\n",
    "\n",
    "final = []\n",
    "for camp in ushmm_camps:\n",
    "    final.append(camp)\n",
    "    final.append(remove_accents(camp))\n",
    "\n",
    "#Delete all duplicates\n",
    "ushmm_camps = list(set(final))\n",
    "ushmm_camps.sort()\n",
    "print (ushmm_camps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the standardization of 'Płaszów' as 'Plaszow' in the list. Both forms are now represented in our dataset, meaning we can develop a rules-based EntityRuler that can find both forms of these words in texts. While we were able to solve the first problem, that of standardized data, we cannot solve the first, however. Should we wish, though, we could add this dataset to our Wikipedia datasets, but as we will see below, a larger dataset presents new challenges."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EHRI and Toponyms in the Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another excellent resource for datasets pertaining to the Holocaust is EHRI Project (https://portal.ehri-project.eu/). Under their authority sets and vocabularies, users have access to well-cultivated datasets, including \"EHRI Camps\" (https://portal.ehri-project.eu/vocabularies). This list contains 1,975 different camps. As we did with the Wikipedia data above, we can easily webscrape this data. It is here, however, that I want to raise a word of caution.\n",
    "\n",
    "Both the EHRI dataset and the Wikipedia larger dataset (both of which include subcamps) contain toponyms, or words that have identical spelling but have different meaning in different contexts. A good example of this is \"Cologne\" which is in our Buchenwald subcamp dataset. Cologne, however, is not strictly a \"camp\", rather it can also be a GPE or LOCATION. In other words, it can also function as Cologne the city in a text. How then do we account for this when trying to prepare a rules-based NER or automate the cultivation of a good training set for an ML model?\n",
    "\n",
    "To generate a rules-based method, we could eliminate all instances of these toponyms being used without the word \"camp\" nearby. We could also train a toponym model that parses all identified camps and then eliminates the ones that don't fit the typical pattern, or any number of other things. There are options. I am less interested in developing a rules-based NER approach for the purpose of deployment, however. I am interested in using a rules-based NER approach to cultivate a training set for a machine learning NER model. Because this is my objective, I would drop all instances of toponyms from my dataset, such as Cologne. The best way to do this is with a content expert and manually eliminate all toponyms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whatever dataset you are trying to cultivate, see if you can find it on the web. As you prepare that dataset, think about the steps discussed in this notebook. Clean the data and consider the various ways in which your dataset could accidently be used to identify false positives. If you can't think about all the different ways that can happen, that's okay. In notebook 09.03 we will start to test our datasets against a corpus. I often do not clean datasets perfectly on my first pass. It is only after a few tests that I realize I can clean them further."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Video"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div align=\"center\">\n",
       "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/XScpSI3RYQI\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<div align=\"center\">\n",
    "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/XScpSI3RYQI\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}