04_NLP_pneumonia_pyConText_targets_and_modifiers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Objectives\n",
    "* Gain hands on experience working with pyConText and its resources\n",
    "* Understand and develop Targets\n",
    "* Understand and develop Modifiers\n",
    "* Graph and visualize Targets and Modifiers together\n",
    "* Gain tools for group project in reducing False Negatives and False Positives (F1 measure)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download NLTK corpora to facilitate sentence splitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m textblob.download_corpora"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# we will definitely need pyConText\n",
    "import pyConTextNLP\n",
    "from pyConTextNLP import pyConTextGraph\n",
    "from pyConTextNLP.itemData import itemData\n",
    "from pyConTextNLP.display.html import mark_document_with_html\n",
    "print(pyConTextNLP.__version__)\n",
    "# useful utilities in RadNLP as well\n",
    "import radnlp\n",
    "import radnlp.view as rview\n",
    "from radnlp.data import classrslts\n",
    "# we will need a few other packages\n",
    "from textblob import TextBlob\n",
    "import urllib\n",
    "import pandas as pd\n",
    "# packages for interaction\n",
    "from IPython.html.widgets import interact, interactive, fixed\n",
    "from IPython.display import display, HTML, Image\n",
    "import ipywidgets\n",
    "# and also our utilities for this class\n",
    "from nlp_pneumonia_utils import Annotation\n",
    "from nlp_pneumonia_utils import AnnotatedDocument\n",
    "from nlp_pneumonia_utils import read_brat_annotations\n",
    "from nlp_pneumonia_utils import read_doc_annotations\n",
    "from nlp_pneumonia_utils import read_annotations\n",
    "from nlp_pneumonia_utils import calculate_prediction_metrics\n",
    "from nlp_pneumonia_utils import mark_text\n",
    "from nlp_pneumonia_utils import clearPyConTextRegularExpressions\n",
    "from nlp_pneumonia_utils import pneumonia_annotation_html_markup\n",
    "from nlp_pneumonia_utils import mark_document_with_html\n",
    "from nlp_pneumonia_utils import view_single_sentence_graph\n",
    "from nlp_pneumonia_utils import markup_sentence\n",
    "\n",
    "from pprint import pprint\n",
    "import xml.dom.minidom as minidom \n",
    "import xml.etree.ElementTree as ElementTree\n",
    "#from pycontext_quiz import identify_target_category\n",
    "#from pycontext_quiz import file_delimiter_quiz\n",
    "#from pycontext_quiz import modifier_directionality_quiz\n",
    "#from pycontext_quiz import second_most_frequent_modifier_quiz\n",
    "\n",
    "print('Imported pneumonia nlp utilities...')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# First thing, let's load our training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "annotated_doc_map = read_doc_annotations('data/training_v2.zip')\n",
    "\n",
    "# let's also use a simple list of documents as well as this map\n",
    "annotated_docs = list(annotated_doc_map.values())\n",
    "\n",
    "print('Total Annotated Documents : {0}'.format(len(annotated_docs)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up an example document to work with\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "example_document = \"\"\"\n",
    "PORTABLE CHEST:  Comparison made to prior film from X:XX a.m. the same day.\n",
    "     \n",
    "The ET tube and nasogastric tube remain in good position. Cardiac and\n",
    "mediastinal contours are stable. No acute changes are seen within the lung\n",
    "parenchyma; specifically, there is no evidence of new infiltrate (skin folds\n",
    "do project over the right lung). No consolidation on either side.\n",
    "\n",
    "IMPRESSION: No evidence of pneumonia.\"\"\"\n",
    "\n",
    "example_sentence = \"\"\"IMPRESSION: No evidence of pneumonia.\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Our first goal is to translate pneumonia \"keywords\"  into pyConText \"targets\" \n",
    "### each target can have:\n",
    "* Standard form\n",
    "* Category \n",
    "* Additional lexical variants (with regular expressions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Before we continue, note that any itemData in pyConText has 4 parts:\n",
    "1. The literal (e.g. \"pneumonia\", \"pneumoniathorax\", \"can rule out\", \"cannot be excluded\", etc)\n",
    "2. The category (e.g. \"EVIDENCE_OF_PNEUMONIA\")\n",
    "3. The regular expression (optional) used to capture the literal in the text. If no regular expression is provided, a regular expression is generated literally from the literal.\n",
    "4. The rule (optional). If the itemData is being used as a modifier, the rule states what direction the modifier operates in the sentence: current valid values are: \"forward\", the item can modify objects following it in the sentence; \"backward\", the item can modify objects preceding it in the sentence; or \"bidirectional\", the item can modify objects preceding and following it in the sentence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additionally, itemData can be instantiated in code or from a file.\n",
    "\n",
    "#### set up some rules for pyConText for EVIDENCE_OF_PNEUMONIA\n",
    "\n",
    "* At this moment, we will just set up these \"concepts\" and we will handle modifiers later\n",
    "* targetswill look like this: \n",
    "\n",
    "```Python\n",
    "targets = itemData([\"literal\", \"CATEGORY\", \"regular expression(s)\", \"empty or forward or backward or bidirectional\"])\n",
    "```\n",
    "* As a CATEGORY we will use `EVIDENCE_OF_PNEUMONIA`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "targets1 = []\n",
    "modifiers1 = []\n",
    "\n",
    "\n",
    "targets1 = itemData([\"pneumonia\", \"EVIDENCE_OF_PNEUMONIA\", \"\", \"\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use this now on example sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "markup = markup_sentence(example_sentence, modifiers1, targets1)\n",
    "# prettier display with IPython display\n",
    "display(markup.nodes(data = True))\n",
    "print(markup.getXML())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This function works on entire documents \n",
    "### Combine all sentence-level objects into one object we can can then graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def markup_context_document(report_text, modifiers, targets):\n",
    "    context = pyConTextGraph.ConTextDocument()\n",
    "    \n",
    "    # we will use TextBlob for breaking up sentences\n",
    "    sentences = [s.raw for s in TextBlob(report_text).sentences]\n",
    "    for sentence in sentences:\n",
    "        m = markup_sentence(sentence, modifiers=modifiers, targets=targets)\n",
    "        context.addMarkup(m)\n",
    "    \n",
    "    return context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quesiton : Will we find a target match on this sentence? Will we match \"pneumonias\"?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "example_sentence_2 = \\\n",
    "\"\"\"Findings consistent with CHF, although underlying bilateral lower lobe pneumonias cannot \n",
    "be excluded.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's see how things look on this sentence\n",
    "markup_sentence_2 = markup_sentence(example_sentence_2, modifiers1, targets1, verbose = True)\n",
    "display(markup_sentence_2.nodes(data = True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### So we didn't mark up a target for \"pneumonias\" \n",
    "#### We only had the singular variant \"pneumonia\". \n",
    "\n",
    "### Let's add that as we augment our target concepts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pneumonia_targets_file = 'KB/pneumonia_targets.tsv'\n",
    "\n",
    "# let's see what we're working with by loading this as a Pandas DataFrame and then we can display it\n",
    "pneumonia_targets_df = pd.read_csv(pneumonia_targets_file, delimiter = '\\t')\n",
    "display(pneumonia_targets_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## This time we will augment our targets by modifying a tab-delimited file (.tsv).  A starter TSV file is included in our course resources:\n",
    "<a href=\"https://github.com/UUDeCART/decart_rule_based_nlp/blob/master/KB/pneumonia_targets.tsv\">KB/pneumonia_targets.tsv</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Our first attempt was very simple targets, so now let's add some additional concepts\n",
    "targets2 = []\n",
    "modifiers2 = []\n",
    "\n",
    "# before we continue, let's clear a mapping of compiled regular expressions which pyConText uses\n",
    "clearPyConTextRegularExpressions()\n",
    "\n",
    "# so now let's set this up with more variants of \"EVIDENCE_OF_PNEUMONIA\"\n",
    "full_targets_path = 'file:///' + os.path.join(os.getcwd(), pneumonia_targets_file)\n",
    "print('Loading pneumonia targets from : ' + full_targets_path)\n",
    "targets2 = pyConTextNLP.itemData.instantiateFromCSVtoitemData(full_targets_path)\n",
    "\n",
    "# let's go ahead and use this again on our updated targets\n",
    "context = markup_context_document(example_document, modifiers2, targets2)\n",
    "# prettier display with IPython display\n",
    "display(context.getDocumentGraph().nodes(data = True))\n",
    "#print(context.getXML())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's look at this markup in HTML with colors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evidence_only_colors = {\n",
    "    \"evidence_of_pneumonia\": \"orange\"\n",
    "}\n",
    "\n",
    "context_html = mark_document_with_html(context, colors = evidence_only_colors, default_color=\"black\")\n",
    "display(HTML(context_html))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's also look again to see if our regular expression for \"pneumonia\" and \"pneumonias\" worked properly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "markup_sentence_2_check = markup_sentence(example_sentence_2, modifiers2, targets2)\n",
    "print(targets2)\n",
    "display(markup_sentence_2_check.nodes(data = True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  We've  added some pyConText targets, so let's utilize them in a classifier \n",
    "### so that we can see that adding targets can increase our Recall even if Precision may still be low\n",
    "* NOTE 1 : We will address Precision in a moment when we start working with ConText Modifiers\n",
    "* NOTE 2 : You don't need to understand all of the code below.  Essentially, it predicts pneumonia and returns 1 anytime it sees at least one target \"EVIDENCE_OF_PNEUMONIA\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ConTextTargetOnlyClassifier(object):\n",
    "    def __init__(self, modifiers, targets):\n",
    "        self.modifiers = modifiers\n",
    "        self.targets = targets\n",
    "    def predict(self, text):\n",
    "        # let's use our other functions in this notebook to perform sentence-wise markup and\n",
    "        # we can then check to see if these contain any EVIDENCE_OF_PNEUMONIA category types\n",
    "        context = markup_context_document(text, self.modifiers, self.targets)\n",
    "        document_graph = context.getDocumentGraph()\n",
    "        \n",
    "        # let's walk through all of the nodes in the graph and see how many are evidence of pneumonia\n",
    "        pneumonia_evidence_count = 0\n",
    "        for node in document_graph.nodes():\n",
    "            category_list = node.getCategory()\n",
    "            for category in category_list:\n",
    "                if category.upper() == 'EVIDENCE_OF_PNEUMONIA':\n",
    "                    pneumonia_evidence_count += 1\n",
    "            \n",
    "        # do we have at least one category of pneumonia evidence here?\n",
    "        return (pneumonia_evidence_count) > 0\n",
    "           \n",
    "# this one has only one target\n",
    "classifier1 = ConTextTargetOnlyClassifier(modifiers1, targets1)\n",
    "# this one has 3...\n",
    "classifier2 = ConTextTargetOnlyClassifier(modifiers2, targets2)\n",
    "\n",
    "# and now we can assess their performance\n",
    "print('****************')\n",
    "print('Performance for Classifier 1 : One total Target')\n",
    "calculate_prediction_metrics(annotated_docs, classifier1.predict)\n",
    "\n",
    "print('****************')\n",
    "print('Performance for Classifier 2 : 3 total Targets')\n",
    "calculate_prediction_metrics(annotated_docs, classifier2.predict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### So we have improved recall, but what are we going to do about Precision?  \n",
    "#### Since both Precision and Recall are measured equally in our F1 measure, we need to address it.  \n",
    "#### The solution to this is to improve our classification pipeline with ConText Modifiers\n",
    "* Modifiers are used to modify other tags or items in the sentence.  For example, we've already added targets for evidence of pneumonia.  We can now use modifiers to treat each of these mentions as affirmed or denied (negated) for a few examples.\n",
    "* Concretely if we had a sentence \"No evidence of pneumonia\" our current pipelines would count this as a positive case of pneumonia when it is in fact the contrary.\n",
    "* We can add a rule \"no evidence of\" to modify all targets which follow it\n",
    "* Modifiers can also modify targets which came before it in a sentence such as \"pneumonia will be ruled out\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modifier Rules can have the following values:\n",
    "1. backward (modify any markup preceding it)\n",
    "2. forward (modify any markup following it)\n",
    "3. bidirectional (modify any markup following or preceding it)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's look at a simple example of a modifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "likely_sentence = \"\"\"likely represents pneumonia\"\"\"\n",
    "\n",
    "simple_modifiers = itemData([\"likely represents\", \"PROBABLE_EXISTENCE\", \"\", \"forward\"])\n",
    "\n",
    "clearPyConTextRegularExpressions()\n",
    "view_single_sentence_graph(likely_sentence, simple_modifiers, targets1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now let's experiment with the directionality of the modifiers (forward, backward, etc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "probable_sentence_1 = \"\"\"probable case of pneumonia\"\"\"\n",
    "probable_sentence_2 = \"\"\"no evidence of pneumonia and probable arthritis\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "   \n",
    "modifiers_forward = itemData([\"probable\", \"PROBABLE_EXISTENCE\", \"\", \"forward\"])\n",
    "modifiers_backward = itemData([\"probable\", \"PROBABLE_EXISTENCE\", \"\", \"backward\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's ee how the forward modifiers work on probable_sentence_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's see how we graph the relationships between targets and modifiers based on 'backward' or 'forward':\n",
    "clearPyConTextRegularExpressions()\n",
    "view_single_sentence_graph(probable_sentence_1, modifiers_forward, targets1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assignment: Experiment with the two sentences and the two modifier lists to see how forward and backward work. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clearPyConTextRegularExpressions()\n",
    "#view_single_sentence_graph(probable_sentence_X, modifiers_X, targets1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## For probable_sentence_2 with modifiers_backward, the modifier has scope over the target, so pyConText would assign the label probable  evidence of pneumonia, which is not correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's see how we graph the relationships between targets and modifiers based on 'backward' or 'forward':\n",
    "clearPyConTextRegularExpressions()\n",
    "view_single_sentence_graph(probable_sentence_2, modifiers_backward, targets1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Another important Modifier rule : \"terminate\"\n",
    "This allows any modifier working forward or backward to stop its modifications if it encounters one of these terms.  Let's demonstrate an example where we want \"probable\" to modify \"arthritis\" as a condition but not \"pneumonia\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "terminate_example_sentence = \"\"\"probable arthritis but not pneumonia\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "temp_targets = itemData([\"pneumonia\", \"EVIDENCE_OF_PNEUMONIA\", \"\", \"\"],\n",
    "                       [\"arthritis\", \"ANOTHER_CONDITION\", \"\", \"\"])\n",
    "\n",
    "modifiers_without_terminate = itemData([\"probable\", \"PROBABLE_EXISTENCE\", \"\", \"forward\"])\n",
    "\n",
    "modifiers_with_terminate = itemData([\"probable\", \"PROBABLE_EXISTENCE\", \"\", \"forward\"],\n",
    "                                   [\"but\", \"CONJ\", \"\", \"terminate\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Without the \"terminate\" modifier, what will \"probable_existence\" be applied to?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clearPyConTextRegularExpressions()\n",
    "view_single_sentence_graph(terminate_example_sentence, modifiers_without_terminate, temp_targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Here with the \"terminate\" modifier \"probable_existence\" is applied only to \"arthritis\" and does not modify beyond the conjunction \"but\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clearPyConTextRegularExpressions()\n",
    "view_single_sentence_graph(terminate_example_sentence, modifiers_with_terminate, temp_targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Developing modifiers takes time and working with a lot of datasets.  \n",
    "### Luckily, many of them have already been developed on various research efforts.  \n",
    "### This file contains a subset of these and are a good place to start our group projects:\n",
    "<a href=\"https://github.com/UUDeCART/decart_rule_based_nlp/blob/master/KB/pneumonia_modifiers.tsv\">KB/pneumonia_modifiers.tsv</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "modifier_file_path = 'file:///' + os.path.join(os.getcwd(), \"KB/pneumonia_modifiers.tsv\")\n",
    "modifier_file = urllib.request.urlopen(modifier_file_path, data=None)\n",
    "# now let's load this in directly into a DataFrame with Pandas and take a look at it\n",
    "modifier_df = pd.read_csv(modifier_file, delimiter = \"\\t\")\n",
    "display(modifier_df.head(5))\n",
    "display(modifier_df.tail(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Before we go on, it might be useful to take inventory of all of the modifier categories that are contained within this file since there are dozens of rows.  We'll use Pandas to group by modifier \"category\" so that we can see what we're working with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(modifier_df.groupby('Type').size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now let's get back to how these are targets and modifiers are used together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "modifiers3 = pyConTextNLP.itemData.instantiateFromCSVtoitemData(modifier_file_path)\n",
    "# let's just use the same targets as above for our third pipeline\n",
    "targets3 = targets2\n",
    "\n",
    "clearPyConTextRegularExpressions()\n",
    "\n",
    "print('Total Modifiers Loaded for pipeline #3 : [{0}]'.format(len(modifiers3)))\n",
    "print('Total Targets Loaded for pipeline #3 : [{0}]'.format(len(targets3)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use HTML to visualize markups with our document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare some colors for displaying any markup we might see\n",
    "colors = {\n",
    "    \"evidence_of_pneumonia\": \"orange\",\n",
    "    \"definite_negated_existence\": \"red\",\n",
    "    \"probable_negated_existence\": \"indianred\",\n",
    "    \"ambivalent_existence\": \"orange\",\n",
    "    \"probable_existence\": \"forestgreen\",\n",
    "    \"definite_existence\": \"green\",\n",
    "    \"historical\": \"goldenrod\",\n",
    "    \"indication\": \"pink\",\n",
    "    \"acute\": \"golden\"\n",
    "}\n",
    "\n",
    "# let's mark up a new context object for our pipeline#3\n",
    "context3 = markup_context_document(example_document, modifiers3, targets3)\n",
    "\n",
    "display(HTML(mark_document_with_html(context3, colors = colors, default_color=\"black\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Take a closer look at the XML to see how this is working behind the scenes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(context3.getXML())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Process all of the documents and then visualize the relationships between Targets and Modifiers for some of these documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "report_results = []\n",
    "print('Marking up all documents...')\n",
    "for anno_doc in annotated_docs:\n",
    "    report_context = markup_context_document(anno_doc.text, modifiers3, targets3)\n",
    "    # package this up into a class that the RadNLP utilities can use\n",
    "    results = classrslts(context_document=report_context, exam_type=\"Chest X-Ray\", report_text=anno_doc.text, classification_result='N/A')\n",
    "    report_results.append(results)\n",
    "    \n",
    "print('DONE Marking up all documents...')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# This function let's us iterate through all documents and view the markup\n",
    "def view_pycontext_graph(class_results, colors):\n",
    "    @interact(i=ipywidgets.IntSlider(min=0, max=len(class_results)-1))\n",
    "    def _view_markup(i):\n",
    "        class_result = class_results[i]\n",
    "        rview.markup_to_pydot(class_result)\n",
    "        display(Image(\"tmp.png\"))\n",
    "        \n",
    "        report_html = mark_document_with_html(class_result.context_document, colors = evidence_only_colors, default_color=\"black\")\n",
    "        \n",
    "        display(HTML(report_html))\n",
    "        \n",
    "# This function let's us iterate through all documents and view the markup\n",
    "def view_annotation_markup(anno_docs, colors):\n",
    "    @interact(i=ipywidgets.IntSlider(min=0, max=len(anno_docs)-1))\n",
    "    def _view_markup(i):\n",
    "        report_html = pneumonia_annotation_html_markup(anno_docs[i])\n",
    "        \n",
    "        display(HTML(report_html))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "view_pycontext_graph(report_results, colors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br/><br/>This material presented as part of the DeCART Data Science for the Health Science Summer Program at the University of Utah in 2017.<br/>\n",
    "Presenters : Dr. Wendy Chapman, Jianlin Shi and Kelly Peterson"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}