02_building_a_simple_index.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 2: Building a Simple Index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this assignment, we will build a simple search index, which we will use later for Boolean retrieval. The assignment tasks are again at the bottom of this document."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "Summaries_file = 'data/cognition_Summaries.pkl.bz2'\n",
    "Abstracts_file = 'data/cognition_Abstracts.pkl.bz2'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle, bz2\n",
    "from collections import namedtuple\n",
    "\n",
    "Summaries = pickle.load( bz2.BZ2File( Summaries_file, 'rb' ) )\n",
    "\n",
    "paper = namedtuple( 'paper', ['title', 'authors', 'year', 'doi'] )\n",
    "\n",
    "for (id, paper_info) in Summaries.items():\n",
    "    \n",
    "    Summaries[id] = paper( *paper_info )\n",
    "    \n",
    "Abstracts = pickle.load( bz2.BZ2File( Abstracts_file, 'rb' ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at what the data looks like for an example of a paper:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "paper(title='Handgrip Strength in Old and Very Old Adults: Mood, Cognition, Function, and Mortality.', authors=['Stessman J', 'Rottenberg Y', 'Fischer M', 'Hammerman-Rozenberg A', 'Jacobs JM'], year=2017, doi='10.1111/jgs.14509')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Summaries[28102890]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'To determine the trajectory of handgrip strength (HGS) from age 70 to 90 and its association with mood, cognition, functional status, and mortality.'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Abstracts[28102890]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Some Utility Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll define some utility functions that allow us to tokenize a string into terms, perform linguistic preprocessing on a list of terms, as well as a function to display information about a paper in a nice way. Note that these tokenization and preprocessing functions are rather naive. We will improve them in a later assignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['lorem', 'ipsum', 'dolor', 'sit', 'amet']\n"
     ]
    }
   ],
   "source": [
    "def tokenize(text):\n",
    "    \"\"\"\n",
    "    Function that tokenizes a string in a rather naive way. Can be extended later.\n",
    "    \"\"\"\n",
    "    return text.split(' ')\n",
    "\n",
    "def preprocess(tokens):\n",
    "    \"\"\"\n",
    "    Perform linguistic preprocessing on a list of tokens. Can be extended later.\n",
    "    \"\"\"\n",
    "    result = []\n",
    "    for token in tokens:\n",
    "        result.append(token.lower())\n",
    "    return result\n",
    "\n",
    "print(preprocess(tokenize(\"Lorem ipsum dolor sit AMET\")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.1111/jgs.14509>Handgrip Strength in Old and Very Old Adults: Mood, Cognition, Function, and Mortality.</a></strong><br>2017. Stessman J, Rottenberg Y, Fischer M, Hammerman-Rozenberg A, Jacobs JM<br>[ID: 28102890]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.1111/jgs.14509>Handgrip Strength in Old and Very Old Adults: Mood, Cognition, Function, and Mortality.</a></strong><br>2017. Stessman J, Rottenberg Y, Fischer M, Hammerman-Rozenberg A, Jacobs JM<br><small><strong>Abstract:</strong> <em>To determine the trajectory of handgrip strength (HGS) from age 70 to 90 and its association with mood, cognition, functional status, and mortality.</em></small><br>[ID: 28102890]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import display, HTML\n",
    "import re\n",
    "\n",
    "def display_summary( id, show_abstract=False, show_id=True, extra_text='' ):\n",
    "    \"\"\"\n",
    "    Function for printing a paper's summary through IPython's Rich Display System.\n",
    "    Trims long author lists, and adds a link to the paper's DOI (when available).\n",
    "    \"\"\"\n",
    "    s = Summaries[id]\n",
    "    lines = []\n",
    "    title = s.title\n",
    "    if s.doi != '':\n",
    "        title = '<a href=http://dx.doi.org/{:s}>{:s}</a>'.format(s.doi, title)\n",
    "    title = '<strong>' + title + '</strong>'\n",
    "    lines.append(title)\n",
    "    authors = ', '.join( s.authors[:20] ) + ('' if len(s.authors) <= 20 else ', ...')\n",
    "    lines.append(str(s.year) + '. ' + authors)\n",
    "    if (show_abstract):\n",
    "        lines.append('<small><strong>Abstract:</strong> <em>{:s}</em></small>'.format(Abstracts[id]))\n",
    "    if (show_id):\n",
    "        lines.append('[ID: {:d}]'.format(id))\n",
    "    if (extra_text != ''):\n",
    "         lines.append(extra_text)\n",
    "    display( HTML('<br>'.join(lines)) )\n",
    "\n",
    "display_summary(28102890)\n",
    "display_summary(28102890, show_abstract=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating our first index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now create an _inverted index_ based on the words in the titles and abstracts of the papers in our dataset. We will implement our inverted index as a Python dictionary with term strings as keys and posting lists (implemented as Python lists) as values. We include all the tokens we can find in the title and (if available) in the abstract:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "inverted_index = defaultdict(list)\n",
    "\n",
    "# This can take a few seconds:\n",
    "for id in sorted(Summaries.keys()):\n",
    "    term_set = set(preprocess(tokenize(Summaries[id].title)))\n",
    "    if id in Abstracts:\n",
    "        term_set.update(preprocess(tokenize(Abstracts[id])))\n",
    "    for term in term_set:\n",
    "        inverted_index[term].append(id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what's in the index for the example term 'amsterdam':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[10547144, 17888546, 18289505, 19004529, 20594617, 20969529, 21482441, 22049191, 23232034, 23802509, 23964225, 25114679, 25598195, 25631781, 25684214, 25877267, 26976043, 27148107, 28462387, 28671504, 29029762, 29209183, 29406610, 29843832, 29848469, 31026198, 31082289, 31547145, 31730165, 33008394, 34203041, 34408904, 34541288, 34925184, 35149261]\n"
     ]
    }
   ],
   "source": [
    "print(inverted_index['amsterdam'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use this inverted index to answer simple one-word queries, for example to show all papers that contain the word 'utrecht':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.1016/j.ejphar.2013.04.055>Sweet memories: 20 years of progress in research on cognitive functioning in diabetes.</a></strong><br>2013. Biessels GJ<br><small><strong>Abstract:</strong> <em>This paper appears in a special issue of the European Journal of Pharmacology that commemorates the retirement of Professor Willem Hendrik Gispen as distinguished professor of Utrecht University and as editor of the European Journal of Pharmacology. The paper provides an overview of a research line on the impact of diabetes on cognition that we started together 20 years ago, and that continues to this day. I will report how we more or less stumbled upon this topic, that was understudied, but proved to be of definite clinical relevance. I will discuss how we tried to establish animal models, how developments from clinical and experimental studies from around the world led us to reconsider our concepts, and how findings from research on diabetic neuropathy, insulin signaling in the brain, Alzheimer's disease and dementia, and vascular disease and stroke converged and helped to create new ideas and refute others. This voyage has not ended yet, because the ultimate goal is to offer patients with diabetes treatment that can protect them against accelerated cognitive decline. Although this could take another 20 years, the research from Willem Hendrik and his group brought us an important step in the right direction. </em></small><br>[ID: 23872409]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.1097/MRR.0000000000000198>Predictors of physical independence at discharge after stroke rehabilitation in a Dutch population.</a></strong><br>2017. Ten Brink AF, Hajos TR, van Bennekom C, Nachtegaal J, Meulenbelt HE, Fleuren JF, Kouwenhoven M, Luijkx MM, Wijffels MP, Post MW<br><small><strong>Abstract:</strong> <em>The aim of this study was to identify predictors, available at admission, of physical independence at discharge from inpatient rehabilitation. Secondary aims were to identify predictors of functional gain and length of stay (LOS). We included 1310 adult stroke patients who were admitted for inpatient rehabilitation in five Dutch rehabilitation centres. Data on the Utrecht Scale for Evaluation of Clinical Rehabilitation at admission and discharge (physical and cognitive independence, mood, pain and fatigue), age, sex and in a subsample stroke characteristics as well were collected. A prediction model was created using random coefficient analysis. None of the stroke characteristics were independently associated with physical independence or functional gain at discharge, or LOS. Higher physical and cognitive independence scores and severe pain at admission were predictors of higher physical independence scores at discharge. Furthermore, lower physical independence scores, higher cognitive independence scores, less pain at admission and younger age predicted more functional gain. Finally, lower physical and cognitive independence scores at admission and younger age predicted longer LOS. Physical independence at admission was the most robust predictor for rehabilitation outcome in a Dutch rehabilitation setting. To a lesser extent, age, cognitive independence and pain predicted rehabilitation outcome after stroke. Treatment of cognition and pain should be taken into account during rehabilitation. Further work needs to be carried out to establish whether focusing on these factors improves outcome after rehabilitation.</em></small><br>[ID: 27749516]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.1080/09602011.2017.1313746>Return to work after mild-to-moderate stroke: work satisfaction and predictive factors.</a></strong><br>2019. van der Kemp J, Kruithof WJ, Nijboer TCW, van Bennekom CAM, van Heugten C, Visser-Meily JMA<br><small><strong>Abstract:</strong> <em>A large proportion of stroke patients are unable to return to work (RTW), although figures vary greatly. A total of 121 mild-to-moderate stroke patients, who had a paid job at the time of their stroke were included (a) to quantify RTW and work satisfaction one-year post-stroke (using the Utrecht Scale for Evaluation of Rehabilitation-Participation) and (b) to determine factors predicting RTW post-stroke, based on stroke-related, personal and neuropsychological variables. Half of the patients were not in work (28%) or were working less (22%) than pre-stroke. Ninety percent of those in fulltime employment post-stroke were satisfied with their occupational situation, against 36% of the unemployed participants. In regards to factors predicting RTW, global cognitive functioning (r = .19, Montreal Cognitive Assessment) and depressive symptoms (r = -.16, Hospital Anxiety and Depression Scale) at two months post-stroke onset were associated with return to work within one year. Only global cognitive functioning was an independent predictor of RTW (11.3% variance, p = .013). Although the explained variance was not that high, neuropsychological factors probably play a pivotal role in returning to work and should be taken into account during rehabilitation after mild and moderate stroke.</em></small><br>[ID: 28441897]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.1080/09602011.2019.1692670>The relationship between social cognition and participation in the long term after stroke.</a></strong><br>2021. Stiekema APM, Nijsse B, de Kort PLM, Spikman JM, Visser-Meily JMA, van Heugten CM<br><small><strong>Abstract:</strong> <em>Social cognitive impairments may play a role in participation restrictions after stroke. Understanding their relationship could inform treatment approaches to improve participation. We investigated the relationship between social cognition and participation in the long term after stroke. Of 395 patients participating in a large prospective cohort study, cross-sectional data were available at 3-4 years post-stroke of 118 patients on tests for emotion recognition, theory of mind, empathy, and behaviour regulation. Participation was assessed with the Utrecht Scale for Evaluation of Rehabilitation - Participation (USER-P). Bivariate and multivariate regression analysis were used to examine the relationship between social cognitive domains and participation. The majority suffered from minor stroke (83.1% scored NIHSS 0-4). Only behaviour regulation was related to participation restrictions in bivariate analysis, but social cognitive impairments did not predict participation restrictions in multivariate regression in this group. To conclude, in a sample of minor stroke patients with mild impairments in theory of mind, emotion recognition and behavioural control, there were no associations with restrictions in participation. Research should examine whether a relationship is present in patients with more severe stroke. In addition, measuring social aspects of participation is necessary to further unravel this relationship, to determine treatment targets for improving participation.</em></small><br>[ID: 31854264]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.1080/09602011.2020.1769687>Which approach to measure cognitive functioning should be preferred when exploring the association between cognitive functioning and participation after stroke?</a></strong><br>2021. de Graaf JA, Nijsse B, Schepers VPM, van Heugten CM, Post MWM, Visser-Meily JMA<br><small><strong>Abstract:</strong> <em>A variety of approaches are currently used to explore the relationship between cognitive functioning and participation after stroke. We aimed to gain insight into the preferred approach to measure cognitive functioning when exploring the association between cognitive functioning and participation in the long term after stroke. In this inception cohort study 128 individuals with stroke participated and were assessed at a single time point three to four years after the event. Participation was measured using the Restrictions subscale of the Utrecht Scale for Evaluation of Rehabilitation-Participation. Subjective cognitive complaints were assessed using the Cognition subscale of the Checklist for Cognitive and Emotional Consequences (CLCE-24-C). Objective cognitive performance was measured using the Montreal Cognitive Assessment (MoCA) and a neuropsychological test battery (NTB) testing multiple cognitive domains. Participation showed a strong correlation (</em></small><br>[ID: 32496877]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.5964/ejop.v16i1.1832>Examining the Relationship Between Engagement and Perceived Stress-Related Cognitive Complaints in the Argentinian Working Population.</a></strong><br>2020. Miranda AR, Rivadero L, Bruera JÁ, Villarreal V, Bernio LY, de Los Ángeles Baydas L, Brizuela ML, Serra SV<br><small><strong>Abstract:</strong> <em>Stress has a negative impact on cognitive functioning and occupational well-being. The aim of this study was to assess the relationship among perceived stress, cognitive complaints and work engagement in public employees from Córdoba, Argentina. In this cross-sectional study, self-report questionnaires were administered to 240 participants. Spanish versions of the following instruments were used: Perceived Stress Scale (PSS), Utrecht Work Engagement Scale (UWES), Memory Failures in Everyday (MFE), Executive Complaint Questionnaire (ECQ). Statistical analysis included ANOVA, path analysis, and multiple logistic regression. Stressed workers showed lower work engagement and more cognitive complaints, even after adjusting for demographic variables. Negative associations were also observed between work engagement and cognitive complaints, suggesting that cognitive difficulties are related to engagement. Given the relation among stress, cognition, and work engagement, it is important to consider these factors to foster workers' health and work productivity.</em></small><br>[ID: 33680167]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "query_word = 'utrecht'\n",
    "for i in inverted_index[query_word]:\n",
    "    display_summary(i, show_abstract=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Your name:** Thomas Norton"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 1\n",
    "\n",
    "Implement the function `and_merge` outlined below. This function takes two posting lists from the index that can be assumed to be sorted already, and it should return the result of the merging of the two lists with AND. The resulting list should therefore include all the elements that appear in both lists. As explained on the slides, this operation should take advantage of the input lists being sorted already, should not perform any additional sorting operation, and should go through each of the input lists just once. Then, test your function with an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 3, 10, 67, 100, 150, 808, 810]\n"
     ]
    }
   ],
   "source": [
    "def and_merge(listA, listB):\n",
    "    duplicates = []\n",
    "    aLength, bLength = len(listA) - 1, len(listB) - 1\n",
    "    iA, iB = 0, 0\n",
    "    while iA <= aLength and iB <= bLength:\n",
    "        if iA > aLength:\n",
    "            iA = aLength\n",
    "        if iB > bLength:\n",
    "            iB = bLength\n",
    "        if listA[iA] == listB[iB]:\n",
    "            duplicates.append(listA[iA])\n",
    "            iA += 1\n",
    "            iB += 1\n",
    "        elif listA[iA] > listB[iB]:\n",
    "            iB += 1\n",
    "        elif listA[iA] < listB[iB]:\n",
    "            iA += 1\n",
    "    return duplicates\n",
    "\n",
    "a = [1, 2, 3, 5, 10, 22, 55, 67, 100, 150, 808, 810]\n",
    "b = [1, 3, 10, 67, 100, 150, 200, 202, 203, 808, 809, 810]\n",
    "\n",
    "print(and_merge(a,b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 2\n",
    "\n",
    "Similarly as above, implement the function `or_merge` outlined below that executes an OR merging of the lists. The resulting list should therefore include all the elements that appear in at least one of the lists (note this is an 'inclusive OR' and not a 'XOR'). Again, this operation should take advantage of the input lists being sorted already, should not perform any additional sorting operation, and should go through each of the input lists just once. Elements that appear in both input list should only appear once in the output list. Test your function again with an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 2, 3, 5, 10, 22, 55, 67, 100, 150, 200, 202, 203, 808, 1002, 1004, 1005, 1006, 1007, 1010]\n"
     ]
    }
   ],
   "source": [
    "def or_merge(listA, listB):\n",
    "    unique = []\n",
    "    aLen, bLen = len(listA) - 1, len(listB) - 1\n",
    "    iA, iB, save = 0, 0, 0\n",
    "    while True:\n",
    "        if iA >= aLen and iB >= bLen:\n",
    "            break\n",
    "        if iA > aLen:\n",
    "            for i in range(aLen - 1, bLen + 1):\n",
    "                unique.append(listB[i])\n",
    "            break\n",
    "        if iB > bLen:\n",
    "            for i in range(bLen, aLen):\n",
    "                unique.append(listA[i])\n",
    "            break\n",
    "        if listA[iA] == listB[iB]:\n",
    "            unique.append(listA[iA])\n",
    "            save = listA[iA]\n",
    "            iA += 1\n",
    "            iB += 1\n",
    "        elif listA[iA] < listB[iB]:\n",
    "            if listA[iA] > save:\n",
    "                unique.append(listA[iA])\n",
    "                save = listA[iA]\n",
    "            iA += 1\n",
    "        elif listB[iB] < listA[iA]:\n",
    "            if listB[iB] > save:\n",
    "                unique.append(listB[iB])\n",
    "                save = listB[iB]\n",
    "            iB += 1\n",
    "    return unique\n",
    "\n",
    "\n",
    "a = [1, 2, 3, 5, 10, 22, 55, 67, 100, 150, 808, 1002, 1004, 1005]\n",
    "b = [1, 3, 10, 67, 100, 150, 200, 202, 203, 808, 1002, 1004, 1006, 1007, 1010]\n",
    "\n",
    "print(or_merge(a,b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 3\n",
    "\n",
    "Construct a function called `and_query` that takes as input a single string, consisting of one or more words, and returns as function value a list of matching documents. `and_query`, as its name suggests, should require that all query terms are present in the documents of the result list.\n",
    "\n",
    "For that, access the variable `inverted_index` from above and use the method `and_merge` that you defined. Also use the `tokenize` and `preprocess` functions we defined above to tokenize and preprocess your query.\n",
    "\n",
    "Again demonstrate the working of your function with an example (choose one that leads to fewer than 100 hits to not overblow this notebook file)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[33661082, 34439604]\n"
     ]
    }
   ],
   "source": [
    "def and_query(input_string):\n",
    "    processed = preprocess(tokenize(input_string))\n",
    "    matching_documents = []\n",
    "    for token in processed:\n",
    "        matching_documents.append(inverted_index[token])\n",
    "    merged = and_merge(matching_documents[0], matching_documents[1])\n",
    "    for i in range(1, len(matching_documents) - 1):\n",
    "        merged = and_merge(merged, matching_documents[i+1])\n",
    "    return merged\n",
    "\n",
    "print(and_query('adult handgrip strength'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 4\n",
    "\n",
    "Construct another function called `or_query` that works in the same way as `and_query` you just implemented, but returns as function value the documents that contain _at least one_ of the words in the query, using the `or_merge` function you defined.\n",
    "\n",
    "Demonstrate the working of this function also with an example (again, choose one that leads to fewer than 100 hits)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[8588536, 10547144, 16137273, 17888546, 18289505, 19004529, 20025431, 20219767, 20495953, 20594617, 20969529, 21482441, 21769735, 22049191, 22546518, 23232034, 23802509, 23936113, 23964225, 24827615, 24862635, 25114679, 25177186, 25598195, 25631781, 25684214, 25877267, 26016893, 26330266, 26511758, 26656425, 26976043, 27148107, 27235850, 27521997, 27828593, 27832209, 28102890, 28105965, 28189666, 28357463, 28462387, 28575126, 28671504, 29029762, 29209183, 29406610, 29522691, 29526916, 29710135, 29743873, 29843832, 29848469, 29893487, 30046894, 30143006, 30210435, 30369219, 30843369, 30931693, 31026198, 31082289, 31237315, 31277579, 31322562, 31390821, 31401637, 31469966, 31547145, 31730165, 31929158, 32210785, 32477460, 32510114, 32587294, 32821140, 33008394, 33271505, 33561081, 33588769, 33600874, 33661082, 33710497, 33720898, 33732918, 33869133, 33879389, 33937294, 34203041, 34208163, 34380435, 34408904, 34439604, 34455171, 34541288, 34805022, 34925184, 35072335, 35149261, 30843369, 30931693, 31237315, 31277579, 31322562, 31390821, 31401637, 31469966, 31929158, 32210785, 32477460, 32510114, 32587294, 32821140, 33271505, 33561081, 33588769, 33600874, 33661082, 33710497, 33720898, 33732918, 33869133, 33879389, 33937294, 34208163, 34380435, 34439604, 34455171, 34805022, 35072335, 35206845, 35390728, 35440016, 35645971, 35737388, 35891808, 35974301, 36152626]\n"
     ]
    }
   ],
   "source": [
    "def or_query(input_string):\n",
    "    processed = preprocess(tokenize(input_string))\n",
    "    matching_documents = []\n",
    "    for token in processed:\n",
    "        matching_documents.append(inverted_index[token])\n",
    "    merged = or_merge(matching_documents[0], matching_documents[1])\n",
    "    for i in range(1, len(matching_documents) - 1):\n",
    "        merged = or_merge(merged, matching_documents[i+1])\n",
    "    return merged\n",
    "\n",
    "print(or_query('amsterdam handgrip'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 5\n",
    "\n",
    "Why does `and_query('adult handgrip strength')` not return our example paper 28102890, even though it mentions handgrip strength and adults in the abstract? (You do not have to implement anything to fix this yet!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Executing the and_query function with the example string 'adult handgrip strength' does not return this paper because the paper uses the word 'adults' instead of 'adult'. Since this is quite a basic implementation, they are treated as completely different words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Submission"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Submit the answers to the assignment via Canvas as a modified version of this Notebook file (file with `.ipynb` extension) that includes your code and your answers.\n",
    "\n",
    "Before submitting, restart the kernel and re-run the complete code (**Kernel > Restart & Run All**), and then check whether your assignment code still works as expected.\n",
    "\n",
    "Don't forget to add your name, and remember that the assignments have to be done **individually**, and that code sharing or copying are **strictly forbidden** and will be punished."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.2 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.2"
  },
  "vscode": {
   "interpreter": {
    "hash": "369f2c481f4da34e4445cda3fffd2e751bd1c4d706f27375911949ba6bb62e1c"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}