04_evaluating.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 4: Evaluating Search Engines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this assignment, we leave aside the code we developed so far, and look into the more general issue of how to evaluate and compare different search engines. The ultimate test for any Information Retrieval system is how well it is able to satisfy the information needs of users."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cohen's Kappa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our evaluation will involve the calculation of [Cohen's kappa](https://en.wikipedia.org/wiki/Cohen's_kappa) to quantify the degree to which two human assessors agree or disagree on whether results are considered relevant or not. To calculate Cohen's kappa, we are going to use the [scikit-learn library](http://scikit-learn.org/stable/):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: scikit-learn in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (0.24.2)\n",
      "Requirement already satisfied: numpy>=1.13.3 in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from scikit-learn) (1.23.4)\n",
      "Requirement already satisfied: joblib>=0.11 in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from scikit-learn) (1.0.1)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from scikit-learn) (2.2.0)\n",
      "Requirement already satisfied: scipy>=0.19.1 in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from scikit-learn) (1.9.3)\n"
     ]
    }
   ],
   "source": [
    "! pip install --user scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import cohen_kappa_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This library expects relevance assessments as lists of elements where `1` stands for _relevant_ and `0` stands for _not relevant_, for example like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "a1=[1,0,1,0,1,0,1,0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This list means that the first document was assessed to be relevant, the second to be not relevant, the third to be relevant etc.\n",
    "\n",
    "We need two assessments in order to calculate Cohen's kappa, so let's make another exemplary list that only differs on the last element:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2=[1,0,1,0,1,0,1,1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now invoke the library as follows to calculate the agreement between the two:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.75"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cohen_kappa_score(a1, a2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This value represents high agreement. We can reach maximal agreement if the two assessments are identical:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cohen_kappa_score(a1, a1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's see what happens for a third assessment that differs on three positions with the first one (the three last positions):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.25"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a3=[1,0,1,0,1,1,0,1]\n",
    "\n",
    "cohen_kappa_score(a1, a3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get a smaller but still positive value, because these two assessments still mostly agree. If we make a further example that differs on 6 of the 8 positions, we get the following result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.5"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a4=[1,0,0,1,0,1,0,1]\n",
    "\n",
    "cohen_kappa_score(a1, a4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The score is now negative, because the two differ on more positions than they agree. The agreement is in fact less than what you would expect to occur just by chance. We get the maximal disagreement if we define a fifth example that disagrees on all positions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a5=[0,1,0,1,0,1,0,1]\n",
    "\n",
    "cohen_kappa_score(a1, a5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Be aware that the kappa score cannot be calculated if you have only `1`s or only `0`s:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/sklearn/metrics/_classification.py:641: RuntimeWarning: invalid value encountered in double_scalars\n",
      "  k = np.sum(w_mat * confusion) / np.sum(w_mat * expected)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "nan"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a6=[1,1,1,1,1,1,1,1]\n",
    "a7=[1,1,1,1,1,1,1,1]\n",
    "\n",
    "cohen_kappa_score(a6, a7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And in the case of a highly skewed set (either vast majority of agreements on `1` or vast majority of agreements on `0`), the kappa score can be counter-intuitive:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.1428571428571428"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a8=[1,1,1,1,1,1,0,1]\n",
    "a9=[1,1,1,1,1,1,1,0]\n",
    "\n",
    "cohen_kappa_score(a8, a9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we understand how this function works, we will apply it below for our specific evaluation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Results and Assessments\n",
    "\n",
    "Next, we will define some auxilary code to deal with lists of URLs from search engines and associated relevance assessments. We will encode result lists like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "urls = [\n",
    "    'https://en.wikipedia.org/wiki/Information_retrieval/',  # 1st result\n",
    "    'http://www.dictionary.com/browse/information',          # 2nd result\n",
    "    'https://nlp.stanford.edu/IR-book/'                      # ...\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we represent corresponding assessments, as above, as lists of the same size containing relevance values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_assessment = [1, 0, 1]\n",
    "another_assessment = [0, 0, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to nicely display URL lists, with or without related assessments, we define a function called `display_results`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display, HTML\n",
    "\n",
    "def display_results(urls, assessment1=None, assessment2=None):\n",
    "    lines = []\n",
    "    lines.append('<table>')\n",
    "    header = '<tr><th>#</th><th>Result URL</th>'\n",
    "    if (assessment1):\n",
    "        header += '<th>Assessment 1</th>'\n",
    "    if (assessment2):\n",
    "        header += '<th>Assessment 2</th>'\n",
    "    header += '</tr>'\n",
    "    lines.append(header)\n",
    "    i = 0\n",
    "    for url in urls:\n",
    "        show_url = url\n",
    "        if (len(url) > 80):\n",
    "            show_url = url[:75] + '...'\n",
    "        line = '<tr><td>{}</td><td><a href=\"{:s}\">{:s}</a></td>'.format(i+1, url, show_url)\n",
    "        if (assessment1):\n",
    "            if (assessment1[i] == 0):\n",
    "                line += '<td><em>Not relevant</em></td>'\n",
    "            else:\n",
    "                line += '<td><strong>Relevant</strong></td>'\n",
    "        if (assessment2):\n",
    "            if (assessment2[i] == 0):\n",
    "                line += '<td><em>Not relevant</em></td>'\n",
    "            else:\n",
    "                line += '<td><strong>Relevant</strong></td>'\n",
    "        line += '</tr>'\n",
    "        lines.append(line)\n",
    "        i = i+1\n",
    "    lines.append('</table>')\n",
    "    display( HTML(''.join(lines)) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use this function to display a list of URLs, optionally together with one or two assessment lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Just a list of URLs:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table><tr><th>#</th><th>Result URL</th></tr><tr><td>1</td><td><a href=\"https://en.wikipedia.org/wiki/Information_retrieval/\">https://en.wikipedia.org/wiki/Information_retrieval/</a></td></tr><tr><td>2</td><td><a href=\"http://www.dictionary.com/browse/information\">http://www.dictionary.com/browse/information</a></td></tr><tr><td>3</td><td><a href=\"https://nlp.stanford.edu/IR-book/\">https://nlp.stanford.edu/IR-book/</a></td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "With one assessment:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table><tr><th>#</th><th>Result URL</th><th>Assessment 1</th></tr><tr><td>1</td><td><a href=\"https://en.wikipedia.org/wiki/Information_retrieval/\">https://en.wikipedia.org/wiki/Information_retrieval/</a></td><td><strong>Relevant</strong></td></tr><tr><td>2</td><td><a href=\"http://www.dictionary.com/browse/information\">http://www.dictionary.com/browse/information</a></td><td><em>Not relevant</em></td></tr><tr><td>3</td><td><a href=\"https://nlp.stanford.edu/IR-book/\">https://nlp.stanford.edu/IR-book/</a></td><td><strong>Relevant</strong></td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "With two assessments:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table><tr><th>#</th><th>Result URL</th><th>Assessment 1</th><th>Assessment 2</th></tr><tr><td>1</td><td><a href=\"https://en.wikipedia.org/wiki/Information_retrieval/\">https://en.wikipedia.org/wiki/Information_retrieval/</a></td><td><strong>Relevant</strong></td><td><em>Not relevant</em></td></tr><tr><td>2</td><td><a href=\"http://www.dictionary.com/browse/information\">http://www.dictionary.com/browse/information</a></td><td><em>Not relevant</em></td><td><em>Not relevant</em></td></tr><tr><td>3</td><td><a href=\"https://nlp.stanford.edu/IR-book/\">https://nlp.stanford.edu/IR-book/</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"Just a list of URLs:\")\n",
    "display_results(urls)\n",
    "\n",
    "print(\"With one assessment:\")\n",
    "display_results(urls, my_assessment)\n",
    "\n",
    "print(\"With two assessments:\")\n",
    "display_results(urls, my_assessment, another_assessment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to perform an actual evaluation, which will involve a substantial amount of manual work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Your name:** Thomas Norton"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 1\n",
    "\n",
    "Think up and formulate a information need (for example in the field of Computer Science or Medicine) for which you think the answer can be found in scientific publications. On page 152 in the book an example of such an information need is shown: \"Information on whether drinking red wine is more effective at reducing the risk of heart attacks than white wine.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer:** Impact of dark chocolate on health"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, write down specifically what documents have to look like to satisfy your information need. For example if your information need is about finding an overview of different cancer types, you could state that a document would need to list at least ten types of cancer to satisfy your information need (among other criteria). Write this down as a protocol with rules and examples. For example, such a protocol could state that at least three out of five given criteria have to be fulfilled for a document to be considered relevant for the information need, and then specify the criteria. Or your protocol could have the form of a sequence of rules, where each rule lets you either label the document as relevant or not relevant, or proceed with the next rule. Such rules and criteria can, for example, be about the general topic of the paper, the concepts mentioned in it, the covered relations between concepts, the type of publication (research paper, overview paper, etc.), the number of references, the types of contained diagrams, and so on, depending on your specified information need."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer:** The paper must mention 4/5 of the following terms:\n",
    "- dark chocolate\n",
    "- health\n",
    "- cacao\n",
    "- impact\n",
    "- health benefits\n",
    "\n",
    "\n",
    "The paper also must have more than 4 references."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 2\n",
    "\n",
    "Formulate a keyword query that represents the information need. For the example on page 152 in the book (see above), the example query \"wine AND red AND white AND heart AND attack AND effective\" is given. (You don't need to use connectors like \"AND\", but if you do, make first sure your chosen search engines below actually support them.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer:** [_Write your keyword query here_]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then submit your query to **two** of the following academic search engines:\n",
    "\n",
    "- [Google Scholar](https://scholar.google.com) (all science disciplines)\n",
    "- [Semantic Scholar](https://www.semanticscholar.org) (all science disciplines)\n",
    "- [PubMed Search](https://www.ncbi.nlm.nih.gov/pubmed) (Life Sciences / biomedicine)\n",
    "\n",
    "The right choice of two from the three search engine depends on the topic of your information need. If your information need is in the Life Sciences and biomedicine, it's probably best to include PubMed Search, but otherwise you should pick Google Scholar and Semantic Scholar.\n",
    "\n",
    "Extract a list of the top 10 URLs of the lists of each of the search engines given the query. To be ensure that your results are reproducible, it is advised to use the private mode of your browser. Try to access the resulting publications. For the publications where that is not possible (because of dead links or because the publication is pay-walled even within the VU network), exclude them from the list and add more publications to the end of your list (that is, append results number 11, then 12, etc. to ensure you have two lists of 10 publications each). In order to deal with paywalls, you should try accessing the articles from the VU network, use\n",
    "[UBVU Off-Campus\n",
    "Access](http://www.ub.vu.nl.vu-nl.idm.oclc.org/nl/faciliteiten/toegang-buiten-de-campus/index.aspx), or try to find the respective documents from alternative sources (Google Scholar, for example, is very good at finding free PDFs of articles). If you get fewer than 10 results for one of the search engines, modify the keyword query above to make it more inclusive, and then redo the steps of this task.\n",
    "\n",
    "Store your two lists of URLs in the form of Python lists as introduced above. Then, use the `display_results` function to nicely display them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th>#</th><th>Result URL</th></tr><tr><td>1</td><td><a href=\"https://academic.oup.com/ajcn/article/87/4/872/4633444\">https://academic.oup.com/ajcn/article/87/4/872/4633444</a></td></tr><tr><td>2</td><td><a href=\"https://heart.bmj.com/content/92/1/119.short\">https://heart.bmj.com/content/92/1/119.short</a></td></tr><tr><td>3</td><td><a href=\"https://www.frontiersin.org/articles/10.3389/fnut.2017.00043/full\">https://www.frontiersin.org/articles/10.3389/fnut.2017.00043/full</a></td></tr><tr><td>4</td><td><a href=\"https://www.nature.com/articles/ejcn201164\">https://www.nature.com/articles/ejcn201164</a></td></tr><tr><td>5</td><td><a href=\"https://www.sciencedirect.com/science/article/pii/S1537189115001135\">https://www.sciencedirect.com/science/article/pii/S1537189115001135</a></td></tr><tr><td>6</td><td><a href=\"https://pubs.acs.org/doi/full/10.1021/pr900607v\">https://pubs.acs.org/doi/full/10.1021/pr900607v</a></td></tr><tr><td>7</td><td><a href=\"https://link.springer.com/article/10.1007/s11936-015-0419-5\">https://link.springer.com/article/10.1007/s11936-015-0419-5</a></td></tr><tr><td>8</td><td><a href=\"https://www.ahajournals.org/doi/full/10.1161/circulationaha.107.738070\">https://www.ahajournals.org/doi/full/10.1161/circulationaha.107.738070</a></td></tr><tr><td>9</td><td><a href=\"https://www.sciencedirect.com/science/article/pii/B9780123984562000190\">https://www.sciencedirect.com/science/article/pii/B9780123984562000190</a></td></tr><tr><td>10</td><td><a href=\"https://www.ahajournals.org/doi/full/10.1161/JAHA.116.005162\">https://www.ahajournals.org/doi/full/10.1161/JAHA.116.005162</a></td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><tr><th>#</th><th>Result URL</th></tr><tr><td>1</td><td><a href=\"https://pubmed.ncbi.nlm.nih.gov/29417473/\">https://pubmed.ncbi.nlm.nih.gov/29417473/</a></td></tr><tr><td>2</td><td><a href=\"https://pubmed.ncbi.nlm.nih.gov/21470061/\">https://pubmed.ncbi.nlm.nih.gov/21470061/</a></td></tr><tr><td>3</td><td><a href=\"https://pubmed.ncbi.nlm.nih.gov/21665390/\">https://pubmed.ncbi.nlm.nih.gov/21665390/</a></td></tr><tr><td>4</td><td><a href=\"https://pubmed.ncbi.nlm.nih.gov/35265879/\">https://pubmed.ncbi.nlm.nih.gov/35265879/</a></td></tr><tr><td>5</td><td><a href=\"https://pubmed.ncbi.nlm.nih.gov/27363823/\">https://pubmed.ncbi.nlm.nih.gov/27363823/</a></td></tr><tr><td>6</td><td><a href=\"https://pubmed.ncbi.nlm.nih.gov/28513635/\">https://pubmed.ncbi.nlm.nih.gov/28513635/</a></td></tr><tr><td>7</td><td><a href=\"https://pubmed.ncbi.nlm.nih.gov/28572069/\">https://pubmed.ncbi.nlm.nih.gov/28572069/</a></td></tr><tr><td>8</td><td><a href=\"https://pubmed.ncbi.nlm.nih.gov/30082608/\">https://pubmed.ncbi.nlm.nih.gov/30082608/</a></td></tr><tr><td>9</td><td><a href=\"https://pubmed.ncbi.nlm.nih.gov/29553824/\">https://pubmed.ncbi.nlm.nih.gov/29553824/</a></td></tr><tr><td>10</td><td><a href=\"https://pubmed.ncbi.nlm.nih.gov/34391456/\">https://pubmed.ncbi.nlm.nih.gov/34391456/</a></td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create two of the lists below, depending on your chosen engines:\n",
    "\n",
    "urls_google = ['https://academic.oup.com/ajcn/article/87/4/872/4633444','https://heart.bmj.com/content/92/1/119.short','https://www.frontiersin.org/articles/10.3389/fnut.2017.00043/full','https://www.nature.com/articles/ejcn201164','https://www.sciencedirect.com/science/article/pii/S1537189115001135','https://pubs.acs.org/doi/full/10.1021/pr900607v','https://link.springer.com/article/10.1007/s11936-015-0419-5','https://www.ahajournals.org/doi/full/10.1161/circulationaha.107.738070','https://www.sciencedirect.com/science/article/pii/B9780123984562000190','https://www.ahajournals.org/doi/full/10.1161/JAHA.116.005162']\n",
    "#urls_semantic = ...\n",
    "urls_pubmed = ['https://pubmed.ncbi.nlm.nih.gov/29417473/','https://pubmed.ncbi.nlm.nih.gov/21470061/','https://pubmed.ncbi.nlm.nih.gov/21665390/','https://pubmed.ncbi.nlm.nih.gov/35265879/','https://pubmed.ncbi.nlm.nih.gov/27363823/','https://pubmed.ncbi.nlm.nih.gov/28513635/','https://pubmed.ncbi.nlm.nih.gov/28572069/','https://pubmed.ncbi.nlm.nih.gov/30082608/','https://pubmed.ncbi.nlm.nih.gov/29553824/','https://pubmed.ncbi.nlm.nih.gov/34391456/']\n",
    "\n",
    "# Call display_results here\n",
    "display_results(urls_google)\n",
    "display_results(urls_pubmed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 3\n",
    "\n",
    "Then, find a fellow student who will **independently**\n",
    "assess the results as \"relevant\" or \"not relevant\" using the protocol that you\n",
    "have defined above, and also help (at least) one other student for his/her\n",
    "assessment. Write down their names here:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Name of the student who assesses my results:** Rafael Pinto\n",
    "\n",
    "**Name of the student who I help to assess his/her results:** Rafael Pinto"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show to the other assessor everything you have written down above for Tasks 1 and 2 (and you might also want to give him/her the PDFs you got for these papers to simplify the process).\n",
    "\n",
    "You as assessors need to stick to the protocol you made in Task 1 and should not discuss with each other, especially when you doubt whether a result is relevant or not. Write down your assessments as lists of relevance values, as introduced above, and make sure they correctly map to the URLs by displaying them together with the `display_results` function.\n",
    "\n",
    "To avoid problems with extreme results, mark in each list at least one paper as 'relevant' and at least one paper as 'not relevant'. That is, if all papers seem relevant, mark the one that seems least relevant 'not relevant', and conversely, if none of the papers seem relevant, mark the one that seems a bit more relevant than the others as 'relevant'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th>#</th><th>Result URL</th><th>Assessment 1</th><th>Assessment 2</th></tr><tr><td>1</td><td><a href=\"https://academic.oup.com/ajcn/article/87/4/872/4633444\">https://academic.oup.com/ajcn/article/87/4/872/4633444</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr><tr><td>2</td><td><a href=\"https://heart.bmj.com/content/92/1/119.short\">https://heart.bmj.com/content/92/1/119.short</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr><tr><td>3</td><td><a href=\"https://www.frontiersin.org/articles/10.3389/fnut.2017.00043/full\">https://www.frontiersin.org/articles/10.3389/fnut.2017.00043/full</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr><tr><td>4</td><td><a href=\"https://www.nature.com/articles/ejcn201164\">https://www.nature.com/articles/ejcn201164</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr><tr><td>5</td><td><a href=\"https://www.sciencedirect.com/science/article/pii/S1537189115001135\">https://www.sciencedirect.com/science/article/pii/S1537189115001135</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr><tr><td>6</td><td><a href=\"https://pubs.acs.org/doi/full/10.1021/pr900607v\">https://pubs.acs.org/doi/full/10.1021/pr900607v</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr><tr><td>7</td><td><a href=\"https://link.springer.com/article/10.1007/s11936-015-0419-5\">https://link.springer.com/article/10.1007/s11936-015-0419-5</a></td><td><strong>Relevant</strong></td><td><em>Not relevant</em></td></tr><tr><td>8</td><td><a href=\"https://www.ahajournals.org/doi/full/10.1161/circulationaha.107.738070\">https://www.ahajournals.org/doi/full/10.1161/circulationaha.107.738070</a></td><td><em>Not relevant</em></td><td><em>Not relevant</em></td></tr><tr><td>9</td><td><a href=\"https://www.sciencedirect.com/science/article/pii/B9780123984562000190\">https://www.sciencedirect.com/science/article/pii/B9780123984562000190</a></td><td><em>Not relevant</em></td><td><em>Not relevant</em></td></tr><tr><td>10</td><td><a href=\"https://www.ahajournals.org/doi/full/10.1161/JAHA.116.005162\">https://www.ahajournals.org/doi/full/10.1161/JAHA.116.005162</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><tr><th>#</th><th>Result URL</th><th>Assessment 1</th><th>Assessment 2</th></tr><tr><td>1</td><td><a href=\"https://pubmed.ncbi.nlm.nih.gov/29417473/\">https://pubmed.ncbi.nlm.nih.gov/29417473/</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr><tr><td>2</td><td><a href=\"https://pubmed.ncbi.nlm.nih.gov/21470061/\">https://pubmed.ncbi.nlm.nih.gov/21470061/</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr><tr><td>3</td><td><a href=\"https://pubmed.ncbi.nlm.nih.gov/21665390/\">https://pubmed.ncbi.nlm.nih.gov/21665390/</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr><tr><td>4</td><td><a href=\"https://pubmed.ncbi.nlm.nih.gov/35265879/\">https://pubmed.ncbi.nlm.nih.gov/35265879/</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr><tr><td>5</td><td><a href=\"https://pubmed.ncbi.nlm.nih.gov/27363823/\">https://pubmed.ncbi.nlm.nih.gov/27363823/</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr><tr><td>6</td><td><a href=\"https://pubmed.ncbi.nlm.nih.gov/28513635/\">https://pubmed.ncbi.nlm.nih.gov/28513635/</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr><tr><td>7</td><td><a href=\"https://pubmed.ncbi.nlm.nih.gov/28572069/\">https://pubmed.ncbi.nlm.nih.gov/28572069/</a></td><td><em>Not relevant</em></td><td><em>Not relevant</em></td></tr><tr><td>8</td><td><a href=\"https://pubmed.ncbi.nlm.nih.gov/30082608/\">https://pubmed.ncbi.nlm.nih.gov/30082608/</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr><tr><td>9</td><td><a href=\"https://pubmed.ncbi.nlm.nih.gov/29553824/\">https://pubmed.ncbi.nlm.nih.gov/29553824/</a></td><td><strong>Relevant</strong></td><td><em>Not relevant</em></td></tr><tr><td>10</td><td><a href=\"https://pubmed.ncbi.nlm.nih.gov/34391456/\">https://pubmed.ncbi.nlm.nih.gov/34391456/</a></td><td><strong>Relevant</strong></td><td><strong>Relevant</strong></td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 0 = not relevant; 1 = relevant\n",
    "\n",
    "# You only need to create 4 of the following 6 lists, again depending on which search engines you chose.\n",
    "\n",
    "# Assessment 1 is from you:\n",
    "\n",
    "assessment1_google = [1,1,1,1,1,1,1,0,0,1]\n",
    "#assessment1_semantic = ...\n",
    "assessment1_pubmed = [1,1,1,1,1,1,0,1,1,1]\n",
    "\n",
    "# Assessment 2 is from your fellow student (don't show him/her your own assessment!):\n",
    "\n",
    "assessment2_google = [1,1,1,1,1,1,0,0,0,1]\n",
    "#assessment2_semantic = ...\n",
    "assessment2_pubmed = [1,1,1,1,1,1,0,1,0,1]\n",
    "\n",
    "# Call display_results here\n",
    "display_results(urls_google, assessment1_google, assessment2_google)\n",
    "display_results(urls_pubmed, assessment1_pubmed, assessment2_pubmed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 4\n",
    "\n",
    "Compute Cohen's kappa to quantify how much the two assessors agreed. Use the function `cohen_kappa_score` demonstrated above to calculate two times the inter-annotator agreement (once for each of the two search engines), and print out the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Kappa for Google Scholar: 0.736842105263158\n",
      "Kappa for PubMed: 0.6153846153846154\n"
     ]
    }
   ],
   "source": [
    "# Add your code here:\n",
    "\n",
    "kappa_google = cohen_kappa_score(assessment1_google,assessment2_google)\n",
    "#kappa_semantic = ...\n",
    "kappa_pubmed = cohen_kappa_score(assessment1_pubmed,assessment2_pubmed)\n",
    "\n",
    "print(\"Kappa for Google Scholar:\", kappa_google)\n",
    "#print(\"Kappa for Semantic Scholar:\", kappa_semantic)\n",
    "print(\"Kappa for PubMed:\", kappa_pubmed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Explain whether the agreement can be considered high or not, based on the interpretation table on [this Wikipedia page](https://en.wikipedia.org/wiki/Fleiss'_kappa#Interpretation) (this Wikipedia page is about a different type of kappa but the interpretation table can also be used for Cohen's kappa)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer:** The Cohen's kappa score for Google Scholar is around 0.73, which is quite high. This means that the agreement between assessor number 1 and assessor number 2 is substantial. The Cohen's kappa score for PubMed was just over 0.6, which means there is moderate to substantial agreement between assessor 1 and assessor 2. Both of these two scores are quite good."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 5\n",
    "\n",
    "Define a function called `precision_at_n` that calculates Precision@n as described in the lecture slides, which takes as input an assessment list and a value for _n_ and returns the respective Precision@n value. Run this function to calculate Precision@10 (that is, n=10) on all four assessments (two assessors and two search engines)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision@10 Google - Assessor 1:  0.8\n",
      "Precision@10 Google - Assessor 2:  0.7\n",
      "Precision@10 PubMed - Assessor 1:  0.9\n",
      "Precision@10 PubMed - Assessor 2:  0.8\n"
     ]
    }
   ],
   "source": [
    "# Add your code here:\n",
    "\n",
    "def precision_at_n(assessment_list, n):\n",
    "    count = 0\n",
    "    for assessment in assessment_list:\n",
    "        count += assessment\n",
    "    return count/n\n",
    "\n",
    "\n",
    "# Print out Precision@10 for all assessments here\n",
    "print('Precision@10 Google - Assessor 1: ', precision_at_n(assessment1_google,10))\n",
    "print('Precision@10 Google - Assessor 2: ', precision_at_n(assessment2_google,10))\n",
    "print('Precision@10 PubMed - Assessor 1: ', precision_at_n(assessment1_pubmed,10))\n",
    "print('Precision@10 PubMed - Assessor 2: ', precision_at_n(assessment2_pubmed,10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Explain what these specific Precision@10 results tell us (or don't tell us) about the quality of the two search engines for your particular information need. You can also refer to the results of Task 4 if necessary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer:** These precision@10 results show us that both assessors agreed that the precision of Google Scholar and PubMed was quite high, with results all being between 0.7 and 0.9. What is interesting is that for both assessors, PubMed scored higher by 0.1, meaning that according to the criteria and the assessors, PubMed performed slightly better than Google Scholar. This is possibly due to the fact that PubMed is more Life Sciences/Biomedically oriented, which is what this query was about. Another thing that these results show is that Assessor 1 (me) was more lenient with how relevant articles was than Assessor 2. \n",
    "\n",
    "When we compare these results to the Cohen's Kappa score, we see that despite scoring higher in this precision@10 exercise, PubMed's results were less agreed upon by the two assessors, leading to a slightly lower agreement score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Submission"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Submit the answers to the assignment via Canvas as a modified version of this Notebook file (file with `.ipynb` extension) that includes your code and your answers.\n",
    "\n",
    "Before submitting, restart the kernel and re-run the complete code (**Kernel > Restart & Run All**), and then check whether your assignment code still works as expected.\n",
    "\n",
    "Don't forget to add your name, and remember that the assignments have to be done **individually**, and that code sharing or copying are **strictly forbidden** and will be punished."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.1 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  },
  "vscode": {
   "interpreter": {
    "hash": "397704579725e15f5c7cb49fe5f0341eb7531c82d19f2c29d197e8b64ab5776b"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}