make_dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building the dataset of research papers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [Entrez](http://biopython.org/DIST/docs/api/Bio.Entrez-module.html) module, a part of the [Biopython](http://biopython.org/) library, will be used to interface with [PubMed](http://www.ncbi.nlm.nih.gov/pubmed).<br>\n",
    "You can download Biopython from [here](http://biopython.org/wiki/Download).\n",
    "\n",
    "In this notebook we will be covering several of the steps taken in the [Biopython Tutorial](http://biopython.org/DIST/docs/tutorial/Tutorial.html), specifically in [Chapter 9  Accessing NCBI’s Entrez databases](http://biopython.org/DIST/docs/tutorial/Tutorial.html#htoc109)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: biopython in /opt/conda/lib/python3.9/site-packages (1.79)\n",
      "Requirement already satisfied: numpy in /opt/conda/lib/python3.9/site-packages (from biopython) (1.20.3)\n"
     ]
    }
   ],
   "source": [
    "!pip install biopython"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "from Bio import Entrez\n",
    "\n",
    "# NCBI requires you to set your email address to make use of NCBI's E-utilities\n",
    "Entrez.email = \"Your.Name.Here@example.org\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The datasets will be saved as serialized Python objects, compressed with bzip2.\n",
    "Saving/loading them will therefore require the [pickle](http://docs.python.org/3/library/pickle.html) and [bz2](http://docs.python.org/3/library/bz2.html) modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle, bz2, os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EInfo: Obtaining information about the Entrez databases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "# accessing extended information about the PubMed database\n",
    "pubmed = Entrez.read( Entrez.einfo(db=\"pubmed\"), validate=False )[u'DbInfo']\n",
    "\n",
    "# list of possible search fields for use with ESearch:\n",
    "search_fields = { f['Name']:f['Description'] for f in pubmed[\"FieldList\"] }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In search_fields, we find 'TIAB' ('Free text associated with Abstract/Title') as a possible search field to use in searches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ALL': 'All terms from all searchable fields',\n",
       " 'UID': 'Unique number assigned to publication',\n",
       " 'FILT': 'Limits the records',\n",
       " 'TITL': 'Words in title of publication',\n",
       " 'WORD': 'Free text associated with publication',\n",
       " 'MESH': 'Medical Subject Headings assigned to publication',\n",
       " 'MAJR': 'MeSH terms of major importance to publication',\n",
       " 'AUTH': 'Author(s) of publication',\n",
       " 'JOUR': 'Journal abbreviation of publication',\n",
       " 'AFFL': \"Author's institutional affiliation and address\",\n",
       " 'ECNO': 'EC number for enzyme or CAS registry number',\n",
       " 'SUBS': 'CAS chemical name or MEDLINE Substance Name',\n",
       " 'PDAT': 'Date of publication',\n",
       " 'EDAT': 'Date publication first accessible through Entrez',\n",
       " 'VOL': 'Volume number of publication',\n",
       " 'PAGE': 'Page number(s) of publication',\n",
       " 'PTYP': 'Type of publication (e.g., review)',\n",
       " 'LANG': 'Language of publication',\n",
       " 'ISS': 'Issue number of publication',\n",
       " 'SUBH': 'Additional specificity for MeSH term',\n",
       " 'SI': 'Cross-reference from publication to other databases',\n",
       " 'MHDA': 'Date publication was indexed with MeSH terms',\n",
       " 'TIAB': 'Free text associated with Abstract/Title',\n",
       " 'OTRM': 'Other terms associated with publication',\n",
       " 'INVR': 'Investigator',\n",
       " 'COLN': 'Corporate Author of publication',\n",
       " 'CNTY': 'Country of publication',\n",
       " 'PAPX': 'MeSH pharmacological action pre-explosions',\n",
       " 'GRNT': 'NIH Grant Numbers',\n",
       " 'MDAT': 'Date of last modification',\n",
       " 'CDAT': 'Date of completion',\n",
       " 'PID': 'Publisher ID',\n",
       " 'FAUT': 'First Author of publication',\n",
       " 'FULL': 'Full Author Name(s) of publication',\n",
       " 'FINV': 'Full name of investigator',\n",
       " 'TT': 'Words in transliterated title of publication',\n",
       " 'LAUT': 'Last Author of publication',\n",
       " 'PPDT': 'Date of print publication',\n",
       " 'EPDT': 'Date of Electronic publication',\n",
       " 'LID': 'ELocation ID',\n",
       " 'CRDT': 'Date publication first accessible through Entrez',\n",
       " 'BOOK': 'ID of the book that contains the document',\n",
       " 'ED': \"Section's Editor\",\n",
       " 'ISBN': 'ISBN',\n",
       " 'PUBN': \"Publisher's name\",\n",
       " 'AUCL': 'Author Cluster ID',\n",
       " 'EID': 'Extended PMID',\n",
       " 'DSO': 'Additional text from the summary',\n",
       " 'AUID': 'Author Identifier',\n",
       " 'PS': 'Personal Name as Subject',\n",
       " 'COIS': 'Conflict of Interest Statements'}"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "search_fields"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ESearch: Searching the Entrez databases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To have a look at the kind of data we get when searching the database, we'll perform a search for papers authored by Haasdijk:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Count': '39', 'RetMax': '20', 'RetStart': '0', 'IdList': ['33501027', '33501026', '33500899', '29311830', '28513205', '28513201', '28323435', '28140628', '26933487', '24977986', '24901702', '24852945', '24708899', '24252306', '23580075', '23144668', '22174697', '22154920', '21870131', '21760539'], 'TranslationSet': [], 'TranslationStack': [{'Term': 'Haasdijk E[Author]', 'Field': 'Author', 'Count': '39', 'Explode': 'N'}, 'GROUP'], 'QueryTranslation': 'Haasdijk E[Author]'}"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_authors = ['Haasdijk E']\n",
    "example_search = Entrez.read( Entrez.esearch( db=\"pubmed\", term=' AND '.join([a+'[AUTH]' for a in example_authors]) ) )\n",
    "example_search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how the result being produced is not in Python's native string format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Bio.Entrez.Parser.StringElement"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type( example_search['IdList'][0] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The part of the query's result we are most interested in is accessible through"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[33501027, 33501026, 33500899, 29311830, 28513205, 28513201, 28323435, 28140628, 26933487, 24977986, 24901702, 24852945, 24708899, 24252306, 23580075, 23144668, 22174697, 22154920, 21870131, 21760539]\n"
     ]
    }
   ],
   "source": [
    "example_ids = [ int(id) for id in example_search['IdList'] ]\n",
    "print(example_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PubMed IDs dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now assemble a dataset comprised of research articles containing the given keyword, in either their titles or abstracts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "search_term = 'cognition'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "Ids_file = 'data/' + search_term + '_Ids.pkl.bz2'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "102128 documents contain the search term \"cognition\".\n"
     ]
    }
   ],
   "source": [
    "if os.path.exists( Ids_file ):\n",
    "    Ids = pickle.load( bz2.BZ2File( Ids_file, 'rb' ) )\n",
    "else:\n",
    "    # determine the number of hits for the search term\n",
    "    search = Entrez.read( Entrez.esearch( db=\"pubmed\", term=search_term+'[TIAB]', retmax=0 ) )\n",
    "    total = int( search['Count'] )\n",
    "    \n",
    "    # `Ids` will be incrementally assembled, by performing multiple queries,\n",
    "    # each returning at most `retrieve_per_query` entries.\n",
    "    Ids_str = []\n",
    "    retrieve_per_query = 10000\n",
    "    \n",
    "    for start in range( 0, total, retrieve_per_query ):\n",
    "        print('Fetching IDs of results [%d,%d]' % ( start, start+retrieve_per_query ) )\n",
    "        s = Entrez.read( Entrez.esearch( db=\"pubmed\", term=search_term+'[TIAB]', retstart=start, retmax=retrieve_per_query ) )\n",
    "        Ids_str.extend( s[ u'IdList' ] )\n",
    "    \n",
    "    # convert Ids to integers (and ensure that the conversion is reversible)\n",
    "    Ids = [ int(id) for id in Ids_str ]\n",
    "    \n",
    "    for (id_str, id_int) in zip(Ids_str, Ids):\n",
    "        if str(id_int) != id_str:\n",
    "            raise Exception('Conversion of PubMed ID %s from string to integer it not reversible.' % id_str )\n",
    "    \n",
    "    # Remove IDs that would cause problems below:\n",
    "    Ids.remove(31430077)\n",
    "    Ids.remove(30566730)\n",
    "    Ids.remove(30561281)\n",
    "    Ids.remove(29127756)\n",
    "    Ids.remove(29031035)\n",
    "    Ids.remove(28155421)\n",
    "    Ids.remove(27936254)\n",
    "    Ids.remove(27732103)\n",
    "    Ids.remove(27570959)\n",
    "    Ids.remove(27570956)\n",
    "    Ids.remove(27570952)\n",
    "    Ids.remove(27570950)\n",
    "    Ids.remove(27568126)\n",
    "    Ids.remove(26218237)\n",
    "    Ids.remove(25590551)\n",
    "    Ids.remove(25590545)\n",
    "    Ids.remove(25367166) \n",
    "    Ids.remove(25148547) \n",
    "    # Save list of Ids\n",
    "    pickle.dump( Ids, bz2.BZ2File( Ids_file, 'wb' ) )\n",
    "    \n",
    "total = len( Ids )\n",
    "print('%d documents contain the search term \"%s\".' % ( total, search_term ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Taking a look at what we just retrieved, here are the last 5 elements of the `Ids` list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[36220089, 36219911, 36219905, 36219802, 36219788]"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Ids[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## ESummary: Retrieving summaries from primary IDs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To have a look at the kind of metadata we get from a call to `Entrez.esummary()`, we now fetch the summary of one of Haasdijk's papers (using one of the PubMed IDs we obtained in the previous section:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Item\n",
      "\t []\n",
      "Id\n",
      "\t 36220089\n",
      "PubDate\n",
      "\t 2022 Oct 10\n",
      "EPubDate\n",
      "\t \n",
      "Source\n",
      "\t Curr Biol\n",
      "AuthorList\n",
      "\t ['Giurfa M']\n",
      "LastAuthor\n",
      "\t Giurfa M\n",
      "Title\n",
      "\t Pollinator cognition: Framing bee memories in an ecological context.\n",
      "Volume\n",
      "\t 32\n",
      "Issue\n",
      "\t 19\n",
      "Pages\n",
      "\t R1015-R1018\n",
      "LangList\n",
      "\t ['English']\n",
      "NlmUniqueID\n",
      "\t 9107782\n",
      "ISSN\n",
      "\t 0960-9822\n",
      "ESSN\n",
      "\t 1879-0445\n",
      "PubTypeList\n",
      "\t ['Journal Article']\n",
      "RecordStatus\n",
      "\t PubMed - indexed for MEDLINE\n",
      "PubStatus\n",
      "\t ppublish\n",
      "ArticleIds\n",
      "\t {'pubmed': ['36220089'], 'medline': [], 'pii': 'S0960-9822(22)01365-3', 'doi': '10.1016/j.cub.2022.08.043', 'rid': '36220089', 'eid': '36220089'}\n",
      "DOI\n",
      "\t 10.1016/j.cub.2022.08.043\n",
      "History\n",
      "\t {'pubmed': ['2022/10/12 06:00'], 'medline': ['2022/10/14 06:00'], 'entrez': '2022/10/11 19:03'}\n",
      "References\n",
      "\t []\n",
      "HasAbstract\n",
      "\t IntegerElement(1, attributes={})\n",
      "PmcRefCount\n",
      "\t IntegerElement(0, attributes={})\n",
      "FullJournalName\n",
      "\t Current biology : CB\n",
      "ELocationID\n",
      "\t doi: 10.1016/j.cub.2022.08.043\n",
      "SO\n",
      "\t 2022 Oct 10;32(19):R1015-R1018\n"
     ]
    }
   ],
   "source": [
    "example_paper = Entrez.read( Entrez.esummary(db=\"pubmed\", id='36220089') )[0]\n",
    "\n",
    "def print_dict( p ):\n",
    "    for k,v in p.items():\n",
    "        print(k)\n",
    "        print('\\t', v)\n",
    "\n",
    "print_dict(example_paper)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For now, we'll keep just some basic information for each paper: title, list of authors, publication year, and [DOI](https://en.wikipedia.org/wiki/Digital_object_identifier).\n",
    "\n",
    "In case you are not familiar with the DOI system, know that the paper above can be accessed through the link  `https://www.doi.org/` followed by the paper's DOI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Pollinator cognition: Framing bee memories in\\xa0an\\xa0ecological context.',\n",
       " ['Giurfa M'],\n",
       " 2022,\n",
       " '10.1016/j.cub.2022.08.043')"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "( example_paper['Title'], example_paper['AuthorList'], int(example_paper['PubDate'][:4]), example_paper['DOI'] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summaries dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to assemble a dataset containing the summaries of all the paper `Ids` we previously fetched.\n",
    "\n",
    "To reduce the memory footprint, and to ensure the saved datasets won't depend on Biopython being installed to be properly loaded, values returned by `Entrez.read()` will be converted to their corresponding native Python types. We start by defining a function for helping with the conversion of strings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "Summaries_file = 'data/' + search_term + '_Summaries.pkl.bz2'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "if os.path.exists( Summaries_file ):\n",
    "    Summaries = pickle.load( bz2.BZ2File( Summaries_file, 'rb' ) )\n",
    "else:\n",
    "    # `Summaries` will be incrementally assembled, by performing multiple queries,\n",
    "    # each returning at most `retrieve_per_query` entries.\n",
    "    Summaries = []\n",
    "    retrieve_per_query = 500\n",
    "    \n",
    "    print('Fetching Summaries of results: ')\n",
    "    for start in range( 0, len(Ids), retrieve_per_query ):\n",
    "        if (start % 10000 == 0):\n",
    "            print('')\n",
    "            print(start, end='')\n",
    "        else:\n",
    "            print('.', end='')\n",
    "        \n",
    "        # build comma separated string with the ids at indexes [start, start+retrieve_per_query)\n",
    "        query_ids = ','.join( [ str(id) for id in Ids[ start : start+retrieve_per_query ] ] )\n",
    "        \n",
    "        s = Entrez.read( Entrez.esummary( db=\"pubmed\", id=query_ids ) )\n",
    "        \n",
    "        # out of the retrieved data, we will keep only a tuple (title, authors, year, DOI), associated with the paper's id.\n",
    "        # (all values converted to native Python formats)\n",
    "        for p in s:\n",
    "            try:\n",
    "                f = [\n",
    "                    ( int( p['Id'] ), (\n",
    "                        str( p['Title'] ),\n",
    "                        [ str(a) for a in p['AuthorList'] ],\n",
    "                        int( p['PubDate'][:4] ),                # keeps just the publication year\n",
    "                        str( p.get('DOI', '') )            # papers for which no DOI is available get an empty string in their place\n",
    "                        ) )\n",
    "                    ]\n",
    "                Summaries.extend( f )\n",
    "            except ValueError as e:\n",
    "                print(\"\\nError with ID \" + p['Id'] + \": \" + str(e))\n",
    "                print(\"Manually remove this ID above and re-run code.\")\n",
    "    \n",
    "    # Save Summaries, as a dictionary indexed by Ids\n",
    "    Summaries = dict( Summaries )\n",
    "    \n",
    "    pickle.dump( Summaries, bz2.BZ2File( Summaries_file, 'wb' ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us take a look at the first 3 retrieved summaries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{36220089: ('Pollinator cognition: Framing bee memories in\\xa0an\\xa0ecological context.',\n",
       "  ['Giurfa M'],\n",
       "  2022,\n",
       "  '10.1016/j.cub.2022.08.043'),\n",
       " 36219911: ('Exposure to fine particulate matter constituents and cognitive function performance, potential mediation by sleep quality: A multicenter study among Chinese adults aged 40-89\\xa0years.',\n",
       "  ['Pan R',\n",
       "   'Zhang Y',\n",
       "   'Xu Z',\n",
       "   'Yi W',\n",
       "   'Zhao F',\n",
       "   'Song J',\n",
       "   'Sun Q',\n",
       "   'Du P',\n",
       "   'Fang J',\n",
       "   'Cheng J',\n",
       "   'Liu Y',\n",
       "   'Chen C',\n",
       "   'Lu Y',\n",
       "   'Li T',\n",
       "   'Su H',\n",
       "   'Shi X'],\n",
       "  2022,\n",
       "  '10.1016/j.envint.2022.107566'),\n",
       " 36219905: ('Chronic cannabis use affects cerebellum dependent visuomotor adaptation.',\n",
       "  ['Blithikioti C',\n",
       "   'Miquel L',\n",
       "   'Paniello B',\n",
       "   'Nuño L',\n",
       "   'Gual A',\n",
       "   'Ballester BR',\n",
       "   'Fernandez A',\n",
       "   'Herreros I',\n",
       "   'Verschure P',\n",
       "   'Balcells-Olivero M'],\n",
       "  2022,\n",
       "  '10.1016/j.jpsychires.2022.10.007')}"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{ id : Summaries[id] for id in Ids[:3] }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EFetch: Downloading full records from Entrez"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Entrez.efetch()` is the function that will allow us to obtain paper abstracts. Let us start by taking a look at the kind of data it returns when we query PubMed's database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "q = Entrez.read( Entrez.efetch(db=\"pubmed\", id='34648188', retmode=\"xml\") )['PubmedArticle']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`q` is a list, with each member corresponding to a queried id. Because here we only queried for one id, its results are then in `q[0]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(list, 1)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(q), len(q)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1\n",
    "At `q[0]` we find a dictionary containing two keys, the contents of which we print below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Bio.Entrez.Parser.DictionaryElement,\n",
       " dict_keys(['MedlineCitation', 'PubmedData']))"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(q[0]), q[0].keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ReferenceList\n",
      "\t [{'Reference': [{'Citation': \"Sadler AG, Booth BM, Mengeling MA, Doebbeling BN. Life span and repeated violence against women during military service: effects on health status and outpatient utilization. J Women's Health. 2004;13(7):799-811.\"}, {'Citation': 'Scherrer JF, Xian H, Kapp JMK, et\\xa0al. Association between exposure to childhood and lifetime traumatic events and lifetime pathological gambling in a twin cohort. J Nerv Ment Dis. 2007;195(1):72-78.'}, {'Citation': 'Katon JG, Lehavot K, Simpson TL, et\\xa0al. Adverse childhood experiences, military service, and adult health. Am J Prev Med. 2015;49(4):573-582.'}, {'Citation': 'Koenen KC, Stellman SD, Sommer Jr JF, Stellman JM. Persisting posttraumatic stress disorder symptoms and their relationship to functioning in Vietnam veterans: a 14-year follow-up. J Trauma Stress. 2008;21(1):49-57.'}, {'Citation': 'Kessler RC. Posttraumatic stress disorder: the burden to the individual and to society. J Clin Psychiatry. 2000;61(5):4-12.'}, {'Citation': 'Weeks WB, Kazis LE, Shen Y, et\\xa0al. Differences in health-related quality of life in rural and urban veterans. Am J Public Health. 2004;94(10):1762-1767.'}, {'Citation': 'Kondo K, Low A, Everson T, et\\xa0al. Health disparities in veterans: a map of the evidence. Med Care. 2017;55:S9-S15.'}, {'Citation': 'Elnitsky CA, Andresen EM, Clark ME, McGarity S, Hall CG, Kerns RD. Access to the US Department of Veterans Affairs health system: self-reported barriers to care among returnees of Operations Enduring Freedom and Iraqi Freedom. BMC Health Serv Res. 2013;13(1):1-10.'}, {'Citation': 'Pietrzak RH, Johnson DC, Goldstein MB, Malley JC, Southwick SM. Perceived stigma and barriers to mental health care utilization among OEF-OIF veterans. Psychiatr Serv. 2009;60(8):1118-1122.'}, {'Citation': \"Kehle-Forbes SM, Harwood EM, Spoont MR, Sayer NA, Gerould H, Murdoch M. Experiences with VHA care: a qualitative study of US women veterans with self-reported trauma histories. BMC Women's Health. 2017;17(1)1-8.\"}, {'Citation': \"Klap R, Darling JE, Hamilton AB, et\\xa0al. Prevalence of stranger harassment of women veterans at Veterans Affairs Medical Centers and impacts on delayed and missed care. Women's Health Issues. 2019;29(2):107-115.\"}, {'Citation': 'Mohr D, Cuijpers P, Lehman K. Supportive accountability: a model for providing human support to enhance adherence to eHealth interventions. J Med Internet Res. 2011;13(1):e30.'}, {'Citation': 'Andersson G, Cuijpers P. Internet-based and other computerized psychological treatments for adult depression: a meta-analysis. Cogn Behav Ther. 2009;38(4):196-205.'}, {'Citation': 'Rothman K, Roddy MK, Doss BD. Completion of a stand-alone versus coach-supported trial of a web-based program for distressed relationships. Fam Rel. 2019;68(4):375-389.'}, {'Citation': 'Erbe D, Eichert HC, Riper H, Ebert DD. Blending face-to-face and internet-based interventions for the treatment of mental disorders in adults: systematic review. J Med Internet Res. 2017;19(9):e306.'}, {'Citation': 'Hobfoll SE, Blais RK, Stevens NR, Walt L, Gengler R. Vets prevail online intervention reduces PTSD and depression in veterans with mild-to-moderate symptoms. J Consult Clin Psychol. 2016;84(1):31-42.'}, {'Citation': 'Lehavot K, Millard SP, Thomas RM, et\\xa0al. A randomized trial of an online, coach-assisted self-management PTSD intervention tailored for women veterans. J Consult Clin Psychol. 2021;89(2):134-142.'}, {'Citation': 'Belsher BE, Kuhn E, Maron D, et\\xa0al. A preliminary study of an Internet-based intervention for OEF/OIF veterans presenting for VA specialty PTSD care. J Trauma Stress. 2015;28(2):153-156.'}, {'Citation': 'Litz BT, Engel CC, Bryant RA, Papa A. A randomized, controlled proof-of-concept trial of an Internet-based, therapist-assisted self-management treatment for posttraumatic stress disorder. Am J Psychiatry. 2007;164(11):1676-1684.'}, {'Citation': 'Cloitre M, Cohen LR, Ortigo KM, Jackson C, Koenen KC. Treating Survivors of Childhood Abuse and Interpersonal Trauma: STAIR Narrative Therapy. Guilford Publications; 2020.'}, {'Citation': 'Cloitre M, Miranda R, Stovall-McClough KC, Han H. Beyond PTSD: emotion regulation and interpersonal problems as predictors of functional impairment in survivors of childhood abuse. Behav Ther. 2005;36(2):119-124.'}, {'Citation': 'Cloitre M, Khan C, Mackintosh MA, et\\xa0al. Emotion regulation mediates the relationship between ACES and physical and mental health. Psychol Trauma. 2019;11(1):82-89.'}, {'Citation': 'Bauer A, Amspoker AB, Fletcher TL, et\\xa0al. A resource building virtual care: programme improving symptoms and social functioning among female and male rural veterans. Eur J Psychotraumatol. 2021;12(1):1860357.'}, {'Citation': \"Maguen S, Cohen B, Ren L, Bosch J, Kimerling R, Seal K. Gender differences in military sexual trauma and mental health diagnoses among Iraq and Afghanistan veterans with posttraumatic stress disorder. Women's Health Issues. 2012;22(1):e61-e66.\"}, {'Citation': \"Skinner KM, Kressin N, Frayne S, et\\xa0al. The prevalence of military sexual assault among female Veterans' Administration outpatients. J Interpers Violence. 2000;15(3):291-310.\"}, {'Citation': 'Suris A, Lind L. Military sexual trauma: a review of prevalence and associated health consequences in veterans. Trauma Violence Abuse. 2008;9(4):250-269.'}, {'Citation': 'Prins A, Bovin MJ, Smolenski DJ, et\\xa0al. The primary care PTSD screen for DSM-5 (PC-PTSD-5): development and evaluation within a veteran primary care sample. J Gen Intern Med. 2016;31(10):1206-1211.'}, {'Citation': 'Kroenke K, Spitzer RL, Williams JB. The Patient Health Questionnaire-2: validity of a two-item depression screener. Med Care. 2003;11:1284-1292.'}, {'Citation': 'Serfaty M, Shafran R, Vickerstaff V, Aspden T. A pragmatic approach to measuring adherence in treatment delivery in psychotherapy. Cogn Behav Ther. 2020;49(5):347-360.'}, {'Citation': 'Weathers FW, Litz BT, Keane TM, Palmieri PA, Marx BP, Schnurr PP. The PTSD Checklist for DSM-5 (PCL-5). National Center for PTSD.'}, {'Citation': 'Bovin MJ, Marx BP, Weathers FW, et\\xa0al. Psychometric properties of the PTSD checklist for diagnostic and statistical manual of mental disorders-fifth edition (PCL-5) in veterans. Psychol Assess. 2016;28(11):1379-1391.'}, {'Citation': 'Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001;16(9):606-613.'}, {'Citation': 'Gratz KL, Roemer L. Multidimensional assessment of emotion regulation and dysregulation: development, factor structure, and initial validation of the difficulties in emotion regulation scale. J Psychopathol Behav Assess. 2004;26(1):41-54.'}, {'Citation': 'Barkham M, Hardy GE, Startup M. The IIP-32: a short version of the Inventory of Interpersonal Problems. Br J Clin Psychol. 1996;35(1):21-35.'}, {'Citation': 'Horvath AO, Greenberg LS. Development and validation of the Working Alliance Inventory. J Counsel Psychol. 1989;36(2):223-233.'}, {'Citation': 'Cohen J. The effect size. Statistical Power Analysis for the Behavioral Sciences. 2nd ed., 77-83. Hillsdale, New Jersey: Lawrence Erlbaum Associates; 1988.'}, {'Citation': \"Sobo EJ, Simmes DR, Landsverk JA, Kurtin PS. Rapid assessment with qualitative telephone interviews: lessons from an evaluation of California's healthy families program & Medi-Cal for children. Am J Eval. 2003;24(3):399-408.\"}, {'Citation': 'Curran GM, Pyne J, Fortney JC, et\\xa0al. Development and implementation of collaborative care for depression in HIV clinics. AIDS Care. 2011;23(12):1626-1636.'}, {'Citation': 'Acierno R, Knapp R, Tuerk P, et\\xa0al. A non-inferiority trial of prolonged exposure for posttraumatic stress disorder: in person versus home-based telehealth. Behav Res Ther. 2017;89:57-65.'}, {'Citation': 'Morland LA, Mackintosh MA, Greene CJ, et\\xa0al. Cognitive processing therapy for posttraumatic stress disorder delivered to rural veterans via telemental health: a randomized noninferiority clinical trial. J Clin Psychiatry. 2014;75(5):470-476.'}, {'Citation': 'Eysenbach G. The law of attrition. J Med Internet Res. 2005;7(1):e11.'}, {'Citation': 'Cullen W, Gulati G, Kelly BD. Mental health in the Covid-19 pandemic. QJM. 2020;113(5):311-312.'}], 'ReferenceList': [], 'Title': 'REFERENCES'}]\n",
      "History\n",
      "\t [DictElement({'Year': '2021', 'Month': '10', 'Day': '15', 'Hour': '6', 'Minute': '0'}, attributes={'PubStatus': 'pubmed'}), DictElement({'Year': '2022', 'Month': '9', 'Day': '24', 'Hour': '6', 'Minute': '0'}, attributes={'PubStatus': 'medline'}), DictElement({'Year': '2021', 'Month': '10', 'Day': '14', 'Hour': '12', 'Minute': '31'}, attributes={'PubStatus': 'entrez'})]\n",
      "PublicationStatus\n",
      "\t ppublish\n",
      "ArticleIdList\n",
      "\t [StringElement('34648188', attributes={'IdType': 'pubmed'}), StringElement('10.1111/jrh.12628', attributes={'IdType': 'doi'})]\n"
     ]
    }
   ],
   "source": [
    "print_dict( q[0][ 'PubmedData' ] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The key `'MedlineCitation'` maps into another dictionary. In that dictionary, most of the information is contained under the key `'Article'`. To minimize the clutter, below we show the contents of `'MedlineCitation'` excluding its `'Article'` member, and below that we then show the contents of `'Article'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SpaceFlightMission\n",
      "\t []\n",
      "CitationSubset\n",
      "\t ['IM']\n",
      "GeneralNote\n",
      "\t []\n",
      "KeywordList\n",
      "\t [ListElement([StringElement('access to care', attributes={'MajorTopicYN': 'N'}), StringElement('telemental health', attributes={'MajorTopicYN': 'N'}), StringElement('trauma', attributes={'MajorTopicYN': 'N'}), StringElement('veterans', attributes={'MajorTopicYN': 'N'}), StringElement('web-based treatment', attributes={'MajorTopicYN': 'N'})], attributes={'Owner': 'NOTNLM'})]\n",
      "OtherAbstract\n",
      "\t []\n",
      "OtherID\n",
      "\t []\n",
      "PMID\n",
      "\t 34648188\n",
      "DateCompleted\n",
      "\t {'Year': '2022', 'Month': '09', 'Day': '23'}\n",
      "DateRevised\n",
      "\t {'Year': '2022', 'Month': '10', 'Day': '03'}\n",
      "MedlineJournalInfo\n",
      "\t {'Country': 'England', 'MedlineTA': 'J Rural Health', 'NlmUniqueID': '8508122', 'ISSNLinking': '0890-765X'}\n",
      "MeshHeadingList\n",
      "\t [{'QualifierName': [], 'DescriptorName': StringElement('Female', attributes={'UI': 'D005260', 'MajorTopicYN': 'N'})}, {'QualifierName': [], 'DescriptorName': StringElement('Health Services Accessibility', attributes={'UI': 'D006297', 'MajorTopicYN': 'N'})}, {'QualifierName': [], 'DescriptorName': StringElement('Humans', attributes={'UI': 'D006801', 'MajorTopicYN': 'N'})}, {'QualifierName': [], 'DescriptorName': StringElement('Internet', attributes={'UI': 'D020407', 'MajorTopicYN': 'N'})}, {'QualifierName': [], 'DescriptorName': StringElement('Mentoring', attributes={'UI': 'D000071252', 'MajorTopicYN': 'Y'})}, {'QualifierName': [], 'DescriptorName': StringElement('Pilot Projects', attributes={'UI': 'D010865', 'MajorTopicYN': 'N'})}, {'QualifierName': [StringElement('psychology', attributes={'UI': 'Q000523', 'MajorTopicYN': 'N'}), StringElement('therapy', attributes={'UI': 'Q000628', 'MajorTopicYN': 'N'})], 'DescriptorName': StringElement('Stress Disorders, Post-Traumatic', attributes={'UI': 'D013313', 'MajorTopicYN': 'Y'})}, {'QualifierName': [StringElement('methods', attributes={'UI': 'Q000379', 'MajorTopicYN': 'N'})], 'DescriptorName': StringElement('Telemedicine', attributes={'UI': 'D017216', 'MajorTopicYN': 'Y'})}, {'QualifierName': [StringElement('psychology', attributes={'UI': 'Q000523', 'MajorTopicYN': 'N'})], 'DescriptorName': StringElement('Veterans', attributes={'UI': 'D014728', 'MajorTopicYN': 'Y'})}]\n"
     ]
    }
   ],
   "source": [
    "print_dict( { k:v for k,v in q[0][ 'MedlineCitation' ].items() if k!='Article' } )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ELocationID\n",
      "\t [StringElement('10.1111/jrh.12628', attributes={'EIdType': 'doi', 'ValidYN': 'Y'})]\n",
      "Language\n",
      "\t ['eng']\n",
      "ArticleDate\n",
      "\t [DictElement({'Year': '2021', 'Month': '10', 'Day': '14'}, attributes={'DateType': 'Electronic'})]\n",
      "Journal\n",
      "\t {'ISSN': StringElement('1748-0361', attributes={'IssnType': 'Electronic'}), 'JournalIssue': DictElement({'Volume': '38', 'Issue': '4', 'PubDate': {'Year': '2022', 'Month': '09'}}, attributes={'CitedMedium': 'Internet'}), 'Title': 'The Journal of rural health : official journal of the American Rural Health Association and the National Rural Health Care Association', 'ISOAbbreviation': 'J Rural Health'}\n",
      "ArticleTitle\n",
      "\t Increasing access to care for trauma-exposed rural veterans: A mixed methods outcome evaluation of a web-based skills training program with telehealth-delivered coaching.\n",
      "Pagination\n",
      "\t {'MedlinePgn': '740-747'}\n",
      "Abstract\n",
      "\t {'AbstractText': [StringElement('While rural veterans with trauma exposure report high rates of posttraumatic stress disorder (PTSD), depression, and functional impairment, utilization of health services is low. This pilot study used mixed qualitative and quantitative methods to evaluate the potential benefits of a transdiagnostic web-based skills training program paired with telehealth-delivered coaching to address a range of symptoms and functional difficulties. The study directed substantial outreach efforts to women veterans who had experienced military sexual trauma given their growing representation in the Veterans Healthcare Administration (VHA) and identified need for services.', attributes={'Label': 'PURPOSE'}), StringElement('Participants were 32 trauma-exposed veterans enrolled in rural-serving VHA facilities who screened positive for either PTSD or depression. Symptoms of PTSD, depression, emotion regulation, and interpersonal problems were assessed at baseline, midpoint, posttreatment, and 3-month follow-up. Veterans completed exit interviews to identify benefits and limitations of the program.', attributes={'Label': 'METHODS'}), StringElement(\"Intent-to-treat analyses revealed significant symptom reduction for all outcomes, with large to moderate effect sizes at 3-month follow-up. Outcomes did not differ by gender or military sexual trauma status. Veterans' rating of the therapeutic alliance was high and interview responses indicated that the presence of the coach was critical to success in the program.\", attributes={'Label': 'RESULTS'}), StringElement('This remotely delivered transdiagnostic intervention provided significant benefits across a range of symptoms and functional outcomes and was viewed positively by veterans. The results indicate that further research (ie, a randomized controlled trial) is warranted. Attention to the role of the coach as a means by which to increase engagement and retention in technology-delivered interventions is warranted.', attributes={'Label': 'CONCLUSION'})], 'CopyrightInformation': '© 2021 National Rural Health Association.'}\n",
      "AuthorList\n",
      "\t ListElement([DictElement({'AffiliationInfo': [{'Identifier': [], 'Affiliation': 'VA HSR&D Houston Center of Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey VA Medical Center, Houston, Texas, USA.'}, {'Identifier': [], 'Affiliation': 'Menninger Department of Psychiatry and Behavioral Sciences, Baylor College of Medicine, Houston, Texas, USA.'}, {'Identifier': [], 'Affiliation': 'VA South Central Mental Illness Research, Education and Clinical Center, Houston, Texas, USA.'}], 'Identifier': [], 'LastName': 'Fletcher', 'ForeName': 'Terri L', 'Initials': 'TL'}, attributes={'ValidYN': 'Y'}), DictElement({'AffiliationInfo': [{'Identifier': [], 'Affiliation': 'VA HSR&D Houston Center of Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey VA Medical Center, Houston, Texas, USA.'}, {'Identifier': [], 'Affiliation': 'VA South Central Mental Illness Research, Education and Clinical Center, Houston, Texas, USA.'}, {'Identifier': [], 'Affiliation': 'Baylor College of Medicine, Department of Medicine, Houston, Texas, USA.'}], 'Identifier': [], 'LastName': 'Amspoker', 'ForeName': 'Amber B', 'Initials': 'AB'}, attributes={'ValidYN': 'Y'}), DictElement({'AffiliationInfo': [{'Identifier': [], 'Affiliation': 'VA HSR&D Houston Center of Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey VA Medical Center, Houston, Texas, USA.'}, {'Identifier': [], 'Affiliation': 'VA South Central Mental Illness Research, Education and Clinical Center, Houston, Texas, USA.'}], 'Identifier': [], 'LastName': 'Wassef', 'ForeName': 'Miryam', 'Initials': 'M'}, attributes={'ValidYN': 'Y'}), DictElement({'AffiliationInfo': [{'Identifier': [], 'Affiliation': 'VA HSR&D Houston Center of Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey VA Medical Center, Houston, Texas, USA.'}, {'Identifier': [], 'Affiliation': 'Menninger Department of Psychiatry and Behavioral Sciences, Baylor College of Medicine, Houston, Texas, USA.'}, {'Identifier': [], 'Affiliation': 'VA South Central Mental Illness Research, Education and Clinical Center, Houston, Texas, USA.'}], 'Identifier': [], 'LastName': 'Hogan', 'ForeName': 'Julianna B', 'Initials': 'JB'}, attributes={'ValidYN': 'Y'}), DictElement({'AffiliationInfo': [{'Identifier': [], 'Affiliation': 'VA HSR&D Houston Center of Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey VA Medical Center, Houston, Texas, USA.'}, {'Identifier': [], 'Affiliation': 'VA South Central Mental Illness Research, Education and Clinical Center, Houston, Texas, USA.'}], 'Identifier': [], 'LastName': 'Helm', 'ForeName': 'Ashley', 'Initials': 'A'}, attributes={'ValidYN': 'Y'}), DictElement({'AffiliationInfo': [{'Identifier': [], 'Affiliation': 'VA New York Harbor Healthcare System, New York, New York, USA.'}], 'Identifier': [], 'LastName': 'Jackson', 'ForeName': 'Christie', 'Initials': 'C'}, attributes={'ValidYN': 'Y'}), DictElement({'AffiliationInfo': [{'Identifier': [], 'Affiliation': 'National Center for PTSD Dissemination and Training Division, VA Palo Alto Health Care System, Palo Alto, California, USA.'}], 'Identifier': [], 'LastName': 'Jacobs', 'ForeName': 'Adam', 'Initials': 'A'}, attributes={'ValidYN': 'Y'}), DictElement({'AffiliationInfo': [{'Identifier': [], 'Affiliation': 'National Center for PTSD Dissemination and Training Division, VA Palo Alto Health Care System, Palo Alto, California, USA.'}], 'Identifier': [], 'LastName': 'Shammet', 'ForeName': 'Rayan', 'Initials': 'R'}, attributes={'ValidYN': 'Y'}), DictElement({'AffiliationInfo': [{'Identifier': [], 'Affiliation': 'National Center for PTSD Dissemination and Training Division, VA Palo Alto Health Care System, Palo Alto, California, USA.'}], 'Identifier': [], 'LastName': 'Speicher', 'ForeName': 'Sarah', 'Initials': 'S'}, attributes={'ValidYN': 'Y'}), DictElement({'AffiliationInfo': [{'Identifier': [], 'Affiliation': 'VA HSR&D Houston Center of Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey VA Medical Center, Houston, Texas, USA.'}, {'Identifier': [], 'Affiliation': 'VA South Central Mental Illness Research, Education and Clinical Center, Houston, Texas, USA.'}, {'Identifier': [], 'Affiliation': 'Baylor College of Medicine, Department of Medicine, Houston, Texas, USA.'}], 'Identifier': [StringElement('0000-0002-2853-0113', attributes={'Source': 'ORCID'})], 'LastName': 'Lindsay', 'ForeName': 'Jan A', 'Initials': 'JA'}, attributes={'ValidYN': 'Y'}), DictElement({'AffiliationInfo': [{'Identifier': [], 'Affiliation': 'National Center for PTSD Dissemination and Training Division, VA Palo Alto Health Care System, Palo Alto, California, USA.'}, {'Identifier': [], 'Affiliation': 'Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, California, USA.'}], 'Identifier': [StringElement('0000-0001-8029-1570', attributes={'Source': 'ORCID'})], 'LastName': 'Cloitre', 'ForeName': 'Marylène', 'Initials': 'M'}, attributes={'ValidYN': 'Y'})], attributes={'CompleteYN': 'Y'})\n",
      "PublicationTypeList\n",
      "\t [StringElement('Journal Article', attributes={'UI': 'D016428'}), StringElement('Randomized Controlled Trial', attributes={'UI': 'D016449'}), StringElement(\"Research Support, U.S. Gov't, Non-P.H.S.\", attributes={'UI': 'D013486'})]\n"
     ]
    }
   ],
   "source": [
    "print_dict( q[0][ 'MedlineCitation' ][ 'Article' ] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A paper's abstract can therefore be accessed with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{34648188: 'While rural veterans with trauma exposure report high rates of posttraumatic stress disorder (PTSD), depression, and functional impairment, utilization of health services is low. This pilot study used mixed qualitative and quantitative methods to evaluate the potential benefits of a transdiagnostic web-based skills training program paired with telehealth-delivered coaching to address a range of symptoms and functional difficulties. The study directed substantial outreach efforts to women veterans who had experienced military sexual trauma given their growing representation in the Veterans Healthcare Administration (VHA) and identified need for services.'}"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{ int(q[0]['MedlineCitation']['PMID']) : str(q[0]['MedlineCitation']['Article']['Abstract']['AbstractText'][0]) }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of the ids in our dataset refer to books from the [NCBI Bookshelf](http://www.ncbi.nlm.nih.gov/books/), a collection of freely available, downloadable, on-line versions of selected biomedical books. For such ids, `Entrez.efetch()` returns a slightly different structure, where the keys `[u'BookDocument', u'PubmedBookData']` take the place of the `[u'MedlineCitation', u'PubmedData']` keys we saw above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Abstracts dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now assemble a dataset mapping paper ids to their abstracts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "Abstracts_file = 'data/' + search_term + '_Abstracts.pkl.bz2'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fetching Abstracts of results: \n",
      "\n",
      "0...................\n",
      "10000...................\n",
      "20000...................\n",
      "30000...................\n",
      "40000...................\n",
      "50000...................\n",
      "60000...................\n",
      "70000...................\n",
      "80000...................\n",
      "90000...................\n",
      "100000...."
     ]
    }
   ],
   "source": [
    "import http.client\n",
    "from collections import deque\n",
    "from xml.dom import minidom\n",
    "import re\n",
    "\n",
    "def ch(node, childtype):\n",
    "    return node.getElementsByTagName(childtype)[0]\n",
    "\n",
    "if os.path.exists( Abstracts_file ):\n",
    "    Abstracts = pickle.load( bz2.BZ2File( Abstracts_file, 'rb' ) )\n",
    "else:\n",
    "    # `Abstracts` will be incrementally assembled, by performing multiple queries,\n",
    "    # each returning at most `retrieve_per_query` entries.\n",
    "    Abstracts = deque()\n",
    "    retrieve_per_query = 500\n",
    "    \n",
    "    print('Fetching Abstracts of results: ')\n",
    "    for start in range( 0, len(Ids), retrieve_per_query ):\n",
    "        if (start % 10000 == 0):\n",
    "            print('')\n",
    "            print(start, end='')\n",
    "        else:\n",
    "            print('.', end='')\n",
    "        \n",
    "        # build comma separated string with the ids at indexes [start, start+retrieve_per_query)\n",
    "        query_ids = ','.join( [ str(id) for id in Ids[ start : start+retrieve_per_query ] ] )\n",
    "        \n",
    "        # issue requests to the server, until we get the full amount of data we expect\n",
    "        while True:\n",
    "            try:\n",
    "                #s = Entrez.read( Entrez.efetch(db=\"pubmed\", id=query_ids, retmode=\"xml\" ) )['PubmedArticle']\n",
    "                s = minidom.parse( Entrez.efetch(db=\"pubmed\", id=query_ids, retmode=\"xml\" ) ).getElementsByTagName(\"PubmedArticle\")\n",
    "            except http.client.IncompleteRead:\n",
    "                print('r', end='')\n",
    "                continue\n",
    "            break\n",
    "        \n",
    "        i = 0\n",
    "        for p in s:\n",
    "            abstr = ''\n",
    "            if (p.getElementsByTagName('MedlineCitation')):\n",
    "                citNode = ch(p,'MedlineCitation')\n",
    "                pmid = ch(citNode,'PMID').firstChild.data\n",
    "                articleNode = ch(citNode,'Article')\n",
    "                if (articleNode.getElementsByTagName('Abstract')):\n",
    "                    try:\n",
    "                        abstr = ch(ch(articleNode,'Abstract'),'AbstractText').firstChild.data\n",
    "                    except AttributeError:\n",
    "                        abstr = ch(ch(articleNode,'Abstract'),'AbstractText').toprettyxml(\"  \")\n",
    "                        abstr = re.sub(r\"\\s+\", \" \", re.sub(\"<[^>]*>\", \"\", abstr))\n",
    "            elif (p.getElementsByTagName('BookDocument')):\n",
    "                bookNode = ch(p,'BookDocument')\n",
    "                pmid = ch(bookNode,'PMID').firstChild.data\n",
    "                if (bookNode.getElementsByTagName('Abstract')):\n",
    "                    try:\n",
    "                        abstr = ch(ch(bookNode,'Abstract'),'AbstractText').firstChild.data\n",
    "                    except AttributeError:\n",
    "                        abstr = ch(ch(bookNode,'Abstract'),'AbstractText').toprettyxml(\"  \")\n",
    "                        abstr = re.sub(r\"\\s+\", \" \", re.sub(\"<[^>]*>\", \"\", abstr))\n",
    "            else:\n",
    "                raise Exception('Unrecognized record type, for id %d (keys: %s)' % (Ids[start+i], str(p.keys())) )\n",
    "            \n",
    "            Abstracts.append( (int(pmid), str(abstr)) )\n",
    "            i += 1\n",
    "    \n",
    "    # Save Abstracts, as a dictionary indexed by Ids\n",
    "    Abstracts = dict( Abstracts )\n",
    "    \n",
    "    pickle.dump( Abstracts, bz2.BZ2File( Abstracts_file, 'wb' ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Taking a look at one paper's abstract:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'While combination antiretroviral therapy (cART) has dramatically increased the life expectancy of people with HIV (PWH), nearly 50% develop HIV-associated neurocognitive disorders. This may be due to previously uncontrolled HIV viral replication, immune activation maintained by residual viral replication or activation from other sources, or cART-associated neurotoxicity. The aim of this study was to determine the effect of cART on cognition and neuroimaging biomarkers in PWH before and after initiation of cART compared with that in HIV-negative controls (HCs) and HIV elite controllers (ECs) who remain untreated.'"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Abstracts[36219802]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ELink: Searching for related items in NCBI Entrez"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To understand how to obtain paper citations with Entrez, we will first assemble a small set of PubMed IDs, and then query for their citations.\n",
    "To that end, we search here for papers published in the Nature journal with our given keyword."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['35896749', '35589843', '35545674', '35478240', '35355009', '35236988', '34937941', '34616074', '34616064', '34614503', '34599306', '34153974', '33790467', '33505022', '33473210', '33361808', '32999460', '32699411', '31915375', '31801999']"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CA_search_term = search_term+'[TIAB] AND Nature[JOUR]'\n",
    "CA_ids = Entrez.read( Entrez.esearch( db=\"pubmed\", term=CA_search_term ) )['IdList']\n",
    "CA_ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'35896749': ('Cortical feedback loops bind distributed representations of working memory.',\n",
       "  ['Voitov I', 'Mrsic-Flogel TD'],\n",
       "  '2022',\n",
       "  'Nature',\n",
       "  '10.1038/s41586-022-05014-3'),\n",
       " '35589843': ('People construct simplified mental representations to plan.',\n",
       "  ['Ho MK', 'Abel D', 'Correa CG', 'Littman ML', 'Cohen JD', 'Griffiths TL'],\n",
       "  '2022',\n",
       "  'Nature',\n",
       "  '10.1038/s41586-022-04743-9'),\n",
       " '35545674': ('Young CSF restores oligodendrogenesis and memory in aged mice via Fgf17.',\n",
       "  ['Iram T', 'Kern F', 'Kaur A', 'Myneni S', 'Morningstar AR', 'Shin H', 'Garcia MA', 'Yerra L', 'Palovics R', 'Yang AC', 'Hahn O', 'Lu N', 'Shuken SR', 'Haney MS', 'Lehallier B', 'Iyer M', 'Luo J', 'Zetterberg H', 'Keller A', 'Zuchero JB', 'Wyss-Coray T'],\n",
       "  '2022',\n",
       "  'Nature',\n",
       "  '10.1038/s41586-022-04722-0'),\n",
       " '35478240': ('Computer-designed repurposing of chemical wastes into drugs.',\n",
       "  ['Wołos A', 'Koszelewski D', 'Roszak R', 'Szymkuć S', 'Moskal M', 'Ostaszewski R', 'Herrera BT', 'Maier JM', 'Brezicki G', 'Samuel J', 'Lummiss JAM', 'McQuade DT', 'Rogers L', 'Grzybowski BA'],\n",
       "  '2022',\n",
       "  'Nature',\n",
       "  '10.1038/s41586-022-04503-9'),\n",
       " '35355009': ('Entropy of city street networks linked to future spatial navigation ability.',\n",
       "  ['Coutrot A', 'Manley E', 'Goodroe S', 'Gahnstrom C', 'Filomena G', 'Yesiltepe D', 'Dalton RC', 'Wiener JM', 'Hölscher C', 'Hornberger M', 'Spiers HJ'],\n",
       "  '2022',\n",
       "  'Nature',\n",
       "  '10.1038/s41586-022-04486-7'),\n",
       " '35236988': (\"FSH blockade improves cognition in mice with Alzheimer's disease.\",\n",
       "  ['Xiong J', 'Kang SS', 'Wang Z', 'Liu X', 'Kuo TC', 'Korkmaz F', 'Padilla A', 'Miyashita S', 'Chan P', 'Zhang Z', 'Katsel P', 'Burgess J', 'Gumerova A', 'Ievleva K', 'Sant D', 'Yu SP', 'Muradova V', 'Frolinger T', 'Lizneva D', 'Iqbal J', 'Goosens KA', 'Gera S', 'Rosen CJ', 'Haroutunian V', 'Ryu V', 'Yuen T', 'Zaidi M', 'Ye K'],\n",
       "  '2022',\n",
       "  'Nature',\n",
       "  '10.1038/s41586-022-04463-0'),\n",
       " '34937941': ('Spatial maps in piriform cortex during olfactory navigation.',\n",
       "  ['Poo C', 'Agarwal G', 'Bonacchi N', 'Mainen ZF'],\n",
       "  '2022',\n",
       "  'Nature',\n",
       "  '10.1038/s41586-021-04242-3'),\n",
       " '34616074': ('The mouse cortico-basal ganglia-thalamic network.',\n",
       "  ['Foster NN', 'Barry J', 'Korobkova L', 'Garcia L', 'Gao L', 'Becerra M', 'Sherafat Y', 'Peng B', 'Li X', 'Choi JH', 'Gou L', 'Zingg B', 'Azam S', 'Lo D', 'Khanjani N', 'Zhang B', 'Stanis J', 'Bowman I', 'Cotter K', 'Cao C', 'Yamashita S', 'Tugangui A', 'Li A', 'Jiang T', 'Jia X', 'Feng Z', 'Aquino S', 'Mun HS', 'Zhu M', 'Santarelli A', 'Benavidez NL', 'Song M', 'Dan G', 'Fayzullina M', 'Ustrell S', 'Boesen T', 'Johnson DL', 'Xu H', 'Bienkowski MS', 'Yang XW', 'Gong H', 'Levine MS', 'Wickersham I', 'Luo Q', 'Hahn JD', 'Lim BK', 'Zhang LI', 'Cepeda C', 'Hintiryan H', 'Dong HW'],\n",
       "  '2021',\n",
       "  'Nature',\n",
       "  '10.1038/s41586-021-03993-3'),\n",
       " '34616064': ('A transcriptomic atlas of mouse cerebellar cortex comprehensively\\xa0defines cell types.',\n",
       "  ['Kozareva V', 'Martin C', 'Osorno T', 'Rudolph S', 'Guo C', 'Vanderburg C', 'Nadaf N', 'Regev A', 'Regehr WG', 'Macosko E'],\n",
       "  '2021',\n",
       "  'Nature',\n",
       "  '10.1038/s41586-021-03220-z'),\n",
       " '34614503': ('Thalamic circuits for independent control of prefrontal signal and noise.',\n",
       "  ['Mukherjee A', 'Lam NH', 'Wimmer RD', 'Halassa MM'],\n",
       "  '2021',\n",
       "  'Nature',\n",
       "  '10.1038/s41586-021-04056-3'),\n",
       " '34599306': ('Hominini-specific regulation of CBLN2 increases prefrontal spinogenesis.',\n",
       "  ['Shibata M', 'Pattabiraman K', 'Muchnik SK', 'Kaur N', 'Morozov YM', 'Cheng X', 'Waxman SG', 'Sestan N'],\n",
       "  '2021',\n",
       "  'Nature',\n",
       "  '10.1038/s41586-021-03952-y'),\n",
       " '34153974': ('Dysregulation of brain and choroid plexus cell types in severe COVID-19.',\n",
       "  ['Yang AC', 'Kern F', 'Losada PM', 'Agam MR', 'Maat CA', 'Schmartz GP', 'Fehlmann T', 'Stein JA', 'Schaum N', 'Lee DP', 'Calcuttawala K', 'Vest RT', 'Berdnik D', 'Lu N', 'Hahn O', 'Gate D', 'McNerney MW', 'Channappa D', 'Cobos I', 'Ludwig N', 'Schulz-Schaeffer WJ', 'Keller A', 'Wyss-Coray T'],\n",
       "  '2021',\n",
       "  'Nature',\n",
       "  '10.1038/s41586-021-03710-0'),\n",
       " '33790467': ('Shared mechanisms underlie the control of working memory and attention.',\n",
       "  ['Panichello MF', 'Buschman TJ'],\n",
       "  '2021',\n",
       "  'Nature',\n",
       "  '10.1038/s41586-021-03390-w'),\n",
       " '33505022': (\"Single-neuronal predictions of others' beliefs in humans.\",\n",
       "  ['Jamali M', 'Grannan BL', 'Fedorenko E', 'Saxe R', 'Báez-Mendoza R', 'Williams ZM'],\n",
       "  '2021',\n",
       "  'Nature',\n",
       "  '10.1038/s41586-021-03184-0'),\n",
       " '33473210': ('Restoring metabolism of myeloid cells reverses cognitive decline in ageing.',\n",
       "  ['Minhas PS', 'Latif-Hernandez A', 'McReynolds MR', 'Durairaj AS', 'Wang Q', 'Rubin A', 'Joshi AU', 'He JQ', 'Gauba E', 'Liu L', 'Wang C', 'Linde M', 'Sugiura Y', 'Moon PK', 'Majeti R', 'Suematsu M', 'Mochly-Rosen D', 'Weissman IL', 'Longo FM', 'Rabinowitz JD', 'Andreasson KI'],\n",
       "  '2021',\n",
       "  'Nature',\n",
       "  '10.1038/s41586-020-03160-0'),\n",
       " '33361808': ('Boundary-anchored neural mechanisms of location-encoding for self and others.',\n",
       "  ['Stangl M', 'Topalovic U', 'Inman CS', 'Hiller S', 'Villaroman D', 'Aghajan ZM', 'Christov-Moore L', 'Hasulak NR', 'Rao VR', 'Halpern CH', 'Eliashiv D', 'Fried I', 'Suthana N'],\n",
       "  '2021',\n",
       "  'Nature',\n",
       "  '10.1038/s41586-020-03073-y'),\n",
       " '32999460': ('A hypothalamic novelty signal modulates hippocampal memory.',\n",
       "  ['Chen S', 'He L', 'Huang AJY', 'Boehringer R', 'Robert V', 'Wintzer ME', 'Polygalov D', 'Weitemier AZ', 'Tao Y', 'Gu M', 'Middleton SJ', 'Namiki K', 'Hama H', 'Therreau L', 'Chevaleyre V', 'Hioki H', 'Miyawaki A', 'Piskorowski RA', 'McHugh TJ'],\n",
       "  '2020',\n",
       "  'Nature',\n",
       "  '10.1038/s41586-020-2771-1'),\n",
       " '32699411': ('Distinct subnetworks of the thalamic reticular nucleus.',\n",
       "  ['Li Y', 'Lopez-Huerta VG', 'Adiconis X', 'Levandowski K', 'Choi S', 'Simmons SK', 'Arias-Garcia MA', 'Guo B', 'Yao AY', 'Blosser TR', 'Wimmer RD', 'Aida T', 'Atamian A', 'Naik T', 'Sun X', 'Bi D', 'Malhotra D', 'Hession CC', 'Shema R', 'Gomes M', 'Li T', 'Hwang E', 'Krol A', 'Kowalczyk M', 'Peça J', 'Pan G', 'Halassa MM', 'Levin JZ', 'Fu Z', 'Feng G'],\n",
       "  '2020',\n",
       "  'Nature',\n",
       "  '10.1038/s41586-020-2504-5'),\n",
       " '31915375': (\"Clonally expanded CD8 T cells patrol the cerebrospinal fluid in Alzheimer's disease.\",\n",
       "  ['Gate D', 'Saligrama N', 'Leventhal O', 'Yang AC', 'Unger MS', 'Middeldorp J', 'Chen K', 'Lehallier B', 'Channappa D', 'De Los Santos MB', 'McBride A', 'Pluvinage J', 'Elahi F', 'Tam GK', 'Kim Y', 'Greicius M', 'Wagner AD', 'Aigner L', 'Galasko DR', 'Davis MM', 'Wyss-Coray T'],\n",
       "  '2020',\n",
       "  'Nature',\n",
       "  '10.1038/s41586-019-1895-7'),\n",
       " '31801999': ('Frontal cortex neuron types categorically encode single decision variables.',\n",
       "  ['Hirokawa J', 'Vaughan A', 'Masset P', 'Ott T', 'Kepecs A'],\n",
       "  '2019',\n",
       "  'Nature',\n",
       "  '10.1038/s41586-019-1816-9')}"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CA_summ = {\n",
    "    p['Id'] : ( p['Title'], p['AuthorList'], p['PubDate'][:4], p['FullJournalName'], p.get('DOI', '') )\n",
    "    for p in Entrez.read( Entrez.esummary(db=\"pubmed\", id=','.join( CA_ids )) )\n",
    "    }\n",
    "CA_summ"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because we restricted our search to papers in an open-access journal, you can then follow their DOIs to freely access their PDFs at the journal's website."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now issue calls to `Entrez.elink()` using these PubMed IDs, to retrieve the IDs of papers that cite them.\n",
    "The database from which the IDs will be retrieved is [PubMed Central](http://www.ncbi.nlm.nih.gov/pmc/), a free digital database of full-text scientific literature in the biomedical and life sciences.\n",
    "\n",
    "A complete list of the kinds of links you can retrieve with `Entrez.elink()` can be found [here](http://eutils.ncbi.nlm.nih.gov/entrez/query/static/entrezlinks.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "CA_citing = {\n",
    "    id : Entrez.read( Entrez.elink(\n",
    "            cmd = \"neighbor\",               # ELink command mode: \"neighbor\", returns\n",
    "                                            #     a set of UIDs in `db` linked to the input UIDs in `dbfrom`.\n",
    "            dbfrom = \"pubmed\",              # Database containing the input UIDs: PubMed\n",
    "            db = \"pmc\",                     # Database from which to retrieve UIDs: PubMed Central\n",
    "            LinkName = \"pubmed_pmc_refs\",   # Name of the Entrez link to retrieve: \"pubmed_pmc_refs\", gets\n",
    "                                            #     \"Full-text articles in the PubMed Central Database that cite the current articles\"\n",
    "            from_uid = id                   # input UIDs\n",
    "            ) )\n",
    "    for id in CA_ids\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'LinkSetDbHistory': [], 'ERROR': [], 'LinkSetDb': [{'Link': [{'Id': '9534763'}, {'Id': '9380260'}, {'Id': '9329517'}, {'Id': '9284159'}, {'Id': '9282170'}, {'Id': '9262333'}, {'Id': '9087306'}, {'Id': '9045815'}, {'Id': '9022003'}, {'Id': '9010663'}, {'Id': '8967821'}, {'Id': '8849135'}, {'Id': '8809184'}, {'Id': '8770604'}, {'Id': '8741155'}, {'Id': '8616578'}, {'Id': '8599015'}, {'Id': '8491944'}, {'Id': '8484055'}, {'Id': '8356785'}, {'Id': '8324299'}, {'Id': '8232037'}, {'Id': '8171116'}, {'Id': '8115883'}, {'Id': '8104958'}, {'Id': '8083070'}, {'Id': '8082085'}, {'Id': '8052096'}, {'Id': '7984585'}, {'Id': '7939081'}, {'Id': '7935083'}, {'Id': '7928425'}, {'Id': '7906913'}, {'Id': '7886003'}, {'Id': '7853322'}, {'Id': '7771962'}, {'Id': '7610550'}, {'Id': '7393175'}, {'Id': '7351792'}, {'Id': '7239658'}, {'Id': '7213979'}, {'Id': '7062473'}, {'Id': '7056889'}], 'DbTo': 'pmc', 'LinkName': 'pubmed_pmc_refs'}], 'DbFrom': 'pubmed', 'IdList': ['31801999']}]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CA_citing['31801999']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have in `CA_citing[paper_id][0]['LinkSetDb'][0]['Link']` the list of papers citing `paper_id`. To get it as just a list of ids, we can do"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['9534763',\n",
       " '9380260',\n",
       " '9329517',\n",
       " '9284159',\n",
       " '9282170',\n",
       " '9262333',\n",
       " '9087306',\n",
       " '9045815',\n",
       " '9022003',\n",
       " '9010663',\n",
       " '8967821',\n",
       " '8849135',\n",
       " '8809184',\n",
       " '8770604',\n",
       " '8741155',\n",
       " '8616578',\n",
       " '8599015',\n",
       " '8491944',\n",
       " '8484055',\n",
       " '8356785',\n",
       " '8324299',\n",
       " '8232037',\n",
       " '8171116',\n",
       " '8115883',\n",
       " '8104958',\n",
       " '8083070',\n",
       " '8082085',\n",
       " '8052096',\n",
       " '7984585',\n",
       " '7939081',\n",
       " '7935083',\n",
       " '7928425',\n",
       " '7906913',\n",
       " '7886003',\n",
       " '7853322',\n",
       " '7771962',\n",
       " '7610550',\n",
       " '7393175',\n",
       " '7351792',\n",
       " '7239658',\n",
       " '7213979',\n",
       " '7062473',\n",
       " '7056889']"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cits = [ l['Id'] for l in CA_citing['31801999'][0]['LinkSetDb'][0]['Link'] ]\n",
    "cits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, one more step is needed, as what we have now are PubMed Central IDs, and not PubMed IDs. Their conversion can be achieved through an additional call to `Entrez.elink()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'LinkSetDbHistory': [], 'ERROR': [], 'LinkSetDb': [{'Link': [{'Id': '36171427'}, {'Id': '35911570'}, {'Id': '35835556'}, {'Id': '35464318'}, {'Id': '35450225'}, {'Id': '35416775'}, {'Id': '35354821'}, {'Id': '34764156'}, {'Id': '34707289'}, {'Id': '34693908'}, {'Id': '34678148'}, {'Id': '34654556'}, {'Id': '34555012'}, {'Id': '34328419'}, {'Id': '34060878'}, {'Id': '34060872'}, {'Id': '33981970'}, {'Id': '33876728'}, {'Id': '33869678'}, {'Id': '33785156'}, {'Id': '33681428'}, {'Id': '33531416'}, {'Id': '33446521'}, {'Id': '33441432'}, {'Id': '33361819'}, {'Id': '33345774'}, {'Id': '33338423'}, {'Id': '33227602'}, {'Id': '32965146'}, {'Id': '32827456'}, {'Id': '32733065'}, {'Id': '32651373'}, {'Id': '32504542'}, {'Id': '32431293'}, {'Id': '32286227'}, {'Id': '32231315'}, {'Id': '32174812'}, {'Id': '32096761'}], 'DbTo': 'pubmed', 'LinkName': 'pmc_pubmed'}], 'DbFrom': 'pmc', 'IdList': ['9534763', '9380260', '9329517', '9284159', '9282170', '9262333', '9087306', '9045815', '9022003', '9010663', '8967821', '8849135', '8809184', '8770604', '8741155', '8616578', '8599015', '8491944', '8484055', '8356785', '8324299', '8232037', '8171116', '8115883', '8104958', '8083070', '8082085', '8052096', '7984585', '7939081', '7935083', '7928425', '7906913', '7886003', '7853322', '7771962', '7610550', '7393175', '7351792', '7239658', '7213979', '7062473', '7056889']}]"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cits_pm = Entrez.read( Entrez.elink( dbfrom=\"pmc\", db=\"pubmed\", LinkName=\"pmc_pubmed\", from_uid=\",\".join(cits)) )\n",
    "cits_pm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'9534763': '36171427',\n",
       " '9380260': '35911570',\n",
       " '9329517': '35835556',\n",
       " '9284159': '35464318',\n",
       " '9282170': '35450225',\n",
       " '9262333': '35416775',\n",
       " '9087306': '35354821',\n",
       " '9045815': '34764156',\n",
       " '9022003': '34707289',\n",
       " '9010663': '34693908',\n",
       " '8967821': '34678148',\n",
       " '8849135': '34654556',\n",
       " '8809184': '34555012',\n",
       " '8770604': '34328419',\n",
       " '8741155': '34060878',\n",
       " '8616578': '34060872',\n",
       " '8599015': '33981970',\n",
       " '8491944': '33876728',\n",
       " '8484055': '33869678',\n",
       " '8356785': '33785156',\n",
       " '8324299': '33681428',\n",
       " '8232037': '33531416',\n",
       " '8171116': '33446521',\n",
       " '8115883': '33441432',\n",
       " '8104958': '33361819',\n",
       " '8083070': '33345774',\n",
       " '8082085': '33338423',\n",
       " '8052096': '33227602',\n",
       " '7984585': '32965146',\n",
       " '7939081': '32827456',\n",
       " '7935083': '32733065',\n",
       " '7928425': '32651373',\n",
       " '7906913': '32504542',\n",
       " '7886003': '32431293',\n",
       " '7853322': '32286227',\n",
       " '7771962': '32231315',\n",
       " '7610550': '32174812',\n",
       " '7393175': '32096761'}"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ids_map = { pmc_id : link['Id'] for (pmc_id,link) in zip(cits_pm[0]['IdList'], cits_pm[0]['LinkSetDb'][0]['Link']) }\n",
    "ids_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And to check these papers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'36171427': ('Thalamus-driven functional populations in frontal cortex support decision-making.',\n",
       "  ['Yang W', 'Tipparaju SL', 'Chen G', 'Li N'],\n",
       "  '2022',\n",
       "  'Nature neuroscience',\n",
       "  '10.1038/s41593-022-01171-w'),\n",
       " '35911570': ('Are Grid-Like Representations a Component of All Perception and Cognition?',\n",
       "  ['Chen ZS', 'Zhang X', 'Long X', 'Zhang SJ'],\n",
       "  '2022',\n",
       "  'Frontiers in neural circuits',\n",
       "  '10.3389/fncir.2022.924016'),\n",
       " '35835556': ('Navigating the Statistical Minefield of Model Selection and Clustering in Neuroscience.',\n",
       "  ['Király B', 'Hangya B'],\n",
       "  '2022',\n",
       "  'eNeuro',\n",
       "  '10.1523/ENEURO.0066-22.2022'),\n",
       " '35464318': ('Memristive LIF Spiking Neuron Model and Its Application in Morse Code.',\n",
       "  ['Fang X', 'Liu D', 'Duan S', 'Wang L'],\n",
       "  '2022',\n",
       "  'Frontiers in neuroscience',\n",
       "  '10.3389/fnins.2022.853010'),\n",
       " '35450225': ('Review on data analysis methods for mesoscale neural imaging <i>in vivo</i>.',\n",
       "  ['Cai Y', 'Wu J', 'Dai Q'],\n",
       "  '2022',\n",
       "  'Neurophotonics',\n",
       "  '10.1117/1.NPh.9.4.041407'),\n",
       " '35416775': ('Neuronal origins of reduced accuracy and biases in economic choices under sequential offers.',\n",
       "  ['Shi W', 'Ballesta S', 'Padoa-Schioppa C'],\n",
       "  '2022',\n",
       "  'eLife',\n",
       "  '10.7554/eLife.75910'),\n",
       " '35354821': ('Donut-like organization of inhibition underlies categorical neural responses in the midbrain.',\n",
       "  ['Mahajan NR', 'Mysore SP'],\n",
       "  '2022',\n",
       "  'Nature communications',\n",
       "  '10.1038/s41467-022-29318-0'),\n",
       " '34764156': ('Economic Choices under Simultaneous or Sequential Offers Rely on the Same Neural Circuit.',\n",
       "  ['Shi W', 'Ballesta S', 'Padoa-Schioppa C'],\n",
       "  '2022',\n",
       "  'The Journal of neuroscience : the official journal of the Society for Neuroscience',\n",
       "  '10.1523/JNEUROSCI.1265-21.2021'),\n",
       " '34707289': ('The orbitofrontal cortex maps future navigational goals.',\n",
       "  ['Basu R', 'Gebauer R', 'Herfurth T', 'Kolb S', 'Golipour Z', 'Tchumatchenko T', 'Ito HT'],\n",
       "  '2021',\n",
       "  'Nature',\n",
       "  '10.1038/s41586-021-04042-9'),\n",
       " '34693908': ('Subpopulations of neurons in lOFC encode previous and current rewards at time of choice.',\n",
       "  ['Hocker DL', 'Brody CD', 'Savin C', 'Constantinople CM'],\n",
       "  '2021',\n",
       "  'eLife',\n",
       "  '10.7554/eLife.70129'),\n",
       " '34678148': ('The learning of prospective and retrospective cognitive maps within neural circuits.',\n",
       "  ['K Namboodiri VM', 'Stuber GD'],\n",
       "  '2021',\n",
       "  'Neuron',\n",
       "  '10.1016/j.neuron.2021.09.034'),\n",
       " '34654556': ('50 years of mnemonic persistent activity: quo vadis?',\n",
       "  ['Wang XJ'],\n",
       "  '2021',\n",
       "  'Trends in neurosciences',\n",
       "  '10.1016/j.tins.2021.09.001'),\n",
       " '34555012': ('A model of naturalistic decision making in preference tests.',\n",
       "  ['Ksander J', 'Katz DB', 'Miller P'],\n",
       "  '2021',\n",
       "  'PLoS computational biology',\n",
       "  '10.1371/journal.pcbi.1009012'),\n",
       " '34328419': ('Sensory coding and the causal impact of mouse cortex in a visual decision.',\n",
       "  ['Zatka-Haas P', 'Steinmetz NA', 'Carandini M', 'Harris KD'],\n",
       "  '2021',\n",
       "  'eLife',\n",
       "  '10.7554/eLife.63163'),\n",
       " '34060878': ('Call for a more balanced approach to understanding orbital frontal cortex function.',\n",
       "  ['Yalcinbas EA', 'Cazares C', 'Gremel CM'],\n",
       "  '2021',\n",
       "  'Behavioral neuroscience',\n",
       "  '10.1037/bne0000450'),\n",
       " '34060872': ('The orbitofrontal cortex in temporal cognition.',\n",
       "  ['Sosa JLR', 'Buonomano D', 'Izquierdo A'],\n",
       "  '2021',\n",
       "  'Behavioral neuroscience',\n",
       "  '10.1037/bne0000430'),\n",
       " '33981970': ('Bi-directional encoding of context-based odors and behavioral states by the nucleus of the lateral olfactory tract.',\n",
       "  ['Tanisumi Y', 'Shiotani K', 'Hirokawa J', 'Sakurai Y', 'Manabe H'],\n",
       "  '2021',\n",
       "  'iScience',\n",
       "  '10.1016/j.isci.2021.102381'),\n",
       " '33876728': ('Robust and distributed neural representation of action values.',\n",
       "  ['Shin EJ', 'Jang Y', 'Kim S', 'Kim H', 'Cai X', 'Lee H', 'Sul JH', 'Lee SH', 'Chung Y', 'Lee D', 'Jung MW'],\n",
       "  '2021',\n",
       "  'eLife',\n",
       "  '10.7554/eLife.53045'),\n",
       " '33869678': ('Is the core function of orbitofrontal cortex to signal values or make predictions?',\n",
       "  ['Zhou J', 'Gardner MPH', 'Schoenbaum G'],\n",
       "  '2021',\n",
       "  'Current opinion in behavioral sciences',\n",
       "  '10.1016/j.cobeha.2021.02.011'),\n",
       " '33785156': ('Integration of value and action in medial prefrontal neural systems.',\n",
       "  ['Kaminska B', 'Caballero JP', 'Moorman DE'],\n",
       "  '2021',\n",
       "  'International review of neurobiology',\n",
       "  '10.1016/bs.irn.2020.11.007'),\n",
       " '33681428': ('Neural circuitry of information seeking.',\n",
       "  ['Bromberg-Martin ES', 'Monosov IE'],\n",
       "  '2020',\n",
       "  'Current opinion in behavioral sciences',\n",
       "  '10.1016/j.cobeha.2020.07.006'),\n",
       " '33531416': ('Coordinated Prefrontal State Transition Leads Extinction of Reward-Seeking Behaviors.',\n",
       "  ['Russo E', 'Ma T', 'Spanagel R', 'Durstewitz D', 'Toutounji H', 'Köhr G'],\n",
       "  '2021',\n",
       "  'The Journal of neuroscience : the official journal of the Society for Neuroscience',\n",
       "  '10.1523/JNEUROSCI.2588-20.2021'),\n",
       " '33446521': ('Orbitofrontal State Representations Are Related to Choice Adaptations and Reward Predictions.',\n",
       "  ['Stalnaker TA', 'Raheja N', 'Schoenbaum G'],\n",
       "  '2021',\n",
       "  'The Journal of neuroscience : the official journal of the Society for Neuroscience',\n",
       "  '10.1523/JNEUROSCI.0753-20.2020'),\n",
       " '33441432': ('Neural Population Dynamics Underlying Expected Value Computation.',\n",
       "  ['Yamada H', 'Imaizumi Y', 'Matsumoto M'],\n",
       "  '2021',\n",
       "  'The Journal of neuroscience : the official journal of the Society for Neuroscience',\n",
       "  '10.1523/JNEUROSCI.1987-20.2020'),\n",
       " '33361819': ('Evolving schema representations in orbitofrontal ensembles during learning.',\n",
       "  ['Zhou J', 'Jia C', 'Montesinos-Cartagena M', 'Gardner MPH', 'Zong W', 'Schoenbaum G'],\n",
       "  '2021',\n",
       "  'Nature',\n",
       "  '10.1038/s41586-020-03061-2'),\n",
       " '33345774': ('Distinct temporal difference error signals in dopamine axons in three regions of the striatum in a decision-making task.',\n",
       "  ['Tsutsui-Kimura I', 'Matsumoto H', 'Akiti K', 'Yamada MM', 'Uchida N', 'Watabe-Uchida M'],\n",
       "  '2020',\n",
       "  'eLife',\n",
       "  '10.7554/eLife.62390'),\n",
       " '33338423': ('Differential encoding in prefrontal cortex projection neuron classes across cognitive tasks.',\n",
       "  ['Lui JH', 'Nguyen ND', 'Grutzner SM', 'Darmanis S', 'Peixoto D', 'Wagner MJ', 'Allen WE', 'Kebschull JM', 'Richman EB', 'Ren J', 'Newsome WT', 'Quake SR', 'Luo L'],\n",
       "  '2021',\n",
       "  'Cell',\n",
       "  '10.1016/j.cell.2020.11.046'),\n",
       " '33227602': ('Approaches to inferring multi-regional interactions from simultaneous population recordings: Inferring multi-regional interactions from simultaneous population recordings.',\n",
       "  ['Kang B', 'Druckmann S'],\n",
       "  '2020',\n",
       "  'Current opinion in neurobiology',\n",
       "  '10.1016/j.conb.2020.10.004'),\n",
       " '32965146': ('Diverse coactive neurons encode stimulus-driven and stimulus-independent variables.',\n",
       "  ['Xia J', 'Marks TD', 'Goard MJ', 'Wessel R'],\n",
       "  '2020',\n",
       "  'Journal of neurophysiology',\n",
       "  '10.1152/jn.00431.2020'),\n",
       " '32827456': ('Transient and Persistent Representations of Odor Value in Prefrontal Cortex.',\n",
       "  ['Wang PY', 'Boboila C', 'Chin M', 'Higashi-Howard A', 'Shamash P', 'Wu Z', 'Stein NP', 'Abbott LF', 'Axel R'],\n",
       "  '2020',\n",
       "  'Neuron',\n",
       "  '10.1016/j.neuron.2020.07.033'),\n",
       " '32733065': ('Dynamic coordination of the perirhinal cortical neurons supports coherent representations between task epochs.',\n",
       "  ['Ohnuki T', 'Osako Y', 'Manabe H', 'Sakurai Y', 'Hirokawa J'],\n",
       "  '2020',\n",
       "  'Communications biology',\n",
       "  '10.1038/s42003-020-01129-3'),\n",
       " '32651373': ('Value and choice as separable and stable representations in orbitofrontal cortex.',\n",
       "  ['Kimmel DL', 'Elsayed GF', 'Cunningham JP', 'Newsome WT'],\n",
       "  '2020',\n",
       "  'Nature communications',\n",
       "  '10.1038/s41467-020-17058-y'),\n",
       " '32504542': ('Behavior- and Modality-General Representation of Confidence in Orbitofrontal Cortex.',\n",
       "  ['Masset P', 'Ott T', 'Lak A', 'Hirokawa J', 'Kepecs A'],\n",
       "  '2020',\n",
       "  'Cell',\n",
       "  '10.1016/j.cell.2020.05.022'),\n",
       " '32431293': ('Mechanisms of competitive selection: A canonical neural circuit framework.',\n",
       "  ['Mysore SP', 'Kothari NB'],\n",
       "  '2020',\n",
       "  'eLife',\n",
       "  '10.7554/eLife.51473'),\n",
       " '32286227': ('Reinforcement biases subsequent perceptual decisions when confidence is low, a widespread behavioral phenomenon.',\n",
       "  ['Lak A', 'Hueske E', 'Hirokawa J', 'Masset P', 'Ott T', 'Urai AE', 'Donner TH', 'Carandini M', 'Tonegawa S', 'Uchida N', 'Kepecs A'],\n",
       "  '2020',\n",
       "  'eLife',\n",
       "  '10.7554/eLife.49834'),\n",
       " '32231315': ('The transition to compulsion in addiction.',\n",
       "  ['Lüscher C', 'Robbins TW', 'Everitt BJ'],\n",
       "  '2020',\n",
       "  'Nature reviews. Neuroscience',\n",
       "  '10.1038/s41583-020-0289-z'),\n",
       " '32174812': ('Canine Adenovirus 2: A Natural Choice for Brain Circuit Dissection.',\n",
       "  ['Lavoie A', 'Liu BH'],\n",
       "  '2020',\n",
       "  'Frontiers in molecular neuroscience',\n",
       "  '10.3389/fnmol.2020.00009'),\n",
       " '32096761': ('Neural mechanisms of economic choices in mice.',\n",
       "  ['Kuwabara M', 'Kang N', 'Holy TE', 'Padoa-Schioppa C'],\n",
       "  '2020',\n",
       "  'eLife',\n",
       "  '10.7554/eLife.49669')}"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{   p['Id'] : ( p['Title'], p['AuthorList'], p['PubDate'][:4], p['FullJournalName'], p.get('DOI', '') )\n",
    "    for p in Entrez.read( Entrez.esummary(db=\"pubmed\", id=','.join( ids_map.values() )) )\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Citations dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have now seen all the steps required to assemble a dataset of citations to each of the papers in our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "Citations_file = 'data/' + search_term + '_Citations.pkl.bz2'\n",
    "Citations = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At least one server query will be issued per paper in `Ids`. Because NCBI allows for at most 3 queries per second (see [here](http://biopython.org/DIST/docs/api/Bio.Entrez-pysrc.html#_open)), this dataset will take a long time to assemble. Should you need to interrupt it for some reason, or the connection fail at some point, it is safe to just rerun the cell below until all data is collected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "import http.client\n",
    "\n",
    "if Citations == [] and os.path.exists( Citations_file ):\n",
    "    Citations = pickle.load( bz2.BZ2File( Citations_file, 'rb' ) )\n",
    "\n",
    "if len(Citations) < len(Ids):\n",
    "    \n",
    "    i = len(Citations)\n",
    "    checkpoint = int(len(Ids) / 100) + 1      # save to hard drive at every 1% of Ids fetched\n",
    "    \n",
    "    for pm_id in Ids[i:]:               # either starts from index 0, or resumes from where we previously left off\n",
    "        \n",
    "        while True:\n",
    "            try:\n",
    "                # query for papers archived in PubMed Central that cite the paper with PubMed ID `pm_id`\n",
    "                c = Entrez.read( Entrez.elink( dbfrom = \"pubmed\", db=\"pmc\", LinkName = \"pubmed_pmc_refs\", id=str(pm_id) ) )\n",
    "                \n",
    "                c = c[0]['LinkSetDb']\n",
    "                if len(c) == 0:\n",
    "                    # no citations found for the current paper\n",
    "                    c = []\n",
    "                else:\n",
    "                    c = [ l['Id'] for l in c[0]['Link'] ]\n",
    "                    \n",
    "                    # convert citations from PubMed Central IDs to PubMed IDs\n",
    "                    p = []\n",
    "                    retrieve_per_query = 500\n",
    "                    for start in range( 0, len(c), retrieve_per_query ):\n",
    "                        query_ids = ','.join( c[start : start+retrieve_per_query] )\n",
    "                        r = Entrez.read( Entrez.elink( dbfrom=\"pmc\", db=\"pubmed\", LinkName=\"pmc_pubmed\", from_uid=query_ids ) )\n",
    "                        # select the IDs. If no matching PubMed ID was found, [] is returned instead\n",
    "                        p.extend( [] if r[0]['LinkSetDb']==[] else [ int(link['Id']) for link in r[0]['LinkSetDb'][0]['Link'] ] )\n",
    "                    c = p\n",
    "            \n",
    "            except http.client.BadStatusLine:\n",
    "                # Presumably, the server closed the connection before sending a valid response. Retry until we have the data.\n",
    "                print('r')\n",
    "                continue\n",
    "            except HTTPError:\n",
    "                print('r')\n",
    "                continue\n",
    "            break\n",
    "        \n",
    "        Citations.append( (pm_id, c) )\n",
    "        if (i % 10000 == 0):\n",
    "            print('')\n",
    "            print(i, end='')\n",
    "        if (i % 100 == 0):\n",
    "            print('.', end='')\n",
    "        i += 1\n",
    "        \n",
    "        if i % checkpoint == 0:\n",
    "            print('\\tsaving at checkpoint', i)\n",
    "            pickle.dump( Citations, bz2.BZ2File( Citations_file, 'wb' ) )\n",
    "    \n",
    "    print('\\n done.')\n",
    "    \n",
    "    # Save Citations, as a dictionary indexed by Ids\n",
    "    Citations = dict( Citations )\n",
    "    \n",
    "    pickle.dump( Citations, bz2.BZ2File( Citations_file, 'wb' ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see that we have indeed obtained the data we expected, you can match the ids below, with the ids listed at the end of last section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "13907952",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m/tmp/ipykernel_4850/2520520158.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mCitations\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m13907952\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m: 13907952"
     ]
    }
   ],
   "source": [
    "Citations[13907952]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Where do we go from here?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running the code above generates multiple local files, containing the datasets we'll be working with. Loading them into memory is a matter of just issuing a call like<br>\n",
    "``data = pickle.load( bz2.BZ2File( data_file, 'rb' ) )``.\n",
    "\n",
    "The Entrez module will therefore no longer be needed, unless you wish to extend your data processing with additional information retrieved from PubMed.\n",
    "\n",
    "Should you be interested in looking at alternative ways to handle the data, have a look at the [sqlite3](http://docs.python.org/3/library/sqlite3.html) module included in Python's standard library, or [Pandas](http://pandas.pydata.org/), the Python Data Analysis Library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}