14 - Lesson - Information Extraction (IE).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ff7334d5",
   "metadata": {},
   "source": [
    "# Information Extraction (IE)\n",
    "### Goal of lesson\n",
    "- What is Information Extraction\n",
    "- Extract knowledge from patterns\n",
    "- Word representation\n",
    "- Skip-Gram architecture\n",
    "- To see how words relate to each other (this is surprising)\n",
    "\n",
    "### What is Information Extraction (IE)\n",
    "- the task of automatically extracting structured information from unstructured and/or semi-structured machine-readable documents ([wiki](https://en.wikipedia.org/wiki/Information_extraction))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06f7c9c6",
   "metadata": {},
   "source": [
    "### Extract knowledge from patterns\n",
    "- Given data knowledge that is fit together - find patterns\n",
    "- Example\n",
    "    - Knowledge given:\n",
    "        - Amazon (1992)\n",
    "        - Facebook (2004)\n",
    "    - Pattern (template) found:\n",
    "        - \"When {company} was founded in {year},\"\n",
    "- This is a simple, but very powerful approach"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d090354",
   "metadata": {},
   "source": [
    "> #### Programming Notes:\n",
    "> - Libraries used\n",
    ">     - [**pandas**](https://pandas.pydata.org) - a data analysis and manipulation tool\n",
    ">     - [**re**](https://docs.python.org/3/library/re.html) regular expressions\n",
    "> - Functionality and concepts used\n",
    ">     - [**CSV**](https://en.wikipedia.org/wiki/Comma-separated_values) file ([Lecture on CSV](https://youtu.be/LEyojSOg4EI))\n",
    ">     - [**read_csv()**](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html) read a comma-separated values (csv) file into **pandas** DataFrame.\n",
    ">     - [**Regular Expression**](https://en.wikipedia.org/wiki/Regular_expression) s a sequence of characters that specifies a search pattern."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "980da212",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ac9bcd4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "books = pd.read_csv('files/books.csv', header=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "74ea5416",
   "metadata": {},
   "outputs": [],
   "source": [
    "book_list = books.values.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5d2128a8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['1984', 'George Orwell'], ['The Help', 'Kathryn Stockett']]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "book_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5994835e",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('files/penguin.html') as f:\n",
    "    corpus = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "dddf0cf1",
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = corpus.replace('\\n', ' ').replace('\\t', ' ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0652325e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1984 - George Orwell\n",
      "-: ge-orwell-with-a-foreword-by-thomas-pynchon/\">1984</a></h2>   <h2 class=\"author\">by George Orwell</h\n",
      "-: eword-by-thomas-pynchon/\">1984</a></h2>   <h2 class=\"author\">by George Orwell</h2>    <div class=\"de\n",
      "-: hon/\">1984</a></h2>   <h2 class=\"author\">by George Orwell</h2>    <div class=\"desc\">We were pretty c\n",
      "The Help - Kathryn Stockett\n",
      "-: /the-help-by-kathryn-stockett/\">The Help</a></h2>   <h2 class=\"author\">by Kathryn Stockett</h2>    <\n",
      "-: -stockett/\">The Help</a></h2>   <h2 class=\"author\">by Kathryn Stockett</h2>    <div class=\"desc\">Thi\n"
     ]
    }
   ],
   "source": [
    "for val1, val2 in book_list:\n",
    "    print(val1, '-', val2)\n",
    "    for i in range(0, len(corpus) - 100, 20):\n",
    "        pattern = corpus[i:i + 100]\n",
    "        if val1 in pattern and val2 in pattern:\n",
    "            print('-:', pattern)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dedf8116",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f0da682d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('/\">',\n",
       " '</a></h2>\\\\ \\\\ \\\\ <h2\\\\ class=\"author\">by\\\\ ',\n",
       " '</h2>\\\\ \\\\ \\\\ \\\\ <div\\\\ class=\"desc\">')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prefix = re.escape('/\">')\n",
    "middle = re.escape('</a></h2>   <h2 class=\"author\">by ')\n",
    "suffix = re.escape('</h2>    <div class=\"desc\">')\n",
    "prefix, middle, suffix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "82676895",
   "metadata": {},
   "outputs": [],
   "source": [
    "regex = f\"{prefix}(.{{0,50}}?){middle}(.{{0,50}}?){suffix}\"\n",
    "results = re.findall(regex, corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "79944531",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('War and Peace', 'Leo Tolstoy'),\n",
       " ('Song of Solomon', 'Toni Morrison'),\n",
       " ('Ulysses', 'James Joyce'),\n",
       " ('The Shadow of the Wind', 'Carlos Ruiz Zafon'),\n",
       " ('The Lord of the Rings', 'J.R.R. Tolkien'),\n",
       " ('The Satanic Verses', 'Salman Rushdie'),\n",
       " ('Don Quixote', 'Miguel de Cervantes'),\n",
       " ('The Golden Compass', 'Philip Pullman'),\n",
       " ('Catch-22', 'Joseph Heller'),\n",
       " ('1984', 'George Orwell'),\n",
       " ('The Kite Runner', 'Khaled Hosseini'),\n",
       " ('Little Women', 'Louisa May Alcott'),\n",
       " ('The Cloud Atlas', 'David Mitchell'),\n",
       " ('The Fountainhead', 'Ayn Rand'),\n",
       " ('The Picture of Dorian Gray', 'Oscar Wilde'),\n",
       " ('Lolita', 'Vladimir Nabokov'),\n",
       " ('The Help', 'Kathryn Stockett'),\n",
       " (\"The Liar's Club\", 'Mary Karr'),\n",
       " ('Moby-Dick', 'Herman Melville'),\n",
       " (\"Gravity's Rainbow\", 'Thomas Pynchon'),\n",
       " (\"The Handmaid's Tale\", 'Margaret Atwood')]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bcfaf700",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "5e9111b2",
   "metadata": {},
   "source": [
    "### One-Hot Representation\n",
    "- Representation word as a vector with a single 1, and with other values as 0\n",
    "- Maybe not useful to have with"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3db796b6",
   "metadata": {},
   "source": [
    "### Distributed Representation\n",
    "- representation of meaning distributed across multiple values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a99e0bd6",
   "metadata": {},
   "source": [
    "### How to define words as vectors\n",
    "- Word is defined by what words suround it\n",
    "- Based on the context\n",
    "- What words happen to show up around it"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2276ff32",
   "metadata": {},
   "source": [
    "### word2vec\n",
    "- model for generating word vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd719e53",
   "metadata": {},
   "source": [
    "### Skip-Gram Architecture\n",
    "- Neural network architecture for predicting context words given a target word\n",
    "    - Given a word - what words show up around it in a context\n",
    "- Example\n",
    "    - Given **target word** (input word) - train the network of which **context words** (right side)\n",
    "    - Then the weights from input node (**target word**) to hidden layer (5 weights) give a representation\n",
    "    - Hence - the word will be represented by a vector\n",
    "    - The number of hidden nodes represent how big the vector should be (here 5)\n",
    "\n",
    "<img src=\"img/word_vectors.png\" width=\"600\" align=\"left\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de22232a",
   "metadata": {},
   "source": [
    "- Idea is as follows\n",
    "    - Each input word will get weights to the hidden layers\n",
    "    - The hidden layers will be trained\n",
    "    - Then each word will be represented as the weights of hidden layers\n",
    "- Intuition\n",
    "    - If two words have similar context (they show up the same places) - then they must be similar - and they have a small distance from each other representations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "caacddbd",
   "metadata": {},
   "source": [
    "> #### Programming Notes:\n",
    "> - Libraries used\n",
    ">     - [**numpy**](http://numpy.org) - scientific computing with Python ([Lecture on NumPy](https://youtu.be/BpzpU8_j0-c))\n",
    ">     - [**scipy**](https://www.scipy.org) - open-source software for mathematics, science, and engineering\n",
    "> - Functionality and concepts used\n",
    ">     - [**cosine**](https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.cosine.html) Compute the Cosine distance between 1-D arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "f63d6343",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from scipy.spatial.distance import cosine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "417c4d90",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('files/words.txt') as f:\n",
    "    words = {}\n",
    "    lines = f.readlines()\n",
    "    for line in lines:\n",
    "        row = line.split()\n",
    "        word = row[0]\n",
    "        vector = np.array([float(x) for x in row[1:]])\n",
    "        words[word] = vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "86719530",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100,)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words['a'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "0037e459",
   "metadata": {},
   "outputs": [],
   "source": [
    "def distance(word1, word2):\n",
    "    return cosine(word1, word2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "1f5c452e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def closest_words(word):\n",
    "    distances = {w: distance(word, words[w]) for w in words}\n",
    "    return sorted(distances, key=lambda w: distances[w])[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6bfe9652",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.19707422881543946"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "distance(words['king'], words['queen'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "83b514e7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.42088794105426874"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "distance(words['king'], words['pope'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "9dd0488c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['queen',\n",
       " 'king',\n",
       " 'empress',\n",
       " 'prince',\n",
       " 'duchess',\n",
       " 'princess',\n",
       " 'consort',\n",
       " 'monarch',\n",
       " 'dowager',\n",
       " 'throne']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "closest_words(words['king'] - words['man'] + words['woman'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ea2b591",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}