249763NLEQuestion3.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/YashMagane/Natural-Language-Engineering/blob/main/249763NLEQuestion3.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Natural Language Engineering G5119\n",
        "## Computer Based Examination, 2023\n",
        "\n",
        "Remember, you can add cells and change their type (between code and text/markdown) as required to answer the questions."
      ],
      "metadata": {
        "id": "Hh003D9cig7w"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# update your candidate number here\n",
        "candidate_number = 249763"
      ],
      "metadata": {
        "id": "dVoZCYKqihTF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Question 3 (50 Marks)\n",
        "\n",
        "This question is about semantics and distributional representations."
      ],
      "metadata": {
        "id": "ZgnCxc1qFUTS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "### do not change the code in this cell\n",
        "# make sure you run this cell\n",
        "import nltk\n",
        "nltk.download('punkt')\n",
        "nltk.download('stopwords')\n",
        "from nltk import word_tokenize\n",
        "from nltk.corpus import stopwords\n",
        "from math import log, sqrt\n",
        "\n",
        "stops = set(stopwords.words('english'))\n",
        "\n",
        "sentences=[\"Being conscious about how and where you spend your money can do a lot to boost your happiness.\",\n",
        "           \"Here is a list of 20 ways you might waste time during your day and what you can do about it.\",\n",
        "           \"These insurance products might be a waste of money, argues Tayne.\",\n",
        "           \"Many parents worry that they don’t spend time with their children.\"]"
      ],
      "metadata": {
        "id": "2pviDy-pFpKF",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a370d942-ddfd-49f6-ff00-1b8293c8bc83"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n",
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "a) Distributional representations are useful for identifying relations of semantic similarity between words. Name three other types of semantic relationship between words (such as might be found in WordNet) and give an example for each. (6 marks)"
      ],
      "metadata": {
        "id": "KGNfRJNfHg0m"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Synonymy: which is the relationship between words that have similar or identical meanings, for example, sad and unhappy.\n",
        "\n",
        "Hyponymy: which is the relationship between a more specific word and a more general word, for instance, a dog is a hyponym of animal.\n",
        "\n",
        "Antonymy: which is the relationship between words that have completely opposite meanings, an example would be sad and happy."
      ],
      "metadata": {
        "id": "lWXF29l1H9pt"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "b)\n",
        "\n",
        "i) Tokenize the strings in the list `sentences`, to produce a list of lists of tokens called `tokenized_sentences`.\n",
        "\n",
        "For example, `[\"This is an example\",\"This is another\"]` would become `[[\"This\",\"is\",\"an\",\"example\"],[\"This\",\"is\",\"another\"]]` (4 marks)"
      ],
      "metadata": {
        "id": "6Quhhm2hFeum"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "B5F9NlzqFQq9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "92220c87-6042-4016-dd9b-54a3097e8bba"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[['Being', 'conscious', 'about', 'how', 'and', 'where', 'you', 'spend', 'your', 'money', 'can', 'do', 'a', 'lot', 'to', 'boost', 'your', 'happiness', '.'], ['Here', 'is', 'a', 'list', 'of', '20', 'ways', 'you', 'might', 'waste', 'time', 'during', 'your', 'day', 'and', 'what', 'you', 'can', 'do', 'about', 'it', '.'], ['These', 'insurance', 'products', 'might', 'be', 'a', 'waste', 'of', 'money', ',', 'argues', 'Tayne', '.'], ['Many', 'parents', 'worry', 'that', 'they', 'don', '’', 't', 'spend', 'time', 'with', 'their', 'children', '.']]\n"
          ]
        }
      ],
      "source": [
        "import nltk\n",
        "tokenized_sentences = [nltk.word_tokenize(sentence) for sentence in sentences]\n",
        "print(tokenized_sentences)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "ii) Case normalise the tokens and remove stopwords and punctuation from the list `tokenised_sentences` and put the result in `normalised_sentences`. (6 marks)"
      ],
      "metadata": {
        "id": "kdBAADlUF-5A"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "from nltk.corpus import stopwords\n",
        "stop_words = stopwords.words('english')\n",
        "normalised_sentences = []\n",
        "for sentence in tokenized_sentences:\n",
        "    normalised_tokens = [token.lower() for token in sentence]\n",
        "    normalised_tokens = [token for token in normalised_tokens if token.isalpha() and token not in stop_words]\n",
        "    normalised_sentences.append(normalised_tokens)\n",
        "print(normalised_sentences)"
      ],
      "metadata": {
        "id": "gmd0pkoxGzQK",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "526549db-d3a2-46a1-8df7-12712c27a926"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[['conscious', 'spend', 'money', 'lot', 'boost', 'happiness'], ['list', 'ways', 'might', 'waste', 'time', 'day'], ['insurance', 'products', 'might', 'waste', 'money', 'argues', 'tayne'], ['many', 'parents', 'worry', 'spend', 'time', 'children']]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "iii) Write a function that takes a list and an index as input and returns a list containing the two context tokens either side of that index, if they exist.\n",
        "\n",
        "For example, given the list `[\"This\", \"is\", \"a\", \"sentence\", \".\"]` and the index `1`, the function would return `[\"This\", \"a\"]`. But given the same list and the index `0` would return `[\"is\"]`. (4 marks)"
      ],
      "metadata": {
        "id": "2uAYbz4RGz0U"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def get_context_tokens(lst, index):\n",
        "    if index > 1 and index < len(lst) - 1:\n",
        "        return [lst[index-1], lst[index+1]]\n",
        "    elif index == 0:\n",
        "        return [lst[index+1]]\n",
        "    elif index == len(lst) - 1:\n",
        "        return [lst[index-1]]\n",
        "    else:\n",
        "        return []"
      ],
      "metadata": {
        "id": "QOyPJu5iI7wv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "iv) Using the function above, compute counts of token co-occurrences in the `normalised_sentences` list. The result should be a dictionary of dictionaries called `word_feature_counts`, with the keys of the outer dictionaries being the word tokens from `normalised_sentences` and the inner keys being the context token features returned by your function. For example, given the normalised sentences `[\"this\", \"is\", \"an\", \"example\"],[\"this\", \"is\", \"another\"]]`, the tokens `this` and `is` co-occur twice. So, `word_feature_counts[\"is\"][\"this\"]` would return the number `2`. (8 marks)"
      ],
      "metadata": {
        "id": "wD6kgDsNI87x"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "word_feature_counts = {}\n",
        "for sentence in normalised_sentences:\n",
        "    for i, token in enumerate(sentence):\n",
        "        context_tokens = get_context_tokens(sentence, i)\n",
        "        if token not in word_feature_counts:\n",
        "            word_feature_counts[token] = {}\n",
        "        for context_token in context_tokens:\n",
        "            if context_token not in word_feature_counts[token]:\n",
        "                word_feature_counts[token][context_token] = 1\n",
        "            else:\n",
        "                word_feature_counts[token][context_token] += 1\n",
        "print(word_feature_counts)"
      ],
      "metadata": {
        "id": "rjafIaofLv2p",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "adac7cce-2758-453e-c7ec-e1e0efe31265"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'conscious': {'spend': 1}, 'spend': {'worry': 1, 'time': 1}, 'money': {'spend': 1, 'lot': 1, 'waste': 1, 'argues': 1}, 'lot': {'money': 1, 'boost': 1}, 'boost': {'lot': 1, 'happiness': 1}, 'happiness': {'boost': 1}, 'list': {'ways': 1}, 'ways': {}, 'might': {'ways': 1, 'waste': 2, 'products': 1}, 'waste': {'might': 2, 'time': 1, 'money': 1}, 'time': {'waste': 1, 'day': 1, 'spend': 1, 'children': 1}, 'day': {'time': 1}, 'insurance': {'products': 1}, 'products': {}, 'argues': {'money': 1, 'tayne': 1}, 'tayne': {'argues': 1}, 'many': {'parents': 1}, 'parents': {}, 'worry': {'parents': 1, 'spend': 1}, 'children': {'time': 1}}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "c) So far, we have built a dictionary of word co-occurrences as an intial step towards constructing a distribuional representation of semantics. Describe the alternative neural network approach as implemented in approaches such as CBOW. (5 marks)"
      ],
      "metadata": {
        "id": "xBRd5gTfLwUI"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "CBOW is a continuous bag of words, it is a model which takes a neural approach for making a distributed representationg of semantics. It does not record the order of words, but it can reocrd the meaning of each word in the context of the words which surround it. in CBOW, a vector of the context words is being turned into a fixed length vector of the word targetted.\n"
      ],
      "metadata": {
        "id": "M1_Qzkrt5WrF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "d)\n",
        "\n",
        "i) Sum over the counts in `word_feature_counts` to produce a total count for all words and feature and a dictionary of total counts for each word, $w$, and a dictionary of total counts for each feature, $f$. (5 marks)"
      ],
      "metadata": {
        "id": "10IbUS6_BpRs"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "total_count = 0\n",
        "word_counts = {}\n",
        "feature_counts = {}\n",
        "for word, features in word_feature_counts.items():\n",
        "    word_count = sum(features.values())\n",
        "    total_count += word_count\n",
        "    word_counts[word] = word_count\n",
        "    for feature, count in features.items():\n",
        "        if feature not in feature_counts:\n",
        "            feature_counts[feature] = count\n",
        "        else:\n",
        "            feature_counts[feature] += count\n",
        "print(total_count)\n",
        "print(word_counts)\n",
        "print(feature_counts)"
      ],
      "metadata": {
        "id": "zRdEtAY3CRPT",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2418f74c-1779-4976-81f7-d8c61e64ca55"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "34\n",
            "{'conscious': 1, 'spend': 2, 'money': 4, 'lot': 2, 'boost': 2, 'happiness': 1, 'list': 1, 'ways': 0, 'might': 4, 'waste': 4, 'time': 4, 'day': 1, 'insurance': 1, 'products': 0, 'argues': 2, 'tayne': 1, 'many': 1, 'parents': 0, 'worry': 2, 'children': 1}\n",
            "{'spend': 4, 'worry': 1, 'time': 4, 'lot': 2, 'waste': 4, 'argues': 2, 'money': 3, 'boost': 2, 'happiness': 1, 'ways': 2, 'products': 2, 'might': 2, 'day': 1, 'children': 1, 'tayne': 1, 'parents': 2}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "ii) Turn these counts into Positive PMI values.\n",
        "\n",
        "$$PPMI = \\max \\left( 0, \\log \\left( \\frac{freq(w,f) \\times freq_{tot}}{freq(w) \\times freq(f)} \\right) \\right)$$\n",
        "\n",
        "Here, $freq(w,f)$ represents the count in `word_feature_counts[w][f]` and $freq_{tot}$ is the sum of these counts for all words and features. $freq(w)$ is the total of the counts for word $w$ and $freq(f)$ is the total of the counts for feature $f$. (5 marks)"
      ],
      "metadata": {
        "id": "TbvcGUia5XMr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import math\n",
        "\n",
        "def get_ppmi(word, feature, word_feature_counts, total_count, word_counts, feature_counts):\n",
        "    fr_wf = word_feature_counts[word][feature]\n",
        "    fr_w = word_counts[word]\n",
        "    fr_f = feature_counts[feature]\n",
        "    ppmi = max(0, math.log((fr_wf * total_count) / (fr_w * fr_f)))\n",
        "    return ppmi\n",
        "\n",
        "ppmi_values = {}\n",
        "\n",
        "for word, features in word_feature_counts.items():\n",
        "    if word not in ppmi_values:\n",
        "        ppmi_values[word] = {}\n",
        "    for feature, count in features.items():\n",
        "\n",
        "print(ppmi_values)\n"
      ],
      "metadata": {
        "id": "t0YaJ4pm5nrd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 130
        },
        "outputId": "9b0cbae1-9b32-434d-f6da-267a24e5a5c6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "IndentationError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-35-990c634066fe>\"\u001b[0;36m, line \u001b[0;32m17\u001b[0m\n\u001b[0;31m    print(ppmi_values)\u001b[0m\n\u001b[0m                      ^\u001b[0m\n\u001b[0;31mIndentationError\u001b[0m\u001b[0;31m:\u001b[0m expected an indented block\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "iii) Define a function that will take two tokens and return their cosine similarity, based on the PPMI representation you have just constructed.\n",
        "\n",
        "$$cos \\left( w_1, w_2 \\right) = \\frac{w_1 \\cdot w_2}{\\sqrt{w_1 \\cdot w_1 \\times w_2 \\cdot w_2}}$$\n",
        "\n",
        "Apply this function to the tokens `\"time\"` and `\"money\"`. (7 marks)"
      ],
      "metadata": {
        "id": "0rpFUxf_5rUD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import math\n",
        "def get_cosine_similarity(token1, token2, ppmi_values):\n",
        "    ppmi1 = ppmi_values[token1]\n",
        "    ppmi2 = ppmi_values[token2]\n",
        "    dot_product = 0\n",
        "    n1 = 0\n",
        "    n2 = 0\n",
        "    for feature, value in ppmi1.items():\n",
        "        if feature in ppmi2:\n",
        "            dot_product += value * ppmi2[feature]\n",
        "        n1 += value**2\n",
        "    for value in ppmi2.values():\n",
        "        n2 += value**2\n",
        "    cosine_similarity = dot_product / (math.sqrt(n1) * math.sqrt(n2))\n",
        "    return cosine_similarity"
      ],
      "metadata": {
        "id": "Qyk0Vav8vVEh"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}