goit_python_ds_hw_12.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "kEJ8sD3NJKs_",
        "ml4mRDRMVPAt",
        "s0P9L_M7NAzZ",
        "YGcWTKVuVH_F",
        "47e9969UU3rR",
        "jDzkFwGEUjbE"
      ],
      "authorship_tag": "ABX9TyOwRoUFRP2dTU5eLfjALv2d",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/lexxai/goit_python_ds_hw_12/blob/main/goit_python_ds_hw_12.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Модуль 12. Основи NLP.\n",
        "\n",
        "*З циклу [домашніх завдань Python Data Science](https://github.com/lexxai/goit_python_data_sciense_homework).*\n",
        "\n",
        "# Домашнє завдання"
      ],
      "metadata": {
        "id": "yyLRXrtSUPez"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Зробіть summary нижчевказаного тексту використовуючи бібліотеки для NLP: `nltk` та `SpaCy`\n",
        "\n",
        "```\n",
        "The Orbiter Discovery, OV-103, is considered eligible for listing in the National Register of Historic Places (NRHP) in the context of the U.S. Space Shuttle Program (1969-2011) under Criterion A in the areas of Space Exploration and Transportation and under Criterion C in the area of Engineering. Because it has achieved significance within the past fifty years, Criteria Consideration G applies. Under Criterion A, Discovery is significant as the oldest of the three extant orbiter vehicles constructed for the Space Shuttle Program (SSP), the longest running American space program to date; she was the third of five orbiters built by NASA. Unlike the Mercury, Gemini, and Apollo programs, the SSP’s emphasis was on cost effectiveness and reusability, and eventually the construction of a space station. Including her maiden voyage (launched August 30, 1984), Discovery flew to space thirty-nine times, more than any of the other four orbiters; she was also the first orbiter to fly twenty missions. She had the honor of being chosen as the Return to Flight vehicle after both the Challenger and Columbia accidents. Discovery was the first shuttle to fly with the redesigned SRBs, a result of the Challenger accident, and the first shuttle to fly with the Phase II and Block I SSME. Discovery also carried the Hubble Space Telescope to orbit and performed two of the five servicing missions to the observatory. She flew the first and last dedicated Department of Defense (DoD) missions, as well as the first unclassified defense-related mission. In addition, Discovery was vital to the construction of the International Space Station (ISS); she flew thirteen of the thirty-seven total missions flown to the station by a U.S. Space Shuttle. She was the first orbiter to dock to the ISS, and the first to perform an exchange of a resident crew. Under Criterion C, Discovery is significant as a feat of engineering. According to Wayne Hale, a flight director from Johnson Space Center, the Space Shuttle orbiter represents a “huge technological leap from expendable rockets and capsules to a reusable, winged, hypersonic, cargo-carrying spacecraft.” Although her base structure followed a conventional aircraft design, she used advanced materials that both minimized her weight for cargo-carrying purposes and featured low thermal expansion ratios, which provided a stable base for her Thermal Protection System (TPS) materials. The Space Shuttle orbiter also featured the first reusable TPS; all previous spaceflight vehicles had a single-use, ablative heat shield. Other notable engineering achievements of the orbiter included the first reusable orbital propulsion system, and the first two-fault-tolerant Integrated Avionics System. As Hale stated, the Space Shuttle remains “the largest, fastest, winged hypersonic aircraft in history,” having regularly flown at twenty-five times the speed of sound.\n",
        "```\n"
      ],
      "metadata": {
        "id": "fGgMtFjfU5Hc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Виконання"
      ],
      "metadata": {
        "id": "Sn4us1W9VFRa"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## TEXT Preprocessor"
      ],
      "metadata": {
        "id": "kEJ8sD3NJKs_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Текст для обробки\n",
        "# text = \"This is an example sentence for tokenization and lemmatization.\"\n",
        "text = \"\"\"The Orbiter Discovery, OV-103, is considered eligible for listing in the National Register of Historic Places (NRHP) in the context of the U.S.\n",
        "Space Shuttle Program (1969-2011) under Criterion A in the areas of Space Exploration and Transportation and under Criterion C in the area of Engineering.\n",
        "Because it has achieved significance within the past fifty years, Criteria Consideration G applies.\n",
        "Under Criterion A, Discovery is significant as the oldest of the three extant orbiter vehicles constructed for the Space Shuttle Program (SSP),\n",
        "the longest running American space program to date; she was the third of five orbiters built by NASA. Unlike the Mercury, Gemini, and Apollo programs,\n",
        "the SSP’s emphasis was on cost effectiveness and reusability, and eventually the construction of a space station.\n",
        "Including her maiden voyage (launched August 30, 1984), Discovery flew to space thirty-nine times, more than any of the other four orbiters;\n",
        "she was also the first orbiter to fly twenty missions.\n",
        "She had the honor of being chosen as the Return to Flight vehicle after both the Challenger and Columbia accidents.\n",
        "Discovery was the first shuttle to fly with the redesigned SRBs, a result of the Challenger accident, and the first shuttle to fly with the Phase II\n",
        "and Block I SSME. Discovery also carried the Hubble Space Telescope to orbit and performed two of the five servicing missions to the observatory.\n",
        "She flew the first and last dedicated Department of Defense (DoD) missions, as well as the first unclassified defense-related mission.\n",
        "In addition, Discovery was vital to the construction of the International Space Station (ISS); she flew thirteen of the thirty-seven total missions\n",
        "flown to the station by a U.S. Space Shuttle. She was the first orbiter to dock to the ISS, and the first to perform an exchange of a resident crew.\n",
        "Under Criterion C, Discovery is significant as a feat of engineering. According to Wayne Hale, a flight director from Johnson Space Center,\n",
        "the Space Shuttle orbiter represents a “huge technological leap from expendable rockets and capsules to a reusable, winged, hypersonic, cargo-carrying spacecraft.\n",
        "” Although her base structure followed a conventional aircraft design, she used advanced materials that both minimized her weight for cargo-carrying purposes\n",
        "and featured low thermal expansion ratios, which provided a stable base for her Thermal Protection System (TPS) materials.\n",
        "The Space Shuttle orbiter also featured the first reusable TPS; all previous spaceflight vehicles had a single-use, ablative heat shield.\n",
        "Other notable engineering achievements of the orbiter included the first reusable orbital propulsion system, and the first two-fault-tolerant Integrated Avionics System.\n",
        "As Hale stated, the Space Shuttle remains “the largest, fastest, winged hypersonic aircraft in history,” having regularly flown at twenty-five times the speed of sound.\n",
        "\"\"\"\n",
        "\n",
        "text = \" \".join(text.split('\\n'))\n",
        "text = \" \".join(text.split())\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TrEwbKlwWBqP",
        "outputId": "d34b985c-b575-449e-fcc5-978406adf860"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['The', 'Orbiter', 'Discovery', ',', 'OV-103', ',', 'is', 'considered', 'eligible', 'for']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Реалізація через spaCy"
      ],
      "metadata": {
        "id": "MEsHbJT4JWvg"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Допоміжні функції"
      ],
      "metadata": {
        "id": "ml4mRDRMVPAt"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "KobA6-jlT5m0"
      },
      "outputs": [],
      "source": [
        "import spacy"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def get_word_frequencies_spacy(doc):\n",
        "  word_frequencies = {}\n",
        "  for word in doc:\n",
        "    word_lower = word.text.lower()\n",
        "    if word.is_stop or word.is_punct or word.is_space:\n",
        "      continue\n",
        "    word_frequencies[word_lower] = word_frequencies.get(word_lower, 0) + 1\n",
        "  return word_frequencies\n",
        "\n",
        "\n",
        "def get_ranked_sentences_index(sentences, word_frequencies):\n",
        "  ranked_sentences_index = {}\n",
        "  for i, s in enumerate(sentences):\n",
        "    sentence_score = sum(word_frequencies.get(word.text.lower(), 0) for word in s)\n",
        "    ranked_sentences_index[i] = sentence_score\n",
        "  return ranked_sentences_index\n",
        "\n",
        "\n",
        "def summarise_spacy(text, num_sentences=4):\n",
        "  nlp = spacy.load('en_core_web_sm')\n",
        "  doc = nlp(text)\n",
        "  sentences = list(doc.sents)\n",
        "  word_frequencies = get_word_frequencies_spacy(doc)\n",
        "  ranked_sentences_index = get_ranked_sentences_index(sentences, word_frequencies)\n",
        "  sort_ranked_sentences = sorted(ranked_sentences_index, key=ranked_sentences_index.get, reverse=True)\n",
        "  summary = [ sentences[i] for i in sort_ranked_sentences[:num_sentences] ]\n",
        "  return summary"
      ],
      "metadata": {
        "id": "YOouYT1mJuEf"
      },
      "execution_count": 39,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### nlargest\n"
      ],
      "metadata": {
        "id": "s0P9L_M7NAzZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from heapq import nlargest\n",
        "\n",
        "def summarise_spacy_nlargest(text, num_sentences=4):\n",
        "  nlp = spacy.load('en_core_web_sm')\n",
        "  doc = nlp(text)\n",
        "  sentences = list(doc.sents)\n",
        "  word_frequencies = get_word_frequencies_spacy(doc)\n",
        "  ranked_sentences_index = get_ranked_sentences_index(sentences, word_frequencies)\n",
        "  summary_index = nlargest(select_length, ranked_sentences_index, key = ranked_sentences_index.get)\n",
        "  summary = [sentences[i] for i in summary_index]\n",
        "  return summary"
      ],
      "metadata": {
        "id": "ApD9xyZINBm0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Резльтат summary"
      ],
      "metadata": {
        "id": "fe0EEKQDVSwt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "summary = summarise_spacy(text)\n",
        "for s in summary:\n",
        "  print(s)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wSWgHyQkMAXf",
        "outputId": "34bb28d7-88c7-4844-e8cf-5914663907db"
      },
      "execution_count": 75,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The Orbiter Discovery, OV-103, is considered eligible for listing in the National Register of Historic Places (NRHP) in the context of the U.S. Space Shuttle Program (1969-2011) under Criterion A in the areas of Space Exploration and Transportation and under Criterion C in the area of Engineering.\n",
            "Under Criterion A, Discovery is significant as the oldest of the three extant orbiter vehicles constructed for the Space Shuttle Program (SSP), the longest running American space program to date; she was the third of five orbiters built by NASA.\n",
            "In addition, Discovery was vital to the construction of the International Space Station (ISS); she flew thirteen of the thirty-seven total missions flown to the station by a U.S. Space Shuttle.\n",
            "According to Wayne Hale, a flight director from Johnson Space Center, the Space Shuttle orbiter represents a “huge technological leap from expendable rockets and capsules to a reusable, winged, hypersonic, cargo-carrying spacecraft. ”\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "summary = summarise_spacy_nlargest(text)\n",
        "for s in summary:\n",
        "  print(s)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tdNBiv0NM258",
        "outputId": "3dd1fc65-4bb9-401b-e68e-55b4c346b20c"
      },
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The Orbiter Discovery, OV-103, is considered eligible for listing in the National Register of Historic Places (NRHP) in the context of the U.S. Space Shuttle Program (1969-2011) under Criterion A in the areas of Space Exploration and Transportation and under Criterion C in the area of Engineering.\n",
            "Under Criterion A, Discovery is significant as the oldest of the three extant orbiter vehicles constructed for the Space Shuttle Program (SSP), the longest running American space program to date; she was the third of five orbiters built by NASA.\n",
            "In addition, Discovery was vital to the construction of the International Space Station (ISS); she flew thirteen of the thirty-seven total missions flown to the station by a U.S. Space Shuttle.\n",
            "According to Wayne Hale, a flight director from Johnson Space Center, the Space Shuttle orbiter represents a “huge technological leap from expendable rockets and capsules to a reusable, winged, hypersonic, cargo-carrying spacecraft. ”\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Реалізація через nltk"
      ],
      "metadata": {
        "id": "j0RLoAm0NTw1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Допоміжні функції"
      ],
      "metadata": {
        "id": "YGcWTKVuVH_F"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "from nltk.corpus import stopwords\n",
        "from nltk.tokenize import word_tokenize, sent_tokenize, punkt"
      ],
      "metadata": {
        "id": "LK3A-2aNNxe6"
      },
      "execution_count": 42,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def get_word_frequencies_nltk(text):\n",
        "  nltk.download('punkt', quiet=True)\n",
        "  nltk.download('stopwords', quiet=True)\n",
        "  tokens = word_tokenize(text)\n",
        "  stop_words = set(stopwords.words('english'))\n",
        "  punkt_data = nltk.data.load('/root/nltk_data/tokenizers/punkt/english.pickle')\n",
        "  punctuation = list(punkt_data.PUNCTUATION)\n",
        "  punctuation.extend(['\\n','(',')','\"','“','’'])\n",
        "\n",
        "  word_frequencies = {}\n",
        "  for word in tokens:\n",
        "    word_lower = word.lower()\n",
        "    if word_lower in stop_words:\n",
        "      continue\n",
        "    if word_lower in punctuation:\n",
        "      continue\n",
        "    word_frequencies[word_lower] = word_frequencies.get(word_lower, 0) + 1\n",
        "  return word_frequencies\n",
        "\n",
        "\n",
        "def get_ranked_sentences_index_nltk(sentences, word_frequencies):\n",
        "  ranked_sentences_index = {}\n",
        "  for i, s in enumerate(sentences):\n",
        "    words = word_tokenize(s)\n",
        "    sentence_score = sum(word_frequencies.get(word.lower(), 0) for word in words)\n",
        "    ranked_sentences_index[i] = sentence_score\n",
        "  return ranked_sentences_index\n",
        "\n",
        "\n",
        "def summarise_nltk(text, num_sentences=4):\n",
        "  sentences = sent_tokenize(text)\n",
        "  word_frequencies = get_word_frequencies_nltk(text)\n",
        "  ranked_sentences_index = get_ranked_sentences_index_nltk(sentences, word_frequencies)\n",
        "  sort_ranked_sentences = sorted(ranked_sentences_index, key=ranked_sentences_index.get, reverse=True)\n",
        "  summary = [ sentences[i] for i in sort_ranked_sentences[:num_sentences] ]\n",
        "  return summary"
      ],
      "metadata": {
        "id": "ic7HkKujNbek"
      },
      "execution_count": 70,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Резльтат summary"
      ],
      "metadata": {
        "id": "47e9969UU3rR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "summary = summarise_nltk(text)\n",
        "for s in summary:\n",
        "  print(s)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fH3DqEkcPY1W",
        "outputId": "38cfc6da-1e7e-4f4e-83f3-314bfa71fe99"
      },
      "execution_count": 72,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The Orbiter Discovery, OV-103, is considered eligible for listing in the National Register of Historic Places (NRHP) in the context of the U.S. Space Shuttle Program (1969-2011) under Criterion A in the areas of Space Exploration and Transportation and under Criterion C in the area of Engineering.\n",
            "Under Criterion A, Discovery is significant as the oldest of the three extant orbiter vehicles constructed for the Space Shuttle Program (SSP), the longest running American space program to date; she was the third of five orbiters built by NASA.\n",
            "In addition, Discovery was vital to the construction of the International Space Station (ISS); she flew thirteen of the thirty-seven total missions flown to the station by a U.S. Space Shuttle.\n",
            "According to Wayne Hale, a flight director from Johnson Space Center, the Space Shuttle orbiter represents a “huge technological leap from expendable rockets and capsules to a reusable, winged, hypersonic, cargo-carrying spacecraft.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Процес вивчення"
      ],
      "metadata": {
        "id": "jDzkFwGEUjbE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nlp = spacy.load('en_core_web_sm')"
      ],
      "metadata": {
        "id": "4g4YsO9jVQ3k"
      },
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Токенізація\n",
        "doc = nlp(text)\n",
        "tokens = [token.text for token in doc]\n",
        "print(tokens[:10])"
      ],
      "metadata": {
        "id": "YB4K7dW3JGZn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# nltk.download('punkt')\n",
        "# nltk.download('stopwords')\n",
        "\n",
        "# tokens = word_tokenize(text)\n",
        "# sentences = sent_tokenize(text)\n",
        "# stop_words = set(stopwords.words('english'))\n",
        "\n",
        "sentences = list(doc.sents)\n",
        "\n"
      ],
      "metadata": {
        "id": "VqfC7xyzjZcy"
      },
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "len(tokens)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OyvuzyjCkbWO",
        "outputId": "b3c047d9-2143-42fe-ac73-a8cac693596f"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "539"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tokens[:5]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CxTsnS64PiDQ",
        "outputId": "913e461c-8b02-4d2d-f849-539e901194e6"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['The', 'Orbiter', 'Discovery', ',', 'OV-103']"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "len(sentences)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LT_TZRZqjvMP",
        "outputId": "52f49b28-9c29-4014-973a-97e59036a739"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "17"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentences[:5]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pRTM5lNRPPgP",
        "outputId": "a9e68300-70ea-4419-a744-c88c047edf9a"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[The Orbiter Discovery, OV-103, is considered eligible for listing in the National Register of Historic Places (NRHP) in the context of the U.S. Space Shuttle Program (1969-2011) under Criterion A in the areas of Space Exploration and Transportation and under Criterion C in the area of Engineering.,\n",
              " Because it has achieved significance within the past fifty years, Criteria Consideration G applies.,\n",
              " Under Criterion A, Discovery is significant as the oldest of the three extant orbiter vehicles constructed for the Space Shuttle Program (SSP), the longest running American space program to date; she was the third of five orbiters built by NASA.,\n",
              " Unlike the Mercury, Gemini, and Apollo programs, the SSP’s emphasis was on cost effectiveness and reusability, and eventually the construction of a space station.,\n",
              " Including her maiden voyage (launched August 30, 1984), Discovery flew to space thirty-nine times, more than any of the other four orbiters; she was also the first orbiter to fly twenty missions.]"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# len(stop_words)"
      ],
      "metadata": {
        "id": "dcgp-gnAkSDz"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# list(stop_words)[:5]"
      ],
      "metadata": {
        "id": "WM3Nh93DPS_7"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# tokens_punkt = list(set(token.text for token in doc if token.is_punct))\n",
        "# tokens_punkt"
      ],
      "metadata": {
        "id": "GaDdHnrDP0Wr"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# punkt_data = nltk.data.load('/root/nltk_data/tokenizers/punkt/english.pickle')\n",
        "# punctuation = list(punkt_data.PUNCTUATION)\n",
        "# punctuation.extend(tokens_punkt)\n",
        "# punctuation.extend(['\\n','’s'])\n",
        "# print(punctuation)"
      ],
      "metadata": {
        "id": "bdH1DTMaqbBN"
      },
      "execution_count": 38,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "word_frequencies = {}\n",
        "for word in doc:\n",
        "  word_lower = word.text.lower()\n",
        "  if word.is_stop or word.is_punct or word.is_space:\n",
        "    continue\n",
        "  word_frequencies[word_lower] = word_frequencies.get(word_lower, 0) + 1"
      ],
      "metadata": {
        "id": "s-cPRJ_7t3jz"
      },
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "list(word_frequencies.items())[:20]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5EvdNGzzt5-U",
        "outputId": "d2314fd9-f310-4c9c-da75-f22bee3cb0a3"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('orbiter', 7),\n",
              " ('discovery', 7),\n",
              " ('ov-103', 1),\n",
              " ('considered', 1),\n",
              " ('eligible', 1),\n",
              " ('listing', 1),\n",
              " ('national', 1),\n",
              " ('register', 1),\n",
              " ('historic', 1),\n",
              " ('places', 1),\n",
              " ('nrhp', 1),\n",
              " ('context', 1),\n",
              " ('u.s.', 2),\n",
              " ('space', 13),\n",
              " ('shuttle', 8),\n",
              " ('program', 3),\n",
              " ('1969', 1),\n",
              " ('2011', 1),\n",
              " ('criterion', 4),\n",
              " ('areas', 1)]"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "list(sorted(word_frequencies.items(), key=lambda x: x[1], reverse=True))[:20]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rmEeGfhFLmLD",
        "outputId": "532a5709-13a8-4c45-f321-141cabfd99d2"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('space', 13),\n",
              " ('shuttle', 8),\n",
              " ('orbiter', 7),\n",
              " ('discovery', 7),\n",
              " ('criterion', 4),\n",
              " ('missions', 4),\n",
              " ('program', 3),\n",
              " ('engineering', 3),\n",
              " ('station', 3),\n",
              " ('flew', 3),\n",
              " ('fly', 3),\n",
              " ('reusable', 3),\n",
              " ('system', 3),\n",
              " ('u.s.', 2),\n",
              " ('c', 2),\n",
              " ('significant', 2),\n",
              " ('vehicles', 2),\n",
              " ('ssp', 2),\n",
              " ('orbiters', 2),\n",
              " ('construction', 2)]"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "ranked_sentences_index = {}\n",
        "\n",
        "for i, s in enumerate(doc.sents):\n",
        "  sentence_score = sum(word_frequencies.get(word.text.lower(), 0) for word in s)\n",
        "  ranked_sentences_index[i] = sentence_score\n",
        "  print(i, \"-------\", sentence_score)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "odOba6nBiEEc",
        "outputId": "6cde0a16-a9fb-476d-a50b-f91edcab08b4"
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0 ------- 82\n",
            "1 ------- 8\n",
            "2 ------- 75\n",
            "3 ------- 30\n",
            "4 ------- 50\n",
            "5 ------- 10\n",
            "6 ------- 39\n",
            "7 ------- 31\n",
            "8 ------- 17\n",
            "9 ------- 70\n",
            "10 ------- 14\n",
            "11 ------- 19\n",
            "12 ------- 69\n",
            "13 ------- 39\n",
            "14 ------- 44\n",
            "15 ------- 28\n",
            "16 ------- 42\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sort_ranked_sentences = sorted(ranked_sentences_index, key=ranked_sentences_index.get, reverse=True)\n",
        "sort_ranked_sentences"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZxajLzGLx48B",
        "outputId": "a0da867d-f9a7-4c51-ae33-873b5f34eb5e"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0, 2, 9, 12, 4, 14, 16, 6, 13, 7, 3, 15, 11, 8, 10, 5, 1]"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "num_sentences = 4\n",
        "summary = [ sentences[i] for i in sort_ranked_sentences[:num_sentences] ]\n"
      ],
      "metadata": {
        "id": "erTnufOj46dC"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "for s in summary:\n",
        "  print(s)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iwgZ19ER6xHr",
        "outputId": "01a80b41-ea76-479a-91d4-3f866d2eaf71"
      },
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The Orbiter Discovery, OV-103, is considered eligible for listing in the National Register of Historic Places (NRHP) in the context of the U.S. Space Shuttle Program (1969-2011) under Criterion A in the areas of Space Exploration and Transportation and under Criterion C in the area of Engineering.\n",
            "Under Criterion A, Discovery is significant as the oldest of the three extant orbiter vehicles constructed for the Space Shuttle Program (SSP), the longest running American space program to date; she was the third of five orbiters built by NASA.\n",
            "In addition, Discovery was vital to the construction of the International Space Station (ISS); she flew thirteen of the thirty-seven total missions flown to the station by a U.S. Space Shuttle.\n",
            "According to Wayne Hale, a flight director from Johnson Space Center, the Space Shuttle orbiter represents a “huge technological leap from expendable rockets and capsules to a reusable, winged, hypersonic, cargo-carrying spacecraft. ”\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "for i in range(4):\n",
        "  print(sentences[sort_ranked_sentences[i]])\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AXOOpB4OyNaO",
        "outputId": "655ee375-a0d2-4783-fa96-59cd57ec485a"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The Orbiter Discovery, OV-103, is considered eligible for listing in the National Register of Historic Places (NRHP) in the context of the U.S. Space Shuttle Program (1969-2011) under Criterion A in the areas of Space Exploration and Transportation and under Criterion C in the area of Engineering.\n",
            "Under Criterion A, Discovery is significant as the oldest of the three extant orbiter vehicles constructed for the Space Shuttle Program (SSP), the longest running American space program to date; she was the third of five orbiters built by NASA.\n",
            "In addition, Discovery was vital to the construction of the International Space Station (ISS); she flew thirteen of the thirty-seven total missions flown to the station by a U.S. Space Shuttle.\n",
            "According to Wayne Hale, a flight director from Johnson Space Center, the Space Shuttle orbiter represents a “huge technological leap from expendable rockets and capsules to a reusable, winged, hypersonic, cargo-carrying spacecraft. ”\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from heapq import nlargest"
      ],
      "metadata": {
        "id": "MiyHb01pOkbM"
      },
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "select_length = 4\n",
        "summary_index = nlargest(select_length, ranked_sentences_index, key = ranked_sentences_index.get)\n",
        "summary = [sentences[i] for i in summary_index]\n",
        "for s in summary:\n",
        "  print(s)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DNa1JxzFPKXT",
        "outputId": "985229ce-05be-4467-96cc-86028642b3ad"
      },
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The Orbiter Discovery, OV-103, is considered eligible for listing in the National Register of Historic Places (NRHP) in the context of the U.S. Space Shuttle Program (1969-2011) under Criterion A in the areas of Space Exploration and Transportation and under Criterion C in the area of Engineering.\n",
            "Under Criterion A, Discovery is significant as the oldest of the three extant orbiter vehicles constructed for the Space Shuttle Program (SSP), the longest running American space program to date; she was the third of five orbiters built by NASA.\n",
            "In addition, Discovery was vital to the construction of the International Space Station (ISS); she flew thirteen of the thirty-seven total missions flown to the station by a U.S. Space Shuttle.\n",
            "According to Wayne Hale, a flight director from Johnson Space Center, the Space Shuttle orbiter represents a “huge technological leap from expendable rockets and capsules to a reusable, winged, hypersonic, cargo-carrying spacecraft. ”\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Висновок"
      ],
      "metadata": {
        "id": "YZUGXTGeV2wU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Завдання було на онсові тексту створити короткий зміст цього тексту.\n",
        "\n",
        "Для фрмування короткого тексту було вибрано було 4 речення, на основі частоти вживання слів у цих реченях.\n",
        "\n",
        "В результаті роботи було викорисанно декілька підходів з викорисатнням різних бібліотек `nltk` та `spaCy`.\n",
        "\n",
        "В результаті було отрманно однаковий результат summary з 4-х речень:"
      ],
      "metadata": {
        "id": "daJnWQLcWUk1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "The Orbiter Discovery, OV-103, is considered eligible for listing in the National Register of Historic Places (NRHP) in the context of the U.S. Space Shuttle Program (1969-2011) under Criterion A in the areas of Space Exploration and Transportation and under Criterion C in the area of Engineering.\n",
        "Under Criterion A, Discovery is significant as the oldest of the three extant orbiter vehicles constructed for the Space Shuttle Program (SSP), the longest running American space program to date; she was the third of five orbiters built by NASA.\n",
        "In addition, Discovery was vital to the construction of the International Space Station (ISS); she flew thirteen of the thirty-seven total missions flown to the station by a U.S. Space Shuttle.\n",
        "According to Wayne Hale, a flight director from Johnson Space Center, the Space Shuttle orbiter represents a “huge technological leap from expendable rockets and capsules to a reusable, winged, hypersonic, cargo-carrying spacecraft.\n",
        "```\n",
        "\n"
      ],
      "metadata": {
        "id": "_iqKl-A-WCQR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Переклад стислого змісту ситемою автоматичного прекладу DeepL:"
      ],
      "metadata": {
        "id": "tHNiFxCXYRsf"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "Орбітальний корабель \"Діскавері\" (OV-103) вважається таким, що має право на внесення до Національного реєстру історичних місць (NRHP) в контексті Програми космічних човників США (1969-2011 рр.) за критерієм А в галузі космічних досліджень і транспорту та за критерієм С в галузі інженерії.\n",
        "За критерієм А \"Діскавері\" має важливе значення як найстаріший з трьох збережених орбітальних кораблів, побудованих для програми \"Спейс Шаттл\" (SSP), найдовшої американської космічної програми на сьогоднішній день; вона була третім з п'яти орбітальних кораблів, побудованих NASA.\n",
        "Крім того, \"Діскавері\" відіграла важливу роль у будівництві Міжнародної космічної станції (МКС); вона здійснила тринадцять з тридцяти семи польотів на станцію за допомогою американських шатлів.\n",
        "За словами Вейна Хейла, керівника польотів Космічного центру імені Джонсона, орбітальний корабель \"Спейс Шаттл\" являє собою \"величезний технологічний стрибок від одноразових ракет і капсул до багаторазового, крилатого, гіперзвукового, вантажного космічного корабля.\n",
        "```"
      ],
      "metadata": {
        "id": "-GKNgqzqX64E"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "jdJFBM_PYd8D"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}