IR_PLDA.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "IR-PLDA.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Information Retrieval for Documents Using PLDA\n",
        "The main process is taken from [here](https://github.com/RaviSoji/plda/blob/master/mnist_demo/mnist_demo.ipynb).\n",
        "\n",
        "**Note: This methods was not good at all so we decided to use others which you can find out them in `DR.TEIT.ipynb` from [here](https://github.com/sharif-multidoc2dial/Docalog-2022).**"
      ],
      "metadata": {
        "id": "MkmR10Bz1dm2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install https://github.com/sadrasabouri/plda/tarball/master"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BxJg8Nfa_UOj",
        "outputId": "65a25389-32e4-4a12-f52d-f605be65d05c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting https://github.com/sadrasabouri/plda/tarball/master\n",
            "  Downloading https://github.com/sadrasabouri/plda/tarball/master\n",
            "\u001b[K     \\ 845 kB 1.7 MB/s\n",
            "\u001b[?25hBuilding wheels for collected packages: plda\n",
            "  Building wheel for plda (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for plda: filename=plda-0.1.0-py3-none-any.whl size=13655 sha256=300c7c59b9f37d195280649631839567e2de9b92e9d4e127f74abcc832537d22\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-91u08quk/wheels/17/57/2b/069666589a33ecf03d21ecebc97313b9fa09b7913577b61dd4\n",
            "Successfully built plda\n",
            "Installing collected packages: plda\n",
            "Successfully installed plda-0.1.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Dataset\n",
        "### Dataset Description\n",
        "\n",
        "- **mutldoc2dial_doc.json** contains the documents that are indexed by key `domain` and `doc_id` . Each document instance includes the following,\n",
        "\n",
        "  - `doc_id`: the ID of a document;\n",
        "  - `title`: the title of the document;\n",
        "  - `domain`: the domain of the document;\n",
        "  - `doc_text`: the text content of the document (without HTML markups);\n",
        "  - `doc_html_ts`: the document content with HTML markups and the annotated spans that are indicated by `text_id` attribute, which corresponds to `id_sp`.\n",
        "  - `doc_html_raw`: the document content with HTML markups and without span annotations.\n",
        "  - `spans`: key-value pairs of all spans in the document, with `id_sp` as key. Each span includes the following,\n",
        "    - `id_sp`: the id of a  span as noted by `text_id` in  `doc_html_ts`;\n",
        "    - `start_sp`/  `end_sp`: the start/end position of the text span in `doc_text`;\n",
        "    - `text_sp`: the text content of the span.\n",
        "    - `id_sec`: the id of the (sub)section (e.g. `<p>`) or title (`<h2>`) that contains the span.\n",
        "    - `start_sec` / `end_sec`: the start/end position of the (sub)section in `doc_text`.\n",
        "    - `text_sec`: the text of the (sub)section.\n",
        "    - `title`: the title of the (sub)section.\n",
        "    - `parent_titles`: the parent titles of the `title`.\n",
        "\n",
        "- **multidoc2dial_dial_train.json** and **multidoc2dial_dial_validation.json**  contain the training and dev split of dialogue data that are indexed by key `domain` . Please note: **For test split, we only include a dummy file in this version.**\n",
        "\n",
        "  Each dialogue instance includes the following,\n",
        "\n",
        "  - `dial_id`: the ID of a dialogue;\n",
        "  - `turns`: a list of dialogue turns. Each turn includes,\n",
        "    - `turn_id`: the time order of the turn;\n",
        "    - `role`: either \"agent\" or \"user\";READ\n",
        "    - `da`: dialogue act;\n",
        "    - `references`: a list of spans with `id_sp` ,  `label` and `doc_id`. `references` is empty if a turn is for indicating previous user query not answerable or irrelevant to the document. **Note** that labels \"*precondition*\"/\"*solution*\" are fuzzy annotations that indicate whether a span is for describing a conditional context or a solution.\n",
        "    - `utterance`: the human-generated utterance based on the dialogue scene.\n",
        "Downloading the training dataset:"
      ],
      "metadata": {
        "id": "IxXgms0k09oh"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Hy_wltIuzTG4",
        "outputId": "feceaff9-6713-4daa-b554-ed4f86e7e3af"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=1Ln4pU93_ofAkbrz1uibsNABB0QsEaOXw\n",
            "To: /content/multidoc2dial.zip\n",
            "100% 6.45M/6.45M [00:00<00:00, 42.3MB/s]\n"
          ]
        }
      ],
      "source": [
        "!gdown --id 1Ln4pU93_ofAkbrz1uibsNABB0QsEaOXw"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Unziping the dataset:"
      ],
      "metadata": {
        "id": "LE6L5c971WXb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!unzip multidoc2dial.zip"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MNiVChSj1Hhx",
        "outputId": "97689e9d-226b-4972-f49e-761d3aff30ef"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Archive:  multidoc2dial.zip\n",
            "   creating: multidoc2dial/\n",
            "  inflating: multidoc2dial/multidoc2dial_dial_validation.json  \n",
            "  inflating: multidoc2dial/multidoc2dial_dial_train.json  \n",
            "  inflating: multidoc2dial/multidoc2dial_dial_test.json  \n",
            "  inflating: multidoc2dial/multidoc2dial_doc.json  \n",
            "  inflating: multidoc2dial/README.md  \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Preprocess the data\n",
        "In this section we'll form the training sampels for the documnet classifier based on PLDA as fallows:\n",
        "\n",
        "$$\n",
        "(X_{ij}, y_i)\n",
        "$$\n",
        "where $X_{ij}$ is the embedding of $j$th span from $i$th document and $y_i$ is the label of $i$th document."
      ],
      "metadata": {
        "id": "w7Fy1-3V_aly"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def clean_text(text):\n",
        "    \"\"\"\n",
        "    Clean the given text.\n",
        "\n",
        "    :param text: input text\n",
        "    :type text: str\n",
        "    :return: cleaned string\n",
        "    \"\"\"\n",
        "    return text.strip()"
      ],
      "metadata": {
        "id": "OjNQtoE_HGlt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import json\n",
        "with open('multidoc2dial/multidoc2dial_doc.json', 'r') as f:\n",
        "    multidoc2dial_doc = json.load(f)"
      ],
      "metadata": {
        "id": "uVGmrY1F1cjv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "multidoc2dial_doc['doc_data']['ssa']['Benefits Planner: Survivors | Planning For Your Survivors | Social Security Administration#1_0']['spans']['1']"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1Al_Qv8GBa9d",
        "outputId": "31eabafb-fec7-4ce3-93c2-be504288ce6a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'end_sec': 61,\n",
              " 'end_sp': 61,\n",
              " 'id_sec': 't_0',\n",
              " 'id_sp': '1',\n",
              " 'parent_titles': [],\n",
              " 'start_sec': 0,\n",
              " 'start_sp': 0,\n",
              " 'tag': 'h2',\n",
              " 'text_sec': '\\n\\nBenefits Planner: Survivors | Planning For Your Survivors \\n',\n",
              " 'text_sp': '\\n\\nBenefits Planner: Survivors | Planning For Your Survivors \\n',\n",
              " 'title': 'Benefits Planner: Survivors | Planning For Your Survivors'}"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "doc_sentence_train = []\n",
        "doc_label_train = []\n",
        "for doc_idx1 in multidoc2dial_doc['doc_data']:\n",
        "    for doc_idx2 in multidoc2dial_doc['doc_data'][doc_idx1]:\n",
        "        for doc_idx3 in multidoc2dial_doc['doc_data'][doc_idx1]\\\n",
        "                                          [doc_idx2]['spans']:\n",
        "            doc_sentence_train.append(clean_text(multidoc2dial_doc['doc_data']\\\n",
        "                                                 [doc_idx1][doc_idx2]['spans']\\\n",
        "                                                 [doc_idx3]['text_sp']))\n",
        "            doc_label_train.append(doc_idx2)"
      ],
      "metadata": {
        "id": "QPmD6QTxCT1Q"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "len(doc_label_train)  # Number of total samples"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7ON_XbIkGnqC",
        "outputId": "15072e65-1943-4889-f93c-8e70933dc83f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "35659"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "len(set(doc_label_train))   # Number of total docs"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VSu2Csx6Gxsv",
        "outputId": "c5645e7f-bebf-4a12-d3c7-306fa3abfdfd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "488"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "for i in [1, 100, 1000, 2000, 5000]:\n",
        "    print(doc_sentence_train[i])\n",
        "    print(doc_label_train[i])\n",
        "    print('--' * 20)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rHmuyw3TDs9D",
        "outputId": "6e9c0bd5-39ef-4a14-e78f-32f591b39de8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "As you plan for the future ,\n",
            "Benefits Planner: Survivors | Planning For Your Survivors | Social Security Administration#1_0\n",
            "----------------------------------------\n",
            "you'll want to think about what your family would need if you should die now.\n",
            "Benefits Planner: Survivors | Planning For Your Survivors | Social Security Administration#2_0\n",
            "----------------------------------------\n",
            "Religious record made before the age of 5 showing your date of birth ;\n",
            "Learn what documents you will need to get a Social Security Card | Social Security Administration#10_0\n",
            "----------------------------------------\n",
            "What happens after I apply?\n",
            "Disability Benefits | Social Security Administration#1_0\n",
            "----------------------------------------\n",
            "For more information about our disability claims process ,\n",
            "Benefits Planner: Disability | How You Qualify | Social Security Administration#2_0\n",
            "----------------------------------------\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Encoding the sentences\n",
        "We use the LaBSE which is a Language-agnostic BERT Sentence Encoder (LaBSE) is a BERT-based model trained for sentence embedding for 109 languages. The pre-training process combines masked language modeling with translation language modeling. The model is useful for getting multilingual sentence embeddings and for bi-text retrieval."
      ],
      "metadata": {
        "id": "hJSIUOdXIB6a"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install transformers"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yG5PITNOGJHu",
        "outputId": "82836e53-417e-444a-84fc-a19182fa9703"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: transformers in /usr/local/lib/python3.7/dist-packages (4.17.0)\n",
            "Requirement already satisfied: sacremoses in /usr/local/lib/python3.7/dist-packages (from transformers) (0.0.49)\n",
            "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from transformers) (4.11.3)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers) (4.63.0)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers) (2.23.0)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from transformers) (21.3)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.7/dist-packages (from transformers) (6.0)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (2019.12.20)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.1.0 in /usr/local/lib/python3.7/dist-packages (from transformers) (0.4.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers) (3.6.0)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (1.21.5)\n",
            "Requirement already satisfied: tokenizers!=0.11.3,>=0.11.1 in /usr/local/lib/python3.7/dist-packages (from transformers) (0.12.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.7/dist-packages (from huggingface-hub<1.0,>=0.1.0->transformers) (3.10.0.2)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->transformers) (3.0.7)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->transformers) (3.7.0)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2021.10.8)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (1.24.3)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.15.0)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (7.1.2)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.1.0)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import AutoTokenizer, AutoModel, AutoConfig\n",
        "import numpy as np\n",
        "import torch\n",
        "from torch.nn.functional import normalize"
      ],
      "metadata": {
        "id": "z5hknCdDIRzE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer_labse = AutoTokenizer.from_pretrained(\"setu4993/LaBSE\")\n",
        "model_labse = AutoModel.from_pretrained(\"setu4993/LaBSE\")"
      ],
      "metadata": {
        "id": "UBpRtQ3DIXdn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### `get_embeddings`\n",
        "In this method we extract the **pooler output** (Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining)."
      ],
      "metadata": {
        "id": "e544tO0fIgyX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def get_embeddings(sentece):\n",
        "    \"\"\"\n",
        "    Return embeddings based on encoder model\n",
        "\n",
        "    :param sentence: input sentence(s)\n",
        "    :type sentence: str or list of strs\n",
        "    :return: embeddings\n",
        "    \"\"\"\n",
        "    tokenized = tokenizer_labse(sentece,\n",
        "                                return_tensors=\"pt\",\n",
        "                                padding=True)\n",
        "    with torch.no_grad():\n",
        "        embeddings = model_labse(**tokenized)\n",
        "    \n",
        "    return np.squeeze(np.array(embeddings.pooler_output))"
      ],
      "metadata": {
        "id": "XhP-sNpLIfwA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "TRAIN_SIZE = len(doc_label_train)  # for final training"
      ],
      "metadata": {
        "id": "nmcq7_LiJLcV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "X = []\n",
        "labels = list(set(doc_label_train))\n",
        "y = []\n",
        "progress = 0\n",
        "for sentence, label in zip(doc_sentence_train[:TRAIN_SIZE],\n",
        "                           doc_label_train[:TRAIN_SIZE]):\n",
        "    X.append(get_embeddings(sentence))\n",
        "    y.append(labels.index(label))\n",
        "    progress += 1\n",
        "    if progress % 50 == 0:\n",
        "        print('Progress Percent = {}%'.format(100 * progress / TRAIN_SIZE))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BdWIe8w0IzZx",
        "outputId": "21d7ec92-ee24-4915-a93f-4e9888d86ccb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Progress Percent = 0.14021705600269216%\n",
            "Progress Percent = 0.2804341120053843%\n",
            "Progress Percent = 0.4206511680080765%\n",
            "Progress Percent = 0.5608682240107686%\n",
            "Progress Percent = 0.7010852800134608%\n",
            "Progress Percent = 0.841302336016153%\n",
            "Progress Percent = 0.9815193920188452%\n",
            "Progress Percent = 1.1217364480215373%\n",
            "Progress Percent = 1.2619535040242296%\n",
            "Progress Percent = 1.4021705600269216%\n",
            "Progress Percent = 1.542387616029614%\n",
            "Progress Percent = 1.682604672032306%\n",
            "Progress Percent = 1.822821728034998%\n",
            "Progress Percent = 1.9630387840376904%\n",
            "Progress Percent = 2.1032558400403825%\n",
            "Progress Percent = 2.2434728960430745%\n",
            "Progress Percent = 2.383689952045767%\n",
            "Progress Percent = 2.523907008048459%\n",
            "Progress Percent = 2.664124064051151%\n",
            "Progress Percent = 2.8043411200538433%\n",
            "Progress Percent = 2.9445581760565354%\n",
            "Progress Percent = 3.084775232059228%\n",
            "Progress Percent = 3.22499228806192%\n",
            "Progress Percent = 3.365209344064612%\n",
            "Progress Percent = 3.505426400067304%\n",
            "Progress Percent = 3.645643456069996%\n",
            "Progress Percent = 3.7858605120726887%\n",
            "Progress Percent = 3.9260775680753808%\n",
            "Progress Percent = 4.066294624078073%\n",
            "Progress Percent = 4.206511680080765%\n",
            "Progress Percent = 4.346728736083457%\n",
            "Progress Percent = 4.486945792086149%\n",
            "Progress Percent = 4.627162848088841%\n",
            "Progress Percent = 4.767379904091534%\n",
            "Progress Percent = 4.907596960094226%\n",
            "Progress Percent = 5.047814016096918%\n",
            "Progress Percent = 5.18803107209961%\n",
            "Progress Percent = 5.328248128102302%\n",
            "Progress Percent = 5.4684651841049945%\n",
            "Progress Percent = 5.608682240107687%\n",
            "Progress Percent = 5.748899296110379%\n",
            "Progress Percent = 5.889116352113071%\n",
            "Progress Percent = 6.029333408115763%\n",
            "Progress Percent = 6.169550464118456%\n",
            "Progress Percent = 6.309767520121148%\n",
            "Progress Percent = 6.44998457612384%\n",
            "Progress Percent = 6.590201632126532%\n",
            "Progress Percent = 6.730418688129224%\n",
            "Progress Percent = 6.870635744131916%\n",
            "Progress Percent = 7.010852800134608%\n",
            "Progress Percent = 7.1510698561373%\n",
            "Progress Percent = 7.291286912139992%\n",
            "Progress Percent = 7.431503968142684%\n",
            "Progress Percent = 7.571721024145377%\n",
            "Progress Percent = 7.7119380801480695%\n",
            "Progress Percent = 7.8521551361507615%\n",
            "Progress Percent = 7.992372192153454%\n",
            "Progress Percent = 8.132589248156146%\n",
            "Progress Percent = 8.272806304158838%\n",
            "Progress Percent = 8.41302336016153%\n",
            "Progress Percent = 8.553240416164222%\n",
            "Progress Percent = 8.693457472166914%\n",
            "Progress Percent = 8.833674528169606%\n",
            "Progress Percent = 8.973891584172298%\n",
            "Progress Percent = 9.11410864017499%\n",
            "Progress Percent = 9.254325696177682%\n",
            "Progress Percent = 9.394542752180374%\n",
            "Progress Percent = 9.534759808183068%\n",
            "Progress Percent = 9.67497686418576%\n",
            "Progress Percent = 9.815193920188452%\n",
            "Progress Percent = 9.955410976191144%\n",
            "Progress Percent = 10.095628032193837%\n",
            "Progress Percent = 10.235845088196529%\n",
            "Progress Percent = 10.37606214419922%\n",
            "Progress Percent = 10.516279200201913%\n",
            "Progress Percent = 10.656496256204605%\n",
            "Progress Percent = 10.796713312207297%\n",
            "Progress Percent = 10.936930368209989%\n",
            "Progress Percent = 11.077147424212681%\n",
            "Progress Percent = 11.217364480215373%\n",
            "Progress Percent = 11.357581536218065%\n",
            "Progress Percent = 11.497798592220757%\n",
            "Progress Percent = 11.63801564822345%\n",
            "Progress Percent = 11.778232704226141%\n",
            "Progress Percent = 11.918449760228834%\n",
            "Progress Percent = 12.058666816231526%\n",
            "Progress Percent = 12.19888387223422%\n",
            "Progress Percent = 12.339100928236912%\n",
            "Progress Percent = 12.479317984239604%\n",
            "Progress Percent = 12.619535040242296%\n",
            "Progress Percent = 12.759752096244988%\n",
            "Progress Percent = 12.89996915224768%\n",
            "Progress Percent = 13.040186208250372%\n",
            "Progress Percent = 13.180403264253064%\n",
            "Progress Percent = 13.320620320255756%\n",
            "Progress Percent = 13.460837376258448%\n",
            "Progress Percent = 13.60105443226114%\n",
            "Progress Percent = 13.741271488263832%\n",
            "Progress Percent = 13.881488544266524%\n",
            "Progress Percent = 14.021705600269216%\n",
            "Progress Percent = 14.161922656271908%\n",
            "Progress Percent = 14.3021397122746%\n",
            "Progress Percent = 14.442356768277293%\n",
            "Progress Percent = 14.582573824279985%\n",
            "Progress Percent = 14.722790880282677%\n",
            "Progress Percent = 14.863007936285369%\n",
            "Progress Percent = 15.003224992288063%\n",
            "Progress Percent = 15.143442048290755%\n",
            "Progress Percent = 15.283659104293447%\n",
            "Progress Percent = 15.423876160296139%\n",
            "Progress Percent = 15.564093216298831%\n",
            "Progress Percent = 15.704310272301523%\n",
            "Progress Percent = 15.844527328304215%\n",
            "Progress Percent = 15.984744384306907%\n",
            "Progress Percent = 16.124961440309598%\n",
            "Progress Percent = 16.26517849631229%\n",
            "Progress Percent = 16.405395552314985%\n",
            "Progress Percent = 16.545612608317676%\n",
            "Progress Percent = 16.68582966432037%\n",
            "Progress Percent = 16.82604672032306%\n",
            "Progress Percent = 16.966263776325754%\n",
            "Progress Percent = 17.106480832328444%\n",
            "Progress Percent = 17.246697888331138%\n",
            "Progress Percent = 17.386914944333828%\n",
            "Progress Percent = 17.527132000336522%\n",
            "Progress Percent = 17.667349056339212%\n",
            "Progress Percent = 17.807566112341906%\n",
            "Progress Percent = 17.947783168344596%\n",
            "Progress Percent = 18.08800022434729%\n",
            "Progress Percent = 18.22821728034998%\n",
            "Progress Percent = 18.368434336352674%\n",
            "Progress Percent = 18.508651392355365%\n",
            "Progress Percent = 18.64886844835806%\n",
            "Progress Percent = 18.78908550436075%\n",
            "Progress Percent = 18.929302560363443%\n",
            "Progress Percent = 19.069519616366136%\n",
            "Progress Percent = 19.209736672368827%\n",
            "Progress Percent = 19.34995372837152%\n",
            "Progress Percent = 19.49017078437421%\n",
            "Progress Percent = 19.630387840376905%\n",
            "Progress Percent = 19.770604896379595%\n",
            "Progress Percent = 19.91082195238229%\n",
            "Progress Percent = 20.05103900838498%\n",
            "Progress Percent = 20.191256064387673%\n",
            "Progress Percent = 20.331473120390363%\n",
            "Progress Percent = 20.471690176393057%\n",
            "Progress Percent = 20.611907232395748%\n",
            "Progress Percent = 20.75212428839844%\n",
            "Progress Percent = 20.89234134440113%\n",
            "Progress Percent = 21.032558400403826%\n",
            "Progress Percent = 21.172775456406516%\n",
            "Progress Percent = 21.31299251240921%\n",
            "Progress Percent = 21.4532095684119%\n",
            "Progress Percent = 21.593426624414594%\n",
            "Progress Percent = 21.733643680417288%\n",
            "Progress Percent = 21.873860736419978%\n",
            "Progress Percent = 22.014077792422672%\n",
            "Progress Percent = 22.154294848425362%\n",
            "Progress Percent = 22.294511904428056%\n",
            "Progress Percent = 22.434728960430746%\n",
            "Progress Percent = 22.57494601643344%\n",
            "Progress Percent = 22.71516307243613%\n",
            "Progress Percent = 22.855380128438824%\n",
            "Progress Percent = 22.995597184441515%\n",
            "Progress Percent = 23.13581424044421%\n",
            "Progress Percent = 23.2760312964469%\n",
            "Progress Percent = 23.416248352449593%\n",
            "Progress Percent = 23.556465408452283%\n",
            "Progress Percent = 23.696682464454977%\n",
            "Progress Percent = 23.836899520457667%\n",
            "Progress Percent = 23.97711657646036%\n",
            "Progress Percent = 24.11733363246305%\n",
            "Progress Percent = 24.257550688465745%\n",
            "Progress Percent = 24.39776774446844%\n",
            "Progress Percent = 24.53798480047113%\n",
            "Progress Percent = 24.678201856473823%\n",
            "Progress Percent = 24.818418912476513%\n",
            "Progress Percent = 24.958635968479207%\n",
            "Progress Percent = 25.098853024481897%\n",
            "Progress Percent = 25.23907008048459%\n",
            "Progress Percent = 25.37928713648728%\n",
            "Progress Percent = 25.519504192489975%\n",
            "Progress Percent = 25.659721248492666%\n",
            "Progress Percent = 25.79993830449536%\n",
            "Progress Percent = 25.94015536049805%\n",
            "Progress Percent = 26.080372416500744%\n",
            "Progress Percent = 26.220589472503434%\n",
            "Progress Percent = 26.360806528506128%\n",
            "Progress Percent = 26.50102358450882%\n",
            "Progress Percent = 26.641240640511512%\n",
            "Progress Percent = 26.781457696514202%\n",
            "Progress Percent = 26.921674752516896%\n",
            "Progress Percent = 27.06189180851959%\n",
            "Progress Percent = 27.20210886452228%\n",
            "Progress Percent = 27.342325920524974%\n",
            "Progress Percent = 27.482542976527665%\n",
            "Progress Percent = 27.62276003253036%\n",
            "Progress Percent = 27.76297708853305%\n",
            "Progress Percent = 27.903194144535743%\n",
            "Progress Percent = 28.043411200538433%\n",
            "Progress Percent = 28.183628256541127%\n",
            "Progress Percent = 28.323845312543817%\n",
            "Progress Percent = 28.46406236854651%\n",
            "Progress Percent = 28.6042794245492%\n",
            "Progress Percent = 28.744496480551895%\n",
            "Progress Percent = 28.884713536554585%\n",
            "Progress Percent = 29.02493059255728%\n",
            "Progress Percent = 29.16514764855997%\n",
            "Progress Percent = 29.305364704562663%\n",
            "Progress Percent = 29.445581760565354%\n",
            "Progress Percent = 29.585798816568047%\n",
            "Progress Percent = 29.726015872570738%\n",
            "Progress Percent = 29.86623292857343%\n",
            "Progress Percent = 30.006449984576125%\n",
            "Progress Percent = 30.146667040578816%\n",
            "Progress Percent = 30.28688409658151%\n",
            "Progress Percent = 30.4271011525842%\n",
            "Progress Percent = 30.567318208586894%\n",
            "Progress Percent = 30.707535264589584%\n",
            "Progress Percent = 30.847752320592278%\n",
            "Progress Percent = 30.987969376594968%\n",
            "Progress Percent = 31.128186432597662%\n",
            "Progress Percent = 31.268403488600352%\n",
            "Progress Percent = 31.408620544603046%\n",
            "Progress Percent = 31.548837600605736%\n",
            "Progress Percent = 31.68905465660843%\n",
            "Progress Percent = 31.82927171261112%\n",
            "Progress Percent = 31.969488768613814%\n",
            "Progress Percent = 32.109705824616505%\n",
            "Progress Percent = 32.249922880619195%\n",
            "Progress Percent = 32.39013993662189%\n",
            "Progress Percent = 32.53035699262458%\n",
            "Progress Percent = 32.67057404862727%\n",
            "Progress Percent = 32.81079110462997%\n",
            "Progress Percent = 32.95100816063266%\n",
            "Progress Percent = 33.09122521663535%\n",
            "Progress Percent = 33.23144227263804%\n",
            "Progress Percent = 33.37165932864074%\n",
            "Progress Percent = 33.51187638464343%\n",
            "Progress Percent = 33.65209344064612%\n",
            "Progress Percent = 33.79231049664881%\n",
            "Progress Percent = 33.93252755265151%\n",
            "Progress Percent = 34.0727446086542%\n",
            "Progress Percent = 34.21296166465689%\n",
            "Progress Percent = 34.35317872065958%\n",
            "Progress Percent = 34.493395776662275%\n",
            "Progress Percent = 34.633612832664966%\n",
            "Progress Percent = 34.773829888667656%\n",
            "Progress Percent = 34.914046944670346%\n",
            "Progress Percent = 35.054264000673044%\n",
            "Progress Percent = 35.194481056675734%\n",
            "Progress Percent = 35.334698112678424%\n",
            "Progress Percent = 35.47491516868112%\n",
            "Progress Percent = 35.61513222468381%\n",
            "Progress Percent = 35.7553492806865%\n",
            "Progress Percent = 35.89556633668919%\n",
            "Progress Percent = 36.03578339269189%\n",
            "Progress Percent = 36.17600044869458%\n",
            "Progress Percent = 36.31621750469727%\n",
            "Progress Percent = 36.45643456069996%\n",
            "Progress Percent = 36.59665161670266%\n",
            "Progress Percent = 36.73686867270535%\n",
            "Progress Percent = 36.87708572870804%\n",
            "Progress Percent = 37.01730278471073%\n",
            "Progress Percent = 37.15751984071343%\n",
            "Progress Percent = 37.29773689671612%\n",
            "Progress Percent = 37.43795395271881%\n",
            "Progress Percent = 37.5781710087215%\n",
            "Progress Percent = 37.718388064724195%\n",
            "Progress Percent = 37.858605120726885%\n",
            "Progress Percent = 37.998822176729576%\n",
            "Progress Percent = 38.13903923273227%\n",
            "Progress Percent = 38.27925628873496%\n",
            "Progress Percent = 38.41947334473765%\n",
            "Progress Percent = 38.559690400740344%\n",
            "Progress Percent = 38.69990745674304%\n",
            "Progress Percent = 38.84012451274573%\n",
            "Progress Percent = 38.98034156874842%\n",
            "Progress Percent = 39.12055862475111%\n",
            "Progress Percent = 39.26077568075381%\n",
            "Progress Percent = 39.4009927367565%\n",
            "Progress Percent = 39.54120979275919%\n",
            "Progress Percent = 39.68142684876188%\n",
            "Progress Percent = 39.82164390476458%\n",
            "Progress Percent = 39.96186096076727%\n",
            "Progress Percent = 40.10207801676996%\n",
            "Progress Percent = 40.24229507277265%\n",
            "Progress Percent = 40.382512128775346%\n",
            "Progress Percent = 40.522729184778036%\n",
            "Progress Percent = 40.66294624078073%\n",
            "Progress Percent = 40.803163296783424%\n",
            "Progress Percent = 40.943380352786114%\n",
            "Progress Percent = 41.083597408788805%\n",
            "Progress Percent = 41.223814464791495%\n",
            "Progress Percent = 41.36403152079419%\n",
            "Progress Percent = 41.50424857679688%\n",
            "Progress Percent = 41.64446563279957%\n",
            "Progress Percent = 41.78468268880226%\n",
            "Progress Percent = 41.92489974480496%\n",
            "Progress Percent = 42.06511680080765%\n",
            "Progress Percent = 42.20533385681034%\n",
            "Progress Percent = 42.34555091281303%\n",
            "Progress Percent = 42.48576796881573%\n",
            "Progress Percent = 42.62598502481842%\n",
            "Progress Percent = 42.76620208082111%\n",
            "Progress Percent = 42.9064191368238%\n",
            "Progress Percent = 43.0466361928265%\n",
            "Progress Percent = 43.18685324882919%\n",
            "Progress Percent = 43.32707030483188%\n",
            "Progress Percent = 43.467287360834575%\n",
            "Progress Percent = 43.607504416837266%\n",
            "Progress Percent = 43.747721472839956%\n",
            "Progress Percent = 43.887938528842646%\n",
            "Progress Percent = 44.028155584845344%\n",
            "Progress Percent = 44.168372640848034%\n",
            "Progress Percent = 44.308589696850724%\n",
            "Progress Percent = 44.448806752853415%\n",
            "Progress Percent = 44.58902380885611%\n",
            "Progress Percent = 44.7292408648588%\n",
            "Progress Percent = 44.86945792086149%\n",
            "Progress Percent = 45.00967497686418%\n",
            "Progress Percent = 45.14989203286688%\n",
            "Progress Percent = 45.29010908886957%\n",
            "Progress Percent = 45.43032614487226%\n",
            "Progress Percent = 45.57054320087495%\n",
            "Progress Percent = 45.71076025687765%\n",
            "Progress Percent = 45.85097731288034%\n",
            "Progress Percent = 45.99119436888303%\n",
            "Progress Percent = 46.13141142488573%\n",
            "Progress Percent = 46.27162848088842%\n",
            "Progress Percent = 46.41184553689111%\n",
            "Progress Percent = 46.5520625928938%\n",
            "Progress Percent = 46.692279648896495%\n",
            "Progress Percent = 46.832496704899185%\n",
            "Progress Percent = 46.972713760901875%\n",
            "Progress Percent = 47.112930816904566%\n",
            "Progress Percent = 47.25314787290726%\n",
            "Progress Percent = 47.39336492890995%\n",
            "Progress Percent = 47.533581984912644%\n",
            "Progress Percent = 47.673799040915334%\n",
            "Progress Percent = 47.81401609691803%\n",
            "Progress Percent = 47.95423315292072%\n",
            "Progress Percent = 48.09445020892341%\n",
            "Progress Percent = 48.2346672649261%\n",
            "Progress Percent = 48.3748843209288%\n",
            "Progress Percent = 48.51510137693149%\n",
            "Progress Percent = 48.65531843293418%\n",
            "Progress Percent = 48.79553548893688%\n",
            "Progress Percent = 48.93575254493957%\n",
            "Progress Percent = 49.07596960094226%\n",
            "Progress Percent = 49.21618665694495%\n",
            "Progress Percent = 49.356403712947646%\n",
            "Progress Percent = 49.496620768950336%\n",
            "Progress Percent = 49.63683782495303%\n",
            "Progress Percent = 49.77705488095572%\n",
            "Progress Percent = 49.917271936958414%\n",
            "Progress Percent = 50.057488992961105%\n",
            "Progress Percent = 50.197706048963795%\n",
            "Progress Percent = 50.337923104966485%\n",
            "Progress Percent = 50.47814016096918%\n",
            "Progress Percent = 50.61835721697187%\n",
            "Progress Percent = 50.75857427297456%\n",
            "Progress Percent = 50.898791328977254%\n",
            "Progress Percent = 51.03900838497995%\n",
            "Progress Percent = 51.17922544098264%\n",
            "Progress Percent = 51.31944249698533%\n",
            "Progress Percent = 51.45965955298803%\n",
            "Progress Percent = 51.59987660899072%\n",
            "Progress Percent = 51.74009366499341%\n",
            "Progress Percent = 51.8803107209961%\n",
            "Progress Percent = 52.0205277769988%\n",
            "Progress Percent = 52.16074483300149%\n",
            "Progress Percent = 52.30096188900418%\n",
            "Progress Percent = 52.44117894500687%\n",
            "Progress Percent = 52.581396001009566%\n",
            "Progress Percent = 52.721613057012256%\n",
            "Progress Percent = 52.861830113014946%\n",
            "Progress Percent = 53.00204716901764%\n",
            "Progress Percent = 53.142264225020334%\n",
            "Progress Percent = 53.282481281023024%\n",
            "Progress Percent = 53.422698337025714%\n",
            "Progress Percent = 53.562915393028405%\n",
            "Progress Percent = 53.7031324490311%\n",
            "Progress Percent = 53.84334950503379%\n",
            "Progress Percent = 53.98356656103648%\n",
            "Progress Percent = 54.12378361703918%\n",
            "Progress Percent = 54.26400067304187%\n",
            "Progress Percent = 54.40421772904456%\n",
            "Progress Percent = 54.54443478504725%\n",
            "Progress Percent = 54.68465184104995%\n",
            "Progress Percent = 54.82486889705264%\n",
            "Progress Percent = 54.96508595305533%\n",
            "Progress Percent = 55.10530300905802%\n",
            "Progress Percent = 55.24552006506072%\n",
            "Progress Percent = 55.38573712106341%\n",
            "Progress Percent = 55.5259541770661%\n",
            "Progress Percent = 55.66617123306879%\n",
            "Progress Percent = 55.806388289071485%\n",
            "Progress Percent = 55.946605345074175%\n",
            "Progress Percent = 56.086822401076866%\n",
            "Progress Percent = 56.227039457079556%\n",
            "Progress Percent = 56.36725651308225%\n",
            "Progress Percent = 56.507473569084944%\n",
            "Progress Percent = 56.647690625087634%\n",
            "Progress Percent = 56.78790768109033%\n",
            "Progress Percent = 56.92812473709302%\n",
            "Progress Percent = 57.06834179309571%\n",
            "Progress Percent = 57.2085588490984%\n",
            "Progress Percent = 57.3487759051011%\n",
            "Progress Percent = 57.48899296110379%\n",
            "Progress Percent = 57.62921001710648%\n",
            "Progress Percent = 57.76942707310917%\n",
            "Progress Percent = 57.90964412911187%\n",
            "Progress Percent = 58.04986118511456%\n",
            "Progress Percent = 58.19007824111725%\n",
            "Progress Percent = 58.33029529711994%\n",
            "Progress Percent = 58.470512353122636%\n",
            "Progress Percent = 58.61072940912533%\n",
            "Progress Percent = 58.75094646512802%\n",
            "Progress Percent = 58.89116352113071%\n",
            "Progress Percent = 59.031380577133405%\n",
            "Progress Percent = 59.171597633136095%\n",
            "Progress Percent = 59.311814689138785%\n",
            "Progress Percent = 59.452031745141475%\n",
            "Progress Percent = 59.59224880114417%\n",
            "Progress Percent = 59.73246585714686%\n",
            "Progress Percent = 59.87268291314955%\n",
            "Progress Percent = 60.01289996915225%\n",
            "Progress Percent = 60.15311702515494%\n",
            "Progress Percent = 60.29333408115763%\n",
            "Progress Percent = 60.43355113716032%\n",
            "Progress Percent = 60.57376819316302%\n",
            "Progress Percent = 60.71398524916571%\n",
            "Progress Percent = 60.8542023051684%\n",
            "Progress Percent = 60.99441936117109%\n",
            "Progress Percent = 61.13463641717379%\n",
            "Progress Percent = 61.27485347317648%\n",
            "Progress Percent = 61.41507052917917%\n",
            "Progress Percent = 61.55528758518186%\n",
            "Progress Percent = 61.695504641184556%\n",
            "Progress Percent = 61.835721697187246%\n",
            "Progress Percent = 61.975938753189936%\n",
            "Progress Percent = 62.11615580919263%\n",
            "Progress Percent = 62.256372865195324%\n",
            "Progress Percent = 62.396589921198014%\n",
            "Progress Percent = 62.536806977200705%\n",
            "Progress Percent = 62.6770240332034%\n",
            "Progress Percent = 62.81724108920609%\n",
            "Progress Percent = 62.95745814520878%\n",
            "Progress Percent = 63.09767520121147%\n",
            "Progress Percent = 63.23789225721417%\n",
            "Progress Percent = 63.37810931321686%\n",
            "Progress Percent = 63.51832636921955%\n",
            "Progress Percent = 63.65854342522224%\n",
            "Progress Percent = 63.79876048122494%\n",
            "Progress Percent = 63.93897753722763%\n",
            "Progress Percent = 64.07919459323033%\n",
            "Progress Percent = 64.21941164923301%\n",
            "Progress Percent = 64.3596287052357%\n",
            "Progress Percent = 64.49984576123839%\n",
            "Progress Percent = 64.64006281724109%\n",
            "Progress Percent = 64.78027987324379%\n",
            "Progress Percent = 64.92049692924647%\n",
            "Progress Percent = 65.06071398524917%\n",
            "Progress Percent = 65.20093104125186%\n",
            "Progress Percent = 65.34114809725455%\n",
            "Progress Percent = 65.48136515325724%\n",
            "Progress Percent = 65.62158220925994%\n",
            "Progress Percent = 65.76179926526262%\n",
            "Progress Percent = 65.90201632126532%\n",
            "Progress Percent = 66.042233377268%\n",
            "Progress Percent = 66.1824504332707%\n",
            "Progress Percent = 66.3226674892734%\n",
            "Progress Percent = 66.46288454527608%\n",
            "Progress Percent = 66.60310160127878%\n",
            "Progress Percent = 66.74331865728148%\n",
            "Progress Percent = 66.88353571328416%\n",
            "Progress Percent = 67.02375276928686%\n",
            "Progress Percent = 67.16396982528954%\n",
            "Progress Percent = 67.30418688129224%\n",
            "Progress Percent = 67.44440393729494%\n",
            "Progress Percent = 67.58462099329762%\n",
            "Progress Percent = 67.72483804930032%\n",
            "Progress Percent = 67.86505510530301%\n",
            "Progress Percent = 68.0052721613057%\n",
            "Progress Percent = 68.1454892173084%\n",
            "Progress Percent = 68.28570627331109%\n",
            "Progress Percent = 68.42592332931378%\n",
            "Progress Percent = 68.56614038531647%\n",
            "Progress Percent = 68.70635744131916%\n",
            "Progress Percent = 68.84657449732185%\n",
            "Progress Percent = 68.98679155332455%\n",
            "Progress Percent = 69.12700860932723%\n",
            "Progress Percent = 69.26722566532993%\n",
            "Progress Percent = 69.40744272133263%\n",
            "Progress Percent = 69.54765977733531%\n",
            "Progress Percent = 69.68787683333801%\n",
            "Progress Percent = 69.82809388934069%\n",
            "Progress Percent = 69.96831094534339%\n",
            "Progress Percent = 70.10852800134609%\n",
            "Progress Percent = 70.24874505734877%\n",
            "Progress Percent = 70.38896211335147%\n",
            "Progress Percent = 70.52917916935417%\n",
            "Progress Percent = 70.66939622535685%\n",
            "Progress Percent = 70.80961328135955%\n",
            "Progress Percent = 70.94983033736224%\n",
            "Progress Percent = 71.09004739336493%\n",
            "Progress Percent = 71.23026444936762%\n",
            "Progress Percent = 71.37048150537031%\n",
            "Progress Percent = 71.510698561373%\n",
            "Progress Percent = 71.6509156173757%\n",
            "Progress Percent = 71.79113267337839%\n",
            "Progress Percent = 71.93134972938108%\n",
            "Progress Percent = 72.07156678538378%\n",
            "Progress Percent = 72.21178384138646%\n",
            "Progress Percent = 72.35200089738916%\n",
            "Progress Percent = 72.49221795339184%\n",
            "Progress Percent = 72.63243500939454%\n",
            "Progress Percent = 72.77265206539724%\n",
            "Progress Percent = 72.91286912139992%\n",
            "Progress Percent = 73.05308617740262%\n",
            "Progress Percent = 73.19330323340532%\n",
            "Progress Percent = 73.333520289408%\n",
            "Progress Percent = 73.4737373454107%\n",
            "Progress Percent = 73.6139544014134%\n",
            "Progress Percent = 73.75417145741608%\n",
            "Progress Percent = 73.89438851341878%\n",
            "Progress Percent = 74.03460556942146%\n",
            "Progress Percent = 74.17482262542416%\n",
            "Progress Percent = 74.31503968142685%\n",
            "Progress Percent = 74.45525673742954%\n",
            "Progress Percent = 74.59547379343223%\n",
            "Progress Percent = 74.73569084943493%\n",
            "Progress Percent = 74.87590790543761%\n",
            "Progress Percent = 75.01612496144031%\n",
            "Progress Percent = 75.156342017443%\n",
            "Progress Percent = 75.29655907344569%\n",
            "Progress Percent = 75.43677612944839%\n",
            "Progress Percent = 75.57699318545107%\n",
            "Progress Percent = 75.71721024145377%\n",
            "Progress Percent = 75.85742729745647%\n",
            "Progress Percent = 75.99764435345915%\n",
            "Progress Percent = 76.13786140946185%\n",
            "Progress Percent = 76.27807846546455%\n",
            "Progress Percent = 76.41829552146723%\n",
            "Progress Percent = 76.55851257746993%\n",
            "Progress Percent = 76.69872963347261%\n",
            "Progress Percent = 76.8389466894753%\n",
            "Progress Percent = 76.979163745478%\n",
            "Progress Percent = 77.11938080148069%\n",
            "Progress Percent = 77.25959785748339%\n",
            "Progress Percent = 77.39981491348608%\n",
            "Progress Percent = 77.54003196948877%\n",
            "Progress Percent = 77.68024902549146%\n",
            "Progress Percent = 77.82046608149415%\n",
            "Progress Percent = 77.96068313749684%\n",
            "Progress Percent = 78.10090019349954%\n",
            "Progress Percent = 78.24111724950222%\n",
            "Progress Percent = 78.38133430550492%\n",
            "Progress Percent = 78.52155136150762%\n",
            "Progress Percent = 78.6617684175103%\n",
            "Progress Percent = 78.801985473513%\n",
            "Progress Percent = 78.9422025295157%\n",
            "Progress Percent = 79.08241958551838%\n",
            "Progress Percent = 79.22263664152108%\n",
            "Progress Percent = 79.36285369752376%\n",
            "Progress Percent = 79.50307075352646%\n",
            "Progress Percent = 79.64328780952916%\n",
            "Progress Percent = 79.78350486553184%\n",
            "Progress Percent = 79.92372192153454%\n",
            "Progress Percent = 80.06393897753723%\n",
            "Progress Percent = 80.20415603353992%\n",
            "Progress Percent = 80.34437308954261%\n",
            "Progress Percent = 80.4845901455453%\n",
            "Progress Percent = 80.624807201548%\n",
            "Progress Percent = 80.76502425755069%\n",
            "Progress Percent = 80.90524131355338%\n",
            "Progress Percent = 81.04545836955607%\n",
            "Progress Percent = 81.18567542555877%\n",
            "Progress Percent = 81.32589248156145%\n",
            "Progress Percent = 81.46610953756415%\n",
            "Progress Percent = 81.60632659356685%\n",
            "Progress Percent = 81.74654364956953%\n",
            "Progress Percent = 81.88676070557223%\n",
            "Progress Percent = 82.02697776157491%\n",
            "Progress Percent = 82.16719481757761%\n",
            "Progress Percent = 82.3074118735803%\n",
            "Progress Percent = 82.44762892958299%\n",
            "Progress Percent = 82.58784598558569%\n",
            "Progress Percent = 82.72806304158838%\n",
            "Progress Percent = 82.86828009759107%\n",
            "Progress Percent = 83.00849715359377%\n",
            "Progress Percent = 83.14871420959645%\n",
            "Progress Percent = 83.28893126559915%\n",
            "Progress Percent = 83.42914832160184%\n",
            "Progress Percent = 83.56936537760453%\n",
            "Progress Percent = 83.70958243360722%\n",
            "Progress Percent = 83.84979948960992%\n",
            "Progress Percent = 83.9900165456126%\n",
            "Progress Percent = 84.1302336016153%\n",
            "Progress Percent = 84.270450657618%\n",
            "Progress Percent = 84.41066771362068%\n",
            "Progress Percent = 84.55088476962338%\n",
            "Progress Percent = 84.69110182562606%\n",
            "Progress Percent = 84.83131888162876%\n",
            "Progress Percent = 84.97153593763146%\n",
            "Progress Percent = 85.11175299363414%\n",
            "Progress Percent = 85.25197004963684%\n",
            "Progress Percent = 85.39218710563954%\n",
            "Progress Percent = 85.53240416164222%\n",
            "Progress Percent = 85.67262121764492%\n",
            "Progress Percent = 85.8128382736476%\n",
            "Progress Percent = 85.9530553296503%\n",
            "Progress Percent = 86.093272385653%\n",
            "Progress Percent = 86.23348944165568%\n",
            "Progress Percent = 86.37370649765838%\n",
            "Progress Percent = 86.51392355366107%\n",
            "Progress Percent = 86.65414060966376%\n",
            "Progress Percent = 86.79435766566645%\n",
            "Progress Percent = 86.93457472166915%\n",
            "Progress Percent = 87.07479177767183%\n",
            "Progress Percent = 87.21500883367453%\n",
            "Progress Percent = 87.35522588967721%\n",
            "Progress Percent = 87.49544294567991%\n",
            "Progress Percent = 87.63566000168261%\n",
            "Progress Percent = 87.77587705768529%\n",
            "Progress Percent = 87.91609411368799%\n",
            "Progress Percent = 88.05631116969069%\n",
            "Progress Percent = 88.19652822569337%\n",
            "Progress Percent = 88.33674528169607%\n",
            "Progress Percent = 88.47696233769875%\n",
            "Progress Percent = 88.61717939370145%\n",
            "Progress Percent = 88.75739644970415%\n",
            "Progress Percent = 88.89761350570683%\n",
            "Progress Percent = 89.03783056170953%\n",
            "Progress Percent = 89.17804761771222%\n",
            "Progress Percent = 89.3182646737149%\n",
            "Progress Percent = 89.4584817297176%\n",
            "Progress Percent = 89.5986987857203%\n",
            "Progress Percent = 89.73891584172299%\n",
            "Progress Percent = 89.87913289772568%\n",
            "Progress Percent = 90.01934995372837%\n",
            "Progress Percent = 90.15956700973106%\n",
            "Progress Percent = 90.29978406573376%\n",
            "Progress Percent = 90.44000112173644%\n",
            "Progress Percent = 90.58021817773914%\n",
            "Progress Percent = 90.72043523374184%\n",
            "Progress Percent = 90.86065228974452%\n",
            "Progress Percent = 91.00086934574722%\n",
            "Progress Percent = 91.1410864017499%\n",
            "Progress Percent = 91.2813034577526%\n",
            "Progress Percent = 91.4215205137553%\n",
            "Progress Percent = 91.56173756975798%\n",
            "Progress Percent = 91.70195462576068%\n",
            "Progress Percent = 91.84217168176338%\n",
            "Progress Percent = 91.98238873776606%\n",
            "Progress Percent = 92.12260579376876%\n",
            "Progress Percent = 92.26282284977145%\n",
            "Progress Percent = 92.40303990577414%\n",
            "Progress Percent = 92.54325696177683%\n",
            "Progress Percent = 92.68347401777952%\n",
            "Progress Percent = 92.82369107378221%\n",
            "Progress Percent = 92.96390812978491%\n",
            "Progress Percent = 93.1041251857876%\n",
            "Progress Percent = 93.24434224179029%\n",
            "Progress Percent = 93.38455929779299%\n",
            "Progress Percent = 93.52477635379567%\n",
            "Progress Percent = 93.66499340979837%\n",
            "Progress Percent = 93.80521046580105%\n",
            "Progress Percent = 93.94542752180375%\n",
            "Progress Percent = 94.08564457780645%\n",
            "Progress Percent = 94.22586163380913%\n",
            "Progress Percent = 94.36607868981183%\n",
            "Progress Percent = 94.50629574581453%\n",
            "Progress Percent = 94.64651280181721%\n",
            "Progress Percent = 94.7867298578199%\n",
            "Progress Percent = 94.9269469138226%\n",
            "Progress Percent = 95.06716396982529%\n",
            "Progress Percent = 95.20738102582798%\n",
            "Progress Percent = 95.34759808183067%\n",
            "Progress Percent = 95.48781513783337%\n",
            "Progress Percent = 95.62803219383606%\n",
            "Progress Percent = 95.76824924983875%\n",
            "Progress Percent = 95.90846630584144%\n",
            "Progress Percent = 96.04868336184414%\n",
            "Progress Percent = 96.18890041784682%\n",
            "Progress Percent = 96.32911747384952%\n",
            "Progress Percent = 96.4693345298522%\n",
            "Progress Percent = 96.6095515858549%\n",
            "Progress Percent = 96.7497686418576%\n",
            "Progress Percent = 96.88998569786028%\n",
            "Progress Percent = 97.03020275386298%\n",
            "Progress Percent = 97.17041980986568%\n",
            "Progress Percent = 97.31063686586836%\n",
            "Progress Percent = 97.45085392187106%\n",
            "Progress Percent = 97.59107097787376%\n",
            "Progress Percent = 97.73128803387644%\n",
            "Progress Percent = 97.87150508987914%\n",
            "Progress Percent = 98.01172214588182%\n",
            "Progress Percent = 98.15193920188452%\n",
            "Progress Percent = 98.29215625788721%\n",
            "Progress Percent = 98.4323733138899%\n",
            "Progress Percent = 98.5725903698926%\n",
            "Progress Percent = 98.71280742589529%\n",
            "Progress Percent = 98.85302448189798%\n",
            "Progress Percent = 98.99324153790067%\n",
            "Progress Percent = 99.13345859390336%\n",
            "Progress Percent = 99.27367564990605%\n",
            "Progress Percent = 99.41389270590875%\n",
            "Progress Percent = 99.55410976191143%\n",
            "Progress Percent = 99.69432681791413%\n",
            "Progress Percent = 99.83454387391683%\n",
            "Progress Percent = 99.97476092991951%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Saving the datas"
      ],
      "metadata": {
        "id": "3wx8lPOsQt4E"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "X = np.array(X)\n",
        "y = np.array(y)\n",
        "print(X.shape, y.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ch_EQ5ruRaO3",
        "outputId": "b83f1737-6cb6-4cd6-8c3d-49cdb1b07ad1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(35659, 768) (35659,)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "with open('doc_spans_LaBSE_Embedding.npy', 'wb') as f:\n",
        "    np.save(f, X)\n",
        "with open('doc_labels.npy', 'wb') as f:\n",
        "    np.save(f, y)"
      ],
      "metadata": {
        "id": "0seaTyxMQznN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "for i in [1, 100, 1000, 2000, 5000]:\n",
        "    print(doc_sentence_train[i])\n",
        "    print(X[i])\n",
        "    print(doc_label_train[i])\n",
        "    print(y[i])\n",
        "    print('--' * 20)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MXncHa3MKFs6",
        "outputId": "a8f32be9-0b9b-4f07-ef66-9c779a8e356b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "As you plan for the future ,\n",
            "[-1.81475043e-01 -5.57618916e-01 -1.24171665e-02 -3.00217479e-01\n",
            " -5.32819152e-01 -1.57167733e-01 -2.91564651e-02  2.33065173e-01\n",
            " -3.90631616e-01  4.06808764e-01 -5.93124554e-02  6.02699742e-02\n",
            "  3.24147969e-01 -7.00683370e-02  5.46456166e-02 -1.70996666e-01\n",
            "  5.40538458e-03  3.65681857e-01 -4.62393641e-01  3.33274901e-01\n",
            " -2.09975570e-01  1.91097274e-01 -4.94019747e-01 -3.16805303e-01\n",
            " -3.73387456e-01  3.16426247e-01 -1.21236019e-01 -4.35519248e-01\n",
            " -6.76226020e-01 -3.96640718e-01 -3.65455709e-02 -6.61699653e-01\n",
            " -1.46537498e-01 -3.36690724e-01  4.28373516e-01 -5.37108481e-01\n",
            " -2.90278226e-01  4.67822343e-01 -1.73657313e-01  3.03228386e-02\n",
            " -2.60386884e-01 -5.60500085e-01  1.58685669e-01 -6.48382902e-01\n",
            " -1.71234593e-01 -9.78930667e-02 -6.56116128e-01  4.51449782e-01\n",
            "  1.58998057e-01 -2.37927601e-01  3.71033043e-01  9.97790322e-02\n",
            "  1.21435821e-01  5.63008189e-01 -5.87908983e-01  5.54587431e-02\n",
            "  5.60545504e-01 -3.33180249e-01 -1.52503267e-01  1.38059795e-01\n",
            "  6.58253789e-01 -1.30362391e-01  2.19391957e-01  7.28462636e-02\n",
            "  3.13197710e-02  3.55559647e-01 -1.78037018e-01 -1.36923283e-01\n",
            " -3.18627775e-01 -4.95219350e-01  3.70955706e-01 -1.93144381e-01\n",
            " -3.55149508e-01  1.65206417e-01 -7.43067145e-01  6.77374125e-01\n",
            "  6.23634338e-01 -3.30129862e-01  4.04664397e-01 -1.16671897e-01\n",
            " -3.81131291e-01 -7.22009838e-01  1.04686506e-01  7.34702289e-01\n",
            " -4.23349053e-01 -1.08051315e-01 -2.16924816e-01  4.90358323e-01\n",
            " -1.60445943e-01  1.97273105e-01  4.83989529e-02 -5.60294867e-01\n",
            " -4.05155838e-01  1.05406055e-02 -3.84021968e-01  4.11710083e-01\n",
            " -4.64190751e-01 -6.56595901e-02 -6.50660768e-02 -4.09220368e-01\n",
            " -4.30992931e-01  2.42517933e-01  3.53504479e-01 -2.08668396e-01\n",
            "  1.90274134e-01 -3.86169225e-01 -4.25614685e-01  3.24973501e-02\n",
            "  4.20253962e-01 -7.22803354e-01  2.95247585e-01  7.55797774e-02\n",
            "  3.77141923e-01  4.94190194e-02 -2.65881330e-01 -2.19104867e-02\n",
            " -3.25198770e-01 -3.57347965e-01  7.86269784e-01  2.39812598e-01\n",
            " -2.17116967e-01  2.06666812e-01  4.68193412e-01  2.82278061e-01\n",
            " -8.54211599e-02  3.50483298e-01 -1.03293531e-01 -7.33851075e-01\n",
            "  7.01592416e-02 -3.12450409e-01 -2.79490113e-01 -1.07377917e-01\n",
            " -3.92339587e-01 -7.10885286e-01 -2.17770249e-01  4.71653603e-02\n",
            " -4.18575197e-01 -7.47316658e-01 -4.34055120e-01 -3.29540282e-01\n",
            " -8.89382362e-02 -5.10772586e-01 -7.43681192e-02 -1.48198709e-01\n",
            "  6.77183717e-02  2.84070224e-01  8.51649493e-02  2.82976151e-01\n",
            "  8.30098316e-02  3.38911712e-01  2.77671456e-01  9.33554024e-02\n",
            "  4.72137272e-01 -3.10013711e-01  4.13715273e-01 -3.39447349e-01\n",
            "  2.89923966e-01 -1.21041358e-01 -2.57961273e-01  6.08897984e-01\n",
            "  1.77683845e-01 -6.10020459e-01 -2.46873260e-01 -3.38459224e-01\n",
            " -4.28099900e-01 -7.85164118e-01 -7.19304830e-02  9.97974537e-03\n",
            " -2.10767776e-01  6.06574956e-03  4.99666840e-01 -3.87455404e-01\n",
            " -2.47274563e-01 -2.29016110e-01  5.07459521e-01 -1.04929186e-01\n",
            " -1.44171968e-01 -5.64849854e-01 -4.25554007e-01 -7.52471685e-01\n",
            " -2.07371086e-01 -3.16362798e-01 -4.40213442e-01  8.81716460e-02\n",
            " -5.52977860e-01  1.23458400e-01  2.86475837e-01 -4.21915829e-01\n",
            " -2.05878690e-01 -2.81888455e-01 -5.13620973e-01 -1.69330165e-01\n",
            " -2.03618631e-01  3.67199808e-01 -8.35470140e-01 -1.35183990e-01\n",
            "  7.04190791e-01 -6.83690190e-01 -3.38686667e-02 -3.31371993e-01\n",
            " -3.29564005e-01 -5.99508464e-01 -1.82508186e-01 -2.40791261e-01\n",
            "  8.18899572e-02  1.89254567e-01 -4.97104824e-01  1.39799947e-02\n",
            "  4.05230671e-01 -1.30777016e-01 -7.12148070e-01  2.41939634e-01\n",
            "  4.66472149e-01 -1.02025211e-01 -1.78160563e-01  1.18689604e-01\n",
            " -9.92608726e-01  4.01292384e-01 -4.89406884e-01 -1.16935596e-01\n",
            " -6.77039400e-02 -1.11706309e-01  4.06495810e-01 -5.72206438e-01\n",
            "  5.04358649e-01 -2.69156843e-01 -4.58230168e-01 -2.03852072e-01\n",
            " -2.18486190e-01 -4.19786602e-01  4.05454874e-01 -3.95749152e-01\n",
            " -6.27237320e-01 -5.06753027e-01  1.37456268e-01  2.53840387e-01\n",
            " -6.09729290e-01 -7.93375149e-02 -5.43511748e-01  3.44642341e-01\n",
            "  3.11411798e-01  2.32943073e-01 -2.16131374e-01 -3.71661603e-01\n",
            "  2.66140878e-01  3.39539319e-01 -5.82916319e-01 -5.64791262e-01\n",
            "  4.79399532e-01  1.93517402e-01 -6.27479106e-02  8.27054560e-01\n",
            " -1.95563078e-01  2.58705646e-01 -3.08304042e-01 -1.11156516e-01\n",
            "  1.37224585e-01 -4.30098087e-01 -5.88099062e-01 -1.96655467e-01\n",
            " -4.91039932e-01 -7.15343416e-01  6.69789374e-01 -2.57517964e-01\n",
            " -5.79438210e-01 -5.30805230e-01  3.47709924e-01 -7.10675538e-01\n",
            "  2.37232268e-01 -9.78933647e-02  2.09276617e-01 -5.40263534e-01\n",
            " -4.51709181e-01  3.21260816e-03  4.11201447e-01 -3.79923195e-01\n",
            " -5.90153813e-01 -1.64674204e-02  1.01744942e-01 -1.18188307e-01\n",
            "  2.91754544e-01 -2.91541368e-01 -6.45161510e-01  3.71955752e-01\n",
            "  4.34653699e-01  5.60700774e-01 -3.89468044e-01  6.10634387e-01\n",
            " -4.03703272e-01  2.15634421e-01  2.44515315e-01  1.87726960e-01\n",
            "  2.30851024e-01 -6.46064639e-01  3.26783836e-01  4.24258828e-01\n",
            "  4.02992666e-02 -1.82895154e-01 -2.00549051e-01 -2.07108378e-01\n",
            "  4.06773061e-01 -5.08041501e-01 -6.90307319e-01  6.98832273e-01\n",
            " -5.75190127e-01  3.74099612e-01 -3.87209117e-01  4.34474766e-01\n",
            " -7.36410394e-02 -1.03679977e-01 -3.79165173e-01  3.83677006e-01\n",
            " -1.03775430e-02 -8.06300581e-01  2.41424516e-01  4.74354804e-01\n",
            " -1.85132116e-01  5.28625131e-01 -5.45246780e-01 -1.10647336e-01\n",
            "  2.94374794e-01  5.96759140e-01  5.70028543e-01  5.70647657e-01\n",
            " -6.79794192e-01  1.52616471e-01 -3.82556617e-01 -8.00353587e-01\n",
            " -2.12504804e-01 -2.76566327e-01 -4.76484224e-02  5.98746762e-02\n",
            "  3.09604853e-02  7.27133989e-01 -2.83689760e-02  6.65325880e-01\n",
            " -3.42091769e-01 -2.47754499e-01 -5.17828107e-01 -6.78302288e-01\n",
            " -3.77784371e-01 -5.03647804e-01  1.85224921e-01 -5.67449741e-02\n",
            " -5.20417392e-01 -5.59971452e-01 -1.54448986e-01 -2.12672517e-01\n",
            " -1.32848784e-01 -2.19294518e-01 -5.16213365e-02 -8.72418046e-01\n",
            "  4.39359754e-01 -6.41110599e-01  3.40585977e-01  4.42358792e-01\n",
            "  3.40656906e-01  5.92119172e-02  1.15923561e-01 -1.42256945e-01\n",
            " -3.56969655e-01 -1.08181881e-02 -1.36980310e-01  2.20885530e-01\n",
            "  6.28185868e-02 -1.72216564e-01 -4.40300107e-01  1.29564628e-01\n",
            " -2.59727746e-01 -5.63183665e-01 -9.49348956e-02 -5.20560503e-01\n",
            " -2.48269409e-01 -3.93103212e-01 -5.37044644e-01 -7.90423453e-01\n",
            "  5.25190756e-02 -3.00081484e-02  1.47783503e-01  2.80510008e-01\n",
            " -3.02746266e-01 -1.68118328e-01  3.13737184e-01 -1.06679171e-01\n",
            "  6.52152121e-01  1.77707970e-01 -7.23489285e-01  4.95795548e-01\n",
            "  4.61121500e-01 -3.93497556e-01 -1.17015198e-01 -5.40452711e-02\n",
            "  2.89929032e-01 -2.49420062e-01 -3.81629094e-02  1.24781996e-01\n",
            " -2.22125307e-01  3.39390635e-01 -5.11565387e-01 -7.53385007e-01\n",
            "  2.33144581e-01 -2.44581506e-01  3.34978342e-01  4.75566387e-01\n",
            "  5.62307596e-01  4.91070867e-01 -4.66410458e-01 -3.24609309e-01\n",
            " -1.28242401e-02  6.27301216e-01  1.26920447e-01 -3.99216354e-01\n",
            " -7.58050263e-01  4.00198475e-02  1.14395462e-01  3.29229422e-02\n",
            " -6.82528168e-02  4.41858560e-01 -8.33478421e-02 -3.89041632e-01\n",
            " -6.59578800e-01  8.40559229e-02 -2.83373386e-01  2.50604302e-01\n",
            "  1.28788725e-02  3.76003355e-01 -3.55762005e-01 -2.88067758e-01\n",
            " -1.27566248e-01 -5.14067113e-01  5.02226427e-02 -2.06471071e-01\n",
            " -2.40015145e-02 -3.05574741e-02  4.14894015e-01 -4.54231590e-01\n",
            " -2.76933491e-01  3.61731976e-01  5.81546724e-01  4.86002982e-01\n",
            " -4.61169183e-01  4.17624682e-01 -3.39847542e-02 -1.24124043e-01\n",
            " -6.45173416e-02 -1.81071505e-01 -2.90435910e-01 -5.84463179e-01\n",
            " -2.45269597e-01 -4.87283617e-02  1.19497571e-02  5.69969773e-01\n",
            " -2.40463689e-01  2.26030536e-02 -1.22690186e-01 -1.20795883e-01\n",
            " -1.49783254e-01  5.48417985e-01 -3.97940755e-01  4.81955260e-01\n",
            " -5.09704947e-01  3.57967556e-01  1.64422661e-01  4.10304219e-01\n",
            " -2.35598162e-01 -3.55319172e-01 -5.79835415e-01 -9.33036149e-01\n",
            " -3.59324753e-01 -2.71248579e-01  1.79242551e-01  4.78274226e-01\n",
            " -7.93118536e-01  2.32665524e-01 -5.03462315e-01 -1.90308481e-01\n",
            " -4.19252276e-01 -3.39723676e-01  1.33517504e-01  5.10889649e-01\n",
            "  2.31566966e-01 -1.26499295e-01 -9.25882999e-03 -5.71456313e-01\n",
            " -6.24685585e-01  4.10031110e-01  2.78829902e-01  5.32636881e-01\n",
            " -8.95930603e-02 -7.98034891e-02  6.00425750e-02 -4.68481660e-01\n",
            "  1.25926599e-01 -1.38712218e-02 -2.61674911e-01  4.22123551e-01\n",
            " -7.35770404e-01 -3.83948088e-01  1.80598617e-01 -2.35241875e-01\n",
            " -1.57096162e-01 -5.91236174e-01  9.71524119e-02  6.87421381e-01\n",
            "  1.68656632e-01  8.22200105e-02 -8.31416622e-02  4.64145482e-01\n",
            " -4.56843138e-01  5.70236683e-01 -8.18167254e-02 -4.87401694e-01\n",
            "  6.59678638e-01  9.82213318e-02 -2.94419497e-01  1.47892982e-01\n",
            " -6.89564526e-01  3.58429462e-01 -9.05355066e-02 -6.90173805e-01\n",
            "  3.53261769e-01  1.18727215e-01  5.44655919e-01 -7.73166344e-02\n",
            " -3.91053915e-01  5.77813387e-01 -2.85893440e-01 -1.12309970e-01\n",
            " -1.96730584e-01 -3.51945497e-03 -5.46186924e-01  1.64177328e-01\n",
            "  6.10579789e-01 -6.13989234e-01  3.41728002e-01  3.70015763e-02\n",
            " -2.51896977e-01 -1.95373133e-01  2.33761162e-01  2.10076123e-02\n",
            "  2.23639369e-01 -4.93957192e-01  6.94727063e-01  7.69512877e-02\n",
            " -5.25155306e-01 -1.98295981e-01 -7.09062934e-01 -4.72022533e-01\n",
            " -2.91647077e-01 -4.07484055e-01 -1.59443036e-01 -6.39414787e-01\n",
            "  4.40510184e-01  4.39646274e-01  5.31897306e-01 -5.26498377e-01\n",
            " -6.16177380e-01 -6.10223234e-01  5.49510896e-01 -7.47664630e-01\n",
            " -7.19825506e-01 -5.52657545e-01  6.12260401e-01  2.40245938e-01\n",
            " -3.71735126e-01 -8.42867047e-02 -4.45362538e-01  3.06943059e-01\n",
            "  4.50487465e-01 -1.01971760e-01 -2.14923248e-01  4.93447095e-01\n",
            " -1.01648577e-01 -5.68686783e-01 -6.56320095e-01 -2.61314362e-01\n",
            "  1.02241643e-01 -3.64076614e-01 -3.09262335e-01 -2.69034147e-01\n",
            "  1.34783939e-01  3.01044524e-01 -2.70648450e-01 -3.33575130e-01\n",
            "  2.39456296e-01  5.47984615e-02  3.13180029e-01  1.49757981e-01\n",
            " -1.89989939e-01 -3.03946203e-04 -7.79817849e-02 -6.90134227e-01\n",
            " -3.99435014e-01 -2.13203147e-01  4.54636842e-01  3.49264979e-01\n",
            " -5.44073164e-01 -1.16845727e-01 -5.83941862e-02 -9.14384723e-01\n",
            " -1.88286662e-01  6.48563743e-01 -3.42268050e-01 -2.52760142e-01\n",
            "  5.55510759e-01 -1.29660472e-01  7.17803240e-01 -4.72158819e-01\n",
            " -3.49694252e-01  3.69453073e-01 -3.91869903e-01 -3.85986835e-01\n",
            "  2.08134755e-01 -1.14011250e-01 -2.31639937e-01 -4.09345269e-01\n",
            " -6.42414093e-01  3.06468099e-01 -1.71408594e-01 -6.29814267e-02\n",
            "  3.56952965e-01 -5.65191925e-01 -3.54115292e-02 -3.04135174e-01\n",
            " -1.31130904e-01 -1.19236916e-01 -3.53473872e-01 -8.80811155e-01\n",
            "  4.97392416e-01  1.25021279e-01 -7.87419975e-02 -6.00226939e-01\n",
            "  3.74395609e-01  2.02767000e-01 -3.48768644e-02 -1.94572479e-01\n",
            "  2.38471538e-01  5.04039191e-02 -2.92341948e-01  5.06469011e-01\n",
            " -4.75123405e-01  5.01765788e-01 -4.96670514e-01 -3.93116832e-01\n",
            "  3.35084766e-01  2.36519054e-01 -4.59560692e-01  1.75853580e-01\n",
            " -8.60672295e-02 -1.92630693e-01 -7.69694969e-02 -1.71383962e-01\n",
            "  3.57984662e-01 -1.69320613e-01 -7.84988105e-02 -5.93200862e-01\n",
            " -3.74906629e-01 -7.81585202e-02 -4.45901424e-01  3.65247577e-01\n",
            "  2.31287211e-01 -2.85133272e-01 -7.57034421e-01 -5.69234848e-01\n",
            "  6.46422524e-03  5.22853397e-02  3.25843424e-01  1.22131355e-01\n",
            "  5.17811358e-01  7.52511084e-01  1.26546072e-02 -2.78374672e-01\n",
            " -5.20271156e-03 -6.95926324e-02  4.79572207e-01 -3.61233413e-01\n",
            " -1.86897948e-01  4.86437231e-02 -2.71297753e-01  8.03687334e-01\n",
            " -2.76130021e-01 -8.00028622e-01 -6.71946406e-01 -6.89329982e-01\n",
            " -4.67561990e-01  2.97801644e-01  4.86536264e-01  1.34876430e-01\n",
            " -9.74655747e-02 -2.46239066e-01 -3.70205939e-01 -6.14783093e-02\n",
            " -7.11252987e-01 -1.81387410e-01 -2.54566997e-01  4.84002292e-01\n",
            "  9.92999822e-02  5.31102538e-01  3.09917718e-01 -5.85489869e-01\n",
            " -2.50267625e-01 -6.65278554e-01  2.54485160e-01 -8.47254992e-02\n",
            "  7.23781884e-02 -1.03524946e-01 -1.14140436e-02  7.39116445e-02\n",
            " -1.08716145e-01 -6.64819777e-01  1.08863764e-01 -6.65600761e-04\n",
            "  1.38673723e-01  1.88759595e-01  7.70889223e-02 -2.85474420e-01\n",
            "  2.51146585e-01 -3.33593115e-02 -5.30917704e-01  3.80696088e-01\n",
            " -1.92329228e-01 -4.25813720e-02  5.91957346e-02 -2.36436367e-01\n",
            "  1.17114797e-01  3.58669192e-01 -3.66473258e-01 -6.07244074e-02\n",
            " -5.75045466e-01  5.44320978e-02  6.99524164e-01  2.19294369e-01\n",
            " -2.44673759e-01  3.32576394e-01  4.93692636e-01 -2.56442148e-02\n",
            " -1.21748947e-01 -1.09792612e-01  1.45945370e-01  1.95749506e-01\n",
            " -1.02372356e-01 -6.56593502e-01 -1.45578459e-01 -4.73522872e-01\n",
            "  3.39854538e-01 -5.27745843e-01  2.02548862e-01 -1.65161669e-01\n",
            " -3.87408398e-02 -5.51155433e-02 -2.36063436e-01 -4.52104181e-01\n",
            "  4.28917170e-01  6.13862634e-01  1.42324477e-01  2.65192300e-01\n",
            "  4.97824490e-01 -2.68486410e-01  1.36520609e-01 -1.60474464e-01\n",
            " -5.85880995e-01  1.72073752e-01 -5.52635081e-02 -5.66955566e-01\n",
            "  3.36075246e-01 -3.66730481e-01 -8.81717563e-01  1.55317053e-01]\n",
            "Benefits Planner: Survivors | Planning For Your Survivors | Social Security Administration#1_0\n",
            "203\n",
            "----------------------------------------\n",
            "you'll want to think about what your family would need if you should die now.\n",
            "[ 2.32434243e-01 -3.96193236e-01  4.92897332e-01 -5.13495922e-01\n",
            "  5.87243140e-01 -8.42683911e-01 -7.21537292e-01 -2.34731555e-01\n",
            "  5.30406475e-01 -6.60753548e-01  7.42822826e-01  4.06079739e-01\n",
            "  8.79054844e-01 -1.07301265e-01 -1.77682042e-01 -1.17090084e-01\n",
            " -1.78512618e-01  2.62760192e-01  7.32529938e-01  7.16462553e-01\n",
            "  7.20109463e-01  5.15195608e-01 -4.53463942e-01 -7.61848390e-01\n",
            " -3.70367557e-01  3.55262935e-01  3.62239450e-01 -3.96118492e-01\n",
            " -7.34290481e-01 -1.79524766e-04 -9.23962533e-01 -8.56978059e-01\n",
            " -4.83666033e-01 -7.96483338e-01 -5.45110822e-01  1.56415198e-02\n",
            "  6.92925900e-02  3.98614556e-01 -8.35062444e-01 -3.96449447e-01\n",
            "  1.74981341e-01 -4.49590534e-01  2.21943650e-02 -1.83040306e-01\n",
            " -3.17930162e-01  2.70596385e-01 -5.95804036e-01  3.78638506e-02\n",
            " -6.15084708e-01 -6.17776275e-01 -3.65824312e-01  4.68281299e-01\n",
            " -5.89323819e-01  2.30121583e-01 -2.62707472e-01 -7.53508508e-01\n",
            " -4.61376667e-01 -3.61632943e-01  2.03076974e-01 -1.20553158e-01\n",
            " -5.85274160e-01 -3.40530127e-01 -6.38391912e-01 -5.37310362e-01\n",
            "  4.80491370e-02  3.61992568e-01 -3.14144582e-01 -1.77630395e-01\n",
            " -8.18296611e-01 -2.85321325e-01  7.39629745e-01 -4.11388606e-01\n",
            " -1.76467970e-01 -4.94776815e-01 -1.45595253e-01  4.68876272e-01\n",
            "  6.95208013e-01 -8.66351485e-01 -4.76597518e-01 -4.72384244e-01\n",
            " -3.38819593e-01 -5.36072910e-01 -1.45099252e-01  1.30560219e-01\n",
            "  2.51292754e-02 -5.85295260e-01 -1.36761814e-01  3.74692529e-02\n",
            " -4.95146692e-01  2.54585236e-01 -1.31762758e-01 -3.07305962e-01\n",
            " -6.72541797e-01  5.92578530e-01 -8.45573187e-01  7.05857813e-01\n",
            " -8.41778100e-01  5.92604935e-01 -4.26684797e-01 -5.05314469e-01\n",
            " -6.41149282e-01  3.02386075e-01  8.06315064e-01 -4.56355035e-01\n",
            " -6.90034747e-01 -3.72089028e-01 -3.53975981e-01  4.03171986e-01\n",
            "  7.54443347e-01 -8.55808973e-01  5.71941316e-01 -4.33783174e-01\n",
            "  3.70807618e-01 -5.35030961e-01  1.35925608e-02  5.01552582e-01\n",
            " -9.22847390e-01  1.88493833e-01  3.73016000e-01  3.72643024e-01\n",
            " -7.30227828e-01  9.42468047e-02 -3.69896233e-01 -3.26605469e-01\n",
            " -2.68926322e-01  2.22058475e-01 -4.06648964e-01 -7.95540929e-01\n",
            " -2.79891156e-02 -2.36210674e-01 -7.19141901e-01 -9.90654975e-02\n",
            " -7.90243745e-01 -3.93199652e-01 -1.49882272e-01 -2.16840044e-01\n",
            " -7.01163352e-01 -4.24711347e-01 -4.27849501e-01 -9.04721260e-01\n",
            "  3.07558447e-01 -7.50693142e-01  6.26682460e-01 -2.07297623e-01\n",
            " -7.67930523e-02  2.97590613e-01  7.36954868e-01  6.62221372e-01\n",
            "  1.73033521e-01  5.00892282e-01 -3.92592192e-01 -2.90818512e-01\n",
            " -8.59384239e-01 -6.49708986e-01 -6.97179317e-01 -3.90074849e-01\n",
            " -9.08907354e-01  1.18578330e-01  6.02015078e-01  3.84527534e-01\n",
            "  3.33273679e-01 -1.63255677e-01 -6.58646464e-01  2.32748225e-01\n",
            " -3.66942167e-01 -7.72247612e-01 -9.62028801e-01  4.23173085e-02\n",
            " -3.94493431e-01 -5.95121920e-01  2.83747055e-02 -5.78607142e-01\n",
            "  1.45681560e-01 -4.04638052e-01  4.94303912e-01 -6.74716473e-01\n",
            " -2.15469584e-01 -4.47580278e-01  2.80545801e-01 -8.89137149e-01\n",
            " -3.03997904e-01 -7.19025791e-01  7.05872253e-02 -3.34357649e-01\n",
            " -5.99075973e-01 -2.74619639e-01 -4.95828949e-02  7.16170490e-01\n",
            "  6.52788222e-01  4.87645924e-01  2.32004538e-01  1.85081497e-01\n",
            "  8.84353593e-02 -4.36755180e-01 -5.08243620e-01 -5.05677611e-02\n",
            "  4.39561665e-01 -5.05162954e-01 -4.77111638e-01 -1.13009915e-01\n",
            "  3.34523842e-02  2.43261233e-01  4.11136717e-01 -5.32611310e-01\n",
            " -1.43082082e-01 -3.56026769e-01 -6.37030840e-01 -9.02979672e-01\n",
            "  3.33473116e-01 -8.24954689e-01 -7.35382736e-01 -5.82524955e-01\n",
            "  1.59475848e-01 -5.37992358e-01 -1.22935986e-02  3.06928337e-01\n",
            " -9.98812258e-01  9.26020816e-02 -2.61419982e-01 -2.23382711e-02\n",
            "  6.31439686e-01 -2.14682400e-01 -8.33391726e-01 -9.23706472e-01\n",
            "  3.98593515e-01 -4.05520827e-01 -6.51916087e-01 -1.34831280e-01\n",
            "  1.05920598e-01 -2.82289803e-01 -4.33518320e-01 -8.34951043e-01\n",
            " -3.46004575e-01  4.41285491e-01  5.58288172e-02 -5.97875059e-01\n",
            " -5.34860909e-01 -5.03956020e-01 -3.13089103e-01 -5.90292513e-01\n",
            " -3.12452167e-01 -7.26504862e-01 -1.51084617e-01  3.00085526e-02\n",
            " -2.11866528e-01 -4.96641546e-01 -1.51474923e-01 -4.96987179e-02\n",
            " -8.00227821e-02 -1.29589260e-01 -6.80485368e-01  7.04243422e-01\n",
            " -5.97606897e-01 -6.65786147e-01 -5.95494211e-01 -7.95049429e-01\n",
            "  2.70452619e-01 -7.81530261e-01 -5.23243070e-01 -3.93437713e-01\n",
            " -8.06137145e-01  4.87747133e-01 -2.88295239e-01 -8.46877038e-01\n",
            " -4.14448321e-01 -3.15647572e-02 -6.64974868e-01  1.04284488e-01\n",
            " -5.66664115e-02 -1.48959115e-01 -3.75815183e-02 -6.62575781e-01\n",
            " -4.50354338e-01 -6.92449749e-01  5.94882727e-01 -7.58056343e-01\n",
            " -5.99991493e-02 -6.27312660e-01  1.07527062e-01 -3.41205984e-01\n",
            " -2.32510671e-01  3.93550903e-01 -5.00006020e-01 -6.52899504e-01\n",
            " -7.77322292e-01 -1.50246367e-01  2.17701033e-01 -1.57688692e-01\n",
            "  2.08667889e-01 -6.14556491e-01 -3.63086253e-01  1.93666071e-01\n",
            "  9.51988548e-02 -6.89315915e-01  1.66147098e-01  3.23884785e-01\n",
            "  3.10156643e-01 -5.85708097e-02 -7.38489106e-02  1.36136651e-01\n",
            " -3.01457077e-01 -6.80069447e-01  4.38275635e-01 -2.48648360e-01\n",
            " -6.71383917e-01 -6.18189991e-01  1.77533180e-02  6.92283139e-02\n",
            " -5.56803346e-01 -1.32385716e-01 -3.17875326e-01 -5.04498005e-01\n",
            "  2.66064227e-01 -6.15953743e-01 -4.69637960e-01 -4.09440696e-01\n",
            "  6.01718910e-02 -7.37672210e-01 -1.80885747e-01  1.53938076e-02\n",
            "  6.85491323e-01 -2.26865739e-01 -4.27515149e-01 -8.65460396e-01\n",
            " -2.81011403e-01 -5.80785573e-01  2.38297090e-01 -9.00286794e-01\n",
            " -7.88763523e-01  5.12848273e-02 -3.74928951e-01  9.70809683e-02\n",
            " -5.08043706e-01  5.22550881e-01 -6.55495167e-01 -5.47431588e-01\n",
            "  3.66057545e-01 -5.49737215e-01 -6.94816649e-01 -2.00135201e-01\n",
            "  1.96755067e-01 -7.85923064e-01 -8.72221708e-01 -3.37507397e-01\n",
            " -8.22764456e-01 -7.47256398e-01 -6.12194717e-01  2.85281867e-01\n",
            "  7.78338313e-02  5.12932897e-01 -4.40875953e-03 -8.36180627e-01\n",
            " -2.42716316e-02 -7.47700214e-01 -3.69780540e-01  2.68973298e-02\n",
            " -2.52680987e-01  3.65706921e-01 -1.36010945e-01  5.32983206e-02\n",
            " -8.45455468e-01 -4.04640257e-01 -5.05710542e-01  6.64305270e-01\n",
            " -6.78112656e-02 -1.08373389e-01 -1.05393603e-01 -2.71761000e-01\n",
            " -2.56516188e-01 -2.19257101e-02  2.77537107e-01 -6.82194412e-01\n",
            " -5.71652353e-01 -4.15983098e-03 -8.42218935e-01 -4.43121463e-01\n",
            " -4.53026503e-01 -1.35492787e-01 -7.10221946e-01 -9.71200243e-02\n",
            " -2.49769539e-01  5.51214814e-01  5.19820333e-01  7.59121537e-01\n",
            " -1.06834762e-01 -4.37500000e-01 -8.58209133e-01 -5.83096862e-01\n",
            " -7.96181858e-01 -8.68096709e-01 -2.12548211e-01 -7.02683687e-01\n",
            "  8.09093893e-01 -3.20503533e-01  6.44499898e-01 -8.13595474e-01\n",
            "  3.88865680e-01 -6.52493000e-01 -8.84678423e-01 -5.93450367e-01\n",
            "  2.82010436e-01 -2.82142818e-01 -8.73611987e-01 -3.31055105e-01\n",
            "  2.79788852e-01 -1.97211713e-01 -6.17807673e-04 -1.79986805e-01\n",
            " -4.59882945e-01  1.27086148e-01 -2.52501741e-02 -7.46910274e-01\n",
            " -3.98899764e-01 -9.85132992e-01  6.51887238e-01 -1.91073999e-01\n",
            "  4.70779315e-02 -3.45354915e-01  3.41820419e-02 -9.35530066e-01\n",
            " -4.40883398e-01 -9.56641495e-01 -3.20317775e-01 -2.84252524e-01\n",
            " -6.73280358e-01  7.76126862e-01  4.45157588e-02 -8.02879989e-01\n",
            " -7.35331476e-01 -1.23447672e-01  2.66723335e-01  3.47589135e-01\n",
            " -8.94383252e-01  8.42329264e-01  4.83617723e-01  1.87908530e-01\n",
            " -9.66403484e-01 -2.12068990e-01 -2.03519270e-01 -4.66661364e-01\n",
            "  9.76146534e-02 -1.16310373e-01 -2.01791391e-01  3.71073544e-01\n",
            "  1.30446240e-01 -1.42264709e-01  2.88433194e-01 -4.44362164e-01\n",
            "  3.83109540e-01  5.61090559e-02 -4.59978431e-02 -1.91966847e-01\n",
            " -8.34990621e-01 -6.86660707e-01 -8.30600202e-01  3.48409802e-01\n",
            "  9.09683034e-02  1.00323342e-01  3.54893923e-01  5.44777401e-02\n",
            " -1.80464357e-01  3.78505290e-01 -9.50615287e-01 -2.00798037e-04\n",
            "  7.02700257e-01 -1.33100435e-01  6.00482106e-01 -3.48364174e-01\n",
            " -8.34009230e-01 -2.59736836e-01 -4.58196133e-01 -6.53575480e-01\n",
            " -4.44166720e-01  4.89161611e-01  6.04488373e-01  2.65389651e-01\n",
            "  2.61876106e-01 -4.95117545e-01  4.40616980e-02 -2.16985285e-01\n",
            " -4.75105613e-01  6.46451771e-01 -8.33298266e-01 -1.65191486e-01\n",
            "  4.85276133e-01  6.26262307e-01  7.09772885e-01  3.52087826e-01\n",
            " -7.02492297e-02  1.37871921e-01 -3.20374966e-01 -1.36272818e-01\n",
            " -6.73690796e-01  4.73577887e-01  3.97620887e-01 -3.43801469e-01\n",
            " -9.48858917e-01  4.58503425e-01 -1.09641992e-01 -1.27445579e-01\n",
            "  4.80667382e-01 -2.41399616e-01 -7.38514662e-02  6.18611038e-01\n",
            " -7.42212117e-01  5.01409054e-01 -6.40399814e-01 -4.23662305e-01\n",
            " -1.09153800e-01 -9.16707590e-02  1.26073554e-01 -4.89251018e-01\n",
            " -3.07868630e-01  3.72222692e-01 -2.43499979e-01 -6.31975889e-01\n",
            " -6.50559247e-01 -7.39183962e-01 -1.12126768e-01 -7.53808618e-01\n",
            " -7.47615576e-01  1.13921791e-01 -3.65747482e-01 -2.28590876e-01\n",
            "  2.77876049e-01  5.49399793e-01 -7.77764022e-01  2.62822121e-01\n",
            " -6.32912338e-01 -3.76985550e-01  7.09757268e-01 -3.62883568e-01\n",
            "  5.15149593e-01  4.78868306e-01  5.97403049e-01 -7.90877104e-01\n",
            " -5.91517270e-01 -6.32385314e-01  2.10290059e-01 -1.31678283e-01\n",
            " -8.15667927e-01  3.80598456e-01 -3.52694720e-01  2.51255482e-01\n",
            "  1.08184360e-01  1.55810490e-01 -8.12517345e-01 -5.70807815e-01\n",
            "  7.38270462e-01 -2.52905279e-01 -2.89922535e-01  2.12479219e-01\n",
            "  1.47722781e-01 -5.16251177e-02  1.39716759e-01 -3.89128029e-02\n",
            "  5.32030463e-01 -2.87590235e-01 -6.01857364e-01 -6.36042714e-01\n",
            "  2.13316754e-01 -6.29203737e-01  7.48397231e-01 -3.32406402e-01\n",
            "  6.96257591e-01 -8.71927857e-01 -9.70406011e-02  3.58570635e-01\n",
            "  7.85942972e-01  5.80057502e-01 -6.80084825e-01  6.07246101e-01\n",
            " -2.04999492e-01 -5.28312624e-01  2.93020695e-01 -6.28059983e-01\n",
            " -3.60926718e-01 -8.06223750e-01 -6.03677511e-01 -1.70277819e-01\n",
            " -6.90538824e-01 -5.62601626e-01  6.00186110e-01 -2.94992030e-01\n",
            " -2.35383317e-01 -4.77885455e-01 -7.13823378e-01  8.27072859e-01\n",
            "  3.05150926e-01 -3.43923837e-01  3.50774020e-01  1.51976675e-01\n",
            "  1.90112546e-01 -7.14987397e-01 -6.37001634e-01 -3.18360925e-01\n",
            " -6.30029023e-01  8.65548104e-03 -2.54963618e-02 -3.26150149e-01\n",
            "  3.68518978e-02 -6.52870893e-01 -4.19402003e-01 -8.38126063e-01\n",
            " -7.03013182e-01  1.78540871e-01  1.76513582e-01 -1.99535251e-01\n",
            "  3.22298557e-01  3.07396203e-01  5.02963483e-01 -5.83998799e-01\n",
            " -6.32493019e-01  3.89287323e-02  1.37934700e-01  4.43665832e-02\n",
            " -2.39319980e-01 -6.58059269e-02  6.67835891e-01 -7.64520705e-01\n",
            " -2.44910195e-02 -7.70784676e-01 -7.02027261e-01 -1.07751526e-01\n",
            " -2.95832574e-01 -8.69532287e-01 -5.04761159e-01 -7.50740588e-01\n",
            " -2.47048974e-01 -7.31924295e-01  8.35218966e-01 -4.03806239e-01\n",
            "  7.24201083e-01 -6.57461405e-01 -3.47407013e-01  3.52554768e-01\n",
            "  3.51351589e-01 -5.71154773e-01 -2.56105691e-01  1.80747926e-01\n",
            " -8.79024267e-01 -4.23049964e-02 -9.09228206e-01 -3.09644863e-02\n",
            "  5.01949608e-01  3.40705633e-01 -4.75081980e-01 -9.45008174e-02\n",
            "  1.92377359e-01 -7.52230763e-01 -2.30289355e-01  4.67255265e-01\n",
            " -5.52091658e-01  1.58703983e-01  2.26298720e-01 -4.52258348e-01\n",
            "  3.01362574e-01 -6.69717908e-01  3.82873833e-01  6.66890740e-02\n",
            "  3.41188490e-01  9.25010741e-02 -4.03243840e-01 -3.93607289e-01\n",
            " -6.44311588e-03 -1.93590388e-01  3.72118771e-01  7.19862700e-01\n",
            " -1.46314343e-02 -9.43550393e-02 -3.56405497e-01  6.72598541e-01\n",
            " -4.01740298e-02  6.31997511e-02  4.44850653e-01 -2.80328304e-01\n",
            " -4.98457178e-02  2.99801230e-01 -8.61427188e-02 -6.58422172e-01\n",
            " -7.45204687e-01 -7.51730800e-01 -8.08279037e-01  3.91895980e-01\n",
            " -1.88984066e-01 -6.33556366e-01  2.43742228e-01  5.50085664e-01\n",
            "  4.25684690e-01 -2.44369775e-01  2.94815954e-02 -7.99247384e-01\n",
            "  7.34733284e-01  5.91248035e-01 -1.75031990e-01  3.97833467e-01\n",
            "  1.44392466e-02 -1.26593381e-01  1.75796047e-01 -1.11326098e-01\n",
            " -7.27042139e-01  1.46802798e-01  4.37912226e-01  8.21124315e-02\n",
            " -3.07631433e-01 -6.24846458e-01 -1.56470358e-01  3.16425145e-01\n",
            " -8.68847221e-02 -8.91554236e-01 -6.24190748e-01 -2.77281314e-01\n",
            "  2.04831362e-01  8.20889175e-01  1.24395974e-01  3.01943064e-01\n",
            "  5.55684030e-01  3.07814807e-01 -6.27993584e-01  4.34510931e-02\n",
            " -4.46706384e-01  7.50535011e-01 -3.88637185e-01 -6.60495937e-01\n",
            " -2.42346495e-01 -6.46513939e-01 -4.64933515e-01  1.41303807e-01\n",
            "  1.11341693e-01 -1.00352593e-01 -3.79787147e-01  4.28192288e-01\n",
            " -1.30891636e-01 -2.50139832e-01  2.85219252e-01 -2.82915443e-01\n",
            " -4.07351196e-01 -1.84776336e-01 -7.41450727e-01 -8.61841857e-01\n",
            " -8.00340772e-01 -3.99420783e-02  2.16875553e-01  2.75890201e-01\n",
            "  5.60940266e-01 -8.41073334e-01 -5.08013368e-01 -7.61213303e-01\n",
            " -5.24091303e-01  1.88705400e-01  1.77128077e-01  1.83113843e-01\n",
            "  1.62683636e-01 -1.41589805e-01 -2.89844275e-01  1.42486006e-01\n",
            " -3.69148582e-01 -4.49272335e-01 -6.94203794e-01  5.18546641e-01\n",
            " -7.12970853e-01 -2.92656541e-01  4.12816256e-01  5.87653100e-01\n",
            " -9.45812538e-02 -8.52961719e-01 -4.59502965e-01  5.46605997e-02]\n",
            "Benefits Planner: Survivors | Planning For Your Survivors | Social Security Administration#2_0\n",
            "274\n",
            "----------------------------------------\n",
            "Religious record made before the age of 5 showing your date of birth ;\n",
            "[ 5.40287375e-01 -2.30133444e-01 -6.81414306e-01 -5.67984283e-01\n",
            " -7.18027174e-01 -5.22438705e-01 -2.54649967e-01  1.01073468e-02\n",
            "  2.64239043e-01 -7.19891071e-01 -5.66601992e-01  1.53849006e-01\n",
            "  1.66281536e-01 -4.11484778e-01  7.08309934e-02  6.12601519e-01\n",
            " -2.31365293e-01 -1.26014441e-01 -7.14766026e-01 -8.72139558e-02\n",
            "  3.29427570e-01  7.90015697e-01  5.13166189e-01 -4.39346910e-01\n",
            "  4.83894587e-01 -7.36472309e-01  2.89927632e-01 -1.63285345e-01\n",
            " -6.47755444e-01 -8.00051212e-01  2.94747174e-01 -8.70757878e-01\n",
            "  2.30858371e-01 -3.11618835e-01 -2.91334152e-01 -5.48265934e-01\n",
            "  2.24791586e-01  1.55342191e-01 -7.65033901e-01 -6.02453113e-01\n",
            " -2.49779046e-01 -1.70571953e-01  3.46280187e-01 -7.40593791e-01\n",
            "  4.02961135e-01 -6.66878939e-01  8.54115933e-05  1.96967959e-01\n",
            " -2.35300124e-01 -1.11291498e-01 -5.31268299e-01 -6.73129976e-01\n",
            " -2.22358555e-01 -4.13092554e-01 -1.32581070e-01 -8.97292852e-01\n",
            " -3.67803186e-01  5.79051197e-01  5.55437386e-01  4.13013607e-01\n",
            " -1.07519336e-01  5.69857419e-01 -8.12488258e-01  6.49150252e-01\n",
            " -3.81251812e-01 -6.20615780e-01  4.60299775e-02 -6.57421291e-01\n",
            "  1.84985413e-03 -1.59808517e-01  4.60626602e-01 -5.08294880e-01\n",
            " -9.28072274e-01 -1.30952105e-01 -1.92819044e-01  9.69149768e-02\n",
            " -5.87294042e-01  1.50744794e-02  1.65371060e-01  5.35864115e-01\n",
            " -4.30809647e-01  2.94538826e-01  2.61020273e-01 -4.26228166e-01\n",
            "  8.08210969e-02  3.22023064e-01 -3.13578248e-01  1.26740083e-01\n",
            " -7.96869040e-01  6.01151645e-01  5.52909255e-01 -3.50639462e-01\n",
            " -3.80006522e-01 -3.64181370e-01  4.51557606e-01  6.50318086e-01\n",
            " -3.91237549e-02 -7.37679482e-01 -6.99330628e-01  2.88440704e-01\n",
            " -4.35721070e-01  6.69355869e-01  2.54650265e-01 -7.08027124e-01\n",
            " -9.06873584e-01 -3.68124932e-01 -4.29142676e-02 -8.95804763e-01\n",
            " -3.97175312e-01 -1.65277511e-01  2.03843802e-01 -8.80732715e-01\n",
            " -1.12977795e-01  8.84848654e-01 -2.67883509e-01 -3.23949665e-01\n",
            " -5.58366120e-01 -4.18254375e-01  5.64194441e-01  6.55582786e-01\n",
            " -4.36477125e-01 -2.55548686e-01  6.59340322e-01 -3.64416301e-01\n",
            " -4.28239882e-01  7.30168819e-01 -1.17606603e-01 -9.61345434e-01\n",
            " -4.06984210e-01 -3.11324388e-01 -1.81723312e-01  2.49585524e-01\n",
            " -3.99325579e-01  5.71305275e-01 -9.28526819e-01  3.79387066e-02\n",
            " -4.67025518e-01  3.40817064e-01 -3.20078433e-01  1.55760124e-01\n",
            "  4.00431216e-01 -7.41192937e-01 -4.33097035e-01 -6.99153662e-01\n",
            " -6.17618442e-01 -6.45133138e-01 -6.21838927e-01  5.05216479e-01\n",
            "  3.58497381e-01 -7.15888560e-01 -7.26042569e-01  4.88587826e-01\n",
            " -2.99351998e-02 -3.54412138e-01  2.12169126e-01 -5.97218871e-01\n",
            " -1.00636795e-01  7.49393523e-01 -1.61446571e-01  3.95636290e-01\n",
            " -3.89240593e-01 -6.02426469e-01 -3.38276207e-01 -6.97486997e-01\n",
            " -3.34594212e-02 -2.30877206e-01 -5.66529334e-01  4.44484919e-01\n",
            " -2.97897995e-01  4.45554256e-01  4.46550906e-01 -3.99061263e-01\n",
            " -7.85397947e-01  2.37903014e-01  1.96901411e-02  2.14983657e-01\n",
            " -2.36819208e-01  4.41515118e-01 -3.18766743e-01 -2.26240367e-01\n",
            " -4.80055869e-01 -4.67036590e-02 -2.29657024e-01  4.55208004e-01\n",
            "  3.53270918e-01 -8.20446074e-01 -3.23023707e-01 -2.69555509e-01\n",
            " -4.62120920e-02 -4.10896987e-01 -3.80041093e-01  1.77013680e-01\n",
            " -3.53696525e-01  1.18870504e-01 -3.06544155e-01  3.32430720e-01\n",
            " -6.80755258e-01  7.24003976e-03 -8.30492616e-01 -1.13784604e-01\n",
            " -6.12592936e-01 -1.09723218e-01 -4.63872254e-01 -3.77661169e-01\n",
            " -2.79087186e-01 -4.48706865e-01 -7.89669514e-01  3.66785258e-01\n",
            " -8.18802938e-02 -1.58386156e-01 -2.00904697e-01  4.45965379e-01\n",
            "  1.24647260e-01 -5.44619441e-01 -1.29172713e-01 -1.68297604e-01\n",
            " -9.95415747e-01 -3.31882715e-01 -7.58660555e-01  2.52766401e-01\n",
            " -4.74948525e-01 -4.56464618e-01 -6.54380798e-01 -2.67702043e-01\n",
            " -3.62460375e-01 -8.31215441e-01 -2.33447533e-02  1.71374559e-01\n",
            " -5.92169404e-01 -8.08229327e-01 -3.17690939e-01 -4.17424858e-01\n",
            " -9.55369055e-01 -4.20192676e-03  1.97042614e-01 -3.07986196e-02\n",
            "  6.48743331e-01 -8.12554598e-01 -4.90222186e-01  1.75345391e-02\n",
            "  3.78520995e-01  8.40205014e-01 -7.33133674e-01 -1.95497259e-01\n",
            "  3.76072854e-01 -3.72780114e-01  4.18732285e-01 -3.37055475e-01\n",
            "  2.00953141e-01 -6.95327103e-01  6.16477609e-01 -7.37176299e-01\n",
            " -1.44860893e-01 -1.82857916e-01 -6.36299491e-01  6.07831001e-01\n",
            "  4.53153878e-01 -8.97925854e-01 -3.42138782e-02  6.77986503e-01\n",
            " -8.59062910e-01  4.82968874e-02  1.37181759e-01 -8.02672088e-01\n",
            "  3.53134394e-01 -8.53288412e-01 -3.74959797e-01 -7.13719308e-01\n",
            "  2.77288884e-01  8.57304096e-01 -2.11550176e-01 -5.49753547e-01\n",
            " -4.08277333e-01 -6.69375539e-01 -7.88270354e-01 -7.20061183e-01\n",
            " -7.46905208e-01 -8.87433767e-01  2.08422273e-01 -5.64289391e-01\n",
            " -7.02939322e-03 -5.72862566e-01 -4.17721123e-01 -6.55543804e-01\n",
            " -5.78704655e-01 -7.39584565e-01 -9.76247434e-03  4.84998561e-02\n",
            " -2.12276485e-02 -2.90843338e-01 -1.75832927e-01  4.67195660e-01\n",
            " -2.57656723e-01 -1.64260402e-01 -6.30299747e-02 -7.99089000e-02\n",
            " -5.76260030e-01 -7.78527439e-01 -6.50664687e-01 -2.70233631e-01\n",
            "  6.51307106e-01 -5.97026885e-01  2.31090009e-01 -3.63534868e-01\n",
            " -7.35863090e-01 -2.27483049e-01  1.67578027e-01  3.86299640e-02\n",
            " -5.00081480e-01 -6.79161191e-01  4.03324336e-01  3.22656065e-01\n",
            "  7.35844553e-01 -8.87055218e-01 -2.01664254e-01 -7.19919503e-01\n",
            " -8.21030587e-02 -2.15061098e-01 -9.31484222e-01 -3.59796822e-01\n",
            " -5.41590273e-01 -7.54944205e-01 -2.35315442e-01  4.87647504e-01\n",
            " -6.87982365e-02 -3.71165633e-01 -8.49306762e-01 -1.02545395e-01\n",
            " -6.72311962e-01 -2.98519552e-01 -8.39977264e-01 -2.96220094e-01\n",
            "  4.35235143e-01 -2.28926167e-02 -8.55827451e-01  6.03749573e-01\n",
            "  6.46136045e-01 -6.27991438e-01 -1.53823197e-01 -6.61152661e-01\n",
            "  9.29401666e-02 -6.99891001e-02 -8.87793422e-01 -3.34353209e-01\n",
            "  2.35034674e-01  2.06301957e-01 -5.93050182e-01 -6.67407095e-01\n",
            " -1.84438094e-01  1.82833776e-01  6.05038464e-01 -5.02575576e-01\n",
            "  2.17069864e-01  1.11018665e-01 -4.29642498e-01 -7.15164363e-01\n",
            " -4.92166728e-02  5.15493393e-01 -4.00432855e-01  4.12787825e-01\n",
            "  3.38016242e-01  2.16170192e-01 -1.00334823e-01 -4.32813078e-01\n",
            " -7.68327653e-01 -6.03041090e-02 -4.19898987e-01 -3.75803411e-02\n",
            " -1.59237355e-01 -7.91520346e-03 -3.00191820e-01 -2.14423612e-01\n",
            " -5.30932188e-01 -2.65490323e-01 -3.53683829e-01  2.20837548e-01\n",
            "  1.13162279e-01  1.41627640e-01  4.32356268e-01  4.39280778e-01\n",
            " -9.07894731e-01  6.50099367e-02 -3.78293335e-01  3.29100728e-01\n",
            " -5.87677360e-01  6.13596559e-01 -5.38864076e-01  5.67442417e-01\n",
            " -6.33681834e-01 -1.10630907e-01 -4.97480273e-01 -2.97506839e-01\n",
            "  4.70659435e-01 -7.49362230e-01  7.11079657e-01  5.26992559e-01\n",
            " -4.07819934e-02  1.80620253e-01 -8.48771751e-01 -9.07245636e-01\n",
            " -3.52208108e-01 -2.92060941e-01  1.40166029e-01  6.21712387e-01\n",
            " -6.87086344e-01  3.32139760e-01 -4.51991946e-01 -9.38145816e-01\n",
            " -7.89847195e-01 -4.13638800e-01 -1.47766694e-01 -9.35322344e-02\n",
            " -6.55206978e-01  2.10419327e-01 -5.46281636e-01  1.29133061e-01\n",
            "  2.96895385e-01 -3.64731491e-01 -3.78573149e-01 -2.48426065e-01\n",
            " -5.85967660e-01 -4.94804025e-01  3.05213213e-01  3.50819290e-01\n",
            " -6.24287426e-01 -2.09226832e-01 -3.33515912e-01 -7.25116432e-01\n",
            "  5.41359603e-01 -8.03234816e-01 -1.36954099e-01 -6.20964646e-01\n",
            " -8.92752632e-02  4.33348954e-01 -3.35000724e-01 -8.14945579e-01\n",
            " -6.91829503e-01 -5.43447137e-01  3.27589840e-01 -2.58752614e-01\n",
            "  1.65230781e-01  6.39684916e-01  4.58631545e-01 -4.91626948e-01\n",
            " -6.94254339e-01 -4.50357161e-02  6.81996346e-02  8.05089831e-01\n",
            "  1.85791254e-01 -5.24057925e-01 -7.01889455e-01 -8.47217590e-02\n",
            " -9.25043643e-01 -8.30482841e-02 -4.11236547e-02  3.63654137e-01\n",
            " -2.17354089e-01 -7.19440520e-01  6.29191458e-01  2.04216674e-01\n",
            "  8.44457090e-01 -8.25559616e-01 -8.43700469e-01  9.11644846e-02\n",
            " -7.89044425e-02 -9.41595912e-01 -2.06297025e-01 -9.33042169e-01\n",
            "  2.25575820e-01  2.15012804e-01 -4.69605535e-01  6.73398376e-02\n",
            " -1.03059724e-01  1.29077241e-01 -8.45343113e-01 -6.94878280e-01\n",
            "  1.45814180e-01  6.15352280e-02 -3.00836444e-01 -1.52444616e-01\n",
            " -5.64250648e-01 -5.44888914e-01 -2.66101390e-01  3.25731456e-01\n",
            " -8.51610482e-01  3.22550088e-01  5.32336235e-01 -4.85062629e-01\n",
            " -2.15940103e-02  6.19048297e-01  1.00249592e-02 -9.46985900e-01\n",
            "  1.93134397e-01 -3.41278821e-01  1.72242388e-01  1.86932296e-01\n",
            " -8.08449626e-01 -9.04137269e-02 -1.47345707e-01 -9.09959301e-02\n",
            " -6.38229132e-01 -4.03603464e-01 -7.31914222e-01 -1.47330955e-01\n",
            "  3.48967165e-01  3.04972202e-01 -5.98020911e-01 -4.00374115e-01\n",
            " -5.08725405e-01  5.23270249e-01  5.17508900e-03  4.97014254e-01\n",
            "  8.64206970e-01  5.57898104e-01 -5.34302115e-01 -3.13135087e-02\n",
            "  2.38434032e-01  3.85585964e-01  1.64380565e-01 -5.42290151e-01\n",
            " -9.26025271e-01  5.36573648e-01 -8.27227160e-02 -2.39909351e-01\n",
            "  2.33746227e-02 -1.68113127e-01  2.83542603e-01 -2.98752397e-01\n",
            " -6.99806273e-01 -3.19086045e-01 -1.00528414e-03  4.61555868e-01\n",
            " -6.37386382e-01  1.37537122e-01  3.48405957e-01  2.41858169e-01\n",
            " -7.76238620e-01 -6.35178626e-01 -4.01111752e-01  7.81764925e-01\n",
            " -3.24759841e-01  1.08115569e-01 -5.92370212e-01 -1.54515475e-01\n",
            " -8.40551257e-01 -4.53013569e-01 -4.61322278e-01  8.80601630e-02\n",
            "  1.03834458e-01 -6.70018494e-01 -5.67977190e-01  5.57001233e-01\n",
            " -4.95178431e-01 -8.37637603e-01  5.43758512e-01 -9.64406252e-01\n",
            " -6.13650501e-01 -1.41821146e-01 -2.88616925e-01 -4.31389123e-01\n",
            " -2.03123882e-01 -8.93175900e-02  3.40911835e-01 -2.21303776e-01\n",
            "  6.88758373e-01 -5.15335083e-01 -3.31167459e-01  1.76518649e-01\n",
            " -7.11125731e-01  7.09060609e-01 -2.93251276e-01 -3.81959707e-01\n",
            " -6.44384682e-01 -8.12627137e-01  1.92787871e-01 -9.79463160e-02\n",
            "  9.28893864e-01  1.00891218e-01 -5.80820441e-01 -3.71964395e-01\n",
            " -3.55098903e-01 -5.94497442e-01 -2.17369035e-01 -2.09628314e-01\n",
            "  1.30884945e-01 -2.13194475e-01 -8.85959566e-01 -4.80243176e-01\n",
            " -4.19587761e-01 -2.38283277e-01 -7.35582590e-01 -6.99221253e-01\n",
            " -9.70788449e-02  4.91503000e-01 -6.19190812e-01 -6.71389341e-01\n",
            " -5.21422684e-01 -7.03059554e-01 -1.82467446e-01 -4.39355522e-01\n",
            "  3.50464582e-01  2.33731076e-01 -1.15313187e-01 -1.92623645e-01\n",
            " -7.69443989e-01 -2.11046427e-01  3.35699737e-01 -5.13551116e-01\n",
            "  5.48038363e-01  7.37040043e-02  1.42977148e-01 -3.43101531e-01\n",
            "  6.49982810e-01  1.53561711e-01 -2.66491026e-01 -8.87477517e-01\n",
            " -2.71187782e-01 -1.65977553e-01  4.38197821e-01 -9.28907454e-01\n",
            " -8.61912787e-01 -7.70051658e-01 -2.32842684e-01 -4.40464526e-01\n",
            " -7.42219806e-01 -4.32457894e-01 -2.86148727e-01 -5.79012573e-01\n",
            " -5.56685440e-02 -3.07525188e-01 -3.14705968e-01 -1.21166259e-01\n",
            " -2.13702530e-01  3.31181139e-01 -7.51060963e-01 -7.58321941e-01\n",
            "  8.73663723e-02 -8.22545309e-03 -3.41518432e-01  2.41798893e-01\n",
            " -9.63220954e-01 -3.67025495e-01 -5.76597273e-01 -4.81196254e-01\n",
            " -2.69626468e-01  3.44642580e-01 -8.02685171e-02 -4.53750491e-01\n",
            "  4.34297100e-02  4.15643930e-01 -1.04429774e-01  2.85035193e-01\n",
            "  7.24825859e-02 -7.84721434e-01 -8.77131447e-02  2.78648168e-01\n",
            " -4.78870012e-02  1.80666536e-01 -5.04586220e-01 -9.15653348e-01\n",
            "  3.06174725e-01 -2.99071312e-01 -5.97301602e-01  3.73918176e-01\n",
            "  1.94977924e-01 -3.92454356e-01 -7.08451569e-01  5.03691733e-01\n",
            " -1.08118095e-02  4.10072535e-01 -5.12158394e-01 -6.50795519e-01\n",
            " -3.91150154e-02 -4.16346081e-02  2.30884701e-02  6.69758141e-01\n",
            " -2.17256099e-01 -7.41500705e-02 -6.24031760e-02  2.75338292e-01\n",
            " -2.25632489e-01 -8.09725285e-01 -9.54447463e-02  5.39290130e-01\n",
            " -6.23674691e-01 -1.32768944e-01 -4.79471743e-01  2.98069626e-01\n",
            " -4.54901546e-01  3.65323424e-01 -3.49908710e-01 -6.51643574e-01\n",
            " -5.70843637e-01 -3.71810675e-01 -7.71398544e-01  6.94610476e-01\n",
            " -3.00314039e-01 -5.78861177e-01 -1.53785914e-01 -1.18774936e-01\n",
            " -6.90654218e-01 -3.19795072e-01  3.11266303e-01 -6.25120029e-02\n",
            " -8.19841474e-02 -2.21434891e-01  2.09488347e-02  1.13371372e-01\n",
            " -9.48986113e-01 -8.08885753e-01 -5.00734329e-01 -3.59180719e-01\n",
            " -9.90884975e-02  5.72343528e-01  6.22489333e-01 -3.78288746e-01\n",
            "  7.66610742e-01 -6.59030318e-01 -7.73107052e-01  3.19088519e-01\n",
            " -8.47735107e-01 -2.04064786e-01 -4.94060636e-01 -4.85347569e-01\n",
            " -4.08827990e-01  1.48306293e-02 -6.55594945e-01 -4.53201741e-01\n",
            " -1.81843922e-01 -5.03777623e-01  1.32679865e-01 -7.48722494e-01\n",
            "  1.35280132e-01  2.60567546e-01 -1.29923180e-01 -5.06256700e-01\n",
            "  8.36090207e-01 -3.84158581e-01  1.37774855e-01 -6.87242031e-01\n",
            " -8.32322061e-01  2.05072105e-01  2.28191450e-01 -2.00997174e-01\n",
            " -4.06722069e-01  1.01336911e-01 -1.87457591e-01 -4.23253655e-01\n",
            " -3.68195593e-01 -7.16464460e-01  6.32616222e-01  2.69170612e-01\n",
            "  3.90720695e-01  1.81250155e-01 -1.04285173e-01 -2.24341929e-01\n",
            "  1.82107165e-01 -7.19903886e-01  5.89017987e-01 -7.25772679e-01\n",
            " -5.71903527e-01 -3.64070356e-01 -5.71519852e-01 -4.85707313e-01\n",
            " -3.10296834e-01 -3.53831440e-01 -8.71606469e-01 -7.26791546e-02]\n",
            "Learn what documents you will need to get a Social Security Card | Social Security Administration#10_0\n",
            "369\n",
            "----------------------------------------\n",
            "What happens after I apply?\n",
            "[-6.75301254e-01  2.43208855e-01  6.39490366e-01 -6.12259865e-01\n",
            " -5.69108188e-01  6.74475310e-03 -7.94484258e-01 -5.40785789e-01\n",
            "  4.84390825e-01 -4.55266833e-01 -1.06093353e-02 -2.72895455e-01\n",
            "  3.45678419e-01 -6.19917214e-01  4.84078676e-02  2.05588758e-01\n",
            " -3.79673451e-01  4.44057018e-01 -7.01903030e-02  2.06228361e-01\n",
            "  1.11977689e-01 -9.33841050e-01  4.13922489e-01  2.82474458e-01\n",
            " -8.61727238e-01  8.07043791e-01  2.63576865e-01  5.15058637e-01\n",
            " -1.68711673e-02 -4.14877206e-01 -8.93229067e-01 -5.89475691e-01\n",
            " -2.85157382e-01 -6.63621351e-02 -1.09635182e-01 -7.64367700e-01\n",
            "  1.29325762e-01 -3.54618728e-01 -4.80651051e-01  7.59767354e-01\n",
            "  3.52080345e-01 -3.73438030e-01  1.34019172e-02 -5.08872151e-01\n",
            " -2.28260428e-01 -7.37419486e-01 -5.40937245e-01  2.62805939e-01\n",
            " -4.46989000e-01 -2.86414474e-01 -3.07230711e-01  5.69890678e-01\n",
            " -5.74865758e-01 -6.17557466e-01 -3.04428130e-01  3.67781907e-01\n",
            "  7.54547834e-01 -9.69621658e-01  6.84183314e-02  7.35907733e-01\n",
            " -2.23434269e-01  2.32859910e-01 -5.74779749e-01  2.71580577e-01\n",
            " -9.21194673e-01 -1.49785146e-01  6.94241345e-01  8.91363993e-02\n",
            " -4.32488501e-01 -7.13933647e-01 -1.69413134e-01 -1.53854210e-02\n",
            " -8.59785557e-01  6.35095090e-02 -3.73376198e-02  2.44069263e-01\n",
            " -2.06035480e-01  7.12401986e-01 -4.08540398e-01  4.19148713e-01\n",
            "  2.16164082e-01 -5.97631276e-01 -4.32310194e-01  6.63718581e-01\n",
            "  5.08043289e-01 -3.16178054e-01 -4.64692980e-01 -6.67567372e-01\n",
            "  5.50750613e-01 -2.42556676e-01 -2.44492814e-02  2.50631981e-02\n",
            " -2.52724469e-01 -6.03448367e-03 -1.95572987e-01  2.92554140e-01\n",
            "  4.77041714e-02 -5.85089087e-01 -1.53183714e-01  3.78784627e-01\n",
            "  5.26313186e-02  8.81119519e-02  1.07490383e-01 -7.62856960e-01\n",
            " -9.85513449e-01 -2.29217708e-01 -7.38208354e-01 -5.11850536e-01\n",
            " -3.17333966e-01 -9.48971987e-01 -1.59997344e-01  9.28940251e-02\n",
            "  3.59758168e-01 -5.85900784e-01  8.71070474e-02  6.72878185e-03\n",
            " -5.63137054e-01 -8.22297633e-01 -3.56999308e-01  3.22723061e-01\n",
            " -6.93406863e-03 -1.88125595e-01 -9.49276865e-01 -7.43669569e-01\n",
            " -8.12266350e-01  8.37831736e-01 -8.30242276e-01 -1.20497786e-01\n",
            " -3.54292750e-01  1.85155332e-01  4.89907950e-01 -6.66490018e-01\n",
            "  3.84287626e-01 -3.60553503e-01  1.73415579e-02 -8.54072928e-01\n",
            " -1.95616946e-01  7.61531472e-01  6.60340488e-01 -2.46485919e-02\n",
            " -4.64415938e-01  5.09971976e-01 -4.76713151e-01  7.47611895e-02\n",
            " -9.54537615e-02 -7.88658023e-01  2.88583577e-01 -3.92636716e-01\n",
            " -5.88806212e-01 -6.46479607e-01  2.59216011e-01 -8.95223379e-01\n",
            "  2.51741648e-01  1.58419490e-01 -1.30688697e-01 -3.89957651e-02\n",
            " -3.36381048e-02  7.58849010e-02  3.32578331e-01  1.41522825e-01\n",
            " -7.55085349e-01  1.42320529e-01  4.33879226e-01 -7.00106561e-01\n",
            "  6.11915767e-01 -8.94084215e-01 -3.29322308e-01  4.07831788e-01\n",
            " -6.31473541e-01 -9.54185128e-01 -5.53590715e-01 -7.07307577e-01\n",
            " -7.07975864e-01 -5.20529091e-01  3.52721661e-01  7.91191310e-02\n",
            "  3.96105617e-01  8.93547356e-01  1.95111573e-01 -9.24575686e-01\n",
            " -6.24791205e-01 -4.61447120e-01 -3.08090895e-01  2.78888136e-01\n",
            " -6.38923824e-01  1.51028395e-01 -4.60109115e-01 -2.18396679e-01\n",
            " -1.23179890e-01 -2.43788615e-01 -2.88650900e-01 -8.95355463e-01\n",
            " -7.80264065e-02  9.64899138e-02 -7.38389432e-01 -6.62551343e-01\n",
            " -4.10455257e-01 -9.30361032e-01  5.96513785e-02  5.80878079e-01\n",
            " -7.53473103e-01 -2.22076029e-01 -1.15026310e-01 -9.73547176e-02\n",
            " -8.46233666e-02  1.57279465e-02 -7.30528414e-01 -7.86915660e-01\n",
            "  3.59384924e-01 -6.40457273e-01 -8.42931986e-01  1.82430595e-01\n",
            " -2.72053301e-01  5.09151995e-01  1.71519220e-01 -8.96161377e-01\n",
            " -9.99459624e-01  4.62995544e-02  6.89007401e-01 -5.99745870e-01\n",
            "  4.17581409e-01 -4.12231505e-01 -5.71495354e-01 -4.93841290e-01\n",
            " -2.25417048e-01  2.90395081e-01 -9.31656778e-01  3.71993929e-01\n",
            " -4.76715773e-01 -7.96581089e-01 -7.42819548e-01 -1.32154822e-01\n",
            " -1.23030812e-01 -8.66281629e-01 -7.44470775e-01 -1.20398998e-01\n",
            " -9.37499255e-02  7.93098137e-02 -4.92129385e-01  2.19077677e-01\n",
            "  2.35140488e-01 -5.99237800e-01 -4.39943641e-01  6.38527215e-01\n",
            " -2.23232672e-01 -7.98369423e-02 -3.02617043e-01  4.00648296e-01\n",
            " -1.36788607e-01  4.57358122e-01 -8.62313569e-01 -8.64867494e-02\n",
            "  1.31243970e-02 -2.81983358e-03 -4.66216892e-01 -7.55478561e-01\n",
            " -1.28665224e-01 -8.61468166e-02  1.20676160e-01  5.53651154e-01\n",
            "  4.03255373e-01 -2.91989833e-01 -1.61207855e-01 -3.80002707e-01\n",
            " -9.70033765e-01 -5.11922419e-01  4.37525064e-01 -4.78114784e-01\n",
            "  6.02399945e-01 -3.38997036e-01 -2.61245131e-01 -4.19411659e-01\n",
            " -2.81993568e-01  5.19895144e-02  1.22510921e-02 -7.72725582e-01\n",
            " -6.61910772e-01  1.04476757e-01 -6.68357551e-01  7.67669320e-01\n",
            "  3.72594088e-01 -4.85255346e-02 -3.84034723e-01  4.43447053e-01\n",
            "  4.90372300e-01 -1.05498627e-01  1.30647793e-01 -9.06500816e-02\n",
            "  1.92238778e-01 -1.78087264e-01 -5.47529936e-01  1.28038093e-01\n",
            "  1.76262990e-01 -4.24140960e-01  5.83332300e-01  4.47138876e-01\n",
            " -6.31806612e-01  1.64386734e-01  5.64961255e-01  9.21851575e-01\n",
            " -9.01527405e-01 -7.85628110e-02 -5.59052452e-02  3.36465150e-01\n",
            " -3.42816740e-01 -3.41370821e-01 -5.12896292e-02  6.27770871e-02\n",
            "  3.32046360e-01 -3.81203473e-01 -2.06105202e-01  1.27744302e-01\n",
            " -4.38036889e-01 -6.97376132e-01  5.68499684e-01  5.97726107e-01\n",
            " -4.77536246e-02 -3.74859512e-01  9.80739519e-02  2.32929736e-01\n",
            "  4.25045669e-01 -2.09932163e-01 -3.02033555e-02 -7.27128237e-02\n",
            "  1.83892965e-01  4.25873846e-01  8.66787210e-02 -5.90002477e-01\n",
            "  1.50602972e-02  5.76616287e-01 -4.38898683e-01  5.78274786e-01\n",
            " -6.17495418e-01 -5.23779690e-01 -8.35001826e-01 -3.73185158e-01\n",
            " -1.61036372e-01 -6.19550884e-01 -4.50880378e-01  4.57113653e-01\n",
            "  1.47983775e-01 -7.91957378e-01  1.91559270e-01  1.40101954e-01\n",
            " -8.12188387e-01  5.18608809e-01 -2.49294445e-01 -7.55193651e-01\n",
            " -5.34884274e-01 -9.19666111e-01 -5.69005430e-01  4.22215372e-01\n",
            " -3.88053283e-02 -4.74238656e-02  4.52271879e-01 -4.87439156e-01\n",
            "  5.68333030e-01 -6.03458285e-01 -2.91377157e-01 -2.42980108e-01\n",
            " -9.05548990e-01  1.04687959e-01 -9.84219790e-01 -9.06664133e-01\n",
            " -5.69155701e-02  4.49401766e-01 -7.84006894e-01 -4.28741395e-01\n",
            " -7.62774169e-01  2.78055191e-01 -4.08844173e-01 -2.41429210e-01\n",
            " -7.44289979e-02 -7.77948678e-01  1.31791523e-02 -8.10703099e-01\n",
            "  1.77887127e-01 -2.21983954e-01 -5.79555154e-01 -7.22793937e-02\n",
            " -3.95540804e-01  1.60710201e-01 -4.26385522e-01  5.62708855e-01\n",
            "  1.96778461e-01 -3.00940961e-01 -8.17572594e-01 -8.22601095e-02\n",
            " -7.62240589e-01 -2.16295972e-01 -8.04529712e-02 -5.89201450e-01\n",
            "  9.90442038e-02  6.16873130e-02 -4.05447513e-01 -5.43811798e-01\n",
            " -1.89956322e-01  5.85051477e-01 -8.69209230e-01 -8.48839760e-01\n",
            "  1.14078879e-01  4.41616148e-01  6.41668379e-01  1.29848614e-01\n",
            " -6.52687371e-01 -6.51509285e-01  6.80307388e-01  9.00951862e-01\n",
            " -3.99913937e-01 -2.05327123e-01 -3.35895777e-01 -2.76946664e-01\n",
            " -1.50369033e-01 -6.39855504e-01  2.11997628e-01 -9.36657488e-01\n",
            " -8.32826346e-02  6.88162088e-01  6.24719867e-03 -1.43242642e-01\n",
            " -8.89148712e-01 -3.17504257e-01  6.34027719e-01 -8.60765696e-01\n",
            "  3.12490135e-01 -2.01141506e-01  3.58179599e-01  3.52231920e-01\n",
            " -8.00973535e-01 -3.27041000e-01 -2.15152398e-01 -1.44018218e-01\n",
            "  1.19972318e-01 -6.30591750e-01  6.09431624e-01 -8.04012001e-01\n",
            " -6.55492127e-01  5.84725559e-01 -7.77808249e-01 -8.87001514e-01\n",
            " -4.48446572e-01  7.49531150e-01 -7.34730840e-01 -1.06628925e-01\n",
            "  8.88963997e-01 -8.68803859e-01 -1.03348911e-01 -7.70966172e-01\n",
            " -9.56821442e-01 -7.42277801e-01  5.69440126e-01  7.11685836e-01\n",
            " -3.24840873e-01  6.86512530e-01  5.72768986e-01 -4.34428662e-01\n",
            " -3.85354936e-01 -1.66600775e-02 -5.07698596e-01  3.19889545e-01\n",
            " -9.72198486e-01  5.11011899e-01 -2.06691027e-01 -6.68470323e-01\n",
            " -2.54428118e-01  3.47973466e-01 -8.05774480e-02 -7.35023439e-01\n",
            "  4.63469885e-02 -6.38726279e-02 -1.06807195e-01 -7.22375512e-01\n",
            "  4.13551241e-01 -1.24499574e-01 -2.51980186e-01 -6.24584019e-01\n",
            "  2.43865818e-01 -5.61847985e-01 -7.53553271e-01  3.04882973e-01\n",
            " -6.46140039e-01 -7.17677414e-01 -8.91692843e-03  5.87735698e-02\n",
            " -7.72078991e-01 -1.05633289e-01 -4.40961495e-02  4.59478259e-01\n",
            " -8.84438634e-01 -9.68284607e-01 -4.90142763e-01  2.56823182e-01\n",
            "  2.65616536e-01 -9.84657884e-01  7.75278270e-01  2.12531716e-01\n",
            " -3.33792716e-01  7.10102022e-01  3.01965892e-01 -2.96527207e-01\n",
            " -1.78140298e-01  4.18722220e-02 -2.14224964e-01  5.20937264e-01\n",
            " -3.87662381e-01 -2.98793644e-01 -8.22642148e-01  5.40598452e-01\n",
            "  1.39272392e-01  4.04132843e-01 -3.04959603e-02  1.61055356e-01\n",
            " -4.76252705e-01 -5.20932198e-01 -5.56569219e-01 -8.47576082e-01\n",
            " -2.56608754e-01 -7.36502588e-01 -7.46982872e-01  6.15054190e-01\n",
            "  1.47463664e-01 -1.78017095e-01 -5.89492440e-01 -5.43358743e-01\n",
            "  5.25041997e-01 -9.03491378e-01 -3.83665375e-02 -3.46955866e-01\n",
            " -6.27488017e-01 -5.81443489e-01  1.12186104e-01 -2.50074286e-02\n",
            " -6.03620000e-02 -1.62923157e-01 -2.77099311e-01  4.79575582e-02\n",
            " -5.23205400e-01 -3.23501021e-01 -6.54048562e-01 -8.65240514e-01\n",
            " -2.36329153e-01 -6.33205593e-01 -9.16430175e-01 -8.58307302e-01\n",
            " -8.60347152e-01 -6.14571095e-01 -9.52463984e-01 -9.63982642e-01\n",
            "  1.64249957e-01  1.03630349e-01 -9.83227432e-01 -6.54186666e-01\n",
            "  3.46967876e-02 -6.80320740e-01  3.03472161e-01 -1.78599842e-02\n",
            "  6.66748150e-04 -9.30159628e-01  2.38788977e-01 -5.79212785e-01\n",
            "  2.84125656e-01 -6.22125864e-01 -5.65064073e-01 -9.67995301e-02\n",
            " -1.89008117e-01 -5.20446897e-01  3.91284168e-01 -5.74012280e-01\n",
            " -2.54103482e-01  2.63365924e-01 -3.69507194e-01 -7.42973328e-01\n",
            "  5.06188750e-01 -3.22737664e-01 -9.63924587e-01  2.14078486e-01\n",
            " -2.73241431e-01  1.09936833e-01 -9.66264665e-01 -7.79076099e-01\n",
            "  3.22680086e-01 -1.43688306e-01 -4.65642542e-01 -7.91138113e-01\n",
            "  3.10345497e-02 -2.63853490e-01 -6.86300844e-02 -9.51297998e-01\n",
            " -4.02778089e-02  3.62397879e-01  3.41681540e-01 -7.46123433e-01\n",
            " -7.46444464e-02 -5.97689390e-01 -8.97248983e-01 -7.22455561e-01\n",
            " -7.11258352e-02  5.95534265e-01 -4.83132571e-01 -8.62191498e-01\n",
            "  5.02192497e-01  4.12748992e-01  5.56971014e-01 -2.95345902e-01\n",
            "  4.85044599e-01 -7.17678726e-01  8.12961236e-02 -5.33039451e-01\n",
            " -5.00487983e-01  5.94324112e-01  5.29818356e-01  5.18935695e-02\n",
            " -1.00112237e-01 -7.62510538e-01  6.01963252e-02 -8.65222454e-01\n",
            "  3.77394557e-02 -1.11411214e-01 -5.56631684e-01  8.07575285e-02\n",
            " -7.59909213e-01 -4.89381939e-01 -2.53132492e-01 -7.08008111e-01\n",
            " -7.67607868e-01  1.31751120e-01 -7.51503885e-01 -6.11719251e-01\n",
            " -4.14770573e-01 -1.15329474e-02  1.78041548e-01  2.10354134e-01\n",
            " -4.34948713e-01 -7.37472236e-01 -7.43520916e-01  1.58238783e-01\n",
            " -7.39676654e-01  1.43353611e-01 -1.85213536e-01 -5.15299141e-01\n",
            " -9.52106833e-01 -3.78139853e-01  6.20963693e-01  3.34906280e-01\n",
            " -8.89645815e-01 -6.35187805e-01  2.81628788e-01 -1.16433918e-01\n",
            " -7.91723058e-02 -2.68079042e-01  3.42758507e-01 -1.94778085e-01\n",
            "  4.65909213e-01 -2.78702140e-01 -5.67518950e-01 -2.31986165e-01\n",
            " -1.80767134e-01 -1.55915506e-04 -9.38211977e-01 -8.66148293e-01\n",
            " -8.56107771e-01 -6.85718119e-01 -1.37259379e-01  6.20021999e-01\n",
            " -2.61318713e-01  3.49534988e-01 -5.41680515e-01  2.96324342e-01\n",
            " -7.71965802e-01 -5.51300347e-01  2.49317646e-01 -5.34510970e-01\n",
            " -6.58604860e-01  1.50785059e-01 -2.49814332e-01  3.96068245e-01\n",
            " -3.49652588e-01 -8.89533877e-01 -7.88587391e-01 -4.03877953e-03\n",
            " -4.07901518e-02 -7.86432028e-01  1.10010549e-01  4.53156710e-01\n",
            "  2.45180503e-01  5.38826406e-01 -2.60828733e-01  5.67427397e-01\n",
            " -6.38699293e-01 -3.01990688e-01 -7.73911178e-01 -5.00025526e-02\n",
            " -3.10450822e-01 -2.74595410e-01 -1.24446057e-01  2.62899399e-01\n",
            "  3.06636125e-01 -4.30812314e-02  8.92875612e-01 -8.34456086e-01\n",
            " -3.65062982e-01 -2.98590869e-01 -4.37341928e-01  5.57210982e-01\n",
            " -3.03423047e-01 -5.71051612e-02  3.94836754e-01  7.19051212e-02\n",
            " -8.25727046e-01 -5.09651244e-01  1.99085772e-01 -6.88574314e-01\n",
            " -3.09027076e-01 -4.22337689e-02  4.53999043e-01  3.31524432e-01\n",
            " -8.89243245e-01 -3.70096534e-01  4.72950399e-01  4.25955951e-02\n",
            "  5.75094700e-01 -6.71658993e-01  2.04535156e-01  7.59299994e-02\n",
            " -2.60845035e-01 -7.66509593e-01 -6.31220877e-01 -4.71115142e-01\n",
            " -3.03779468e-02 -6.73974216e-01 -6.60859421e-02 -2.84599904e-02\n",
            " -2.52298024e-02 -6.94178224e-01  7.66507164e-03 -4.36943799e-01\n",
            "  2.82530099e-01 -2.58642972e-01 -5.31300306e-01 -6.75079346e-01\n",
            " -9.04618740e-01 -4.91668582e-01 -9.98254195e-02 -3.65166426e-01\n",
            "  1.67325974e-01 -2.43680865e-01 -4.34819877e-01 -5.45022547e-01\n",
            " -5.26176274e-01  1.43723115e-01  2.61676669e-01 -6.20925307e-01\n",
            " -6.67249382e-01  7.10840002e-02 -1.72978461e-01  3.76797348e-01\n",
            " -3.75086159e-01 -8.12659442e-01 -3.65181476e-01 -4.25765187e-01\n",
            "  2.42674887e-01 -4.38169360e-01  4.84772354e-01  2.85498668e-02\n",
            " -9.26420212e-01  3.54662746e-01 -6.44717395e-01  9.02513489e-02]\n",
            "Disability Benefits | Social Security Administration#1_0\n",
            "384\n",
            "----------------------------------------\n",
            "For more information about our disability claims process ,\n",
            "[-2.91586399e-01  3.66558075e-01  3.74047130e-01 -2.84308523e-01\n",
            " -5.86941898e-01  5.16724214e-02 -1.42908201e-01  3.66933085e-02\n",
            "  1.64970800e-01 -5.17306626e-01 -4.42202866e-01 -5.60043752e-01\n",
            " -2.39579856e-01 -1.43662468e-01 -4.42769229e-02 -6.48269877e-02\n",
            "  3.58544350e-01 -4.33479637e-01 -4.73740399e-01  2.83699334e-01\n",
            "  1.43103033e-01  3.50578785e-01  4.13697660e-01  3.33252072e-01\n",
            " -8.50658119e-02  8.45112652e-02 -1.28351137e-01 -5.21748424e-01\n",
            "  9.90808681e-02 -5.53797901e-01  3.56719583e-01 -1.29383937e-01\n",
            "  2.86517233e-01  4.12109286e-01  1.79186448e-01 -4.72330958e-01\n",
            " -5.74704468e-01  5.81123948e-01 -5.95605373e-01  3.09279442e-01\n",
            " -1.41086638e-01 -5.94384015e-01  1.89881384e-01 -1.21385165e-01\n",
            "  1.60846800e-01  4.15690541e-01  6.40346885e-01  3.77445281e-01\n",
            " -3.47169131e-01  3.92285556e-01  3.28552365e-01  3.57884854e-01\n",
            " -1.98659729e-02 -3.71819466e-01 -1.44710513e-02 -1.59877658e-01\n",
            " -1.51570112e-01  5.65637589e-01  5.83223343e-01 -2.34269410e-01\n",
            "  5.45949996e-01  5.16339064e-01 -8.00858364e-02 -4.93342429e-01\n",
            "  5.18129170e-01  5.96861839e-01  5.18492818e-01  2.76539594e-01\n",
            "  5.06803453e-01 -6.18433952e-01 -1.34786695e-01  4.67839152e-01\n",
            "  1.31172746e-01 -6.42888546e-01 -2.29974926e-01 -2.01675296e-01\n",
            " -6.38069436e-02 -1.75204232e-01  2.96704054e-01 -2.63990536e-02\n",
            " -6.22857690e-01 -6.29802123e-02 -4.12446260e-02 -1.26217335e-01\n",
            " -6.16307974e-01 -3.44637990e-01  1.02951974e-01  1.77783996e-01\n",
            " -1.47062346e-01  5.60722768e-01  2.60593966e-02 -7.36200929e-01\n",
            " -1.51913241e-01 -1.63736179e-01 -4.16574299e-01  7.90624470e-02\n",
            "  1.03015848e-01  1.94768414e-01  1.57442112e-02  6.25233233e-01\n",
            " -4.76386368e-01  5.79095185e-01 -3.04473370e-01 -2.65350133e-01\n",
            " -1.05726533e-01 -2.80545026e-01 -9.97233093e-02 -7.84934819e-01\n",
            " -1.49080217e-01 -5.81864536e-01  3.26666266e-01 -6.15478575e-01\n",
            " -3.31376255e-01 -6.18350625e-01  6.16635859e-01 -1.06920108e-01\n",
            "  5.27704537e-01 -7.77288079e-02 -2.84927636e-01 -6.76194429e-01\n",
            "  2.63502419e-01 -2.61285692e-01 -3.57286900e-01 -2.53796339e-01\n",
            " -5.72041690e-01 -1.54375315e-01  8.48663375e-02 -6.44307137e-01\n",
            "  1.48801371e-01 -2.09404111e-01  6.93022013e-02 -5.50954103e-01\n",
            "  4.41544563e-01  3.44523489e-01  4.75366771e-01  1.33880883e-01\n",
            " -2.86939234e-01  1.10495267e-02 -2.85811633e-01 -4.00750905e-01\n",
            " -3.46639752e-01 -4.50977534e-01 -2.51335770e-01  1.27926320e-01\n",
            " -1.32036954e-01  3.19112629e-01 -2.58782148e-01  6.13870502e-01\n",
            "  3.59213501e-01  1.09462561e-02  4.74845588e-01  3.15427899e-01\n",
            " -2.72339415e-02  9.89684761e-02  2.23367661e-01  1.04085900e-01\n",
            "  1.48285449e-01 -1.45777643e-01  2.92534709e-01  4.51048046e-01\n",
            " -7.78538436e-02 -5.96367002e-01  2.27373078e-01 -2.66104341e-01\n",
            "  6.22540653e-01 -4.05162752e-01  2.72941142e-01  2.57831186e-01\n",
            "  2.36925796e-01 -8.72394517e-02 -1.08016238e-01 -6.78756982e-02\n",
            "  7.79631972e-01 -6.36891663e-01 -7.95810938e-01 -4.11642849e-01\n",
            "  1.31786793e-01  4.85714823e-01 -7.53040195e-01 -3.82042289e-01\n",
            " -1.24862805e-01  6.62801862e-01 -1.72581956e-01  4.45188314e-01\n",
            "  1.02898188e-01 -1.42167524e-01 -2.31717601e-01  6.35404468e-01\n",
            " -4.66741204e-01  5.50975464e-02 -1.45380661e-01 -1.75377503e-01\n",
            " -1.05272032e-01 -3.60373229e-01 -4.87341702e-01 -2.40671579e-02\n",
            "  2.27516681e-01  3.58467668e-01 -1.37412772e-02 -2.22874239e-01\n",
            " -8.73966932e-01  4.58748378e-02 -4.34726700e-02 -3.36965561e-01\n",
            " -5.67454891e-03  3.38533103e-01 -8.58416930e-02 -4.51675147e-01\n",
            "  1.79918036e-01  3.78442764e-01  3.99309881e-02  2.08633259e-01\n",
            " -7.39250332e-02  2.48972580e-01 -5.60340822e-01 -1.08654886e-01\n",
            " -9.94578600e-01 -2.77061582e-01 -3.05492412e-02  4.89310920e-02\n",
            " -6.62299991e-01  3.23428288e-02 -3.55894089e-01  2.80548543e-01\n",
            " -3.18871826e-01 -4.90001529e-01 -8.13301146e-01 -6.03587985e-01\n",
            "  6.55933917e-02  4.52778935e-01  1.61189854e-01  9.59901139e-02\n",
            "  3.40582430e-01 -4.91785318e-01  9.86958966e-02  1.85940385e-01\n",
            "  3.63629043e-01 -5.15565097e-01 -2.90667474e-01 -1.44975437e-02\n",
            " -2.77562708e-01 -6.15305424e-01 -4.05493170e-01  2.39316821e-01\n",
            "  4.91226129e-02  3.41445133e-02 -4.85204726e-01 -6.69142842e-01\n",
            "  3.78030509e-01 -1.96151391e-01  1.80616617e-01  1.03902876e-01\n",
            " -7.92799890e-02  1.20619215e-01 -4.98677015e-01 -2.19516773e-02\n",
            "  1.72299951e-01 -1.83861196e-01 -2.59133190e-01  2.23967023e-02\n",
            "  7.49088079e-02 -5.07209659e-01 -3.03912282e-01 -1.16054587e-01\n",
            " -1.61500201e-01  5.78450203e-01 -3.83290380e-01 -7.92198896e-01\n",
            " -2.78286606e-01 -5.07186115e-01 -1.99276395e-02 -6.06823683e-01\n",
            " -8.51967037e-02 -8.02716494e-01  4.33366448e-01 -7.52363980e-01\n",
            "  5.79274952e-01 -4.40349579e-01  3.76112431e-01  2.88741678e-01\n",
            " -3.84237468e-01 -2.15903148e-01  6.84104741e-01  2.71142721e-01\n",
            "  2.00351495e-02 -2.25773707e-01  3.12258631e-01  3.60557109e-01\n",
            " -2.73889094e-03 -1.48385242e-01 -7.15052068e-01 -4.38756436e-01\n",
            "  5.74099898e-01 -7.07262397e-01 -6.07429802e-01  3.29452127e-01\n",
            " -5.58188975e-01  5.59502542e-02 -2.29474320e-03 -2.68355638e-01\n",
            "  1.81972280e-01 -6.66884184e-01 -6.20820820e-01  5.60229123e-01\n",
            "  4.02780771e-02  3.26785713e-01  2.29041558e-02 -3.73092815e-02\n",
            " -1.24880426e-01 -5.16500950e-01  2.82090575e-01 -3.22915792e-01\n",
            "  2.36452729e-01 -3.48637365e-02  2.40039542e-01 -4.91547436e-01\n",
            " -8.15867931e-02  5.50472677e-01 -5.02517283e-01 -1.77034691e-01\n",
            "  6.95944667e-01  4.21834514e-02 -1.62130043e-01  2.37990141e-01\n",
            " -4.64510284e-02  1.76534608e-01  4.97994214e-01  1.14052460e-01\n",
            " -1.28537729e-01  3.74505192e-01 -5.16348243e-01  1.64669007e-01\n",
            " -2.10016668e-01 -4.93476912e-02  3.02201778e-01 -1.72856644e-01\n",
            " -1.82218000e-01  3.97691317e-02  1.54351071e-01  2.94528693e-01\n",
            " -4.12311882e-01  8.16490501e-02 -7.06033766e-01 -1.62826553e-01\n",
            " -9.35819298e-02 -8.24104369e-01  5.58776438e-01 -1.83011860e-01\n",
            "  2.74282396e-01 -7.23482251e-01  6.22562086e-03 -6.65690660e-01\n",
            " -3.33384931e-01  1.29704371e-01  8.24834555e-02 -1.70225337e-01\n",
            "  9.88859800e-04  1.35598347e-01 -2.67731637e-01 -7.40459144e-01\n",
            "  6.09641254e-01 -2.07594976e-01 -3.50584626e-01 -4.16603051e-02\n",
            "  3.31576556e-01 -1.44874036e-01  5.50958276e-01 -7.05946863e-01\n",
            "  3.83756161e-01  8.59787688e-02 -3.82900953e-01 -1.23151727e-01\n",
            " -5.91408312e-01 -4.72274087e-02  1.73749954e-01 -1.53551430e-01\n",
            "  3.36103201e-01 -6.56952381e-01  8.83490294e-02  2.79002666e-01\n",
            "  4.35929239e-01  4.28802192e-01  5.20321667e-01  6.98869526e-02\n",
            "  4.00560588e-01  3.52432616e-02 -6.16437793e-01  4.84388977e-01\n",
            "  7.03636646e-01 -1.03402130e-01 -7.13511333e-02 -1.08016595e-01\n",
            "  1.33965224e-01 -3.02979171e-01 -3.88251871e-01 -8.35086286e-01\n",
            "  6.23889193e-02 -3.39169204e-01  9.11630914e-02 -2.77245224e-01\n",
            " -4.76585418e-01 -5.78941822e-01 -6.24475837e-01 -9.37548727e-02\n",
            "  2.35771403e-01  4.46888924e-01  5.51463425e-01 -5.28588593e-02\n",
            "  3.29777479e-01  1.48810983e-01 -4.63499516e-01  1.67805791e-01\n",
            " -7.81773403e-02  2.40162313e-01 -7.12602446e-03  4.87671018e-01\n",
            " -3.07211041e-01  3.76327038e-01  3.02790165e-01 -4.22729366e-02\n",
            "  8.89181867e-02  3.31368521e-02  1.81815863e-01  1.98349237e-01\n",
            "  6.57268614e-02 -1.74607292e-01  4.64112014e-01 -7.24994719e-01\n",
            " -2.41675764e-01 -1.00357637e-01 -4.18082863e-01 -3.72058362e-01\n",
            " -7.19212353e-01  1.61901399e-01  5.15817463e-01 -6.12024605e-01\n",
            "  1.11243717e-01 -3.43525231e-01  5.61156511e-01  1.09588943e-01\n",
            "  1.35550961e-01  7.38714874e-01 -5.88200271e-01  4.81007844e-02\n",
            " -2.76707798e-01 -2.80794859e-01 -1.40932500e-01  4.05027270e-01\n",
            " -8.02298307e-01  2.41916284e-01  3.88364643e-01 -3.56059670e-01\n",
            " -2.07499191e-01 -9.46340486e-02 -5.59591293e-01 -3.42400521e-01\n",
            "  5.32050878e-02 -2.05395296e-01  3.46994728e-01 -3.15687507e-01\n",
            " -4.12039459e-01 -6.48547411e-01  4.54347841e-02 -1.62424102e-01\n",
            "  4.26026076e-01 -5.15951753e-01  5.09240687e-01 -3.01903486e-01\n",
            " -1.90465569e-01  6.75348759e-01  1.86768353e-01 -1.94959387e-01\n",
            " -4.17722791e-01  5.49465239e-01 -2.42546842e-01  4.83189598e-02\n",
            "  5.01367629e-01 -4.42130566e-02 -5.65027773e-01  1.12419687e-02\n",
            " -5.25404930e-01 -1.73542678e-01 -4.00373898e-03  3.73070180e-01\n",
            "  9.42473635e-02  1.23560935e-01 -4.78282452e-01 -5.50888419e-01\n",
            "  8.55497569e-02  2.81461418e-01  1.63591564e-01  3.30302417e-02\n",
            "  2.54751235e-01 -5.60923934e-01  3.38981718e-01 -5.93021095e-01\n",
            " -2.67501771e-01  1.92426950e-01 -5.24659872e-01 -2.56025285e-01\n",
            "  5.40965736e-01 -4.20027934e-02 -3.26609910e-01  7.68460557e-02\n",
            " -1.05418362e-01  3.90424639e-01 -1.38878837e-01 -3.00845504e-01\n",
            " -1.31327793e-01  1.29507378e-01 -5.07789314e-01 -1.48954868e-01\n",
            "  4.25515436e-02  3.05893928e-01 -5.58288932e-01 -4.95465070e-01\n",
            "  2.84297734e-01 -3.52968648e-02 -1.30162701e-01 -3.36898118e-01\n",
            " -4.38105285e-01  4.16035615e-02 -5.24140596e-02 -2.54006475e-01\n",
            " -4.72772047e-02  3.10221046e-01 -5.21400392e-01  7.29287267e-01\n",
            "  3.76785338e-01  4.51720923e-01 -1.63822994e-01 -3.57547939e-01\n",
            " -6.96752191e-01 -1.51049152e-01 -1.98309541e-01  1.40892103e-01\n",
            " -2.20704243e-01 -3.29686731e-01 -5.09953141e-01 -5.01875699e-01\n",
            "  2.50885397e-01 -2.58325815e-01 -2.33135641e-01 -3.80963475e-01\n",
            " -2.51302123e-01 -4.97385949e-01 -2.30863750e-01 -8.51861775e-01\n",
            " -1.81282744e-01  7.03540862e-01  6.34830952e-01 -5.73922336e-01\n",
            " -4.14158642e-01 -1.15757450e-01  3.73946428e-01 -3.05624217e-01\n",
            "  2.29708999e-01 -2.90832937e-01  6.19049277e-03 -5.99062182e-02\n",
            "  1.22539006e-01  2.07791924e-01 -6.55192852e-01  7.07561135e-01\n",
            " -3.30294698e-01 -5.15588760e-01 -4.52891022e-01  5.63767180e-02\n",
            " -9.82074253e-03  8.37004483e-02  7.75984108e-01  1.72913238e-01\n",
            "  1.67282030e-01 -1.63565233e-01  3.02185100e-02 -7.65381083e-02\n",
            "  6.13158584e-01 -2.24322110e-01 -4.63158101e-01 -5.05889475e-01\n",
            " -3.97202134e-01 -7.76344061e-01 -5.73563218e-01 -3.46901923e-01\n",
            " -2.48612925e-01 -4.79467690e-01 -4.03350949e-01 -2.90716112e-01\n",
            " -6.18023202e-02  7.71844713e-03 -3.06133300e-01 -8.41543555e-01\n",
            " -2.79004365e-01 -3.26715946e-01 -2.38006678e-03  5.31792819e-01\n",
            "  5.45006871e-01 -1.73669457e-01  4.51992601e-02 -7.47064292e-01\n",
            " -2.02011898e-01  5.15978098e-01  5.33138692e-01 -2.09474236e-01\n",
            "  4.56197917e-01 -6.06832914e-02  2.84903347e-01 -7.04592586e-01\n",
            " -3.68301898e-01 -4.07528013e-01 -3.91666859e-01 -7.28066787e-02\n",
            " -2.94998169e-01 -2.18443088e-02 -9.67729241e-02 -3.46776843e-01\n",
            "  5.50233305e-01  5.13861835e-01  3.03335139e-04  6.70679808e-01\n",
            " -6.05202556e-01 -3.20210755e-01 -6.58836961e-01 -4.19530779e-01\n",
            "  1.23048924e-01  1.78511590e-01 -6.82274222e-01  6.43106759e-01\n",
            " -2.29310051e-01 -2.31077626e-01  7.82497466e-01  2.37521693e-01\n",
            " -1.87465116e-01 -6.17033124e-01 -3.48322093e-01  6.70809001e-02\n",
            "  3.79954964e-01 -3.95664901e-01  8.96757394e-02  2.74304032e-01\n",
            "  1.95602421e-04 -3.49557370e-01 -4.17750448e-01 -1.91236138e-01\n",
            " -1.28691211e-01  1.18309081e-01 -6.65212035e-01 -1.10881075e-01\n",
            " -3.48924607e-01  1.18077934e-01 -6.76679552e-01 -3.90679389e-02\n",
            " -6.70853972e-01 -4.70835865e-01 -3.01062971e-01 -1.73525419e-02\n",
            "  5.27253389e-01 -1.01819269e-01 -7.51409352e-01 -9.67734531e-02\n",
            "  1.55250892e-01 -1.10549793e-01 -6.49129450e-01  5.06661892e-01\n",
            " -3.93850535e-01  4.02028501e-01  4.93910640e-01  9.39036310e-02\n",
            "  2.62249589e-01  7.64853358e-02 -4.53615397e-01 -2.17435047e-01\n",
            " -4.26682174e-01  2.09063396e-01  6.72699213e-02 -5.71253970e-02\n",
            "  4.39250380e-01 -2.36977071e-01 -4.49310273e-01  5.38317323e-01\n",
            "  1.66730836e-01 -2.46039703e-01  4.55787480e-01 -5.24559058e-02\n",
            "  1.92489754e-02 -1.11841857e-01  1.28480166e-01  2.34962642e-01\n",
            "  1.19166784e-01 -6.25388801e-01  4.08225328e-01 -5.27860284e-01\n",
            "  4.94813100e-02  1.09167561e-01  1.51515231e-01 -6.30268216e-01\n",
            "  2.85391569e-01 -3.34192067e-01 -6.43594086e-01 -6.41542748e-02\n",
            " -1.03902504e-01  8.03514600e-01  5.95620498e-02  2.00009663e-02\n",
            "  2.01861396e-01 -7.95183778e-01 -4.68884975e-01  4.37190458e-02\n",
            "  1.30496219e-01 -5.17846644e-01 -1.72823831e-01 -4.08436984e-01\n",
            "  1.37345186e-02  9.12005752e-02  5.30932963e-01  4.40162927e-01\n",
            "  5.84353447e-01  1.57690160e-02  5.11076331e-01  4.11527097e-01\n",
            "  6.23393714e-01  2.25874811e-01  2.22932443e-01 -4.20529097e-01\n",
            " -7.11285770e-01 -8.59886289e-01 -2.26196259e-01 -2.77443558e-01\n",
            " -5.69253862e-02  3.74669395e-02  4.01997536e-01  3.01548570e-01\n",
            "  5.28065078e-02  2.00785920e-01 -4.75359768e-01 -5.17894208e-01\n",
            " -1.82106167e-01 -3.82460743e-01 -4.71564025e-01 -2.90099531e-01\n",
            " -6.27713323e-01 -3.44684869e-01 -4.82126558e-03 -3.71051520e-01\n",
            "  1.54826224e-01  4.10538435e-01 -2.71952659e-01  3.18833977e-01\n",
            " -7.47692108e-01  8.11640203e-01  2.92371571e-01  1.65330440e-01\n",
            " -7.91805089e-01 -7.46647567e-02  4.23600584e-01  2.92107999e-01\n",
            "  5.30923605e-01 -5.92866242e-01 -4.46403414e-01 -5.08783400e-01\n",
            "  2.29918882e-01  1.73061222e-01  3.70183289e-01 -1.54328600e-01\n",
            " -2.91109353e-01  1.89991280e-01 -5.20559490e-01  1.63162738e-01]\n",
            "Benefits Planner: Disability | How You Qualify | Social Security Administration#2_0\n",
            "436\n",
            "----------------------------------------\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## PLDA Training"
      ],
      "metadata": {
        "id": "VPje5bvNK2km"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import plda"
      ],
      "metadata": {
        "id": "vHd-AfZaLRri"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Fit model for overfit classifier\n",
        "This method overfits the PLDA instead of this method we could also use bellow method:\n",
        "\n",
        "```\n",
        "better_classifier = plda.Classifier()\n",
        "better_classifier.fit_model(X, y, n_principal_components=5)\n",
        "```"
      ],
      "metadata": {
        "id": "2tE5FSSvK7Al"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "PLDA_classifier = plda.Classifier()\n",
        "PLDA_classifier.fit_model(np.array(X),\n",
        "                          np.array(y))"
      ],
      "metadata": {
        "id": "vczqDQCNK1u0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import pickle\n",
        "with open('plda_clf.pkl', 'wb') as f:\n",
        "    pickle.dump(PLDA_classifier, f)"
      ],
      "metadata": {
        "id": "Bfpk4FxdR36c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 202
        },
        "outputId": "a345dfa2-ba3d-4e77-d325-d4861c5f19f7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-9-4eddaea89546>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mpickle\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'plda_clf.pkl'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'wb'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m     \u001b[0mpickle\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdump\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mPLDA_classifier\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m: name 'PLDA_classifier' is not defined"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## PLDA Testing\n",
        "In this section we wanted to test the trained PLDA model."
      ],
      "metadata": {
        "id": "QJeCrgd0MLxI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def predict_doc(query):\n",
        "    \"\"\"\n",
        "    Predict which document is matched to the given query.\n",
        "\n",
        "    :param query: input query\n",
        "    :type query: str (or list of strs)\n",
        "    :return: return the document name\n",
        "    \"\"\"\n",
        "    query_embedding = get_embeddings(query)\n",
        "    predictions, log_p_predictions = PLDA_classifier.predict(query_embedding)\n",
        "    return labels[predictions]"
      ],
      "metadata": {
        "id": "u5zEl21tMR1m"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "test_queries = [\"I'm looking for information regarding benefits planning, can you help me?\",\n",
        "                \"I want to know about the benefits plan for survivors, can you give me more information about this?\",\n",
        "                \"What are Social Security credits?\",\n",
        "                \"Do you have any knowledge of Adult Disability Report? What if my spouse and I are no longer together?\"]\n",
        "test_labels = [\"Benefits Planner: Survivors | Planning For Your Survivors | Social Security Administration#1_0\",\n",
        "               \"Benefits Planner: Survivors | Planning For Your Survivors | Social Security Administration#1_0\",\n",
        "               \"Benefits Planner: Survivors | Planning For Your Survivors | Social Security Administration#1_0\",\n",
        "               \"Benefits Planner: Survivors | Planning For Your Survivors | Social Security Administration#1_0\"]"
      ],
      "metadata": {
        "id": "1DDpXqvmOHl_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "for query in test_queries:\n",
        "    print(predict_doc(query))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OqQr1-PGRi4J",
        "outputId": "85e2d3cd-ed79-4e6e-f626-8f4c365fa56f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "How To Apply For The GI Bill | Veterans Affairs#1_0\n",
            "Benefits Planner: Disability | Social Security Administration#1_0\n",
            "Benefit Verification Letter  | Social Security Administration#1_0\n",
            "Benefits Planner: Disability | How You Qualify | Social Security Administration#1_0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Final Test"
      ],
      "metadata": {
        "id": "EHF39mFm6Q3W"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!gdown --id 1ALs7qEOVzY8B-JF1BKuOJzGcS_1DcGbz"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LzAW4zRQ6Qbv",
        "outputId": "2ab20527-55f0-4abc-9619-e0c65db2c2b1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=1ALs7qEOVzY8B-JF1BKuOJzGcS_1DcGbz\n",
            "To: /content/plda_clf.pkl\n",
            "\r  0% 0.00/16.6M [00:00<?, ?B/s]\r100% 16.6M/16.6M [00:00<00:00, 230MB/s]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "doc_sentence_train = []\n",
        "doc_label_train = []\n",
        "for doc_idx1 in multidoc2dial_doc['doc_data']:\n",
        "    for doc_idx2 in multidoc2dial_doc['doc_data'][doc_idx1]:\n",
        "        for doc_idx3 in multidoc2dial_doc['doc_data'][doc_idx1]\\\n",
        "                                          [doc_idx2]['spans']:\n",
        "            doc_sentence_train.append(clean_text(multidoc2dial_doc['doc_data']\\\n",
        "                                                 [doc_idx1][doc_idx2]['spans']\\\n",
        "                                                 [doc_idx3]['text_sp']))\n",
        "            doc_label_train.append(doc_idx2)\n",
        "labels = list(set(doc_label_train))"
      ],
      "metadata": {
        "id": "4XqrJVc46nA_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import pickle\n",
        "with open('plda_clf.pkl', 'rb') as f:\n",
        "    PLDA_classifier = pickle.load(f)"
      ],
      "metadata": {
        "id": "mIqztxyJ62CK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def if_predicted(query, predicted):\n",
        "    if isinstance(predicted, (int, np.uint8)):\n",
        "        return query == predicted\n",
        "    return True if query in predicted else False"
      ],
      "metadata": {
        "id": "Fc2jZWulK4we"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def predict_doc_at(query, k=1):\n",
        "    \"\"\"\n",
        "    Predict which document is matched to the given query.\n",
        "\n",
        "    :param query: input query\n",
        "    :type query: str (or list of strs)\n",
        "    :param k: number of returning docs\n",
        "    :type k: int \n",
        "    :return: return the document name\n",
        "    \"\"\"\n",
        "    query_embedding = get_embeddings(query)\n",
        "    predictions, log_p_predictions = PLDA_classifier.predict(query_embedding,\n",
        "                                                             n_best=3)\n",
        "    predictions = predictions[:k]\n",
        "    sum_log = np.sum(np.exp(-log_p_predictions))\n",
        "    accuracy = list(map(lambda x: np.exp(-x) / sum_log,\n",
        "                        log_p_predictions[predictions]))\n",
        "    predictions = list(map(lambda x: labels[x], predictions))\n",
        "    return accuracy, predictions"
      ],
      "metadata": {
        "id": "DwTrT9WERC0t"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "for query in test_queries:\n",
        "    accs, preds = predict_doc_at(query, k=5)\n",
        "    print(accs)\n",
        "    print(preds)\n",
        "    print('-' * 20)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 358
        },
        "id": "qhzog-WlUirm",
        "outputId": "9f9ae4bc-434e-46ab-cb00-2715acbe34f2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "TypeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-27-1b6edff3a3fd>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mquery\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtest_queries\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m     \u001b[0maccs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpreds\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpredict_doc_at\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mquery\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0maccs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpreds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'-'\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;36m20\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-26-91292f270516>\u001b[0m in \u001b[0;36mpredict_doc_at\u001b[0;34m(query, k)\u001b[0m\n\u001b[1;32m     11\u001b[0m     \u001b[0mquery_embedding\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_embeddings\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mquery\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     12\u001b[0m     predictions, log_p_predictions = PLDA_classifier.predict(query_embedding,\n\u001b[0;32m---> 13\u001b[0;31m                                                              n_best=3)\n\u001b[0m\u001b[1;32m     14\u001b[0m     \u001b[0mpredictions\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpredictions\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m     \u001b[0msum_log\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0mlog_p_predictions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mTypeError\u001b[0m: predict() got an unexpected keyword argument 'n_best'"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import json\n",
        "with open('multidoc2dial/multidoc2dial_dial_train.json', 'r') as f:\n",
        "    multidoc2dial_dial_train = json.load(f)"
      ],
      "metadata": {
        "id": "6gcVU-G6VwrQ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "multidoc2dial_dial_train['dial_data']['dmv'][0]['turns'][0]['utterance']"
      ],
      "metadata": {
        "id": "5H8HeQbjdSIW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "multidoc2dial_dial_train['dial_data']['dmv'][0]['turns'][0]['references'][0]['doc_id']"
      ],
      "metadata": {
        "id": "31qqk1iDdjnj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "doc_sentence_test = []\n",
        "doc_label_test = []\n",
        "for doc_idx1 in multidoc2dial_dial_train['dial_data']:\n",
        "    for dial in multidoc2dial_dial_train['dial_data'][doc_idx1]:\n",
        "        for turns in dial['turns']:\n",
        "            if turns['role'] == \"user\":\n",
        "                doc_sentence_test.append(turns['utterance'])\n",
        "                doc_label_test.append(turns['references'][0]['doc_id'])"
      ],
      "metadata": {
        "id": "8-GHz2gAdqMi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "for i in [1, 100, 1000, 2000, 5000]:\n",
        "    print(doc_sentence_test[i])\n",
        "    print(doc_label_test[i])\n",
        "    print('--' * 20)"
      ],
      "metadata": {
        "id": "T5P76rvXf0nv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "TEST_SIZE = len(doc_sentence_test)\n",
        "TEST_SIZE"
      ],
      "metadata": {
        "id": "2auqkgCBhBbo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "prec_at_500 = 0\n",
        "prec_at_100 = 0\n",
        "prec_at_50 = 0\n",
        "prec_at_10 = 0\n",
        "prec_at_5 = 0\n",
        "prec_at_1 = 0\n",
        "sample_till_now = 0\n",
        "ranks = []\n",
        "for query, act_doc in zip(doc_sentence_test[:TEST_SIZE],\n",
        "                          doc_label_test[:TEST_SIZE]):\n",
        "    accs, preds = predict_doc_at(query, k=500)\n",
        "    ranks.append(1/ (preds.index(act_doc) + 1))\n",
        "    # print(accs)\n",
        "    # print(preds)\n",
        "    # print(act_doc)\n",
        "    # print('-' * 20)\n",
        "    if act_doc == preds[0]:\n",
        "        prec_at_1 += 1\n",
        "    if act_doc in preds[:5]:\n",
        "        prec_at_5 += 1\n",
        "    if act_doc in preds[:10]:\n",
        "        prec_at_10 += 1\n",
        "    if act_doc in preds[:50]:\n",
        "        prec_at_50 += 1\n",
        "    if act_doc in preds[:100]:\n",
        "        prec_at_100 += 1\n",
        "    if act_doc in preds[:500]:\n",
        "        prec_at_500 += 1\n",
        "    sample_till_now += 1\n",
        "    if sample_till_now % 10 == 0:\n",
        "        print(\"MRR: mean={}, var={}\".format(np.array(ranks).mean(), np.array(ranks).var()))\n",
        "        print(\"Prec@(1) = {} | Prec@(5) = {} | Prec@(10) = {} | Prec@(50) = {} | Prec@(100) = {} | Prec@(500) = {} | NUMBER_OF_SAMPLES = {}\".\\\n",
        "              format(prec_at_1 / sample_till_now, prec_at_5 / sample_till_now,\n",
        "                     prec_at_10 / sample_till_now, prec_at_50 / sample_till_now,\n",
        "                     prec_at_100 / sample_till_now, prec_at_500 / sample_till_now,\n",
        "                     sample_till_now))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 611
        },
        "id": "VXPXMkWGhU1I",
        "outputId": "de8501c0-121f-4f2c-905e-91633e0262c0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:19: RuntimeWarning: overflow encountered in exp\n",
            "/usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py:86: RuntimeWarning: overflow encountered in reduce\n",
            "  return ufunc.reduce(obj, axis, dtype, out, **passkwargs)\n",
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:20: RuntimeWarning: overflow encountered in exp\n",
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:20: RuntimeWarning: invalid value encountered in double_scalars\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Prec@(1) = 0.0 | Prec@(5) = 0.0 | Prec@(10) = 0.0 | Prec@(50) = 0.2 | Prec@(100) = 0.3 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 10\n",
            "Prec@(1) = 0.0 | Prec@(5) = 0.0 | Prec@(10) = 0.0 | Prec@(50) = 0.3 | Prec@(100) = 0.4 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 20\n",
            "Prec@(1) = 0.0 | Prec@(5) = 0.0 | Prec@(10) = 0.0 | Prec@(50) = 0.3 | Prec@(100) = 0.36666666666666664 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 30\n",
            "Prec@(1) = 0.0 | Prec@(5) = 0.0 | Prec@(10) = 0.0 | Prec@(50) = 0.275 | Prec@(100) = 0.4 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 40\n",
            "Prec@(1) = 0.0 | Prec@(5) = 0.0 | Prec@(10) = 0.0 | Prec@(50) = 0.26 | Prec@(100) = 0.4 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 50\n",
            "Prec@(1) = 0.0 | Prec@(5) = 0.0 | Prec@(10) = 0.0 | Prec@(50) = 0.21666666666666667 | Prec@(100) = 0.3333333333333333 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 60\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-67-3d811f786056>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      8\u001b[0m for query, act_doc in zip(doc_sentence_test[:TEST_SIZE],\n\u001b[1;32m      9\u001b[0m                           doc_label_test[:TEST_SIZE]):\n\u001b[0;32m---> 10\u001b[0;31m     \u001b[0maccs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpreds\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpredict_doc_at\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mquery\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m500\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     11\u001b[0m     \u001b[0;31m# print(accs)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     12\u001b[0m     \u001b[0;31m# print(preds)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-65-52f194d2dcf9>\u001b[0m in \u001b[0;36mpredict_doc_at\u001b[0;34m(query, k)\u001b[0m\n\u001b[1;32m     14\u001b[0m                                            to_space='U_model')\n\u001b[1;32m     15\u001b[0m     logpps_k, K = PLDA_classifier.calc_logp_pp_categories(data,\n\u001b[0;32m---> 16\u001b[0;31m                                                           False)\n\u001b[0m\u001b[1;32m     17\u001b[0m     \u001b[0mbest_k_idx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlogpps_k\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margsort\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     18\u001b[0m     \u001b[0mpredictions\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mK\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mbest_k_idx\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/plda/classifier.py\u001b[0m in \u001b[0;36mcalc_logp_pp_categories\u001b[0;34m(self, data, normalize_logps)\u001b[0m\n\u001b[1;32m    158\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    159\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mk\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mK\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 160\u001b[0;31m             \u001b[0mlogpps_k\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcalc_logp_posterior_predictive\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    161\u001b[0m             \u001b[0mlogpps_by_category\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlogpps_k\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    162\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/plda/model.py\u001b[0m in \u001b[0;36mcalc_logp_posterior_predictive\u001b[0;34m(self, U_model, category)\u001b[0m\n\u001b[1;32m    110\u001b[0m         \u001b[0mcov_diag\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mposterior_predictive_params\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mcategory\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'cov_diag'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    111\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 112\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mgaussian\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmean\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdiag\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcov_diag\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlogpdf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mU_model\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    113\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    114\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mcalc_logp_marginal_likelihood\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mU_model\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/scipy/stats/_multivariate.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, mean, cov, allow_singular, seed)\u001b[0m\n\u001b[1;32m    361\u001b[0m         return multivariate_normal_frozen(mean, cov,\n\u001b[1;32m    362\u001b[0m                                           \u001b[0mallow_singular\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mallow_singular\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 363\u001b[0;31m                                           seed=seed)\n\u001b[0m\u001b[1;32m    364\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    365\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_process_parameters\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdim\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmean\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcov\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/scipy/stats/_multivariate.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, mean, cov, allow_singular, seed, maxpts, abseps, releps)\u001b[0m\n\u001b[1;32m    734\u001b[0m         self.dim, self.mean, self.cov = self._dist._process_parameters(\n\u001b[1;32m    735\u001b[0m                                                             None, mean, cov)\n\u001b[0;32m--> 736\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcov_info\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_PSD\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcov\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mallow_singular\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mallow_singular\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    737\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mmaxpts\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    738\u001b[0m             \u001b[0mmaxpts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m1000000\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdim\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/scipy/stats/_multivariate.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, M, cond, rcond, lower, check_finite, allow_singular)\u001b[0m\n\u001b[1;32m    154\u001b[0m         \u001b[0;31m# Note that eigh takes care of array conversion, chkfinite,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    155\u001b[0m         \u001b[0;31m# and assertion that the matrix is square.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 156\u001b[0;31m         \u001b[0ms\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mu\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mscipy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlinalg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0meigh\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mM\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlower\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mlower\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcheck_finite\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcheck_finite\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    157\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    158\u001b[0m         \u001b[0meps\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_eigvalsh_to_eps\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcond\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrcond\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/scipy/linalg/decomp.py\u001b[0m in \u001b[0;36meigh\u001b[0;34m(a, b, lower, eigvals_only, overwrite_a, overwrite_b, turbo, eigvals, type, check_finite)\u001b[0m\n\u001b[1;32m    430\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0meigvals\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    431\u001b[0m             w, v, info = evr(a1, uplo=uplo, jobz=_job, range=\"A\", il=1,\n\u001b[0;32m--> 432\u001b[0;31m                              iu=a1.shape[0], overwrite_a=overwrite_a)\n\u001b[0m\u001b[1;32m    433\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    434\u001b[0m             \u001b[0;34m(\u001b[0m\u001b[0mlo\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhi\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0meigvals\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Label-wise Test (Much Faster)"
      ],
      "metadata": {
        "id": "mCEyJjpYsTUI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "label_embeddings = []\n",
        "progress = 0\n",
        "TRAIN_SIZE = len(labels)\n",
        "for label in labels:\n",
        "    label_embeddings.append(get_embeddings(label))\n",
        "    progress += 1\n",
        "    if progress % 50 == 0:\n",
        "        print('Progress Percent = {}%'.format(100 * progress / TRAIN_SIZE))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W7d240KlryjR",
        "outputId": "2b557506-ba59-4f23-d07b-9b45ca96c17d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Progress Percent = 10.245901639344263%\n",
            "Progress Percent = 20.491803278688526%\n",
            "Progress Percent = 30.737704918032787%\n",
            "Progress Percent = 40.98360655737705%\n",
            "Progress Percent = 51.22950819672131%\n",
            "Progress Percent = 61.47540983606557%\n",
            "Progress Percent = 71.72131147540983%\n",
            "Progress Percent = 81.9672131147541%\n",
            "Progress Percent = 92.21311475409836%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def predict_labelwise_doc_at(query, k=1):\n",
        "    \"\"\"\n",
        "    Predict which document is matched to the given query.\n",
        "\n",
        "    :param query: input query\n",
        "    :type query: str (or list of strs)\n",
        "    :param k: number of returning docs\n",
        "    :type k: int \n",
        "    :return: return the document name\n",
        "    \"\"\"\n",
        "    query_embedding = get_embeddings(query)\n",
        "    similarities = list(map(lambda x: np.dot(x, query_embedding) /\n",
        "                            (np.linalg.norm(query_embedding) * np.linalg.norm(x)),\n",
        "                            label_embeddings))\n",
        "    similarities = np.array(similarities)\n",
        "    best_k_idx = similarities.argsort()[::-1][:k]\n",
        "    predictions = list(map(lambda x: labels[x], best_k_idx))\n",
        "    accuracy = similarities[best_k_idx]\n",
        "    return accuracy, predictions"
      ],
      "metadata": {
        "id": "lPaLWGnii2oJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "for query in test_queries:\n",
        "    accs, preds = predict_labelwise_doc_at(query, k=5)\n",
        "    print(accs)\n",
        "    print(preds)\n",
        "    print('-' * 20)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pks9lYdatYBZ",
        "outputId": "658453ca-e96c-4df2-9a40-eebd8616977c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[0.4170898  0.4135436  0.41314107 0.41062284 0.40333804]\n",
            "['Benefits Planner: Disability | How You Apply | Social Security Administration#2_0', 'Benefits Planner: Disability | Are You Working | Social Security Administration#2_0', 'Benefits Planner: Disability | How You Apply | Social Security Administration#1_0', 'Benefits Planner: Disability | Are You Working | Social Security Administration#1_0', 'Contracting Information | Federal Student Aid#1_0']\n",
            "--------------------\n",
            "[0.38753018 0.3701287  0.36652112 0.35735554 0.3544231 ]\n",
            "['VA Education Benefits For Survivors And Dependents | Veterans Affairs#1_0', 'Benefits Planner: Survivors | Planning For Your Survivors | Social Security Administration#2_0', 'Benefits Planner: Survivors | Planning For Your Survivors | Social Security Administration#1_0', 'Benefits Planner: Survivors | If You Are The Survivor | Social Security Administration#2_0', 'Benefits Planner: Survivors | If You Are The Survivor | Social Security Administration#1_0']\n",
            "--------------------\n",
            "[0.49750158 0.49483535 0.49121538 0.48960638 0.48805937]\n",
            "['Learn what documents you will need to get a Social Security Card | Social Security Administration#6_0_1_2_3', 'Learn what documents you will need to get a Social Security Card | Social Security Administration#7_0_1_2', 'Learn what documents you will need to get a Social Security Card | Social Security Administration#6_0_1_2_3_4', 'Learn what documents you will need to get a Social Security Card | Social Security Administration#6_0_1_2', 'Supplemental Security Income (SSI) Benefits | Social Security Administration#1_0']\n",
            "--------------------\n",
            "[0.33226582 0.32709485 0.31967327 0.3110455  0.3053561 ]\n",
            "[\"VA Individual Unemployability If You Can't Work | Veterans Affairs#1_0\", 'What To Expect After You Get A Disability Rating | Veterans Affairs#1_0', 'About VA Disability Ratings | Veterans Affairs#1_0', 'Benefits For Spouses And Dependents (VA DIC) | Veterans Affairs#1_0', 'Benefits Planner: Retirement | If You Are Divorced | Social Security Administration#1_0']\n",
            "--------------------\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "prec_at_500 = 0\n",
        "prec_at_100 = 0\n",
        "prec_at_50 = 0\n",
        "prec_at_10 = 0\n",
        "prec_at_5 = 0\n",
        "prec_at_1 = 0\n",
        "sample_till_now = 0\n",
        "for query, act_doc in zip(doc_sentence_test[:TEST_SIZE],\n",
        "                          doc_label_test[:TEST_SIZE]):\n",
        "    accs, preds = predict_labelwise_doc_at(query, k=500)\n",
        "    # print(accs)\n",
        "    # print(preds)\n",
        "    # print(act_doc)\n",
        "    # print('-' * 20)\n",
        "    if act_doc == preds[0]:\n",
        "        prec_at_1 += 1\n",
        "    if act_doc in preds[:5]:\n",
        "        prec_at_5 += 1\n",
        "    if act_doc in preds[:10]:\n",
        "        prec_at_10 += 1\n",
        "    if act_doc in preds[:50]:\n",
        "        prec_at_50 += 1\n",
        "    if act_doc in preds[:100]:\n",
        "        prec_at_100 += 1\n",
        "    if act_doc in preds[:500]:\n",
        "        prec_at_500 += 1\n",
        "    sample_till_now += 1\n",
        "    if sample_till_now % 100 == 0:\n",
        "        print(\"Prec@(1) = {} | Prec@(5) = {} | Prec@(10) = {} | Prec@(50) = {} | Prec@(100) = {} | Prec@(500) = {} | NUMBER_OF_SAMPLES = {}\".\\\n",
        "              format(prec_at_1 / sample_till_now, prec_at_5 / sample_till_now,\n",
        "                     prec_at_10 / sample_till_now, prec_at_50 / sample_till_now,\n",
        "                     prec_at_100 / sample_till_now, prec_at_500 / sample_till_now,\n",
        "                     sample_till_now))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 628
        },
        "id": "c-1fAIxetc1K",
        "outputId": "1a898130-ac20-4191-8691-b6a5658c4a51"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Prec@(1) = 0.05 | Prec@(5) = 0.17 | Prec@(10) = 0.31 | Prec@(50) = 0.47 | Prec@(100) = 0.58 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 100\n",
            "Prec@(1) = 0.09 | Prec@(5) = 0.19 | Prec@(10) = 0.295 | Prec@(50) = 0.49 | Prec@(100) = 0.64 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 200\n",
            "Prec@(1) = 0.10666666666666667 | Prec@(5) = 0.22333333333333333 | Prec@(10) = 0.32666666666666666 | Prec@(50) = 0.49333333333333335 | Prec@(100) = 0.66 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 300\n",
            "Prec@(1) = 0.1075 | Prec@(5) = 0.225 | Prec@(10) = 0.3375 | Prec@(50) = 0.53 | Prec@(100) = 0.6875 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 400\n",
            "Prec@(1) = 0.124 | Prec@(5) = 0.244 | Prec@(10) = 0.35 | Prec@(50) = 0.556 | Prec@(100) = 0.706 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 500\n",
            "Prec@(1) = 0.12333333333333334 | Prec@(5) = 0.24 | Prec@(10) = 0.33666666666666667 | Prec@(50) = 0.5416666666666666 | Prec@(100) = 0.6916666666666667 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 600\n",
            "Prec@(1) = 0.12571428571428572 | Prec@(5) = 0.23 | Prec@(10) = 0.32 | Prec@(50) = 0.54 | Prec@(100) = 0.6828571428571428 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 700\n",
            "Prec@(1) = 0.14125 | Prec@(5) = 0.2425 | Prec@(10) = 0.32875 | Prec@(50) = 0.5425 | Prec@(100) = 0.685 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 800\n",
            "Prec@(1) = 0.13111111111111112 | Prec@(5) = 0.2411111111111111 | Prec@(10) = 0.32666666666666666 | Prec@(50) = 0.5466666666666666 | Prec@(100) = 0.6922222222222222 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 900\n",
            "Prec@(1) = 0.131 | Prec@(5) = 0.251 | Prec@(10) = 0.336 | Prec@(50) = 0.554 | Prec@(100) = 0.695 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 1000\n",
            "Prec@(1) = 0.13545454545454547 | Prec@(5) = 0.25727272727272726 | Prec@(10) = 0.3409090909090909 | Prec@(50) = 0.5554545454545454 | Prec@(100) = 0.7009090909090909 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 1100\n",
            "Prec@(1) = 0.1375 | Prec@(5) = 0.255 | Prec@(10) = 0.335 | Prec@(50) = 0.5408333333333334 | Prec@(100) = 0.6908333333333333 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 1200\n",
            "Prec@(1) = 0.13615384615384615 | Prec@(5) = 0.2592307692307692 | Prec@(10) = 0.33692307692307694 | Prec@(50) = 0.5376923076923077 | Prec@(100) = 0.693076923076923 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 1300\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-31-24baf366ed62>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      8\u001b[0m for query, act_doc in zip(doc_sentence_test[:TEST_SIZE],\n\u001b[1;32m      9\u001b[0m                           doc_label_test[:TEST_SIZE]):\n\u001b[0;32m---> 10\u001b[0;31m     \u001b[0maccs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpreds\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpredict_labelwise_doc_at\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mquery\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m500\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     11\u001b[0m     \u001b[0;31m# print(accs)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     12\u001b[0m     \u001b[0;31m# print(preds)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-27-e8763278e8db>\u001b[0m in \u001b[0;36mpredict_labelwise_doc_at\u001b[0;34m(query, k)\u001b[0m\n\u001b[1;32m      9\u001b[0m     \u001b[0;34m:\u001b[0m\u001b[0;32mreturn\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mdocument\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m     \"\"\"\n\u001b[0;32m---> 11\u001b[0;31m     \u001b[0mquery_embedding\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_embeddings\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mquery\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     12\u001b[0m     similarities = list(map(lambda x: np.dot(x, query_embedding) /\n\u001b[1;32m     13\u001b[0m                             \u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlinalg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnorm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mquery_embedding\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlinalg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnorm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-12-93659ddcbeb4>\u001b[0m in \u001b[0;36mget_embeddings\u001b[0;34m(sentece)\u001b[0m\n\u001b[1;32m     11\u001b[0m                                 padding=True)\n\u001b[1;32m     12\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mno_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 13\u001b[0;31m         \u001b[0membeddings\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel_labse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mtokenized\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     14\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqueeze\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0membeddings\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpooler_output\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)\u001b[0m\n\u001b[1;32m   1004\u001b[0m             \u001b[0moutput_attentions\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moutput_attentions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1005\u001b[0m             \u001b[0moutput_hidden_states\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moutput_hidden_states\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1006\u001b[0;31m             \u001b[0mreturn_dict\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mreturn_dict\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1007\u001b[0m         )\n\u001b[1;32m   1008\u001b[0m         \u001b[0msequence_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mencoder_outputs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)\u001b[0m\n\u001b[1;32m    590\u001b[0m                     \u001b[0mencoder_attention_mask\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    591\u001b[0m                     \u001b[0mpast_key_value\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 592\u001b[0;31m                     \u001b[0moutput_attentions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    593\u001b[0m                 )\n\u001b[1;32m    594\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions)\u001b[0m\n\u001b[1;32m    475\u001b[0m             \u001b[0mhead_mask\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    476\u001b[0m             \u001b[0moutput_attentions\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moutput_attentions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 477\u001b[0;31m             \u001b[0mpast_key_value\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself_attn_past_key_value\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    478\u001b[0m         )\n\u001b[1;32m    479\u001b[0m         \u001b[0mattention_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself_attention_outputs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions)\u001b[0m\n\u001b[1;32m    407\u001b[0m             \u001b[0mencoder_attention_mask\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    408\u001b[0m             \u001b[0mpast_key_value\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 409\u001b[0;31m             \u001b[0moutput_attentions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    410\u001b[0m         )\n\u001b[1;32m    411\u001b[0m         \u001b[0mattention_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutput\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself_outputs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhidden_states\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions)\u001b[0m\n\u001b[1;32m    288\u001b[0m             \u001b[0mvalue_layer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpast_key_value\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue_layer\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdim\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    289\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 290\u001b[0;31m             \u001b[0mkey_layer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtranspose_for_scores\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhidden_states\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    291\u001b[0m             \u001b[0mvalue_layer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtranspose_for_scores\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhidden_states\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    292\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/linear.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m    101\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    102\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 103\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mF\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlinear\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    104\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    105\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mextra_repr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py\u001b[0m in \u001b[0;36mlinear\u001b[0;34m(input, weight, bias)\u001b[0m\n\u001b[1;32m   1846\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhas_torch_function_variadic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1847\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mhandle_torch_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlinear\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbias\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1848\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_C\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_nn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlinear\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1849\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1850\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### History (last three sentences)"
      ],
      "metadata": {
        "id": "mxu4jo39ilSB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "prec_at_500 = 0\n",
        "prec_at_100 = 0\n",
        "prec_at_50 = 0\n",
        "prec_at_10 = 0\n",
        "prec_at_5 = 0\n",
        "prec_at_1 = 0\n",
        "sample_till_now = 0\n",
        "for i in range(2, TEST_SIZE):\n",
        "    query = '.'.join(doc_sentence_test[i-2:i+1])\n",
        "    act_doc = doc_label_test[i]\n",
        "    accs, preds = predict_labelwise_doc_at(query, k=500)\n",
        "    # print(accs)\n",
        "    # print(preds)\n",
        "    # print(act_doc)\n",
        "    # print('-' * 20)\n",
        "    if act_doc == preds[0]:\n",
        "        prec_at_1 += 1\n",
        "    if act_doc in preds[:5]:\n",
        "        prec_at_5 += 1\n",
        "    if act_doc in preds[:10]:\n",
        "        prec_at_10 += 1\n",
        "    if act_doc in preds[:50]:\n",
        "        prec_at_50 += 1\n",
        "    if act_doc in preds[:100]:\n",
        "        prec_at_100 += 1\n",
        "    if act_doc in preds[:500]:\n",
        "        prec_at_500 += 1\n",
        "    sample_till_now += 1\n",
        "    if sample_till_now % 100 == 0:\n",
        "        print(\"Prec@(1) = {} | Prec@(5) = {} | Prec@(10) = {} | Prec@(50) = {} | Prec@(100) = {} | Prec@(500) = {} | NUMBER_OF_SAMPLES = {}\".\\\n",
        "              format(prec_at_1 / sample_till_now, prec_at_5 / sample_till_now,\n",
        "                     prec_at_10 / sample_till_now, prec_at_50 / sample_till_now,\n",
        "                     prec_at_100 / sample_till_now, prec_at_500 / sample_till_now,\n",
        "                     sample_till_now))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 559
        },
        "id": "TesiuHr8ipne",
        "outputId": "3fe992f4-2c06-4579-b844-ecb4ff4535e5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Prec@(1) = 0.05 | Prec@(5) = 0.13 | Prec@(10) = 0.22 | Prec@(50) = 0.46 | Prec@(100) = 0.7 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 100\n",
            "Prec@(1) = 0.085 | Prec@(5) = 0.15 | Prec@(10) = 0.215 | Prec@(50) = 0.49 | Prec@(100) = 0.66 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 200\n",
            "Prec@(1) = 0.09666666666666666 | Prec@(5) = 0.17 | Prec@(10) = 0.25333333333333335 | Prec@(50) = 0.5033333333333333 | Prec@(100) = 0.6833333333333333 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 300\n",
            "Prec@(1) = 0.1075 | Prec@(5) = 0.22 | Prec@(10) = 0.3 | Prec@(50) = 0.55 | Prec@(100) = 0.7325 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 400\n",
            "Prec@(1) = 0.094 | Prec@(5) = 0.242 | Prec@(10) = 0.316 | Prec@(50) = 0.56 | Prec@(100) = 0.742 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 500\n",
            "Prec@(1) = 0.08666666666666667 | Prec@(5) = 0.23333333333333334 | Prec@(10) = 0.3 | Prec@(50) = 0.5366666666666666 | Prec@(100) = 0.73 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 600\n",
            "Prec@(1) = 0.09571428571428571 | Prec@(5) = 0.2357142857142857 | Prec@(10) = 0.3028571428571429 | Prec@(50) = 0.5228571428571429 | Prec@(100) = 0.7114285714285714 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 700\n",
            "Prec@(1) = 0.10625 | Prec@(5) = 0.2525 | Prec@(10) = 0.32125 | Prec@(50) = 0.54125 | Prec@(100) = 0.73 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 800\n",
            "Prec@(1) = 0.09666666666666666 | Prec@(5) = 0.23666666666666666 | Prec@(10) = 0.3111111111111111 | Prec@(50) = 0.5288888888888889 | Prec@(100) = 0.7333333333333333 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 900\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-32-32c107264e32>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      9\u001b[0m     \u001b[0mquery\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'.'\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdoc_sentence_test\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m     \u001b[0mact_doc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdoc_label_test\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m     \u001b[0maccs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpreds\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpredict_labelwise_doc_at\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mquery\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m500\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     12\u001b[0m     \u001b[0;31m# print(accs)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m     \u001b[0;31m# print(preds)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-27-e8763278e8db>\u001b[0m in \u001b[0;36mpredict_labelwise_doc_at\u001b[0;34m(query, k)\u001b[0m\n\u001b[1;32m      9\u001b[0m     \u001b[0;34m:\u001b[0m\u001b[0;32mreturn\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mdocument\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m     \"\"\"\n\u001b[0;32m---> 11\u001b[0;31m     \u001b[0mquery_embedding\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_embeddings\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mquery\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     12\u001b[0m     similarities = list(map(lambda x: np.dot(x, query_embedding) /\n\u001b[1;32m     13\u001b[0m                             \u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlinalg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnorm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mquery_embedding\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlinalg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnorm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-12-93659ddcbeb4>\u001b[0m in \u001b[0;36mget_embeddings\u001b[0;34m(sentece)\u001b[0m\n\u001b[1;32m     11\u001b[0m                                 padding=True)\n\u001b[1;32m     12\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mno_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 13\u001b[0;31m         \u001b[0membeddings\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel_labse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mtokenized\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     14\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqueeze\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0membeddings\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpooler_output\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)\u001b[0m\n\u001b[1;32m   1004\u001b[0m             \u001b[0moutput_attentions\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moutput_attentions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1005\u001b[0m             \u001b[0moutput_hidden_states\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moutput_hidden_states\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1006\u001b[0;31m             \u001b[0mreturn_dict\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mreturn_dict\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1007\u001b[0m         )\n\u001b[1;32m   1008\u001b[0m         \u001b[0msequence_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mencoder_outputs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)\u001b[0m\n\u001b[1;32m    590\u001b[0m                     \u001b[0mencoder_attention_mask\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    591\u001b[0m                     \u001b[0mpast_key_value\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 592\u001b[0;31m                     \u001b[0moutput_attentions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    593\u001b[0m                 )\n\u001b[1;32m    594\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions)\u001b[0m\n\u001b[1;32m    475\u001b[0m             \u001b[0mhead_mask\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    476\u001b[0m             \u001b[0moutput_attentions\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moutput_attentions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 477\u001b[0;31m             \u001b[0mpast_key_value\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself_attn_past_key_value\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    478\u001b[0m         )\n\u001b[1;32m    479\u001b[0m         \u001b[0mattention_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself_attention_outputs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions)\u001b[0m\n\u001b[1;32m    407\u001b[0m             \u001b[0mencoder_attention_mask\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    408\u001b[0m             \u001b[0mpast_key_value\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 409\u001b[0;31m             \u001b[0moutput_attentions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    410\u001b[0m         )\n\u001b[1;32m    411\u001b[0m         \u001b[0mattention_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutput\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself_outputs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhidden_states\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions)\u001b[0m\n\u001b[1;32m    289\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    290\u001b[0m             \u001b[0mkey_layer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtranspose_for_scores\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhidden_states\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 291\u001b[0;31m             \u001b[0mvalue_layer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtranspose_for_scores\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhidden_states\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    292\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    293\u001b[0m         \u001b[0mquery_layer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtranspose_for_scores\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmixed_query_layer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/linear.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m    101\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    102\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 103\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mF\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlinear\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    104\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    105\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mextra_repr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py\u001b[0m in \u001b[0;36mlinear\u001b[0;34m(input, weight, bias)\u001b[0m\n\u001b[1;32m   1846\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhas_torch_function_variadic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1847\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mhandle_torch_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlinear\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbias\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1848\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_C\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_nn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlinear\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1849\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1850\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Label-wise + TF-IDF"
      ],
      "metadata": {
        "id": "chQpG2n03jd9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "doc_texts_train = []\n",
        "for doc_idx1 in multidoc2dial_doc['doc_data']:\n",
        "    for doc_idx2 in multidoc2dial_doc['doc_data'][doc_idx1]:\n",
        "        doc_texts_train.append(multidoc2dial_doc['doc_data'][doc_idx1]\\\n",
        "                                          [doc_idx2]['doc_text'].strip())"
      ],
      "metadata": {
        "id": "IE679JD_53xx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "doc_texts_train[0]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 126
        },
        "id": "HIKSnjyt6PAx",
        "outputId": "d917a720-dded-4606-875a-fe26866a5a67"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\"Benefits Planner: Survivors | Planning For Your Survivors \\nAs you plan for the future , you'll want to think about what your family would need if you should die now. Social Security can help your family if you have earned enough Social Security credits through your work. You can earn up to four credits each year. In 2019 , for example , you earn one credit for each $1,360 of wages or self - employment income. When you have earned $5,440 , you have earned your four credits for the year. The number of credits needed to provide benefits for your survivors depends on your age when you die. No one needs more than 40 credits 10 years of work to be eligible for any Social Security benefit. But , the younger a person is , the fewer credits they must have for family members to receive survivors benefits. Benefits can be paid to your children and your spouse who is caring for the children even if you don't have the required number of credits. They can get benefits if you have credit for one and one - half years of work 6 credits in the three years just before your death. \\n\\nFor Your Widow Or Widower \\nThere are about five million widows and widowers receiving monthly Social Security benefits based on their deceased spouse's earnings record. And , for many of those survivors, particularly aged women, those benefits are keeping them out of poverty. Widows and widowers can receive : reduced benefits as early as age 60 or full benefits at full retirement age or older. benefits as early as age 50 if they're disabled AND their disability started before or within seven years of your death. benefits at any age , if they have not remarried , and if they take care of your child who is under age 16 or disabled and receives benefits on your record. If applying for disability benefits on a deceased worker s record , they can speed up the application process if they complete an Adult Disability Report and have it available at the time of their appointment. We use the same definition of disability for widows and widowers as we do for workers. \\n\\nFor Your Surviving Divorced Spouse \\nIf you have a surviving divorced spouse , they could get the same benefits as your widow or widower provided that your marriage lasted 10 years or more. Benefits paid to a surviving divorced spouse won't affect the benefit amounts your other survivors will receive based on your earnings record. If your former spouse is caring for your child who is under age 16 or disabled and gets benefits on your record , they will not have to meet the length - of - marriage rule. The child must be your natural or legally adopted child. \\n\\nFor Your Children \\nYour unmarried children who are under 18 up to age 19 if attending elementary or secondary school full time can be eligible to receive Social Security benefits when you die. And your child can get benefits at any age if they were disabled before age 22 and remain disabled. Besides your natural children , your stepchildren, grandchildren, step grandchildren or adopted children may receive benefits under certain circumstances. For further information , view our publication. \\n\\nFor Your Parents \\nYou must have been providing at least half of your parent s support and your parent must not be eligible to receive a retirement benefit that is higher than the benefit we could pay on your record. Generally, your parent also must not have married after your death ; however, there are some exceptions. In addition to your natural parent , your stepparent or adoptive parent may receive benefits if they became your parent before you were age 16. \\n\\nHow Much Would Your Survivors Receive \\nHow much your family could receive in benefits depends on your average lifetime earnings. The higher your earnings were , the higher their benefits would be. We calculate a basic amount as if you had reached full retirement age at the time you die. These are examples of monthly benefit payments : Widow or widower, full retirement age or older 100 percent of your benefit amount ; Widow or widower , age 60 to full retirement age 71 to 99 percent of your basic amount ; Disabled widow or widower , age 50 through 59 71 percent ; Widow or widower , any age, caring for a child under age 16 75 percent ; A child under age 18 19 if still in elementary or secondary school or disabled 75 percent ; and Your dependent parent , age 62 or older : One surviving parent 82 percent. Two surviving parents 75 percent to each parent. Percentages for a surviving divorced spouse would be the same as above. There may also be a special lump - sum death payment. \\n\\nMaximum Family Amount \\nThere's a limit to the amount that family members can receive each month. The limit varies , but it is generally equal to between 150 and 180 percent of the basic benefit rate. If the sum of the benefits payable to family members is greater than this limit , the benefits will be reduced proportionately. Any benefits paid to a surviving divorced spouse based on disability or age won't count toward this maximum amount. Get your online or check our Benefit Calculators for an estimate of the benefits your family could receive if you died right now. \\n\\nOther Things You Need To Know \\nThere are limits on how much survivors may earn while they receive benefits. Benefits for a widow, widower, or surviving divorced spouse may be affected by several additional factors : If your widow, widower, or surviving divorced spouse remarries before they reach age 60 age 50 if disabled , they cannot receive benefits as a surviving spouse while they're married. If your widow, widower, or surviving divorced spouse remarries after they reach age 60 age 50 if disabled , they will continue to qualify for benefits on your Social Security record. However , if their current spouse is a Social Security beneficiary , they may want to apply for spouse's benefits on their record. If that amount is more than the widow's or widower's benefit on your record , they will receive a combination of benefits that equals the higher amount. If your widow, widower, or surviving divorced spouse receives benefits on your record , they can switch to their own retirement benefit as early as age 62. This assumes they're eligible for retirement benefits and their retirement rate is higher than their rate as a widow, widower, or surviving divorced spouse. In many cases , a widow or widower can begin receiving one benefit at a reduced rate and then, at full retirement age, switch to the other benefit at an unreduced rate. If your widow, widower, or surviving divorced spouse will also receive a pension based on work not covered by Social Security, such as government or foreign work , their Social Security benefits as a survivor may be affected.\""
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "words = set()\n",
        "doc_texts_train_tokenized = []\n",
        "for doc in doc_texts_train:\n",
        "    tokenized_doc = [s.lower() for s in tokenizer_labse.tokenize(doc)]\n",
        "    doc_texts_train_tokenized.append(tokenized_doc) \n",
        "    words = set(tokenized_doc).union(words)\n",
        "len(words)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LgKcGxx9pqVg",
        "outputId": "50da321b-d39d-4d0d-ee8e-3b78bc9218c7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "8446"
            ]
          },
          "metadata": {},
          "execution_count": 55
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "words2IDF = {}\n",
        "N_doc = len(doc_texts_train)\n",
        "for i, word in enumerate(words):\n",
        "    n_word = 0\n",
        "    for doc in doc_texts_train_tokenized:\n",
        "        if word in doc:\n",
        "            n_word += 1\n",
        "    words2IDF[word] = np.log(N_doc / (n_word + 1))\n",
        "    if i % 1000 == 0:\n",
        "        print(word, words2IDF[word])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TpF6K8girTmN",
        "outputId": "33c2fd13-6bfe-4d59-97d4-5f4561b30720"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "nature 3.792420133054777\n",
            "nobody 5.0917031171850375\n",
            "potentially 4.804021044733257\n",
            "examination 3.8877303128591016\n",
            "##wal 3.5512580762378887\n",
            "appear 2.5793974932089228\n",
            "participate 3.245876426686707\n",
            "##mig 3.6253660483916104\n",
            "go 1.1730355690382228\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def calc_idf_score(sentence):\n",
        "    tokenzied_sentence = [s.lower() for s in tokenizer_labse.tokenize(sentence)]\n",
        "    score = 0\n",
        "    for token in tokenzied_sentence:\n",
        "        if token in words2IDF:\n",
        "            score += words2IDF[token]\n",
        "        else:\n",
        "            score += np.log(N_doc)\n",
        "    return score / len(tokenzied_sentence)"
      ],
      "metadata": {
        "id": "LnrVBcdQujE4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def predict_labelwise_doc_at_history(query_h2, query_h1, query_h0, k=1):\n",
        "    \"\"\"\n",
        "    Predict which document is matched to the given query.\n",
        "\n",
        "    :param query: input query\n",
        "    :type query: str (or list of strs)\n",
        "    :param k: number of returning docs\n",
        "    :type k: int \n",
        "    :return: return the document name\n",
        "    \"\"\"\n",
        "    query_h2_embedding = get_embeddings(query_h2)\n",
        "    similarities2 = list(map(lambda x: np.dot(x, query_h2_embedding) /\n",
        "                            (np.linalg.norm(query_h2_embedding) * np.linalg.norm(x)),\n",
        "                            label_embeddings))\n",
        "    similarities2 = np.array(similarities2)\n",
        "    idf_score2 = calc_idf_score(query_h2)\n",
        "\n",
        "    query_h1_embedding = get_embeddings(query_h1)\n",
        "    similarities1 = list(map(lambda x: np.dot(x, query_h1_embedding) /\n",
        "                            (np.linalg.norm(query_h1_embedding) * np.linalg.norm(x)),\n",
        "                            label_embeddings))\n",
        "    similarities1 = np.array(similarities1)\n",
        "    idf_score1 = calc_idf_score(query_h1)\n",
        "\n",
        "    query_h0_embedding = get_embeddings(query_h0)\n",
        "    similarities0 = list(map(lambda x: np.dot(x, query_h0_embedding) /\n",
        "                            (np.linalg.norm(query_h0_embedding) * np.linalg.norm(x)),\n",
        "                            label_embeddings))\n",
        "    similarities0 = np.array(similarities0)\n",
        "    idf_score0 = calc_idf_score(query_h0)\n",
        "\n",
        "    similarities = (idf_score0 * similarities0 + \\\n",
        "                    idf_score1 * similarities1 + \\\n",
        "                    idf_score2 * similarities2) / \\\n",
        "                    (idf_score0 + idf_score1 + idf_score2)\n",
        "    best_k_idx = similarities.argsort()[::-1][:k]\n",
        "    predictions = list(map(lambda x: labels[x], best_k_idx))\n",
        "    accuracy = similarities[best_k_idx]\n",
        "    return accuracy, predictions"
      ],
      "metadata": {
        "id": "Li5Cw6c3vZe9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "prec_at_500 = 0\n",
        "prec_at_100 = 0\n",
        "prec_at_50 = 0\n",
        "prec_at_10 = 0\n",
        "prec_at_5 = 0\n",
        "prec_at_1 = 0\n",
        "sample_till_now = 0\n",
        "for i in range(2, TEST_SIZE):\n",
        "    act_doc = doc_label_test[i]\n",
        "    query_h2 = doc_sentence_test[i-2]\n",
        "    query_h1 = doc_sentence_test[i-1]\n",
        "    query_h0 = doc_sentence_test[i]\n",
        "    accs, preds = predict_labelwise_doc_at_history(query_h2,\n",
        "                                                   query_h1,\n",
        "                                                   query_h0,\n",
        "                                                   k=500)\n",
        "    # print(accs)\n",
        "    # print(preds)\n",
        "    # print(act_doc)\n",
        "    # print('-' * 20)\n",
        "    if act_doc == preds[0]:\n",
        "        prec_at_1 += 1\n",
        "    if act_doc in preds[:5]:\n",
        "        prec_at_5 += 1\n",
        "    if act_doc in preds[:10]:\n",
        "        prec_at_10 += 1\n",
        "    if act_doc in preds[:50]:\n",
        "        prec_at_50 += 1\n",
        "    if act_doc in preds[:100]:\n",
        "        prec_at_100 += 1\n",
        "    if act_doc in preds[:500]:\n",
        "        prec_at_500 += 1\n",
        "    sample_till_now += 1\n",
        "    if sample_till_now % 100 == 0:\n",
        "        print(\"Prec@(1) = {} | Prec@(5) = {} | Prec@(10) = {} | Prec@(50) = {} | Prec@(100) = {} | Prec@(500) = {} | NUMBER_OF_SAMPLES = {}\".\\\n",
        "              format(prec_at_1 / sample_till_now, prec_at_5 / sample_till_now,\n",
        "                     prec_at_10 / sample_till_now, prec_at_50 / sample_till_now,\n",
        "                     prec_at_100 / sample_till_now, prec_at_500 / sample_till_now,\n",
        "                     sample_till_now))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "fgPK-pPluDkh",
        "outputId": "b3cd4c1a-e169-41de-f8ce-bcbf6a850e8e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Prec@(1) = 0.06 | Prec@(5) = 0.15 | Prec@(10) = 0.23 | Prec@(50) = 0.47 | Prec@(100) = 0.69 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 100\n",
            "Prec@(1) = 0.045 | Prec@(5) = 0.155 | Prec@(10) = 0.25 | Prec@(50) = 0.5 | Prec@(100) = 0.715 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 200\n",
            "Prec@(1) = 0.06 | Prec@(5) = 0.19 | Prec@(10) = 0.29 | Prec@(50) = 0.5366666666666666 | Prec@(100) = 0.76 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 300\n",
            "Prec@(1) = 0.085 | Prec@(5) = 0.2325 | Prec@(10) = 0.335 | Prec@(50) = 0.59 | Prec@(100) = 0.795 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 400\n",
            "Prec@(1) = 0.108 | Prec@(5) = 0.266 | Prec@(10) = 0.364 | Prec@(50) = 0.622 | Prec@(100) = 0.816 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 500\n",
            "Prec@(1) = 0.105 | Prec@(5) = 0.25333333333333335 | Prec@(10) = 0.345 | Prec@(50) = 0.595 | Prec@(100) = 0.8083333333333333 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 600\n",
            "Prec@(1) = 0.11714285714285715 | Prec@(5) = 0.2542857142857143 | Prec@(10) = 0.3457142857142857 | Prec@(50) = 0.6 | Prec@(100) = 0.8142857142857143 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 700\n",
            "Prec@(1) = 0.14 | Prec@(5) = 0.2825 | Prec@(10) = 0.3675 | Prec@(50) = 0.615 | Prec@(100) = 0.82125 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 800\n",
            "Prec@(1) = 0.13111111111111112 | Prec@(5) = 0.2788888888888889 | Prec@(10) = 0.36333333333333334 | Prec@(50) = 0.62 | Prec@(100) = 0.8211111111111111 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 900\n",
            "Prec@(1) = 0.139 | Prec@(5) = 0.303 | Prec@(10) = 0.388 | Prec@(50) = 0.636 | Prec@(100) = 0.827 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 1000\n",
            "Prec@(1) = 0.13818181818181818 | Prec@(5) = 0.30454545454545456 | Prec@(10) = 0.39181818181818184 | Prec@(50) = 0.6454545454545455 | Prec@(100) = 0.8354545454545454 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 1100\n",
            "Prec@(1) = 0.14166666666666666 | Prec@(5) = 0.30416666666666664 | Prec@(10) = 0.3908333333333333 | Prec@(50) = 0.6375 | Prec@(100) = 0.8225 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 1200\n",
            "Prec@(1) = 0.14 | Prec@(5) = 0.30538461538461537 | Prec@(10) = 0.39076923076923076 | Prec@(50) = 0.6346153846153846 | Prec@(100) = 0.8153846153846154 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 1300\n",
            "Prec@(1) = 0.14 | Prec@(5) = 0.30714285714285716 | Prec@(10) = 0.3942857142857143 | Prec@(50) = 0.6328571428571429 | Prec@(100) = 0.8135714285714286 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 1400\n",
            "Prec@(1) = 0.13133333333333333 | Prec@(5) = 0.29133333333333333 | Prec@(10) = 0.37733333333333335 | Prec@(50) = 0.6126666666666667 | Prec@(100) = 0.802 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 1500\n",
            "Prec@(1) = 0.131875 | Prec@(5) = 0.29375 | Prec@(10) = 0.37875 | Prec@(50) = 0.613125 | Prec@(100) = 0.804375 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 1600\n",
            "Prec@(1) = 0.13941176470588235 | Prec@(5) = 0.3052941176470588 | Prec@(10) = 0.39176470588235296 | Prec@(50) = 0.62 | Prec@(100) = 0.8035294117647059 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 1700\n",
            "Prec@(1) = 0.14833333333333334 | Prec@(5) = 0.3194444444444444 | Prec@(10) = 0.4038888888888889 | Prec@(50) = 0.6344444444444445 | Prec@(100) = 0.8122222222222222 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 1800\n",
            "Prec@(1) = 0.1463157894736842 | Prec@(5) = 0.32105263157894737 | Prec@(10) = 0.4068421052631579 | Prec@(50) = 0.6363157894736842 | Prec@(100) = 0.8105263157894737 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 1900\n",
            "Prec@(1) = 0.144 | Prec@(5) = 0.317 | Prec@(10) = 0.4035 | Prec@(50) = 0.6365 | Prec@(100) = 0.8115 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 2000\n",
            "Prec@(1) = 0.14142857142857143 | Prec@(5) = 0.3119047619047619 | Prec@(10) = 0.4 | Prec@(50) = 0.6309523809523809 | Prec@(100) = 0.8042857142857143 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 2100\n",
            "Prec@(1) = 0.1440909090909091 | Prec@(5) = 0.3118181818181818 | Prec@(10) = 0.4022727272727273 | Prec@(50) = 0.6331818181818182 | Prec@(100) = 0.8045454545454546 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 2200\n",
            "Prec@(1) = 0.14043478260869566 | Prec@(5) = 0.30695652173913046 | Prec@(10) = 0.39608695652173914 | Prec@(50) = 0.6247826086956522 | Prec@(100) = 0.7917391304347826 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 2300\n",
            "Prec@(1) = 0.14 | Prec@(5) = 0.30875 | Prec@(10) = 0.39916666666666667 | Prec@(50) = 0.6241666666666666 | Prec@(100) = 0.7895833333333333 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 2400\n",
            "Prec@(1) = 0.1508 | Prec@(5) = 0.3208 | Prec@(10) = 0.4088 | Prec@(50) = 0.6324 | Prec@(100) = 0.7932 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 2500\n",
            "Prec@(1) = 0.15576923076923077 | Prec@(5) = 0.32269230769230767 | Prec@(10) = 0.41115384615384615 | Prec@(50) = 0.6361538461538462 | Prec@(100) = 0.7942307692307692 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 2600\n",
            "Prec@(1) = 0.1514814814814815 | Prec@(5) = 0.31666666666666665 | Prec@(10) = 0.4048148148148148 | Prec@(50) = 0.6292592592592593 | Prec@(100) = 0.7892592592592592 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 2700\n",
            "Prec@(1) = 0.15035714285714286 | Prec@(5) = 0.3182142857142857 | Prec@(10) = 0.4067857142857143 | Prec@(50) = 0.6353571428571428 | Prec@(100) = 0.7932142857142858 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 2800\n",
            "Prec@(1) = 0.15482758620689654 | Prec@(5) = 0.32275862068965516 | Prec@(10) = 0.4106896551724138 | Prec@(50) = 0.6427586206896552 | Prec@(100) = 0.7979310344827586 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 2900\n",
            "Prec@(1) = 0.15466666666666667 | Prec@(5) = 0.325 | Prec@(10) = 0.411 | Prec@(50) = 0.6416666666666667 | Prec@(100) = 0.795 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 3000\n",
            "Prec@(1) = 0.1529032258064516 | Prec@(5) = 0.3193548387096774 | Prec@(10) = 0.40419354838709676 | Prec@(50) = 0.6329032258064516 | Prec@(100) = 0.7864516129032258 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 3100\n",
            "Prec@(1) = 0.150625 | Prec@(5) = 0.313125 | Prec@(10) = 0.3959375 | Prec@(50) = 0.6259375 | Prec@(100) = 0.7828125 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 3200\n",
            "Prec@(1) = 0.15636363636363637 | Prec@(5) = 0.3184848484848485 | Prec@(10) = 0.4009090909090909 | Prec@(50) = 0.6296969696969696 | Prec@(100) = 0.7851515151515152 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 3300\n",
            "Prec@(1) = 0.1588235294117647 | Prec@(5) = 0.3197058823529412 | Prec@(10) = 0.40176470588235297 | Prec@(50) = 0.6294117647058823 | Prec@(100) = 0.785 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 3400\n",
            "Prec@(1) = 0.15942857142857142 | Prec@(5) = 0.32285714285714284 | Prec@(10) = 0.406 | Prec@(50) = 0.6328571428571429 | Prec@(100) = 0.7882857142857143 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 3500\n",
            "Prec@(1) = 0.15916666666666668 | Prec@(5) = 0.3252777777777778 | Prec@(10) = 0.4086111111111111 | Prec@(50) = 0.6347222222222222 | Prec@(100) = 0.79 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 3600\n",
            "Prec@(1) = 0.15972972972972974 | Prec@(5) = 0.3256756756756757 | Prec@(10) = 0.41027027027027024 | Prec@(50) = 0.635945945945946 | Prec@(100) = 0.7902702702702703 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 3700\n",
            "Prec@(1) = 0.16210526315789472 | Prec@(5) = 0.3294736842105263 | Prec@(10) = 0.4131578947368421 | Prec@(50) = 0.6386842105263157 | Prec@(100) = 0.791578947368421 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 3800\n",
            "Prec@(1) = 0.16256410256410256 | Prec@(5) = 0.33179487179487177 | Prec@(10) = 0.4153846153846154 | Prec@(50) = 0.642051282051282 | Prec@(100) = 0.7933333333333333 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 3900\n",
            "Prec@(1) = 0.16225 | Prec@(5) = 0.333 | Prec@(10) = 0.41675 | Prec@(50) = 0.643 | Prec@(100) = 0.795 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 4000\n",
            "Prec@(1) = 0.15902439024390244 | Prec@(5) = 0.3282926829268293 | Prec@(10) = 0.4119512195121951 | Prec@(50) = 0.6404878048780488 | Prec@(100) = 0.7919512195121952 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 4100\n",
            "Prec@(1) = 0.1580952380952381 | Prec@(5) = 0.32595238095238094 | Prec@(10) = 0.40976190476190477 | Prec@(50) = 0.6361904761904762 | Prec@(100) = 0.7885714285714286 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 4200\n",
            "Prec@(1) = 0.15813953488372093 | Prec@(5) = 0.3251162790697674 | Prec@(10) = 0.4095348837209302 | Prec@(50) = 0.6358139534883721 | Prec@(100) = 0.7888372093023256 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 4300\n",
            "Prec@(1) = 0.15977272727272726 | Prec@(5) = 0.3265909090909091 | Prec@(10) = 0.4102272727272727 | Prec@(50) = 0.6370454545454546 | Prec@(100) = 0.7897727272727273 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 4400\n",
            "Prec@(1) = 0.16422222222222221 | Prec@(5) = 0.33355555555555555 | Prec@(10) = 0.41733333333333333 | Prec@(50) = 0.6426666666666667 | Prec@(100) = 0.7935555555555556 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 4500\n",
            "Prec@(1) = 0.1676086956521739 | Prec@(5) = 0.336304347826087 | Prec@(10) = 0.41956521739130437 | Prec@(50) = 0.6419565217391304 | Prec@(100) = 0.792608695652174 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 4600\n",
            "Prec@(1) = 0.16595744680851063 | Prec@(5) = 0.33382978723404255 | Prec@(10) = 0.41638297872340424 | Prec@(50) = 0.6406382978723404 | Prec@(100) = 0.7931914893617021 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 4700\n",
            "Prec@(1) = 0.16354166666666667 | Prec@(5) = 0.3329166666666667 | Prec@(10) = 0.414375 | Prec@(50) = 0.6404166666666666 | Prec@(100) = 0.7939583333333333 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 4800\n",
            "Prec@(1) = 0.16285714285714287 | Prec@(5) = 0.3320408163265306 | Prec@(10) = 0.413265306122449 | Prec@(50) = 0.6414285714285715 | Prec@(100) = 0.7953061224489796 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 4900\n",
            "Prec@(1) = 0.163 | Prec@(5) = 0.3342 | Prec@(10) = 0.416 | Prec@(50) = 0.6454 | Prec@(100) = 0.798 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 5000\n",
            "Prec@(1) = 0.16156862745098038 | Prec@(5) = 0.3298039215686275 | Prec@(10) = 0.41019607843137257 | Prec@(50) = 0.6372549019607843 | Prec@(100) = 0.7903921568627451 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 5100\n",
            "Prec@(1) = 0.15846153846153846 | Prec@(5) = 0.32461538461538464 | Prec@(10) = 0.4040384615384615 | Prec@(50) = 0.6301923076923077 | Prec@(100) = 0.7828846153846154 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 5200\n",
            "Prec@(1) = 0.16226415094339622 | Prec@(5) = 0.3290566037735849 | Prec@(10) = 0.4088679245283019 | Prec@(50) = 0.6343396226415094 | Prec@(100) = 0.7858490566037736 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 5300\n",
            "Prec@(1) = 0.16333333333333333 | Prec@(5) = 0.32944444444444443 | Prec@(10) = 0.4090740740740741 | Prec@(50) = 0.6342592592592593 | Prec@(100) = 0.7861111111111111 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 5400\n",
            "Prec@(1) = 0.16472727272727272 | Prec@(5) = 0.3321818181818182 | Prec@(10) = 0.412 | Prec@(50) = 0.6367272727272727 | Prec@(100) = 0.7883636363636364 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 5500\n",
            "Prec@(1) = 0.16839285714285715 | Prec@(5) = 0.3357142857142857 | Prec@(10) = 0.41517857142857145 | Prec@(50) = 0.6407142857142857 | Prec@(100) = 0.7910714285714285 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 5600\n",
            "Prec@(1) = 0.1694736842105263 | Prec@(5) = 0.33614035087719296 | Prec@(10) = 0.4149122807017544 | Prec@(50) = 0.638421052631579 | Prec@(100) = 0.7889473684210526 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 5700\n",
            "Prec@(1) = 0.1703448275862069 | Prec@(5) = 0.3375862068965517 | Prec@(10) = 0.41620689655172416 | Prec@(50) = 0.64 | Prec@(100) = 0.7894827586206896 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 5800\n",
            "Prec@(1) = 0.17135593220338982 | Prec@(5) = 0.33864406779661016 | Prec@(10) = 0.41627118644067795 | Prec@(50) = 0.6393220338983051 | Prec@(100) = 0.7888135593220339 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 5900\n",
            "Prec@(1) = 0.17166666666666666 | Prec@(5) = 0.3431666666666667 | Prec@(10) = 0.42033333333333334 | Prec@(50) = 0.643 | Prec@(100) = 0.7916666666666666 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 6000\n",
            "Prec@(1) = 0.17295081967213113 | Prec@(5) = 0.3431147540983607 | Prec@(10) = 0.419344262295082 | Prec@(50) = 0.6409836065573771 | Prec@(100) = 0.79 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 6100\n",
            "Prec@(1) = 0.17419354838709677 | Prec@(5) = 0.34403225806451615 | Prec@(10) = 0.4208064516129032 | Prec@(50) = 0.6419354838709678 | Prec@(100) = 0.7916129032258065 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 6200\n",
            "Prec@(1) = 0.176984126984127 | Prec@(5) = 0.3466666666666667 | Prec@(10) = 0.4234920634920635 | Prec@(50) = 0.643968253968254 | Prec@(100) = 0.7930158730158731 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 6300\n",
            "Prec@(1) = 0.17828125 | Prec@(5) = 0.3484375 | Prec@(10) = 0.425625 | Prec@(50) = 0.64609375 | Prec@(100) = 0.79421875 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 6400\n",
            "Prec@(1) = 0.17876923076923076 | Prec@(5) = 0.348 | Prec@(10) = 0.4247692307692308 | Prec@(50) = 0.6456923076923077 | Prec@(100) = 0.7933846153846154 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 6500\n",
            "Prec@(1) = 0.17757575757575758 | Prec@(5) = 0.3462121212121212 | Prec@(10) = 0.4222727272727273 | Prec@(50) = 0.6427272727272727 | Prec@(100) = 0.7903030303030303 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 6600\n",
            "Prec@(1) = 0.17746268656716419 | Prec@(5) = 0.34776119402985073 | Prec@(10) = 0.4241791044776119 | Prec@(50) = 0.644179104477612 | Prec@(100) = 0.792089552238806 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 6700\n",
            "Prec@(1) = 0.175 | Prec@(5) = 0.34397058823529414 | Prec@(10) = 0.42 | Prec@(50) = 0.6407352941176471 | Prec@(100) = 0.7891176470588235 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 6800\n",
            "Prec@(1) = 0.17246376811594202 | Prec@(5) = 0.3389855072463768 | Prec@(10) = 0.4146376811594203 | Prec@(50) = 0.6385507246376811 | Prec@(100) = 0.7876811594202898 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 6900\n",
            "Prec@(1) = 0.17057142857142857 | Prec@(5) = 0.33585714285714285 | Prec@(10) = 0.41214285714285714 | Prec@(50) = 0.6405714285714286 | Prec@(100) = 0.7892857142857143 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 7000\n",
            "Prec@(1) = 0.16816901408450705 | Prec@(5) = 0.33154929577464787 | Prec@(10) = 0.4074647887323944 | Prec@(50) = 0.6388732394366197 | Prec@(100) = 0.787887323943662 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 7100\n",
            "Prec@(1) = 0.1663888888888889 | Prec@(5) = 0.32916666666666666 | Prec@(10) = 0.40555555555555556 | Prec@(50) = 0.6411111111111111 | Prec@(100) = 0.7890277777777778 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 7200\n",
            "Prec@(1) = 0.1643835616438356 | Prec@(5) = 0.3254794520547945 | Prec@(10) = 0.40150684931506847 | Prec@(50) = 0.64 | Prec@(100) = 0.7882191780821918 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 7300\n",
            "Prec@(1) = 0.16243243243243244 | Prec@(5) = 0.3222972972972973 | Prec@(10) = 0.3994594594594595 | Prec@(50) = 0.6436486486486487 | Prec@(100) = 0.7908108108108108 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 7400\n",
            "Prec@(1) = 0.16026666666666667 | Prec@(5) = 0.3188 | Prec@(10) = 0.3964 | Prec@(50) = 0.6464 | Prec@(100) = 0.7929333333333334 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 7500\n",
            "Prec@(1) = 0.15842105263157893 | Prec@(5) = 0.31723684210526315 | Prec@(10) = 0.3964473684210526 | Prec@(50) = 0.65 | Prec@(100) = 0.7952631578947369 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 7600\n",
            "Prec@(1) = 0.15688311688311687 | Prec@(5) = 0.31545454545454543 | Prec@(10) = 0.395974025974026 | Prec@(50) = 0.652987012987013 | Prec@(100) = 0.7970129870129871 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 7700\n",
            "Prec@(1) = 0.15487179487179487 | Prec@(5) = 0.3119230769230769 | Prec@(10) = 0.3923076923076923 | Prec@(50) = 0.652051282051282 | Prec@(100) = 0.7951282051282051 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 7800\n",
            "Prec@(1) = 0.1530379746835443 | Prec@(5) = 0.3081012658227848 | Prec@(10) = 0.38772151898734175 | Prec@(50) = 0.6456962025316456 | Prec@(100) = 0.789113924050633 | Prec@(500) = 1.0 | NUMBER_OF_SAMPLES = 7900\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-62-d7133f0d8e66>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     14\u001b[0m                                                    \u001b[0mquery_h1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m                                                    \u001b[0mquery_h0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 16\u001b[0;31m                                                    k=500)\n\u001b[0m\u001b[1;32m     17\u001b[0m     \u001b[0;31m# print(accs)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     18\u001b[0m     \u001b[0;31m# print(preds)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-58-cb4f50ab229c>\u001b[0m in \u001b[0;36mpredict_labelwise_doc_at_history\u001b[0;34m(query_h2, query_h1, query_h0, k)\u001b[0m\n\u001b[1;32m     23\u001b[0m     \u001b[0midf_score1\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcalc_idf_score\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mquery_h1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     24\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 25\u001b[0;31m     \u001b[0mquery_h0_embedding\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_embeddings\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mquery_h0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     26\u001b[0m     similarities0 = list(map(lambda x: np.dot(x, query_h0_embedding) /\n\u001b[1;32m     27\u001b[0m                             \u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlinalg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnorm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mquery_h0_embedding\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlinalg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnorm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-12-93659ddcbeb4>\u001b[0m in \u001b[0;36mget_embeddings\u001b[0;34m(sentece)\u001b[0m\n\u001b[1;32m     11\u001b[0m                                 padding=True)\n\u001b[1;32m     12\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mno_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 13\u001b[0;31m         \u001b[0membeddings\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel_labse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mtokenized\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     14\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqueeze\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0membeddings\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpooler_output\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)\u001b[0m\n\u001b[1;32m   1004\u001b[0m             \u001b[0moutput_attentions\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moutput_attentions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1005\u001b[0m             \u001b[0moutput_hidden_states\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moutput_hidden_states\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1006\u001b[0;31m             \u001b[0mreturn_dict\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mreturn_dict\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1007\u001b[0m         )\n\u001b[1;32m   1008\u001b[0m         \u001b[0msequence_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mencoder_outputs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)\u001b[0m\n\u001b[1;32m    590\u001b[0m                     \u001b[0mencoder_attention_mask\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    591\u001b[0m                     \u001b[0mpast_key_value\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 592\u001b[0;31m                     \u001b[0moutput_attentions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    593\u001b[0m                 )\n\u001b[1;32m    594\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions)\u001b[0m\n\u001b[1;32m    512\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    513\u001b[0m         layer_output = apply_chunking_to_forward(\n\u001b[0;32m--> 514\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfeed_forward_chunk\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchunk_size_feed_forward\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mseq_len_dim\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mattention_output\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    515\u001b[0m         )\n\u001b[1;32m    516\u001b[0m         \u001b[0moutputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mlayer_output\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0moutputs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py\u001b[0m in \u001b[0;36mapply_chunking_to_forward\u001b[0;34m(forward_fn, chunk_size, chunk_dim, *input_tensors)\u001b[0m\n\u001b[1;32m   2470\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moutput_chunks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdim\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mchunk_dim\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2471\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2472\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mforward_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput_tensors\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py\u001b[0m in \u001b[0;36mfeed_forward_chunk\u001b[0;34m(self, attention_output)\u001b[0m\n\u001b[1;32m    523\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    524\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mfeed_forward_chunk\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mattention_output\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 525\u001b[0;31m         \u001b[0mintermediate_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mintermediate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mattention_output\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    526\u001b[0m         \u001b[0mlayer_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutput\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mintermediate_output\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mattention_output\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    527\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mlayer_output\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, hidden_states)\u001b[0m\n\u001b[1;32m    424\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    425\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhidden_states\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 426\u001b[0;31m         \u001b[0mhidden_states\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdense\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhidden_states\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    427\u001b[0m         \u001b[0mhidden_states\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mintermediate_act_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhidden_states\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    428\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mhidden_states\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/linear.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m    101\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    102\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 103\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mF\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlinear\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    104\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    105\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mextra_repr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py\u001b[0m in \u001b[0;36mlinear\u001b[0;34m(input, weight, bias)\u001b[0m\n\u001b[1;32m   1846\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhas_torch_function_variadic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1847\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mhandle_torch_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlinear\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbias\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1848\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_C\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_nn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlinear\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1849\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1850\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "XJSnAiOYwv7u"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}