two_stage_retrieval_evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "42fc5c20-860d-46ac-834e-83c3773682c4",
   "metadata": {
    "id": "42fc5c20-860d-46ac-834e-83c3773682c4"
   },
   "source": [
    "# MyScale Two-Stage Retrieval Evaluation\n",
    "\n",
    "**Understanding Metrics in Retrieval Evaluation**:\n",
    "\n",
    "To gauge the efficacy of our retrieval system, we primarily relied on two widely-accepted metrics: **Hit Rate** and **Mean Reciprocal Rank (MRR)**. Let's delve into these metrics to understand their significance and how they operate.\n",
    "\n",
    "- **Hit Rate**:\n",
    "    \n",
    "    Hit rate calculates the fraction of queries where the correct answer is found within the top-k retrieved documents. In simpler terms, it's about how often our system gets it right within the top few guesses.\n",
    "    \n",
    "- **Mean Reciprocal Rank (MRR)**:\n",
    "    \n",
    "    For each query, MRR evaluates the system's accuracy by looking at the rank of the highest-placed relevant document. Specifically, it's the average of the reciprocals of these ranks across all the queries. So, if the first relevant document is the top result, the reciprocal rank is 1; if it's second, the reciprocal rank is 1/2, and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c05bfb0a-369c-444a-bd30-2bd263a55c93",
   "metadata": {
    "scrolled": true,
    "id": "c05bfb0a-369c-444a-bd30-2bd263a55c93",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "fe981d19-8b80-446a-e26c-84e7f3330cb8"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Collecting llama-index\n",
      "  Downloading llama_index-0.9.27-py3-none-any.whl (15.8 MB)\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m15.8/15.8 MB\u001B[0m \u001B[31m41.9 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[?25hCollecting sentence-transformers\n",
      "  Downloading sentence-transformers-2.2.2.tar.gz (85 kB)\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m86.0/86.0 kB\u001B[0m \u001B[31m10.1 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[?25h  Preparing metadata (setup.py) ... \u001B[?25l\u001B[?25hdone\n",
      "Requirement already satisfied: protobuf in /usr/local/lib/python3.10/dist-packages (3.20.3)\n",
      "Collecting pypdf\n",
      "  Downloading pypdf-3.17.4-py3-none-any.whl (278 kB)\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m278.2/278.2 kB\u001B[0m \u001B[31m15.4 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[?25hCollecting clickhouse-connect\n",
      "  Downloading clickhouse_connect-0.6.23-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (964 kB)\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m964.5/964.5 kB\u001B[0m \u001B[31m24.2 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[?25hRequirement already satisfied: SQLAlchemy[asyncio]>=1.4.49 in /usr/local/lib/python3.10/dist-packages (from llama-index) (2.0.24)\n",
      "Requirement already satisfied: aiohttp<4.0.0,>=3.8.6 in /usr/local/lib/python3.10/dist-packages (from llama-index) (3.9.1)\n",
      "Collecting beautifulsoup4<5.0.0,>=4.12.2 (from llama-index)\n",
      "  Downloading beautifulsoup4-4.12.2-py3-none-any.whl (142 kB)\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m143.0/143.0 kB\u001B[0m \u001B[31m19.3 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[?25hCollecting dataclasses-json (from llama-index)\n",
      "  Downloading dataclasses_json-0.6.3-py3-none-any.whl (28 kB)\n",
      "Collecting deprecated>=1.2.9.3 (from llama-index)\n",
      "  Downloading Deprecated-1.2.14-py2.py3-none-any.whl (9.6 kB)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from llama-index) (2023.6.0)\n",
      "Collecting httpx (from llama-index)\n",
      "  Downloading httpx-0.26.0-py3-none-any.whl (75 kB)\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m75.9/75.9 kB\u001B[0m \u001B[31m7.7 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[?25hRequirement already satisfied: nest-asyncio<2.0.0,>=1.5.8 in /usr/local/lib/python3.10/dist-packages (from llama-index) (1.5.8)\n",
      "Requirement already satisfied: networkx>=3.0 in /usr/local/lib/python3.10/dist-packages (from llama-index) (3.2.1)\n",
      "Requirement already satisfied: nltk<4.0.0,>=3.8.1 in /usr/local/lib/python3.10/dist-packages (from llama-index) (3.8.1)\n",
      "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from llama-index) (1.23.5)\n",
      "Collecting openai>=1.1.0 (from llama-index)\n",
      "  Downloading openai-1.7.0-py3-none-any.whl (224 kB)\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m224.7/224.7 kB\u001B[0m \u001B[31m20.2 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[?25hRequirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from llama-index) (1.5.3)\n",
      "Requirement already satisfied: requests>=2.31.0 in /usr/local/lib/python3.10/dist-packages (from llama-index) (2.31.0)\n",
      "Requirement already satisfied: tenacity<9.0.0,>=8.2.0 in /usr/local/lib/python3.10/dist-packages (from llama-index) (8.2.3)\n",
      "Collecting tiktoken>=0.3.3 (from llama-index)\n",
      "  Downloading tiktoken-0.5.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.0 MB)\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m2.0/2.0 MB\u001B[0m \u001B[31m15.9 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[?25hRequirement already satisfied: typing-extensions>=4.5.0 in /usr/local/lib/python3.10/dist-packages (from llama-index) (4.5.0)\n",
      "Collecting typing-inspect>=0.8.0 (from llama-index)\n",
      "  Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB)\n",
      "Requirement already satisfied: transformers<5.0.0,>=4.6.0 in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (4.35.2)\n",
      "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (4.66.1)\n",
      "Requirement already satisfied: torch>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (2.1.0+cu121)\n",
      "Requirement already satisfied: torchvision in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (0.16.0+cu121)\n",
      "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (1.2.2)\n",
      "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (1.11.4)\n",
      "Collecting sentencepiece (from sentence-transformers)\n",
      "  Downloading sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m1.3/1.3 MB\u001B[0m \u001B[31m53.9 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[?25hRequirement already satisfied: huggingface-hub>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (0.20.1)\n",
      "Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from clickhouse-connect) (2023.11.17)\n",
      "Requirement already satisfied: urllib3>=1.26 in /usr/local/lib/python3.10/dist-packages (from clickhouse-connect) (2.0.7)\n",
      "Requirement already satisfied: pytz in /usr/local/lib/python3.10/dist-packages (from clickhouse-connect) (2023.3.post1)\n",
      "Collecting zstandard (from clickhouse-connect)\n",
      "  Downloading zstandard-0.22.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.4 MB)\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m5.4/5.4 MB\u001B[0m \u001B[31m71.5 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[?25hCollecting lz4 (from clickhouse-connect)\n",
      "  Downloading lz4-4.3.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m1.3/1.3 MB\u001B[0m \u001B[31m54.1 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[?25hRequirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.6->llama-index) (23.2.0)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.6->llama-index) (6.0.4)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.6->llama-index) (1.9.4)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.6->llama-index) (1.4.1)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.6->llama-index) (1.3.1)\n",
      "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.6->llama-index) (4.0.3)\n",
      "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4<5.0.0,>=4.12.2->llama-index) (2.5)\n",
      "Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.10/dist-packages (from deprecated>=1.2.9.3->llama-index) (1.14.1)\n",
      "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (3.13.1)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (6.0.1)\n",
      "Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (23.2)\n",
      "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk<4.0.0,>=3.8.1->llama-index) (8.1.7)\n",
      "Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nltk<4.0.0,>=3.8.1->llama-index) (1.3.2)\n",
      "Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.10/dist-packages (from nltk<4.0.0,>=3.8.1->llama-index) (2023.6.3)\n",
      "Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from openai>=1.1.0->llama-index) (3.7.1)\n",
      "Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai>=1.1.0->llama-index) (1.7.0)\n",
      "Requirement already satisfied: pydantic<3,>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from openai>=1.1.0->llama-index) (1.10.13)\n",
      "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from openai>=1.1.0->llama-index) (1.3.0)\n",
      "Collecting typing-extensions>=4.5.0 (from llama-index)\n",
      "  Downloading typing_extensions-4.9.0-py3-none-any.whl (32 kB)\n",
      "Collecting httpcore==1.* (from httpx->llama-index)\n",
      "  Downloading httpcore-1.0.2-py3-none-any.whl (76 kB)\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m76.9/76.9 kB\u001B[0m \u001B[31m6.1 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[?25hRequirement already satisfied: idna in /usr/local/lib/python3.10/dist-packages (from httpx->llama-index) (3.6)\n",
      "Collecting h11<0.15,>=0.13 (from httpcore==1.*->httpx->llama-index)\n",
      "  Downloading h11-0.14.0-py3-none-any.whl (58 kB)\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m58.3/58.3 kB\u001B[0m \u001B[31m7.1 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[?25hRequirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.31.0->llama-index) (3.3.2)\n",
      "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/dist-packages (from SQLAlchemy[asyncio]>=1.4.49->llama-index) (3.0.3)\n",
      "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.6.0->sentence-transformers) (1.12)\n",
      "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.6.0->sentence-transformers) (3.1.2)\n",
      "Requirement already satisfied: triton==2.1.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.6.0->sentence-transformers) (2.1.0)\n",
      "Requirement already satisfied: tokenizers<0.19,>=0.14 in /usr/local/lib/python3.10/dist-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers) (0.15.0)\n",
      "Requirement already satisfied: safetensors>=0.3.1 in /usr/local/lib/python3.10/dist-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers) (0.4.1)\n",
      "Collecting mypy-extensions>=0.3.0 (from typing-inspect>=0.8.0->llama-index)\n",
      "  Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)\n",
      "Collecting marshmallow<4.0.0,>=3.18.0 (from dataclasses-json->llama-index)\n",
      "  Downloading marshmallow-3.20.1-py3-none-any.whl (49 kB)\n",
      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m49.4/49.4 kB\u001B[0m \u001B[31m3.6 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[?25hRequirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->llama-index) (2.8.2)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->sentence-transformers) (3.2.0)\n",
      "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.10/dist-packages (from torchvision->sentence-transformers) (9.4.0)\n",
      "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->openai>=1.1.0->llama-index) (1.2.0)\n",
      "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.1->pandas->llama-index) (1.16.0)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.6.0->sentence-transformers) (2.1.3)\n",
      "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.6.0->sentence-transformers) (1.3.0)\n",
      "Building wheels for collected packages: sentence-transformers\n",
      "  Building wheel for sentence-transformers (setup.py) ... \u001B[?25l\u001B[?25hdone\n",
      "  Created wheel for sentence-transformers: filename=sentence_transformers-2.2.2-py3-none-any.whl size=125923 sha256=188382ab9d2c018346f910c3bc749d3bf727103a79cb40189e755a67c59833ff\n",
      "  Stored in directory: /root/.cache/pip/wheels/62/f2/10/1e606fd5f02395388f74e7462910fe851042f97238cbbd902f\n",
      "Successfully built sentence-transformers\n",
      "Installing collected packages: sentencepiece, zstandard, typing-extensions, pypdf, mypy-extensions, marshmallow, lz4, h11, deprecated, beautifulsoup4, typing-inspect, tiktoken, httpcore, clickhouse-connect, httpx, dataclasses-json, openai, sentence-transformers, llama-index\n",
      "  Attempting uninstall: typing-extensions\n",
      "    Found existing installation: typing_extensions 4.5.0\n",
      "    Uninstalling typing_extensions-4.5.0:\n",
      "      Successfully uninstalled typing_extensions-4.5.0\n",
      "  Attempting uninstall: beautifulsoup4\n",
      "    Found existing installation: beautifulsoup4 4.11.2\n",
      "    Uninstalling beautifulsoup4-4.11.2:\n",
      "      Successfully uninstalled beautifulsoup4-4.11.2\n",
      "\u001B[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "llmx 0.0.15a0 requires cohere, which is not installed.\n",
      "tensorflow-probability 0.22.0 requires typing-extensions<4.6.0, but you have typing-extensions 4.9.0 which is incompatible.\u001B[0m\u001B[31m\n",
      "\u001B[0mSuccessfully installed beautifulsoup4-4.12.2 clickhouse-connect-0.6.23 dataclasses-json-0.6.3 deprecated-1.2.14 h11-0.14.0 httpcore-1.0.2 httpx-0.26.0 llama-index-0.9.27 lz4-4.3.3 marshmallow-3.20.1 mypy-extensions-1.0.0 openai-1.7.0 pypdf-3.17.4 sentence-transformers-2.2.2 sentencepiece-0.1.99 tiktoken-0.5.2 typing-extensions-4.9.0 typing-inspect-0.9.0 zstandard-0.22.0\n"
     ]
    }
   ],
   "source": [
    "!pip install llama-index sentence-transformers protobuf pypdf clickhouse-connect"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "915a3c6b-0e43-4461-a86e-64007ebf9ae5",
   "metadata": {
    "id": "915a3c6b-0e43-4461-a86e-64007ebf9ae5"
   },
   "source": [
    "## Setting Up the Environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e6989d8-25cc-4c53-b42b-3c9315a831ca",
   "metadata": {
    "id": "5e6989d8-25cc-4c53-b42b-3c9315a831ca"
   },
   "outputs": [],
   "source": [
    "from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext\n",
    "from llama_index.storage.storage_context import StorageContext\n",
    "from llama_index.vector_stores import MyScaleVectorStore\n",
    "from llama_index.node_parser import SimpleNodeParser\n",
    "\n",
    "# LLM\n",
    "from llama_index.llms import OpenAI\n",
    "\n",
    "# Embeddings\n",
    "from llama_index.embeddings import OpenAIEmbedding\n",
    "\n",
    "# Retrievers\n",
    "from llama_index.retrievers import (\n",
    "    BaseRetriever,\n",
    "    VectorIndexRetriever,\n",
    ")\n",
    "\n",
    "# Rerankers\n",
    "from llama_index.indices.query.schema import QueryBundle, QueryType\n",
    "from llama_index.schema import NodeWithScore\n",
    "from llama_index.indices.postprocessor import SentenceTransformerRerank\n",
    "from llama_index.finetuning.embeddings.common import EmbeddingQAFinetuneDataset\n",
    "\n",
    "# Evaluator\n",
    "from llama_index.evaluation import (\n",
    "    generate_question_context_pairs,\n",
    "    EmbeddingQAFinetuneDataset,\n",
    ")\n",
    "from llama_index.evaluation import RetrieverEvaluator\n",
    "\n",
    "\n",
    "from typing import List\n",
    "import pandas as pd\n",
    "import openai\n",
    "\n",
    "import nest_asyncio\n",
    "import clickhouse_connect\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f650d35-fbd6-4e4e-968d-808ff9fbb923",
   "metadata": {
    "id": "3f650d35-fbd6-4e4e-968d-808ff9fbb923"
   },
   "source": [
    "## Settingup API Keys and MyScale Configs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5313e42-d71c-4427-a0e2-0578f3831c4c",
   "metadata": {
    "id": "f5313e42-d71c-4427-a0e2-0578f3831c4c"
   },
   "outputs": [],
   "source": [
    "openai_api_key = 'YOUR_OPENAI_API_KEY'\n",
    "openai_api_base = 'YOUR_OPENAI_API_BASE'\n",
    "myscale_host = 'MYSCALE_HOST'\n",
    "myscale_username = 'MYSCALE_USERNAME'\n",
    "myscale_password = 'MYSCALE_PASSWORD'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d54c37b-3d4f-4b99-83fd-f98e97f2bcda",
   "metadata": {
    "id": "2d54c37b-3d4f-4b99-83fd-f98e97f2bcda"
   },
   "source": [
    "## Download Data\n",
    "\n",
    "We will use Llama2 paper for this experiment. Let's download the paper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ef4be8e-f307-45a2-8dcf-319738b55051",
   "metadata": {
    "id": "5ef4be8e-f307-45a2-8dcf-319738b55051",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "6011cc30-e7de-419d-8f1c-0cb4019e643f"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "--2024-01-09 02:15:53--  https://arxiv.org/pdf/2307.09288.pdf\n",
      "Resolving arxiv.org (arxiv.org)... 151.101.195.42, 151.101.3.42, 151.101.131.42, ...\n",
      "Connecting to arxiv.org (arxiv.org)|151.101.195.42|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 13661300 (13M) [application/pdf]\n",
      "Saving to: ‘llama2.pdf’\n",
      "\n",
      "llama2.pdf          100%[===================>]  13.03M  27.2MB/s    in 0.5s    \n",
      "\n",
      "2024-01-09 02:15:53 (27.2 MB/s) - ‘llama2.pdf’ saved [13661300/13661300]\n"
     ]
    }
   ],
   "source": [
    "!wget --user-agent \"Mozilla\" \"https://arxiv.org/pdf/2307.09288.pdf\" -O \"llama2.pdf\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28633c1f-86ba-4272-afc9-ddd3a17c8593",
   "metadata": {
    "id": "28633c1f-86ba-4272-afc9-ddd3a17c8593"
   },
   "source": [
    "## Loading the Data\n",
    "\n",
    "Let’s load the data. We will use Pages from start to 36 for the experiment which excludes table of contents, references and appendix.\n",
    "\n",
    "This data was then parsed by converted to nodes, which represent chunks of data we'd like to retrieve. We did use chunk_size as 512."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43f367cc-7ed9-4cf3-9f19-de223eec0841",
   "metadata": {
    "id": "43f367cc-7ed9-4cf3-9f19-de223eec0841"
   },
   "outputs": [],
   "source": [
    "documents = SimpleDirectoryReader(input_files=['llama2.pdf']).load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3971e2d-2b91-4bfa-bb77-c87fef94d8af",
   "metadata": {
    "id": "e3971e2d-2b91-4bfa-bb77-c87fef94d8af"
   },
   "outputs": [],
   "source": [
    "node_parser = SimpleNodeParser.from_defaults(chunk_size=512)\n",
    "nodes = node_parser.get_nodes_from_documents(documents[:37])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22f83542-cc39-45fa-a728-c8a94f3e1be3",
   "metadata": {
    "id": "22f83542-cc39-45fa-a728-c8a94f3e1be3"
   },
   "source": [
    "## Generating Question-Context Pairs\n",
    "\n",
    "For evaluation purposes, we created a dataset of question-context pairs. This dataset can be seen as a set of questions and their corresponding context from our data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31753606-fece-4c64-9152-b7623be16bca",
   "metadata": {
    "id": "31753606-fece-4c64-9152-b7623be16bca"
   },
   "source": [
    "Let's initialize prompt template to generate question-context pairs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66a231e0-06cd-4027-8cb9-f88ea1910b93",
   "metadata": {
    "id": "66a231e0-06cd-4027-8cb9-f88ea1910b93"
   },
   "outputs": [],
   "source": [
    "# Prompt to generate questions\n",
    "qa_generate_prompt_tmpl = \"\"\"\\\n",
    "Context information is below.\n",
    "\n",
    "---------------------\n",
    "{context_str}\n",
    "---------------------\n",
    "\n",
    "Given the context information and not prior knowledge.\n",
    "generate only questions based on the below query.\n",
    "\n",
    "You are a Professor. Your task is to setup \\\n",
    "{num_questions_per_chunk} questions for an upcoming \\\n",
    "quiz/examination. The questions should be diverse in nature \\\n",
    "across the document. The questions should not contain options, not start with Q1/ Q2. \\\n",
    "Restrict the questions to the context information provided.\\\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffd8318d-68a2-4808-934a-6302501fae7b",
   "metadata": {
    "id": "ffd8318d-68a2-4808-934a-6302501fae7b",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "87d063ca-3354-4f2d-c427-8cdcf649f3b6"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "100%|██████████| 109/109 [02:27<00:00,  1.35s/it]\n"
     ]
    }
   ],
   "source": [
    "llm = OpenAI(api_key=openai_api_key, api_base=openai_api_base)\n",
    "qa_dataset = generate_question_context_pairs(\n",
    "    nodes, llm=llm, num_questions_per_chunk=2, qa_generate_prompt_tmpl=qa_generate_prompt_tmpl\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ee59cb1-0d42-4bc5-bd34-b053450463b4",
   "metadata": {
    "id": "1ee59cb1-0d42-4bc5-bd34-b053450463b4"
   },
   "source": [
    "Function to filter out sentences such as `Here are 2 questions based on provided context`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "290473b1-c568-490e-9526-fb4032a8de09",
   "metadata": {
    "id": "290473b1-c568-490e-9526-fb4032a8de09"
   },
   "outputs": [],
   "source": [
    "# function to clean the dataset\n",
    "def filter_qa_dataset(qa_dataset):\n",
    "    \"\"\"\n",
    "    Filters out queries from the qa_dataset that contain certain phrases and the corresponding\n",
    "    entries in the relevant_docs, and creates a new EmbeddingQAFinetuneDataset object with\n",
    "    the filtered data.\n",
    "\n",
    "    :param qa_dataset: An object that has 'queries', 'corpus', and 'relevant_docs' attributes.\n",
    "    :return: An EmbeddingQAFinetuneDataset object with the filtered queries, corpus and relevant_docs.\n",
    "    \"\"\"\n",
    "\n",
    "    # Extract keys from queries and relevant_docs that need to be removed\n",
    "    queries_relevant_docs_keys_to_remove = {\n",
    "        k for k, v in qa_dataset.queries.items()\n",
    "        if 'Here are 2' in v or 'Here are two' in v\n",
    "    }\n",
    "\n",
    "    # Filter queries and relevant_docs using dictionary comprehensions\n",
    "    filtered_queries = {\n",
    "        k: v for k, v in qa_dataset.queries.items()\n",
    "        if k not in queries_relevant_docs_keys_to_remove\n",
    "    }\n",
    "    filtered_relevant_docs = {\n",
    "        k: v for k, v in qa_dataset.relevant_docs.items()\n",
    "        if k not in queries_relevant_docs_keys_to_remove\n",
    "    }\n",
    "\n",
    "    # Create a new instance of EmbeddingQAFinetuneDataset with the filtered data\n",
    "    return EmbeddingQAFinetuneDataset(\n",
    "        queries=filtered_queries,\n",
    "        corpus=qa_dataset.corpus,\n",
    "        relevant_docs=filtered_relevant_docs\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c0f70e6-eb3f-4823-b457-64e08ee7612a",
   "metadata": {
    "id": "9c0f70e6-eb3f-4823-b457-64e08ee7612a"
   },
   "outputs": [],
   "source": [
    "# filter out pairs with phrases `Here are 2 questions based on provided context`\n",
    "qa_dataset = filter_qa_dataset(qa_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f757e90-3f95-4c1d-ba2a-e172a0fd7645",
   "metadata": {
    "id": "4f757e90-3f95-4c1d-ba2a-e172a0fd7645"
   },
   "source": [
    "## Initialize Embeddings and Retrievers\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aab9d1ec-2ca4-4cd3-bd8b-0dde31f51d73",
   "metadata": {
    "id": "aab9d1ec-2ca4-4cd3-bd8b-0dde31f51d73",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 316,
     "referenced_widgets": [
      "c9dcab4e4637465087d92b313255a976",
      "22db0cd0329a43faa1df835859d5e9d7",
      "73f932e03c3b426383099c0f10e53899",
      "5667b89d4b764a58b3e0f702b37d2055",
      "64a4205e41d94930aefce2e5c461d79d",
      "aa957beadd934ec5a1fb6acc35383e97",
      "676f71a4c1d145e4a94f76dd1f4b9c29",
      "005a4ad88ecf453f9efbf1c7b474d5fe",
      "0e346f9b23f3414cad6c2b365c0049bb",
      "8b704e4e0c3b465f82b07537c0bbebae",
      "0d7c9fa4088442099b4f0595a63c61c1",
      "6a308d8655e346a5831859d571eff2f6",
      "5dcb1a1599c44ca9a2a7a76a2e444482",
      "8078afb29d7d4c33adf9b9a3d2f64861",
      "1bde5825a1554d309480074b43f3782d",
      "a57ef29dbdd34d9da7945901723a9289",
      "f0593c2fb61e43d1912ec0ce1da3bffc",
      "8687f3fb20af4d8f9c1ab190109ea8e3",
      "bae57545c8284a71ae4948dca54eb445",
      "2c22fe755608475a97060db49baa344b",
      "0554727ec82d41c7807d3900b1284e2e",
      "6be52bf329eb447795df2764a299f318",
      "1749ba660711402e8ad348ddf9ffec7a",
      "8f573f660d8041daa504b1e859e50d1e",
      "7d9e84dc24ab472a843c4161a8973fa6",
      "621bc77f3cae4348964362a089867609",
      "1bf5e4518b0e4d90871f9ffd4968f3e1",
      "9409053d36b04c8f9cd206d2172aa00f",
      "ce5ccc31d0254d8180c3468d945a6037",
      "906c010c13dd4e14a22463d666811ac3",
      "748dbc5b40984536aa9f872e7dad13fb",
      "85d1914d2c4f4240a8fe761f21c9de6c",
      "a7993a3b50674d3e88ae90ee22fcd4f0",
      "5228b540158a4c1f89fa36a7799df896",
      "7df296daa6444ba8972916a8ac77d1c9",
      "37b9eff9ad5c48debcb799a9110f2fab",
      "81221316383147b5a0fbe9317313adac",
      "7338d4ad577a44729d90fd026e93504e",
      "d3824103a1d243d184f893851e9705ff",
      "ff0ffdbb6937476c8f0e349a36fab918",
      "4dc4b08440924190afb396f54e5e8b91",
      "b125b56d6cfe4f89a52b16d98788eec5",
      "8a4f96c33379417a8e57335cd9d4e407",
      "de55283ff55c473dbf6ce3ad906fcaad",
      "77f26ea201a041f0a05a0f2274f84ab7",
      "36031fd44cc94970b66d1634c3469bf2",
      "e43ca9ab7bf14c9d9a66d202ca7c0d29",
      "9b193c70a2e14762b47eea1031d520ce",
      "a0ee08ffcf6e4b93aea0c6898da7f339",
      "cf0babe97285474a86a07a2821de083d",
      "7591b26caf7c4fd5aaed0191006782cc",
      "e65127db9c1949f5b0cd32bad51356c8",
      "7146f0931d264e97865d7ce620dc3383",
      "13b6144dbd5243a08760063b2c639bed",
      "f899851effea4b74be0080e5b4a3ff52",
      "b35a7f6c65204abbaf57fa41f5447ff2",
      "ac927faee6a542ef99bb50e4b4086be0",
      "e734d3dc37da4fa2bffad127130aadb7",
      "717d90080bc549dc832ca3a4022c62a4",
      "02145253e17047acbd078622eefb249d",
      "477c4217778b4e07a900473f5cd93f02",
      "43bc99b52c2b46fa8b6ab897fb87dec4",
      "e15b189542bc4da6b2a88fb34e7ff945",
      "143ca06b543b468d8da6781aa2cfc352",
      "c322cfb893a4414783a437010e6c4f6a",
      "3740f2bf346a4d4498d399e98b9c1910"
     ]
    },
    "outputId": "e2b075cd-bbce-423b-e6c0-15e0a0202a12"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:72: UserWarning: \n",
      "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
      "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
      "You will be able to reuse this secret in all of your notebooks.\n",
      "Please note that authentication is recommended but still optional to access public models or datasets.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "output_type": "display_data",
     "data": {
      "text/plain": [
       "config.json:   0%|          | 0.00/799 [00:00<?, ?B/s]"
      ],
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "c9dcab4e4637465087d92b313255a976"
      }
     },
     "metadata": {}
    },
    {
     "output_type": "display_data",
     "data": {
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/1.11G [00:00<?, ?B/s]"
      ],
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "6a308d8655e346a5831859d571eff2f6"
      }
     },
     "metadata": {}
    },
    {
     "output_type": "display_data",
     "data": {
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/443 [00:00<?, ?B/s]"
      ],
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "1749ba660711402e8ad348ddf9ffec7a"
      }
     },
     "metadata": {}
    },
    {
     "output_type": "display_data",
     "data": {
      "text/plain": [
       "sentencepiece.bpe.model:   0%|          | 0.00/5.07M [00:00<?, ?B/s]"
      ],
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "5228b540158a4c1f89fa36a7799df896"
      }
     },
     "metadata": {}
    },
    {
     "output_type": "display_data",
     "data": {
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/17.1M [00:00<?, ?B/s]"
      ],
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "77f26ea201a041f0a05a0f2274f84ab7"
      }
     },
     "metadata": {}
    },
    {
     "output_type": "display_data",
     "data": {
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/279 [00:00<?, ?B/s]"
      ],
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "b35a7f6c65204abbaf57fa41f5447ff2"
      }
     },
     "metadata": {}
    }
   ],
   "source": [
    "# Define embeddings and rerankers for text processing\n",
    "# For embeddings, we utilize OpenAI's Embedding model\n",
    "EMBEDDINGS = {\n",
    "    \"OpenAI\": OpenAIEmbedding(api_key=openai_api_key, api_base=openai_api_base)\n",
    "}\n",
    "\n",
    "# For rerankers, we have two options for evalutaion:\n",
    "# 1. 'WithoutReranker' which implies no reranking is applied\n",
    "# 2. 'BgeRerankerBase' uses BAAI/bge-reranker-base reranking model to refine results, limited to top 5 results\n",
    "\n",
    "RERANKERS = {\n",
    "    \"WithoutReranker\": \"None\",\n",
    "    \"BgeRerankerBase\": SentenceTransformerRerank(model=\"BAAI/bge-reranker-base\", top_n=5),\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a5533b3-eb9c-4628-b197-a23826178a84",
   "metadata": {
    "id": "6a5533b3-eb9c-4628-b197-a23826178a84"
   },
   "source": [
    "### Define a function to display results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bdc5076-7806-4cec-a168-a18f1639a5b8",
   "metadata": {
    "id": "1bdc5076-7806-4cec-a168-a18f1639a5b8"
   },
   "outputs": [],
   "source": [
    "def display_results(embedding_name, reranker_name, eval_results):\n",
    "    \"\"\"Display results from evaluate.\"\"\"\n",
    "\n",
    "    metric_dicts = []\n",
    "    for eval_result in eval_results:\n",
    "        metric_dict = eval_result.metric_vals_dict\n",
    "        metric_dicts.append(metric_dict)\n",
    "\n",
    "    full_df = pd.DataFrame(metric_dicts)\n",
    "\n",
    "    hit_rate = full_df[\"hit_rate\"].mean()\n",
    "    mrr = full_df[\"mrr\"].mean()\n",
    "\n",
    "    metric_df = pd.DataFrame(\n",
    "        {\"Embedding\": [embedding_name], \"Reranker\": [reranker_name], \"hit_rate\": [hit_rate], \"mrr\": [mrr]}\n",
    "    )\n",
    "\n",
    "    return metric_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba1106e7-1135-461e-82a6-8ac758d60ab0",
   "metadata": {
    "id": "ba1106e7-1135-461e-82a6-8ac758d60ab0"
   },
   "source": [
    "## Define Retriever and Evaluate\n",
    "\n",
    "To identify the optimal retriever, we employ a combination of an embedding model and a reranker. Initially, we establish a base VectorIndexRetriever. Upon retrieving the nodes, we then introduce a reranker to further refine the results. It's worth noting that for this particular experiment, we've set similarity_top_k to 5. However, feel free to adjust this parameter based on the needs of your specific experiment."
   ]
  },
  {
   "cell_type": "code",
   "source": [
    "results_df = pd.DataFrame()\n",
    "client = clickhouse_connect.get_client(\n",
    "    host=myscale_host,\n",
    "    port=443,\n",
    "    username=myscale_username,\n",
    "    password=myscale_password\n",
    ")\n",
    "\n",
    "# Loop over embeddings\n",
    "for embed_name, embed_model in EMBEDDINGS.items():\n",
    "\n",
    "    service_context = ServiceContext.from_defaults(llm=None, embed_model=embed_model)\n",
    "    vector_store = MyScaleVectorStore(myscale_client=client)\n",
    "    storage_context = StorageContext.from_defaults(vector_store=vector_store)\n",
    "    vector_index = VectorStoreIndex(nodes, service_context=service_context, storage_context=storage_context)\n",
    "    vector_retriever = VectorIndexRetriever(index=vector_index, similarity_top_k=10)\n",
    "\n",
    "    # Loop over rerankers\n",
    "    for rerank_name, reranker in RERANKERS.items():\n",
    "\n",
    "        print(f\"Running Evaluation for Embedding Model: {embed_name} and Reranker: {rerank_name}\")\n",
    "\n",
    "        # Define Retriever\n",
    "        class CustomRetriever(BaseRetriever):\n",
    "            \"\"\"Custom retriever that performs both Vector search and Knowledge Graph search\"\"\"\n",
    "\n",
    "            def __init__(\n",
    "                self,\n",
    "                vector_retriever: VectorIndexRetriever,\n",
    "            ) -> None:\n",
    "                \"\"\"Init params.\"\"\"\n",
    "\n",
    "                self._vector_retriever = vector_retriever\n",
    "\n",
    "            def _retrieve(self, query_bundle: QueryBundle) -> List[NodeWithScore]:\n",
    "                \"\"\"Retrieve nodes given query.\"\"\"\n",
    "\n",
    "                retrieved_nodes = self._vector_retriever.retrieve(query_bundle)\n",
    "                if reranker != 'None':\n",
    "                    retrieved_nodes = reranker.postprocess_nodes(retrieved_nodes, query_bundle)\n",
    "                else:\n",
    "                    retrieved_nodes = retrieved_nodes[:5]\n",
    "                return retrieved_nodes\n",
    "\n",
    "            async def _aretrieve(self, query_bundle: QueryBundle) -> List[NodeWithScore]:\n",
    "                \"\"\"Asynchronously retrieve nodes given query.\n",
    "\n",
    "                Implemented by the user.\n",
    "\n",
    "                \"\"\"\n",
    "                return self._retrieve(query_bundle)\n",
    "\n",
    "            async def aretrieve(self, str_or_query_bundle: QueryType) -> List[NodeWithScore]:\n",
    "                if isinstance(str_or_query_bundle, str):\n",
    "                    str_or_query_bundle = QueryBundle(str_or_query_bundle)\n",
    "                return await self._aretrieve(str_or_query_bundle)\n",
    "\n",
    "        custom_retriever = CustomRetriever(vector_retriever)\n",
    "\n",
    "        retriever_evaluator = RetrieverEvaluator.from_metric_names(\n",
    "            [\"mrr\", \"hit_rate\"], retriever=custom_retriever\n",
    "        )\n",
    "        eval_results = await retriever_evaluator.aevaluate_dataset(qa_dataset)\n",
    "        current_df = display_results(embed_name, rerank_name, eval_results)\n",
    "        results_df = pd.concat([results_df, current_df], ignore_index=True)"
   ],
   "metadata": {
    "id": "rAP1GbZJnIPW"
   },
   "id": "rAP1GbZJnIPW",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "0b5cde62-5923-4caa-a6c3-c0c081716a73",
   "metadata": {
    "id": "0b5cde62-5923-4caa-a6c3-c0c081716a73"
   },
   "source": [
    "## Check Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "605bb11c-97f2-4405-9603-9cacd44fb535",
   "metadata": {
    "id": "605bb11c-97f2-4405-9603-9cacd44fb535",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "66acf39b-b056-4b25-ebbf-b7c2e4d94b25"
   },
   "outputs": [
    {
     "metadata": {
      "tags": null
     },
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Embedding         Reranker  hit_rate       mrr\n",
      "0    OpenAI  WithoutReranker  0.854545  0.640303\n",
      "1    OpenAI  BgeRerankerBase  0.895455  0.707652\n"
     ]
    }
   ],
   "source": [
    "# Display final results\n",
    "print(results_df)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "colab": {
   "provenance": [],
   "gpuType": "T4"
  },
  "accelerator": "GPU",
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "c9dcab4e4637465087d92b313255a976": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_22db0cd0329a43faa1df835859d5e9d7",
       "IPY_MODEL_73f932e03c3b426383099c0f10e53899",
       "IPY_MODEL_5667b89d4b764a58b3e0f702b37d2055"
      ],
      "layout": "IPY_MODEL_64a4205e41d94930aefce2e5c461d79d"
     }
    },
    "22db0cd0329a43faa1df835859d5e9d7": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_aa957beadd934ec5a1fb6acc35383e97",
      "placeholder": "​",
      "style": "IPY_MODEL_676f71a4c1d145e4a94f76dd1f4b9c29",
      "value": "config.json: 100%"
     }
    },
    "73f932e03c3b426383099c0f10e53899": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_005a4ad88ecf453f9efbf1c7b474d5fe",
      "max": 799,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_0e346f9b23f3414cad6c2b365c0049bb",
      "value": 799
     }
    },
    "5667b89d4b764a58b3e0f702b37d2055": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_8b704e4e0c3b465f82b07537c0bbebae",
      "placeholder": "​",
      "style": "IPY_MODEL_0d7c9fa4088442099b4f0595a63c61c1",
      "value": " 799/799 [00:00&lt;00:00, 50.3kB/s]"
     }
    },
    "64a4205e41d94930aefce2e5c461d79d": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "aa957beadd934ec5a1fb6acc35383e97": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "676f71a4c1d145e4a94f76dd1f4b9c29": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "model_module_version": "1.5.0",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "005a4ad88ecf453f9efbf1c7b474d5fe": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0e346f9b23f3414cad6c2b365c0049bb": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "model_module_version": "1.5.0",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "8b704e4e0c3b465f82b07537c0bbebae": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0d7c9fa4088442099b4f0595a63c61c1": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "model_module_version": "1.5.0",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "6a308d8655e346a5831859d571eff2f6": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_5dcb1a1599c44ca9a2a7a76a2e444482",
       "IPY_MODEL_8078afb29d7d4c33adf9b9a3d2f64861",
       "IPY_MODEL_1bde5825a1554d309480074b43f3782d"
      ],
      "layout": "IPY_MODEL_a57ef29dbdd34d9da7945901723a9289"
     }
    },
    "5dcb1a1599c44ca9a2a7a76a2e444482": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f0593c2fb61e43d1912ec0ce1da3bffc",
      "placeholder": "​",
      "style": "IPY_MODEL_8687f3fb20af4d8f9c1ab190109ea8e3",
      "value": "model.safetensors: 100%"
     }
    },
    "8078afb29d7d4c33adf9b9a3d2f64861": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_bae57545c8284a71ae4948dca54eb445",
      "max": 1112206140,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_2c22fe755608475a97060db49baa344b",
      "value": 1112206140
     }
    },
    "1bde5825a1554d309480074b43f3782d": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0554727ec82d41c7807d3900b1284e2e",
      "placeholder": "​",
      "style": "IPY_MODEL_6be52bf329eb447795df2764a299f318",
      "value": " 1.11G/1.11G [00:06&lt;00:00, 63.8MB/s]"
     }
    },
    "a57ef29dbdd34d9da7945901723a9289": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f0593c2fb61e43d1912ec0ce1da3bffc": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8687f3fb20af4d8f9c1ab190109ea8e3": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "model_module_version": "1.5.0",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "bae57545c8284a71ae4948dca54eb445": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2c22fe755608475a97060db49baa344b": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "model_module_version": "1.5.0",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "0554727ec82d41c7807d3900b1284e2e": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6be52bf329eb447795df2764a299f318": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "model_module_version": "1.5.0",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "1749ba660711402e8ad348ddf9ffec7a": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_8f573f660d8041daa504b1e859e50d1e",
       "IPY_MODEL_7d9e84dc24ab472a843c4161a8973fa6",
       "IPY_MODEL_621bc77f3cae4348964362a089867609"
      ],
      "layout": "IPY_MODEL_1bf5e4518b0e4d90871f9ffd4968f3e1"
     }
    },
    "8f573f660d8041daa504b1e859e50d1e": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9409053d36b04c8f9cd206d2172aa00f",
      "placeholder": "​",
      "style": "IPY_MODEL_ce5ccc31d0254d8180c3468d945a6037",
      "value": "tokenizer_config.json: 100%"
     }
    },
    "7d9e84dc24ab472a843c4161a8973fa6": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_906c010c13dd4e14a22463d666811ac3",
      "max": 443,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_748dbc5b40984536aa9f872e7dad13fb",
      "value": 443
     }
    },
    "621bc77f3cae4348964362a089867609": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_85d1914d2c4f4240a8fe761f21c9de6c",
      "placeholder": "​",
      "style": "IPY_MODEL_a7993a3b50674d3e88ae90ee22fcd4f0",
      "value": " 443/443 [00:00&lt;00:00, 23.1kB/s]"
     }
    },
    "1bf5e4518b0e4d90871f9ffd4968f3e1": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9409053d36b04c8f9cd206d2172aa00f": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ce5ccc31d0254d8180c3468d945a6037": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "model_module_version": "1.5.0",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "906c010c13dd4e14a22463d666811ac3": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "748dbc5b40984536aa9f872e7dad13fb": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "model_module_version": "1.5.0",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "85d1914d2c4f4240a8fe761f21c9de6c": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a7993a3b50674d3e88ae90ee22fcd4f0": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "model_module_version": "1.5.0",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "5228b540158a4c1f89fa36a7799df896": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_7df296daa6444ba8972916a8ac77d1c9",
       "IPY_MODEL_37b9eff9ad5c48debcb799a9110f2fab",
       "IPY_MODEL_81221316383147b5a0fbe9317313adac"
      ],
      "layout": "IPY_MODEL_7338d4ad577a44729d90fd026e93504e"
     }
    },
    "7df296daa6444ba8972916a8ac77d1c9": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d3824103a1d243d184f893851e9705ff",
      "placeholder": "​",
      "style": "IPY_MODEL_ff0ffdbb6937476c8f0e349a36fab918",
      "value": "sentencepiece.bpe.model: 100%"
     }
    },
    "37b9eff9ad5c48debcb799a9110f2fab": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4dc4b08440924190afb396f54e5e8b91",
      "max": 5069051,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_b125b56d6cfe4f89a52b16d98788eec5",
      "value": 5069051
     }
    },
    "81221316383147b5a0fbe9317313adac": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_8a4f96c33379417a8e57335cd9d4e407",
      "placeholder": "​",
      "style": "IPY_MODEL_de55283ff55c473dbf6ce3ad906fcaad",
      "value": " 5.07M/5.07M [00:00&lt;00:00, 22.7MB/s]"
     }
    },
    "7338d4ad577a44729d90fd026e93504e": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d3824103a1d243d184f893851e9705ff": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ff0ffdbb6937476c8f0e349a36fab918": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "model_module_version": "1.5.0",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4dc4b08440924190afb396f54e5e8b91": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b125b56d6cfe4f89a52b16d98788eec5": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "model_module_version": "1.5.0",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "8a4f96c33379417a8e57335cd9d4e407": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "de55283ff55c473dbf6ce3ad906fcaad": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "model_module_version": "1.5.0",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "77f26ea201a041f0a05a0f2274f84ab7": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_36031fd44cc94970b66d1634c3469bf2",
       "IPY_MODEL_e43ca9ab7bf14c9d9a66d202ca7c0d29",
       "IPY_MODEL_9b193c70a2e14762b47eea1031d520ce"
      ],
      "layout": "IPY_MODEL_a0ee08ffcf6e4b93aea0c6898da7f339"
     }
    },
    "36031fd44cc94970b66d1634c3469bf2": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_cf0babe97285474a86a07a2821de083d",
      "placeholder": "​",
      "style": "IPY_MODEL_7591b26caf7c4fd5aaed0191006782cc",
      "value": "tokenizer.json: 100%"
     }
    },
    "e43ca9ab7bf14c9d9a66d202ca7c0d29": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_e65127db9c1949f5b0cd32bad51356c8",
      "max": 17098107,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_7146f0931d264e97865d7ce620dc3383",
      "value": 17098107
     }
    },
    "9b193c70a2e14762b47eea1031d520ce": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_13b6144dbd5243a08760063b2c639bed",
      "placeholder": "​",
      "style": "IPY_MODEL_f899851effea4b74be0080e5b4a3ff52",
      "value": " 17.1M/17.1M [00:00&lt;00:00, 98.3MB/s]"
     }
    },
    "a0ee08ffcf6e4b93aea0c6898da7f339": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "cf0babe97285474a86a07a2821de083d": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7591b26caf7c4fd5aaed0191006782cc": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "model_module_version": "1.5.0",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "e65127db9c1949f5b0cd32bad51356c8": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7146f0931d264e97865d7ce620dc3383": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "model_module_version": "1.5.0",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "13b6144dbd5243a08760063b2c639bed": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f899851effea4b74be0080e5b4a3ff52": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "model_module_version": "1.5.0",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b35a7f6c65204abbaf57fa41f5447ff2": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_ac927faee6a542ef99bb50e4b4086be0",
       "IPY_MODEL_e734d3dc37da4fa2bffad127130aadb7",
       "IPY_MODEL_717d90080bc549dc832ca3a4022c62a4"
      ],
      "layout": "IPY_MODEL_02145253e17047acbd078622eefb249d"
     }
    },
    "ac927faee6a542ef99bb50e4b4086be0": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_477c4217778b4e07a900473f5cd93f02",
      "placeholder": "​",
      "style": "IPY_MODEL_43bc99b52c2b46fa8b6ab897fb87dec4",
      "value": "special_tokens_map.json: 100%"
     }
    },
    "e734d3dc37da4fa2bffad127130aadb7": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_e15b189542bc4da6b2a88fb34e7ff945",
      "max": 279,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_143ca06b543b468d8da6781aa2cfc352",
      "value": 279
     }
    },
    "717d90080bc549dc832ca3a4022c62a4": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "model_module_version": "1.5.0",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c322cfb893a4414783a437010e6c4f6a",
      "placeholder": "​",
      "style": "IPY_MODEL_3740f2bf346a4d4498d399e98b9c1910",
      "value": " 279/279 [00:00&lt;00:00, 15.7kB/s]"
     }
    },
    "02145253e17047acbd078622eefb249d": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "477c4217778b4e07a900473f5cd93f02": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "43bc99b52c2b46fa8b6ab897fb87dec4": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "model_module_version": "1.5.0",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "e15b189542bc4da6b2a88fb34e7ff945": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "143ca06b543b468d8da6781aa2cfc352": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "model_module_version": "1.5.0",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "c322cfb893a4414783a437010e6c4f6a": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "model_module_version": "1.2.0",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3740f2bf346a4d4498d399e98b9c1910": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "model_module_version": "1.5.0",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}