tutorials/different-types-text-chunking-in-RAG/Text_Chunking_on_RAG_application_with_LanceDB.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yzGy7Oi-EoNo"
   },
   "source": [
    "## Retrieval Augmented Generation with LanceDB  \n",
    "\n",
    "**Objective:**\n",
    "Use Llama 2.0, Langchain and LanceDB to create a Retrieval Augmented Generation (RAG) system.\n",
    "\n",
    "This will allow us to ask questions about our documents (that were not included in the training data), without fine-tunning the Large Language Model (LLM).\n",
    "\n",
    "Here Text Splitting will help LLM to give accurate answers without hallucination.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ENgV8B98FP37"
   },
   "source": [
    "## What is a Retrieval Augmented Generation (RAG) system?\n",
    "\n",
    "Retrieval-augmented generation (RAG) is an AI framework for improving the quality of LLM-generated responses by grounding the model on external sources of knowledge to supplement the LLM’s internal representation of information. Implementing RAG in an LLM-based question answering system has two main benefits:\n",
    "1. It ensures that the model has access to the most current, reliable facts, and that users have access to the model’s sources, ensuring that its claims can be checked for accuracy and ultimately trusted.\n",
    "2. RAG has additional benefits. By grounding an LLM on a set of external, verifiable facts, the model has fewer opportunities to pull information baked into its parameters. This reduces the chances that an LLM will leak sensitive data, or ‘hallucinate’ incorrect or misleading information.\n",
    "\n",
    "\n",
    "The orchestration of the retriever and generator will be done using Langchain. A specialized function from Langchain allows us to create the receiver-generator in one line of code.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "gIFJpZ_3D-wu",
    "outputId": "4086ba33-b814-48a8-9c48-312e7fcda031"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (4.34.1)\n",
      "Requirement already satisfied: accelerate in /usr/local/lib/python3.10/dist-packages (0.23.0)\n",
      "Requirement already satisfied: einops in /usr/local/lib/python3.10/dist-packages (0.7.0)\n",
      "Requirement already satisfied: langchain in /usr/local/lib/python3.10/dist-packages (0.0.321)\n",
      "Requirement already satisfied: xformers in /usr/local/lib/python3.10/dist-packages (0.0.22.post4)\n",
      "Requirement already satisfied: bitsandbytes in /usr/local/lib/python3.10/dist-packages (0.41.1)\n",
      "Requirement already satisfied: lancedb in /usr/local/lib/python3.10/dist-packages (0.3.1)\n",
      "Requirement already satisfied: sentence_transformers in /usr/local/lib/python3.10/dist-packages (2.2.2)\n",
      "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers) (3.12.4)\n",
      "Requirement already satisfied: huggingface-hub<1.0,>=0.16.4 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.17.3)\n",
      "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (1.23.5)\n",
      "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (23.2)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (6.0.1)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (2023.6.3)\n",
      "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers) (2.31.0)\n",
      "Requirement already satisfied: tokenizers<0.15,>=0.14 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.14.1)\n",
      "Requirement already satisfied: safetensors>=0.3.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.4.0)\n",
      "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers) (4.66.1)\n",
      "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate) (5.9.5)\n",
      "Requirement already satisfied: torch>=1.10.0 in /usr/local/lib/python3.10/dist-packages (from accelerate) (2.1.0+cu118)\n",
      "Requirement already satisfied: SQLAlchemy<3,>=1.4 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.0.22)\n",
      "Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /usr/local/lib/python3.10/dist-packages (from langchain) (3.8.6)\n",
      "Requirement already satisfied: anyio<4.0 in /usr/local/lib/python3.10/dist-packages (from langchain) (3.7.1)\n",
      "Requirement already satisfied: async-timeout<5.0.0,>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from langchain) (4.0.3)\n",
      "Requirement already satisfied: dataclasses-json<0.7,>=0.5.7 in /usr/local/lib/python3.10/dist-packages (from langchain) (0.6.1)\n",
      "Requirement already satisfied: jsonpatch<2.0,>=1.33 in /usr/local/lib/python3.10/dist-packages (from langchain) (1.33)\n",
      "Requirement already satisfied: langsmith<0.1.0,>=0.0.43 in /usr/local/lib/python3.10/dist-packages (from langchain) (0.0.49)\n",
      "Requirement already satisfied: pydantic<3,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain) (1.10.13)\n",
      "Requirement already satisfied: tenacity<9.0.0,>=8.1.0 in /usr/local/lib/python3.10/dist-packages (from langchain) (8.2.3)\n",
      "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (4.5.0)\n",
      "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (1.12)\n",
      "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (3.2)\n",
      "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (3.1.2)\n",
      "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (2023.6.0)\n",
      "Requirement already satisfied: triton==2.1.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (2.1.0)\n",
      "Requirement already satisfied: deprecation in /usr/local/lib/python3.10/dist-packages (from lancedb) (2.1.0)\n",
      "Requirement already satisfied: pylance==0.8.3 in /usr/local/lib/python3.10/dist-packages (from lancedb) (0.8.3)\n",
      "Requirement already satisfied: ratelimiter~=1.0 in /usr/local/lib/python3.10/dist-packages (from lancedb) (1.2.0.post0)\n",
      "Requirement already satisfied: retry>=0.9.2 in /usr/local/lib/python3.10/dist-packages (from lancedb) (0.9.2)\n",
      "Requirement already satisfied: attrs>=21.3.0 in /usr/local/lib/python3.10/dist-packages (from lancedb) (23.1.0)\n",
      "Requirement already satisfied: semver>=3.0 in /usr/local/lib/python3.10/dist-packages (from lancedb) (3.0.2)\n",
      "Requirement already satisfied: cachetools in /usr/local/lib/python3.10/dist-packages (from lancedb) (5.3.1)\n",
      "Requirement already satisfied: click>=8.1.7 in /usr/local/lib/python3.10/dist-packages (from lancedb) (8.1.7)\n",
      "Requirement already satisfied: pyarrow>=10 in /usr/local/lib/python3.10/dist-packages (from pylance==0.8.3->lancedb) (13.0.0)\n",
      "Requirement already satisfied: torchvision in /usr/local/lib/python3.10/dist-packages (from sentence_transformers) (0.16.0+cu118)\n",
      "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (from sentence_transformers) (1.2.2)\n",
      "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from sentence_transformers) (1.11.3)\n",
      "Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (from sentence_transformers) (3.8.1)\n",
      "Requirement already satisfied: sentencepiece in /usr/local/lib/python3.10/dist-packages (from sentence_transformers) (0.1.99)\n",
      "Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (3.3.0)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (6.0.4)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.9.2)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.4.0)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.3.1)\n",
      "Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/dist-packages (from anyio<4.0->langchain) (3.4)\n",
      "Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.10/dist-packages (from anyio<4.0->langchain) (1.3.0)\n",
      "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<4.0->langchain) (1.1.3)\n",
      "Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /usr/local/lib/python3.10/dist-packages (from dataclasses-json<0.7,>=0.5.7->langchain) (3.20.1)\n",
      "Requirement already satisfied: typing-inspect<1,>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from dataclasses-json<0.7,>=0.5.7->langchain) (0.9.0)\n",
      "Requirement already satisfied: jsonpointer>=1.9 in /usr/local/lib/python3.10/dist-packages (from jsonpatch<2.0,>=1.33->langchain) (2.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2.0.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2023.7.22)\n",
      "Requirement already satisfied: decorator>=3.4.2 in /usr/local/lib/python3.10/dist-packages (from retry>=0.9.2->lancedb) (4.4.2)\n",
      "Requirement already satisfied: py<2.0.0,>=1.4.26 in /usr/local/lib/python3.10/dist-packages (from retry>=0.9.2->lancedb) (1.11.0)\n",
      "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/dist-packages (from SQLAlchemy<3,>=1.4->langchain) (3.0.0)\n",
      "Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nltk->sentence_transformers) (1.3.2)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->sentence_transformers) (3.2.0)\n",
      "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.10/dist-packages (from torchvision->sentence_transformers) (9.4.0)\n",
      "Requirement already satisfied: mypy-extensions>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7,>=0.5.7->langchain) (1.0.0)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.10.0->accelerate) (2.1.3)\n",
      "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.10.0->accelerate) (1.3.0)\n"
     ]
    }
   ],
   "source": [
    "# Installation\n",
    "\n",
    "!pip install transformers accelerate einops langchain xformers bitsandbytes lancedb sentence_transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tfb94kwvLtU1"
   },
   "outputs": [],
   "source": [
    "# imports\n",
    "\n",
    "from torch import cuda, bfloat16\n",
    "import torch\n",
    "import transformers\n",
    "from transformers import AutoTokenizer\n",
    "from time import time\n",
    "import lancedb\n",
    "from langchain.llms import HuggingFacePipeline\n",
    "from langchain.document_loaders import TextLoader\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "from langchain.embeddings import HuggingFaceEmbeddings\n",
    "from langchain.chains import RetrievalQA\n",
    "from langchain.vectorstores import LanceDB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "NGvhj6XWI2YV",
    "outputId": "4ea9aa3d-3492-47f2-864b-75db252ecbe1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-01-25 01:04:50--  https://gist.githubusercontent.com/PrashantDixit0/10fd4ab8a7d0d37de361af2a06ecfbe2/raw/indianEconomy.txt\n",
      "Resolving gist.githubusercontent.com (gist.githubusercontent.com)... "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "64:ff9b::b9c7:6c85, 64:ff9b::b9c7:6d85, 64:ff9b::b9c7:6e85, ...\n",
      "Connecting to gist.githubusercontent.com (gist.githubusercontent.com)|64:ff9b::b9c7:6c85|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 2620 (2.6K) [text/plain]\n",
      "Saving to: ‘indianEconomy.txt’\n",
      "\n",
      "indianEconomy.txt   100%[===================>]   2.56K  --.-KB/s    in 0s      \n",
      "\n",
      "2024-01-25 01:04:51 (11.4 MB/s) - ‘indianEconomy.txt’ saved [2620/2620]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Dataset(in txt format)\n",
    "\n",
    "!wget https://gist.githubusercontent.com/PrashantDixit0/10fd4ab8a7d0d37de361af2a06ecfbe2/raw/indianEconomy.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8IoDr-jZJFVm"
   },
   "outputs": [],
   "source": [
    "# load data\n",
    "\n",
    "dataloader = TextLoader(\"indianEconomy.txt\", encoding=\"utf8\")\n",
    "documents = dataloader.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sh05Xli7MJMh"
   },
   "source": [
    "## Text Chunking\n",
    "\n",
    "We have discussed various types Text Chunking for LLM Applications and Tips and Tricks related to it.\n",
    "\n",
    "Refer - https://medium.com/p/a420efc96a13/edit\n",
    "\n",
    "Here you can try out different Text Spplitting Strategies according to your data and Tips and Tricks discussed in Blog.\n",
    "\n",
    "For Now, we are going to use Recursive Text Splitting using LangChain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "r0om7g1nMEOk"
   },
   "outputs": [],
   "source": [
    "recursive_text_splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size=10000, chunk_overlap=200\n",
    ")\n",
    "all_splits = recursive_text_splitter.split_documents(documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DxqPNUpgNprN"
   },
   "source": [
    "## Embeddings Generator\n",
    "\n",
    "Creating embeddings using Sentence Transformer with HuggingFace embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "UCuE42JMNmrz"
   },
   "outputs": [],
   "source": [
    "model_name = \"sentence-transformers/all-mpnet-base-v2\"\n",
    "model_kwargs = {\"device\": \"cuda\"}\n",
    "\n",
    "embeddings = HuggingFaceEmbeddings(model_name=model_name, model_kwargs=model_kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_Og-UmALOYwV"
   },
   "source": [
    "## LanceDB for vector storage and searching\n",
    "\n",
    "Initialize LanceDB with the Recursive Text Chunking, the *embeddings*   Sentence Transformer object will be used for extract embeddings from all the text splits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vqO6knYcOe1K"
   },
   "outputs": [],
   "source": [
    "db = lancedb.connect(\"/tmp/lancedb\")\n",
    "table = db.create_table(\n",
    "    \"rag_table\",\n",
    "    data=[\n",
    "        {\n",
    "            \"vector\": embeddings.embed_query(\"Indian Economoy\"),\n",
    "            \"text\": \"Current and future details of Indian Economy\",\n",
    "            \"id\": \"1\",\n",
    "        }\n",
    "    ],\n",
    "    mode=\"overwrite\",\n",
    ")\n",
    "\n",
    "vectordb = LanceDB.from_documents(documents, embeddings, connection=table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zBW-Yp7mPxQY"
   },
   "source": [
    "## Initialize RAG Chain\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Aa95aAbP6ukc"
   },
   "source": [
    "## Chat Models\n",
    "\n",
    "Here you can change to any other LLM for Chat Model.\n",
    "\n",
    "Refer to LangChain, There are few Chat Models which can be used as Chat model to generate answers in RAG.\n",
    "https://python.langchain.com/docs/integrations/chat/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-3oNrEEZy6qe"
   },
   "outputs": [],
   "source": [
    "from langchain.chat_models import ChatOpenAI\n",
    "\n",
    "llm = ChatOpenAI(\n",
    "    temperature=0,\n",
    "    openai_api_key=\"YOUR_API_KEY\",\n",
    "    openai_organization=\"YOUR_ORGANIZATION_ID\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zdhU5x1aPwma"
   },
   "outputs": [],
   "source": [
    "# Retreiver\n",
    "retriever = vectordb.as_retriever()\n",
    "\n",
    "qa = RetrievalQA.from_chain_type(\n",
    "    llm=llm, chain_type=\"stuff\", retriever=retriever, verbose=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8XxQSWUHOiPG"
   },
   "outputs": [],
   "source": [
    "# Results of RAG\n",
    "\n",
    "query = \"What is growth of Indian Economy?\"\n",
    "\n",
    "result = qa({\"query\": query})\n",
    "\n",
    "print(result)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "T4",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}