18-better-rag-with-lotr-and-rerank.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Here we are about to discuss 2 key enhancements for RAG\n",
    "##### 1. When you have diferent kind of documents, you can't create a single index. you have to create multiple indexs and have to index via all of them seperately to perform the RAG. we are going to see how we can do a better RAG in this case using Merger Retriever (LOTR)\n",
    "##### 2. Lost in the Middle ! When your Context is long, we organize your documents as high-rank (high similarity) documents in top and bottom and put low / average documents in the middle using `LongContextReorder`. Because in `Lost in the Middle` what happens is response synthesizer only focus on the top and bottom documents and ignore the middle documents. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Merger Retriever (LOTR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os, yaml, cohere\n",
    "from langchain.vectorstores import Chroma\n",
    "from langchain.chat_models import AzureChatOpenAI\n",
    "from langchain.embeddings import HuggingFaceBgeEmbeddings\n",
    "from langchain.document_loaders import PyPDFLoader, TextLoader\n",
    "from langchain.retrievers.merger_retriever import MergerRetriever\n",
    "from langchain.document_transformers import (\n",
    "                                            EmbeddingsRedundantFilter,\n",
    "                                            EmbeddingsClusteringFilter,\n",
    "                                            LongContextReorder\n",
    "                                            )\n",
    "from langchain.retrievers.document_compressors import DocumentCompressorPipeline\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "from langchain.retrievers import ContextualCompressionRetriever"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('cadentials.yaml') as f:\n",
    "    credentials = yaml.load(f, Loader=yaml.FullLoader)\n",
    "\n",
    "os.environ['OPENAI_API_KEY'] = credentials['OPENAI_API_KEY']\n",
    "os.environ['OPENAI_API_TYPE'] = credentials['OPENAI_API_TYPE']\n",
    "os.environ['AZURE_OPENAI_ENDPOINT'] = credentials['AD_OPENAI_API_BASE']\n",
    "os.environ['OPENAI_API_VERSION'] = credentials['AD_OPENAI_API_VERSION']\n",
    "os.environ[\"COHERE_API\"] = credentials['COHERE_API']\n",
    "os.environ['ENGINE'] = credentials['ENGINE']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding = HuggingFaceBgeEmbeddings(\n",
    "                                    model_name=\"BAAI/bge-small-en-v1.5\",\n",
    "                                    model_kwargs={'device': 'mps'},\n",
    "                                    encode_kwargs={'normalize_embeddings': True}\n",
    "                                    )\n",
    "\n",
    "llm = AzureChatOpenAI(\n",
    "                    deployment_name=credentials['AD_DEPLOYMENT_ID'],\n",
    "                    model_name=credentials['AD_ENGINE'],\n",
    "                    temperature=0.9, \n",
    "                    max_tokens=256\n",
    "                    )\n",
    "\n",
    "co = cohere.Client(os.environ[\"COHERE_API\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "loader_un_sdg = PyPDFLoader(\"data/political/UN SDG.pdf\")\n",
    "documents_un_sdg = loader_un_sdg.load()\n",
    "text_splitter_un_sdg = RecursiveCharacterTextSplitter(\n",
    "                                                    chunk_size=1000,\n",
    "                                                    chunk_overlap=100\n",
    "                                                    )\n",
    "texts_un_sdg = text_splitter_un_sdg.split_documents(documents_un_sdg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(page_content='TRANSFORMING OUR WORLD:\\nTHE 2030 AGENDA FOR \\nSUST AINABLE DEVELOPMENTUNITED NA TIONS', metadata={'source': 'data/political/UN SDG.pdf', 'page': 0})"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "texts_un_sdg[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "loader_paris_agreement = PyPDFLoader(\"data/political/english_paris_agreement.pdf\")\n",
    "documents_paris_agreement = loader_paris_agreement.load()\n",
    "text_splitter_paris_agreement = RecursiveCharacterTextSplitter(\n",
    "                                                                chunk_size=1000,\n",
    "                                                                chunk_overlap=100\n",
    "                                                                )\n",
    "texts_paris_agreement = text_splitter_paris_agreement.split_documents(documents_paris_agreement)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(page_content='PARIS AGREEMENT \\n(mm \\nUNITED NATIONS \\n2015', metadata={'source': 'data/political/english_paris_agreement.pdf', 'page': 0})"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "texts_paris_agreement[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vector Store"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not os.path.exists(\"db/18/un_sdg_chroma_cosine\"):\n",
    "    un_sdg_store = Chroma.from_documents(\n",
    "                                        texts_un_sdg, \n",
    "                                        embedding, \n",
    "                                        collection_metadata={\"hnsw:space\": \"cosine\"}, \n",
    "                                        persist_directory=\"db/18/un_sdg_chroma_cosine\"\n",
    "                                        )\n",
    "else:\n",
    "    un_sdg_store = Chroma(\n",
    "                        persist_directory=\"db/18/un_sdg_chroma_cosine\",\n",
    "                        embedding_function=embedding\n",
    "                        )\n",
    "\n",
    "if not os.path.exists(\"db/18/paris_chroma_cosine\"):\n",
    "    paris_agreement_store = Chroma.from_documents(\n",
    "                                        texts_paris_agreement, \n",
    "                                        embedding, \n",
    "                                        collection_metadata={\"hnsw:space\": \"cosine\"}, \n",
    "                                        persist_directory=\"db/18/paris_chroma_cosine\"\n",
    "                                        )\n",
    "else:\n",
    "    paris_agreement_store = Chroma(\n",
    "                        persist_directory=\"db/18/paris_chroma_cosine\",\n",
    "                        embedding_function=embedding\n",
    "                        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build Merged Retriever"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MergerRetriever(retrievers=[VectorStoreRetriever(tags=['Chroma', 'HuggingFaceBgeEmbeddings'], vectorstore=<langchain.vectorstores.chroma.Chroma object at 0x107e146a0>, search_kwargs={'k': 3, 'include_metadata': True}), VectorStoreRetriever(tags=['Chroma', 'HuggingFaceBgeEmbeddings'], vectorstore=<langchain.vectorstores.chroma.Chroma object at 0x2b47d50d0>, search_kwargs={'k': 3, 'include_metadata': True})])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retriever_un_sdg = un_sdg_store.as_retriever(\n",
    "                                            search_type = \"similarity\", \n",
    "                                            search_kwargs = {\n",
    "                                                            \"k\":3, \n",
    "                                                            \"include_metadata\": True\n",
    "                                                            }\n",
    "                                            )\n",
    "\n",
    "retriever_paris_agreement = paris_agreement_store.as_retriever(\n",
    "                                                                search_type = \"similarity\", \n",
    "                                                                search_kwargs = {\n",
    "                                                                                \"k\":3, \n",
    "                                                                                \"include_metadata\": True\n",
    "                                                                                }\n",
    "                                            )\n",
    "lotr = MergerRetriever(retrievers=[retriever_un_sdg, retriever_paris_agreement])\n",
    "lotr\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "resilience+ and+ adaptive+ capacity+ to+ climateJrelated+ hazards+ and+ natural+disasters+in+all+countries++13.2+Integrate+climate+change+measures+into+national+policies,+strategies+and+planning++13.3+Improve+education,+awarenessJraising+and+human+and+institutional+capacity+on+climate+change+mitigation,+adaptation,+impact+reduction+and+early+warning+13.a+Implement+the+commitment+undertaken+by+developedJcountry+parties+to+the+United+Nations+Framework+Convention+on+Climate+Change+to+a+goal+of+mobilizing+jointly+$100+billion+annually+by+2020+from+all+sources+to+address+the+needs+of+developing+countries+in+the+context+of+meaningful+mitigation+actions+and+transparency+on+implementation+and+fully+operationalize+the+Green+Climate+Fund+through+its+capitalization+as+soon+as+possible+++*+Acknowledging+that+the+United+Nations+Framework+Convention+on+Climate+Change+is+the+primary+international,+intergovernmental+forum+for+negotiating+the+global+response+to+climate+change.+\n",
      "finance should  represent a progression beyond previous efforts. \n",
      "4. The provision of scaled-up financial resources should aim to achieve a \n",
      "balance between adaptation and mitigation, taking into account country-driven \n",
      "strategies, and the priorities and needs of developing country Parties, especially \n",
      "those that are particularly vulnerable to the adverse effects of climate change and \n",
      "have significant capacity constraints, such as the least developed countries and \n",
      "small island developing States, considering the need for public and grant-based \n",
      "resources for adaptation. \n",
      "5. Developed country Parties shall biennially communicate indicative \n",
      "quantitative and qualitative information related to paragraphs 1 and 3 of this \n",
      "Article, as applicable, including, as available, projected levels of public financial \n",
      "resources to be provided to developing country Parties. Other Parties providing \n",
      "resources are encouraged to communicate biennially such information on a \n",
      "voluntary basis.\n",
      "and+ adaptation+ to+ climate+ change,+ resilience+ to+ disasters,+ and+ develop+ and+implement,+in+line+with+the+Sendai+Framework+for+Disaster+Risk+Reduction+2015J2030,+holistic+disaster+risk+management+at+all+levels++11.c+Support+least+developed+countries,+including+through+financial+and+technical+assistance,+in+building+sustainable+and+resilient+buildings+utilizing+local+materials+++!!Goal!12.!Ensure!sustainable!consumption!and!production!patterns!+12.1+Implement+the+10JYear+Framework+of+Programmes+on+Sustainable+Consumption+and+Production+ Patterns,+ all+ countries+ taking+ action,+ with+ developed+ countries+ taking+ the+ lead,+taking+into+account+the+development+and+capabilities+of+developing+countries+12.2+By+2030,+achieve+the+sustainable+management+and+efficient+use+of+natural+resources++\n",
      "8. Each Party should also provide information related to climate change \n",
      "impacts and adaptation under Article 7, as appropriate. \n",
      "9. Developed country Parties shall, and other Parties that provide support \n",
      "should, provide information on financial, technology transfer and capacity-building \n",
      "support provided to developing country Parties under Articles 9,  10 and 11.\n",
      "environmental+ degradation.+ The+ global+ nature+ of+ climate+ change+ calls+ for+ the+ widest+possible+international+cooperation+aimed+at+accelerating+the+reduction+of+global+greenhouse+gas+emissions+and+addressing+adaptation+to+the+adverse+impacts+of+climate+change.+We+note+with+grave+concern+the+significant+gap+between+the+aggregate+effect+of+parties’+mitigation+pledges+ in+ terms+ of+ global+ annual+ emissions+ of+ greenhouse+ gases+ by+ 2020+ and+ aggregate+emission+pathways+consistent+with+having+a+likely+chance+of+holding+the+increase+in+global+average+ temperature+ below+ 2+ degrees+ Celsius+ or+ 1.5+ degrees+ Celsius+ above+ preJindustrial+levels.+32.+Looking+ahead+to+the+twentyJfirst+session+of+the+Conference+of+the+Parties+in+Paris,+we+underscore+ the+ commitment+ of+ all+ States+ to+ work+ for+an+ ambitious+ and+ universal+\n",
      "5. Accelerating, encouraging and enabling innovation is critical for an \n",
      "effective, long-term global response to climate change and promoting economic \n",
      "growth and sustainable development. Such effort shall be, as appropriate, \n",
      "supported, including by the Technology Mechanism and, through financial means, \n",
      "by the Financial Mechanism of the Convention, for collaborative approaches to \n",
      "research and development, and facilitating access to technology, in particular for \n",
      "early stages of the technology cycle, to developing country Parties. \n",
      "6. Support, including  financial support, shall be provided to developing country \n",
      "Parties for the implementation of this Article, including for strengthening \n",
      "cooperative action on technology development and transfer at different stages of \n",
      "the technology cycle, with a view to achieving a balance between support for \n",
      "mitigation and adaptation. The global stocktake referred to in Article 14 shall take\n"
     ]
    }
   ],
   "source": [
    "for chunks in lotr.get_relevant_documents(\"Is there any framework available to tackle the climate change?\"):\n",
    "    print(chunks.page_content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='resilience+ and+ adaptive+ capacity+ to+ climateJrelated+ hazards+ and+ natural+disasters+in+all+countries++13.2+Integrate+climate+change+measures+into+national+policies,+strategies+and+planning++13.3+Improve+education,+awarenessJraising+and+human+and+institutional+capacity+on+climate+change+mitigation,+adaptation,+impact+reduction+and+early+warning+13.a+Implement+the+commitment+undertaken+by+developedJcountry+parties+to+the+United+Nations+Framework+Convention+on+Climate+Change+to+a+goal+of+mobilizing+jointly+$100+billion+annually+by+2020+from+all+sources+to+address+the+needs+of+developing+countries+in+the+context+of+meaningful+mitigation+actions+and+transparency+on+implementation+and+fully+operationalize+the+Green+Climate+Fund+through+its+capitalization+as+soon+as+possible+++*+Acknowledging+that+the+United+Nations+Framework+Convention+on+Climate+Change+is+the+primary+international,+intergovernmental+forum+for+negotiating+the+global+response+to+climate+change.+', metadata={'page': 26, 'source': 'data/political/UN SDG.pdf'}),\n",
       " Document(page_content='finance should  represent a progression beyond previous efforts. \\n4. The provision of scaled-up financial resources should aim to achieve a \\nbalance between adaptation and mitigation, taking into account country-driven \\nstrategies, and the priorities and needs of developing country Parties, especially \\nthose that are particularly vulnerable to the adverse effects of climate change and \\nhave significant capacity constraints, such as the least developed countries and \\nsmall island developing States, considering the need for public and grant-based \\nresources for adaptation. \\n5. Developed country Parties shall biennially communicate indicative \\nquantitative and qualitative information related to paragraphs 1 and 3 of this \\nArticle, as applicable, including, as available, projected levels of public financial \\nresources to be provided to developing country Parties. Other Parties providing \\nresources are encouraged to communicate biennially such information on a \\nvoluntary basis.', metadata={'page': 14, 'source': 'data/political/english_paris_agreement.pdf'}),\n",
       " Document(page_content='and+ adaptation+ to+ climate+ change,+ resilience+ to+ disasters,+ and+ develop+ and+implement,+in+line+with+the+Sendai+Framework+for+Disaster+Risk+Reduction+2015J2030,+holistic+disaster+risk+management+at+all+levels++11.c+Support+least+developed+countries,+including+through+financial+and+technical+assistance,+in+building+sustainable+and+resilient+buildings+utilizing+local+materials+++!!Goal!12.!Ensure!sustainable!consumption!and!production!patterns!+12.1+Implement+the+10JYear+Framework+of+Programmes+on+Sustainable+Consumption+and+Production+ Patterns,+ all+ countries+ taking+ action,+ with+ developed+ countries+ taking+ the+ lead,+taking+into+account+the+development+and+capabilities+of+developing+countries+12.2+By+2030,+achieve+the+sustainable+management+and+efficient+use+of+natural+resources++', metadata={'page': 25, 'source': 'data/political/UN SDG.pdf'}),\n",
       " Document(page_content='8. Each Party should also provide information related to climate change \\nimpacts and adaptation under Article 7, as appropriate. \\n9. Developed country Parties shall, and other Parties that provide support \\nshould, provide information on financial, technology transfer and capacity-building \\nsupport provided to developing country Parties under Articles 9,  10 and 11.', metadata={'page': 18, 'source': 'data/political/english_paris_agreement.pdf'}),\n",
       " Document(page_content='environmental+ degradation.+ The+ global+ nature+ of+ climate+ change+ calls+ for+ the+ widest+possible+international+cooperation+aimed+at+accelerating+the+reduction+of+global+greenhouse+gas+emissions+and+addressing+adaptation+to+the+adverse+impacts+of+climate+change.+We+note+with+grave+concern+the+significant+gap+between+the+aggregate+effect+of+parties’+mitigation+pledges+ in+ terms+ of+ global+ annual+ emissions+ of+ greenhouse+ gases+ by+ 2020+ and+ aggregate+emission+pathways+consistent+with+having+a+likely+chance+of+holding+the+increase+in+global+average+ temperature+ below+ 2+ degrees+ Celsius+ or+ 1.5+ degrees+ Celsius+ above+ preJindustrial+levels.+32.+Looking+ahead+to+the+twentyJfirst+session+of+the+Conference+of+the+Parties+in+Paris,+we+underscore+ the+ commitment+ of+ all+ States+ to+ work+ for+an+ ambitious+ and+ universal+', metadata={'page': 11, 'source': 'data/political/UN SDG.pdf'}),\n",
       " Document(page_content='5. Accelerating, encouraging and enabling innovation is critical for an \\neffective, long-term global response to climate change and promoting economic \\ngrowth and sustainable development. Such effort shall be, as appropriate, \\nsupported, including by the Technology Mechanism and, through financial means, \\nby the Financial Mechanism of the Convention, for collaborative approaches to \\nresearch and development, and facilitating access to technology, in particular for \\nearly stages of the technology cycle, to developing country Parties. \\n6. Support, including  financial support, shall be provided to developing country \\nParties for the implementation of this Article, including for strengthening \\ncooperative action on technology development and transfer at different stages of \\nthe technology cycle, with a view to achieving a balance between support for \\nmitigation and adaptation. The global stocktake referred to in Article 14 shall take', metadata={'page': 16, 'source': 'data/political/english_paris_agreement.pdf'})]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs = lotr.get_relevant_documents(\"Is there any framework available to tackle the climate change?\")\n",
    "docs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Long Context Reorder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='finance should  represent a progression beyond previous efforts. \\n4. The provision of scaled-up financial resources should aim to achieve a \\nbalance between adaptation and mitigation, taking into account country-driven \\nstrategies, and the priorities and needs of developing country Parties, especially \\nthose that are particularly vulnerable to the adverse effects of climate change and \\nhave significant capacity constraints, such as the least developed countries and \\nsmall island developing States, considering the need for public and grant-based \\nresources for adaptation. \\n5. Developed country Parties shall biennially communicate indicative \\nquantitative and qualitative information related to paragraphs 1 and 3 of this \\nArticle, as applicable, including, as available, projected levels of public financial \\nresources to be provided to developing country Parties. Other Parties providing \\nresources are encouraged to communicate biennially such information on a \\nvoluntary basis.', metadata={'page': 14, 'source': 'data/political/english_paris_agreement.pdf'}),\n",
       " Document(page_content='8. Each Party should also provide information related to climate change \\nimpacts and adaptation under Article 7, as appropriate. \\n9. Developed country Parties shall, and other Parties that provide support \\nshould, provide information on financial, technology transfer and capacity-building \\nsupport provided to developing country Parties under Articles 9,  10 and 11.', metadata={'page': 18, 'source': 'data/political/english_paris_agreement.pdf'}),\n",
       " Document(page_content='5. Accelerating, encouraging and enabling innovation is critical for an \\neffective, long-term global response to climate change and promoting economic \\ngrowth and sustainable development. Such effort shall be, as appropriate, \\nsupported, including by the Technology Mechanism and, through financial means, \\nby the Financial Mechanism of the Convention, for collaborative approaches to \\nresearch and development, and facilitating access to technology, in particular for \\nearly stages of the technology cycle, to developing country Parties. \\n6. Support, including  financial support, shall be provided to developing country \\nParties for the implementation of this Article, including for strengthening \\ncooperative action on technology development and transfer at different stages of \\nthe technology cycle, with a view to achieving a balance between support for \\nmitigation and adaptation. The global stocktake referred to in Article 14 shall take', metadata={'page': 16, 'source': 'data/political/english_paris_agreement.pdf'}),\n",
       " Document(page_content='environmental+ degradation.+ The+ global+ nature+ of+ climate+ change+ calls+ for+ the+ widest+possible+international+cooperation+aimed+at+accelerating+the+reduction+of+global+greenhouse+gas+emissions+and+addressing+adaptation+to+the+adverse+impacts+of+climate+change.+We+note+with+grave+concern+the+significant+gap+between+the+aggregate+effect+of+parties’+mitigation+pledges+ in+ terms+ of+ global+ annual+ emissions+ of+ greenhouse+ gases+ by+ 2020+ and+ aggregate+emission+pathways+consistent+with+having+a+likely+chance+of+holding+the+increase+in+global+average+ temperature+ below+ 2+ degrees+ Celsius+ or+ 1.5+ degrees+ Celsius+ above+ preJindustrial+levels.+32.+Looking+ahead+to+the+twentyJfirst+session+of+the+Conference+of+the+Parties+in+Paris,+we+underscore+ the+ commitment+ of+ all+ States+ to+ work+ for+an+ ambitious+ and+ universal+', metadata={'page': 11, 'source': 'data/political/UN SDG.pdf'}),\n",
       " Document(page_content='and+ adaptation+ to+ climate+ change,+ resilience+ to+ disasters,+ and+ develop+ and+implement,+in+line+with+the+Sendai+Framework+for+Disaster+Risk+Reduction+2015J2030,+holistic+disaster+risk+management+at+all+levels++11.c+Support+least+developed+countries,+including+through+financial+and+technical+assistance,+in+building+sustainable+and+resilient+buildings+utilizing+local+materials+++!!Goal!12.!Ensure!sustainable!consumption!and!production!patterns!+12.1+Implement+the+10JYear+Framework+of+Programmes+on+Sustainable+Consumption+and+Production+ Patterns,+ all+ countries+ taking+ action,+ with+ developed+ countries+ taking+ the+ lead,+taking+into+account+the+development+and+capabilities+of+developing+countries+12.2+By+2030,+achieve+the+sustainable+management+and+efficient+use+of+natural+resources++', metadata={'page': 25, 'source': 'data/political/UN SDG.pdf'}),\n",
       " Document(page_content='resilience+ and+ adaptive+ capacity+ to+ climateJrelated+ hazards+ and+ natural+disasters+in+all+countries++13.2+Integrate+climate+change+measures+into+national+policies,+strategies+and+planning++13.3+Improve+education,+awarenessJraising+and+human+and+institutional+capacity+on+climate+change+mitigation,+adaptation,+impact+reduction+and+early+warning+13.a+Implement+the+commitment+undertaken+by+developedJcountry+parties+to+the+United+Nations+Framework+Convention+on+Climate+Change+to+a+goal+of+mobilizing+jointly+$100+billion+annually+by+2020+from+all+sources+to+address+the+needs+of+developing+countries+in+the+context+of+meaningful+mitigation+actions+and+transparency+on+implementation+and+fully+operationalize+the+Green+Climate+Fund+through+its+capitalization+as+soon+as+possible+++*+Acknowledging+that+the+United+Nations+Framework+Convention+on+Climate+Change+is+the+primary+international,+intergovernmental+forum+for+negotiating+the+global+response+to+climate+change.+', metadata={'page': 26, 'source': 'data/political/UN SDG.pdf'})]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reordering = LongContextReorder()\n",
    "reordered_docs = reordering.transform_documents(docs)\n",
    "reordered_docs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Rerank"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['resilience+ and+ adaptive+ capacity+ to+ climateJrelated+ hazards+ and+ natural+disasters+in+all+countries++13.2+Integrate+climate+change+measures+into+national+policies,+strategies+and+planning++13.3+Improve+education,+awarenessJraising+and+human+and+institutional+capacity+on+climate+change+mitigation,+adaptation,+impact+reduction+and+early+warning+13.a+Implement+the+commitment+undertaken+by+developedJcountry+parties+to+the+United+Nations+Framework+Convention+on+Climate+Change+to+a+goal+of+mobilizing+jointly+$100+billion+annually+by+2020+from+all+sources+to+address+the+needs+of+developing+countries+in+the+context+of+meaningful+mitigation+actions+and+transparency+on+implementation+and+fully+operationalize+the+Green+Climate+Fund+through+its+capitalization+as+soon+as+possible+++*+Acknowledging+that+the+United+Nations+Framework+Convention+on+Climate+Change+is+the+primary+international,+intergovernmental+forum+for+negotiating+the+global+response+to+climate+change.+',\n",
       " 'and+ adaptation+ to+ climate+ change,+ resilience+ to+ disasters,+ and+ develop+ and+implement,+in+line+with+the+Sendai+Framework+for+Disaster+Risk+Reduction+2015J2030,+holistic+disaster+risk+management+at+all+levels++11.c+Support+least+developed+countries,+including+through+financial+and+technical+assistance,+in+building+sustainable+and+resilient+buildings+utilizing+local+materials+++!!Goal!12.!Ensure!sustainable!consumption!and!production!patterns!+12.1+Implement+the+10JYear+Framework+of+Programmes+on+Sustainable+Consumption+and+Production+ Patterns,+ all+ countries+ taking+ action,+ with+ developed+ countries+ taking+ the+ lead,+taking+into+account+the+development+and+capabilities+of+developing+countries+12.2+By+2030,+achieve+the+sustainable+management+and+efficient+use+of+natural+resources++',\n",
       " 'finance should  represent a progression beyond previous efforts. \\n4. The provision of scaled-up financial resources should aim to achieve a \\nbalance between adaptation and mitigation, taking into account country-driven \\nstrategies, and the priorities and needs of developing country Parties, especially \\nthose that are particularly vulnerable to the adverse effects of climate change and \\nhave significant capacity constraints, such as the least developed countries and \\nsmall island developing States, considering the need for public and grant-based \\nresources for adaptation. \\n5. Developed country Parties shall biennially communicate indicative \\nquantitative and qualitative information related to paragraphs 1 and 3 of this \\nArticle, as applicable, including, as available, projected levels of public financial \\nresources to be provided to developing country Parties. Other Parties providing \\nresources are encouraged to communicate biennially such information on a \\nvoluntary basis.',\n",
       " '8. Each Party should also provide information related to climate change \\nimpacts and adaptation under Article 7, as appropriate. \\n9. Developed country Parties shall, and other Parties that provide support \\nshould, provide information on financial, technology transfer and capacity-building \\nsupport provided to developing country Parties under Articles 9,  10 and 11.',\n",
       " '5. Accelerating, encouraging and enabling innovation is critical for an \\neffective, long-term global response to climate change and promoting economic \\ngrowth and sustainable development. Such effort shall be, as appropriate, \\nsupported, including by the Technology Mechanism and, through financial means, \\nby the Financial Mechanism of the Convention, for collaborative approaches to \\nresearch and development, and facilitating access to technology, in particular for \\nearly stages of the technology cycle, to developing country Parties. \\n6. Support, including  financial support, shall be provided to developing country \\nParties for the implementation of this Article, including for strengthening \\ncooperative action on technology development and transfer at different stages of \\nthe technology cycle, with a view to achieving a balance between support for \\nmitigation and adaptation. The global stocktake referred to in Article 14 shall take',\n",
       " 'environmental+ degradation.+ The+ global+ nature+ of+ climate+ change+ calls+ for+ the+ widest+possible+international+cooperation+aimed+at+accelerating+the+reduction+of+global+greenhouse+gas+emissions+and+addressing+adaptation+to+the+adverse+impacts+of+climate+change.+We+note+with+grave+concern+the+significant+gap+between+the+aggregate+effect+of+parties’+mitigation+pledges+ in+ terms+ of+ global+ annual+ emissions+ of+ greenhouse+ gases+ by+ 2020+ and+ aggregate+emission+pathways+consistent+with+having+a+likely+chance+of+holding+the+increase+in+global+average+ temperature+ below+ 2+ degrees+ Celsius+ or+ 1.5+ degrees+ Celsius+ above+ preJindustrial+levels.+32.+Looking+ahead+to+the+twentyJfirst+session+of+the+Conference+of+the+Parties+in+Paris,+we+underscore+ the+ commitment+ of+ all+ States+ to+ work+ for+an+ ambitious+ and+ universal+']"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rerank_docs = co.rerank(\n",
    "                        query=\"Is there any framework available to tackle the climate change?\", \n",
    "                        documents=[doc_.page_content for doc_ in docs], \n",
    "                        top_n=6, \n",
    "                        model=\"rerank-english-v2.0\"\n",
    "                        )\n",
    "rerank_docs = [doc.document[\"text\"] for doc in rerank_docs]\n",
    "rerank_docs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### You can clearly see documents have been re-ordered"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llamaindex",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}