1_LangChain_Obsidian_loader.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LangChain with data loaded from Obsidian \n",
    "\n",
    "[Obsidian](https://obsidian.md/) is a powerful and extensible knowledge base that works on top of your local folder of plain text files.   \n",
    "Obsidian store Markdown files on disk as a folder. For more details see [how Obsidian stores data](\n",
    "https://help.obsidian.md/Files+and+folders/How+Obsidian+stores+data#:~:text=Obsidian%20stores%20your%20notes%20as,file%20system%2C%20including%20any%20subfolders).  \n",
    "\n",
    "\n",
    "LangChain allow apply prompt to documents. It owns documents loader including for obsidian.   \n",
    "For more detail see the [documentation](https://python.langchain.com/docs/integrations/document_loaders/obsidian). \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load envirenmental variables from .env file \n",
    "from dotenv import load_dotenv, find_dotenv\n",
    "_ = load_dotenv(find_dotenv()) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'C:\\\\Users\\\\otols\\\\Documents\\\\Obsidian Vault'"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "documents_path = os.path.join(os.getenv('USERPROFILE'), 'Documents')\n",
    "obsidian_path = os.path.join(documents_path, 'Obsidian Vault')\n",
    "\n",
    "obsidian_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert os.path.exists(obsidian_path), f\"Obsidian path does not exist: {obsidian_path}\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content=\"\\n\\nI encourage you to specify why you're interested in the company (scaleway)\", metadata={'source': 'Conseils RH.md', 'path': 'C:\\\\Users\\\\otols\\\\Documents\\\\Obsidian Vault\\\\Conseils RH.md', 'created': 1694683825.1194148, 'last_modified': 1694683862.9424913, 'last_accessed': 1696340970.6644685}),\n",
       " Document(page_content='\\ndispensing with  - обходясь без', metadata={'source': 'English.md', 'path': 'C:\\\\Users\\\\otols\\\\Documents\\\\Obsidian Vault\\\\English.md', 'created': 1693818242.0307033, 'last_modified': 1693818259.6911194, 'last_accessed': 1696340970.6644685}),\n",
       " Document(page_content=' AWS Summit,\\n Pycon \\n HashiConf', metadata={'source': \"Evenements d'ingénérie.md\", 'path': \"C:\\\\Users\\\\otols\\\\Documents\\\\Obsidian Vault\\\\Evenements d'ingénérie.md\", 'created': 1694507620.1801715, 'last_modified': 1695743942.5252457, 'last_accessed': 1696340970.6650107})]"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.document_loaders import ObsidianLoader\n",
    "\n",
    "loader = ObsidianLoader(obsidian_path, encoding='UTF-8', collect_metadata=False) \n",
    "docs = loader.load()\n",
    "\n",
    "docs[:3]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create your index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com\n",
      "Requirement already satisfied: chromadb in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (0.4.13)\n",
      "Requirement already satisfied: pydantic>=1.9 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from chromadb) (2.4.1)\n",
      "Requirement already satisfied: uvicorn[standard]>=0.18.3 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from chromadb) (0.23.2)\n",
      "Requirement already satisfied: requests>=2.28 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from chromadb) (2.31.0)\n",
      "Requirement already satisfied: importlib-resources in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from chromadb) (6.1.0)\n",
      "Requirement already satisfied: numpy>=1.22.5 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from chromadb) (1.26.0)\n",
      "Requirement already satisfied: fastapi>=0.95.2 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from chromadb) (0.103.2)\n",
      "Requirement already satisfied: onnxruntime>=1.14.1 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from chromadb) (1.16.0)\n",
      "Requirement already satisfied: typer>=0.9.0 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from chromadb) (0.9.0)\n",
      "Requirement already satisfied: tqdm>=4.65.0 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from chromadb) (4.66.1)\n",
      "Requirement already satisfied: chroma-hnswlib==0.7.3 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from chromadb) (0.7.3)\n",
      "Requirement already satisfied: overrides>=7.3.1 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from chromadb) (7.4.0)\n",
      "Requirement already satisfied: typing-extensions>=4.5.0 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from chromadb) (4.8.0)\n",
      "Requirement already satisfied: pypika>=0.48.9 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from chromadb) (0.48.9)\n",
      "Requirement already satisfied: posthog>=2.4.0 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from chromadb) (3.0.2)\n",
      "Requirement already satisfied: bcrypt>=4.0.1 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from chromadb) (4.0.1)\n",
      "Requirement already satisfied: tokenizers>=0.13.2 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from chromadb) (0.13.3)\n",
      "Requirement already satisfied: pulsar-client>=3.1.0 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from chromadb) (3.3.0)\n",
      "Requirement already satisfied: starlette<0.28.0,>=0.27.0 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from fastapi>=0.95.2->chromadb) (0.27.0)\n",
      "Requirement already satisfied: anyio<4.0.0,>=3.7.1 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from fastapi>=0.95.2->chromadb) (3.7.1)\n",
      "Requirement already satisfied: coloredlogs in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from onnxruntime>=1.14.1->chromadb) (15.0.1)\n",
      "Requirement already satisfied: protobuf in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from onnxruntime>=1.14.1->chromadb) (4.24.3)\n",
      "Requirement already satisfied: packaging in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from onnxruntime>=1.14.1->chromadb) (23.1)\n",
      "Requirement already satisfied: sympy in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from onnxruntime>=1.14.1->chromadb) (1.12)\n",
      "Requirement already satisfied: flatbuffers in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from onnxruntime>=1.14.1->chromadb) (23.5.26)\n",
      "Requirement already satisfied: monotonic>=1.5 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from posthog>=2.4.0->chromadb) (1.6)\n",
      "Requirement already satisfied: python-dateutil>2.1 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from posthog>=2.4.0->chromadb) (2.8.2)\n",
      "Requirement already satisfied: six>=1.5 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from posthog>=2.4.0->chromadb) (1.16.0)\n",
      "Requirement already satisfied: backoff>=1.10.0 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from posthog>=2.4.0->chromadb) (2.2.1)\n",
      "Requirement already satisfied: certifi in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from pulsar-client>=3.1.0->chromadb) (2023.7.22)\n",
      "Requirement already satisfied: pydantic-core==2.10.1 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from pydantic>=1.9->chromadb) (2.10.1)\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from pydantic>=1.9->chromadb) (0.5.0)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from requests>=2.28->chromadb) (2.0.5)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from requests>=2.28->chromadb) (3.2.0)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from requests>=2.28->chromadb) (3.4)\n",
      "Requirement already satisfied: colorama in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from tqdm>=4.65.0->chromadb) (0.4.6)\n",
      "Requirement already satisfied: click<9.0.0,>=7.1.1 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from typer>=0.9.0->chromadb) (8.1.7)\n",
      "Requirement already satisfied: h11>=0.8 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from uvicorn[standard]>=0.18.3->chromadb) (0.14.0)\n",
      "Requirement already satisfied: pyyaml>=5.1 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from uvicorn[standard]>=0.18.3->chromadb) (6.0.1)\n",
      "Requirement already satisfied: python-dotenv>=0.13 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from uvicorn[standard]>=0.18.3->chromadb) (1.0.0)\n",
      "Requirement already satisfied: httptools>=0.5.0 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from uvicorn[standard]>=0.18.3->chromadb) (0.6.0)\n",
      "Requirement already satisfied: websockets>=10.4 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from uvicorn[standard]>=0.18.3->chromadb) (11.0.3)\n",
      "Requirement already satisfied: watchfiles>=0.13 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from uvicorn[standard]>=0.18.3->chromadb) (0.20.0)\n",
      "Requirement already satisfied: zipp>=3.1.0 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from importlib-resources->chromadb) (3.17.0)\n",
      "Requirement already satisfied: exceptiongroup in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from anyio<4.0.0,>=3.7.1->fastapi>=0.95.2->chromadb) (1.1.3)\n",
      "Requirement already satisfied: sniffio>=1.1 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from anyio<4.0.0,>=3.7.1->fastapi>=0.95.2->chromadb) (1.3.0)\n",
      "Requirement already satisfied: humanfriendly>=9.1 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from coloredlogs->onnxruntime>=1.14.1->chromadb) (10.0)\n",
      "Requirement already satisfied: mpmath>=0.19 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from sympy->onnxruntime>=1.14.1->chromadb) (1.3.0)\n",
      "Requirement already satisfied: pyreadline3 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from humanfriendly>=9.1->coloredlogs->onnxruntime>=1.14.1->chromadb) (3.4.1)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: You are using pip version 22.0.4; however, version 23.2.1 is available.\n",
      "You should consider upgrading via the 'c:\\Users\\otols\\Desktop\\dev\\PromptEngineering\\venv\\Scripts\\python.exe -m pip install --upgrade pip' command.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com\n",
      "Requirement already satisfied: tiktoken in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (0.5.1)\n",
      "Requirement already satisfied: regex>=2022.1.18 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from tiktoken) (2023.8.8)\n",
      "Requirement already satisfied: requests>=2.26.0 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from tiktoken) (2.31.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from requests>=2.26.0->tiktoken) (3.2.0)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from requests>=2.26.0->tiktoken) (2.0.5)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from requests>=2.26.0->tiktoken) (2023.7.22)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\otols\\desktop\\dev\\promptengineering\\venv\\lib\\site-packages (from requests>=2.26.0->tiktoken) (3.4)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: You are using pip version 22.0.4; however, version 23.2.1 is available.\n",
      "You should consider upgrading via the 'c:\\Users\\otols\\Desktop\\dev\\PromptEngineering\\venv\\Scripts\\python.exe -m pip install --upgrade pip' command.\n"
     ]
    }
   ],
   "source": [
    "%pip install chromadb\n",
    "%pip install tiktoken "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.indexes import VectorstoreIndexCreator\n",
    "\n",
    "# Create a vectorstore index from loaders\n",
    "index = VectorstoreIndexCreator().from_loaders([loader])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'question': 'What are the limits I have listed on my note about LangChain?',\n",
       " 'answer': ' The limits listed on the note about LangChain are: Large Language Model (LLM), two types of LLM (base and instructions tuned), Transformers, predicting tokens in a chain, multi-modal LLM trained on video and images, API with temperature parameter, and limiting the number of tokens.\\n',\n",
       " 'sources': 'Maximiser sa productivité avec les LLM et Chat GPT.md'}"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = \"What are the limits I have listed on my note about LangChain?\"\n",
    "\n",
    "index.query_with_sources(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'question': 'Give me the main points of my note about Maximiser sa productivité avec les LLM et Chat GPT?',\n",
       " 'answer': \" The main points of the note about Maximiser sa productivité avec les LLM et Chat GPT are: Large Language Models (LLM) use a lot of data and scientific review to generate text using the same logic; LLM has two types, base and instructions tuned; Chat GPT is limited; it is important to have keywords; there are 6 alternatives to Chat GPT; a data scientist junior needs to generate a motivation letter using their competences and the elements mentioned; multi-modal LLM can be trained on video and images; the API can be used to manage the parameters; the note also mentions Kafka + RabbitMQ, a GCP project, Sopra Steria, Google's vector search technology, HuggingFace datasets, Langchain, Open AI, and hallucination models.\\n\",\n",
       " 'sources': 'Maximiser sa productivité avec les LLM et Chat GPT.md'}"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = \"Summarize my thoughts on my note?\"\n",
    "query = \"Give me the main points of my note about Maximiser sa productivité avec les LLM et Chat GPT?\"\n",
    "index.query_with_sources(query)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}