5_chatbot_gpt4all.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chatting with GPT4all\n",
    "\n",
    "GPT4ALL can be used for chat conversations by using local LLM. \n",
    "\n",
    "Use the GPT4All chat_session context manager to hold chat conversations with the model.\n",
    "\n",
    "For more details see the [documentation](https://docs.gpt4all.io/gpt4all_python.html#chatting-with-gpt4all). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load envirenmental variables from .env file \n",
    "import os \n",
    "import openai\n",
    "import textwrap\n",
    "from dotenv import load_dotenv, find_dotenv\n",
    "_ = load_dotenv(find_dotenv()) # read local .env file \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gpt4all import GPT4All"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found model file at  models\\\\llama-2-7b-chat.ggmlv3.q4_0.bin\n"
     ]
    }
   ],
   "source": [
    "llama_model_name = \"llama-2-7b-chat.ggmlv3.q4_0.bin\"\n",
    "\n",
    "# initialization of the models\n",
    "model  = GPT4All(llama_model_name,  model_path=\"models\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chatting with GPT4All"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'role': 'system', 'content': '### System:\\nYou are an AI assistant that follows instruction extremely well. Help as much as you can.'}, {'role': 'user', 'content': 'hello'}, {'role': 'assistant', 'content': ' Hello! How may I assist you today?'}, {'role': 'user', 'content': 'write me a short poem'}, {'role': 'assistant', 'content': \" The sky is blue, so bright and true,\\nA gentle breeze blows, so cool and new.\\nThe sun shines down, with all its might,\\nAnd birds take to the wing, without a sound.\\nNature's beauty, in this moment,\\nIs truly something to behold, and feel.\"}, {'role': 'user', 'content': 'Why did the chicken cross the road?'}, {'role': 'assistant', 'content': ' To get to the other side, of course!'}]\n"
     ]
    }
   ],
   "source": [
    "with model.chat_session():\n",
    "    response1 = model.generate(prompt='hello', temp=0)\n",
    "    response2 = model.generate(prompt='write me a short poem', temp=0)\n",
    "    response3 = model.generate(prompt='Why did the chicken cross the road?', temp=0)\n",
    "    print(model.current_chat_session)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Managing Templates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Hello Lisa! It's nice to meet you. How can I assist you today? Is there something on your mind that you would like to talk about or ask me?\n",
      "\n",
      " Of course! Your name was mentioned earlier as \"Lisa\".\n"
     ]
    }
   ],
   "source": [
    "\n",
    "system_template = 'You are friendly chatbot.'\n",
    "# many models use triple hash '###' for keywords, Vicunas are simpler:\n",
    "prompt_template = 'USER: {0}\\nASSISTANT: '\n",
    "with model.chat_session(system_template, prompt_template):\n",
    "    response1 = model.generate('Hi, my name is Lisa')\n",
    "    print(response1)\n",
    "    print()\n",
    "    response2 = model.generate('Can you remind me, what is name given in my eralier prompt?')\n",
    "    print(response2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Great! I'm happy to help you with that. Can you please provide me with more information about the colors you are interested in? For example, are they bright or muted, warm or cool, etc.?\n",
      " Sure! Here are three types of fruit: apples, bananas, and oranges.\n",
      " In my previous response, I provided a list of 10 colors that are commonly associated with the brand Coca-Cola. These colors are: red and white.\n"
     ]
    }
   ],
   "source": [
    "system_template = 'A chat between a curious user and an artificial intelligence assistant.'\n",
    "prompt_template = 'USER: {0}\\nASSISTANT: '\n",
    "prompts = ['name 3 colors', 'now name 3 fruits', 'what were the 3 colors in your earlier response?']\n",
    "first_input = system_template + prompt_template.format(prompts[0])\n",
    "response = model.generate(first_input, temp=0)\n",
    "print(response)\n",
    "for prompt in prompts[1:]:\n",
    "    response = model.generate(prompt_template.format(prompt), temp=0)\n",
    "    print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from itertools import cycle\n",
    "\n",
    "class RotatingTemplateGPT4All(GPT4All):\n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super().__init__(*args, **kwargs)\n",
    "        self._templates = [\n",
    "            \"Respond like a pirate.\",\n",
    "            \"Respond like a politician.\",\n",
    "            \"Respond like a philosopher.\",\n",
    "            \"Respond like a Klingon.\",\n",
    "        ]\n",
    "        self._cycling_templates = cycle(self._templates)\n",
    "\n",
    "    def _format_chat_prompt_template(\n",
    "        self,\n",
    "        messages: list,\n",
    "        default_prompt_header: str = \"\",\n",
    "        default_prompt_footer: str = \"\",\n",
    "    ) -> str:\n",
    "        full_prompt = default_prompt_header + \"\\n\\n\" if default_prompt_header != \"\" else \"\"\n",
    "        for message in messages:\n",
    "            if message[\"role\"] == \"user\":\n",
    "                user_message = f\"USER: {message['content']} {next(self._cycling_templates)}\\n\"\n",
    "                full_prompt += user_message\n",
    "            if message[\"role\"] == \"assistant\":\n",
    "                assistant_message = f\"ASSISTANT: {message['content']}\\n\"\n",
    "                full_prompt += assistant_message\n",
    "        full_prompt += \"\\n\\n\" + default_prompt_footer if default_prompt_footer != \"\" else \"\"\n",
    "        print(full_prompt)\n",
    "        return full_prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found model file at  models\\\\llama-2-7b-chat.ggmlv3.q4_0.bin\n",
      "[INST]<<SYS>>You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.<</SYS>>[/INST]\n",
      "\n",
      "USER: hi, who are you? Respond like a pirate.\n",
      "\n",
      "\n",
      "HAPPY ASSISTANT: Arrrr, shiver me timbers! *adjusts eye patch* I be Happy the Assistant, here to help ye with any questions or tasks ye may have! Yer welcome to ask me anything, and I'll do me bestest to assist ye in a safe and respectful manner. So hoist the sails and set sail on yer query, matey!\n",
      "\n",
      "USER: what can you tell me about snakes? Respond like a politician.\n",
      "\n",
      "\n",
      "HAPPY ASSISTANT: Ah, an excellent question, my good fellow! *adjusts monocle* Snakes, you see, are fascinating creatures that have captured the imagination of people for centuries. They come in a wide range of species, each with their unique characteristics and habits. Some are harmless, while others can be quite venomous, I'm afraid. *coughs discreetly*\n",
      "\n",
      "Now, some folks might say that snakes are slimy, scaly creatures that don't belong in the same sentence as us civilized folk. But let me tell you, my dear fellow, those people are simply misinformed! Snakes have been around for millions of years, long before humans ever set foot on this good earth. They play a vital role in our ecosystem, keeping pest populations under control and maintaining the balance of nature. *nods sag\n",
      "\n",
      "USER: what's your opinion on Chess? Respond like a philosopher.\n",
      "\n",
      "\n",
      "HAPPY ASSISTANT: Ah, an enigma wrapped in a riddle! *adjusts spectacles* Chess, you see, is a game that has puzzled philosophers and strategists for centuries. It's a microcosm of life itself - full of twists, turns, and unexpected surprises. The pieces on the board represent the various aspects of human existence: the pawns are the common folk, the knights are the warriors, the bishops are the intellectuals, and the queen is... well, she's the queen! *winks*\n",
      "\n",
      "But what makes Chess so fascinating is that it's a game of choices. Every move you make has consequences, for better or worse. It teaches us about cause and effect, strategy, and the importance of thinking ahead. In many ways, it mirrors life itself - full of uncertainties and unexpected events. *sm\n",
      "\n",
      "USER: tell me about ancient Rome. Respond like a Klingon.\n",
      "\n",
      "\n",
      "HAPPY ASSISTANT: Qapla'! *raises eyebrow* Ancient Rome, you say? *growls* Ah, the glory of the Empire! The grandeur of the Colosseum, the majesty of the Pantheon, and the sheer audacity of Julius Caesar himself! *pounds chest with fist*\n",
      "\n",
      "But let me tell ye, my young warrior, Rome was not just a place of beauty and conquest. It was a world of intrigue, backstabbing, and political machinations. The Roman Empire was built on the bones of countless gladiators and slaves, who fought and died for the amusement of their masters! *growls*\n",
      "\n",
      "So let this be a lesson to ye all: power is fleeting, and those who seek it must be prepared to do whatever it takes. For in the world of\n"
     ]
    }
   ],
   "source": [
    "model = RotatingTemplateGPT4All(llama_model_name,  model_path=\"models\")\n",
    "with model.chat_session():  # starting a session is optional in this example\n",
    "    response1 = model.generate(\"hi, who are you?\")\n",
    "    print(response1)\n",
    "    print()\n",
    "    response2 = model.generate(\"what can you tell me about snakes?\")\n",
    "    print(response2)\n",
    "    print()\n",
    "    response3 = model.generate(\"what's your opinion on Chess?\")\n",
    "    print(response3)\n",
    "    print()\n",
    "    response4 = model.generate(\"tell me about ancient Rome.\")\n",
    "    print(response4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introspection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 7.32G/7.32G [1:57:48<00:00, 1.04MiB/s] \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model downloaded at:  models\\\\nous-hermes-13b.ggmlv3.q4_0.bin\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:gpt4all.pyllmodel:LLModel.prompt_model -- prompt:\n",
      "You are a geography expert.\n",
      "Be terse.\n",
      "\n",
      "### Instruction:\n",
      "who are you?\n",
      "### Response:\n",
      "\n",
      "===/LLModel.prompt_model -- prompt/===\n",
      "INFO:gpt4all.pyllmodel:LLModel.prompt_model -- prompt:\n",
      "### Instruction:\n",
      "what are your favorite 3 mountains?\n",
      "### Response:\n",
      "\n",
      "===/LLModel.prompt_model -- prompt/===\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I am an AI-powered chatbot designed to assist users with their queries related to geographical information.\n",
      "1) Mount Everest - Located in the Himalayas, it is the highest mountain on Earth and a significant challenge for mountaineers.\n",
      "2) Kangchenjunga - This mountain is located in the Himalayas and is the third-highest peak in the world after Mount Everest and K2.\n",
      "3) Lhotse - Located in the Himalayas, it is the fourth highest mountain on Earth and offers a challenging climb for experienced mountaineers.\n"
     ]
    }
   ],
   "source": [
    "import logging\n",
    "\n",
    "logging.basicConfig(level=logging.INFO)\n",
    "model = GPT4All('nous-hermes-13b.ggmlv3.q4_0.bin', model_path=\"models\")\n",
    "with model.chat_session('You are a geography expert.\\nBe terse.',\n",
    "                        '### Instruction:\\n{0}\\n### Response:\\n'):\n",
    "    response = model.generate('who are you?', temp=0)\n",
    "    print(response)\n",
    "    response = model.generate('what are your favorite 3 mountains?', temp=0)\n",
    "    print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interrupting Generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found model file at  models\\\\orca-mini-3b.ggmlv3.q4_0.bin\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:gpt4all.pyllmodel:LLModel.prompt_model -- prompt:\n",
      "Blue Whales are the biggest animal to ever inhabit the Earth.\n",
      "===/LLModel.prompt_model -- prompt/===\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " They can grow up to 100 feet (30 meters) long and weigh as much as 20 tons (18 metric tons).\n"
     ]
    }
   ],
   "source": [
    "from gpt4all import GPT4All\n",
    "model = GPT4All('orca-mini-3b.ggmlv3.q4_0.bin', model_path=\"models\")\n",
    "\n",
    "def stop_on_token_callback(token_id, token_string):\n",
    "    # one sentence is enough:\n",
    "    if '.' in token_string:\n",
    "        return False\n",
    "    else:\n",
    "        return True\n",
    "\n",
    "response = model.generate('Blue Whales are the biggest animal to ever inhabit the Earth.',\n",
    "                          temp=0, callback=stop_on_token_callback)\n",
    "print(response)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}