diff --git a/src/pages/post/2023-09-06-diy-chatbot.md b/src/pages/post/2023-09-06-diy-chatbot.md new file mode 100644 index 0000000..f34fef1 --- /dev/null +++ b/src/pages/post/2023-09-06-diy-chatbot.md @@ -0,0 +1,124 @@ +--- +title: Make your own AI Chatbot +layout: "../../layouts/BlogLayout.astro" +preview: "Not affiliated with ChatGPT" +datePublished: 2023-09-06 +--- + +## Introduction + +AI chatbots like ChatGPT are pretty cool, no one's denying that, but also they train it on your inputs by default and you're at the mercy of OpenAI/Google/Other AI Overlord as to when you can use it. +Why not make/host your own? + +## Setup + +If you haven't got Python installed, you _probably_ want to install [miniconda](//docs.conda.io/projects/miniconda/en/latest/miniconda-install.html). + +We need to install a few things first (scope it to the project): + +- [Chainlit](https://docs.chainlit.io/installation) +- [`llama-cpp-python`](https://docs.chainlit.io/installation) +- [LangChain](https://python.langchain.com/docs/get_started/installation.html) + +and you also want to install a llama model. +This is more complicated than it needs to be, but you sign up to ~~Facebook's~~ Meta's llama access [here](https://ai.meta.com/resources/models-and-libraries/llama-downloads/) and then download one of the `.gguf` files [here](https://huggingface.co/TheBloke/Llama-2-13B-chat-GGUF). +The one I'm using is called [llama-2-13b-chat.Q4_K_M.gguf](https://huggingface.co/TheBloke/Llama-2-13B-chat-GGUF/blob/main/llama-2-13b-chat.Q4_K_M.gguf). + +## Chainlit + +Chainlit is super easy to use, if you want an interactive app with most of the work done for you, then that's enough! + +If you create a file called `app.py`, pop this into it + +``` py +import chainlit as cl + +@cl.on_message +async def main(message: str): + await cl.Message(content=f"Received: {message}").send() +``` + +and then run the following command + +``` sh +chainlit run app.py -w +``` + +The `-w` flag turns on auto-reloading, so you can edit `app.py` and it'll update live at (not http**s** for some reason). + +## LangChain + +``` py +from langchain.llms import LlamaCpp + +llm = LlamaCpp( + model_path="./models/llama-2-13b-chat.Q4_K_M.gguf", # Make sure you change this to the path you downloaded it to earlier + max_tokens=2048, +) + +output = llm("Q: Name the planets in the solar system? A: ", max_tokens=32, stop=["Q:", "\n"], echo=True) +print(output) +``` + +produces + +``` +{ + "id": "cmpl-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", + "object": "text_completion", + "created": 1679561337, + "model": "./models/7B/ggml-model.bin", + "choices": [ + { + "text": "Q: Name the planets in the solar system? A: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune and Pluto.", + "index": 0, + "logprobs": None, + "finish_reason": "stop" + } + ], + "usage": { + "prompt_tokens": 14, + "completion_tokens": 28, + "total_tokens": 42 + } +} +``` + +## Combining them + +Now if we stick them together, we find this, a pretty simple minimal working example for a chat bot. + +``` py +import chainlit as cl +from langchain.llms import LlamaCpp +from langchain import PromptTemplate, LLMChain + +template = """You are a helpful and friendly AI chatbot. Try and answer the following if you can. +Question: {question} +Answer: """ + +@cl.on_chat_start +def main(): + # Make sure the model path is correct for your system! + llm = LlamaCpp( + model_path="./models/llama-2-13b-chat.Q4_K_M.gguf", + max_tokens=2048, + ) + + llm_chain = LLMChain(prompt=prompt, llm=llm, verbose=True) + + # Store the chain in the user session + cl.user_session.set("llm_chain", llm_chain) + + +@cl.on_message +async def main(message: str): + # Get chain from the user session + llm_chain = cl.user_session.get("llm_chain") + + # Call the chain asynchronously + res = await llm_chain.acall(message) + + await cl.Message(content=res["text"]).send() + +```