Getting started with Chat Abilities
Requirements: ExLlama is required for routing to work. For a real-time experience, a 30X0 or better is needed.
The AI assistant application leverages large language models like Llama to engage in dynamic and contextually rich conversations. Users can ask questions, seek information, or request the model to perform text-based tasks such as writing, editing, or brainstorming ideas. The model can comprehend the input query and generate responses in natural language, making the interaction smooth and user-friendly. In addition to accessing LLMs, the primary advantage of using Spell Book over a vanilla LLM chat client is its function calling and model routing abilities. To take advantage of these, two models are required to be running: Llama 7B (ExLlama) and T5 Large v2. These two models are used in conjunction with two Loras to provide the function calling services for Spell Book.
Models can be downloaded via the UI on each backend server that is running. Start by navigating to Servers > golem-xxxxxx > Available Skills > Embedding. You will need to download both T5 Large v2 and Llama 7B (ExLlama) for routing to work; they are highlighted by the UI to make them easier to find. T5 is found under embeddings, and Llama 7B is found under Language Models. Download any other models you are interested in using; if you do not see the model you want to use, it can be added with the Custom Skill button.
Once the models have finished downloading, they will appear in the Servers > golem-xxxxxx > Installed Skills area. They can be run by clicking on the icon on the far right that looks like a computer screen.
Shortcuts allow you to access chat abilities and language models when automatic routing is not sending the request to the correct model or chat ability. Shortcuts are emojis that are prepended to a chat request. There are also several pre-defined shortcuts to control the context that is sent to the model. If function and skill pinning are turned on, manual selection also extends the routing definitions, so similar requests will be routed to the chat ability or language model automatically. Below are some examples of how to use shortcuts.
Each chat ability can be accessed with the shortcut emoji that appears at the beginning of its definition. This emoji also appears below the selected skill icon in the chat session.
Example: 🌞 What is the current temp in Chicago?
You can also access language models with a shortcut. The emoji that represents which language model a request can be sent to is found at the beginning of its name in the Online Skills section of the Servers configuration. This emoji is also output in the chat session in the left-hand column under the circular icon.
Example: ⚡ How do I open port 9000 on a Linux box?
You can also control the number of past messages that are sent to the language model by using the 👉 and 👆 emojis. This is useful if only a small part of the conversation is relevant to the next chat round and you are trying to save money on API calls to OpenAI. In the two examples below, the first will send only the one message to GPT-4. The second example sends the last 5 user/assistant responses to GPT-3.5.
Example: 👉✨ What is wrong with this code block?
Example: 👆5️⃣⚡ Summarize our conversation so far.
The Chat Ability and Skill pinning is functionality that allows the user to extend the definitions for routing. When pinning is enabled and a manual route is selected with an emoji, the function calling system will save the definition of the guessed function or knowledge domain to the database. Future similar chat requests will automatically be routed based on these extended definitions, making the function calling features work better with more use and manually selecting routes.
You can define your own models to be used by the various chat abilities on the Servers page. Click on the Add Skill button. The most important fields are explained below. For the most part, you can copy and paste an existing definition and change a few key fields.
Label Must be unique.
Routing Key Must be unique.
Moe Domain Definitions for model routing.
Moe Function Definitions for model routing.
Model > Name The last part of the HuggingFace URL for the model repository.