Click to open the initial README
Get the desktop app | Read the blog post
TinyAgent aims to enable complex reasoning and function calling capabilities in Small Language Models (SLMs) that can be deployed securely and privately at the edge. Traditional Large Language Models (LLMs) like GPT-4 and Gemini-1.5, while powerful, are often too large and resource-intensive for edge deployment, posing challenges in terms of privacy, connectivity, and latency. TinyAgent addresses these challenges by training specialized SLMs with high-quality, curated data, and focusing on function calling with LLMCompiler. As a driving application, TinyAgent can interact with various macOS applications, helping users with day-to-day tasks such as composing emails, managing contacts, scheduling calendar events, and organizing Zoom meetings.
TinyAgent is equipped with 16 different functions that can interact with different applications on Mac, which includes:
- Compose New Email
- Start a new email with options for adding recipients and attachments.
- Example Query: “Email Sid and Nick about the meeting with attachment project.pdf.”
- Reply to Emails
- Respond to received emails, optionally adding new recipients and attachments.
- Example Query: “Reply to Alice's email with the updated budget document attached.”
- Forward Emails
- Forward an existing email to other contacts, including additional attachments if necessary.
- Example Query: “Forward the project briefing to the marketing team.”
- Retrieve phone numbers and email addresses from the contacts database.
- Example Query: “Get John’s phone number” or “Find Alice’s email address.”
- Send text messages to contacts directly from TinyAgent.
- Example Query: “Send an SMS to Canberk saying ‘I’m running late.’”
- Create new calendar events with specified titles, dates, and times.
- Example Query: “Create an event called 'Meeting with Sid' on Friday at 3 PM.”
- Find directions or open map locations for points of interest via Apple Maps.
- Example Query: “Show me directions to the nearest Starbucks.”
- Create, open, and append content to notes stored in various folders.
- Example Query: “Create a note called 'Meeting Notes' in the Meetings folder.”
- File Reading
- Open and read files directly through TinyAgent.
- Example Query: “Open the LLM Compiler.pdf.”
- PDF Summarization
- Generate summaries of PDF documents, enhancing content digestion and review efficiency.
- Example Query: “Summarize the document LLM Compiler.pdf and save the summary in my Documents folder.”
- Set reminders for various activities or tasks, ensuring nothing is forgotten.
- Example Query: “Remind me to call Sid at 3 PM about the budget approval.”
- Schedule and organize Zoom meetings, including setting names and times.
- Example Query: “Create a Zoom meeting called 'Team Standup' at 10 AM next Monday.”
- Write and configure specific instructions for your TinyAgent.
- Example Query: “Always cc team members Nick and Sid in all emails.”
You can choose to enable/disable certain apps by going to the Preferences window.
Depending on the task simplicity, TinyAgent orchestrates the execution of different more specialized or smaller LMs. TinyAgent currently can operate LMs that can summarize a PDF, write emails, or take notes.
See the Customization section to see how to add your own sub-agentx.
When faced with challenging tasks, SLM agents require appropriate tools and in-context examples to guide them. If the model sees irrelevant examples, it can hallucinate. Likewise, if the model sees the descriptions of the tools that it doesn’t need, it usually gets confused, and these tools take up unnecessary prompt space. To tackle this, TinyAgent uses ToolRAG to retrieve the best tools and examples suited for a given query. This process has minimal latency and increases the accuracy of TinyAgent substantially. Please take a look at our blog post and our ToolRAG model for more details.
You need to first install our ToolRAG model from Hugging Face and enable it from the TinyAgent settings to use it.
TinyAgent also accepts voice commands through both the OpenAI Whisper API and local whisper.cpp deployment. For whisper.cpp, you need to setup the local whisper server and provide the server port number in the TinyAgent settings.
You can use with your OpenAI key, Azure deployments, or even your own local models!
You need to provide OpenAI API Key and the models you want to use in the 'Preferences' window.
You need to provide your deployment name and the endpoints for the main agent/sub-agents/embedding model as well as the context length of the agent models in the 'Preferences' window.
You can plug-and-play every part of TinyAgent with your local models! TinyAgent can use an OpenAI-compatible server to run models locally. There are several options you can take:
-
LMStudio : For models already on Huggingface, LMStudio provides an easy-to-use to interface to get started with locally served models.
-
llama.cpp server: However, if you want more control over your models, we recommend using the official llama.cpp server to get started. Please read through the tagged documentation to get started with it.
All TinyAgent needs is the port numbers that you are serving your model at and its context length.
We also provide our own fine-tuned open source models, TinyAgent-1.1B and TinyAgent-7B! We curated a dataset of 40.000 real-life use cases for TinyAgent and fine-tuned two small open-source language models on this dataset with LoRA. After fine-tuning and using ToolRAG, both TinyAgent-1.1B and TinyAgent-7B exceed the performance of GPT-4-turbo. Check out our for the specifics of dataset generation, evaluation, and fine-tuning.
| Model | Success Rate |
|---|---|
| GPT-3.5-turbo | 65.04% |
| GPT-4-turbo | 79.08% |
| TinyLLama-1.1B-32K-Instruct | 12.71% |
| WizardLM-2-7b | 41.25% |
| TinyAgent-1.1B + ToolRAG / [hf] [gguf] | 80.06% |
| TinyAgent-7B + ToolRAG / [hf] [gguf] | 84.95% |
You can customize your TinyAgent by going to ~/Library/Application Support/TinyAgent/tinyagent-llmcompiler directory
and changing the code yourself.
You can use TinyAgent programmatically by just passing in a config file.
from src.tiny_agent.tiny_agent import TinyAgent
from src.tiny_agent.config import get_tiny_agent_config
config_path = "..."
tiny_agent_config = get_tiny_agent_config(config_path=config_path)
tiny_agent = TinyAgent(tiny_agent_config)
await TinyAgent.arun(query="Create a meeting with Sid and Lutfi for tomorrow 2pm to discuss the meeting notes.")- Navigate to
src/tiny_agent/models.pyand add your tool;s name toTinyAgentToolName(Enum)
class TinyAgentToolName(Enum):
...
CUSTOM_TOOL_NAME = "custom_tool"- Navigate to
src/tiny_agent/tiny_agent_tools.pyand define your tool.
def get_custom_tool(...) -> list[Tool]:
async def tool_coroutine(...) -> str: # Needs to return a string
...
return ...
custom_tool = Tool(
name=TinyAgentToolName.CUSTOM_TOOL_NAME,
func=tool_coroutine,
description=(
f"{TinyAgentToolName.CUSTOM_TOOL_NAME.value}(...) -> str\n"
"<The description of the tool>"
)
)
return [custom_tool]- Add your tools to the
get_tiny_agent_toolsfunction.
def get_tiny_agent_tools(...):
...
tools += get_custom_tools(...)
...Note: Adding your own tools only works for GPT models since our open-source models and ToolRAG were only fine-tuned on the original TinyAgent toolset.
You can also add your own custom subagents. To do so, please follow these steps:
- All sub-agents inherit from
SubAgentclass insrc/tiny_agent/sub_agents/sub_agent.py. Your custom agent should inherit from this abstract class and define the__call__method.
class CustomSubAgent(SubAgent):
async def __call__(self, ...) -> str:
...
return response- Add your custom agent to
TinyAgentinsrc/tiny_agent/tiny_agent.py
from src.tiny_agent.sub_agents.custom_agent import CustomAgent
class TinyAgent:
...
custom_agent: CustomAgent
def __init__(...):
...
self.custom_agent = CustomAgent(
sub_agent_llm, config.sub_agent_config, config.custom_instructions
)
...- After defining your custom agent and adding it to TinyAgent, you should create a tool that calls this agent. Please refer to the Adding your own tools section to see how to do so
We would appreciate it if you could please cite our blog post if you found TinyAgent useful for your work:
@misc{tiny-agent,
title={TinyAgent: Function Calling at the Edge},
author={Erdogan, Lutfi Eren and Lee, Nicholas and Jha, Siddharth and Kim, Sehoon and Tabrizi, Ryan and Moon, Suhong and Hooper, Coleman and Anumanchipalli, Gopala and Keutzer, Kurt and Gholami, Amir},
howpublished={\url{https://bair.berkeley.edu/blog/2024/05/29/tiny-agent/}},
year={2024}
}
As a Python environment Poetry is used. To create a new environment use the poetry shell command. Once the virtual
environment is started, install all the necessary dependencies using the poetry install command.
To use OpenAI models please specify OPENAI_API_KEY in the .env file in the root of the repository. To use
Open-Source models you should additionally specify HF_TOKEN in the .env file. Example of the .env file
is .env-sample and located in the root of the repository.
Also, the keys for OpenAI API or HuggingFace API should be specified in the configs (config_mistral.json or config_openai.json).
If you encountered with the following problem:
UnboundLocalError: local variable 'llm_response' referenced before assignmentMost likely that you didn't specify the key in the configuration file.
To use an agent fully, locally you have to set up a dedicated LLM server. The easiest way to do that is to use LM Studio or vllm (TODO)
Once you have installed all the necessary dependencies, declared environmental variables and started the local server, you can finally use the agent. To run an agent programmatically, run the run_programmatically.py script using poetry:
poetry run python3 run_programmatically.pyDuring the first start, an agent will ask you for a permission to the internal directories or macOS applications.