agent.md

Agent

Agentic Design Frameworks

• Agentic Design Frameworks focus on managing autonomous or semi-autonomous AI agents for complex tasks.
• e.g., Autogen, and crewAI
• Agent Leaderboard

Agent Design Patterns

• The Rise and Potential of Large Language Model Based Agents: A Survey: The papers list for LLM-based agents [cnt] / git [14 Sep 2023]
• AgentBench Evaluating LLMs as Agents: Assess LLM-as Agent’s reasoning and decision-making abilities. [7 Aug 2023]
• Agentic Design Patterns ref [Mar 2024]
  • Reflection: LLM self-evaluates to improve.
    • Self-Refine [30 Mar 2023]
    • Reflexion [20 Mar 2023]
    • CRITIC [19 May 2023]
  • Tool use: LLM uses tools for information gathering, action, or data processing.
    • Gorilla [24 May 2023]
    • MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action [20 Mar 2023]
    • Efficient Tool Use with Chain-of-Abstraction Reasoning [30 Jan 2024]
  • Planning: LLM devises and executes multistep plans to reach goals.
    • Chain-of-Thought Prompting Elicits Reasoning in Large Language Models [28 Jan 2022]
    • HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face [30 Mar 2023]
    • Understanding the planning of LLM agents: A survey [5 Feb 2024]
  • Multi-agent collaboration: Multiple AI agents collaborate for better solutions.
    • Communicative Agents for Software Development [16 Jul 2023]
    • AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation [16 Aug 2023]
    • MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework [1 Aug 2023]
    • Framework: Autogen / LangGraph / crewAI
• Generate the code ref [Jun 2024]
  • AgentCoder: Multiagent-Code Generation with Iterative Testing and Optimisation [20 Dec 2023]
  • LDB: A Large Language Model Debugger via Verifying Runtime Execution Step by Step [25 Feb 2024]
  • SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering [6 May 2024]
• 5 Agentic AI Design Patterns: Reflection, Tool use, ReAct, Planning, Multi-agent pattern [24 Jan 2025]
• Generative AI Design Patterns for Agentic AI Systems: Design Patterns for Agentic solutions in Azure [May 2023]

Agent Design Reference

• Exploring Generative AI (martinfowler.com): Memos on how LLMs are being used to enhance software delivery practices, including Toochain, Test-Driven Development (TDD) with GitHub Copilot, pair programming, and multi-file editing. [26 Jul 2023 ~ ]
• Cognitive Architectures for Language Agents: Cognitive Architectures for Language Agents (CoALA). Procedural (how to perform tasks), Semantic (long-term store of knowledge), Episodic Memory (recall specific past events) ref [5 Sep 2023]
• Language Agent Tree Search Method (LATS): LATS leverages an external environment and an MCTS (Monte Carlo Tree Search)-based search [6 Oct 2023]
• AI Agents That Matter: AI agent evaluations for optimizing both accuracy and cost. Focusing solely on accuracy can lead to overfitting and high costs. retry, warming, escalation [1 Jul 2024]
• Automated Design of Agentic Systems: Automated Design of Agentic Systems (ADAS), which aims to automatically create powerful agentic system designs, including inventing novel building blocks and/or combining them. [15 Aug 2024]
• The Different Ochestration Frameworks:💡Orchestration frameworks for LLM applications: Micro-orchestration / Macro-orchestration / Agentic Design Frameworks / Optimizer frameworks [11 Oct 2024]
• Agent-as-a-Judge: Evaluate Agents with Agents. DevAI, a new benchmark of 55 realistic automated AI development tasks. Agent-as-a-Judge > LLM-as-a-Judge > Human-as-a-Judge [14 Oct 2024]
• Google AI Agents Whitepaper [12 Nov 2024]
• Generative Agent Simulations of 1,000 People: a generative agent architecture that simulates more than 1,000 real individuals using two-hour qualitative interviews. 85% accuracy in General Social Survey. [15 Nov 2024]
• Agents Are Not Enough: Proposes an ecosystem comprising agents (task executors), sims (user preferences and behavior), and assistants (human-in-the-loop). [19 Dec 2024]
• Hugging Face Agents Course 🤗 Hugging Face Agents Course. [Jan 2025]

Tool use: LLM to Master APIs

• Berkeley Function-Calling Leaderboard V2 [Aug 2024]

• Gorilla: An API store for LLMs: [cnt]: Gorilla: Large Language Model Connected with Massive APIs git [24 May 2023]

  1. Used GPT-4 to generate a dataset of instruction-api pairs for fine-tuning Gorilla.
  2. Used the abstract syntax tree (AST) of the generated code to match with APIs in the database and test set for evaluation purposes.

  Another user asked how Gorilla compared to LangChain; Patil replied: LangChain is a terrific project that tries to teach agents how to use tools using prompting. Our take on this is that prompting is not scalable if you want to pick between 1000s of APIs. So Gorilla is a LLM that can pick and write the semantically and syntactically correct API for you to call! A drop in replacement into LangChain! cite [04 Jul 2023]

• Meta: Toolformer: [cnt]: Language Models That Can Use Tools, by MetaAI git [9 Feb 2023]

• ToolLLM: [cnt]: : Facilitating Large Language Models to Master 16000+ Real-world APIs git [31 Jul 2023]

• APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets [26 Jun 2024]

• ToolShed: Toolshed Knowledge Bases & Advanced RAG-Tool Fusion, optimized for storing and retrieving tools in a vector database for large-scale agents. To address the limitations of primary methods, two approaches are: 1. tuning-based tool calling via LLM fine-tuning, and 2. retriever-based tool selection and planning. [18 Oct 2024]

Computer use & MCP

• ACU - Awesome Agents for Computer Use
• Anthropic Claude's computer use:

1. CogAgent: An open-sourced end-to-end VLM-based GUI Agent [Dec 2023]
2. OpenInterpreter starts to support Computer Use API
3. Agent.exe: Electron app to use computer use APIs. [Oct 2024]
4. 5ire: a cross-platform desktop AI assistant, MCP client. [Oct 2024]
5. x-ref > UFO: Windows Control
6. Self-Operating Computer Framework: A framework to enable multimodal models to operate a computer. [Nov 2023]
7. Open-Interface: LLM backend (GPT-4V, etc), supporting Linux, Mac, Windows. [Jan 2024]
8. Computer Use OOTB: Out-of-the-box (OOTB) GUI Agent for Windows and macOS. [Oct 2024]
9. goose:💡An open-source, extensible AI agent with support for the Model Context Protocol (MCP). Developed by Block, a company founded in 2009 by Jack Dorsey. [Jan 2025]
10. UI-TARS: An agent model built on Qwen-2-VL for seamless GUI interaction, by ByteDance. git / Application git [21 Jan 2025]
11. OpenAI Operator x-ref [23 Jan 2025]
12. Open Operator: a web agent based on Browserbase [24 Jan 2025]

Memory Layer

• x-ref

Agent Framework

1. Huginn: A hackable version of IFTTT or Zapier on your own server for building agents that perform automated tasks. [Mar 2013]
2. Botpress Cloud: The open-source hub to build & deploy GPT/LLM Agents. [Nov 2016]
3. n8n: A workflow automation tool for integrating various tools. [Jan 2019]
4. phidata: Build AI Assistants with memory, knowledge, and tools [May 2022]
5. Cheshire-Cat (Stregatto): Framework to build custom AIs with memory and plugins [Feb 2023]
6. MetaGPT: Multi-Agent Framework. Assign different roles to GPTs to form a collaborative entity for complex tasks. e.g., Data Interpreter [Jun 2023]
7. SuperAGI: Autonomous AI Agents framework [May 2023]
8. AgentVerse: Primarily providing: task-solving and simulation. [May 2023]
9. OpenAgents: Three distinct agents: Data Agent for data analysis, Plugins Agent for plugin integration, and Web Agent for autonomous web browsing. [Aug 2023]
10. AgentOps:Python SDK for AI agent monitoring, LLM cost tracking, benchmarking. [Aug 2023]
11. Lagent: Inspired by the design philosophy of PyTorch. A lightweight framework for building LLM-based agents. [Aug 2023]
12. Autogen:💡Customizable and conversable agents framework [Aug 2023]
13. LangGraph: Built on top of LangChain [Aug 2023]
14. crewAI: Framework for orchestrating role-playing, autonomous AI agents. [Oct 2023]
15. XAgent: Autonomous LLM Agent for complex task solving like data analysis, recommendation, and model training [Oct 2023]
16. Qwen-Agent: Agent framework built upon Qwen1.5, featuring Function Calling, Code Interpreter, RAG, and Chrome extension. [Sep 2023]
17. Open AI Assistant API [6 Nov 2023]
18. Burr: Create an application as a state machine (graph/flowchart) for managing state, decisions, human feedback, and workflows. [Jan 2024]
19. TaskingAI: A BaaS (Backend as a Service) platform for LLM-based Agent Development and Deployment. [Jan 2024]
20. AgentScope: To build LLM-empowered multi-agent applications. [Jan 2024]
21. composio: Integration of Agents with 100+ Tools [Feb 2024]
22. maestro: A Framework for Claude Opus, GPT, and local LLMs to Orchestrate Subagents [Mar 2024]
23. UpSonic: (previously GPT Computer Assistant(GCA)) an AI agent framework designed to make computer use. [May 2024]
24. Atomic Agents: an extremely lightweight and modular framework for building Agentic AI pipelines [Jun 2024]
25. AWS: Multi-Agent Orchestrator: a framework for managing multiple AI agents and handling complex conversations. [Jul 2024]
26. SwarmZero: SwarmZero's SDK for building AI agents, swarms of agents. [Aug 2024]
27. Dynamiq: An orchestration framework for RAG, agentic AI, and LLM applications [Sep 2024]
28. PySpur: Drag-and-Drop. an AI agent builder in Python. [Sep 2024]
29. Bee Agent Framework: IBM. The TypeScript framework for building scalable agentic applications. [Oct 2024]
30. Agent-S: To build intelligent GUI agents that autonomously learn and perform complex tasks on your computer. [Oct 2024]
31. OpenAI Swarm: An experimental and educational framework for lightweight multi-agent orchestration. [11 Oct 2024]
32. PydanticAI: Agent Framework / shim to use Pydantic with LLMs. Model-agnostic. Type-safe. [29 Oct 2024]
33. smolagents:🤗a smol library to build great agents! [Dec 2024]
34. Agentarium: a framework for creating and managing simulations populated with AI-powered agents. [Dec 2024]
35. Microsoft Agent Frameworks x-ref
36. Agent Framework used in MLE-bench: GPT-4o (AIDE) earned the highest score x-ref
    • AIDE: The state-of-the-art machine learning engineer agent [Apr 2024]
    • OpenHands: OpenHands (formerly OpenDevin), a platform for software development agents [Mar 2024]
    • MLAB ResearchAgent: Evaluating Language Agents on Machine Learning Experimentation [Aug 2023]

Agent Application

1. Khoj: Open-source, personal AI agents. Cloud or Self-Host, Multiple Interfaces. Python Django based [Aug 2021]
2. PR-Agent: Efficient code review and handle pull requests, by providing AI feedbacks and suggestions [Jan 2023]
3. Auto-GPT: Most popular [Mar 2023]
4. CAMEL: CAMEL: Communicative Agents for “Mind” Exploration of Large Scale Language Model Society [Mar 2023]
5. babyagi: Simplest implementation - Coworking of 4 agents [Apr 2023]
6. SuperAGI: GUI for agent settings [May 2023]
7. AgentGPT: Assemble, configure, and deploy autonomous AI agents in your browser [Apr 2023]
8. GPT Pilot: The first real AI developer. Dev tool that writes scalable apps from scratch while the developer oversees the implementation [Jul 2023]
9. OpenDAN : Your Personal AIOS: OpenDAN, an open-source Personal AI OS consolidating various AI modules in one place [May 2023]
10. RasaGPT: Built with Rasa, FastAPI, Langchain, and LlamaIndex [Apr 2023]
11. ChatDev: Virtual software company. Create Customized Software using LLM-powered Multi-Agent Collaboration [Sep 2023]
12. skyagi: Simulating believable human behaviors. Role playing [Apr 2023]
13. tabby: a self-hosted AI coding assistant, offering an open-source and on-premises alternative to GitHub Copilot. [Aug 2023]
14. Letta ADE: a graphical user interface for ADE (Agent Development Environment) by Letta (previously MemGPT) [12 Oct 2023]
15. AppAgent-TencentQQGYLab: Multimodal Agents as Smartphone Users. [Dec 2023]
16. AIOS: LLM Agent Operating System [Jan 2024]
17. SeeAct: GPT-4V(ision) is a Generalist Web Agent, if Grounded git [Jan 2024]
18. WrenAI: Open-source SQL AI Agent for Text-to-SQL [Mar 2024]
19. Devon: An open-source pair programmer. [Mar 2024]
20. Project Astra: Google DeepMind, A universal AI agent that is helpful in everyday life [14 May 2024]
21. SakanaAI AI-Scientist: Towards Fully Automated Open-Ended Scientific Discovery [Aug 2024]
22. Integuru: An AI agent that generates integration code by reverse-engineering platforms' internal APIs. [Oct 2024]
23. skyvern: Automate browser-based workflows with LLMs and Computer Vision [Feb 2024]
24. LaVague: Automate automation with Large Action Model framework. Generate Selenium code. [Feb 2024]
25. Mobile-Agent: The Powerful Mobile Device Operation Assistant Family. [Jan 2024]
26. TEN Agent: The world’s first real-time multimodal agent integrated with the OpenAI Realtime API. [Jun 2024]
27. Agent Zero: An open-source framework for autonomous AI agents with task automation and code generation. [Jun 2024]
28. Agentless: an agentless approach to automatically solve software development problems. AGENTLESS, consisting of three phases: localization, repair, and patch validation (self-reflect). [1 Jul 2024]
29. browser-use: Make websites accessible for AI agents. [Nov 2024]
30. Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks [Nov 2024]
31. Riona-AI-Agent: automation tool designed for Instagram to automate social media interactions such as posting, liking, and commenting. [Jan 2025]
32. Realtime API Agents Demo: a simple demonstration of more advanced, agentic patterns built on top of the Realtime API. OpenAI. [Jan 2025]
33. PaSa: an advanced paper search agent. Bytedance. [Jan 2025]

OSS Alternatives for OpenAI Code Interpreter (aka. Advanced Data Analytics)

1. OpenAI Code Interpreter Integration with Sandboxed python execution environment [23 Mar 2023]
   • We provide our models with a working Python interpreter in a sandboxed, firewalled execution environment, along with some ephemeral disk space.
2. SlashGPT The tool integrated with "jupyter" agent [Apr 2023]
3. gpt-code-ui An open source implementation of OpenAI's ChatGPT Code interpreter. [May 2023]
4. OSS Code Interpreter A LangChain implementation of the ChatGPT Code Interpreter. [Jul 2023]
5. Open Interpreter:💡Let language models run code on your computer. [Jul 2023]

Domain-specific

1. 5 Top AI Agents for Earth Snapshots VLMs and LLMs for Geospatial Intelligent Analysis: GeoChat | GEOBench-VLM | RS5M | VHM | EarthGPT
2. MLE-agent: LLM agent for machine learning engineers and researchers [Apr 2024]
3. DrugAgent: Automating AI-aided Drug Discovery [24 Nov 2024]
4. FinRobot: AI Agent for Equity Research and Valuation [13 Nov 2024]
5. An LLM Agent for Automatic Geospatial Data Analysis [24 Oct 2024]
6. Director: Think of Director as ChatGPT for videos. AI video agents framework for video interactions and workflows. [Oct 2024]
7. ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoning: ChemAgent leverages an innovative self-improving memory system to significantly enhance performance in complex scientific tasks, with a particular focus on Chemistry. [11 Jan 2025]
8. landing.ai: Vision Agent: A agent frameworks to generate code to solve your vision task. [Feb 2025]

Deep research

1. STORM: Simulating Expert Q&A, iterative research, structured outline creation, and grounding in trusted sources to generate Wikipedia-like reports. [Apr 2024]
2. SakanaAI AI-Scientist: x-ref [Aug 2024]
3. Company Researcher: a free and open-source tool that helps you instantly understand any company inside out. [Nov 2024]
4. Open Deep Research: Open source alternative to Gemini Deep Research. [Dec 2024]
5. Agent Laboratory: E2E autonomous research workflow. Using LLM Agents as Research Assistants. [8 Jan 2025]
6. OpenAI deep research: x-ref [2 Feb 2025]
7. Ollama Deep Researcher: a fully local web research assistant that uses any LLM hosted by Ollama [Feb 2025]
8. smolagents: Open Deep Research > examples/open_deep_research. By HuggingFace [Feb 2025]
9. open source deep research: Firecrawl Search based backend & UI [Feb 2025]
10. DeepSearcher: DeepSearcher integrates LLMs and Vector Databases for precise search, evaluation, and reasoning on private data, providing accurate answers and detailed reports. [Feb 2025]
11. Felo.ai Deep Research [8 Feb 2025]
12. LangChain Open Deep Research: Formerly Report mAIstro, a web research assistant for generating comprehensive reports on any topic. [13 Feb 2025]
13. Perplexity Deep Research [14 Feb 2025]
14. Accelerating scientific breakthroughs with an AI co-scientist: Google introduces AI co-scientist, a multi-agent AI system built with Gemini 2.0 as a virtual scientific collaborator [19 Feb 2025]