[🌐 Website] • [📜 Paper] • [🐱 GitHub]• [ Zhihu]• [ OpenReview]

This is the repo for the paper OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use. This paper conducts a comprehensive survey on OS Agents, which are (M)LLM-based agents that use computing devices (e.g., computers and mobile phones) by operating within the environments and interfaces (e.g., Graphical User Interface (GUI)) provided by operating systems (OS) to automate tasks. The survey is aimed to consolidates the state of OS Agents research, providing insights to guide both academic inquiry and industrial development. In this repository, we have listed relevant papers related to our work in four areas: Foundation Models, Agent Frameworks, Evaluation & Benchmarks, and Safety & Privacy and this collection will be continuously updated. We aim to provide you with comprehensive knowledge in the OS Agent field, hoping it can help you quickly familiarize yourself with this research direction.

This paper was rejected by arXiv surprisingly with the justification: "Our moderators determined that your submission does not contain sufficient original or substantive scholarly research and is not of interest to arXiv." This reasoning appears to be inconsistent with the content and contribution of the paper. We attempted an appeal, but unfortunately, this was unsuccessful, and no further explanation was provided. A resubmission did not resolve the issue either. As a result, the ONLY way to access the paper at the moment is through our GitHub repository or via OpenReview Archive. We are disappointed by the lack of transparency in arXiv’s moderation process.

(Some teams involved in this survey is hiring. Information will be continuously updated, please stay tuned. Detailed information is here.)

OPPO is seeking algorithm interns, new graduates, and experienced candidates for its Personal AI Team. Focused on multimodal LLMs, AI Agents, and personalization, the team works to develop AI Native Phones. Interested candidates, contact: [email protected].

01.AI is hiring algorithm interns, new graduates, and experienced candidates for its post-training team. Join the team behind Yi-Lightning, ranked #1 in China and #6 globally in the LMSys Chatbot Arena updated on October 14, 2024. Interested candidates, contact: [email protected].

• Overview of OS Agent Survey
• Tables
• Full List
  • Foundation Models
  • Agent Frameworks
  • Evaluation & Benchmark
  • Safety & Privacy
• Hiring
• Useful Links
• Contact
• Citation

This survey aims to advance the research and development of OS Agents by providing a detailed exploration of their fundamental capabilities, methodologies for building them using (M)LLMs, and emerging trends in the field. While OS Agents are still in the early stages of growth, the rapid evolution of technology continues to introduce innovative approaches and applications. This work seeks to highlight ongoing challenges, future opportunities, and the latest developments, encouraging further research and industrial adoption. Ultimately, we hope this study will serve as a catalyst for innovation, driving meaningful progress in both academia and industry.

Table 1: Recent foundation models for OS Agents. Arch.: Architecture, Exist.: Existing, Mod.: Modified, Concat.: Concatenated, PT: Pre-Train, SFT: Supervised Fine-Tune, RL: Reinforcement Learning.

Table 2: Recent agent frameworks for OS Agents. TD: Textual Description, GS: GUI Screenshots, VG: Visual Grounding, SG: Semantic Grounding, DG: Dual Grounding, GL: Global, IT: Iterative, AE: Automated Exploration, EA: Experience-Augmented, MA: Management, IO: Input Operations, NO: Navigation Operations, EO: Extended Operations.

Table 3: Recent benchmarks for OS Agents. We divided the Benchmarks into three sections based on the Platform and sorted them by release date. The following is an explanation of the abbreviations. BS: Benchmark Settings, M/P: Mobile, PC: Desktop, IT: Interactive, ST: Static, OET: Operation Environment Types, RW: Real-World, SM: Simulated, GG: GUI Grounding, IF: Information Processing, AT: Agentic, CG: Code Generation.

1. [2024/12/13] Falcon-UI: Understanding GUI Before Following User Instructions. [paper]
2. [2024/12/13] AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials. [paper]
3. [2024/12/05] Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction. [paper]
4. [2024/11/26] ShowUI: One Vision-Language-Action Model for GUI Visual Agent. [paper]
5. [2024/11/22] ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data. [paper]
6. [2024/11/18] Improved GUI Grounding via Iterative Narrowing. [paper]
7. [2024/10/31] AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents. [paper]
8. [2024/10/30] OS-ATLAS: A Foundation Action Model for Generalist GUI Agents. [paper]
9. [2024/10/28] AutoGLM: Autonomous Foundation Agents for GUIs. [paper]
10. [2024/10/25] EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data. [paper]
11. [2024/10/24] Ferret-UI One: Mastering Universal User Interface Understanding Across Platforms. [paper]
12. [2024/10/22] ShowUI: One Vision-Language-Action Model for Generalist GUI Agent. [paper]
13. [2024/10/17] Harnessing Webpage UIs for Text-Rich Visual Understanding. [paper]
14. [2024/10/09] TinyClick: Single-Turn Agent for Empowering GUI Automation. [paper]
15. [2024/10/07] Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents. [paper]
16. [2024/10/03] NNetscape Navigator: Complex Demonstrations for Web Agents Without a Demonstrator. [paper]
17. [2024/09/30] MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning. [paper]
18. [2024/09/24] Synatra: Turning indirect knowledge into direct demonstrations for digital agents at scale. [paper]
19. [2024/09/23] MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understanding. [paper]
20. [2024/09/22] MobileViews: A Large-Scale Mobile GUI Dataset. [paper]
21. [2024/08/30] UI-Hawk: Unleashing the screen stream understanding for gui agents. [paper]
22. [2024/07/19] GUI Action Narrator: Where and When Did That Action Take Place? [paper]
23. [2024/07/05] MobileFlow: A Multimodal LLM for Mobile GUI Agent. [paper]
24. [2024/06/20] VGA: Vision GUI Assistant - Minimizing Hallucinations through Image-Centric Fine-Tuning. [paper]
25. [2024/06/12] GUI Odyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices. [paper]
26. [2024/06/12] Tell Me What's Next: Textual Foresight for Generic UI Representations. [paper]
27. [2024/05/08] Visual Grounding for User Interfaces. [paper]
28. [2024/05/05] Visual grounding for desktop graphical user interfaces. [paper]
29. [2024/05/05] Android in the Zoo：Chain-of-Action-Thought for GUI Agents. [paper]
30. [2024/05/01] Navigating WebAI: Training Agents to Complete Web Tasks with Large Language Models and Reinforcement Learning. [paper]
31. [2024/04/16] Search Beyond Queries: Training Smaller Language Models for Web Interactions via Reinforcement Learning. [paper]
32. [2024/04/12] Training a Vision Language Model as Smartphone Assistant. [paper]
33. [2024/04/09] Autonomous Evaluation and Refinement of Web Agents. [paper]
34. [2024/04/08] Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs. [paper]
35. [2024/04/04] AutoWebGLM: A Large Language Model-based Web Navigating Agent. [paper]
36. [2024/04/02] Octopus v2: On-device language model for super agent. [paper]
37. [2024/03/30] Large Language Models Can Self-Improve At Web Agent Tasks. [paper]
38. [2024/02/08] WebLINX: Real-World website navigation with Multi-Turn dialogue. [paper]
39. [2024/02/07] ScreenAI: A Vision-Language Model for UI and Infographics Understanding. [paper]
40. [2024/02/06] Dual-View Visual Contextualization for Web Navigation. [paper]
41. [2024/01/20] E-ANT: A Large-Scale Dataset for Efficient Automatic GUI NavigaTion. [paper]
42. [2024/01/17] SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents. [paper]
43. [2024/01/17] GUICourse: From General Vision Language Models to Versatile GUI Agents. [paper]
44. [2023/12/25] WebVLN: Vision-and-Language Navigation on Websites. [paper]
45. [2023/12/25] UINav: A Practical Approach to Train On-Device Automation Agents. [paper]
46. [2023/12/14] CogAgent: A Visual Language Model for GUI Agents. [paper]
47. [2023/12/04] Intelligent Virtual Assistants with LLM-based Process Automation. [paper]
48. [2023/11/30] Exposing Limitations of Language Model Agents in Sequential-Task Compositions on the Web. [paper]
49. [2023/10/27] Android in the wild: A large-scale dataset for android device control. [paper]
50. [2023/10/08] UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model. [paper]
51. [2023/10/07] ILuvUI: Instruction-tuned Language-Vision modeling of UIs from Machine Conversations. [paper]
52. [2023/10/07] Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation API. [paper]
53. [2023/07/24] A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis. [paper]
54. [2023/05/31] From pixels to UI actions: Learning to follow instructions via graphical user interfaces. [paper]
55. [2023/05/19] Multimodal Web Navigation with Instruction-Finetuned Foundation Models. [paper]
56. [2023/01/30] WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics. [paper]
57. [2023/01/23] Lexi: Self-Supervised Learning of the UI Language. [paper]
58. [2022/10/06] Towards Better Semantic Understanding of Mobile Interfaces. [paper]
59. [2022/09/29] Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus. [paper]
60. [2022/07/04] WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. [paper]
61. [2022/05/23] Meta-gui: Towards multi-modal conversational agents on mobile gui. [paper]
62. [2022/02/16] A data-driven approach for learning to control computers. [paper]

1. [2024/12/02] Ponder & Press: Advancing Visual GUI Agent towards General Computer Control. [paper]
2. [2024/11/20] AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations. [paper]
3. [2024/11/18] Improved GUI Grounding via Iterative Narrowing. [paper]
4. [2024/11/15] The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use. [paper]
5. [2024/11/10] Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents. [paper]
6. [2024/11/01] WebOlympus: An Open Platform for Web Agents on Live Websites. [paper]
7. [2024/10/29] Auto-Intent: Automated Intent Discovery and Self-Exploration for Large Language Model Web Agents. [paper]
8. [2024/10/25] OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization. [paper]
9. [2024/10/24] OSCAR: Operating System Control via State-Aware Reasoning and Re-Planning. [paper]
10. [2024/10/24] AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant. [paper]
11. [2024/10/24] Infogent: An Agent-Based Framework for Web Information Aggregation. [paper]
12. [2024/10/22] Large Language Models Empowered Personalized Web Agents. [paper]
13. [2024/10/21] Beyond Browsing: API-Based Web Agents. [paper]
14. [2024/10/17] AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agents. [paper]
15. [2024/10/17] MobA: A Two-Level Agent System for Efficient Mobile Task Automation. [paper]
16. [2024/10/11] VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planning. [paper]
17. [2024/10/10] Agent S: An Open Agentic Framework that Uses Computers Like a Human. [paper]
18. [2024/10/09] ClickAgent: Enhancing UI Location Capabilities of Autonomous Agents. [paper]
19. [2024/10/01] Dynamic Planning for LLM-based Graphical User Interface Automation. [paper]
20. [2024/10/01] Multimodal Auto Validation For Self-Refinement in Web Agents. [paper]
21. [2024/09/25] Turn Every Application into an Agent: Towards Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents. [paper]
22. [2024/09/23] From Commands to Prompts: LLM-based Semantic File System for AIOS. [paper]
23. [2024/09/23] Steward: Natural Language Web Automation. [paper]
24. [2024/09/16] NaviQAte: Functionality-Guided Web Application Navigation. [paper]
25. [2024/09/14] PeriGuru: A Peripheral Robotic Mobile App Operation Assistant based on GUI Image Understanding and Prompting with LLM. [paper]
26. [2024/09/12] Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale. [paper]
27. [2024/09/11] Agent Workflow Memory. [paper]
28. [2024/08/28] Webpilot: A versatile and autonomous multi-agent system for web task execution with strategic exploration. [paper]
29. [2024/08/24] AutoWebGLM: A Large Language Model-based Web Navigating Agent. [paper]
30. [2024/08/24] Intelligent Agents with LLM-based Process Automation. [paper]
31. [2024/08/01] OpenWebAgent: An Open Toolkit to Enable Web Agents on Large Language Models. [paper]
32. [2024/08/01] Omniparser for pure vision based gui agent. [paper]
33. [2024/07/21] Towards LLMCI: Multimodal AI for LLM-Vision UI Operation. [paper]
34. [2024/07/17] Agent-e: From autonomous web navigation to foundational design principles in agentic systems. [paper]
35. [2024/07/04] MobileExperts: A Dynamic Tool-Enabled Agent Team in Mobile Devices. [paper]
36. [2024/07/01] Tree Search for Language Model Agents. [paper]
37. [2024/06/27] Read anywhere pointed: Layout-aware gui screen reading with tree-of-lens grounding. [paper]
38. [2024/06/11] CAAP: Context-Aware Action Planning Prompting to Solve Computer Tasks with Front-End UI Only. [paper]
39. [2024/06/03] Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration. [paper]
40. [2024/05/23] AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. [paper]
41. [2024/05/17] Latent State Estimation Helps UI Agents to Reason. [paper]
42. [2024/04/28] MMAC-Copilot: Multi-modal Agent Collaboration Operating System Copilot. [paper]
43. [2024/04/09] Autonomous Evaluation and Refinement of Web Agents. [paper]
44. [2024/04/03] PromptRPA: Generating Robotic Process Automation on Smartphones from Textual Prompts. [paper]
45. [2024/03/25] AIOS: LLM Agent Operating System. [paper]
46. [2024/03/05] Android in the zoo: Chain-of-action-thought for gui agents. [paper]
47. [2024/03/05] Cradle: Empowering Foundation Agents Towards General Computer Control. [paper]
48. [2024/02/23] On the Multi-turn Instruction Following for Conversational Web Agents. [paper]
49. [2024/02/19] CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation. [paper]
50. [2024/02/12] OS-Copilot: Towards Generalist Computer Agents with Self-Improvement. [paper]
51. [2024/02/09] ScreenAgent: A Vision Language Model-driven Computer Control Agent. [paper]
52. [2024/02/08] Ufo: A ui-focused agent for windows os interaction. [paper]
53. [2024/02/06] Dual-View Visual Contextualization for Web Navigation. [paper]
54. [2024/01/29] Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception. [paper]
55. [2024/01/25] WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models. [paper]
56. [2024/01/17] Seeclick: Harnessing gui grounding for advanced visual gui agents. [paper]
57. [2024/01/04] MobileAgent: enhancing mobile control via human-machine interaction and SOP integration. [paper]
58. [2024/01/03] GPT-4V(ision) is a Generalist Web Agent, if Grounded. [paper]
59. [2023/12/21] AppAgent: Multimodal Agents as Smartphone Users. [paper]
60. [2023/12/20] Assistgui: Task-oriented desktop graphical user interface automation. [paper]
61. [2023/12/04] MobileGPT: Augmenting LLM with Human-like App Memory for Mobile Task Automation. [paper]
62. [2023/12/04] Intelligent Virtual Assistants with LLM-based Process Automation. [paper]
63. [2023/11/13] GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation. [paper]
64. [2023/10/24] WebWISE: Web Interface Control and Sequential Exploration with Large Language Models. [paper]
65. [2023/10/12] A Zero-Shot Language Agent for Computer Control with Structured Reflection. [paper]
66. [2023/09/20] You only look at screens: Multimodal chain-of-action agents. [paper]
67. [2023/09/15] Laser: Llm agent with state-space exploration for web navigation. [paper]
68. [2023/08/29] AutoDroid: LLM-powered Task Automation in Android. [paper]
69. [2023/07/24] A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis. [paper]
70. [2023/06/14] Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control. [paper]
71. [2023/06/09] Mind2Web: Towards a Generalist Agent for the Web. [paper]
72. [2023/05/30] Sheetcopilot: Bringing software productivity to the next level through large language models. [paper]
73. [2023/05/23] Hierarchical prompting assists large language model on web navigation. [paper]
74. [2023/05/09] InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Language. [paper]
75. [2023/03/30] Language Models can Solve Computer Tasks. [paper]
76. [2022/09/19] Enabling conversational interaction with mobile ui using large language models. [paper]
77. [2022/05/23] Meta-gui: Towards multi-modal conversational agents on mobile gui. [paper]

1. [2024/12/09] BigDocs: An Open and Permissively-Licensed Dataset for Training Multimodal Models on Document and Code Tasks. [paper]
2. [2024/12/05] Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction. [paper]
3. [2024/10/31] AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents. [paper]
4. [2024/10/28] AssistEditor: Multi-Agent Collaboration for GUI Workflow Automation in Video Creation. [paper]
5. [2024/10/28] Shopping MMLU: A Massive Multi-Task Online Shopping Benchmark for Large Language Models. [paper]
6. [2024/10/28] MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI. [paper]
7. [2024/10/24] VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks. [paper]
8. [2024/10/22] Large Language Models Empowered Personalized Web Agents. [paper]
9. [2024/10/19] SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation. [paper]
10. [2024/10/17] MobA: A Two-Level Agent System for Efficient Mobile Task Automation. [paper]
11. [2024/10/07] Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents. [paper]
12. [2024/09/22] MobileViews: A Large-Scale Mobile GUI Dataset. [paper]
13. [2024/09/12] Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale. [paper]
14. [2024/09/06] WebQuest: A Benchmark for Multimodal QA on Web Page Sequences. [paper]
15. [2024/08/01] Omniparser for pure vision based gui agent. [paper]
16. [2024/07/26] OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automation. [paper]
17. [2024/07/22] AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks? [paper]
18. [2024/07/15] Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows? [paper]
19. [2024/07/13] RealWeb: A Benchmark for Universal Instruction Following in Realistic Web Services Navigation. [paper]
20. [2024/07/11] UICrit: Enhancing Automated Design Evaluation with a UI Critique Dataset. [paper]
21. [2024/07/07] WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks. [paper]
22. [2024/07/04] MobileExperts: A Dynamic Tool-Enabled Agent Team in Mobile Devices. [paper]
23. [2024/07/03] Amex: Android multi-annotation expo dataset for mobile gui agents. [paper]
24. [2024/07/01] CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents. [paper]
25. [2024/07/01] Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents. [paper]
26. [2024/06/27] Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding. [paper]
27. [2024/06/20] E-ANT: A Large-Scale Dataset for Efficient Automatic GUI NavigaTion. [paper]
28. [2024/06/20] Identifying User Goals from UI Trajectories. [paper]
29. [2024/06/19] GUI Action Narrator: Where and When Did That Action Take Place? [paper]
30. [2024/06/18] WebCanvas: Benchmarking Web Agents in Online Environments. [paper]
31. [2024/06/17] GUICourse: From General Vision Language Models to Versatile GUI Agents. [paper]
32. [2024/06/14] VideoGUI: A Benchmark f om Instructional Videos. [paper]
33. [2024/06/12] GUI Odyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices. [paper]
34. [2024/06/12] GUI-WORLD: A Dataset for GUI-oriented Multimodal LLM-based Agents. [paper]
35. [2024/06/12] MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents. [paper]
36. [2024/06/06] On the Effects of Data Scale on UI Control Agents. [paper]
37. [2024/06/05] WebOlympus: An Open Platform for Web Agents on Live Websites. [paper]
38. [2024/06/05] UGIF-DataSet: A New Dataset for Cross-lingual, Cross-modal Sequential actions on the UI. [paper]
39. [2024/06/01] WebSuite: Systematically Evaluating Why Web Agents Fail. [paper]
40. [2024/05/23] AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. [paper]
41. [2024/05/23] AGILE: A Novel Reinforcement Learning Framework of LLM Agents. [paper]
42. [2024/05/17] Latent state estimation helps ui agents to reason. [paper]
43. [2024/05/07] Mapping natural language instructions to mobile UI action sequences. [paper]
44. [2024/05/05] Visual grounding for desktop graphical user interfaces. [paper]
45. [2024/04/28] MMAC-Copilot: Multi-modal Agent Collaboration Operating System Copilot. [paper]
46. [2024/04/25] Benchmarking mobile device control agents across diverse configurations. [paper]
47. [2024/04/15] MMInA: Benchmarking multihop multimodal internet agents. [paper]
48. [2024/04/12] Llamatouch: A faithful and scalable testbed for mobile ui task automation. [paper]
49. [2024/04/11] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. [paper]
50. [2024/04/09] VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding? [paper]
51. [2024/04/09] GUIDE: Graphical User Interface Data for Execution. [paper]
52. [2024/04/08] Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs. [paper]
53. [2024/04/04] Autowebglm: Bootstrap and reinforce a large language model-based web navigating agent. [paper]
54. [2024/03/29] Evaluating language model agents on realistic autonomous tasks. [paper]
55. [2024/03/29] Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want. [paper]
56. [2024/03/26] AgentStudio: A Toolkit for Building General Virtual Agents. [paper]
57. [2024/03/18] Tur[k]ingBench: A Challenge Benchmark for Web Agents. [paper]
58. [2024/03/15] Computer User Interface Understanding. A New Dataset and a Learning Framework. [paper]
59. [2024/03/12] Workarena: How capable are web agents at solving common knowledge work tasks? [paper]
60. [2024/03/06] PPTC-R benchmark: Towards Evaluating the Robustness of Large Language Models for PowerPoint Task Completion. [paper]
61. [2024/03/05] Android in the Zoo:Chain-of-Action-Thought for GUI Agents. [paper]
62. [2024/02/27] Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. [paper]
63. [2024/02/23] On the Multi-turn Instruction Following for Conversational Web Agents. [paper]
64. [2024/02/09] Understanding the weakness of large language model agents within a complex android environment. [paper]
65. [2024/02/08] WebLINX: Real-World website navigation with Multi-Turn dialogue. [paper]
66. [2024/01/29] Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception. [paper]
67. [2024/01/25] Webvoyager: Building an end-to-end web agent with large multimodal models. [paper]
68. [2024/01/25] GPTVoiceTasker: Advancing Multi-step Mobile Task Efficiency Through Dynamic Interface Exploration and Learning. [paper]
69. [2024/01/24] Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. [paper]
70. [2024/01/24] Agentboard: An analytical evaluation board of multiturn LLM agents. [paper]
71. [2024/01/17] Seeclick: Harnessing gui grounding for advanced visual gui agents. [paper]
72. [2023/12/26] AutoTask: Executing Arbitrary Voice Commands by Exploring and Learning from Mobile GUI. [paper]
73. [2023/12/25] WebVLN: Vision-and-Language Navigation on Websites. [paper]
74. [2023/12/20] ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation. [paper]
75. [2023/11/30] Exposing Limitations of Language Model Agents in Sequential-Task Compositions on the Web. [paper]
76. [2023/11/21] GAIA: a benchmark for general AI assistants. [paper]
77. [2023/11/13] GPT-4V in wonderland: Large multimodal models for Zero-Shot smartphone GUI navigation. [paper]
78. [2023/11/03] Pptc benchmark: Evaluating large language models for powerpoint task completion. [paper]
79. [2023/10/16] OpenAgents: An Open Platform for Language Agents in the Wild. [paper]
80. [2023/09/20] You Only Look at Screens:Multimodal Chain-of-Action Agents. [paper]
81. [2023/08/29] AutoDroid: LLM-powered Task Automation in Android. [paper]
82. [2023/08/07] Agentbench: Evaluating llms as agents. [paper]
83. [2023/07/25] Webarena: A realistic web environment for building autonomous agents. [paper]
84. [2023/07/19] Android in the wild: A large-scale dataset for android device control. [paper]
85. [2023/06/09] Mind2Web: Towards a Generalist Agent for the Web. [paper]
86. [2023/05/30] Sheetcopilot: Bringing software productivity to the next level through large language models. [paper]
87. [2023/05/25] On the tool manipulation capability of open-source large language models. [paper]
88. [2023/05/14] Mobile-env: A universal platform for training and evaluation of mobile interaction. [paper]
89. [2023/05/14] Mobile-env: Building qualified evaluation benchmarks for llm-gui interaction. [paper]
90. [2023/04/14] Droidbot-gpt: Gpt-powered ui automation for android. [paper]
91. [2023/04/10] OpenAGI: When LLM Meets Domain Experts. [paper]
92. [2023/03/13] Vision-Language models as success detectors. [paper]
93. [2022/11/14] Ugif: Ui grounded instruction following. [paper]
94. [2022/10/08] Understanding html with large language models. [paper]
95. [2022/09/29] MUG: Interactive Multimodal Grounding on User Interfaces. [paper]
96. [2022/09/16] ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots. [paper]
97. [2022/07/04] Webshop: Towards scalable real-world web interaction with grounded language agents. [paper]
98. [2022/05/23] Meta-gui: Towards multi-modal conversational agents on mobile gui. [paper]
99. [2022/02/04] A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility. [paper]
100. [2021/04/17] Mobile app tasks with iterative feedback (motif): Addressing task feasibility in interactive visual environments. [paper]
101. [2021/01/23] Websrc: A dataset for web-based structural reading comprehension. [paper]
102. [2018/08/28] Mapping natural language commands to web elements. [paper]
103. [2017/11/06] Building natural language interfaces to web apis. [paper]
104. [2017/08/06] World of bits: An open-domain platform for web-based agents. [paper]

1. [2024/11/04] Attacking Vision-Language Computer Agents via Pop-ups. [paper]
2. [2024/10/23] MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Control. [paper]
3. [2024/10/22] Advweb: Controllable black-box attacks on vlm-powered web agents. [paper]
4. [2024/10/11] Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents. [paper]
5. [2024/10/09] ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents. [paper]
6. [2024/09/17] Eia: Environmental injection attack on generalist web agents for privacy leakage. [paper]
7. [2024/08/05] Caution for the Environment: Multimodal Agents are Susceptible to Environmental Distractions. [paper]
8. [2024/07/12] Security Matrix for Multimodal Agents on Mobile Devices: A Systematic and Proof of Concept Study. [paper]
9. [2024/06/18] Adversarial Attacks on Multimodal Agents. [paper]
10. [2024/02/26] WIPI: A New Web Threat for LLM-Driven Web Agents. [paper]
11. [2023/08/03] From Prompt Injections to SQL Injection Attacks: How Protected is Your LLM-Integrated Web Application? [paper]
12. [2024/12/19] Agent-SafetyBench: Evaluating the Safety of LLM Agents. [paper]

OPPO is dedicated to developing the most advanced foundation model technologies and creating personalized user experiences. Our mission is focused on developing revolutionary AI Native Phones that will shape the future. The Personal AI Team is primarily focused on cutting-edge research in (multimodal) LLMs, AI Agents and personalization (e.g., team's new work AI PERSONA). The team leverages OPPO’s ample platform resources, including data and computing power, to continuously invest in these key areas. We are currently hiring algorithm interns, new graduates, and experienced candidates. We expect you to be proficient in algorithms or engineering of large foundation models. If interested, please contact via email: [email protected].

01.AI is a global leader in technology and applications of large foundation models. Its latest flagship model, Yi-Lightning, ranked #1 in China and #6 globally in the LMSys Chatbot Arena updated on October 14, 2024. The post-training team is responsible for cutting-edge research and engineering of post-training techniques for foundation models. We are currently hiring algorithm interns, new graduates, and experienced candidates. We expect you to be proficient in algorithms or engineering of large foundation models. If interested, please contact via email: [email protected].

To better foster community collaboration, we have listed some repositories related to OS Agents：

1. aialt/awesome-mobile-agents: https://github.com/aialt/awesome-mobile-agents
   
2. OSU-NLP-Group/GUI-Agents-Paper-List: https://github.com/OSU-NLP-Group/GUI-Agents-Paper-List
   
3. vyokky/LLM-Brained-GUI-Agents-Survey: https://github.com/vyokky/LLM-Brained-GUI-Agents-Survey
   
4. wendell0218/GVA-Survey: https://github.com/wendell0218/GVA-Survey
   

The repo is still being updated rapidly🚀. Please let us know if you notice any mistakes or would like any work related to OS Agents to be included in our list by e-mail: [email protected].

Considering that the current bib citation points to our repository, we will update it to point to the paper as soon as the preprint server is available. Please stay tuned for updates. Before this, if you find our repository helpful, we would appreciate it if you could cite:

@misc{hu2024osagents,  
  title        = {OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use},  
  author       = {Xueyu Hu and Tao Xiong and Biao Yi and Zishu Wei and Ruixuan Xiao and Yurun Chen and Jiasheng Ye and Meiling Tao and Xiangxin Zhou and Ziyu Zhao and Yuhuai Li and Shengze Xu and Shawn Wang and Xinchen Xu and Shuofei Qiao and Kun Kuang and Tieyong Zeng and Liang Wang and Jiwei Li and Yuchen Eleanor Jiang and Wangchunshu Zhou and Guoyin Wang and Keting Yin and Zhou Zhao and Hongxia Yang and Fan Wu and Shengyu Zhang and Fei Wu},  
  year         = {2024},  
  howpublished = {\url{https://github.com/OS-Agent-Survey/OS-Agent-Survey/}},  
}