Make Every Application An AI Agent

Research by Microsoft suggests that AI agents can operate more efficiently by interacting with application programming interfaces (APIs) instead of graphical user interfaces (GUIs). The paper highlights that relying on APIs can minimize the latency and errors associated with UI interactions, making task completion quicker and more reliable. Multimodal large language models also enhance AI agents' performance by allowing them to interact with UIs through a combination of text, images, and buttons. While there are challenges in converting some GUI tasks to APIs, a hybrid approach ensures better task efficiency and coverage.