<p>In a recent study, researchers at Apple have proposed a new multimodal large language model called Ferret-UI. This model is specifically designed to improve the understanding and interaction with mobile user interfaces (UI). Ferret-UI incorporates referring, grounding, and reasoning abilities, allowing it to excel beyond most open-source UI models. With its enhanced capabilities, Ferret-UI can perform a range of tasks including icon recognition, finding text, and widget listing. What sets Ferret-UI apart is its ‘any resolution’ capability, which enables it to handle various screen sizes accurately and reliably. In fact, Ferret-UI surpasses GPT-4V on elementary UI tasks. This model developed by Apple represents a significant advancement in the field of mobile UI understanding and holds great potential for improving user experiences.</p>


Collections

Researchers at Apple have developed Ferret-UI, a multimodal large language model that enhances mobile UI understanding and interaction. It has referring, grounding, and reasoning abilities, can perform tasks like icon recognition and finding text, and handles various screen sizes accurately.

Enhancing Mobile UI Understanding with Ferret-UI: A Multimodal Large Language Model