[ad_1]
Updated on April 13th: Article originally posted on April 10th.
This may be the latest buzzword about Android, but Apple’s iPhone is not considered an AI-powered smartphone. That’s about to change, and we now know one way Tim Cook and his team plan to catch up.
The details are in a newly published research paper by Cornell University researchers working with Apple. Titled “Ferret-UI: Understanding Grounded Mobile UI with Multimodal LLM”, it focuses on what appears on the screen, especially the elements of the mobile user interface (for example, the iPhone).
Thanks to a large amount of training data, you can select icons, search for text, parse widgets, describe what’s on the screen in text, parse interface elements, and navigate the display guided by open-ended instructions. Is possible. Display a prompt.
Ferret was released in October 2023 and was designed to recognize exhibit content by analyzing photos and images. This nifty-titled upgrade to his Ferret-UI offers several benefits for iPhone users and could easily be adapted to the improved AI-powered Siri.
Being able to describe screens independent of the app opens up richer avenues for accessibility apps, eliminating the need to pre-program responses and actions. People who want to perform complex tasks or find obscure options on their phone can ask Siri to open complex apps and use obscure functions hidden deep within the menu system. can do.
Developers can use Ferret-UI as a testing tool to make the MLMM perform tasks as if it were a 14-year-old with little experience with social networking, or a 75-year-old trying to connect to Facetime. You can also simulate users. their grandchildren.
Update: Saturday, April 13th: Backend code along with academic paper pointing to Siri’s AI upgrade Discovered by Nicolas Alvarez Refers to new server-side tools for individual iPhones.
The features are labeled “Safari Browsing Assistant” and “Encrypted Visual Search.” Both of these tie in with the features described in the Ferret-UI study, but there are some caveats to these findings. Since this is server-side code, it is easy to change these functions and use different code. Rather than leveraging AI, it can be tied to more mundane code. Or these may be placeholders for use in future products or not.
Visual Search appears to be built into the code of visionOS and Apple Vision Pro headsets, but it’s worth noting that the feature has not yet been released.
These aren’t strong signs of Apple’s AI path in and of themselves, but they are part of a growing body of evidence about Apple’s approach.
Google publicly began its AI-first smartphone launch rush on October 4, just over three weeks after the iPhone 15 was released. Tim Cook and his team didn’t make any big announcements or draw attention to the AI improvements hidden in their product. Photo processing and text auto-correction could give Android a head start on AI, giving hope to Google’s mobile platform.
Apple’s Worldwide Developer Conference will be held in June and will be the first moment Apple can interact with the public about its AI plans as it lays the groundwork for the September launch of the iPhone 16 and iPhone 16 Pro.
Until then, we need to continue with the academic side of Apple’s AI approach.
Read why iPhone’s approach to AI disrupts iPhone 16 and iPhone 16 Plus specs…
follow me twitter Or LinkedIn. check out my website.
[ad_2]
Source link
