Google's AI Mode gets Gemini's multimodal powers making it a visual search expert

What you need to know

• Google announced today that it will be integrating Gemini's multimodal capabilities into AI Mode on Google Search.
• This will help expand Search's capabilities by answering comprehensive questions in one go.
• Google Search's AI Mode becomes available to more "Labs" users in the U.S.

Google announced today that it will give AI Mode on Google Search more powers to perform its tasks. In a press release the company said that it is expanding its Gemini Capabilities to AI Mode on Google search, with the help of Google Lens.

"This experience brings together powerful visual search capabilities in Google Lens with a custom version of Gemini, so you can easily ask complex questions about what you see," Google stated.

It allows users who have access to AI Mode to be able to snap a photo or record a video and ask the chatbot a comprehensive question about what they see. For instance, users can ask exploratory and open-ended questions in AI mode, along with more complicated tasks — "like comparing two products, exploring how-tos or planning a trip," the Keyword added.

AI Mode gets this ability from Gemini’s multimodal capabilities. The tech giant added that once this mode is enabled, it can understand every scene in an image, including the different materials, colors or shapes, and arrangements of multiple objects, as seen in the gif above. For example, AI mode is able to read the titles of the books, capture the theme and commonalities of these books, and suggest similar high-rated books to the user.

It uses a “query fan-out” technique, just like the text-based search option and AI Mode, then issues multiple queries about the image as a whole and the objects within the image. It is important to note that AI Mode will be made available on both mobile and desktop search versions. However, it remains unclear if it's the same for the Lens feature, as the gif shows only a phone using this new capability.

That said, Google started testing the AI Mode only last month and has since received positive feedback from Google One AI Premium subscribers who were able to try it out. Users felt that AI Mode was able to quickly answer their complex and nuanced questions and also gave them a lot more information about a certain topic than a general Google Search.

Lastly, Google added that it's constantly testing to improve AI Mode based on feedback that comes from users in "Labs." AI Mode is now available on both Android and iOS via the Google app for those with a Google One AI Premium subscription.