Multimodal search lets people search with more than one type of input, such as text, images, voice and video, sometimes all at once. Point your camera at a plant and ask "why are the leaves turning yellow?", and you've done a multimodal search.
It's growing quickly, and it changes what search engines need from your website. Your images, videos and product data are no longer decoration. They're part of how people find you.
What is multimodal search?
Multimodal search is a search experience that accepts and understands multiple "modes" of information, typically text, images, audio and video, and can combine them in a single query. The results can be multimodal too: a written answer alongside images, product listings, videos and links.
Google Lens is the best-known example. You can photograph an object and search for it, add a text question on top of the photo, or record a short video while asking a question aloud. Google described these video and voice features in Lens in October 2024. AI tools like ChatGPT and Gemini also accept images and files in a conversation.
Under the hood, multimodal search depends on models that can relate different kinds of content to each other, so a photo of a red velvet sofa can be matched to text describing "red velvet sofa."
Why multimodal search matters
The volume is significant. In October 2024, Google said people use Lens for nearly 20 billion visual searches every month, and that about 20% of Lens searches are shopping-related.
It's also moving into AI search. In April 2025, Google added multimodal search to AI Mode, combining Lens with a custom version of Gemini so people could ask questions about a photo and get a detailed answer with links. In September 2025, Google added visual exploration to AI Mode, letting people refine image results conversationally. And as of May 2026, Google says the redesigned Search box in AI Mode accepts text, images, files and videos.
For businesses, the implication is straightforward. If your products, places or work are visual, people increasingly search for them visually. Pages with clear, well-labeled images and videos give these systems more to match.
How multimodal search works
Understanding each input
Vision models identify objects, text, colors and materials in images. Speech recognition turns spoken questions into text. Language models interpret the question. Google says AI Mode uses Lens to identify specific objects while Gemini interprets the whole scene, including how objects relate to each other.
Placing content in a shared space
Many systems use multimodal vector embeddings, which represent images and text in the same numerical space. That lets a photo be compared with product descriptions, captions and web pages by meaning.
Fanning out queries
Google describes a visual search fan-out technique that analyzes the main subject and secondary objects in an image and runs several background queries. This builds on the broader query fan-out approach.
Assembling the answer
The system combines what it found into a response: text, images, shopping listings, videos and source links. For shopping, Google draws on its Shopping Graph, which it said in September 2025 includes more than 50 billion product listings.
Multimodal search in practice
Imagine a hypothetical independent furniture store. A shopper sees a mid-century armchair at a friend's house, snaps a photo and asks, "where can I buy a chair like this in green?"
The search system identifies the chair's style, shape and materials, then looks for matching products and pages. The store has a chance to show up if its product pages have sharp, original photos taken from several angles, descriptive file names, accurate alt text, Product structured data with color and price, and an up-to-date Merchant Center feed. A store with tiny, generic stock photos and no product details gives the system much less to connect.
How to optimize for multimodal search
- Use original, high-quality images. Show your actual products, locations and work. Google recommends sharp images that are still optimized for speed in its image SEO best practices.
- Make images crawlable. Use standard HTML image elements. Google says it doesn't index CSS background images.
- Write descriptive alt text and file names. Describe the image accurately in context, and avoid stuffing keywords, which Google warns against.
- Place images near relevant text. Surrounding headings and captions help systems understand what an image shows.
- Add structured data. Product, LocalBusiness and VideoObject markup clarify what your images and videos represent. Keep it consistent with visible content.
- Give videos a proper home. Google's video SEO guidance recommends dedicated watch pages, unique titles and descriptions, and video sitemaps.
- Keep product and business data current. Google recommends keeping Merchant Center and Business Profile information up to date for its AI features.
Common mistakes and misconceptions
- Relying on stock photos. Generic images look like everyone else's and give visual search little to distinguish your product.
- Putting key information only in images. Text baked into graphics, like prices or specs, should also appear as HTML text.
- Treating alt text as a keyword field. Alt text is for describing the image to people who can't see it. Accurate descriptions serve both accessibility and search.
- Assuming it's only for retailers. Restaurants, contractors, clinics and service firms all benefit from clear images of their work, spaces and teams.
How Butter approaches multimodal search
Butter doesn't offer photography or video production, so our role in multimodal search is the technical and structural side. In our GEO+SEO plan, unlimited on-page optimization can cover alt text, image placement and file naming, and our technical SEO fixes address crawl issues that keep images or videos from being found. Our schema markup guidance covers the Product, LocalBusiness and VideoObject types that help systems understand visual content. We keep it practical: make the visual assets you already have easy for search engines to read.
Frequently asked questions
Is multimodal search the same as visual search?
Visual search is one part of it. Multimodal search covers any combination of inputs, like a photo plus a text question, or a video plus a spoken question.
Does alt text help with multimodal search?
Google's image guidance says it uses alt text, along with computer vision and the page's content, to understand what an image shows. Alt text also improves accessibility. Write accurate, concise descriptions rather than lists of keywords.
Which tools support multimodal search?
As of October 2026, Google Lens, Google AI Mode, ChatGPT and Gemini all accept images, and several accept voice or video. Features vary by country and device.