Multimodal Search: Getting Recommended When Customers Search With a Photo

A growing number of searches don't start with words anymore. Someone photographs a pair of shoes on the street, screenshots a product from a video, or points their phone at a broken part under the sink and asks the AI “where can I get this” or “who can fix this.” The interface changed and most businesses haven't noticed, because they're still optimizing for typed phrases while a real slice of demand now arrives as pixels. If you sell anything visual — products, spaces, food, physical work — this is worth understanding now rather than after it's obvious.

Here's the direct answer to how it works: an AI handling an image search runs two steps, not one. First it identifies what's in the image and turns it back into words. Then it recommends using those words, exactly the way it would with a typed query. Almost every business that loses in multimodal search loses at the first step, and the fix for the first step is different from anything in a normal SEO or AEO playbook.

What actually happens when someone searches with an image

When you hand an AI a photo, it can't recommend a picture. It has to convert what it sees into language it can reason about — “a white ceramic pendant light with a brass stem, mid-century style” or “a cracked P-trap under a bathroom sink” or “a storefront that looks like a specialty coffee roaster.” That conversion is the identification step, and everything downstream depends on it being right. Once the model has words, it's back on familiar ground: it runs essentially the same assembly process it uses for any recommendation, pulling names it can corroborate and describe for that now-textual query.

So the uncomfortable truth is that multimodal search still gets decided in text. The image just determines which text query you end up competing in. Get identified as the wrong thing and you're competing for a customer who was never going to buy from you. Get identified precisely and you're in a query you can actually win with the authority work you'd do anyway.

The identification step is where most businesses lose

The model identifies an image using whatever visual and contextual evidence it can find. For a product, that means the model does better when your item is visually documented on the open web in a way that ties the picture to a name: clear product photos from multiple angles, associated with a specific product name and category, ideally echoed across more than one source. If the only images of your product are on a single page with a vague caption, the model has thin evidence to match against and will often round you off to a generic category — or to a bigger competitor whose product it can identify confidently.

For a physical business — a restaurant, a studio, a storefront — identification leans on the association between images of your place and your name and location across the web: your listings, your reviews with photos, mentions that pair a picture with who and where you are. A business that's photographed often and consistently labeled is easy to identify. A business with two stock photos and no visual footprint is a coin flip.

There's a compounding effect worth naming. Every correctly-labeled image of your product or place doesn't just help with that one photo — it strengthens the model's overall association between your look and your name, which makes the next identification easier too. A business with a rich, consistent visual footprint builds a kind of recognition equity, where even a partial or awkward customer photo still resolves to it. A business with a thin footprint stays a coin flip indefinitely, because there's never enough evidence for the association to firm up. This is why I treat visual documentation as an asset you accumulate, not a task you check off.

I want to be honest about the limits of my confidence here: the exact weighting of these systems isn't public and it's moving fast. But the direction is not subtle. The businesses that are easy to identify from an image are the ones whose visual presence on the web is abundant, labeled, and consistent. That's the lever.

Text still decides the recommendation

Once you've been correctly identified, the second step is the familiar one, and it rewards the same fundamentals I write about constantly. The model recommends the identified thing based on corroboration, describability, recency, and fit to any qualifier in the ask. “Where can I buy this pendant light” becomes a sourcing question, and it'll favor sellers it can confirm actually carry it. “Who can fix this” becomes a local-service question, and it'll favor providers whose specialty and service area are clearly stated and independently confirmed.

This is actually good news. It means you don't need a separate strategy for the recommendation half of multimodal search — the authority, review, and consistency work that makes you recommendable in text already carries over. What's genuinely new, and what most people are neglecting, is the identification half.

The playbook for visual visibility

If image-started searches matter for your category, here's where I'd put the effort.

• Document your products and spaces abundantly. Multiple clear images, multiple angles, good lighting, on more than one property — your site, your listings, review platforms, wherever your things legitimately appear. Sparse visual evidence is the single most common reason a business can't be identified.
• Label every image in plain language. Tie each picture to a specific name, category, and the qualifiers that matter, in captions, alt text, and surrounding copy. You're teaching the model the words to convert the image into.
• Keep visual details consistent with your text. If your photos show one thing and your description says another, identification gets shaky. The picture and the words about it should agree.
• Encourage customer photos. Reviews and posts with images of your product or place are powerful corroboration — independent visual evidence that pairs your look with your name.
• Then do the ordinary authority work. Once identification is solid, everything that makes you recommendable in text — corroborated specialty, clear service area, freshness, consistency — does the rest.

How this plays out by what you sell

The identification problem looks different depending on your category, and it's worth being specific about your own. If you sell distinctive physical products — furniture, jewelry, apparel, specialty equipment — you have the most to gain and the most to lose, because a well-documented item can be matched to your name from a single photo, while an under-documented one gets handed to whichever competitor the model can confidently name. Abundant, labeled product imagery is close to non-negotiable here.

If you run a local physical business — a restaurant, gym, clinic, or shop — identification rides on the association between images of your place and your name and location. The work is making sure your storefront, interior, and offerings are photographed and correctly labeled across your listings and reviews, so a snapshot of your sign or your space resolves to you rather than to a generic “coffee shop near here.” If you provide a repair or trade service, the customer's image is usually of the problem, not of you — a leaking valve, a cracked screen, a dead furnace. There, the identification step is about the object, and your job is on the recommendation side: make it unmistakable that you fix exactly that thing in exactly that area. And if you sell something purely abstract — software, consulting, finance — image search barely touches you today, and you can spend your attention elsewhere with a clear conscience.

My honest read on where this goes

I don't think image-first search replaces typed search; I think it becomes a normal on-ramp alongside it, especially on phones and especially for anything you can see before you can name. That makes visual documentation a form of visibility infrastructure, the same way a clean, consistent web presence became infrastructure for text. The businesses that get ahead won't be the ones with the most artistic photography — they'll be the ones whose products and places are documented so thoroughly and labeled so plainly that a machine can look at a customer's blurry phone snapshot and know, without guessing, exactly who makes that and where to get it. That's an unglamorous, do-it-once kind of project, which is exactly why the businesses that do it will quietly own the queries nobody else prepared for.

About the author

Scott Tischler is the Founder & Chairman of AIrecommend.ai and a practitioner-authority on AI search and Answer Engine Optimization. With 20+ years in marketing technology — including American Express, MetLife, and UBS — and executive and professional study at Wharton, Harvard, and Oxford, he helps businesses become the ones AI recommends.