Answer Volatility: Why AI Recommends You One Day and Skips You the Next

Here's the direct answer: AI assistants don't return the same recommendation every time you ask. Ask ChatGPT, Gemini or Perplexity the same question three times and you can get three different lists of businesses. That instability isn't a bug you can eliminate — it's how these systems are built. The practical consequence is that a single check tells you almost nothing. If you want to know whether AI actually recommends you, you have to measure the pattern across many runs, not the result of one lucky prompt.

I call this answer volatility, and in my experience it's the single most misunderstood thing about AI visibility. Business owners screenshot one good answer and assume they've won. Or they see one bad answer and panic. Both are reading noise as signal. Let me walk through why the volatility exists, what it means for how you measure, and how I actually track it.

Why the same question gives different answers

There are three separate sources of variance stacked on top of each other, and it helps to keep them apart.

The first is sampling randomness inside the model. Language models generate text by predicting one token at a time from a probability distribution. Most systems deliberately introduce a little randomness into that choice so the output feels natural rather than robotic. That's why the wording changes run to run — and sometimes the wording change is small, but sometimes it flips which business gets named first, or whether a fourth option makes the cut.

The second is retrieval variance. Modern assistants don't answer purely from memory; they run live searches and read pages before composing a reply. The search results underneath can shift between two requests made minutes apart — a page gets re-indexed, a source goes stale, the engine picks a slightly different set of documents to ground its answer. Different source documents in, different names out.

The third is context sensitivity. The exact phrasing of the prompt, the conversation that came before it, the location and account signals attached to the session — all of it steers the answer. "Best physical therapist near me" and "recommend a physical therapist in [city]" are, to the model, meaningfully different questions, and they can surface different winners.

Stack those three together and you get a system that is directionally stable but never identically stable. My read, not a guarantee: the businesses with genuine authority tend to show up consistently across the noise, and the marginal ones flicker in and out. Which is exactly why the flicker is worth measuring.

The single-check trap

The most common mistake I see is treating one answer as the truth. Someone asks ChatGPT "who's the best [category] in [town]," sees their name, and closes the laptop satisfied. Or doesn't see their name, and concludes AI has blacklisted them. Neither conclusion is supported by one data point.

Think about it the way you'd think about a coin you suspect is weighted. One flip tells you nothing. Ten flips start to tell you something. A hundred flips tell you the real bias. AI recommendations work the same way: your visibility isn't a yes/no state, it's a frequency — the share of times you get named when a relevant question is asked. A business that appears in eight of ten runs is in a completely different position from one that appears in two of ten, even though a single check could have shown either one "winning" or "losing."

This is also why competitive intelligence based on screenshots is so misleading. Your competitor sends around a screenshot where the AI raves about them. That screenshot is real — but it's one flip. It doesn't tell you whether they win 20% of the time or 90% of the time, and that difference is the entire game.

How to measure volatility instead of fighting it

Once you accept that the output is a distribution rather than a fact, the measurement approach falls out naturally. You stop asking "did I appear?" and start asking "how often, and how prominently, do I appear across a realistic set of questions and repeated runs?"

Here's the framework I use.

Build a prompt set, not a prompt. Write down the ten to thirty ways a real customer would actually phrase the question that should surface you. Include the obvious ones ("best [category] in [city]") and the messy real-world ones ("who should I call to [solve specific problem]"). This set is your measurement instrument, and it should stay fixed so your readings are comparable over time.

Run each prompt multiple times. Because of the randomness above, one pass per prompt isn't enough. Run each several times, ideally in fresh sessions so prior context doesn't leak in. Now instead of one answer you have a spread, and the spread is the point.

Record presence and position, not just presence. Getting named last in a list of six is not the same as being the first business the model recommends. Track both whether you appear and where. Over enough runs, an average position emerges that's far more honest than any single answer.

Compute a frequency, then watch it move. The number that matters is something like "named in 6 of 10 runs, usually second." That's a baseline. The value isn't the absolute figure on day one — it's the trend after you do authority-building work. If you go from 3-of-10 to 7-of-10 over a quarter, that's real movement you can see through the noise.

This is the core of what we do at AIrecommend.ai, and it's why I keep pushing owners away from the screenshot habit. You cannot manage what you're measuring one flip at a time.

A worked example from the field

Let me make this concrete with the kind of case I run into constantly. Say you own a med spa and you want to know whether AI recommends you. You ask ChatGPT "best med spa in [your city]" once, see your name in third place, and feel good. If you'd stopped there, you'd have logged a win.

But run that same prompt ten times in fresh sessions and you might see this: named in four of ten, position ranging from second to "not mentioned." Then run the variations a real customer would type — "where should I go for [specific treatment] near me," "who does the best [service] in [neighborhood]," "affordable med spa [city]" — and the picture fragments further. You dominate the treatment-specific prompts because your site talks about that treatment in plain language, and you vanish on the generic "best med spa" prompts because a competitor has more third-party coverage.

That fragmented picture is the truth, and no single screenshot could have shown it to you. It also tells you exactly what to do: you're already winning the specific-intent questions, so the growth is in building the broad authority that wins the generic ones. One check would have told you "we're fine." Thirty checks told you where the money is.

How often to take a reading

People ask me whether they need to measure daily. Usually not. The engines change, the web changes, and your own authority work takes weeks to land, so a sensible cadence for most businesses is a fuller re-measurement monthly, with lighter spot-checks in between if you're actively running a campaign. Measuring too often just means you're staring at day-to-day noise and tempted to react to it. Measuring too rarely means you miss a slide until it's a cliff. Monthly is a reasonable middle for most owners, more often during an active push.

The other discipline is to keep the instrument fixed. If you change your prompt set every time you measure, you can't compare readings — you've swapped the ruler mid-measurement. Lock the prompts, lock the method, and let the numbers move. When you do add prompts because customer language shifted, note it, so you know a jump came from a new question rather than a real change in your standing.

What volatility tells you about your authority

The amount of volatility is itself a signal, and this is the part most people miss. A business that appears rock-solid in nearly every run has authority the model is confident about — consistent facts across the web, strong corroboration, an unambiguous identity. A business that flickers wildly is one the model is unsure about. It has just enough of a footprint to sometimes surface, but not enough to be a reliable answer.

So high volatility is a diagnosis: you're on the bubble. The work that reduces it — getting your facts consistent everywhere, earning third-party mentions, making your positioning legible — is the same work that lifts your frequency. As your authority solidifies, you'll typically see two things happen together: you get named more often, and the run-to-run wobble shrinks. Stability is what winning looks like from the inside.

The reverse is worth watching too. If you were consistently named and then start flickering, something changed — a competitor earned new coverage, your information went stale, a contradiction crept into your web presence. Volatility rising is an early warning that shows up before you fully drop out. You only catch it if you're measuring continuously rather than checking once a quarter.

The honest limits of this

I want to be careful not to oversell precision here. You are measuring a moving target with a noisy instrument, and the engines change their behavior without notice. Your frequency numbers are estimates, not audited figures, and I'd treat anyone who quotes you a decimal-point-precise "AI visibility score" with some skepticism. What you're really building is a relative picture — you versus your competitors, this quarter versus last — and that relative picture is reliable enough to steer real decisions even when the absolute numbers wobble.

The mindset shift is the whole thing. AI recommendation is not a light switch that's on or off. It's a weather pattern. You don't ask "is it raining right now" and call that the climate. You watch the pattern across many days and act on the trend. Do that, and the volatility stops being confusing and starts being one of the most useful diagnostics you have.

About the author

Scott Tischler is the Founder & Chairman of AIrecommend.ai and a practitioner-authority on AI search and Answer Engine Optimization. With 20+ years in marketing technology — including American Express, MetLife, and UBS — and executive and professional study at Wharton, Harvard, and Oxford, he helps businesses become the ones AI recommends.