If you ask ChatGPT the same question twice, you can get two different answers. Ask it ten times and you might be named in six, skipped in four, and described three slightly different ways. So when someone runs a single test, sees their business named, and declares "we're winning in AI," they've measured almost nothing. The honest answer to "how visible are we in AI?" isn't a yes or no from one check. It's a rate, and a rate needs repetition to be real.
This is the measurement problem underneath all of answer engine optimization, and it's the one people most want to skip. I run these tests constantly, and the single most common mistake I see — from smart operators — is treating one lucky (or unlucky) AI answer as the truth. Here's how I think about testing prompts enough times to actually trust the number, without pretending it's more precise than it is.
Why one test tells you nothing
AI answers are non-deterministic by design. The same prompt can produce different outputs because of the randomness built into how models generate text, differences in what gets retrieved at that moment, personalization and location, and ongoing changes on the model's side. None of that is a bug you can eliminate. It's the environment you're measuring in.
So a single result is a sample of one from a distribution you can't see. If you're named 50% of the time and you test once, you'll conclude either "we're in" or "we're out" — and you'll be confidently wrong half the time. If you're named 20% of the time, one lucky test makes you declare victory over a position that's actually weak. The number you care about isn't "did I appear?" It's "what share of the time do I appear?" — and you cannot estimate a share from one trial.
Think in rates, not verdicts
The mental shift that fixes most of this: stop asking "am I recommended?" and start asking "how often am I recommended, out of how many tries?" That reframing turns a coin flip into a measurable quantity. Ten tests where you appear seven times is a 70% appearance rate. That's a number you can track, compare against a competitor, and watch move after you do the work.
Once you're thinking in rates, the natural next question is how many trials you need before the rate is trustworthy. And here the honest answer is: more than you'd like, but fewer than you might fear — and it depends entirely on how precise you need to be.
How many runs is enough?
There's no single magic number, but there are useful anchors. The core tension is that the fewer times you test, the wider your uncertainty. A little intuition, without pretending this is a lab experiment:
- One to two runs: a smoke test, not a measurement. Useful only to confirm the prompt works and to eyeball whether you're anywhere in the conversation. Never report a rate off this.
- Around five runs: a rough read. Enough to tell "basically never" from "sometimes" from "usually." Good for a quick directional gut-check across many prompts, but the margin of error is large — a 3-out-of-5 could easily be a true rate anywhere from roughly a third to over 80%.
- Around ten runs: a working baseline. This is where I'd start trusting a rate for a single prompt for practical decisions. You can distinguish strong from weak positions and track meaningful movement, while accepting the interval is still wide.
- Twenty to thirty-plus runs: a tighter estimate. When a specific prompt really matters — it's your money query, or you're proving a result to a client — this is what it takes to narrow the uncertainty enough to make confident claims about a few-point change.
The reason more runs help isn't mysterious. Estimating a rate from samples gets more precise roughly with the square root of the number of trials, which means going from 5 to 20 tests roughly halves your uncertainty, but going from 20 to 80 only halves it again. There are diminishing returns, so past a point you're better off testing more prompts than hammering one prompt hundreds of times. My rule of thumb: ten runs per prompt as a default, more only when a specific prompt's precision genuinely drives a decision.
What to hold constant, and what to vary
Repetition only means something if you're repeating the right thing. A few disciplines make the number honest.
Fix the prompt wording, then also test variants separately. Run the exact same prompt N times to get a clean rate for that phrasing. But remember that real users phrase things dozens of ways, so also measure a handful of natural variants as their own little tests. One prompt's rate is not your overall visibility; it's one data point in a set.
Control for the obvious confounders. Personalization and history can skew results, so test in a clean state — logged out or in a fresh session — unless you're deliberately studying the personalized case. Location matters for anything local, so hold it fixed or test each market on purpose. Write down the model and mode you used, because a rate from one model doesn't transfer to another.
Space the tests out. Ten runs back-to-back in one minute can share transient conditions. Spreading them across sessions, and re-checking over days, separates a stable position from a momentary one. Volatility over time is itself a finding — a rate that swings week to week tells you your position is fragile.
Define "appeared" before you start. Decide up front what counts: named in the answer, named in the top few, cited as a source, described accurately? Being mentioned once at the bottom is not the same as being the lead recommendation. Score the same way every time, or your rate is measuring your mood, not the model.
Reading the number without fooling yourself
When you've got a rate, resist the urge to over-read small differences. If you appear 6 out of 10 times this week and 7 out of 10 next week, that is almost certainly noise, not progress — the intervals around those two numbers overlap heavily. Real movement, at ten runs, looks like going from 2-of-10 to 7-of-10, not a one-test wobble. When a change is small and you need to know if it's real, the answer is more runs, not a bolder conclusion.
The same caution applies to competitor comparisons. If you're named 70% of the time and a rival 60%, don't declare you've won the category off ten runs each — that gap is within the noise. Treat close rates as effectively tied and either test more or focus on gaps big enough to survive the uncertainty.
A simple protocol you can actually run
Here's the lightweight version I'd hand to a team that wants trustworthy numbers without a research budget. Pick your prompt set — the real questions customers ask, in a few natural phrasings each. For each prompt, run it ten times, in clean sessions, spaced out rather than all at once, with model and location held constant and written down. Score each run against a fixed definition of "appeared." Record the rate as "named X of 10," not as a pass or fail. Re-run the whole set on a regular cadence so you're watching a trend line, not a snapshot. When a specific prompt is high-stakes, bump it to 20 or 30 runs before you make a claim about it.
That's it. It's not glamorous and it won't give you the false comfort of a single green checkmark. But it gives you something far more useful: a defensible number you can track, compare, and act on — and the discipline to not celebrate or panic over noise.
Measure like it's a rate, because it is
The uncomfortable truth of AI visibility is that it isn't a state you're in or out of — it's a probability, and probabilities only reveal themselves through repetition. One test is a rumor. Ten tests, scored consistently and tracked over time, is a measurement. Thirty is close to proof for the prompt that matters.
My honest read is that the teams who win at this won't be the ones with the best single screenshot of ChatGPT naming them. They'll be the ones who measured their true appearance rate, watched it move as they built corroboration and authority, and knew the difference between real progress and a lucky roll. Test enough to earn the number. Then trust it.
Key takeaways
- AI answers are non-deterministic, so a single test measures almost nothing — you'll be confidently wrong roughly half the time on a 50/50 position.
- Stop asking 'am I recommended?' and start asking 'how often, out of how many tries?' Visibility is a rate, not a verdict.
- Ten runs per prompt is a solid working baseline; bump high-stakes prompts to 20–30 to narrow the uncertainty enough to claim a few-point change.
- Precision improves with the square root of trials, so returns diminish fast — past a point, test more prompts rather than hammering one prompt hundreds of times.
- Hold the wording, model, location and 'appeared' definition constant and space runs out over time; volatility across weeks is itself a finding.
- Don't over-read small gaps — 6/10 vs 7/10, or you at 70% vs a rival at 60%, is almost certainly noise; treat close rates as tied or test more.
Frequently asked questions
About the author
Scott Tischler is the Founder & Chairman of AIrecommend.ai and a practitioner-authority on AI search and Answer Engine Optimization. With 20+ years in marketing technology — including American Express, MetLife, and UBS — and executive and professional study at Wharton, Harvard, and Oxford, he helps businesses become the ones AI recommends.
Want to be the business AI recommends?
See how AIrecommend.ai builds the entity authority answer engines reward.
Explore AIrecommend.ai